Table of Contents
Identify the Source of Failure
In order to learn from failure, you must first recognize that you did indeed fail. However, personal failure or team failure is easy to gloss over. There are plenty of excuses you can tell yourself to protect your—or your team’s—ego. Maybe the competition was tough, or the market was saturated, or a vendor did a poor job. The reality can be much more difficult to accept. Perhaps you made something confusing, or didn’t test an important feature, or marketing missed, or you didn’t focus enough on sales. But how can you remove the filter and see the failure for what it truly is?
Review Your Metrics
The first step in identifying the source of failure is to review whatever metrics indicate failure. For example, did you expect users to purchase more items from your store than they actually did on average? What was your expectation for the metric? Was anything put in place to drive improvement of that metric? What was done? Could it have been done better? Would some other change—even if it is hard to implement—move your metrics closer toward what you expect?
Consider the Motivating Factors of Metrics
Contrary to what you might expect, business metrics are not the summation of your business. The data doesn’t lie, but it doesn’t tell the whole story. What you measure is important. As Eli Goldratt—the originator of the Theory of Constraints—said, “tell me how you measure me and I will tell you how I will behave.” In other words, an illogical measurement may result in illogical behavior. Consider the ways your end users (or even your team) might abuse a metric. If your goal is to sell N products per user session, might your team find a way to designate add-on items or accessories as full-fledged products? Behavior like this in a metrics-driven company is common. Just look at the Wells Fargo account scandal: bankers were highly motivated to open more accounts, even if they were not what customers wanted.
How Do You Find the Right Metrics?
This isn’t to say metrics are inherently bad. Stealing a page from Michael Pollan’s take on food, here is my advice on metrics: Measure results. Not too many. All in alignment. In other words, ensure you are measuring only the results that truly matter to your business. If you give a team metrics to hit, they will likely try to hit them at all costs. Setting the incorrect metrics—say, a metric for user session time, which doesn’t correlate directly with making a purchase—will lead people to focus on optimizing the wrong thing. Additionally, setting too many metrics can lead to confusion or competition when deciding on what to optimize next. Finally, ensure all of your goals are in alignment. Have just enough metrics in place to identify and learn from failure.
The knee-jerk reaction managers love to give is “of course all of our goals are in alignment! We’re all on the same team!” That is not necessary true. Especially as companies grow, there are many competing factors between departments or teams that organically occur as a result of people trying to optimize the metrics they (1) care about most and (2) can influence the most. Therefore, don’t accept your own hand-waving at questions of alignment or conflicting goals. Map it out. See if you can draw logical relationships between the goals, objectives, and metrics your team has established. If you can’t, go back to the drawing board.
Utilize the Theory of Constraints
Speaking of logically mapping out relationships, consider adopting tools from the Theory of Constraints (TOC). TOC is a management paradigm you can apply to how you think through problems. Its key hypothesis is that every system typically has one constraint, and at most a few constraints. These tools help you identify the objectives needed to reach your goals. In turn, it also helps you identify metrics to measure your progress toward goals. These tools are simple, too. You can distill much of the constraint talk down to the Five Whys, a tool for finding the causal relationships in problem symptoms and sources of failure.
Once you identify a problem in your metrics, start asking yourself why things are the way they are. Try to avoid the inevitable existential crisis that follows. Then use some of the data points below to analyze the failures. Build a cause-and-effect narrative using these tools: “If, according to Google Analytics, the average user drops off after clicking 3 links, and it takes 5 clicks to reach a payment page, then most users will not reach the payment page.” That contrived example may sound simple, but obvious insights like that are difficult to notice if you don’t spend time analyzing what went wrong. In hindsight, things may appear obvious. But it takes some reflection to reach such obvious conclusions.
Analyze Points of Failure
User Analytics
If you believe your problem lies in the path of converting visitors into users, free users into paid users, or user attribution, review your analytics. Mobile and web analytics describe the behavior of users. Analytics show you where users spend their time in your app. It also shows you what they avoid, when they user your product, and how they use your product. For example, Google Analytics has a behavior flow report to visualize the path users take in your app or website. Behavior flows are useful in reviewing your conversion funnel. That is, you can see the percentage of users who drop off at any stage as you lead them to your end goal.
Bug Reports
If your problem is technical in nature, review your bug reports and crash logs. A good bug reporting tool like Critic gives you actionable information about what happened. This includes 8 data points to help diagnose software problems. By reviewing device statistics, application logs, user comments, and app- or user-specific data, you can pinpoint what went wrong and why.
Customer Surveys
Outside of technical issues, it’s best to directly ask your customers or end users where things went wrong. Surveys conducted through your application, via email, or through your website can give your users an opportunity to explicitly say what they like and don’t like about your product. However, surveys are a one-size-fits-all feedback mechanism. While they are a great way to collect feedback from a multitude of users, the dialog is one-sided.
Customer Interviews
The highest quality feedback on your product will come from customer interviews. As opposed to surveys, interviews are interactive. You are engaging a customer—typically one-on-one—in a face-to-face meeting or phone call. Though you will still start by asking prompting questions, customers may quickly deviate and discuss what they really care about. Your questions about the user experience may be met with responses like “the UX is great, but the app doesn’t actually make my life better.” Those types of interactions lead to more valuable insight than a survey that simply asks “On a scale of 1-10, how much do you like the UX?”
Regardless of the feedback mechanism, you will need to compile results and analyze them for common trends, insights, and unmet needs. What do users say they really care about? Do their words (responses) match their actions (analytics)? Use this information to validate or invalidate your theories about what went wrong.
Respond to Failure
Learn from Failure
Once you’ve determined where the failure truly lies, it’s time to learn from your failure. Failure is often glorified in tech startups because there is so much to learn when your ideas are invalidated. Certainly no one wants to fail, but a botched product launch, software update, or public relations issue is valuable. These experiences give you data points to look at and behaviors—that of you, your team, and your users—to reflect upon for future improvement.
Using the previously mentioned tools, including those from the Theory of Constraints, determine what could be done differently. Describe your desired future reality and the actions that must be taken to attain that reality. While TOC future reality trees can look complicated, the process is simple. Write down your desired end results (e.g., “increase average lifetime value of customers”). Then examine what you have in place right now. What behaviors create your current, lesser result? Write those down and connect the dots logically. “If X occurs, then Y occurs.” Finally, ask yourself what happens if you replace or augment some of those behaviors (“if we present a today-only discount on accessories at checkout, more users will purchase accessories”).
Once you’ve identified what to change, you can begin implementing changes.
Create a Plan
After you learn from failure, form your response plan. This should be outlined in your desired future reality described above. There are specific actions or behaviors you want to promote to get a better result. Write down a concrete plan—detailed steps to take, who will take those steps, and what results they should expect—to implement your changes. Ensure that your team understands the changes and why they are important. Without the “why”, people will be more likely to criticize change. But if you can rally the team around your vision of the future, they will be more likely to give your new plan a chance. Otherwise, expect team members to blindly execute orders without critically evaluating the results. You want team members to be bought in to the desired future reality so they will try to adapt if the plan doesn’t create the results you are all striving for.
Establish Metrics for Success
To determine whether the plan is working, you need to set metrics to monitor. Metrics should not be an end goal. That is, metrics indicate whether you are trending in the right direction. If you establish a metric as the goal—for example, setting a minimum annual sales target—people will work hard to meet that minimum and then stop. It’s not that they’re lazy or want to do a poor job. Meeting the metric is just an indicator that they have done their job.
But for most businesses, meeting a minimum is not the real goal. If you hit $10MM in monthly recurring revenue, wouldn’t $11MM be even nicer? Using metrics as goals will dampen the growth potential of your business and your team. Instead, teach them to strive for something better than what they’re already doing. Proper metrics encourage this professional growth and business growth. “Congratulations on hitting $10MM! That’s the new baseline. Let’s see if we can do even better.”
I’ve seen firsthand the pitfalls in establishing a metric as a goal. I once wanted to set certain code quality standards on a team to prevent defective work from getting out the door. Therefore, I enforced a minimum test coverage number of 80% coverage. 80% felt like a safe number for the average project, even though I knew some critical (life-and-death) projects would require higher test coverage. Would you like to guess where almost every landed? Certainly, 80% test coverage was not good enough for some projects, but most people that hit that minimum number felt like they had done their job. After all, why should they put in more work after their success metric is satisfied? Avoid this undesirable behavior by looking at trends instead of goal numbers. Focus on ever-improving results instead of a minimum desirable result.
Follow Through
Monitor your metrics at least weekly. Expect that metrics won’t always indicate a positive trend; everything has its ups and downs. If you monitor metrics too frequently, you may find yourself optimizing non-problems. Asking why site visits decrease one Monday isn’t that helpful. However, asking why site visits decrease over a week-long or month-long period is useful. Perhaps you will find some seasonality in site traffic, or will identify an overall downward trend that needs correcting.
If you don’t monitor progress toward your desired future state, neither will your team. Only by demonstrating that you are focused on your goals—in your everyday words and actions—will your team focus on your goals. Otherwise, they will not believe you when you say the goals are important. If they’re so important, why aren’t they worth your time as well?
Be Ready to Adjust
Finally, be prepared to change as the situation changes. You may recover from one failure beautifully, improving metrics that ensure you won’t experience a repeat of the same failure. However, as TOC teaches us, constraints in a system move. Just because you addressed one failure does not mean there are no failures. There is always a constraint preventing you from attaining more throughput. Throughput, for example, may be number of products shipped, tickets resolved, or revenue acquired. So even when failure seems like a thing of the past, take a moment and ask yourself how your business could be better.
Need Help?
If you would like to bounce ideas off of someone else to resolve a failure, I’m always happy to talk to you. Also, please try Critic for customer feedback and bug reports. Then users can tell you what problems they are experiencing. Critic bug reports include data that can help you diagnose and learn from failure.