Software development has evolved tremendously over the last decade. At companies like Facebook, every engineer is empowered to be able to run live experiments. According to Mark Zuckerberg, “we are going to learn a lot more by experimenting and letting teams explore the things that are worth exploring than by having a heavy hand in that.” This approach to learning and risk-taking is rapidly transforming product organisations across tech companies. Techniques that were previously regarded as a niche, such as experimentation, feature management, and controlled rollouts, are nowadays seen as indispensable parts of the product development toolkit.

However, many organisations still struggle to keep up with these innovations. Like any great trend, the benefits are easily lost if companies do not have a plan to manage key pitfalls. There are six principles for product development that all enterprises should adhere to build a culture of experimentation.

>See also: Poland’s Gliwice emerges as hotspot for software development 

Measure more user behaviours

Experimentation is one of the most powerful levers for moving your metrics; however, most companies analyse too few metrics to detect improvements. In Optimizely’s Global Optimisation Benchmark of over 100,000 experiments, two-thirds of all experiments only measure three metrics, with the most common being revenue and conversion rate. This is similar to a football manager measuring all team players by only goals and assists when most positions should focus on different objectives.

Great experimentation requires understanding how responsibilities differ across areas of your products and balancing them to reach your overall goals. The survey also revealed that a sixth of experiments had at least seven goals, and were twice as likely to detect improvements as a result. Even if these experiments didn’t improve the most important metric, they showed an improvement in user behaviour that teams could learn from and continue to build on. Teams with a narrow metrics focus can fail to detect the small victories, miss valuable learning opportunities, and lose motivation.

>See also: How CTOs can introduce blockchain to the enterprise

Use experimentation as a compass instead of a speedometer

For many companies who are earlier in their experimentation maturity, their product direction is already decided, and experiments happen after that roadmap. Experimentation acts as a speedometer, telling teams how fast they are going, instead of a compass, identifying the direction that is best. Before you run an experiment, ask yourself, what will you do if the experiment loses or is insignificant? Will this feedback help you understand how to serve your customers better? Great product teams use a number of techniques to test their riskiest assumptions or the value of their prototypes before deciding on a large up-front investment that is hard to turn away from later.

>See also: How to remove malware

Develop noticeable and meaningful improvements

In statistics, the smaller your sample, the larger the improvement you need to detect a significant change. Simply put, if you are testing on low traffic areas of your product, you need to make bold changes that users will react to. A large company like Google has the traffic to detect the optimal shade of blue out of forty variants. But a B2B company must test meaningful changes that users will notice to be measurable. If the area of the product you’re testing on gets fewer than 30,000 visits a month, you will need much bolder ideas to effect detectable changes in user behaviour.

Teach teams and stakeholders how to work with statistics

The only thing worse than not running experiments is running experiments that are misinterpreted. There are several ways in which companies misunderstand the statistics behind experiments. Firstly, companies are overly reactive to early returns. Early on during experiments there are few conversions and experiment results swing wildly. When teams “peak early” at results, they frequently overvalue the data and end experiments prematurely. It is very common for the direction of a metric to swing over the course of an experiment and teams that do not have the patience to wait are at the mercy of random chance.

>See also: How to choose a managed service provider

Secondly, stakeholders often create arbitrary pressures and deadlines to get answers early. In many business processes, management can improve productivity by introducing pressure and deadlines for teams. However, in the realm of science, this behaviour causes the opposite of the intended effect. Ordering teams to give results of by a certain date often causes teams to interpret insignificant data through gut feel. While these decisions can feel scientific to executives, they are all too often incorrect and give a false certainty about the wrong direction. Great experimentation teams run many experiments concurrently and know to wait before interpreting results.

Question your results

Twyman’s Law should always be your reference point. If your scientific results appear extreme or out of the ordinary, they are likely to be wrong. Results that seem too good to be true should be analysed and retested. If a particular metric appears to move for no reason, investigate it. For example, if you have an experiment with ten goals and 90% significance, odds are at least one of these goals will be a false positive. To minimise these risks, you should make sure your experimentation platform uses modern approaches like sequential testing, false discovery rate control, and outlier filtering.

Don’t work in a silo

As with every other aspect of a business, you cannot afford to keep experimentation siloed. The data and insights gathered can be useful to any number of teams across an organisation so organisations must centralise data and reporting to allow for scalable impact. Insights should be shared with the broader company, not just a single team. Additionally, great ideas need input from other areas of the business. At many companies, the customer service teams and frontline employees have tremendous insights into the most common problems users face. Not involving them in ideation can cause the most urgent improvement areas to be missed while teams invest effort into low-yield ideas. These silos will grow naturally in many organisations, especially if there is a risk-averse culture that penalises ‘failure.’ As Microsoft CEO Satya Nadella says, “how does one achieve success? It’s through a series of failures, a series of hypothesis testing. That’s in some sense the real pursuit.”

>See also: How to secure legacy systems and protect against IoT attacks

Experimentation has begun to hit the mainstream in the last few years, as an important and effective way for businesses to improve returns and manage the risks of radical ideas. As Nadella says, “our industry does not value tradition – it values innovation.” For businesses to reap the rewards of innovation, they need to build a culture that learns from failure, measures improvements scientifically, and empowers all areas of the organisation to contribute transformative ideas.

By Hazjier Pourkhalkhali, Global Head of Strategy of Optimizely