International DevOps Certification Academy™
How Do You Improve Your Hypotheses With DevOps And Empower Your Experiment And Learning-Driven DevOps Organization?
In typical IT organizations you used to build, most likely you are still building multiple releases of products and product features without validating if the desired business outcomes from these products and features are fulfilled or even if they are being used by some living human beings at all. The most inefficient way to test a business model or a product and feature idea is that: You build a complete software product and service to see whether the predicted business demand already exists and your idea can meet this demand.
Before you build a full-blown product and service you need to ask: “Why should I build this?”, “Is this really worth of my time and resources?”. You need to create and run fastest and cheapest experiments possible to validate if your ideas, products and features meet your desired business outcomes.
Long time statistical experiments and measurements with various enterprise product suits have shown that: Only 1/3 of all performed changes can considerably improve a key business metric such as revenues, conversions, orders, subscriptions, new customer acquisition and so on. In other words 2/3 of changes make from no to negligible improvement or they make results of the metric even worse. For these 2/3 of changes, your organization would be better off giving entire team a vacation, instead of building these non-value adding features.
Your counter measure to overcome this challenge is user research by conducting A/B testing. You test results of your features before you build them. You integrate feedback from user research to your software engineering process to make sure that you are not only correctly building with DevOps methodology, but also you are building the correct thing. The faster you can experiment, learn and integrate client feedback into your software engineering life cycle, the better ability we will have to outperform your competitors in your particular market.
A/B testing technique became first popular with direct response marketing via postal mail. By changing wording, color schema, design layout, headline, copy text and so on, marketers have been testing numerous variations of flyers and post cards in order to find out the versions which perform and sell more than others. British government has been doing A/B testing to identify the best performing letter to collect overdue tax revenue from its citizen. A/B testing is nowadays a mainstream method not only to test marketing and communication elements, but also to identify what ideas, products and features work and what don’t.
In order to adopt A/B testing in your DevOps team, you need to be able to quickly and easily deploy multiple versions of product features and present them to various different user segments. With your comprehensive telemetry and business layer metrics you identify if the feature produces better business results. If yes, which version of this feature performs best.
An example: Your DevOps organization wants to release a new checkout flow for an e-commerce portal. Before this new checkout flow becomes a major new work for your DevOps team, it is only a hypothesis for better business results. Only after this hypothesis is investigated and validated, your new checkout flow can turn into a real project which will consume significant time, resource and energy from your team. The first work package of your DevOps team with this checkout flow is:
Validate Hypothesis: We Believe that a new version of checkout flow, Can Result In improved conversion rates. We Will Be Confident to fully build and deliver this feature when we see that one of experimental check-out flow versions presented to 1% of visitors increases sales at least 5% during next 7 days.
CONCLUSION
Instead of merely relying on your gut feeling and best practices you and your DevOps team have been learning and observing so far, the focus must be getting real people in the real world to really perform in your experiments.
DevOps Product Owners should see their product and feature ideas as hypothesises to be validated. These hypothesises can be comprehensively designed, built and tested only after they are rationally proven to be good ideas.
DEVOPS-CERTIFICATION.ORG
