3.14. Experiment Factory

Have you ever thought why some people are more successful than others and how they were able to achieve what they have? Maybe you’ve wished to become invisible and sneak in to a place to find out the secret of what actions to take to become successful? Or maybe you’ve wished to have a superpower to be able to read minds and get to know one’s way of thinking and how the most important decisions are made. If you’ve answered ‘yes’ to at least one of those questions, then keep on reading as today we’re going to open the curtain on actions to take for the magic to happen!

Let’s Learn From the Best!

Skyeng is an international EdTech company, an English language academy, and the largest educational software developer in Europe. Not only that, it’s also one of the apt examples of how to structure your work, which actions to take, and how to use resources to achieve goals and hack the grow of the company’s success. Skyeng developed and presented a detailed scheme of what they are doing in order to introduce (or not) a new feature to a product. Let’s learn from them!

From the Start to the End

Everything starts from an idea. Accordingly, there are different sources of ideas — sources for hypotheses. One of the sources is customer development, when you are provided deep quality data. What you do, is:
1a. Search for and find customer segments that are interested in your product or services;
1b. Conduct interviews and ask customers directly or indirectly using open projective questions about their problem;
1c. Process the provided data, aggregate feedback received from the customers and identify recurring insights.
There are other ways where ideas come from:
1d. Team ideas. Nobody knows the product better than the team that has developed it. Thus, it a great source of hypotheses;
1e. Feedback from the existing customers and users. It is a wise thought to have a separate chat with your most loyal customers who are going to provide you with their experiences with your product or services, on the basis of which you can generate hypotheses;
1f. Analysts’ insights. Product analysts monitor how the product is used by people, analyse findings and in case they find some anomalies in the use, provide valuable data for further improvement.
Having gathered all the information, the next step is to:
2. Use the identified recurring insights to create a job story, a description of what you are going to do, on the basis of which a solution will be created.
3. Create an MVP solution hypothesis to complete the job story. MVP must be developed quickly, without putting in a lot of efforts, and it has to be cheap. It must provide clear-cut answers whether a solution will be solved.
Afterwards, it is necessary to work with the responses and reactions of people to the MVP.
4. Funnel testing on product sync is the analysis of users’ behaviour related to the product or service throughout the whole AAARRR funnel.
5. Scoring. All data and information are counted and go through prioritisation in order to get a significant result.
6. Finally, the job story, a hypothesis, is placed into the best hypotheses backlog to be improved, reviewed, further developed, and/or brought into action.
7. Analysts take a hypothesis from the backlog to carry out some further research. They check whether there is any historical data and go through various sources to find out if the hypothesis was already run by some other companies or businesspersons. Depending on whether they find any data, there are two possible ways.
8a. If analysts found historical data, then they search for correlations and define casuality.
8aa. Having found the necessary information, the question is if there is proof and verification. If it’s possible to verify, then continue to step 13. If there’s no verification — you must kill the feature and reject the hypothesis.
As a matter of fact, the aim of the whole scheme is to kill the feature or a hypothesis at an early as possible stage so that you don’t spend additional time, money and/or resources on something that in the end will have to go the recycle bin.

8b. If analysts did not find historical data, then it’s necessary to check if it’s possible to carry out a regression testing. The mentioned test will help to understand whether any changes will have any effects on the metrics.

9a. If it’s possible to implement a regression testing, then we carry out a regression experiment and collect all the necessary data.
9aa. All the data collected in step 9a should be used to understand if there are any changes that may affect the metrics. For example, you think that people don’t use your Website Builder because you have only 200 templates. You believe that if you had more, people would definitely use your product. But before spending time on creating new templates, why not to worsen the situation and see what happens then. Thus, instead of creating 200 new templates, you delete your existing ones and leave only 3. What’s the result? Nothing changed. The conversion with 3 templates is the same as with 200 templates. There is no need to create more, as it will be a waste of time and efforts.
9aaNO. If there are no changes — you must kill the feature and reject the hypothesis.
9aaYES. If there are changes — continue to step 9b.

9b. If it’s not possible to implement a regression testing, then we can start developing an MVP project.

10. With the MVP project we’ve developed, we create a scenario for checking information with the users.
11. Having created a scenario, it’s necessary to validate a solution. Will your MVP solution solve the users’ problems?
12. Accordingly, was the solution validated? Did the product/feature/design indeed solve the problem that was initially intended to be solved?
12a. If the initial solution was not validated, then it is necessary to check if there’s another solution. If there are no more solutions, then you must kill the feature and reject the hypothesis. If another solution does exist, then you must return to step 3 and start all over again.
12b. If the initial solution was validated, then continue to 13.

13. Regardless of the fact how you got here, you’ve already carried out a lot of work. Now it’s time to draw layouts.
14. Once the layouts are drawn, it’s necessary to check if they are suitable for UX-testing.
14a. If layouts are suitable for UX-testing, then you need to develop a prototype, create a scenario, and carry out an UX-test.
14aa. Depending on the results, you need to understand whether the solution passed the UX-test.
14aaYES. If the solution passed the test, then go to step 15.
14aaNO. If the solution did not pass the test, then check whether there’s another solution. If there are no more solutions, then you must kill the feature and reject the hypothesis. If another solution does exist, then you must return to step 13 and start all over again.
14b. If layouts are not suitable for UX-testing, go to step 15.

15. The front-end development. On this step you need to develop the graphical interface of the website so that the users can interact and try out your product. Having developed, it’s time for front-end testing and implementation of it to get significant results.
16. On this step you need to test functionality. Does your product do what it is supposed to do? Do all its components and functions work as they are supposed to work?
17. Now the time has come to start a split test. As a matter of fact, split testing is considered to be the most expensive element of hypothesis verification. Accordingly, it’s better to kill the features or a hypothesis during earlier stages and carry out split testing for those that you really believe in. Having carried out the split test, sum up its results and continue to step 18.
18. So, the question is if the solution took off?
You’ve spent a lot of time, efforts and resources on understanding whether your hypothesis is right and the new feature that you’ve decided to develop will have a major effect on your business. Don’t stop here! You’re almost there!

18a. If the solution did not take off, then you need to observe user sessions in the WebVisor, prepare a task for CustDev, create a scenario and conduct interviews to get as much information as possible to understand what you have done wrong. If you don’t have the slightest idea of what is not right, then you must kill the feature and reject the hypothesis. If you’ve got some insights and realized what was the matter, then return to step 3 and start all over again. Don’t get upset, at least now you know and you probably learned some valuable lessons in the process.

18b. If the solution took off, then you delete the split test, present your development to the audience, and, finally, put the feature in production! Congrats! You made it to the end!

Final Words

So, you’ve seen that success doesn’t fall from the sky and is not something easily achieved. It is the result of hard work, persistent effort, and strong spirit not to give up; it’s the result of numerous analyses, testing, and experiments. Indeed, if you want to hack the growth of your own company or business, then this should be something you also need to do, and instead of using your intuition or advice or anything else — analyse! test! and experiment!