Growth experiments: Testing growth hypotheses at low traffic volumes

Product thinking, User researches, Intentionaly

Meta is a mental-health service. It is a marketplace where clients find psytherapists. Smart algorithm allows our service to offer specialists who will best cope with a request. The client then selects an appropriate specialist and arranges a meeting.

In the fall of 2021, we worked on the stage of conversion rate from “Complete the survey” to “Chose a psychotherapist”.

After completing the survey, the client would get to the page of the most suitable specialist.

60% of users who completed the survey don’t choose a psychotherapist

We started by studying the problem — why clients who completed the survey don’t choose a psychotherapist? Together with the product manager, we studied 40 hours of screen recordings, conducted 30 problem understanding interviews and several ux sessions with users. Here’s what we learned:

• After the selection, clients see one open profile; some don’t realise that they can switch to other therapists;
• It is not clear why some specialists are more suitable than others;
• Three more therapists would be enough to choose from;
• In a list of 30+ therapists, it is difficult to choose the one, who is most suitable; users face choice paralysis;
• There is no option to view and change your request, only to create a new one;
• 88% of users don’t understand whether the therapist will really help to cope with their problem;
• Clients don’t understand the difference between therapists;
• There is insufficient descriptive information about the therapist. this information does not give the understanding of what the therapist’s speech is like. As we have found, speech is a very important criterion when making a choice — sometimes there is a desire to see a video or make a short free call;
• Texts written by therapists themselves are of poor quality: there are punctuation and spelling errors, lack of structure, a lot of redundant information;
• Clients don’t understand psychotherapeutic terms.

User interviews — my favourite part of research

88% of users don’t understand whether the therapist will help to cope with their problem

We formulated several hypotheses on how we can help users:

• If we show the client a list of therapists instead of an open profile, it will remove the misunderstanding that there is only one therapist to choose from;
• If we show the client a limited list of therapists (more than 3, but less than 10) this will not lead to paralysis of choice, but it will give the opportunity to have a choice;
• If each therapist adds a video or audio story about themselves, it will help the client to get to know the therapist better;
• By explaining to the user why we show these particular therapists only, will remove the choice barrier;
• Making the “see all” block more visible will help more clients move on to the list of all therapists in the event, when they were unable to choose from those offered;
• If you give a client to change his or her request, the client will be able to choose a therapist from another category (for example — price range), in case when he could not choose from those offered;
• If you help each therapist make a high-quality structured text about their experience, it will help a client to compare therapists and make the right choice;
• ...

We evaluated the hypotheses according to the criteria “Development time” / “Expected impact on conversion”, and we have prioritized the order in which the experiments shall be carried out.

First, we decided on the feature, which we call the “Top 5”. After completing the survey, a client sees a list of 5 most suitable specialists.

When the prototyping was completed, we conducted 8 interviews with users: we compared the previous and new solutions Side-by-Side to make sure that users see a qualitative improvement in the selection process.

Having received good reviews from users, we launched a feature in production. Later, we conducted experiments for two months on how and which list is best to be shown to customers:

• List of the 5 most suitable therapists;
• List of the 5 most suitable therapists + the opportunity to see all suitable specialists;
• List of the 7 most suitable therapists + the opportunity to see all suitable specialists;
• List of the 5 most suitable therapists + the opportunity to see all specialists in our database;
• List of the 5 most suitable therapists + the opportunity to see all suitable specialists + the opportunity to change your request.

According to the results of the experiment, we settled on the design version, which showed the best conversion results. We show clients the 5 most appropriate therapists with the opportunity to see all the right ones. The experiment helped us raise the conversion in that part of the funnel by 6 percentage points.

A surge in conversion was noted at the end of february, which was caused by the outbreak of war in ukraine

With little traffic, conversion fluctuates — it shows irregular behavior. Under these conditions, it is difficult to conduct experiments on the change in conversion: it is not always clear what affects growth and fall — whether its a product solution or fluctuation.