The best way to validate a value proposition is by testing if the solution delivers the value the customer is looking for. To test your value proposition, you need a way to deliver the same value to the customer as the final product will deliver. So you need an experiment that mimics the value of the eventual product. 

As a startup, you have a limited amount of resources. So, you want to maximize the number of learnings, for the minimum amount of resources. That is why we use “The Wizard of Oz prototyping” and “Concierge Testing” experiments. These two methods help you get insights into the needed value proposition without building the entire solution. The Wizard of Oz and Concierge experiments have a lot of similarities, which makes it hard to have a clear distinction between the two methods. In this article, we will discuss the differences, and when you should use what method. But first, we will describe both experiments separately.

Concierge testing

With the Concierge experiment, product features are being imitated. The tasks to imitate those features are being executed by a human. The person who takes the role of a concierge provides the solution for the customer by executing the tasks manually. An important part of this method is that the customer is fully aware that the features are being mimicked by a person. 

The concierge provides the entire solution, which in some circumstances betters the customer experience in comparison to a (technological) product. Human interaction can bring extra perceived value to the solving of a customer’s problem. This is because a person that has empathy for your problems, feels more trustworthy than an automated system that has no feelings at all. Plus, the concierge provides the whole solution without any complications for the customer. While technology will always require a certain understanding to use correctly, which lowers the customer experience. The value proposition provided by the concierge can be significantly higher than what the automated version would ultimately be able to bring up. As a consequence, the result of the experiment can be misinterpreted quite easily. This should be taken into consideration when picking a method and setting up learnings for the experiment.

Wizard of Oz prototyping

The essence of the Wizard of Oz experiment is to get the answer on: Does my solution bring enough value for the customer? The method is named after the movie The Wizard of Oz, In which a man pretends to be a powerful wizard. In reality, he is pulling levers behind a curtain. The idea of the method is to create a simple solution and mimic the features of your future product. This is similar to Concierge testing. But in this case, the customer perceives the solution as an already working, automated system. In reality, it is a person imitating the product’s features by executing tasks in the backend. Only the features which are necessary to make the customer perceive that it is a working technology will be built. 

To validate the value proposition correctly with this method, the delivered value proposition must come as close to the real value proposition as possible, only the delivery time might be delayed. Using the Wizard of Oz prototyping method will allow you to test your value proposition while saving a lot of time and resources. 

When should I use what method?

Now that you understand both methods, it is time to distinguish one from the other. 

Because using a method with the wrong intentions could lead to a false positive or a false negative, which can cost a lot of time and money. If you have a clear value proposition, and you want to validate or invalidate it, then The Wizard of Oz is the method you need. With the Wizard of Oz prototyping, you can come as close to the real value proposition as possible, which makes it the perfect method to test your defined value propositions. The Wizard of Oz should be used to answer the question: 

• Does my value proposition solve the problem of the customer? 
• What interactions does our customer need to make with our product?
• Do the features that we came up with help solve our customer’s problem? 
• Does our product create value for our customers? 
• Is our product a must-have/must-use for our customers? 
• Are customers willing to pay for our solution? 

If you’re not sure how to shape your value proposition yet, or you want to get more insights into the customer problem before defining a value proposition, Concierge Testing is your go-to method. It purely focuses on what the customer problem looks like, and how it could be solved. The personal interaction makes asking for feedback easier, which is essential for learning from your experiments. But, as mentioned before, it is important to be aware that the extra customer value that comes with personal interaction, can be misleading. 

Concierge Testing can be used to answer the following questions:

• Does the solution solve a real customer problem?
• How can the problem be solved?
• What is the minimum feature set required to implement a solution?
• What are the greatest problems in effecting a solution for the customer?

Conclusion

The Wizard of Oz prototyping and Concierge Testing are two experimenting methods to test solutions for customer problems. Both methods involve human interaction which makes them seem similar, but they’re not. The difference is the fact that with Concierge Testing, the customer knows that a human is providing the solution. While with The Wizard of Oz prototyping, the customer thinks it’s a working technology that is providing the solution. 

The two methods can not be used interchangeably. But, they are a good addition to each other. Concierge testing can get more insights into the customer problem and how to solve it. Once you got enough insides to create a value proposition, you can test it with Wizard of Oz prototyping.

If you want to read more about Concierge Testing vs Wizard of Oz prototyping, we suggest you read: Concierge Testing vs. Wizard of Oz prototyping from Tristan Kromer. He wrote an article on the same topic. In the article, he used some great practical examples, which helps to get a better understanding of the differences between the two experimenting methods. If you want to learn more about experimenting to validate assumptions, we suggest you read: Validated Learning: The ‘Build, Measure, Learn’ loop