Testing is an integral part of developing any digital product, but when it comes to AI features, the complexity level kicks up a notch. In the realm of AI, where outputs aren't always predictable and user interactions can vary widely, traditional testing methods often fall short.

Luckily, in this case, there is a magical solution. It’s called Wizard of Oz testing. 

Testing AI is Hard

Testing digital products with users is hard. There’s a whole realm of User Research dedicated to it (don’t get us started!). But testing AI features is even harder. It adds other layers of complexity to the problem: 

• Lack of control: The lack of control over AI model outputs adds uncertainty to the testing process, making it difficult to create and evaluate intricate user flows.
• The right data: achieving realistic tests requires not only the right AI capabilities but also a proper integration into user workflows and, of course, the availability of relevant data.
• Slow & expensive: while the process of building AI has become more accessible, the development of new features still demands time and resources before they can undergo thorough testing.

A Tasty Example 

Imagine we're developing a recipe scanner feature that adds ingredients to your shopping list. To effectively test this feature, we need to ensure several things: 

• that the ingredient extraction works accurately across various recipes, 
• that the ingredient database is comprehensive and searchable, 
• and that the system can propose alternative ingredients if needed. 

However, building such a feature from scratch just for testing purposes is neither practical nor efficient. Remember, we first want to test the appetite for this feature (sorry not sorry).

The Genius of the Wizard

Enter the Wizard of Oz approach. Much like the iconic character, this method involves creating the illusion of AI functionality without actually relying on complex algorithms.

Instead, a human operator behind the scenes simulates the AI's responses or actions. By doing so, testers gain the flexibility to assess user interactions realistically, without being constrained by the limitations of AI models.

Back to the Recipe

In applying Wizard of Oz testing to our recipe scanning feature, we uncovered much-needed insights. Firstly, even minor errors in ingredient extraction rendered the feature virtually unusable for most users. This allowed us to identify and mitigate feasibility risks early in the development process. 

Secondly, by isolating the AI component, we were able to uncover additional UX flaws that might have otherwise gone unnoticed. Addressing these issues ultimately resulted in a smoother and more intuitive user experience.

Let the magic unfold

In AI-driven technology, traditional testing approaches may not always suffice. By embracing methods like Wizard of Oz testing, developers & user researchers can gain deeper insights into user interactions, refine their products more effectively, and ultimately deliver a superior user experience. 

So, the next time you're faced with testing an AI feature, consider channelling your inner wizard. And let the magic unfold.