Wie is de Mol? / AVROTROS

Major challenges

The first testing challenge for Polteq lay in the high number of expected users. Directly after launching the app, and before the first television broadcast, many people create their account and invite friends to play the suspicion game together. Therefore, the new app had to handle a huge peak load right from start.

A second challenge was that the course of the game show was unknown during app development.  As a result, all possible scenarios had to be tested both in the app and in the backend.

The third major challenge was the “first time right” principle: the app has only one release at the start of the season. After release, everything had to run flawlessly for millions of users right away.

Finally, the app also had to provide real-time data, such as statistics for the national mole suspicions on behalf of the talk show Moltalk.

Polteq is proud of its flawlessly running app, which achieved a No. 1 position in the list of most downloaded apps at the start of the season.

Technical test background

For years Wie is de Mol? (WIDM) has been among the best-watched programs on television, and for all that time viewers have been able to play along at home with the accompanying app. In 2023, it was decided to have the app rebuilt in terms of both frontend and backend. Polteq tested this updated app extensively, where we took on the role of the Mole and tried to sabotage the app.

For this new app, AVROTROS commissioned digital agency Q42 to build a modern app with a clean look and great stability. Moreover, the app had to be future-proof and able to handle a growing number of users. Both the iOS and Android apps were completely redeveloped, along with the entire backend required for registrations and point calculations, among other things. All this had to be accomplished within a tight deadline, as the app had to function perfectly before it would appear in the app stores and more importantly, before the first episode would air on television. This required tight planning and much, much testing.

To make this project a success, the dream team consisted of the product owner and designer from AVROTROS, two Android developers, two iOS developers and two backend developers from Q42. From Polteq, we started with two testers, one for the native apps and one for the performance testing of the backend.

Extensive backend testing

Performance and stress tests were performed both on the backend to identify bottlenecks and in conjunction with app usage to test the app’s behaviour during peak loads. The key question of the performance test was: can the backend of the app, including the API and database, handle the expected load of users? To test this, the expected numbers were further developed into three episodes, each with their own load test scripts and load profiles. Each episode represented a peak moment in the WIDM season: a. episode one, b. episode X, c. finals. For each episode, the required API end-points were collected in a script and then incrementally put under load.

These steps were:

• Baseline (validation load profiles setup)
• Increment (putting backend under load)
• Peak (real load test) with 1,000,000 users
• Stress test (when does backend go down)

To simulate the required load, a group of virtual machines was created in the Google Cloud environment. These machines are completely self-equipped to generate the load without succumbing to the pressure themselves.

A second challenge was that the API consisted of gRPC end points instead of the standard REST HTTP end points. This protocol required a switch from JMeter to the K6 performance tooling.

A third challenge lay in sharing test data such as registrations, authentication tokens, pools and join-tokens. On each virtual machine, Docker was running with Redis to store and retrieve lists of test data. A nice added challenge was that the number of pools and the amount of participants in a pool had to follow the desired statistics.

A fourth challenge was that the entire backend was rebuilt with new end-points, so the load from previous years could not serve as an example for creating load profiles.

The performance testing was conducted in parallel with the app and backend development, so that the initial testing could be done as early as possible and there was enough time to understand how the system behaves under load, and to run all the necessary tests.

In total, more than 50 load tests were conducted, including in conjunction with the Android and iOS app, to validate the behaviour under expected loads of up to 1,000,000 users during the various episodes and provide advice on what scaling is needed to accommodate the load.

Total flexibility because the entertainment format is a mirror maze

For user acceptance testing, Polteq has written templates that allow any possible scenario to be tested by making just a few adjustments to the settings. Thanks to these templates, the test suite is as dynamic as the program itself and can be used by anyone.

Unlike participants who unexpectedly return after dropping out, the revamped app would not get a second chance. At launch, the app had to function 100% right away; there is no room for updates during the season. To ensure that the app would actually work properly on users’ phones, Polteq conducted a thorough device analysis to get a picture of the devices used by all users of the WIDM app. Based on this, we put together a mobile test lab to test with a representative selection of the user base.

Throughout the development of the project, the entire team held weekly test sessions with these devices in hand, playing parts or full seasons. Additional functionality was added at each iteration, with extra attention paid to these additions. As the app became increasingly complex, we increasingly used scenarios to ensure that every aspect of the app was tested during its seasonal run-through. With the deadline approaching, Polteq additionally helped by scaling up and deploying four additional mobile testers to test additional seasonal runs. In this way, Polteq was able to support the developers by taking work off their hands without sacrificing quality.

All in all, Polteq was able to contribute to the success of the app with:

• Native app testing
• Mobile test lab
• Device analysis API testing
• Performance testing
• Security review
• User Acceptance Testing

Polteq is able to test a mobile app on all aspects of the process!