Polteq iQ®: testing AI and complex systems
Under the banner of “Digital Transformation,” organizations are increasingly embracing digital technology to respond faster to changing market needs. Traditional organizations are suddenly transforming into data-driven organizations because of the big data these digital technologies produce. Thanks to “Artificial Intelligence” and “Machine Learning,” organizations are enabled to quickly and accurately analyze large amounts of data, gain new insights and make the correct decisions. While these AI applications contribute in accelerating and improving work processes, in practice they often prove to be opaque and complex, and organizations risk losing control of their internal decision-making. Polteq iQ® (increment Quotient) gives the organizations insight into what impact changes in ‘Artificial Intelligence’ and ‘Machine Learning’ applications have on results.
The challenge: AI as a ‘black box’
Artificial Intelligence and Machine Learning assist in performing analysis, gaining new insights and making decisions based on big data. However, these technologies are both complex. In practice, the underlying algorithms turn out to be a “black box” for most people, yet these algorithms are increasingly influencing our lives.
Traditional approaches to software testing provide insight into the quality of such “black boxes” by comparing results desired in advance with actual results. Based on these comparisons, it can be indicated whether the software is “right” or “wrong.” However, testing AI systems requires a different approach. Due to the dynamic nature of AI systems, traditional testing approaches are insufficient to demonstrate the quality of these types of systems. In AI systems, the behaviour of the system can evolve without human intervention, has its own feedback loop and can learn continuously from new data. One of the risks here is that the AI system will behave differently after go-live than it did during the development process.
So how do you measure the quality of this process, how can you trust such a self-learning system and how do you know that the algorithms are doing what they are supposed to do?
The solution: Polteq iQ®
To optimize the quality of AI systems, it is essential to understand the impact of changes made to the underlying algorithms (and thus the operation of the product). Polteq has developed a structured testing approach for this purpose with a corresponding test framework (Polteq iQ®). Polteq iQ® allows you to compare different versions of a system and determine – prior to going into production – whether the product or service has improved or worsened because of the modifications. Polteq iQ® communicates with the API of the target system to perform automated test cases. The results of the output on the different systems are automatically compared, filling a dashboard with metrics.
The visualization of the results in a dashboard facilitates interpretation. Polteq iQ® is suitable for multiple applications because the metrics can be set up on key performance indicators appropriate to the domain being researched.
The insights gained can be used to rearrange the algorithm. After the change(s) are made, they are measured again, providing insight into the outcome of the changes. This gives you control over the quality of your AI and/or “Machine Learning” applications.
The approach
Polteq iQ® for testing AI systems includes the following steps:
1. domain analysis
Configuring Polteq-iQ® tooling 2.
Setting up required test data
Linking Polteq-iQ® with API 4.
5. test execution
6. analysis metrics and reporting
Applications Polteq iQ®
A good example of a system with underlying AI is a “recommendation engine”.
For example, a “recommendation engine” recommends products based on the buying behaviour of others who have a similar profile. However, a result that is fine for one user may be unacceptable or unusable for another. A new version of the system may then have a different impact on different users. Polteq iQ® is ideally suited to analyze the impact of changes on the system.
Other areas where Polteq iQ® would be well-suited include:
- Search engine optimization
- Virtual Assistants
- Fraud Detection
- Online customer support/chatbots
- Automated editors
- Surveillance
- Music and movie recommendations
- Social media
- Medical applications
- Object Recognition
- Virtual Reality/Augmented Reality
Want to gain control over the impact of changes to your Artificial Intelligence and Machine Learning applications? If so, please contact us.
