Generative AI’s Role in Software Testing and Quality Assurance

The introduction of generative AI in software testing has made a significant difference in the approach, quality, and productivity of software testing.

Before generative AI, software testing was about manual work that consumed time, shackled productivity, and didn’t always yield the optimal outcome.

Let’s take, for example, test cases that are manually written, leaving the risk of unidentified scenarios open — what you expect is what you get.

Generative AI has automated multiple actions in software testing, including test case generation, code mutation (where code is deliberately manipulated), and automated bug detection with speed and accuracy, freeing up software testers from repetitive tasks to focus on more critical and creative assignments.

Generative AI has, however, certain limitations. For example, ChatGPT is unable to perform testing on mobile devices — it has no access to all, indeed any, devices or emulators to run tests on.

Generative AI has been evolving fast, and soon it can enter into unexplored areas of software testing. But it begs a question, though — what happens to the software testers? Where do they go from here?

Software Testing Before Generative AI

Before generative AI, software testing was a manual process, which included:

• Creating test cases

Testers would create test cases based on their understanding of the requirements and scenarios. The test cases provide the steps to test the software and the expected outcome.

• Manual execution

Based on the test cases, the software tester would test the software and match the outcome with that written in the test case.

• Regression testing

Testers would then write test cases for regression tests. When a new feature is added, updated, or removed, regression tests find out the effects of the change on the software. It’s helpful to identify any issues because of the change.

• Exploratory testing

Testers would put themselves in the shoes of the target audience of the software and explore the software. The process yields defects from the perspective of a customer.

• Performance testing

The software would be put through different levels of user loads to simulate performance under scenarios like other user loads, page or screen load time, and resource consumption.

• Security testing

The tester would try to test the security of the software through various means, such as entering the wrong credentials and attempting other ways to gain access to the software. ?

Manual testing is a time-consuming process that is also prone to errors.

Software Testing After Generative AI

Generative AI has automated multiple areas of software testing and can increase speed and productivity significantly:

• Test data generation

Availability of test data before generative AI was a problem, and it took a while to generate the data. Generative AI can generate quality test data quickly for various scenarios, edge cases and create input combinations that are difficult or time-consuming to do manually.

• Test case generation

Manual test case creation can miss scenarios, whereas with generative AI you can input the software details such as requirements, specifications, and design, and provide appropriate prompts. Generative AI will generate a comprehensive suite of test cases.

• Mutation testing

Mutation testing is about testing a mutated code and finding the difference between the original and mutated code. Modifying code to create a mutated version manually is a difficult process, but generative AI can create multiple mutated versions of the code, with each test being a step toward ensuring code quality.

• Code review and analysis?

Code review, analysis, and refactoring are substantial but necessary efforts, and manual efforts need more manpower and time. Generative AI can review code, provide suggestions and examples of improved code, and even refactor code.

Case Study

This case study is on how generative AI-enabled superior testing of the Messenger app from Meta.

Messenger is a popular app that needs no introduction. However, manual testing was challenging.

For example, users access the app in various ways and on different devices such as mobile, tablet, and laptops, bringing diverse scenarios of app access and use. Identifying so many scenarios and creating test cases manually is a big challenge, and failure impacts the quality of the app and the user experience.

Generative AI simulated user interactions in real-time and captured the data. This enabled the software to know how users think, versus what the software testers assume while writing test cases. Based on the user interaction data, Generative AI generated test cases and data and then automated the testing of the app.

The main benefits of deploying generative AI were a better understanding of user behavior, a superior testing process, and automated testing. Meta saw improved productivity, higher user satisfaction, fewer bugs, and a better testing workflow.

It reinforced the idea that manual testing can capture user intentions to an extent, but generative AI can constantly learn from user behavior and quirks.

Limitations of Generative AI

For all the benefits it brings and the excitement it generates, generative AI is still evolving in the software testing niche because it still has limitations.

For example, ChatGPT struggles to test apps on mobile devices because it doesn’t have access to multiple different mobile devices. It can’t parse the scripts or details of the different mobile devices and write test cases.

For example, ChatGPT cannot load an app on a particular Samsung mobile device and test the login screen of the app.

Since it’s an evolving area, we can’t yet say for sure that generative AI is the answer to the problems of manual software testing.

The Bottom Line

Generative AI is a potent force and is here to stay. It operates on a model of continuous learning, and it’s reasonable to assume that it will overcome its limitations one by one.

But will it replace manual testers? The answer depends on multiple factors, such as whether generative AI can perform testing in extremely complex applications with high stakes.

Will the stakeholders trust generative AI enough to test extremely complex applications?

The answer may be a mix of automated and manual testing because complex applications require human analysis and finesse.

Overall, the future is not decided, but it certainly looks like manual involvement and automated testing will learn to co-exist.