Quality Engineering Trends From State Of Software Quality Report

Katalon is a huge player in Quality Engineering tools. It creates Katalon Studio which is among most popular test automation tools. Few years back they started creating a report on State of the Quality and publish it to the Quality Engineering community. Katalon published this year’s report a few days ago. The report contains analysis of the survey answered by almost 4000 people relevant to the Quality Engineering. This is my analysis of the upcoming Quality engineering trends.

The most adopted Quality Engineering practices

As you may have guessed it, test automation is the most used QE practice. However, even with the largest percentage of respondents confirming that they are using test automation the most, this percentage is only 57%. This figure was increased compared to last year’s 46%. Unit testing is in second place. It is a choice of 49% of respondents which shows an increase from 34% reported in 2022. Code review takes high third place with 43%. Only13% of survey respondents select some of well known opposite practices to code review like pair programming. Another surprising figure is – 30% of people claim that they do monitoring and testing in production. As a second side of the medal to Shift Left testing, shift right testing should be more utilized. According to the report, the monitoring and testing in production has decreased in last two years from 38% to current 30%.

QE practices and their effectiveness

The report also questioned the effectiveness of different quality engineering practices. Although automated testing is recognized as the most effective practice, the results show evidence of high effectiveness of practices such as test driven development (TDD) and behavior driven development (BDD). Having in mind that only 29% and 26% of respondents said that they are using TDD and BDD respectively, it seems like these practices are not used enough.

After all, the quality engineering community seems satisfied with overall state of the software quality after release. High percentage of 73% of respondents say that their software quality is either very good or good. I would be very much interested to compare those results with software quality from customer perspective.

Challenges in Quality Engineering

The survey also addressed the biggest challenges in Quality Assurance. Some of the most popular answers encompass: lack of time to ensure quality, challenges in test automation, frequent requirement changes.

There is nothing new here. These are the challenges we all face, yet our approach to overcoming them is what makes difference.

Better communication in the team is the best solution for frequent requirement changes. When the entire team is notified about the change simultaneously they can address the changes more effectively.

Applying test automation is also a matter of better planning. Whether it is unit testing or API or UI automation Quality Engineers must plan it according to the overall development plan. One way we can achieve this is Shift Left implementation.

Lack of time to ensure quality is an old issue. We can address it only by acknowledging that quality is essential part of the software development. Testing is part of development we cannot skip it or leave it for later. It has to be embedded in the development process and planned accordingly. Lack of time cannot be an excuse for shipping unverified software. The team must do task prioritization, analyze risks and implement right tools for the job.

Test automation adoption brings higher return of investment (ROI)

The report tells us that the test automation adoption is increased in 2024 compared to previous years. Web applications, API testing and mobile applications show increase in adoption of test automation. Only desktop applications show decline in trend of test automation. Selenium is still number one option for test automation with 51% of votes. Appium is 5th with 29% and Cypress takes 10th place with 18%. Interestingly, Playwright is located in 28th place with only 2%.

The respondents claim that they have been able to notice higher ROI after adopting test automation. One third of answers show 20% savings while 23% of respondents claim 20 – 50% savings. It is clear from the report that the ROI is higher after more years of applying test automation. Respondents with over 5 years of test automation implementation (15%) say that their savings are higher than 50%. Apart from ROI there are other benefits of test automation. Being able to test more (49%), shorter time to deliver (42%) and higher quality of software (51%) are just some of them.

However, test automation still suffers from a number of challenges. Astonishing 45% of people claim that they face lack of skills and experience in test automation. Compared to last years 37% we are showing an increase in this category.

What can we learn from this? The organizations should explore more options for test automation because of higher savings. The current state of test automation, although increasing it is still underutilized. QA teams are obviously focused on well established and known tools with large communities. The choice of right tools should be based on the tool that solves the problem in the easiest way. The popularity or the tool, the familiarity of the team with the tool and technical prowess should not be key factors.

Is AI taking over Quality Engineering?

The AI definitely is one of Quality engineering trends. It is used mostly for generation of test cases, test data and automated tests. Among those companies who haven’t implemented test automation, 69% use AI for test case generation. ChatGPT is used the most with 58% of votes, while Github Copilot is in second place with 28%. All the others are used a lot less. Some of the obstacles to achieving higher usage of AI tools are privacy concerns and lack of capable AI tools. Around 30% of respondents also say that AI tools are unreliable for their type of work.

However, AI tools seem to be more and more used by newer generations of QA engineers. Less experienced engineers with 1 – 3 of experience tend to use them more (65%) while this percentage drops with years of experience. Engineers with 10 – 20 years of experience use AI only 33%. We can understand this in a way that AI cannot replace years of experience. Less experienced engineers are trying to fill the skills gap by using AI.

From these results we can conclude that we can achieve some lower skill activities by using AI, but these tools are still not capable of performing more complicated tasks. There is a valid use case for using these tools in Quality engineering but only as a side help in daily activities. It is still difficult to imagine that AI tools will be able to take over the entire set of tasks performed by experienced Quality engineer.

The future Quality engineering trends?

The community identified several topics to focus on in the future. High 57% wants to focus on improving processes and practices followed by 56% of those whose focus will be to increase test coverage. Whopping 47% wants to increase usage of AI in test practices. The respondents would like to see AI develop more in the following directions: 52% wants to use it for generation of test cases, 51% wants to use it in generation of test cases and scripts in test automation, 46% would like to use it more for test data generation. The report also shows further need to increase test automation adoption, unit testing and test driven development.