How Shift Right Testing Enhances Quality And User Experience

We talked extensively about Shift Left and its practical implementation in some previous articles. Now we have to acknowledge the other side of the coin – Shift Right testing. Shift Right has become increasingly important in recent years although this concept is not new or revolutionary. It is simply a complementary technique to Shift Left. If we agree that we should test early in the development cycle, we must complement such testing with testing software after release to production.

Focus on Post-Production Testing

There are a number of techniques implemented to get the best out of Shift Right testing. I strongly believe that we can catch more bugs in early stages of the development cycle when we introduce Shift Left. However, the true benefit of Shift Right is understanding user behavior in real world examples. During postmortem analysis I have noticed in a number of occasions that we were not able to catch some bug in pre-production environment because the client had a specific set of data which emphasizes the bug and it becomes visible in production. This supports the idea of monitoring and testing in higher environments. Understanding real user behavior and performance is of key value for creating better software. Agile Manifesto points out the importance of interactions and collaboration with aim of effective software development. This is indirectly fostered with Shift Left implementations.

Continuous Feedback Loop

Learning from user’s behavior is invaluable for software development team. By following user interaction with our software we can gather information that can help us with future development. There are a number of tools which can be used for user actions in our software. Analytic tools in software do not help us only with providing the best experience to the user. We can also utilize them for filtering the unwanted or less attractive behaviors which may prevent the software from correct functioning.

Another, more hands-on approach is to perform the direct testing in production environment and gain insight about the feature with a specific set of data or some customer settings or customization. We can use A/B testing or even some tools which can help with visual testing. Of course, this requires a setup of proper user roles which the testers can use to test the production environment without affecting the real client’s data.

Types of Shift-Right Testing

Canary Testing: This is a way of reducing risk in case something bad happens after release. You first deploy the latest changes to a selected number of clients and then do monitoring and testing. When there is enough data to confirm that there are no critical issues introduced by this change you can push the release to the rest of the clients.

Blue-Green Deployments: This is also a safety net for deployment. In this case you have two production environments: blue and green. Blue is where the clients work on and green is a working copy of the blue environment, yet no one is using it. When there is a release, you can push the changes to just one of the environments, for example green. Next you would swap the environments, put green to be customer facing and blue becomes backup. You monitor the feedback from clients for some time and if there are no issues you push the same release to the blue environment. In case that the clients do report some issues on green, you would swap the environments again and the blue environment without the released changes becomes customer facing.

Chaos Engineering: This is a bit extreme engineering technique which was introduced long time ago but expanded by Netflix. Their development team believed that the failures are inevitable and therefore created a tool called Chaos Monkey. Chaos Monkey would just turn off random servers on Netflix’s AWS. If the system was well designed there wouldn’t be any downtime. It is an interesting approach, but have I mentioned that the Chaos Monkey works with production servers? In this way Netflix wanted to ensure that their engineers keep resilience of the system always on their mind when programming.

Tools and Technologies Supporting Shift-Right

Obviously the first thing is to have monitoring tools that can provide you with enough information to get some conclusions. There are a number of monitoring tools. Some are part of your CI/CD tools like Azure App Insights and some are independent monitoring tools like Grafana or DataDog. For user interactions you can use Google Analytics or Pendo (not affiliated with any of these btw) or something similar where you get high quality data for your purpose. For setup of Blue-Green or Canary deployments this should be available by any large deployment tool. I’m sure you can find a suitable tool after a little investigation.

Challenges of Shift-Right Testing

Production environments can become increasingly complex depending on the system and setup. If you consider Shift Right testing as an option I would suggest first to help yourself by reducing your system complexity. The system should be designed in such way to allow the testers to access it and use it without affecting the operations of real clients. This requires setup of specific roles and access rights which can be used only by testers. It is useful not only for testing new releases but also for troubleshooting purposes. Another challenge is the cost of the supporting tools which can become too high if it is not carefully planned. In this case return of the investment might not be as expected.

Best Practices for Implementing Shift-Right

It is always better to have small releases and release often. With the Shift left and Right set in place having small and frequent releases should not be a problem. Monitoring after the release in combination with collaboration with users for feedback is a winning combination.

The company should define metrics which will show you if you are on the right track or not. Whether you are looking for performance issues by following error rates, downtime, latency, spikes in memory usage or bugs in specific user flows you should manage your team’s expectations. Metrics can help you a lot here especially if you are interested to compare results between releases.

Reporting the issues, clear communication in the team, fast responses after failure are a must for a seamless experience. The team must work together and knowledge about any novelty must be shared across the team.

Conclusion: The Future of Testing

Ok, it may be a bit excessive to say that Shift right is the future of testing. But it has the potential to reduce the amount of frustration for both development teams and clients. It is an important leverage in software testing and the benefits of it are nothing new in software development world. It should be considered as candidate for test strategy as an opposite to Shift left. If your team already performs Shift left then it would be the right time to introduce Shift right as well.