Testing in production, or 'production testing,' means evaluating software in its live environment. This practice was once met with skepticism but increasingly recognized for its value. As the software industry evolves rapidly, perceptions about testing in production are changing. What was once frowned upon is now becoming an endorsed practice in many scenarios. Despite this shift, the lingering doubts about its effectiveness and the fear of potential backlash remain for many. If you are skeptical about the merits of testing in production, this post will help you address your concerns and shed some light on it.
Understanding testing in production
Testing in production refers to the practice within software development where new code changes are evaluated directly with live user traffic instead of being tested in a controlled staging environment. This approach is intended not to replace other functional and non-functional testing types but to enhance and complement them. Adopting testing in production often hinges on risk management considerations for the product's launch. While testing in production can offer valuable insights from real-world usage and contribute to the application's improvement, it necessitates careful execution to mitigate potential adverse effects on users.
The adoption of testing in production comes with its set of challenges:
- There's the danger of deploying flawed code.
- Sensitive data might be inadvertently exposed.
- System capacity could be overwhelmed, leading to performance issues.
- Tracking and analytics data could become corrupted.
- Introducing poorly designed products or features to your users is risky.
Pro tip
At Global App Testing, we use a crowd of professional testers from around the world to provide real-time feedback and insights, facilitating continuous testing and improvement of the product before, during, and after release. This continuous evaluation provides immediate insights into the risks associated with new code submissions, enhancing the ability to deliver timely and error-free software. Automated production testing, covering web, mobile, desktop, and APIs, can significantly increase both efficiency and effectiveness.
Assessing the viability of testing in production
Why would you consider testing in production? In certain situations, you might find yourself without alternative options. Creating a staged testing environment might be too complex or costly, or you may need to capture genuine usage data, necessitating a direct approach to the live environment. Certain tests, particularly those assessing an application's scalability, are more accurately conducted within the production environment. Load testing, for instance, is most effective when performed on the actual system used by your users.
Ultimately, the intricacies of software development play a pivotal role in this decision. Despite having a robust QA strategy, employing the latest best practices, and utilizing advanced tools, some issues are bound to slip into the live environment. Thus, testing or monitoring your application in production serves as a crucial safeguard, acting as a final barrier to identifying and addressing these unforeseen bugs.
How to test in a production environment
How to effectively conduct testing in production? Here are the key strategies that harness the benefits of production testing:
- Real User Monitoring (RUM): Tracks and analyzes the interactions of real users with the application, providing insights into performance issues, user behavior, and overall experience.
- A/B testing: Compares two versions of a webpage or app to see which performs better on live traffic, enabling data-driven decisions about features or designs.
- Canary releases: Gradually rolls out changes to a small subset of users before a wider release, allowing teams to monitor and evaluate the impact of new updates in the production environment.
- Feature flagging: Allows the toggling on or off of specific features without deploying new code, enabling testing of feature impacts on user experience and system performance.
- Load testing: Simulates real-life load conditions on the application to understand how the system behaves under heavy traffic or data processing demands.
- Chaos engineering: Intentionally introduces disturbances or failures in the production environment to test the resilience and reliability of the system.
- Synthetic monitoring: Uses automated scripts to simulate user interactions with the application, helping identify issues before they affect real users.
- Shadow testing: Routes incoming traffic to both the production and a clone of the production environment, allowing comparison and testing without impacting the live system.
- Blue/green deployments: Switches traffic between two identical production environments, one hosting the current version and the other the new version, to minimize downtime and risk during updates.
Which tools to use?
In the production environment, choosing the right testing tools is crucial, covering a broad spectrum to meet various testing needs. Essential tools include Application Performance Monitoring (APM) tools like New Relic and Datadog for real-time performance insights and user analytics platforms like Google Analytics to understand user interactions and identify potential UX issues.
Feature flagging services like LaunchDarkly enable the safe testing of new features by allowing features to be toggled without a code deployment. Automated testing tools, such as Selenium for the web, Appium for mobile applications, and JMeter and LoadRunner for load testing, are critical for efficiently identifying and resolving issues. Platforms like JIRA and TestRail support manual testing, while Real User Monitoring (RUM) tools like Raygun and LogRocket provide direct insights into user experiences and errors. When used effectively, this toolkit significantly improves application quality and reliability, blending automated and manual testing methods for a seamless user experience.
Global App Testing further enhances this approach, supporting both automated and manual testing with the flexibility to meet project-specific requirements, from exploratory tests to large-scale test execution, ensuring high-quality product delivery.
Insights from industry professionals
So, should you test in a production environment? Don't just trust us – we consulted three testing experts actively utilizing production testing, gathering insights on its advantages and potential pitfalls.
Benefits of testing in production
So, what are the benefits? Well, it turns out that there are a number of them!
1. Authentic User Experience
There is no margin for error in terms of production environments. You are testing in the real-life environment of the user, so you can effectively test their user experience.
2. Preparedness for unexpected user actions
Testers are skilled at evaluating software for issues. Still, with their unpredictable usage patterns, customers are likely to encounter or cause unforeseen problems in your app. It's wise to anticipate these scenarios and be ready to address them.
3. Real-world payment verification
There are definitely some elements of testing that benefit from the production environment. Ajeet Dhaliwal highlighted the importance of testing in production when looking at payment functions. You cannot precisely match the payment process, so conducting the test in the production environment can be beneficial.
Drawbacks of testing in production
However, testing in production does not always work. It can have some real drawbacks if it isn't used correctly or if it is used unnecessarily. As explained by software engineer Cindy Sridharan in her extremely thorough article on the subject:
"When done poorly or haphazardly, "testing in production" does, in fact, very much live up to this reputation. Testing in production is by no means a substitute to pre-production testing nor is it, by any stretch, easy."
What Cindy means is that testing in production is very difficult to get right. Assuming this is an easy fix would not be correct. Here are the drawbacks our experts highlighted:
1. Operational restrictions
Performance testing in production can negatively affect all users, including customers, due to inherent limitations and legal constraints on using real data. In fact, some QA experts and engineers hear the term and run a mile. That's because there are a lot of scenarios where testing in production is a big 'no.'
2. Potential for customer loss
Directly testing in production risks exposing users to bugs, potentially leading to dissatisfaction or losing potential customers. Your customer may encounter issues, churn, or simply write off your software as buggy. That's why companies who test in production need to have the ability to take risks and not suffer hugely if they go wrong.
3. Risk of catastrophic failures
Poorly executed production testing can introduce severe issues, highlighting the need for a skilled and cautious approach. As Cindy Sridharan says, testing in production is a cautious art. You have to have a highly skilled team willing to take risks. Otherwise, you could end up with a real mess you did not want on your hands.
In conclusion
Testing in production is a tricky subject matter. While it has benefits, the QA community takes a cautious approach - tread carefully, they say! But that doesn't mean you should run a mile from the prospect of testing in production. In fact, it can be a handy tool in your testing toolkit. Take a risk assessment of how your software could deal with the potential drawbacks, and see how it levels up regarding the benefits. Testing in production could be right for you. And, if you want to learn more about how Global App Testing can help you during any phase of your product development, sign up and schedule a call today!
We can help you test your product while being aligned with your business goals
Contact us
FAQ
1. Is testing in production risky?
While there are risks, such as the potential for affecting real users with testing actions, these can be managed with careful planning and techniques like canary releases, feature flagging, and monitoring tools. The benefits of identifying and fixing issues before they impact a broader user base often outweigh the risks.
2. How does testing in production fit into the CI/CD pipeline?
In a CI/CD pipeline, testing in production is often the final stage, following development, unit testing, integration testing, and staging testing. It ensures that the software is continuously and automatically tested and deployed, maintaining high quality and user satisfaction.
3. Can testing in production replace pre-production testing?
No, it complements but does not replace pre-production testing. Testing in production is an additional step that catches issues not found during pre-production stages, ensuring the application performs well in the real world.
4. What are the best practices for testing in production?
Best practices include:
- Starting with a clear plan.
- Using appropriate tools and techniques for your needs.
- Closely monitoring the impact of tests.
- Being ready to revert changes if necessary quickly.
- Always prioritizing user experience and data privacy.
Keep learning
Automated testing vs Manual testing - What's the difference?
5 Best practices for testing web applications
Quality Assurance vs Quality Control
