What is End-to-End Testing? E2E Testing Full Guide (2024)

What is End-to-End Testing?

End-to-end testing is also referred to as E2E testing. Thanks to end-to-end testing, testers gain insights into how the application functions from the end user’s perspective, giving them a more comprehensive understanding the software quality before release.

Importance of End-to-end Testing

End-to-end testing is essential as modern software has grown to be intricate with dozens of systems interacting with each other simultaneously.

Even if these components function perfectly fine individually, they may still fail when integrated due to the miscommunication between components. End to end testing is there to verify the flow of information through the AUT, including all possible paths and dependencies. If there is an issue in any “touchpoint” between software components and/or subsystems, testers can easily locate the root cause and troubleshoot immediately.

Testers would want to achieve the highest level of coverage with E2E testing. All subsystems or components of the application, such as the user interface, application server, database, as well as any external systems that the application may interact with, should undergo E2E testing.

End-to-End Testing Examples

Here's an example of end-to-end testing for an eCommerce website.

Let's take eBay as an example. End-to-end testing ensures that the entire purchasing process on eBay works seamlessly. This guide outlines the necessary tests to validate the complete flow from product selection to receiving a confirmation email.

Steps for End-to-End Testing

1. Product Page and Product Selection Features

• Test Objective: Verify that users can view and select products accurately.
• Actions:
  • Navigate to a specific product page, such as "Apple iPhone 13 Pro."
  • Check that product details (name, description, price, images) are displayed correctly.
  • Select different product options (e.g., color: "Sierra Blue", storage: "256GB") and verify that changes are reflected correctly.
  • Add the product to the cart and ensure it appears in the shopping cart with the correct details.

2. Customer Information Form Data Validation

• Test Objective: Ensure the form correctly validates customer input.
• Actions:
  • Enter valid data into the customer information form (e.g., name: "John Doe", address: "123 Main St, San Francisco, CA, 94101").
  • Enter invalid data (e.g., an invalid email address "john.doe@invalid") and verify that appropriate error messages are displayed.
  • Ensure that the form accepts valid data without displaying errors and allows proceeding to the payment step.

3. Payment Information Validation

• Test Objective: Validate that the payment information entered by the customer is correct and secure.
• Actions:
  • Enter valid payment details (e.g., credit card number: "4111 1111 1111 1111", expiration date: "12/25", CVV: "123").
  • Enter invalid payment details (e.g., incorrect credit card number: "1234 5678 9012 3456") and verify that error messages appear.
  • Ensure that valid payment details are accepted, and the customer can proceed to the payment confirmation step.

4. Payment Processing Feature

• Test Objective: Verify that the payment processing system works correctly.
• Actions:
  • Submit the payment using a valid payment method.
  • Check that the payment processing system initiates and completes without errors.
  • Verify that the system handles different payment methods (e.g., credit card, PayPal) and processes them correctly.

5. Payment Success Confirmation

• Test Objective: Ensure that a successful payment triggers the correct response on the website.
• Actions:
  • Confirm that a success message is displayed on the website after the payment is processed, stating "Thank you for your purchase! Your order number is 123456."
  • Ensure that the success message contains relevant details like the order number and payment amount.

6. Email Confirmation

• Test Objective: Verify that a confirmation email is sent and received.
• Actions:
  • Check the customer’s email inbox for a confirmation email from eBay.
  • Ensure that the email arrives promptly after the purchase is completed.

7. Email Content Validation

• Test Objective: Ensure the confirmation email contains accurate purchase details.
• Actions:
  • Open the confirmation email.
  • Verify that the email contains the correct product name ("Apple iPhone 13 Pro"), quantity ("1"), price ("$999.99"), and order number ("123456").

8. Order History Verification

• Test Objective: Validate that the purchase is correctly listed in the customer's order history.
• Actions:
  • Log in to the customer’s account on eBay.
  • Navigate to the order history section.
  • Ensure that the recent purchase ("Apple iPhone 13 Pro") is listed with correct details (product name, price, order date, etc.)

Read More: 100+ Test Cases For eCommerce Website

Benefits of End-to-End Testing

End-to-end testing has been more reliable and widely adopted because of the following benefits:

• Quality management across multiple application levels: modern applications are built upon complex architecture consisting of multiple layers with interconnected workflow. These layers may work fine individually but conflict with each other once connected. E2E testing can verify the interactions between these individual layers and components.
• Backend QA: E2E testing first verifies the backend layers, especially the database of the application, which feeds critical information to other layers for the application to work.
• Ensure consistent application quality across environments: E2E testing verifies the frontend, ensuring that the application works as intended across a wide range of browsers, devices, and platforms.Cross-browser testing is frequently performed for this purpose.
• Third-party applications testing:there are external systems integrated into the application to perform highly specific tasks. End to end testing ensures the compatibility between external and internal systems, as well as the data communication between them.

End-to-End Testing Lifecycle

An end-to-endsoftware testing life cycle consists of four components: Test planning, Test design, Test execution, and Results analysis.

• Test planning: This phase takes place when integration testing is complete. In a test plan, test objectives are outlined based on the initial client requirements and application architecture.
• Test design: A proper test environment is set up based on the requirements. Usually the test environment is already configured for previous test runs, and testers can leverage these existing configs. Risk analysis and usage analysis is also conducted to more effectively allocate the suitable resources to achieve test objectives. After that, testers start creating the necessary test cases, utilizing automation testing tools to minimize time spent on test design
• Test execution: Test cases are executed, locally and remotely. Monitor test progress for consistency with the original test plan
• Results analysis: Analyzes test results and pinpoint the root cause of the bug. These discoveries will be sent to the development team for immediate intervention. Then a project retrospective is implemented to evaluate processes and discuss areas of improvements. Learn more about building a test report here.

Types of End-to-End Testing

1. Horizontal E2E test

Horizontal end to end testing is what the end user thinks of when they hear “end-to-end testing”. In this approach, testers focus on ensuring that each individual workflow in the application works correctly.

The eCommerce example above is exactly what horizontal E2E testing looks like from the end user’s perspective. The tests to be performed involve verifying the Product Search feature, the Product Page functionalities, Product Ordering steps, and Shipping details.

In other words, with horizontal end-to-end testing, QA professionals want to put themselves in the position of the user and test how they experience every customer-facing aspect of the application.

Read More: Manual Testing: A Comprehensive Guide

2. Vertical E2E test

Vertical E2E testing is a more technical approach. This is what the tester thinks when they hear end-to-end testing. Unlike end users who only experience the frontend, testers also work with the backend, and they want to make sure that the right data is transferred to the right place, at the right time, so that all background processes keeping the application running can be smoothly executed.

In the eCommerce website context, vertical E2E testing involves verifying the Account Registration & Verification process, account-related features, the product database, product updates, and eventually the UI (frontend). These vertical E2E tests happen in sequential, hierarchical order.

End-to-End Testing Success Metrics

Some of the many testing metrics used for E2E testing include:

1. Test Case Preparation Status

This metric tracks the percentage of test cases that have been prepared relative to the total number of planned test cases. It provides insights into the readiness of test cases before execution begins.

Formula:

PreparationProgress = (NumberofTestCasesPrepared / TotalPlannedTestCases) × 100

2. Test Progress Tracking

Metrics:

• Execution Progress: Measures the percentage of test cases executed relative to the total planned.
• Pass Percentage: Indicates the percentage of executed test cases that passed.
• Fail Percentage: Indicates the percentage of executed test cases that failed, providing insights into areas needing improvement.

Formulas:

ExecutionProgress = (NumberofExecutedTestCases / TotalPlannedTestCases) × 100

PassPercentage = (NumberofExecutedTestCases / NumberofPassedTestCases​) × 100

FailPercentage = (NumberofFailedTestCases / NumberofExecutedTestCases) × 100

3. Defects Status and Details

Metrics:

• Open Defect Percentage: Measures the percentage of defects that are still unresolved.
• Closed Defect Percentage: Measures the percentage of defects that have been successfully resolved.
• Defect Distribution: Analyzes defects based on severity (critical, major, minor) and priority (high, medium, low), helping prioritize fixes.

Formulas:

OpenDefectPercentage = (NumberofOpenDefectsTotalDefects) × 100

OpenDefectPercentage = (TotalDefects / NumberofOpenDefects​) × 100

ClosedDefectPercentage = (TotalDefects / NumberofClosedDefects​) × 100

4. Environment Availability

It measures the availability of the testing environment compared to the scheduled operational hours. It ensures sufficient testing time and resource availability.

Formula:

EnvironmentUptime = (ActualOperationalHours / ScheduledHours) × 100

End-to-end Testing vs Functional Testing

End-to-end tests are not just several unit tests andfunctional tests strung together – they are more complex and carry more risks. We’ve listed the main differences between functional and E2E tests to illustrate this further.

End-to-End Testing vs Integration Testing

End to end testing is quite similar tointegration testing, and indeed they do overlap in several aspects. Here are some major differences between them:

Read More: End-to-end Testing vs. Integration Testing: A Detailed Comparison

End-to-End Testing Main Challenges

Detecting bugs in complex workflows certainly has its challenges.

1. End-to-end testing can be time-consuming

Creating the necessary test suites and matching them to the user’s navigation within the application may require the running of thousands of tests and can be quite time-consuming. Usually QA teams leverageautomation testing tools to assist them in such tasks. These tools should give testers the collaboration capability so that they can align with the development teams on what to test.

2. Setting up the right test environment

Accessing a proper test environment is not always straightforward and can include having to install local agents and logging into virtual machines. To reduce the difficulties, it is better to adopt an on-cloud testing system which gives testers access to a wider range of platforms and environments to execute their tests on, saving the initial investment into physical machines.

End-to-End Testing Best Practices

1. Focus on your application’s critical workflows first

It's important to prioritize critical parts of the application. Focus on what most people use in your application and create tests for that in advance.

After selecting the most critical workflows, start by breaking them down into smaller steps. This gives you a more focused perspective into how your tests should be carried out and minimizes the number of unrelated tests.

2. Avoid exception testing

Exception testing is the process of testing the behavior or system when it encounters an error condition or exceptional event. Although it is a highly recommended practice, end to end testing is not suitable for exception testing. This type of test may identify that an error occurred but does not necessarily tell us why the error happened or how it will affect the application or system.

3. Build End-to-End tests that minimize UI flakiness

When it comes to end-to-end tests, UI tests are usually involved, and these types of test regularly fail due to the flaky nature of interacting with the system from the perspective of the end user. Network issues, server slowness, and many other aspects can affect the test results and generate false positives.

To deal with such flakiness, it is recommended to account for unexpected system issues while performing your tests. For example, withKatalon, testers can utilize the Smart Wait feature to wait until all elements on the screen have been loaded before continuing the predefined actions.

4. Leverage automation testing

Unlike other types of testing, end-to-end testing involves testing the application's functionality from end-to-end, covering all possible user scenarios, interactions, and interfaces. It requires a higher level of coordination and collaboration between teams, as it involves testing the integration of various modules, APIs, and systems.

For end to end testing, we need more than just test automation tools. We need a software quality management platform. Such a platform provides a comprehensive test management system that allows testers to manage and organize test cases, requirements, test plans, and defects in a single place.

End-to-End Test Automation with Katalon Studio

Katalonis an excellent software quality management platform that can make your end-to-end testsless complicated. With Katalon, software teams can have an end-to-end perspective of their testing activities, from collaboration for test planning, test authoring, test execution, test artifact management, to test analytics and reporting.

Demo speaks louder than words. Here's a sneak peek of how Katalon works:

Katalon supports a wide range of testing types, and both developers and manual testers can easily automate UI or functional testing for web, API, desktop, and mobile applications, all in 1 place. Tools used across the software testing life cycle can natively integrate with Katalon, giving you a comprehensive testing experience. (your existing CI/CD pipeline and test management tools).

Katalon has several noticeable features to support your E2E testing activities:

1. On-cloud Test Execution

Performing end-to-end testing across multiple browsers and operating systems can be a daunting task, especially when it comes to setting up physical machines to execute tests. This process can be time-consuming and resource-intensive, leading to delays in the end-to-end testing process.

By leveraging Katalon, you can easily run tests on the cloud, on a wide range of multiple browsers, devices, and operating systems simultaneously, ensuring comprehensive test coverage. This feature helps organizations save time and resources while improving the quality of their web applications.

Read More: Katalon TestCloud is The New and Better Way To Test Across Environments

2. Katalon Recorder

Different use cases are often mixed together in different orders and variations. But we can call each grouping of use cases a user journey. Technically, a user journey is a collection of steps in which each step corresponds to a user action. Collectively, they represent a typical user session.

Katalon offers the Recorder feature on the web to help you accomplish the task of creating user journeys without any hassle. The Recorder essentially watches and records all your movements on the application so that you can focus on creating the journey itself.

3. Built-in keywords

A user journey usually consists of hundreds of steps. When a test case – which represents a user journey – fails, it may be easy to pinpoint superficial causes (e.g. a change in a Web element’s property). However, it is harder to diagnose the actual cause solely based on the fact that the test has failed. It could have been because the current page is not the same page that was recorded because at some points the test case went off the right track.

To ensure that the user journey has to go through certain milestones, test oracles are necessary. Test oracle refers to a mechanism to ensure that the test case is being executed as expected. In Katalon, a set of built-in keywords that implement this concept is provided. You can assert or verify a web element against certain text that you expect it to contain, or its properties against some expected properties and many other types of expectations.

4. Custom keywords

Depending on the business logic of your application, there may be behaviors that occur across different pages but differ only in some known characteristics. In such cases, it is a best practice to capture these behaviors in a behavior template that can be filled with specific information when necessary.

In Katalon, acustom keyword can be defined to represent such a behavior template. Once defined, it can be reused in different test cases and even in different test projects. You can even share your custom keyword as a plug-in through Katalon Store so that others can benefit from it.