Overview

Timberland is a major apparel brand in the UK, we were involved in testing various aspects of the website, most notably the user experience of the website. The website used to go through various phases of A/B testing as well to decide which design would engage more people.

Challenges

• The main challenge with this project was that we’d receive the scope of what had to be tested right before the actual testing started, which meant that the QA had to quickly understand what the updates were across the website. A major challenge was often the limited timeframe for testing, which may result in insufficient time to fully test all areas of the website, potentially leading to missed bugs or other issues. Due to the small timeframe for testing, the testing process needed to be efficient and accurate, which could have resulted in focusing only on the changes that need to be tested, potentially overlooking other areas of the website that may have been affected. The client would provide us with the TSR which was the Testing status report, which contained all of the necessary information about the project and what changes have been implemented in the website, and also which area of the website should the QA focus on.
• While conducting A/B testing, the load balancer selected which user receives which version of the website, which can be a challenge when trying to test specific variations. Determining the appropriate sample size and duration for A/B testing used to be a challenge, as tests that run for too short a period may not capture enough data to detect meaningful differences, while tests that run for too long may delay the implementation of changes. Other confounding variables, such as seasonality or marketing campaigns, impacted user behaviour and affected the results of A/B testing. As an e-commerce website, testing of the add-to-cart and checkout processes was also critical, and any issues that arise during testing may have a significant business impact.

Solution and Strategy

• Since we had a limited timeframe for testing, we prioritized testing based on risk and impact. The QA team worked with stakeholders to identify critical areas that must be tested, and we allocated testing time accordingly.
• To improve testing efficiency, we leveraged testing frameworks and methodologies, such as agile testing or exploratory testing, to help identify critical areas for testing and prioritize testing efforts.
• As we were receiving the scope of changes at the last minute, we worked with stakeholders to establish clear communication channels and ensured that changes are communicated in a timely and consistent manner. This can help ensure that the QA team is prepared to test the changes as soon as they are implemented.
• To determine the appropriate sample size and duration for A/B testing, the QA team can work with stakeholders to establish clear success metrics and goals for the test. This can help ensure that the test is conducted for an appropriate length of time and that the sample size is sufficient to detect meaningful differences. For the impact of confounding variables on A/B testing, we work with stakeholders to identify potential variables that may impact the results and establish methods for controlling or accounting for these variables.

Conclusion

By prioritizing testing based on risk and impact, leveraging testing frameworks and methodologies, establishing clear communication channels, and working closely with stakeholders to determine appropriate sample sizes and success metrics, the QA team was able to successfully navigate the challenges presented by the limited timeframe, confounding variables, and critical e-commerce processes. As a result, the website was thoroughly tested, and any issues were identified and resolved before launch, ensuring a high-quality user experience for customers.