Usability testing focuses on whether the product meets its intended purpose among its users effectively without frustrating the user and if possible, to the user’s delight.
User Testing and A/B(Multivariate testing) are two key methods employed today to understand usability of a product.
User Testing is a method to observe a single user’s interaction with a product to see if it successfully serves the intended purpose. Multiple such interactions are recorded(typically ~“5 users” ) to obtain diverse feedback.
A/B Testing is an experiment to observe which of the two variants of the product design is preferred by the users and results in positive impact for the business. Two is the minimum number of variants, and in many cases, multiple variants of the design are tested.
Both User Testing and A/B Testing remain excellent choices for usability testing, but an appropriate choice may depend on adoption of the product, the time available to run experiments/gather feedback, and of course, the resources available in the team.
#####Early stage vs Highly adopted product: In the earlier stages of the product, when the adoption is low, the best way to obtain quick feedback about usability is by doing User testing.
The coolest thing about user testing is that it can be performed on any kind of designs, from simple wireframes to very detailed working mockups, to actual product. It is entirely up to the product manager to decide the level of detail, that would be needed at a given stage. For a highly adopted product, it could still turn out to be useful, to optimize certain usability pain points, however, at this stage, comparison of multiple designs with the existing users may provide greater insights. Perhaps, a better use at this stage would be come up with the potential alternates for A/B testing using User Testing.
A/B Testing would not make much sense at an early stage, as the adoption is low, and the key is get users on-boarded to a single design that solves the customer problem. On a highly adopted product the product team can use it to test multiple variations on their design.
#####Time available for Testing: Direct User feedback is quick and void of any ambiguities and the user testing does exactly that. It is more descriptive and qualitative, and may sometimes involve some surprise discoveries/feedback from the users that the product team may not thought about. This direct feedback can help in quicker resolution and essentially some quick wins.
A/B Testing helps in deciding which of the designs/variants are preferred by the users. Smaller changes bring in smaller gains resulting in a compounding gain over a period of time. However, these tests cannot be rushed. The testing has to reach statistical significance and that takes time.
#####Resource availability: There are other resource constraints that needs to be considered when deciding the type of usability testing.
User testing changes have to be watched closely to see if any other key metric is being adversely affected. There may be a need to revert back should any of the key metrics be drastically affected.
Not all teams may have a resident A/B Testing expert, who can develop appropriate hypothesis, create and run experiments. The results of the A/B Testing are more definitive and the changes result in most definite small incremental successes. It can be very exciting to see a variant winning in the first few days and the experimenters should also be aware of false positives, and wait for the test to reach statistical significance.
Both the User Testing and A/B Testing have their respective advantages, and if possible should be paired together. In fact A/B Testing designs can be done based on the feedback from User Testing, and thus a more precise design change can be made on a positive hypothesis.
Usability Testing is qualitative in nature, i.e., it focuses on the ‘Why’.
A/B Testing is quantitative in nature, i.e., it focuses on the “How Many”
It is absolutely important for the product teams to understand which test is needed when, the different factors that influence these tests and make a right choice for their team and their product.
Also, published in Linked In : “User Testing Vs A/B Testing”