A/B testing is the ideal method for testing and understanding store performance, marketing efficacy (both paid and organic), and customer behavior in eCommerce; however, insufficient levels of traffic often prevent new sellers from being able to conduct legitimate split tests. Alternatives include:

  • Non-scientific A/B testing
  • User experience research
  • Surveys/polls
  • Customer interviews

While these alternatives can provide valuable insights, sellers need to be cautious when drawing conclusions from anecdotal evidence — which is highly subject to bias, heuristics, and framing effects.

Non-Scientific A/B Testing

It is possible to replicate A/B testing with insufficient levels of traffic, but you will not be able to draw conclusions with the same level of confidence. Extrapolating on non-scientific “findings” presents an array of problems, but making modest changes based on what the data seems to be suggesting could be beneficial for your store. Common areas of non-scientific exploration include:

  • Pricing
  • Product details
  • Photography and graphics
  • Product offerings and/or bundles
  • Color scheme
  • Banner details

For Example: Your store is fairly new, and your traffic levels are modest. You would like to experiment with price on one of your product pages ahead of peak retail season, so you plan to create two versions of the same page — one is the control, and the testing page features the product for $1 less. If conversion is disproportionately higher for the test, you may want to consider making this change.

User Experience Research

User experience (UX) research encompasses both qualitative and quantitative aspects, which yield insight into a user’s experience with a website or app. Unlike A/B testing, UX does not require high levels of traffic or near-perfect controls for experimentation — it is a broad term that includes things like engaging members of your target audience through test sessions, using software to observe user sessions remotely, and tracking user behavior through applications on your website.

For Example: Your store typically has an add-to-cart (ATC) rate of 12-15%, an initiated checkout (IC) rate of 6-8%, and a conversion (CR) rate of 3-4%. While evaluating statistics for one of your pages, you notice that there is an above-average 18% ATC rate, but only 3% of visitors initiated checkout, and only 2% converted. This unusual data indicates that something might be wrong with your cart — since that’s where the numbers shift from above to below average. Using UX research tools to evaluate your shopping cart (the look, feel, functionality, etc.) could help you to find and plug the leak.

Helpful Tools:

  • Lucky Orange: Lucky Orange is a service that leverages heat maps, screen recordings, and other user monitoring functions to aggregate behavioral data from your store.
  • Google Analytics: Google Analytics is a program that tracks, evaluates, and reports user data. By adding a track code to each of your pages, you can use Google Analytics to synthesize user behavior data and generate reports on a variety of important key performance indicators (KPIs) to help you better understand your customers.

Surveys/Polls

As another alternative to A/B testing, surveys and polls are an easy way to engage website users and learn about their experiences with your online store. You can send general surveys or surveys to certain customer segments, depending on your interests. Some things to consider asking:

  • Demographics: What is your gender, location, etc.?
  • Recipient: For whom were you buying our product(s)? Was it a gift?
  • Packout: Was the product packaging/presentation acceptable? Would you like a gift-wrapping option? Would you be willing to pay more for customization?
  • Shipping: Were you happy with the shipping speed? Would you like the option to expedite? How much would you be willing to spend to expedite delivery?
  • Brand: What kinds of content would you like to see more of on our website? 

Unfortunately, survey response rates for most companies hover between 5%-30%, so gaining abundant data can be difficult with this method. To overcome this, consider using an engaging survey and offering discount codes or giveaways to entice participation.

Helpful Tools:

  • Google Forms: Google Forms is a free tool that allows you to collect and evaluate data in a useful way. Each form can be highly customized to reflect your brand, and forms automatically generate an associated Google Sheets form for expedient data analysis.
  • Typeform: Typeform is a dynamic SasS offering for creating interactive surveys and uncovering actionable insights.

Customer Interviews

Interviews are a highly personal way to glean information from existing customers who are already familiar with your business and they are particularly beneficial for those in consulting or who otherwise sell high-ticket items. Since these customers have already purchased from you, confirmation bias is a concern (this also applies to data from surveys and polls of existing customers); regardless, customers who interact with your brand on a regular basis and have traveled through your sales funnel are likely to provide valuable insights for improvement.

While your interviews may happen on a more ad hoc basis, especially if you are interviewing over email, it is important to be prepared and to ask structured questions about topics that are important to your business — although, a few open-ended questions can help to add a level of richness that might otherwise remain elusive in general surveys and polls). As usual, look for commonality in responses — both positive and negative.

Conclusion

It cannot be emphasized enough: drawing conclusions from non-scientific (anecdotal) data is risky as inadequate samples are especially susceptible to bias, heuristics, and framing effects — meaning that any conclusions drawn from there could easily be misleading. That said, alternatives like those discussed in this article do have a place in eCommerce and can often point sellers in the right direction — especially for conclusions which seem to be consistently upheld either over time and/or across tests.