data

Analytics, A/B Testing: if you run an ecommerce company, you have spent time looking at and analyzing the data available. But are you looking for the right results? Or are you making some major analyzing mistakes? Below is a list of the most frequently made mistakes – so you can watch out for these common pitfalls:

1. Making Conclusions Before Finishing Tests

When you set a test to run, the longest part is always going to be waiting for results. As humans, we are impatient for progress and results, which means most people will likely check the tests daily – and as soon as they see a statistically significant result, they stop the test and declare a result!

The problem with this method is that every time you check and make a conclusion, your chances of drawing a bad conclusion double. This means that after 5 times, your chances of a wrong conclusion have jumped from 5% to 23%!

This means as soon as you start your test, leave it alone. Wait until the testing is completed, and only then come back and analyze the results. Fight the urge to look at results while the test runs.

2. Misinterpreting Influence on Data Variations

One of the great things about having an ecommerce site is being able to try a variety of methods to increase revenue. However, just as you are likely to quickly switch to a new Facebook ad campaign, you are equally likely to interpret a change in statistics as something you did. In ecommerce, changes are common and random; often one day will be busier for no particular reason, and if the difference is low, it could easily be a coincidence.

Do not jump the gun and assume that every wave in data was caused by you. Only if there is crystal clear evidence that your peak was caused by something other than random forces – meaning, a statistically significant difference between group A and group B – should you study the data. Otherwise, wait and see if your changes actually make an impact.

3. A/B Errors

With A/B testing, you can make two errors: you can conclude A is a winner, when in reality it does not perform better than B (or even performs worse). You can call this a false positive. The second error is concluding A is no different from option B, when in fact it was. That is a false negative.

If you do not run the test for long enough, also called an under-powered test, you run the risk of drawing the wrong conclusion. That means you have just wasted precious time and money. The answer is to know the power of the test, and make sure it is about 70-80% – and run your test to the end.

4. Not Deciding with Data

There is an unbelievable amount of information you can draw out of data – and with great information, comes great power. The more you are able to make decisions based on solid data, the better served your company will be.

Of course, this is contingent upon having someone who can read and correctly interpret the data provided. Oftentimes companies do not have a Data Scientist, and while the analytical systems available are making it easier to break down the results, there may be some overlooked points that only qualified personnel could interpret. Improperly analyzing data and making decisions based on those results could lead to harmful outcomes for the company.

5. Answering Trivial Questions

Do not make the mistake of answering small questions with the data. You might want to answer “what” questions, but the correct usage is about “why” questions. Big data is about joining data sets that have not been joined, asking questions that have not been previously asked. It is about finding out why customers and employees do the things they do.

Be careful not to veer too far to one side, and overcomplicate things. To answer “why” questions, you do not need a huge team of analysts, or overly expensive big data tools. Make sure you have a foundation in place to utilize learnings based on strategic questions and goals. Start with basic tools and grow from there.

6. Too Much Data

Do not think too big. Big data deserves the buzz, but start small in leveraging the data results. These projects can be very expensive and create recurring costs. Start by solving real problems and expand on the solutions and you build out.

Also, make sure you are data-informed and not data-driven. Giving your data too much decision-making power can lead to ignoring impactful opportunities that are not in the center of the data tools. Use the data as an influence on your journey, not as the final voice in your roadmap.