Overfitting: is your website "too good"?

By Mischa Andrews

When we focus too much on imperfect measurements – and many of our measurements are imperfect – we can fall victim to "overfitting" and miss our goals completely.

To illustrate overfitting in the everyday world, let's imagine you're trying to be more healthy. There are many different ways to measure what "healthy" looks like, but it's difficult to get started unless we pick some kind of measurement. So, let's say you decide to aim for a particular weight. By using the metric of body weight as a proxy for your health, you can trick yourself into thinking you're becoming more healthy by skipping important meals or taking unhealthy supplements. By your chosen method of measurement, you may have succeeded. But by your original goals, you may have failed.

Similarly, without stepping back and re-evaluating how you measure success for your website and digital communications, you can get caught chasing the numbers at the cost of success.

So how can you avoid overfitting?

First, acknowledge that web and social analytics are imperfect measurements. You probably know this already: one metric that's well known for its imperfections is website page views. When you have more people loading pages on your website, that might mean more engagement or new traffic – but it can also mean your users are having difficulty finding information, or have come to the wrong website altogether. As a metric, page views are well known for being imperfect, but plenty of our other metrics are imperfect in some way or in some context too. High engagement on social media, for example, can indicate both positive and negative feelings.

Next, analyse change with more than just your key metrics in mind. If your metrics show some spike in activity try not to immediately assume that's good or bad based on your key metrics. Cross-check against other measures or conduct user research to try to understand that change.

Finally, recognise when your analytics are failing and redefine how you measure success. Key metrics should change as your business grows, or your technology changes, or users' expectations change.

Web and social analytics are an integral part of effective digital communication. But doing whatever you can to meet your target metrics won’t always lead to a good outcome. With occasional reflection, thoughtful investigation and the courage to change, you can avoid overfitting – and find success.

If you want to talk more analytics, get in touch with us! Maria Sjögren, Consultant at +46 76 109 05 55 or maria.sjogren@comprend.com.

This article was inspired by Algorithms to Live By: the Computer Science of Human Decisions by Brian Christian and Tom Griffiths.