Everyone likes chasing big shiny objects all the time. What's not to like. They are big. They are shiny. :)But a lot of progress in life comes from doing the everyday small things better. A small improvement, every single day, to move the ball a little more forward.
A best practice I've developed is to take a step back and reflect on if I have a good balance between chasing shiny objects and making incremental progress on small every day things.
Today's post is from the latter category. It is a story about dealing with a complicated everyday challenge in an incrementally better way.
It is not quite about web analytics, not quite about data you can just get from WebTrends or Google Analytics, and not quite about just making yet another pretty chart. Rather it is about trying to step back and to think differently about a particular problem and, this is so scary, fighting to not simplify things to the point of uselessness.
I deeply wish that the problem we confront in this post did not exist at all. I wish we did not have this desire to pulpify a bunch of metrics to produce something recognizable. But it does exist, and so let us figure out how to incrementally improve the insights we can deliver.
Here's a summary of our journey in this delightful post…
~ What are Compound Metrics?
~ The Challenge with Compound Metrics: Social Media Edition
~ Compound Metrics "Decrapification" Strategy: Social: Standard Approach
~ Compound Metrics "Decrapification" Strategy: Social: Awesome Approach
~ Compound Metrics "Decrapification" Strategy: Search: Awesome Approach
~ The Challenge with Compound Metrics: Social Media Edition
~ Compound Metrics "Decrapification" Strategy: Social: Standard Approach
~ Compound Metrics "Decrapification" Strategy: Social: Awesome Approach
~ Compound Metrics "Decrapification" Strategy: Search: Awesome Approach
Ready? Let's go…
What are Compound Metrics?
Compound metrics are everywhere. An example from the world of sports is the NFL Passer Rating (don't worry, no one really understands what it is). In a government context, Consumer Confidence Index and Consumer Price Index are good examples. One that you likely care about a lot is your FICO credit score. Perhaps the most famous digital compound metric is PageRank .
In a digital analytics context a compound metric might be…
Visit Quality = [(% of downloads + % of visits with more than 68 pageviews - % of visits with less 2 pages + % visits with Facebook shares + % of visits with store searches)/5)*1476]
The goal would be to communicate, perhaps, the quality of the site or the quality of each visit or quality of the outcomes, to the Dear Leader in your company using an "one simple easy metric."
I understand why people want to create compound metrics.
We have too much data on the web. We can measure way too much nuance, and sub-nuance and sub-sub-nuance. Because there is so much, and so much of our leadership is full of simple minds (not a ding, we should not expect them to be Analysts), the instinct is to "make it all simple, make it into one simple number" in the hope that they'll understand what's going on.
The intent behind creating compound metrics is good, sadly the outcome never is.
The Challenge with Compound Metrics: Social Media Edition
Go back and look at the Visit Quality compound metric someone's defined above. Do you see the problem? Do you see why it would be nearly impossible for you to look at the number that get's spit out every day/week/month and learn almost nothing from it – even if the number moves every time ?
For a bit more on the other reasons please see: Actively avoid insights: Use Compound Metrics
Here's a summary: When you cross-breed a bunch of metrics to produce a, I'm being charitable here, hybrid "simple number," the process, by design, hides insights, hinders the ability to understand performance and almost never allows the management team to identify root-causes.
Let's look at a practical example that demonstrates why I'm allergic to compound metrics.
Tool X below computes a social influence score for me. I'm a 83.
(Even if you can recognize this tool can I please request that you not name it in comments? They are hardly unique in creating compound metrics in the social analytics space. Thank you.)
Nothing.
I just know that I'm 83 and a peer of mine is 87 and another one is 76. So what? More importantly, what do I do now?
There is a trend in the image. I can see it goes up and down.
Why does it do that? Not clear at all.
What contributes to 83 that causes the trend to go up and down, or for that matter what causes the 87?
No clarity.
So what do we do with the 83? Well… It depends on what you are measuring. Oh, it's not clear what it is measuring. Hmm… so it is useless?
They do share how much of this influence is from my presence on Twitter, Google+, Facebook and Wikipedia. Oh… Maybe that helps?
So what's the 83 and what can you do with it?
Nothing.
This is the problem with compound metrics. You don't know what causes the final number to happen. You don't know the focusing factors, hence you are blind to why things go up/down and what actions you can take to improve.
In this case this Tool X is deliberately hiding all the focusing factors because they don't want you to game the system. I respect that. But for me as the recipient of this "boiled down from all the complexity simple metric that anyone can understand" it is a little less than useless.
Compound metrics. #arrrrhhhh
Let's look at another example and bring the problem closer to a company level challenge.
(Again, I just pulled a random example. If you know the tool or the provider, please do not mention it in comments. They are not unique in being guilty. Thank you.)
Tool Y shares this comparison of two companies.
Yes. One seems high and other other seems low, in extremely tiny percentage terms.
But can you learn anything about what Mercedes-Benz is doing right or wrong? How about Audi? What might they be doing right or wrong?
Nothing. Zip. Nada.
You don't even know if 0.670% is awesome or completely sucky. You might if you knew that the upper limit or benchmark for Engagement Rate is 100%. That would give you incredible context. But you don't get that above. That in turn implies you don't even know if Mercedes-Benz is just a little less than awesome or really totally completely sucky!
What if you were kindly told by Tool Y that this is the formula used to compute Engagement Rate…
Does it help?
Now can you tell me what 0.290% and 0.670% mean?
And, always more importantly, why 0.670% became 0.670%? What caused Audi's number to be higher than MB?
Let's say this was not Audi and MB.
In April your Website Engagement Rate (WER) was 29 and in May your WER was 67. Would the score of that compound metric be able to tell you anything about what you did better or worse? Would your management team know what action to take when you plonk this compound metric in a dashboard?
Extremely highly improbable. Hence my allergy to compound metrics.
[PS: The above formula is not even the most sub-optimal one. There are social/search/website engagement/quality/experience compound metrics that are much, much worse. Those exquisite beauties contain weights, multipliers and other sundry "values" attached to each element in the formula. "Values" that are sourced from nothing more than our biases and blind spots. The end result is unique type of awful that you have to experience to believe.]
Compound Metrics "Decrapification" Strategy: Social: Standard Approach
If you can avoid compound metrics, that is the optimal. Rather can creating unrecognizable pulp, use the Digital Marketing and Measurement Model process to identify the best direct key performance metric. Use them to deliver insights that directly drive ultimate business profitability.
In our social scenario above I'll take inspiration from the Digital Marketing and Measurement Model process to create an alternative simple approach to using compound metrics.
Rather than "simplify" things and put five metrics into a blender and puke out an "easy to understand" number, my strategy would be to expose the focusing factors in order to encourage our leadership to look a little deeper to understand performance.
Instead of calculating the "Average Social Engagement Rate," I would much rather (as recommended in the Best Social Media Metrics post) show the management team this…
The first three, Conversation Rate, Amplification Rate, Applause Rate, show, wait for it…. engagement (!) on the social channels. As you look across you can see how well or badly you are doing on individual channels and the last row beautifully shows how much money we make off each social visit to your digital existence.
Sweet, right?
What about Audi and Mercedes-Benz?
Instead of the cute infographic compound metric view with two other value deficient metrics, number of fans and total "interactions," would it not be magnificently been better to show this…
But would you agree that the above view makes it really, really, easy to understand social engagement AND social's impact on the business of each company?
So, if you are willing to trade a small amount of pain (the discussion you have to have with your Dear Leader that a "one simple easy metric" might actually not be so simple or easy or valuable, and that you'll show they a bit more data), then it is entirely possible to completely avoid having to use a compound metric. Imagine the waves of pleasure that will wash over your body because you no longer have to create a crazy formula based on little more than biases and blind spots (exemplified by the Engagement Rate formula above).
Oh, and this works beautifully even if you look at just one company and their performance over a period of time.
Let's say we work for Audi (hurray!) and want to show social performance. Rather than saying the Average Engagement Rate was 0.67%, 0.68%, 0.57%, we can just show this table….
Progress. Prosperity.
Compound Metrics "Decrapification" Strategy: Social: Awesome Approach
What if you are in a situation where your good intentions are insufficient?
What if you are a little minion, but a poor humble soul, who no one cares about and who's being whipped into creating compound metrics?
It does happen. Sometimes you simply can't avoid it. There is too much pressure to "simplify" things.
What do you do? How to ensure that you still deliver something of value?
Here's my suggestion: Give them want they want, but package with it what they need. Oh and while they are at it, include the one disruptive thing that will give them critical performance context!
Show them this….
But we don't stop there! We also include something that is incredibly valuable for Dear Leaders around the world in trying to understand performance: We include critical context to illustrate if the performance seen is good or bad.
For the last part (the big gray box) I've used 1. an available benchmark for the metric or 2. an upper limit I'm aware of or have computed from existing data or 3. a target we as a company are shooting for.
Now the Dear Leader has what they wanted, but we've also provided what they needed to go from data to action.
It should be quite obvious why your company will make smarter decisions now.
Oh, and you don't have to use my best social media metrics . You can use anything you want for your formula that computes your compound metric.
For example you can do this….
The point is, you should embrace the framework. You can use any reasonably relevant collection of metrics, for the compound metric you are trying to create. Just make sure your focusing factors (metrics you use) are as complete as possible, and ideally independent from each other.
Switching back to my original four focusing factors… you can now see exactly how this would look like if we wanted to compare the Social Engagement Scores of Audi and Mercedes-Benz…
Even though the compound metric is there on top, you'll deliver a deeper understanding of why Mercedes is at 0.290% and Audi is at 0.670%. Both companies can learn from the other and figure out how to fix their social strategy to rock more.
And remember the above visualization could be for the month of May and June for one company. Still works beautifully in explaining the performance of the "one simple easy metric."
Compound Metrics "Decrapification" Strategy: Search: Awesome Approach
As I'd mentioned earlier, this approach could work in other contexts as well – any time you end up with a compound metric.
You follow the same three step process for decrapification outlined above:
1. Show the compound metric.
2. Show the contributing metric's performance.
3. Show that against a benchmark or upper limit or target.
2. Show the contributing metric's performance.
3. Show that against a benchmark or upper limit or target.
Step three above is something I've started to do a lot recently, my tiny incremental innovation if you will.
Let's look at one more example to really solidify this three step process for you. Since we've beaten Social to death already, let's look at another exiting area: Search.
This example is one where I was unable to run away from a compound metric. I had to create one. Boo!
The mandate was to try and figure out how to show any company their "overall Search performance" using a "one simple easy metric."
How is that even possible given the insane complexity that is the digital Search business? But mandates demand that we deliver, so off we go.
Here's my humble attempt at a Search Performance Index (SPI)…
I'll let your imagination run wild as to how the score itself is computed. (If there is interest, I'll share it in a future blog post.)
The cool part is not that there is an overall score that the Dear Leader can look at and be thrilled with. The cool part is that she/he will be forced to glance, even if for a moment, at the focusing factors that explain why the Search Performance Index is at 643. The coolest part is that they'll know which area needs more attention because of the distance between the current performance and what's possible (the benchmark/upper limit/target).
Now it is easy for them to prioritize where we should devote resources to improve the SPI. Action!
That is, dear blog readers, to borrow a phrase, priceless.
I'm sure you noticed that while there is a lot of data behind the simple visualization above, there is almost no data overtly present. Just a bunch of bars.
That is by design. Sharing a more nuanced view of reality does not have to be complicated. Take everything away, until you just have the things you absolutely need left.
Once you have that nailed the focusing factors and the compound metric formula, it is easy to do cool stuff with it (if you have access to the data).
You can look at the compound metric across competitors and smile (or cry)….
And even if you are ZQ Inc (643) you can see that you are at least good at one thing (ad creativity) and you can figure out how to use that strength – even as you fix the other focusing factors .
You can/should do this to understand your own performance over time. And you don't have to use my choices above. I want to stress the process and not prescribe the metrics you should use.
Internalize the process. Spend some time understanding your own unique business needs, get advice from experts out there and create an extra-special magical formula that you believe will deliver glorious insights. Remove the metric you don't need above, plonk in your new BFF.
The next time your Awesome Search Performance Agency is doing their quarterly review, ask them to open their presentation with this view of the SPI….
You are happy because you snuck in just enough information to make the index easier to understand.
Your Agency is delighted because you are asking for a much more sophisticated understanding of reality to show the value of your huge investment in the Agency and Search. Agencies want their feet held to the fire, they love accountability.
Win – win – win.
Summary
So often we report (puke) data, we diligently produce (puke) dashboards, we lovingly create (puke) slides. But it is rare that our efforts drive action. It's not because our puking was not great, it is simply because we are unable to succinctly bring forth the underlying focusing factors and present them in a way that is accretive to quick internalization of which of the focusing factors need killing and which need feeding.
Solve for that.
And now you know how.
Ok, it is your turn.
Do you agree that compound metrics are sub-optimal? Are there compound metrics your company uses that you've simply failed to kill? Or other compound metrics that are your favorite? How do you deal with the insights that delivery actionability challenge that mashing five metrics into one to produce a paste presents? Do you think showing the focusing factors beneath the "one simple easy metric" will work inside your company? How can I make the above approach even better?
Please share your ideas, critique, praise, improvements, examples in comments below.
Thank you.
Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics is a post from: Occam's Razor by Avinash Kaushik
DIGITAL JUICE
No comments:
Post a Comment
Thank's!