I got bored today and took out the old spread sheet and tried to see if I could get some numerical support for some of the hunches I had with how well Klout and PeerIndex correlated with various Twitter metrics.
For the purposes of this test, I took the 50 accounts from the InfluencersinTravel.com list of independent travel influencers.
I then created a spreadsheet in Excel and imported the following data for all 50 people on the list:
- Klout Score
- PeerIndex Score
- Number of Tweets
- Number of Followers
- Number of people following
- Ratio of followers to following
- Number of Twitter lists
(4 of the 50 people did not have PeerIndex accounts or did not make their data public)
I then plotted the data for Klout and PeerIndex scores vs the various metrics and put a trendline on the data with a corresponding R^2 value. A value of 1.00 would be a perfect correlation.
Correlation with Followers
The most obvious thing you’d think would factor into a score would be the number of followers you have. Here is the data:
PeerIndex scored correlate much more strongly than Klout scores with the number of followers you have. In fact, the correlation with Klout seems to be so low that it looks like Klout doesn’t even use it as a metic at all. The correlation between Klout score and followers was the lowest of anything I measured.
On one hand, I think that the size of your audience does in fact have something to do with your influence. After all, if you are only influential with 20 people, you aren’t really that influential.
On the other hand, there are many people on Twitter with inflated numbers of followers. This is usually done by auto-following whoever follows you. You can usually filter this out by looking at the followers/following ratio. That is what I looked at next:
Follower/Following Ratio
The key here to remember is that a high ratio means you have many more people following you than you are following.
Again, there is almost no correlation with Klout scores with the ratio. It correlates slight more than with the number of followers you have, but not much. You can follow every spammer, bot and porn star and Klout doesn’t hold it against you. (Read about how one guy got a bot to get a Klout score over 60)
Number of people following
The next thing I checked is how score correlates with the number of people you follow. I wasn’t even sure I should do this because I’m sure neither company actually uses the number of people you follow as a positive metric. It would make no sense and would be very easy to game. I figured it would be a very small number for each service.
I was wrong:
Wow. Klout correlates FAR more with the number of people you follow than the number of people who follow you!! Either there is something really messed up with the data set I used, or there is something really wrong with Klout.
PeerIndex score correlates less with following than it does with followers (which makes sense), but the difference isn’t that great and it is half that of Klout.
Twitter Lists
Next I checked Twitter lists. Someone who is influential you would think would be placed on Twitter lists more than someone without influence.
Both scores correlate high with Twitter lists, which is what you would expect. PeerIndex has a higher correlation, however. Almost 50% higher.
Tweets
Finally I looked at the thing which I suspected would correlate the highest: number of tweets. One thing that I don’t have data for is retweets. The closest thing which would approximate that with the data I had was the total number of tweets. If you assume an average number of retweets per tweet across all of Twitter, then it would make sense that total tweets would correlate with retweets.
The correlation between Klout score and total tweets was the highest of anything I measured. It was 4x greater than the correlation with PeerIndex.
I have personally observed many people who stopped tweeting briefly because of a trip and saw a drop in their Klout score. I wasn’t surprised to see a high correlation.
Conclusion
If you want to increase your Klout score you should follow as many people as you can and tweet as much as possible. The person with the highest Klout score on this list has both the highest number of tweets and also follows the largest number of people. Both tweeting and following people are things which are under control of the user.
If you want to increase your PeerIndex score, get people to follow you and put you on Twitter lists. These things are not under the control of the user, so it would seem much harder to game PeerIndex. The person with the highest PeerIndex score had the largest number of follower, second most lists and the highest ratio.
While any measuring of “influence” is inherently subjective in terms of what you think determines influence, PeerIndex seems to me to be better than Klout if for no other reasons other than it correlates with things which make more sense in terms of influence. The highest factors that correlate with Klout score and factors which anyone can control themselves. This makes it open to gaming.
Notes
- All the data I used is available here on Google Docs.
- 4 people did not have PeerIndex data.
- One user had over 10x the number of followers than anyone else. It was such a larger outlier I removed the data point to see how much it effected overall correlation. Because it was only 1/50th of the datapoints, it had very little impact so I left the data point in.
- Klout also considers personal Facebook accounts in determining its score.
- PeerIndex considers Facebook, LinkedIn and other sites in determing its score.
9 replies on “Klout vs PeerIndex”
Hi, I am the ceo and one of the founders of Klout. This was a fun article, thanks for taking the time to look at our data.
Having a little bit of insight into our algorithm there are a few notes I can add:
– You are right, we don’t care how many followers you have. We have found that this isn’t a good indication of influence as it’s too easily gamed.
– The number of people you follow does not play into the algorithm at all. Not sure why that is showing up as being highly correlated.
– We believe that influence is the ability to drive action. The reason you see total tweets as being correlated to Klout scores is that a person needs to consistently engage with their audience in order to build influence. If your audience isn’t responding to the tweets you put out though your score will actually drop. To be clear here, just tweeting will never raise your Klout score. If people interact with your content your score will rise.
Thanks!
Joe
Joe, I’ve read the comments you’ve left on several blogs who mention Klout. My issues are the following:
– If people are so influential, how come they can’t get traffic to their blog or fans on Facebook? Real action is getting people to click and read, not just retweet. If you are getting retweets on someone else’s content, who has the real influence: the person with the tweet or the person with the compelling content?
The problem is, Klout defines engagement by what is measurable, not by what really matters.
– Who are your followers actually matters a great deal. Oprah is extremely influential even if she doesn’t interact with her fans. You are correct that raw numbers are gameable on twitter, but who your followers are is not. If Robert Scoble or Bill Gates follows you, that means more than if a spammer or a bot follows you.
– Likewise, if most of your followers are bots and spammers, why should you be rewarded if you are just getting auto retwetted? I’m not sure why following tens of thousands of spammers shouldn’t be a negative mark on an influence score. If Klout to the lead in penalizing those who autofollow tens of thousands of people, it might go a long way in cleaning up the Twitter world.
– By focusing on retweets, it takes emphasis on what is really important…what is being tweeted. The fact that someone is tweeting someone else’s content doesn’t make them influential. The person who’s content is being tweeted has the real influence. That is like saying the person who quotes other people is smarter than the person who originally came up with the quote.
– Influence doesn’t go away when you go on vacation. I’m in the travel space. I know many people who go on trips and are away from the internet for an extended period of time. Whenever they do, their Klout score dips. Their influence hasn’t dipped however, just their short term Twitter activity. So long as you are just counting retweets, you wind up with the nonsensical idea that influence can go up and down over a period of says. Influence takes a long time to build and doesn’t disappear over night.
– Twitter doesn’t exist in a vacuum. A person will real influence might have a popular Facebook fan page, website or other online outposts.
As Klout is now, it is just a retweet index. It isn’t influence.
While I find this to be a very informative post, I think we need to separate the concepts of causality and correlation. This post indicates “correlation” very clearly, but then seems to draw conclusions based on an underlying assumption of causality. To say “something really wrong with Klout” for instance due only to a correlation doesn’t make sense. Even Joe said in his comment here that he is not sure why following count is highly correlated with Klout score. Yet I think this is valuable information – and instead of chalking this up to a problem with klout, we should be thinking about why other metrics klout does use end up correlating in this fashion – and how this can be turned into meaningful insight for behavior.
I do like the point made here that audience size does matter to some extent – e.g. if you only have 20 followers, it is extremely unlikely that you are influential. I’d expect klout to have some kind of lower-bound threshold along these lines, but on the other hand it probably doesn’t need it, as its other metrics already correlate appropriately for this scenario.
One last thing to consider: the sample size and restriction to a group in a fairly narrow domain (travel influencers) makes it difficult to really draw any true conclusions from the correlations.
I think it’s perhaps more interesting to consider why anybody cares about a proprietary metric that is so obviously inaccurate. Why are we even discussing “Klout?”
I think the answer is that, in the social media/marketing community, there are a lot of freelancers who are desperate for validation of their consulting services. Quite understandably, they really need some way of differentiating themselves. Klout fills this need.
So you have social media “experts” seeking validation from an absurd scoring system, which then in turn benefits from self-congratulatory tweeting of those “experts.” I wouldn’t call this a scam, but it’s pretty irrelevant to the real world.
Gary
Thanks for this great piece of analysis – it’s good that you took the time to do the analysis.
I apologise for the delay in getting back to you on this – I have been travelling for the bulk of the day without reasonable and reliable internet access.
So – to be clear, all the bulk of factors we consider in our calculations as observed characteristics dependent on how other people respond to you. We don’t look at raw list counts or how many people you follow (except in certain special instances). We generally look at things that are much harder for you to game without getting *lots* of people to regularly and repeatedly collude with you.
To give you a sense of how we identify which features to consider:
* We used a model of the world (‘a truth’) against which we test the data we sample and analyse, we extract features which appear to explain the truth better than other features.
* We used some robust theoretical models on identifying trusted nodes in graphs as a basis (the model we used is similar to PageRank, adapted slightly to our needs).
We add features and change weights on features relatively frequently.
Some examples (and not by any means exhaustive) of features that we use:
> EigenVector centrality measure (apologies, this section assumes you have a basic grounding in network analysis) – for every user we build a topic-specific graph of the messages they pass in the graph. In other words if users B retweets something from user A that is about travel, we draw an ‘edge’ from B to A. We end up with a topic-graph on, say, travel of all the user who have tweeted/status updated, etc on travel and who responded to whom. We then extract the ‘eigenvector centrality’ (a measure of the trust of authority) of each node – as a feature. (As a centrality measure, EC walks the whole of the graph – and takes into consideration the trust levels of all the nodes connected to it).
To increase this measure, you need to increase the number and quality of actions people take on your tweets / status updates.
> @ mentions to you – we look at the number of people who @ mention you (both in absolute terms and as a proportion of your follower base)
> Follower/friend ratio – we look at the follower friend ratio
> Actvitity rates in that topic relative to all topics
> You can see some of the other things we measure on your PeerIndex dashboard. (Lots of other thing we measure – like list memberships we don’t yet show on the dashboard)
Additionally measure your PeerIndex on a series of sliding historical windows (measured in months rather than days); although we recalculate as often as nightly.
The correlations you will have seen will be a result of the underlying structures in the world. Someone who generates lots of high quality retweets and engagement will have a high PeerIndex. They are also likely to have a large number of list memberships – because other people will have responded to their quality tweeting by adding them to lists.
Likewise with the correlations to the number of followers – again not something we generally measure directly (although sometimes it flags spam accounts) – but simply an artifact that could quality tweeters get retweeted and so increase their follow base. To Eric’s point – there may be correlations, but they don’t necessarily imply causality.
A final observation is that the data set ends up being a bit limited. And you may find that it’s not a representative sample of the population.
Thanks for this great piece of work! Talk soon.
Azeem,
I’m aware the data set I’m using is pretty limited. This was sort of a back of the envelope calculation. To do more I think I’d have to write something to access API’s and get access to several thousand random accounts at minimum.
I think the fact that they are travel people doesn’t matter, but I could be wrong and I can’t prove that I’m right.
This started because I noticed, via anecdotal observation, that people who auto-followed and tweeted a LOT had very high scores. Higher than would be expected based on their standing based on content creation, off-line influence and personal standing.
Because Klout and PeerIndex value retweets so highly, if you can game a ton of followers and then throw enough crap against the wall, you’ll get retweets. That correlates with following people and tweeting.
When dealing with ranking social media profiles I can’t help but think of what Albert Einstein said: Not everything that can be counted counts, and not everything that counts can be counted.
HI Gary
Couple of thoughts:
1. I think it matters less that they are all travel, but more that they are all at the top of your list – so what is a useful secondary exercise would be to select 50 random names within travel. (basically, it’s the bias in the sample set – as if you surveyed 50 American nobel prizewinners to make an estimate of how good the education system was)
2. Followers does raise retweets, but less than you would imagine within PeerIndex – the reason is that we run our algorithm’s recursively so you generally need to be @’d or RT’d by people who have high status (rather like on the Web, 1 link for CNN counts for more than 100 links from spam sites). (I say generally – there is of course some noise in the index which we aim to stamp out).
Yeah – agree you probably don’t need API access – in fact let me think on it and figure out if there is a fun test we can do.
This dynamic dialogue shows why we cannot often quantify what really matters – and how we often pretend to show results in quantified statements.
It’s fun for those who enjoy numbers as many of us do. It’s also dangerous if we assume we can use numerics to show human value and intellectual acumen. It takes qualitative extensions to illustrate value.
The servant leader – for instance – engages with far less demands on others and far more capability to open opportunities for all. Numbers lose that nuance:-) Still, it’s fun – and it’s also a reminder of how we warped many systems that slow down innovation in this era.
Thanks for the interesting post!
Late to the discussion, but…
I really liked Rivas’s comment about Klout itself and why it has gained traction:
“I think the answer is that, in the social media/marketing community, there are a lot of freelancers who are desperate for validation of their consulting services. Quite understandably, they really need some way of differentiating themselves. Klout fills this need.”
I just would add that this need is not simply a made up thing – a bunch of nobodies trying to invent influence. It is rather that without such metrics social media actions lack ANY context in many business conversations. Businesses often are simply responding to a Zeitgeist, when in fact what is needed are hard reasons for positioning oneself. Klout is part of that move of social media towards translating itself into business terms.
The question isn’t whether Klout or Peerindex is “accurate” – for even if they are, one would have to ask: Accurate to what, to what end? – it is whether social media would be better off without it/them. As aggravating as Klout is – and I pay as much attention to someone’s Klout score as I do their profile pic, less actually – it has driven the conversation in a positive direction. It is ONE factor among many. Peerindex I pay even less attention to, not because it is somehow less “accurate” than Klout, but because it is playing an even lesser role in the conversation.
To return to the first point Rivas makes, I think we need to remember that social media is a SOCIAL media. That is one of the best things that it brings forward. And so how social media types feel about a metric, how it impacts them and how they interact indeed is an important factor. Klout and Peerindex are relatively deceptive metrics lets admit. They are like a gold start system. So are far cruder but still powerful metrics like “followers” listed under your profile pic in Tweetdeck. But in fast, dynamic, information dense media shorthands for standing can and do play a role. I’m all for correcting these metrics towards the more important factors – and this article and other push-backs force these products to improve. But one of the lessons of social media is that statistical measures have emergent properties, and how these emergent properties are represented to people involved in that medium is an important and useful social dynamic, even when flawed.
Kevin
@mediasres