In response to the video views metrics misreporting scandal in August, Facebook has today revealed several other miscalculations and errors it has found in its reporting, alongside a raft of improvement measures it believes will offer more transparency, “clarity and confidence” in its metrics.
Confidence in the robustness of Facebook’s metrics was shaken in August when it emerged video views had been over-reported by as much as 80% for two years. Today (17 November) it has acknowledged at least four more reporting errors including over-stating time spent with Instant Articles, over-reporting organic reach counts, discrepancy in video completions and over-stated app referrals.
Ahead of the announcement, Carolyn Everson, Facebook’s vice president of global marketing solutions, tells AdNews the changes are about making sure marketers can be confident in the platform’s metrics.
“Feedback [from clients] is around making sure we are consistent with communications and clients can trust the validity of the metrics we put out so they can continue to invest with confidence,” she says.
Errors found in metrics
• A bug in organic reach page insights since May 2016 – page reach was miscalculated as repeat visitors were not de-duplicated. Seven-day averages will be 33% lower while 28-day averages will now be 55% lower in the new corrected measurement.
• A discrepancy in video completion lengths which undercounts the ‘video metrics at 100%’ metric. The fix may increase reported watches by 35%.
• Average time spent in Instant Articles has been over-reported by 7-8% since August 2015. There was a “calculation error” as Facebook was calculating the average time spent across a ‘histogram’ of time sent – not a total time sent divided by total views.
• App referrals was “miscalculated”. It was meant to count clicks that went directory to a client’s app or website – but it counted all clicks on content posted produced by apps – even that which opened directly within Facebook. At the top end this was overstated by 6%.
It's worth noting that while none of the errors were on a directly billable metric, each one would have impacted decisions marketers made on where to invest budgets and in how they view the strength and success of content and ads on Facebook.
Likewise, publishers should be concerned about the overstating of Instant Article traffic. The offering was launched as the solution to publisher woes in terms of monetising content on Facebook, whereby the social network has control of its audience and traffic, making it more difficult to monetise in their own platforms.
Threefold fixes
There are three components Facebook is putting into effect today. Firstly, it is setting up a global measurement council, secondly it is expanding its third party measurement and thirdly it is committing to better and more transparent communication around metrics.
The global measurement council will operate similarly to its global client ad creative councils which meet regularly to discuss issues affecting advertisers and how Facebook can improve its operations.
“[The client council] has been a great mechanism for Facebook to get feedback from clients over the past six years so it’s a logical extension for us to be a listening company,” says Everson.
Rising pressure for independent verification
Since 2014 Facebook has been working with partners such as Moat, Integral Ad Science, comScore and Nielsen on measuring areas such as video. It is now adding third party oversight across display advertising “so all areas can be verified”.
It is also integrating Facebook video and Live video with Nielsen’s Digital Content Ratings to “allow for comparable digital and TV metrics” in Nielsen’s Total Audience Measurement.
Third party measurement has been a huge issue for agency partners who have long called for better, independent, verification from sources outside Facebook.
There have been rising concerns among marketers and agencies about the lack of independent verification of Facebook’s numbers with many saying that unless advertisers put pressure on the social network to make improvements, it would continue to operate its walled garden/black box approach. Senior global figures including Unilever’s Keith Weed and Sir Martin Sorrell have accused Facebook of marking its own homework and raised concerns over TV and video view comparisons.
Earlier this year, when Facebook’s video misreporting was revealed, one senior agency exec told AdNews: “The danger is that these digital publishers can cut the numbers whatever way they like in their favour. It just lacks integrity.”
Commitment to transparency
Facebook is also today launching the Metrics FYI Blog, which Everson says is intended to be a “fully transparent” and ongoing channel of communication to keep advertisers and clients informed about updates and changes to metrics and measurements.
“On a strategic level, the FYI blog is a commitment to having a source of truth where we can have ongoing updates to the marketing community about metrics as they evolve and we develop our products. We wanted to demonstrate that this is the level of transparency we are going to operate going forward,” she says.
The first post on the blog outlines a litany of errors found amid a “thorough review” of Facebook’s some 220 metrics which was undertaken following the August revelations of video misreporting.
Everson described it as a “handful” of metrics where errors have been identified but the blog post describes no less than four instances of “bugs, discrepancies, miscalculations, over-reporting, under-reporting, double-counting, redefinitions and clarifications”.
Everson was clear about how seriously Facebook takes the errors, but also downplayed their extent.
“We undertook a thorough review and I’m proud of the way we approached that with rigour and discipline and the highest level of integrity,” she says.
She also suggested that while the first-time advertisers or publishers are made aware of the errors and miscalculations, it appears more dramatic than it is, reiterating that the errors were not on “billable” metrics.
”What I will say is that some [of the erroneous metrics] are rarely used, some are more so … the first reaction can be ‘whoa’. It sounds big, but when you peel back the onion and understand what it means, [clients] are grateful that we’re being so robust.”
When asked by AdNews why it took the dramatic revelations of two years’ worth of over-reporting video metrics, to come out now, Everson was quick to point out that much of what is being publicly announced now has been in the works since February this year, or even before.
“There is no doubt that we took [The August video misreporting] very seriously and we doubled down our commitment to a thorough review of every metric, but work has been going on since earlier this year on the formation of the measurement council, an ongoing effort on third party verification and looking at our definitions. This is not our first announcement third party verification; it’s been in the works since 2015.
“What changed from August? Look, no one likes to find errors and we certainly didn’t like to find the error in August. What we wanted to do was a thorough review of 220 metrics and we approached that with real intensity over the last two months.
“What I’ve seen in the six years I’ve been at Facebook is that when we double down and get serious, whether it’s pivoting to mobile or whatever the topic of focus is, we approach things with a very significant level of intensity and we did that with this. That is a result of finding an error in August.”
“It will be an ongoing review. We won’t ever be perfect, but we want to be as accurate as possible and fully transparent.”
Comparing apples to apples
One of the biggest complaints against digital measurement, as popularised by Professor Mark Ritson and others this year, is that online video metrics such as those offered by Facebook, appear to show massive numbers in favour of digital over traditional channels such as TV when it comes to so-called views. But the comparisons are completely at odds, particularly when Facebook counts a video view as three-seconds of video being viewable – not necessarily viewed - while television’s long-established metrics offer an average viewer per minute figure.
When asked by AdNews if she believes it is taking enough steps to satisfy critics that say Facebook is “marking its own homework”, she said: “It’s universally recognised that as an industry we [digital] are still nascent. TV has been around 70 years. We have new products and new behaviours and it will continue to evolve. I’m not saying there won’t be a mistake again, we will be rigorous about it, transparent and have ongoing communication, and I really believe that intent [to improve] matters a lot here with marketers.”
When asked about the likelihood of Facebook providing an average per minute audience figure for video to make it truly comparable to television and allow advertisers to compare apples to apples, she said by integrating video and Live video into Nielsen’s Digital Content Ratings Facebook it's making an aggregate metric possible, which is a positive step towards being able to measure across media.
She added there is more to be done by the industry in that respect and she would like to see a move away from proxy metrics such as impressions and more results measured in outcomes for clients.“I don’t think the industry is asking enough about outcomes, and [talking] too much on proxy measures. At the end of the day a CMO has one job – growth. ‘Where should I spend my money to drive growth,’ but it’s hard to change and evolve and industry overnight.”
What Facebook is announcing:
• Increased third party verification – the ability to verify display impression data through third party partners
• Facebook video and Live viewership will be included in Nielsen’s Digital Content Ratings allowing comparable digital and TV metrics in Nielsen’s Total Audience Measurement
• Launching a global Measurement Council to facilitate feedback and co-creation. Nestle will be among brand members. A regional/local council may come into effect down the line.
• More regular and clearer communications on metrics – Launching the Metrics FYI Blog to communicate updates on measurement.
• Updates to clarify ads metrics and definitions: more descriptive names, clarified calculations to reflect how they’re being used by advertisers; more consistent definitions; noting which are estimates; better categorisation in reporting.
• Clarifying the definition of video views metrics to include the term ‘aggregate’.
Other impacts from the updates coming into effect:
• A 5% drop in followers for most profile lists as the Interests Lists feature is being retired.
• Reach counts will be redefined to only count reach once a post reaches a user’s screen – not as previously when a post was placed on the feed. Counts will be re-calculated based on this new viewable impressions definition. Reported reach will drop 20% on average.
What do you think of today's news? Email us and share your views.
Have something to say on this? Share your views in the comments section below. Or if you have a news story or tip-off, drop me a line at rosiebaker@yaffa.com.au
Sign up to the AdNews newsletter, like us on Facebook or follow us on Twitter for breaking stories and campaigns throughout the day. Need a job? Visit adnewsjobs.com.au.