Metrics matter. The purpose of this article is aimed at how one would take raw data and use it build metrics to make business decisions, confirm quality, and provide feedback into operations. Good metrics can inform decisions related to reducing video distribution costs, improve video quality, predict traffic, and gain competitive advantages for distributors that stream the same content. Ensuring video quality-of-experience is not just based upon measuring buffering incidents, video startup time, failures/exits, and bitrates delivered. The idea is to take these measurements and overlay them on various dimensions to help paint a better picture of the consumer experience, specific to the user and general to the viewing population.
Datazoom captures and transports the raw data that drives quality of experience tools and is the basis for analytics. More importantly, data is captured in near real-time. Detecting and mitigating anomalies quickly, especially in the area of live streaming, is an important factor for a successful video streaming business. Real time data enables streaming video providers to create feedback loops that enable critical configuration adjustments, which ultimately allow them to mimic and then surpass the quality of a traditional television broadcast. This is especially important in a world where there is choice for a subscriber.
- Consider that streaming video is delivered over unreliable networks. Momentary outages, traffic prioritization, and last-mile concerns are all challenges of a public distribution network.
- Decide what's important. Measuring the number of video starts might be important, but what happens when you consider video completions? Is a started video that was completed more valuable when considering the content of the video and ultimately user retention? Are your users sampling then leaving because they were disinterested in the video subject and never coming back? What of actual Quality of Experience? Are users abandoning because of rebuffering instances, jarring bitrate shifts, or slow startup times?
- Set targets
- Establish baselines. (example: low percent buffering and highest bitrate at the expense of slow video startup time)
- Establish thresholds (example: number of failed video starts over x minutes. What number of failures can you tolerate given item 1?)
- Consider live content versus on-demand content and how viewing behaviors will impact measurement.
- Consider the viewing platform and how a customer is using it. For example, it's not uncommon for a consumer using a web browser to background video while they are doing something else. How does backgrounding video or switching tabs while a video is playing manifest itself in your QoE metrics?
- Decide on outliers to remove from QoE calculations. When considering video start failures or exits before video starts, does a failure for an auto-started clip count as a failure when a user swipes away to another part of your app before the clip started? Knowing user behaviors can help you fine-tune your alerting thresholds.
- Efficient encode profiles. Invest in perceptual video metrics tools like SSIMWave. Feedback from these video quality tools will assist in tuning your encode profile to match devices to quality perception.
This is a fairly simple metric to measure, until you consider the variables that can impact your analysis. You'll need to understand your playback environment, per platform, to get a good idea of what that number means. When does the startup-time clock begin? When the player fully loads, the first 3 chunks of video are cached, and after the play button is pressed? Or, is it the time difference between when the play button is pressed and the first frame of video appears? Do you consider ad pre-roll in video startup or does the clock start after the ad completes? Once you understand startup time behavior for your platforms, you can then begin to add dimensions like startup time by device, by stream type, by region, by connection type, by bitrate, or by number of concurrent attempts. Each of these dimensions can have an impact.
It's safe to say that any stalled playback results in unhappy consumers. This is especially impactful when viewing a live event like an awards show or sporting event. When the stream returns, the viewer can be pushed further back from "live". That could expose the viewer to "spoilers", when a comment on social media is posted about something that the particular viewer hasn't seen yet, but has already happened.
Things you can do with rebuffering data:
- count the number of rebuffering events that happened during a given period of time. (example: 4 rebuffering events in the last minute)
- calculate the length of time the total number of rebuffering events happened during a given period of time (example:10 seconds of rebuffering in the last minute)
- calculate the percentage of time a consumer is experiencing rebuffering (example: 10 seconds of rebuffering over the past 60 seconds)
- calculate rebuffering as a percentage of total video playback time (example: 90 seconds of buffering over total video playback time)
Having rebuffering data and using it as a reference as you make changes to your player, encode, and/or distribution platform will help inform you if the changes are making a positive impact to reduce rebuffering. For example making changes to your encode profile that delivers the least amount of bits at a given bitrate to achieve the best possible perceived quality will help in reducing rebuffering.
"Playback Failures" is a general term that is used when there was something that caused a video to fail. This can be ads or your content. You can do better than that. Your player will have specific error codes. Use them. Further, collaborating with your CDN partner will allow you to know what to look for in request/response headers that can help understand distribution related errors.
While not playback specific, Authentication errors prevent a consumer from getting to video. Tracking those is equally important to understand the health of your authentication and authorization services. Every service provider needs to assess for themselves if it's more critical to get the consumer to a playable video experience, bypassing a malfunctioning authentication service or adhering to strict authorization protocols.
Considering playback errors during an ad break is critical. This is especially important in the a world with ad exchanges. VAST calls to an ad that calls another ad that calls another ad are common. Capturing advertising error codes in real-time is important as the data can be fed back to your ad operations team to mitigate revenue loss and to fix what could be a negative user experience.
Abandonment (Exit before video start)
Data for this metric relates to the percentage of consumers that have clicked the play button, but have decided to move on and not watch before the video has started. Understanding how this data is represented will help you better know how to address. This metric generally relates to long startup times. That said, if startup times are short and abandonment is high, you might have a case where consumers are dropping during an ad pre-roll.
Bitrate and Upshifts/Downshifts
Understanding the quality of video you are delivering is important for a number of reasons. Primarily you would like to deliver the best quality (brand rep) at the lowest given bitrate (lower cost) as quickly as possible (positive consumer experience). Given the current nature of content delivery via unmanaged networks, video delivery is designed to take this variability into consideration. Therefore you can measure things like the number of Upshifts/Downshifts over a given period of time. This could expose areas in the distribution chain that aren't optimized.