《Akamai:测量视频质量和性能:最佳实践白皮书(英文版)(17页).pdf》由会员分享,可在线阅读,更多相关《Akamai:测量视频质量和性能:最佳实践白皮书(英文版)(17页).pdf(17页珍藏版)》请在三个皮匠报告上搜索。
1、Measuring Video Quality and Performance:Best PracticesVideo delivery performance directly impacts the quality of a viewers experience which has a direct impact on business outcome.WHITE PAPERMeasuring Video Quality and Performance:Best PracticesTable of ContentsMeasuring Video Quality and Performanc
2、e:Why It Matters .2Defining the Success of Digital Video Providers Through Viewer Experience .2Meeting Viewers Quality Expectations .2How Video Quality Impacts Engagement .3Creating a Robust Video Quality and Performance Measurement Strategy .5Creating a Common Language for Measuring Video Quality a
3、nd Performance .6Quality of Experience(QoE)Metrics .7Quality of Service(QoS)Metrics .8Quality of Performance(QoP)Metrics .8Using the Best Methodology to Measure Streaming Quality .9Methodology for Measuring End-User or QoE Metrics .9Measuring CDN Edge Performance or CDN Quality of Performance(QoP).1
4、0Defining the CDN Edge Performance or CDN Quality of Performance(QoP).11Choosing the Right Tools for Effective Measurement .12Quality of Experience(QoE)Measurement Tools .12Quality of Service(QoS)Measurement Tools .13Summary .13Appendix .14Table 1:QoE Metrics for End-User Viewing Experience Measurem
5、ent .14Table 2:Rebuffering Metrics and Definitions .14Table 3:QoS Metrics for CDN Performance Measurement .15Table 4:Synthetic and Real User Monitoring .152Measuring Video Quality and Performance:Best PracticesMeasuring Video Quality and Performance:Why It MattersVideo quality and performance direct
6、ly impact a viewers quality of experience,which directly impacts media organizations business outcomes.In this white paper,we will provide an understanding of the video quality metrics,methodology,and measurement tools,as well as some of the best practices in designing a video quality and performanc
7、e measurement framework that will help your organization improve viewer experience,retain viewership,and ensure that your services can stand up to increasing competition.Defining the Success of Digital Video Providers Through Viewer ExperienceMeeting viewers quality expectations as digital video con
8、sumption is going mainstream is an important success factor for service providers.There is a higher likelihood of churn if your service doesnt provide a better or comparable experience.This section covers the importance of certain video streaming attributes,the impact of quality,and key findings fro
9、m recent research.Meeting Viewers Quality ExpectationsThe U.S.Census Bureau estimates that by 2023,the worlds population will near 8 billion,and digital video viewership will rise more than 39%,from 35%in 2019.1Digital video viewership is on an aggressive growth course globally.Countries like India
10、and China have relatively low broadband penetration,but high rates of mobile usage for online video consumption.On the other hand,Western markets,including the United States and parts of Western Europe,see a strong penetration of connected TV infrastructures.Increased video streaming choices and the
11、 low cost of switching providers are leading viewers to churn if the viewing experience is poor.With over-the-top(OTT)media services rapidly becoming a key part of digital entertainment globally,viewers expect high-quality experiences.“As the online delivery of video matures,so does the requirement
12、to qualitatively measure the service,performance,and experience delivered to the end user.No system can be improved if it cannot be measured.The generation of metrics at each stage of the video delivery value chain,combined with a feedback and monitoring system,allows OTT service providers to evolve
13、 from a best-effort implementation model to a quality-assured,finely tuned,and robust entertainment distribution system.”Will Law,Chief Architect,Akamai3Measuring Video Quality and Performance:Best PracticesAccording to a survey conducted by PwC in October 2019 involving more than 2,000 people in th
14、e United States from ages 18 to 59 years,“streamlining the streaming experience”is increasingly important.Ease of use and reliability are linked to engagement,and are essential for driving retention and preference among video services.The February 2020 edition of the Nielsen Total Audience Report hi
15、ghlights the importance of streaming and playback quality when it comes to the importance of video streaming attributes.The chart below shows the importance of video streaming attributes,and the highlighted boxes show which attributes are affected by the content delivery network(CDN)platform being u
16、sed for video delivery.Importance of Video Streaming Attributes(Rated Extremely or Very Important)2How Video Quality Impacts EngagementThe primary business objective for most OTT service providers is to grow and retain their audience over a given period to ensure a return on investment(ROI).The qual
17、ity of experience on a platform impacts a viewers perception and engagement with the brand,which impacts the overall business model.84%79%74%58%52%40%81%77%71%56%48%38%37%CostEase of UseVariety/Availability of ContentStreaming/Playback QualitySpeed(Menu Selection/Loading Content)Accessibility/Search
18、 of Desired ContentAvailability Across DevicesResolution Available(4K UHD,HD)Skipping Ad FeaturesAd-FreeContent Available for DownloadingMenu RecommendationContent Available Live4Measuring Video Quality and Performance:Best PracticesIn order to clearly understand viewer behavior and its correlation
19、to video quality,Akamai and Sensum,3 a biometric research firm,measured the reactions of more than 1,000 people.Respondents were divided into two groups.One group was shown a video clip delivered at a high resolution(2160p UHD video,streamed at 5 Mbps)with no rebuffering.The second group was shown t
20、he same video at a lower resolution(1080p HD video,streamed at 1.6 Mbps)with a rebuffering event during a key scene.The reactions to both videos were measured using tools like galvanic skin response monitors and facial coding software.Key Findings Higher-quality streams generated 19.8%more emotional
21、 engagement according to the galvanic skin response test results Rebuffering caused a 16%increase in negative emotions,9%increase in disgust,7%increase in sadness,and 8%decrease in focus,according to facial coding technique results 76%of participants would stop using a service if a problem like rebu
22、ffering occurred several times,according to survey responsesIn the first test(graph below),respondents were asked to watch the high-and low-quality videos while their emotional engagement was measured.With no intense action,the higher-quality resolution video produced a 10.4%increase in viewer engag
23、ement compared with the lower-quality resolution according to the galvanic skin response results.5Measuring Video Quality and Performance:Best PracticesIn a second test,in addition to the high-and low-quality videos,a two-second rebuffering event was introduced during a high-intensity scene.When par
24、ticipants viewed the action scene,there was a spike in skin conductance that you can see reflected in the graph below.In this moment of high intensity,when the action scene was introduced,the difference in viewer engagement between the high-and low-quality video was 19.8%.The secondary bump at 0:46
25、in the low-quality experience was most likely due to a moment of negative response to the rebuffering period.Creating a Robust Video Quality and Performance Measurement Strategy Video delivery performance directly impacts a viewers quality of experience,which directly impacts the video service provi
26、ders desired business outcome.“Is there a link between video quality and user retention?Probably so.When we first started,the number 1 reason for churn,according to customers,was because of video quality and rebuffering.That has now improved by 90%.As we improved,it went from the number 1 problem to
27、 immaterial.”COO,SVOD ServiceMeasures Video Delivery QualityCan the video be delivered to the end users flawlessly without any issues?Can viewers enjoy a consistent and high-quality experience without interruptions?Would the viewing experience result in increased viewership,watch time,and revenues?D
28、elivery PerformanceMeasures Users Viewing ExperienceQuality of ExperienceMeasures Business ResultsBusiness OutcomesImpactsImpacts6Measuring Video Quality and Performance:Best PracticesThere is much debate about what metrics to use to define a users viewing experience.Each aspect of digital video del
29、ivery(components within the online video delivery value chain,from content production to distribution)is measured differently,adding even more complexity to defining the video delivery performance and viewing experience.For example,in a livestreaming workflow,the performance of the video preparation
30、 solutions is determined by how reliably and efficiently the video is prepared.For the distribution solutions including the CDN platform,performance is defined by the availability,scalability,and reliability of the platform.Without consistency in measurements across video streaming services,it is di
31、fficult to standardize acceptable measurements.For example,there can be multiple interpretations to the statement“Rebuffering rate of 5%”:Does this mean the average viewer spent 5%of viewing time experiencing rebuffering?Did 5%of viewers have at least one rebuffer event?Was this calculated as a mean
32、 or median value?How long does a rebuffer event need to exist for it to be counted?There have been efforts underway globally to standardize measurements to examine streaming quality.The Customer Technology Association(CTA)established the R04 WG20 working group that recently published the streaming q
33、uality of experience events,properties,and metrics standard as CTA-2066.The standard specifies a set of media player events,properties,quality of experience(QoE)metrics,and associated terminology for representing the streaming quality of experience across systems,media players,and analytics vendors.
34、Creating a Common Language for Measuring Video Quality and Performance The first step toward standardization is to determine what the metrics are,what the methodology is,and what kind of measurement tools should be used to ensure transparency and consistency in measuring video quality and performanc
35、e.010203MetricsDefine a common language of METRICS for measuring streaming quality and performance that customers can relate toMethodologyObjectively measure streaming QoS using a documented METHODOLOGY to ensure consistencyMeasurementQuantitatively measure video delivery performance with tools that
36、 provide the required insights and granularity of MEASUREMENT7Measuring Video Quality and Performance:Best PracticesLets begin by walking through what defines the viewer experience.The viewer experience journey starts with identifying two major aspects:1.A viewers perception about the video quality
37、determined by their viewing experience on their device represented by QoE metrics2.The service and performance quality delivered by the components in the video delivery value chain represented by QoS metricsQuality of Experience(QoE)Metrics QoE metrics are the parameters that determine the viewers e
38、xperience and perception of the service quality measured on the client device.Refer to the Appendix Table 1 for some of the common QoE metrics measured by most video analytics services.Although these listed metrics are the ones provided by most analytics tools,the methodology used to define them oft
39、en varies,which can create inconsistencies in measurements and reporting.For example,rebuffering is measured by different analytics tools using different formulas.They can be represented either as rebuffering count,rebuffering duration,or rebuffering frequency as mean,median,count,minutes,or percent
40、ages.The methodology varies based on how and when the rebuffering events are captured.Appendix Table 2 represents some of the common rebuffering metrics and their definitions at a broad level.“Quality of experience is the degree of delight or annoyance of the user of an application or service.It is
41、a measure of the perceived improvement or degradation of the audio or video and the viewers satisfaction with the media experience.”Customer Technology Association(CTA)QoS MetricsNetwork MetricsHardware MetricsCDN MetricsAvailabilityStartup TimesBitrateRebuffingCombination of QoS and QoE MetricsQoS
42、vs QoEQoS and QoEMethodologyMeasurementMetrics,Methodology,and Measurement to Improve Viewer ExperienceQoE MetricsPerformanceBest PracticesViewer Experience8Measuring Video Quality and Performance:Best PracticesQuality of Service(QoS)Metrics QoS metrics are parameters used to define the service qual
43、ity of each component of the video delivery value chain(i.e.,content preparation,production,processing systems,content distribution systems,and internet service providers ISPs involved in last-mile delivery to the client device).In order to overcome the QoE inconsistencies and to accurately pinpoint
44、 the source of degradation in performance,QoS metrics must be defined for each component of the video delivery value chain,including the content preparation infrastructure,the CDN,and the viewers ISP.QoS can be defined as“the description or measurement of the overall performance of a service,such as
45、 a telephony or computer network or a cloud computing service.To quantitatively measure quality of service,several related aspects of the network service are often considered,such as availability,throughput,transmission delay,jitter,etc.”4 For example,when it comes to the CDN component of the video
46、delivery value chain,its important to understand the QoS metrics for a CDN.Appendix Table 3 provides some of the common QoS metrics used to measure CDN performance.Quality of Performance(QoP)Metrics QoS metrics are often characterized as operational metrics,or transactional data points mainly used f
47、or real-time decision-making,which differentiates them from QoE metrics,which are characterized as business metrics mainly used for effective decision-making.QoP MetricsOperational DataQoS MetricsNetwork DataCDN DataTransactional Data for Real-Time ActionsCHARACTERISTICSDay-to-Day OperationsHigh Vol
48、ume and ScaleLow Latency AccessIn the MomentOperational DataQoE MetricsUser PerceptionDemography DataTransactional Data for Real-Time ActionsCHARACTERISTICSDay-to-Day OperationsHigh Volume and ScaleLow Latency AccessIn the Moment9Measuring Video Quality and Performance:Best PracticesEnsuring a unifi
49、ed view of the viewer experience across business and operations teams requires a unique set of metrics that provide the QoS for the respective component of the video delivery value chain and closely relate to the industry metrics for measuring quality of experience.QoP metrics are parameters that ca
50、n be used to define operational data to improve system efficiencies and provide visibility into the business aspects by serving as proxy QoE metrics.An example of a QoP metric:In an adaptive bitrate streaming scenario,if a customer has video segments that are six seconds long and the CDN takes more
51、than six seconds to deliver the segment,there will be rebuffering on the client player.A QoP metric can be used to measure this CDN service quality behavior and also serve as a proxy metric to client-side rebuffering.Best Practice:Define effective quality of performance(QoP)metrics working with your
52、 video technology service provider to create consistent metrics that provide visibility into the quality of service(QoS)and also relate to the viewers quality of experience(QoE).Using the Best Methodology to Measure Streaming Quality Since OTT services and other digital video applications rely on mu
53、ltiple purpose-built solutions across the video delivery value chain,it is important to understand the performance from a real end users perspective(QoE metrics measurement)and also understand the network,hardware,and CDN health(QoS metrics measurement).Methodology for Measuring End-User or QoE Metr
54、icsWhile there are multiple ways to monitor the performance of systems,network,and infrastructure,there are two popular methods of monitoring performance from the end-user perspective:Active monitoring or synthetic monitoring:Uses automated tests that run at scheduled times to collect performance da
55、ta.These tests simulate the behavior of a real user in a preconfigured environment.They run in a specified device type over a specific network type,and test probes can be deployed either as backbone nodes(i.e.,test machines sitting in data centers all around the globe)or as last-mile nodes(i.e.,test
56、 machines sitting behind a real homelike internet connection).Passive monitoring or real user monitoring(RUM):RUM reports real-time performance data as experienced by a real end user.The data is collected directly from real users using the service and captures the behavior of each one.RUM monitoring
57、 requires the service provider to implement a RUM SDK(or a client-side code)in either the player itself or the browser or app using the player.Appendix Table 4 provides use cases and benefits of synthetic and real user monitoring.10Measuring Video Quality and Performance:Best PracticesMeasuring CDN
58、Edge Performance or CDN Quality of Performance(QoP)CDNs today manage a significant portion of the worlds video streaming traffic and are ubiquitous in mitigating the toughest challenges of delivering video content over the internet.According to the IDC MarketScape:Worldwide Commercial CDN 2019 Vendo
59、r Assessment report,which positioned Akamai in the“Leaders”category,CDNs will manage 72%of internet traffic by 2022,up from 56%in 2017.5 Along with end-to-end performance visibility,it is also imperative to methodically measure the performance of the purpose-built solutions across the video delivery
60、 value chain,which includes measuring the CDNs platform performance.The diagram here illustrates a high-level workflow in a video streaming use case,identifying the key elements and associated performance visibility elements:COMPONENTSDESCRIPTIONMETHODOLOGYOrigin VisibilityVisibility into the round-
61、trip data flow between the CDN edge server and the content preparation systems origin infrastructureQoS metrics for origin performance monitoring defined in Appendix Table 3CDN VisibilityVisibility into the CDN platform between the first byte request received by the CDN edge server from the client d
62、evice and the response back by the CDN platformCombination of edge and midgress QoS metrics defined in Appendix Table 3Round-Trip VisibilityVisibility into the round trip between the client device and the CDN edge server(nearest to the end user)Multiple Factors:Last-mile ISP performance,last-mile ho
63、me router or cellular performance(in case of mobile devices),the client device itselfOrigin InfrastructureOrigin VisibilityCDN VisibilityRequest Start Response StartEnd-to-End VisibilityDNS+TCP+SSLRound TripISPCDNPlatformHome Router or Cell TowerReal User11Measuring Video Quality and Performance:Bes
64、t PracticesMany RUM and synthetic testing tools help measure end-to-end performance,as well as individual component performance such as the DNS performance;however,it is generally difficult to use these tools to get granular component-level details,such as performance of the CDN network(QoS metrics)
65、.This calls for specialized methods to be defined to measure the CDN performance using QoP metrics that provide visibility into a CDNs performance(CDN QoS)and,at the same time,also relate to the end users viewing experience(QoE metrics).Defining the CDN Edge Performance or CDN Quality of Performance
66、(QoP)Who should define the methodology?Every CDN has a different network-level architecture and implements different types of optimizations within their networks.It is your CDNs responsibility to provide clear and transparent performance measurement methods.What should the methodology achieve?Given
67、that the methodology is meant to measure the video quality and performance,the target should be to define the specific CDN QoP metrics that measure performance for video traffic(manifests and segments).Since the methodology is specific to the individual CDN,a definitive methodology should be adopted
68、 to make these metrics easily relevant and relatable to the QoE metrics commonly used by RUM and synthetic testing tools.Defining QoP metrics as discussed in the previous section would be a step toward that end.How should the performance visibility be provided?Data Feeds:Data feeds bring low-latency
69、 data feeds on video delivery performance,CDN health,latency,errors,and events through raw logs mainly targeted for operational visibility.Reports:Reports bring visibility into CDN platform performance through customizable and low-latency reports and dashboards.Insights:Insights show performance tre
70、nds and how they relate to regions,content type,consumption,network conditions,performance benchmarks,and key performance indicators(KPIs)to measure a viewers quality of experience.Best Practice:Reinforce the need for CDN accountability to define video quality and performance methodology using QoS a
71、nd QoP metrics delivered as raw feeds,reports,and insights for operational and business needs.12Measuring Video Quality and Performance:Best PracticesChoosing the Right Tools for Effective MeasurementOnce the metrics to measure video delivery performance have been established and the methodology to
72、measure these metrics has been identified,the next important task is to identify or build the tools that will be used to quantitatively measure the video delivery performance.Quality of Experience(QoE)Measurement Tools There are multiple RUM and synthetic measurement tools available for QoE measurem
73、ent and monitoring,including tools from Conviva,Nice People At Work,Mux,Bitmovin,Touchstream,and Telestream.Akamai also offers the Media Analytics solution for QoE measurement,which is made up of two key modules:Quality of Service(QoS)Monitor:Provides visibility into the quality of video playback an
74、d viewing experience to help gain audience engagement insights Audience Analytics:Provide a comprehensive overview of key trends around audience behavior as they engage with video contentNote:RUM and synthetic tools are ideal for understanding an end users QoE;however,there can be multiple scenarios
75、 in which the insights from these tools might not provide a complete picture of video quality improvement or degradation and might make it difficult to pinpoint what is responsible for the issue.It is helpful to have a checklist of questions to evaluate these metrics and insights:How have the QoE cl
76、ients been implemented in the player?Incorrect test object implementations can cause erroneous performance representation.What methodology is being used?There is a clear distinction between QoE and QoS and the associated methodologies.Why was the specific time frame selected?It is most effective to
77、evaluate multiple scenarios across a length of time.Metrics from a specific time frame with nonoptimal performance might be a misrepresentation.Are the stats for a specific region and network,or for a CDN and percentile of users?A nonrepresentative sample can show an unrealistic state of affairs.Whi
78、ch tool is used to compare the data?Different tools can result in different statistics for the same metric and can result in incorrect inferences.13Measuring Video Quality and Performance:Best PracticesQuality of Service(QoS)Measurement Tools QoS measurement tools measure the performance of specific
79、 components of the video delivery value chain,including origin infrastructure and CDN video delivery performance.In order to help organizations measure CDN performance and CDN QoS,Akamai offers Media Reports to monitor video streaming quality to ensure it reaches the end user with the highest qualit
80、y for a superior user experience.Providing media customers with visibility into platform performance helps organizations make timely decisions.This is a key area of focus for Akamai.Initiatives and enhancements are in progress to provide platform performance visibility using QoP metrics;provide low-
81、latency,raw log data for operational monitoring;provide greater visibility into platform capacity;and troubleshooting of individual end-user requests.Best Practice:Use a real user monitoring(RUM)or synthetic measurement tool to understand the viewers quality of experience.It is also helpful to use t
82、he list of questions on the previous page to evaluate metrics and insights.Use service-quality tools that incorporate quality of performance(QoP)metrics to provide visibility into CDN performance.SummaryAmid the growth of OTT consumption in recent years,viewer expectations around video quality and v
83、iewing experience has matured.What started off as a“good to have”requirement has now become a“mission critical”necessity.Understanding video quality and how to measure it is the starting point.The next step is to implement a framework to measure video quality and performance,not just from the viewer
84、s perception,but also the quality of service being delivered by the components in the video delivery value chain.These steps will better engage the viewers and help organizations monetize the engagement from advertisers and subscribers to deliver a successful business impact.14Measuring Video Qualit
85、y and Performance:Best PracticesAppendixTable 1:QoE Metrics for End-User Viewing Experience MeasurementTable 2:Rebuffering Metrics and DefinitionsQoE METRICDESCRIPTIONIMPACT IF POORAvailabilityContent availability at the time a request is sent by the end user Inability to stream or interruptions wat
86、ching streamsBitrateVideo quality indicated by the number of bits transmitted over a set length of timeLower-quality streams often translated to grainy and lower-resolution videoPlayback ErrorHow many playbacks were abandoned because of a playback error after the initial rebuffering was completedPre
87、mature end of the video or the player/browser crashingRebufferingA stall in video delivery caused by video segments downloading slower than play rate Stuttering in video/frozen video and audio,and possibly the“wheel of death”on the screenStartup TimeThe time from when playback is requested until the
88、 video starts playingLong wait time for the video to start,or long switching time between live or linear channelsVideo Start FailureHow often playback attempts are terminated during video startup before the first video frame is shownNo video shown at allMETRICREPRESENTATIONQoS DEFINITIONRebuffering
89、PercentagePercentageAmount of time video spent rebuffering as a percentage of the total time video was requestedRebuffering FrequencyNumberHow often the video rebuffered per minute of video playback that the viewer attempted to watchRebuffering DurationMedian or PercentileHow long the viewer has to
90、wait for their video to reload as a percentage of video watch time lost to rebufferingRebuffering CountMedian or PercentileThe number of rebuffering instances aggregated across the viewing duration15Measuring Video Quality and Performance:Best PracticesTable 3:QoS Metrics for CDN Performance Measure
91、mentTable 4:Synthetic and Real User MonitoringQoS METRICSDEFINITIONVARIABLESTraffic/HitsTotal volume of data getting delivered across the CDN platform,in terms of hits and bytesOrigin,midgress,and edge hits and bytesThroughputBandwidth available to deliver content across the CDN platformThroughput i
92、n MbpsOffloadTotal volume of requests that did not go back to the origin,in terms of hits and bytesOffload hits and offload volumeAvailabilityHits that resulted in successful responses,compared with the total number of hitsEdge and origin responsesACTIVE OR SYNTHETIC MONITORINGPASSIVE OR REAL USER M
93、ONITORINGUsing synthetic monitoring,one can:Simulate a controlled test environment with preconfigured variables Schedule tests to run when needed Determine what needs to be tested and build tests based on the monitoring requirementUsing passive monitoring or RUM,one can:Vary test environments with e
94、very user to aggregate real user data Collect data for every user accessing the application Configure variables to track and collect data in real timeActive or synthetic monitoring helps to:Reduce resolution time by actively tracking potential issues before they impact end users Set alerts for when
95、any metric value falls below or rises above set threshold Monitor proactively in a pre-production environmentPassive monitoring or RUM helps to:Capture performance of real users from different devices,browsers,geolocation,etc.Correlate user engagement with service performance Get historical data to
96、predict performance trends,business outcomes16Measuring Video Quality and Performance:Best PracticesAkamai secures and delivers digital experiences for the worlds largest companies.Akamais intelligent edge platform surrounds everything,from the enterprise to the cloud,so customers and their business
97、es can be fast,smart,and secure.Top brands globally rely on Akamai to help them realize competitive advantage through agile solutions that extend the power of their multi-cloud architectures.Akamai keeps decisions,apps,and experiences closer to users than anyone and attacks and threats far away.Akam
98、ais portfolio of edge security,web and mobile performance,enterprise access,and video delivery solutions is supported by unmatched customer service,analytics,and 24/7/365 monitoring.To learn why the worlds top brands trust Akamai,visit ,or Akamai on Twitter.You can find our global contact information at 05/20.SOURCES 1)eMarketer Global Digital Video Report 20192)Nielsen Total Audience Report,February 20203)https:/ MarketScape:Worldwide Commercial CDN 2019 Vendor Assessment