YouTubeGuidelines.pdf
-
Upload
marco-fernandes -
Category
Documents
-
view
214 -
download
0
Transcript of YouTubeGuidelines.pdf
-
8/15/2019 YouTubeGuidelines.pdf
1/64
YouTube Rating Guidelines
Version 1.1
January 8, 2016
1. Introduction2. Important Notes3. Types of YouTube Search Results
3.1. Video Results3.2. Channel Results3.3. Playlist Results3.4. Movie Results3.5. Pay-Restricted Results
4. Assigning Utility Rating
4.1. Utility Rating for Video Results4.1.1. Video Result Utility Rating Examples
4.2. Utility Rating for Channel Results4.2.1. Channel Result Utility Rating Examples
4.3. Utility Rating for Playlist Results4.3.1. Playlist Result Utility Rating Examples
4.4. Utility Rating for Movie Results4.4.1. Movie Result Utility Rating Examples
4.5. Utility Rating for Pay-Restricted Results4.5.1. Pay-Restricted Result Utility Rating Examples
5. Assigning Quality Rating5.1. Quality Rating for Video Results
5.1.1. Video Result Quality Rating Examples
5.2. Quality Rating for Playlist Results5.2.1. Playlist Result Quality Rating Examples
5.3. Quality Rating for Channel Results5.3.1. Channel Result Quality Rating Examples
5.4. Quality Rating for Pay-Restricted Movies and Videos5.4.1. Channel Result Quality Rating Examples
6. Assigning Flags to YouTube Result Blocks6.1. Assigning flags to video results6.2. Assigning flags to non-video results
7. Reporting Results with Duplicate Landing Pages7.1. Pre-Identified Duplicates7.2. Rater-Identified Duplicates7.3. Reporting Duplicate Results
8. Assigning an Overall Side by Side Score (if requested)
1
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
2/64
1. Introduction
The goal of YouTube search is to provide users with relevant, engaging videos. The data derived from your rating
tasks is used to inform YouTube search quality engineers about what can be done to achieve this goal. These
guidelines are designed to help make your responses on rating tasks as useful as possible.
In YouTube Block Utility tasks, you will be given a query that was issued on www.youtube.com and one or two sets of
YouTube result blocks to evaluate.
In all YouTube Block Utility tasks, you will:
1. Assign a Utility Rating to each YouTube result block using the given Utility Rating slider.
2. Assign a Quality Rating to each YouTube result block using the given Quality Rating slider.
3. Assign Flags to YouTube result blocks, when appropriate.
4. Report Duplicate Result Blocks (results that have the same landing page).
In some YouTube Block Utility tasks, you will also be asked to assign an Overall Side by Side rating. You will be
required to leave a comment if you think one side is better than the other side.
These guidelines provide information about the different types of results provided by YouTube and how to evaluate
them. Please note that YouTube continually makes changes to the appearance of pages on the site, so some of the
screenshots in these guidelines may not exactly match what you see on the site.
2. Important Notes
Result blocks are sometimes slow to load: Before assigning any ratings, please scan the task page to make sure
that all result blocks have loaded. Please note that result blocks may be slow to load and/or that the task page may
need reloading if there are empty result blocks. If the result blocks fail to load after refreshing a few times, please release the task and send a note to your employer.
Sometimes a side will not have any result blocks : If one side displays the message “This side did not generate
any results”, please do not release the task or send a note to your employer. Please assign utility ratings to each
result block using your best judgment. Please also assign an overall SxS rating, if the task asks for one.
Sometimes the task will not be divided into two sides: You will be given just one set of result blocks to evaluate.
Do not refresh the task page after assigning ratings: If you refresh the page after assigning ratings, you will lose
them and they will have to be entered again.
2
Proprietary and Confidential - 2016
u=hkpjohpj
http://www.youtube.com/
-
8/15/2019 YouTubeGuidelines.pdf
3/64
3. Types of YouTube Search Results
Most YouTube search results are videos, channels, or playlists. These guidelines provide guidance on rating these
types of results.
Result Type Examples
Videohttps://www.youtube.com/watch?v=uBWrpVrazzA
http://www.youtube.com/watch?v=W795W63n7mA
Channel
https://www.youtube.com/user/PewDiePie
http://www.youtube.com/user/whitehouse
https://www.youtube.com/channel/UCE80FOXpJydkkMo-BYoJdEg (topic channel)
Playlist
http://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQ
http://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rew
https://www.youtube.com/watch?v=EVBsypHzF3U&list=RDEVBsypHzF3U (mix)
3
Proprietary and Confidential - 2016
http://www.youtube.com/watch?v=qr5VqBP1voA&list=FL81VD6eeuLLSfyY_D-N8sVwhttps://www.youtube.com/watch?v=EVBsypHzF3U&list=RDEVBsypHzF3Uhttp://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rewhttp://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rewhttp://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcN-Y7HIuyE0Rewhttp://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQhttp://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQhttp://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQhttp://www.youtube.com/user/whitehouse?https://www.youtube.com/channel/UCE80FOXpJydkkMo-BYoJdEghttp://www.youtube.com/user/whitehousehttp://www.youtube.com/user/madonna?http://www.youtube.com/user/madonna?https://www.youtube.com/user/PewDiePiehttp://www.youtube.com/watch?v=W795W63n7mAhttps://www.youtube.com/watch?v=uBWrpVrazzA
-
8/15/2019 YouTubeGuidelines.pdf
4/64
This is how YouTube result blocks for these result types were displayed at the time these guidelines were written:
Result Type Result Block
Video Result
Channel Result
Notice the “CHANNEL”label
Playlist Result
Notice the thumbnaildifference. Usercurated playlist resultand YouTubeauto-generated mixplaylist result could lookdifferently.
YouTube provides other types of search results, such as movie results, artist watch cards, and show watch cards, and
you may occasionally be asked to rate these other result types. Please use your judgment and assign reasonable
ratings based on the utility of the landing page to the query.
This is how YouTube result blocks for these result types were displayed at the time these guidelines were written:
Result Type Result Block
Movie Result
4
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
5/64
Artist Watch Card
Show Watch Card
5
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
6/64
3.1. Video Results
Video results are the most common type of YouTube search result. Here is a link to the video that is displayed when
the user clicks on the title 10 Cutest Cat Moments of this video search result:
The video result page looks like this at the time these guidelines were written:
6
Proprietary and Confidential - 2016
http://www.youtube.com/watch?v=q1dpQKntj_w
-
8/15/2019 YouTubeGuidelines.pdf
7/64
3.2. Channel Results
A channel is a collection of YouTube webpages. Here are some signs that usually indicate that a particular YouTube
webpage is part of a channel:
● A channel specific banner at the top the page, followed by the channel title, a “Subscribe” button, and the number of subscribers.
● Navigational tabs, such as “Home”, “Videos”, “Popular”, “Channels”, “Discussions”, “About”, etc.
● Prominent display of the word “channel” on the page.
● A search box titled “Search Channel”.
All channels have at least one of these signs. With a little experience you will soon be able to recognize channels at a
glance.
Here are some examples. Please visit these channel pages by clicking the screenshots or the titles above the
screenshots.
Channel Example 1: Taylor Swift Channel Channel Example 2: Kitesurf Channel
Examples 3 and 4 below are known as Topic channels. Topic channels are created automatically by YouTube, and
the topic channel names usually start with a hashtag (#).
7
Proprietary and Confidential - 2016
http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/bartezkitetv?http://www.youtube.com/user/ladygagavevo?https://www.youtube.com/user/TaylorSwiftVEVO
-
8/15/2019 YouTubeGuidelines.pdf
8/64
Example 3: Taylor Swift Topic Channel Example 4: Minecraft Topic Channel
When you click on the about tab, you’ll see topic channels are auto-generated by YouTube.
8
Proprietary and Confidential - 2016
http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0http://www.youtube.com/channel/HCfvBjoGqrZq0https://www.youtube.com/channel/UCQvWX73GQygcwXOTSf_VDVghttp://www.youtube.com/channel/HCQrFlYXe6zIIhttps://www.youtube.com/channel/UCPC0L1d253x-KuMNwa05TpA
-
8/15/2019 YouTubeGuidelines.pdf
9/64
9
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
10/64
3.3. Playlist Results
A playlist is a list of videos arranged in a linear order. The arrangement of the videos is controlled by the registered
YouTube user who owns and creates the playlist, or by the automatic mix algorithm from YouTube. Under the default
playlist settings, videos in the playlist will play sequentially with no further action from users. Users may click videos in
the playlist if they wish to watch videos in a different order.
This is how YouTube Playlist result blocks were displayed at the time these guidelines were written:
The playlist result page looks similar to a video page, with an additional list of videos on the right side above the
related videos panel.
10
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
11/64
11
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
12/64
3.4. Movie Results
Some movies on YouTube are free for users. Others require payment and users would not be able to view them
without paying. When assigning a utility rating to such movies, please assume that the movie plays as advertised. If
the title of the movie is a good fit for the query, it should get a good utility rating. You will learn more about the YouTube video rating scale in Section 4.
Here are examples of free movies and movies that require payment.
Movie Result
TypeResult Block
Free Movie Result
Paid Movie Result
Notice the dollar signblue label and the“$4.99” fee
12
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
13/64
3.5. Pay-Restricted Results
Pay-restricted results are YouTube search results that cannot be viewed without first making a payment. These
results have a price icon somewhere in the result block which looks like this:
Some of the movie and video results are pay-restricted. Upon clicking on a pay-restricted result, the preview trailer
might start to play. Please do not confuse such preview trailers with actual movies!
Sometimes, the video result directs to a YouTube page with a video player informing users that payment is required
before the video can be viewed.
13
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
14/64
14
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
15/64
4. Assigning Utility Rating
Understanding the query and user intent
As with other types of rating, you should familiarize yourself with the possible interpretations of the query and develop an understanding of the user’s intent. If the meaning of the query is not perfectly clear, do some research: look at the
YouTube search results in the task and/or try issuing the query on google.com or wikipedia.com. You may also be
able to find definitions of unfamiliar terms at urbandictionary.com. Try to imagine the various reasons a user might
have issued the query on YouTube and the kind of videos that the user expects to see.
Evaluating YouTube Video Result Blocks
The same rating scale will be used when evaluating all types of YouTube results; however, you will be given individual
instructions for rating these four types: videos, channels, playlists, and movies.
4.1. Utility Rating for Video Results
The usefulness of a YouTube video result is determined by the video result (the main video on the landing page) and
by the user’s intent. If the user has a particular result (video) in mind, then that result is very useful. Results that have
nothing to do with the user’s intent are not relevant and therefore not useful.
You will assign a utility rating to YouTube video results according to the following scale:
Rating Description
VeryUseful
Use this rating when the video is not only clearly relevant to the query, but is a video that a user typing the querywould expect to see and be pleased to see.
UsefulUse this rating when the video has a clear connection to the query, and is a result that some users would be
pleased to see but would not appeal to as wide an audience as a Very Useful
video.
MarginallyUseful
Use this rating when the video has a connection to the query, but it would not appeal to the vast majority of userswho issue the query.
This rating is also appropriate for video results that serve a minor interpretation of the query.
Useless
Use this rating:
● When the video has no clear connection to the query and would evoke the response “I have no ideawhy they are showing me this!” in the vast majority of users.
● The video does not play. You will also check the Did Not Load flag.● The video is unratable because it is in a foreign language and you are unable to determine utility. You
will also check the Foreign Language flag.
This rating also applies when you cannot determine the utility of the video within the allotted time.
IMPORTANT: Do NOT use this rating when the video requires payment to play. Some movie results requirepayment to play. For movies that do not play without payment, please assume that the movie is what it claims tobe.
Important: You are not required to assign the Useless rating to foreign language videos. If you believe that a foreign
language video may have some utility, you may select another rating. Here are two possible scenarios:
15
Proprietary and Confidential - 2016
http://urbandictionary.com/http://wikipedia.com/http://google.com/
-
8/15/2019 YouTubeGuidelines.pdf
16/64
● The video doesn’t need language to be appreciated. For the query [skateboarding], English (US), this Chinese video may have some utility for English users.
● The user is looking for a foreign language video. For the query [Como la Flor by Selena], English (US)
this Spanish video is exactly what the user is looking for. In fact, this video is Very Useful even though it is in
a foreign language.
4.1.1. Video Result Utility Rating Examples
Query URL Rating Description
[baby pandasneeze]
http://www.youtube.com/watch?v=FzRH3iTQPrk
Very UsefulVideo page with a video of a baby pandasneezing
[zidane headbutt] http://www.youtube.com/watch?v=zAjWi663kXc Very Useful
Video page with a video of Zinedine Zidane
giving a headbutt to an Italian soccer playerduring the 2006 FIFA World Cup match.
[pug] http://www.youtube.com/watch?v=Njo28CN5XuY
UsefulVideo page with a video of a talking pug on theMontel Williams TV Show.
[elf] http://www.youtube.com/watch?v=9jyCfRHumHU
UsefulVideo page with a video clip of the movie “TheElf”.
[snakes on a plane] http://www.youtube.com/watch?v=J2IZkwgXl8M
Marginally UsefulVideo page with a video of the University ofSouthern California Marching Band playing asong from the movie “Snakes on a Plane”.
[Star Wars] http://www.youtube.com/watch?v=hVrIyEu6h_E
Marginally UsefulVideo page with a video of “Grocery StoreWars” with animated fruits and vegetablesplaying characters from the movie, “Star Wars”
[hot dog] https://www.youtube.com/watch?v=lzQ8GDBA8Is
UselessVideo page with a video with a recording ofElvis Presley singing “Hound Dog”.
[oscar ceremony] https://www.youtube.com/watch?v=Avp-KwMmZA8
UselessVideo page with a video of a Japaneseanimation with a main character named Oscar.
[modern family] http://www.youtube.com/watch?v=59twO1fJwtQ
Useless Video page with video from the TV show “TheView”, not “Modern Family”
16
Proprietary and Confidential - 2016
http://www.youtube.com/watch?v=59twO1fJwtQhttp://www.youtube.com/watch?v=59twO1fJwtQhttp://www.youtube.com/watch?v=OU6EHnDAAvQhttps://www.youtube.com/watch?v=Avp-KwMmZA8https://www.youtube.com/watch?v=Avp-KwMmZA8http://www.youtube.com/watch?v=hVrIyEu6h_Ehttp://www.youtube.com/watch?v=hVrIyEu6h_Ehttp://www.youtube.com/watch?v=hVrIyEu6h_Ehttp://www.youtube.com/watch?v=J2IZkwgXl8Mhttp://www.youtube.com/watch?v=J2IZkwgXl8Mhttp://www.youtube.com/watch?v=J2IZkwgXl8Mhttp://www.youtube.com/watch?v=9jyCfRHumHUhttp://www.youtube.com/watch?v=9jyCfRHumHUhttp://www.youtube.com/watch?v=9jyCfRHumHUhttp://www.youtube.com/watch?v=Njo28CN5XuYhttp://www.youtube.com/watch?v=Njo28CN5XuYhttp://www.youtube.com/watch?v=zAjWi663kXchttp://www.youtube.com/watch?v=zAjWi663kXchttp://www.youtube.com/watch?v=zAjWi663kXchttp://www.youtube.com/watch?v=FzRH3iTQPrkhttp://www.youtube.com/watch?v=FzRH3iTQPrkhttps://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdfhttps://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdfhttp://www.youtube.com/watch?v=gXqwO9ZsPLchttp://www.youtube.com/watch?v=N-AdUM0Y0ychttp://www.youtube.com/watch?v=N-AdUM0Y0yc
-
8/15/2019 YouTubeGuidelines.pdf
17/64
4.2. Utility Rating for Channel Results
The first step in evaluating a channel result is to determine whether the query is channel-seeking (“navigational”) or
non-channel-seeking (“non-navigational”). In other words, you should try to determine whether the query is looking for
a particular channel. Note: the term “navigational” is used here in the context of YouTube channel results, not Web
results.
● Channel-seeking queries. We assume that the user is looking for a YouTube channel when:
a. The word “channel” or “official” is contained in the query. Here are some examples: [madonna
official], [taylor swift channel], [sesame street official channel], [elton john vevo]. Note: Vevo is a
music website, and channels of Vevo-affiliated artists are displayed on YouTube.
b. The query is an unusual word or name which is only associated with a registered YouTube channel
user. Here are some examples: [CJTClol], [kjgophish], [jimmykimmellive].
● Non-channel-seeking queries. We will assume that the user is not looking for a particular YouTube
channel result for all other queries.
Users who visit channel pages are typically interested in either going to a particular channel or finding videos on aparticular topic.
● A particular channel. A user may visit Lady Gaga’s channel to see if Lady Gaga has posted any recent
videos or to see if Lady Gaga or her fans have posted new messages. Similar content may be available on
other channels, but the user is interested in this particular channel.
● Videos on a particular topic. A user may visit a particular channel to find a good collection of videos on a
particular topic. The channel owner and the specific channel may not be important to the user – it is the topic
that is of interest. For example, a user interested in snowboarding may visit a channel devoted to
snowboarding to peruse the collection of videos on that channel.
It is important to understand these two cases because the rating you assign to channel results should depend on the
user intent.
You will assign a utility rating to channel results for channel-seeking queries according to the following scale:
Rating Description
Very UsefulUse this rating when you are certain that the user is looking for the particular channel. For example:[madonna channel]
UsefulUse this rating when you believe it is
likely or possible
but not certain that the user is looking for theparticular channel. For example, the query has multiple interpretations and this interpretation is likely orpossible, but not dominant.
Marginally Useful
Please do not use this rating when the query is channel-seeking. The channel will either be the target of
the query (
Very Useful
), one of several likely targets of the query (
Useful
), or the wrong channel( Useless).
Useless Use this rating when you believe the user is looking for a different channel.
For non-channel-seeking queries, please watch parts of the three videos on the channel that seem to be most
relevant to the query. Based on your impression of these three videos, please assign a rating according to the
following scale:
17
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
18/64
Rating Description
Very Useful Please do not use this rating when the query is non-channel-seeking.
Useful
Use this rating when the following conditions are true:
● The ratio of good-to-bad videos is very high. Very Useful and Useful videos are consideredgood videos. Marginally Useful and Useless videos are considered bad videos.
● Most of the best videos on the channel are easy to find on the channel landing page; that is, thepage that loads when you click the title of the channel result.
Marginally UsefulUse this rating if one or more of the above conditions is somewhat lacking. For example, the ratio ofgood to bad videos is even lower and/or many of the best videos are hard to find.
UselessUse this rating when most of the videos are bad and/or the good videos are difficult to find, i.e., the goodvideos are not found on the landing page. For example, if you have to go to another page or if you haveto use the search box on the channel page to find good videos, the good videos are difficult to find.
4.2.1. Channel Result Utility Rating Examples
QueryChannel-Seeking?
ResultTarget ofthe Query?
Rating Explanation
[rihannachannel]
YesRihanna’sChannel
Yes Very UsefulThis channel is the target of the query, soit is Very Useful .
[rihannachannel]
YesRihanna - Stayft. Mikky Ekko
No, not achannelresult
UsefulThis is a specific video from Rihanna'schannel, but since the query is channelseeking, the best video results are Useful .
[lada gaga vevo] YesLady Gaga’sVevo Channel
Yes Very Useful The channel is the target of the query, so itis Very Useful .
[lady gaga] NoLady Gaga’sVevo Channel
Not achannel-seeking query
Useful
This channel is Useful because it isrelated to the artist, the ratio ofgood-to-bad videos is high, and the goodvideos are easy to find.
[chris brownofficial]
YesMadonna’sChannel
No UselessEven though this is a high quality channel,it is not the channel the user is seeking.
[til it happens Toyou]
NoLady Gaga’sVevo Channel
Not achannel-see
king query
Useful
The user seeks a specific video which iseasily found (at this writing) on the channellanding page. Other videos on the channelare Useful
to the query.
[bad romance] NoLady Gaga’sVevo Channel
Not achannel-seeking query
MarginallyUseful
Even though the channel has a high ratioof good-to-bad videos, “Bad Romance”(the video specified in the query) is not onthe landing page making it hard to find.This channel is therefore MarginallyUseful .
18
Proprietary and Confidential - 2016
http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/madonna?http://www.youtube.com/user/madonna?http://www.youtube.com/user/madonna?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?http://www.youtube.com/user/LadyGagaVEVO?https://www.youtube.com/watch?v=JF8BRvqGCNshttps://www.youtube.com/watch?v=JF8BRvqGCNshttp://www.youtube.com/user/rihanna?http://www.youtube.com/user/rihanna?http://www.youtube.com/user/rihanna?
-
8/15/2019 YouTubeGuidelines.pdf
19/64
QueryChannel-Seeking?
ResultTarget ofthe Query?
Rating Explanation
[skateboardingtwins official]
Yes
The PuehseTwins'SkateboardingChannel
Yes Very Useful
The Puehse twins are a pair of particularlywell-known skateboarding twins. Evenwithout their name in the query, researchshows that they are the target of the query.
This is their official skateboarding channel.
[puehse twinschannel]
Yes
The PuehseTwins'SkateboardingChannel
Yes Very Useful The Puehse twins are well known for theirskateboarding. This channel is the targetof the query.
[puehse twinschannel]
YesThe PuehseTwins’ TennisChannel
Likely orpossible,but notcertain
Useful
The Puehse twins have taken up tennis.This is their tennis channel. This is not thedominant interpretation of the query, but itis somewhat likely.
[CJTClol] Yes CJTClol’sChannel
Yes Very Useful
Even though the query doesn’t contain
“channel” or “official,” it is navigational.There are no reasonable interpretations forthe query other than this YouTubechannel.
[st mary’scollege official]
Yes
St. Mary’sCollege ofCaliforniaChannel
St Mary’sCollege ofMarylandChannel
Likely orpossible,but notcertain
Likely orpossible,but notcertain
Useful
Useful
The query is navigational, but we can’t tellwhich St. Mary’s channel the user islooking for. Channels for both colleges areUseful .
[Yosemite] NoYosemiteNational ParkTopic hannel
Not achannel-seeking query
Useful
This topic channel is Useful because it isrelated to the query, the ratio ofgood-to-bad videos is high, and the goodvideos are easy to find.
[dogs] NoCoyote Topic Channel
Not achannel-seeking query
MarginallyUseful
Even though the videos are high quality,they have just a marginal connection to thequery. The vast majority of users whoissue the query are looking for videos ofhousehold dogs, not coyotes that live inthe wild.
Please note: Even though topic channels are created differently, they are rated in the same way as other channel
results.
19
Proprietary and Confidential - 2016
http://www.youtube.com/topic/lhJQqib8bvI/gray-wolfhttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttps://www.youtube.com/channel/UCz7vWYmm1ZIz4m-WphbPR0ghttp://www.youtube.com/topic/LOWWLdWAnmY/yosemite-national-parkhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttps://www.youtube.com/channel/UCrQ8BMjljGKzRH4bY2kXKYQhttp://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/smcm?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/stmarysca?http://www.youtube.com/user/CJTClol?http://www.youtube.com/user/CJTClol?http://www.youtube.com/user/CJTClol?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/puehsetennis?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?http://www.youtube.com/user/groms?
-
8/15/2019 YouTubeGuidelines.pdf
20/64
4.3. Utility Rating for Playlist Results
When evaluating playlist results, you are not required to watch all of the videos in the playlist.
You will form an opinion of the videos in the playlist by watching parts of the first three videos
that play when you
click on the title or thumbnail in the rating task. Please base your rating on these three videos:
● The video that plays when you click the title or thumbnail in the rating task, and
● The following two videos that appear in the playlist right hand side feeder.
Please ignore the remaining videos in the playlist.
It is possible that one or more of the three videos you should watch doesn’t load or fails to play. If this happens,
please do the following:
● Flag the playlist as Did Not Load to indicate there is a problem with at least one of the videos.
● Watch the other videos and assign a rating according to the guidelines below.
Important: Please do not watch any extra videos to compensate for videos that do not load.
Evaluating Playlist Results
The first step in evaluating a playlist result is to determine whether the query is “playlist-seeking” or
“non-playlist-seeking”.
● Playlist-seeking queries. When the query includes words like “playlist”, “songs”, “top hits”, or “videos”, we
will assume the user is looking for a playlist result. Notice that “songs”, “top hits”, and “videos” are pluralized,
which indicates the user is looking for more than one song or video.
● Non-playlist-seeking queries. We will assume that the user is not looking for a playlist result when the
above words (or similar types of words) are not included in the query.
20
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
21/64
You will assign a utility rating to playlist results for playlist-seeking queries according to the following scale:
Rating Description
Very Useful Use this rating when all or most of the three videos you watch are Very Useful for the query.
Useful Use this rating when all or most of the three videos you watch are Useful for the query.
Marginally Useful Use this rating when all or most of the three videos you watch are only Marginally Useful for the query.
UselessUse this rating when all or most of the three videos you watch are Useless to the query and most userswould wonder why they are being shown the playlist result.
You will assign a utility rating to playlist results for non-playlist-seeking queries according to the following scale:
Rating Description
Very Useful Please do not use this rating when the query is non-playlist-seeking.
Useful Use this rating when all or most of the three videos you watch are Very Useful for the query.
Marginally Useful Use this rating when all or most of the three videos you watch are Useful for the query.
Useless Use this rating when all or most of the three videos are Useless for the query.
Please note that the rating standards are high when the query is non-playlist-seeking. To be rated Useful, all or most of the three individual videos you watch must be
Very Useful . To be Marginally Useful, all or most of the three videos you watch must be Useful.
21
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
22/64
4.3.1. Playlist Result Utility Rating Examples
QueryPlaylist
Seeking?
Playlist
Result
Description Rating Explanation
[Madonna
playlist]Yes
Madonna
YouTube
Playlist
Result 1
The main video is Madonna’s
“Like a Virgin” video. The
next two videos are her
“Express Yourself” and
“Erotica” videos.
Very
Useful
Since the query has “playlist” in it,
we know it is playlist-seeking. The
main video and the next two videos
are Madonna songs. Most users
would consider these videos Very
Useful or Useful.
[Madonna
songs]Yes
Madonna
YouTube
Playlist
Result 1
The main video is Madonna’s
“Like a Virgin” video. The
next two videos are her
“Express Yourself” and
“Erotica” videos.
Very
Useful
Since the query has “songs” in it, we
know it is playlist-seeking. The main
video and the next two videos are
Madonna songs. Most users would
consider these videos Very Useful
or Useful.
[Madonna] No
Madonna
YouTube
Playlist
Result 1
The main video is Madonna’s
“Like a Virgin” video. The
next two videos are her
“Express Yourself” and
“Erotica” videos.
Useful
For non-playlist-seeking queries,
Useful is the highest rating that can
be assigned.
[Madonna
songs]Yes
Madonna
YouTube
Playlist
Result 2
The main video is a video of
an interview Madonna did on
a TV show. The second
video is a promotional
documentary of “W.E.,” a
movie directed by Madonna.
The third video is a trailer for
the “W.E.” movie.
Useless
Even though the query is
playlist-seeking and the result is a
playlist, this particular playlist result
is Useless for the query. User intent
is to watch videos of Madonna’s
songs, not watch interviews,documentaries, etc.
[Madonna] No
Madonna
YouTube
Playlist
Result 2
The main video is a video of
an interview Madonna did on
a TV show. The second
video is a promotional
documentary of “W.E.,” a
movie directed by Madonna.
The third video is a trailer for
the “W.E.” movie.
Marginally
Useful
Most users would be looking for
music videos and not interviews,
documentaries, etc. All of these
videos are Marginally Useful for the
user intent, and because the query is
not playlist-seeking, the rating is
demoted one level.
[sesame
street
songs]
Yes
SesameStreet
Songs
Playlist
Result
The main video
and the nexttwo videos are music videos
from the children’s show
“Sesame Street.”
Very
Useful
Since the query has “songs” in it, we
know it is playlist-seeking. The mainvideo and the next two videos are
songs from the children’s show
“Sesame Street.” Most users would
consider these videos Very Useful
or Useful.
22
Proprietary and Confidential - 2016
http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=0_pq6Vujkzs&list=UU81VD6eeuLLSfyY_D-N8sVw&index=7&feature=plpp_videohttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906Ehttp://www.youtube.com/watch?v=s__rX_WL100&feature=results_main&playnext=1&list=PL59C8F79CD73A906E
-
8/15/2019 YouTubeGuidelines.pdf
23/64
[sesame
street ]
No
Sesame
Street
Songs
Playlist
Result
The main video
and the next
two videos are music videos
from the children’s show
“Sesame Street.”
Useful
For non-playlist-seeking queries,
Useful is the highest rating that can
be assigned.
4.4. Utility Rating for Movie Results
The first step in evaluating a movie result is to determine what the query is looking for. In some cases, the query is
looking for a particular movie. In other cases, the query is looking for a celebrity or a subject. You must try to imagine
user intent when assigning a rating to movie results.
You will assign a utility rating to movie results according to the following scale:
Rating Description
Very Useful Use this rating when you are certain that the user is looking for the movie.
UsefulUse this rating when the movie has a clear connection to the query, and is a result that some userswould be pleased to see but would not appeal to as wide an audience as a Very Useful movie.
Marginally Useful
Use this rating when the movie has a connection to the query, but it would not appeal to the vast majorityof users who issue the query.
This rating is also appropriate for movie results that serve a minor interpretation of the query.
Useless
Use this rating:
● When the movie has no clear connection to the query and would evoke the response “I have noidea why they are showing me this!” in the vast majority of users.
● The movie does not play. You will also check the Did Not Load play flag.● The video is unratable because it is in a foreign language and you are unable to determineutility. You will also check the Foreign Language flag.
This rating also applies when you cannot determine the utility of the movie within the allotted time.
IMPORTANT: Do NOT use this rating when the movie requires payment to play. For movies that do notplay without payment, please assume that the movie is what it claims to be.
Important: You are not required to assign the Useless rating to foreign language movies. If you believe that a foreign
language movie may have some utility, you may select another rating. Here are two possible scenarios:
● The user is looking for a foreign language movie. For the query [Die Legende von Paul und Paula],
English (US)
this German movie is exactly what the user is looking for. In fact, this movie is
Very Useful even though it is in a foreign language.
23
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=7bBtTgzaF4khttp://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0http://www.youtube.com/watch?v=fCZoEqJbizo&feature=results_main&playnext=1&list=PL2A77493B44DE29E0
-
8/15/2019 YouTubeGuidelines.pdf
24/64
4.4.1. Movie Result Utility Rating Examples
Query URL Rating Description
[fletch]http://www.youtube.com/movie?v=-ssId2r7F1I
Very Useful
This movie matches the query. It is what the user is
looking for.
This movie requires payment. Although we can’twatch the movie without paying, we will assume thatthe movie is what it claims to be.
[gone in 60seconds]
http://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_sr
Very Useful
This movie matches the query. It is what the user islooking for.
This movie requires payment. Although we can’twatch the movie without paying, we will assume thatthe movie is what it claims to be.
[police]
http://www.youtube.com/
movie?v=QuHMBFTBD5s
MarginallyUseful
This movie has a connection to the query, but it wouldnot appeal to the vast majority of users who issue the
query. Most users who issue this query are notlooking for a movie; they want to watch videos of theband “The Police”.
4.5. Utility Rating for Pay-Restricted Results
If the result requires payment, you are not required to pay and are not required to watch. We still ask that you rate the
block, even without the ability to view the video. Use the title, snippet, and thumbnail to give your best guess. Do not
assign the "Did Not Load" flag to pay-restricted results. (The Did Not Load flag signals to us that there may be
a problem with the result, but pay-restricted results typically function as we intend them to function.)
4.5.1. Pay-Restricted Result Utility Rating Examples
Query [adventure time]
Pay-RestrictedResult Block
Rating Useful
DescriptionThe user seeks episodes or clips from the Adventure Time TV show. The video uploader(CartoonNetworkEps) seems like the official owner of this TV show and this seems like an episode from theshow. Many users would find this result appealing if it were free, so it should be rated Useful.
Query [adventure time jiggler]
24
Proprietary and Confidential - 2016
http://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=QuHMBFTBD5shttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=vt1AFkNitcU&ob=av1n&feature=mv_srhttp://www.youtube.com/movie?v=-ssId2r7F1Ihttp://www.youtube.com/movie?v=-ssId2r7F1Ihttp://www.youtube.com/movie?v=-ssId2r7F1I
-
8/15/2019 YouTubeGuidelines.pdf
25/64
Pay-RestrictedResult Block
Rating Very Useful Description The user seeks a particular episode of this TV show. This is exactly what the user is looking for.
Query [adventure time jiggler]
Pay-RestrictedResult Block
Rating Marginally Useful
DescriptionThe user is looking for a particular episode of this TV show. This is the wrong episode, so it will have limitedappeal.
Query [adventure time jiggler free]
Pay-RestrictedMovie ResultBlock
Rating Useless to Marginally Useful
Description The user seeks a particular episode of this TV show that can be viewed without charge. Thispay-restricted result has limited appeal.
25
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
26/64
5. Assigning Quality Rating
The goal of YouTube Quality rating is to evaluate how well a YouTube page achieves its purpose. Unlike most web
pages, most YouTube pages only have one or two main purposes:
1. to provide a video that users will want to watch
2. to help users decide what video to watch next
To assign a YouTube quality rating you will need to know the purpose of the page. Here are the main purposes for
various result types.
Result
type
Purpose Example
Video 1,Section 5.1 for rating guidelines
Playlist 1 & 2,Section 5.2 for rating guidelines
Channel 2,Section 5.3 for rating guidelines
Pay-restrictedvideos andmovies
1,Section 5.4 for rating guidelines
5.1. Quality Rating for Video Results
If the video does not load or play for you, assign the Lowest possible quality rating.
Guiding principle when evaluating video quality: did content require effort and skill to produce?
You cannot be an expert judge of content quality for every niche interest on YouTube, but you can probably make a
guess about how much effort or skill was required to produce a video or collection of videos. Effort and skill is a useful
indicator of content quality so rely on this heavily when assigning Quality ratings.
26
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
27/64
Neutral or Worthless Indicators of quality (these simple counts can be aggregated by machines. If we wanted to
count these things we would ask a machine, not you!):
● View Count
● Subscriber Count
● Likes/Dislike count or ratio
Negative indicators of effort and skill include:
Copied content It is easy to upload someone else’s video (or a movie or TV show
you didn’t make). If the channel on which the video was uploaded
does not seem like an official or authoritative source for the
content, it is probably copied content. NOTE: parodies involving
subtle or clever audio/video editing DO take effort to produce and
do not deserve a harsh penalty if they seem well-done.
Low-quality, blurry, grainy or
scratchy audio or video
N/A
Distracting video annotations N/A
Useless video metadata For example, keyword stuffing or links to spammy websites.
Positive indicators of effort and skill include:
High “production value” Videos that show evidence of post-processing, video editing, or
sound mixing. For example, multiple camera shots in a single
scene or impressive special video or audio effects.
High quality, crisp and clearaudio and video
N/A
A strong narrative or plot; story
development; multiple scenes
N/A
Effort from multiple people to
produce.
For example, if a large cast of actors actively participates in the
video.
Use of stage props. Did the video uploader build, buy, or find something impressive,
expensive or rare? Compare this video of a rocket made out of
wooden matches and this video showing testing of a
machine-milled steel rocket nozzle in a rocket lab: the props in the
latter video are harder to make.
Useful video metadata. For example, lyrics to songs or links to related content. NOTE: not
all video metadata is useful. For example, keyword stuffing and
links to spammy sites are not helpful.
27
Proprietary and Confidential - 2016
https://www.google.com/evaluation/url?q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DndgvIbtZ5hUhttps://www.google.com/evaluation/url?q=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfDkiuu1pALA
-
8/15/2019 YouTubeGuidelines.pdf
28/64
Quality indicators not related to effort and skill include:
Authority, expertise, or fame Videos from an official source (like the official White House
channel) or an authority (like the Mayo Clinic) or a famous source
(like a pop star) are often perceived as high-quality.
Misleading video thumbnailimages or video titles
These are indicators of low-quality videos. Users do not want to
be tricked or misled. If the video thumbnail and title seem to
advertise something the video does not provide, users will not be
satisfied.
Strong sentiment directed
towards content in user
comments
If there are a lot of comments about content that express strong
feelings, positive or negative, this may be evidence that the
content was engaging and generated an emotional response,
which may be a sign of high quality. However, comments directed
towards other commenters or to anything other than the content,
no matter how vehement, do not provide much evidence about
content quality.
Unique access Video of an interesting recent newsworthy event, for example
video of a disaster or protest, can have high quality if it is the only
video footage of its kind. For example, this video shows a plane
crash (warning: fatal accident footage). This footage is difficult to
obtain, so low audio and video quality can be somewhat excused.
Your gut feeling You may have watched a lot internet videos, as well as TV shows
and movies. Based on your personal experience, do you think
this video will appeal to users? If you have a strong opinion you
can allow it to influence your rating, especially if the video is
related to your interests.
5.1.1. Video Result Quality Rating Examples
Example 1
result block
28
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=AtG3RUindC4
-
8/15/2019 YouTubeGuidelines.pdf
29/64
landing page
URL https://www.youtube.com/watch?v=Lm1fya3Qx4s
content No useful content: just static images with no relation to the title.
qualityfactors
● The video required little effort to produce.● Result title and thumbnail are misleading/deceptive.
29
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=Lm1fya3Qx4s
-
8/15/2019 YouTubeGuidelines.pdf
30/64
● Audio/image quality are low because there is no audio andimages are static.
● Video description is not useful. The “click here” link provided inthe description is deceptive.
● The video content is not provided by an authority or expert.
quality ratingLowest
Example 2
resultblock
30
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
31/64
landingpage
URL https://www.youtube.com/watch?v=KIdTKE5g6bc
content Man remarks that a soccer game was well-played.
qualityfactors
● The video was not made with care or effort.● Result title and thumbnail are misleading/deceptive.● Image/audio quality is very low.● Video description is distracting and unhelpful.● The video content does not come from an authority or expert.
31
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=KIdTKE5g6bc
-
8/15/2019 YouTubeGuidelines.pdf
32/64
qualityrating
Lowest
Example 3
result
block
landingpage
URL https://www.youtube.com/watch?v=3vE2WABErEM
32
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=3vE2WABErEM
-
8/15/2019 YouTubeGuidelines.pdf
33/64
content A compilation of short amusing video clips.
qualityfactors
● The video is a compilation of videos from other users. Selectingthese videos and editing them into this single video probablyrequired some time and effort.
● Result title is okay but thumbnail is misleading and deceptive. Itsuggests that the video has racy content but there is very littleracy content in the video.
● Image/audio quality is fair. The “annotation” overlays/framesseem distracting and unnecessary.
● Video description contains some links and info of moderate valueand some keyword stuffing near the bottom.
● The video content is not produced by an authority or expert.
qualityrating
Low
Example 4
resultblock
33
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
34/64
landingpage
URL https://www.youtube.com/watch?v=1-sv8JGEdYw
content Clips of women and girls failing at various tasks.
qualityfactors
● The video is a compilation of videos from other users. Selectingthese videos and editing them into this single video probably
required some time and effort.● Result title and thumbnail are deceptive: they advertise racy
content but the video is not very racy.● Image and audio quality is poor.● Description consists mainly of keyword stuffing.● The video content is not produced by an authority or expert.
qualityrating
Low - Lowest
34
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=1-sv8JGEdYw
-
8/15/2019 YouTubeGuidelines.pdf
35/64
Example 5
resultblock
landingpage
URL https://www.youtube.com/watch?v=VuAo_YpZk8A
content Clips of people failing at various tasks.
35
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=VuAo_YpZk8A
-
8/15/2019 YouTubeGuidelines.pdf
36/64
qualityfactors
● The video is a compilation of videos from other users. Selectingthese videos and editing them into this single video probablyrequired some time and effort.
● Title and thumbnail are mostly accurate. The thumbnail imagescene is not present in the video but it is suggestive of the videocontent.
● Image and audio quality is mostly good.● Description consists mainly of keyword stuffing but the three links
it provides may be of some use.● The author has a website devoted to funny “fail” videos, with
content stretching back to 2009. We may regard him or her as aminor authority on fail videos.
qualityrating
Medium or Low+
Example 6
result
block
36
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
37/64
landingpage
URL https://www.youtube.com/watch?v=5Cod_wgM7n0
content A fan talks about an episode of a cartoon.
qualityfactors
● Audio is corrupted by white noise in several places in the video.The image quality is not very good.
● The author did not make this video with effort and care, otherwisehe would have fixed the audio issues.
● Result title suggests that the video is an actual episode of a cartoonrather than a review of the cartoon. The thumbnail is accurate.
● Video description is essentially empty.● The video content is not from an expert or authority.
quality
ratingLowest
Example 7
37
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=5Cod_wgM7n0
-
8/15/2019 YouTubeGuidelines.pdf
38/64
resultblock
landingpage
URL The video was removed from YouTube due to copyright violations.
content A complete episode of a cartoon shrunk to fit inside a frame or border.
qualityfactors
● This is directly copied content - it required little effort to produce.● Result title and thumbnail are accurate.● Image quality is marred by the distracting frame or border.
● The description has keyword stuffing.● The user who uploaded the video is not an authority or expert.
qualityrating
Lowest
Example 8
38
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
39/64
resultblock
landingpage
URL https://www.youtube.com/watch?v=i6anvsoeOGw
content A short clip from a TV show.
qualityfactors
● This is copied content. However, the user who uploaded thevideo selected this particular scene for a particular purpose (i.e.to provide a preview) rather than uploading the entire TV show
episode, so it does not deserve the “lowest” quality rating justbecause it is unoriginal.● Result title and thumbnail are accurate.● Image and audio quality are poor.● The description is brief but accurate. It does not improve or
harm quality.● The video is provided by a devoted fan on a channel containing
content related only to this TV show. The user who uploaded the
39
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=i6anvsoeOGw
-
8/15/2019 YouTubeGuidelines.pdf
40/64
video may be generously regarded as a minor authority orexpert.
qualityrating
Low - Medium
Example 9
resultblock
landingpage
URL https://www.youtube.com/watch?v=FRTf3UXCpiE
content The corporate anthem of Fujitsu Ltd. with lyrics slideshow.
qualityfactors
● This seems like copied content. However, finding this obscure oldsong and adding a lyrics slideshow requires some effort and care.
40
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=FRTf3UXCpiEhttps://www.youtube.com/watch?v=FRTf3UXCpiE
-
8/15/2019 YouTubeGuidelines.pdf
41/64
● Result title and thumbnail are accurate.● Image quality is fine, but the audio is a little scratchy.● The description includes helpful information about the singer.● Little information about the user who uploaded the video is
available. There’s no reason to think that the user who uploadedthe video is an authority or expert.
qualityrating
Medium - High
Example 10
resultblock
41
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
42/64
landingpage
URL https://www.youtube.com/watch?v=dys8KUnwGGg
content A moaning “Ahhh” noise set to jarring music and disturbing images.
qualityfactors
● The author seems to have animated this video and made thesoundtrack on his own. This required effort and care.
● Result title and thumbnail are accurate.● Image and audio quality are good.
42
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=dys8KUnwGGg
-
8/15/2019 YouTubeGuidelines.pdf
43/64
● The video description contains links and facts that may behelpful.
● The author is an authority or expert for this video because he hasbuilt a brand around this type of content, including a channel ofsimilar videos and an online store. His Twitter feed says “I writesongs that drive you insane.”
qualityrating
High - Highest
Example 11
resultblock
landingpage
URL https://www.youtube.com/watch?v=61G8iXGuSUg
content Software tutorial
43
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=61G8iXGuSUg
-
8/15/2019 YouTubeGuidelines.pdf
44/64
Qualityfactors
● The video required care and effort to produce. The authordemonstrates his expertise about the software and usedartistic/design skills in the video as well.
● Video title and thumbnail image are accurate.● Image quality is fine. Audio could be a bit clearer or louder.● Description contains links that may be helpful.● Because the author has a YouTube channel and a separate
website devoted to this type of software tutorial he can beregarded as a minor authority or expert on this type of content.
qualityrating
High
Example 12
resultblock
44
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
45/64
landingpage
URL https://www.youtube.com/watch?v=LMs-VIbc_48
content “Trip to the esthetician” role playing video in ASMR style
Quality
factors
● The video seems to have required effort and care to produce. It was
done in a single 40 minute shot. The author paid careful attention tolighting and audio -- note especially the separate left and right audiotracks.
● Result title and thumbnail are accurate.● Image and audio quality are excellent.● Video description is extensive.
45
Proprietary and Confidential - 2016
http://en.wikipedia.org/wiki/Autonomous_sensory_meridian_responsehttps://www.youtube.com/watch?v=LMs-VIbc_48
-
8/15/2019 YouTubeGuidelines.pdf
46/64
● The author is an authority or expert in this area because she makesmany of these videos, and according to her webpage, people donatemoney to support her to do this.
qualityrating
High - Highest
Example 13
resultblock
46
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
47/64
landingpage
URL https://www.youtube.com/watch?v=T_vGAPvjOc8
content Man narrates a video game adventure
quality
factors
● The video required some planning, effort and editing to create
and narrate.● Result title and thumbnail are accurate.● Image and video quality are excellent.● Description contains links that may be helpful.● The author is an authority or expert in this area because he has
built a brand (including an online store) and a following aroundthis type of video game adventure video.
47
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=T_vGAPvjOc8
-
8/15/2019 YouTubeGuidelines.pdf
48/64
qualityrating
High - Highest
Example 14
result
block
landingpage
URL https://www.youtube.com/watch?v=-3a2qoyONVA
content Music video
qualityfactors
● The video required care and effort to create. It seems to haverequired filming at multiple locations and lots of editing and audioprocessing.
● Result title and thumbnail describe the video well.● Image and audio quality are good.● The description is sparse but accurate.
48
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=-3a2qoyONVA
-
8/15/2019 YouTubeGuidelines.pdf
49/64
● RobynVEVO, the channel that hosts this video, is the primaryauthority on YouTube for music videos from the artist Robyn.
qualityrating
Highest
Example 15
resultblock
landingpage
URL https://www.youtube.com/watch?v=V2MBiS1kc_0
content Instructions on how to perform a neurological examination.
qualityfactors
● This seems to be a reliable instructional video produced by amedical professional. It requires a unique skill -- knowledge ofneurological examination techniques -- to produce.
49
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=V2MBiS1kc_0
-
8/15/2019 YouTubeGuidelines.pdf
50/64
● The instructor in the video is a board-certified nurse practitionerwith a doctorate in nursing practice. She is an expert in thisarea.
● Result title and thumbnail are accurate.● Video audio and image quality are good.● Video description is sparse but accurate.
qualityrating
Highest
5.2. Quality Rating for Playlist Results
The purpose of a playlist is to provide the user with something to watch right away and a good collectionof videos to watch next. Ideally, the quality rating for a playlist would be the average of the quality ratingsfor the videos it contains plus or minus an adjustment for the cohesiveness, variety, order, size, andplayability of the collection considered as a whole. This ideal is impossible to achieve because you will
not have time to evaluate the quality of all the videos in a playlist. When evaluating a playlist, please dothe following two things, then make your best guess about what the ideal quality rating would be.
1. Skim the first three playlist videos to get an idea about Video Quality. An image below showswhere to find the first three videos of the playlist.
2. Visit the playlist list view and skim titles and thumbnails. The first screenshot below shows wherethe link to list view can be found, and the second image shows what list view looks like. Look atthe list view and ask yourself if the playlist seems to contain a good collection of videos, keepingin mind the following points:
a. cohesiveness is necessary: videos in a good playlist must have an obvious commontheme or topic -- like “amazing soccer goals” or “15th century lute music” or “birds ofprey”, and the title of the playlist should describe the topic.
b. variety helps the quality of a playlist, so long as the added variety does not causedepartures from the theme or topic.
c. order is important: if there is a “right” order for videos (like Episode 1, Episode 2, Episode3 of a TV show, or Acts 1, 2, 3 of a play) then playlists should put videos in the right order.
d. size helps quality: longer playlists are generally better than shorter playlists. However,length is less important than cohesiveness, variety, and order.
e. videos should be playable: sometimes playlists contain videos that no longer play, orare personalized and play only for a select group of people designated by the person whouploaded the video. These are obvious in playlist list view; see the second image belowfor example.
50
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
51/64
Playlist List view: displays all the videos in the playlist:
51
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
52/64
5.2.1. Playlist Result Quality Rating Examples
Example 1
resultblock
52
Proprietary and Confidential - 2016
https://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdf
-
8/15/2019 YouTubeGuidelines.pdf
53/64
playlist listpage
URL https://www.youtube.com/playlist?list=PLJvQXRgtxlulYgB1tznRR0h8biN2QCiM7
content Footage from Metallica concerts from 2014.
qualityfactors
● Audio quality is generally good, which is important formusic-related videos.
● There is high production value: multiple shots and evidence ofvideo post-processing.
● The playlist and videos come from the official Metallica YouTube
channel.● Unique access: videos show backstage footage of the band, which
would would be difficult for most people to obtain.● The playlist is organized around an obvious theme: Metallica
concerts in 2014.● Videos are arranged in an obvious order: date of concert.
53
Proprietary and Confidential - 2016
https://www.youtube.com/playlist?list=PLJvQXRgtxlulYgB1tznRR0h8biN2QCiM7https://www.youtube.com/playlist?list=PLJvQXRgtxlulYgB1tznRR0h8biN2QCiM7
-
8/15/2019 YouTubeGuidelines.pdf
54/64
qualityrating
Highest
Example 2
result
block
landingpage
URL https://www.youtube.com/playlist?list=PLoip94oFJ4XuhyreC_ckBvihCya2CmLaB
content List of bad or funny auditions, typically a few minutes long, from variousTV-audition shows.
qualityfactors
● Many of the videos lack audio-visual crispness.● Many of these are uploaded by non-official uploaders (like
“shaunuk21”) and are thus copied content, but some areapparently from authoritative sources (“Britain’s Got Talent”, “The X Factor UK”).
● The collection as a whole is not organized around a tight theme:there are singing audition clips and music audition clips fromseveral years.
● The collection is not particularly comprehensive.
54
Proprietary and Confidential - 2016
https://www.youtube.com/playlist?list=PLoip94oFJ4XuhyreC_ckBvihCya2CmLaBhttps://www.youtube.com/playlist?list=PLoip94oFJ4XuhyreC_ckBvihCya2CmLaB
-
8/15/2019 YouTubeGuidelines.pdf
55/64
qualityrating
Medium
Example 3
result
block
landingpage
URL https://www.youtube.com/playlist?list=PLC944E2DDE82E9904
content Eight music videos from various country music artists.
qualityfactors
● Some of the videos have low quality due to issues aroundaudio-visual crispness (like the first video, a cell-phone video of alive concert), or copied content.
● The bigger issue is that the playlist content has little to do with
the playlist title: the title indicates that the list has to do with aparticular song by a particular artist, but most of the videos inthe list have little or nothing to do with that song.
qualityrating
Low
Example 4
55
Proprietary and Confidential - 2016
https://www.youtube.com/playlist?list=PLC944E2DDE82E9904
-
8/15/2019 YouTubeGuidelines.pdf
56/64
resultblock
landingpage
URL https://www.youtube.com/playlist?list=PLF9D86C3DCFEDF1EE
content Official Lady Gaga music videos alternating with “Loading Video…” spamvideos (see screenshot).
qualityfactors
There are several copies of a video titled “Loading Video …”. This shortvideo has no content and seems intended to raise video view countstatistics for the uploader who curated this playlist: from the typicaluser’s perspective it is spam, with no value.
qualityrating
Lowest
5.3. Quality Rating for Channel Results
Channels show content uploaded by one particular video uploader, which may be an individual likeJenna Marbles or an entity like Sony Pictures Entertainment Group. To assign a quality rating to thechannel, click on the uploaded videos link (illustrated below), pick any three uploaded videos on the
56
Proprietary and Confidential - 2016
https://www.youtube.com/user/SonyPictureshttps://www.youtube.com/user/JennaMarbleshttps://www.youtube.com/playlist?list=PLF9D86C3DCFEDF1EE
-
8/15/2019 YouTubeGuidelines.pdf
57/64
channel page and skim them to get an idea about the quality of the videos uploaded to the channel.Then make a guess about the average quality of the ratings on the channel, and assign that for a qualityrating.
5.3.1. Channel Result Quality Rating Examples
Example 1
resultblock
57
Proprietary and Confidential - 2016
https://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdf
-
8/15/2019 YouTubeGuidelines.pdf
58/64
landingpage
URL https://www.youtube.com/user/NationalGeographic/videos
content Videos from National Geographic
quality
factors
This is the official channel for this well-known source of educational,
scientific and documentary content. Videos have high production quality,involve many people and props, and clearly took a great deal of effort toproduce. Audio visual quality is quite good.
qualityrating
Highest
Example 2
resultblock
58
Proprietary and Confidential - 2016
https://www.youtube.com/user/NationalGeographic/videos
-
8/15/2019 YouTubeGuidelines.pdf
59/64
landingpage
URL https://www.youtube.com/user/FranzLisztFerentz/videos
content Classical music recordings
quality
factors
● The recordings have good audio quality.
● The videos seem to be copied content.
qualityrating
Medium
Example 3
resultblock
59
Proprietary and Confidential - 2016
https://www.youtube.com/user/FranzLisztFerentz/videos
-
8/15/2019 YouTubeGuidelines.pdf
60/64
landingpage
URL URL not given: this content is in violation of YouTube terms of service and
will be removed.
content Full-length movies
qualityfactors
● These full-length movies have low audio-visual quality, manydistracting video annotations, keyword stuffing in video metadata,and they are copied content.
● The channel has uploaded only these three videos.● The channel name suggests that all videos will be horror movies, but
this is not the case.
quality
rating
Lowest
5.4. Quality Rating for Pay-Restricted Movies and Videos
Always assign the highest quality rating to pay-restricted movies and videos.
5.4.1. Channel Result Quality Rating Examples
Example
resultblock
60
Proprietary and Confidential - 2016
https://www.google.com/evaluation/portal/portal_files/VideoResultRating.pdf
-
8/15/2019 YouTubeGuidelines.pdf
61/64
landingpage
URL https://www.youtube.com/watch?v=sIbhaVkWDes
content Pay-restricted cartoon episode.
qualityfactors
Please assume that pay-restricted content is highest-quality.
qualityrating
Highest
61
Proprietary and Confidential - 2016
https://www.youtube.com/watch?v=sIbhaVkWDes
-
8/15/2019 YouTubeGuidelines.pdf
62/64
6. Assigning Flags to YouTube Result Blocks
You will assign the following flags as needed: Porn / Offensive, Foreign Language, Did Not Load.
6.1. Assigning flags to video results
Porn / Offensive: Use this flag when the content of the video result is offensive or shocking because of sex-related
content, violence, bad language, etc.
Foreign Language: Use this flag for videos that would be regarded as foreign by most users in your locale. In
general, the Useless rating is appropriate for foreign videos unless you think the video has some utility despite being
foreign. In that case, please assign a rating from the rating scale.
Did Not Load: Use this flag when the video does not play for any reason, e.g. taken down, marked as private,
prohibited for the locale, etc. Do NOT use this flag for paid results, which typically function as intended after payment.
6.2. Assigning flags to non-video results
If a result page contains multiple videos (such as playlists, channels), assign the appropriate flag when some of the
first few videos on the page are porn / offensive, foreign language or did not load.
7. Reporting Results with Duplicate Landing Pages
In addition to assigning utility ratings to each display block result, we would like you to tell us whether two results are
duplicates (dupes).
7.1. Pre-Identified Duplicates
Some duplicate results are automatically detected and will be pre-identified for you. They will be highlighted by a grey
or green background, and annotated by either "Same as..." text right below the flags and above the result block.
This is what a pre-identified “Same as…” result block look like:
Same as… pre-identified duplicate result
62
Proprietary and Confidential - 2016
-
8/15/2019 YouTubeGuidelines.pdf
63/64
Please note:
● Results annotated by “Same as…” may
not be given different ratings/flags/comments. Their utility ratings, page quality ratings, flags, and comments will be automatically transferred to each other.
● You cannot un-check dupes that have been automatically detected and pre-identified.
7.2. Rater-Identified Duplicates
We would like your help identifying duplicate results that have not been automatically detected.
Encountering a list of identical or very similar results on the YouTube search results page is generally a bad
experience for users. Use your judgment when applying this definition: two video results are duplicates if they have
similar or identical content and the user who issued the query would not find the second video at all engaging after
having enjoyed the first to its fullest extent. A video result should NOT be marked as a duplicate of a playlist or
channel result.
7.3. Reporting Duplicate Results
When you notice that the results in two or more display blocks are duplicates, please click on the Report Dupe button
of one of the results. The current result (the result you clicked on) will be highlighted by a thick solid red border and the
button's name will change to Select Dupes. You can then check results that are duplicates of the current result, and
the checked results will be highlighted by a red dotted border.
The duplicate results that you checked will be annotated by "Dupe of..." text right below the flags and above the
display block. After selecting all dupes, please click the red Select Dupes button to return to the normal rating mode.
The button's name will change back to Report Dupe, and you will be able to report other sets of dupes (if there are
any). If you change your mind, you can always un-check a result. Please note that rater-identified duplicate results
annotated by "
Dupe of..." may be given different utility ratings/flags/comments.
Please identify dupes both within the same side and across sides. Even for cross-side results, you should still
ask yourself the question "Would you want to see both results if they were returned by the same search engine?"
Here are some examples:
Dupes
QUERY: [Princess Mononoke Trailer]
URL 1: http://www.youtube.com/watch?v=oZ4pxURy5-I
63
Proprietary and Confidential - 2016
http://www.youtube.com/watch?v=oZ4pxURy5-I
-
8/15/2019 YouTubeGuidelines.pdf
64/64
URL 2: http://www.youtube.com/watch?v=pkWWWKKA8jY
Reason: These two video results are duplicates because they are movie trailers that differ only in aspect ratio. Even
though they were uploaded by different YouTube users and their run times are slightly different, they are duplicates.
Users would not want to see both results returned by YouTube.
Not Dupes
QUERY: [Por una cabeza]
URL 1: http://www.youtube.com/watch?v=ZbyIUd6tfp0URL 2: h ttp://www.youtube.com/watch?v=F2zTd_YwTvo
Reason: These videos show actors dancing the tango to “Por una Cabeza” from two different movies (“True Lies” and
“Scent of a Woman”). Even though the song is the same, the videos are very different. Users could benefit by
YouTube returning both videos for the query.
8. Assigning an Overall Side by Side Score (if requested)
When you are done assigning ratings and flags and flagging pages with duplicate landing pages, please decide which
side is better according to the SxS Guidelines .
Reminder : You will only be asked to assign an Overall Side by Side score in some Block Utility tasks.
https://www.google.com/evaluation/portal/portal_files/side-by-side.pdfhttp://www.youtube.com/watch?v=F2zTd_YwTvohttp://www.youtube.com/watch?v=F2zTd_YwTvohttp://www.youtube.com/watch?v=ZbyIUd6tfp0http://www.youtube.com/watch?v=pkWWWKKA8jY