from keywords to better user experience

from penalties to better understanding of user behavior

from static to dynamic ranking factors

from desktop to mobile search

Karina Kumykova

Evolution of Yandex’s Algorithms

Introduction• Before 2009 Yandex SERP was quite easy to manipulate. All you needed was links, and lots of them! (Easy to get at brokers like Sape.ru )

• Only a few reasons for getting the penalty: non visible text on the website, doorway sites redirecting users to other domains, link farming.

• Yandex had one single kind of penalty for webmasters conducting black hat SEO – remove the website from their index.

• It was not difficult to be in the index again. All you needed is to get rid of all those black hat SEO practices on your website, write to Yandex, wait for a month, and the site’s positions were restored in Yandex search results.

• Today it is much more complicated. Yandex evolved its algorithms, and penalties can be applied not only to a website, but also to its certain pages OR to the pages in respond to the certain keywords.

• Today not always clear what algorithm exactly behind a certain penalty AND for what SEO activities. What yesterday was count as a normal SEO practice, today can be seen as a black hat SEO.

• The algorithm has become very complex and, as Yandex claims, intelligent. Each version on the algorithm made SEO more and more complex, while every new one was aiming to improve a certain area of search.

• Just like with Google, nobody knows all the factors taken into consideration by Yandex.

• Just like with Google, there are best practices that work.

• Just like with Google, there are the are certain technologies behind the algorithms.

MatrixNet (2009)

MatrixNet is Russia’s page priority algorithm and is in essence a technology of machine learning. MatrixNet measures site popularity and the sum and relevancy of backlinks to determine a thematic index citation (TIC) score, which plays into MatrixNet. Rather than scoring on a scale of 0-10 like PageRank, TIC is scored on a scale of 0-10,000. The TIC score is also the same for every page on the website rather than on a page-by-page basis. A TIC score quantifies the importance of a piece of content, which is included in SERPs to help users determine the quality and credibility of the content.

With implementation of Matrixnet, behavioural factors has become an important part of optimizing websites for Yandex. Yandex learns what people wants and refines the SERPs based on the learnings. MatrixNet builds a formula based on tens of thousands of factors.

Spectrum & MatrixNet

Spectrum (2010)• returns a whole spectrum of search results matching different user intents based

on the frequency of user searches;• based on query statistics;• able to make better sense of the meaning of searches based on its own

classification system;• analyses search queries and divides them into categories, then selecting the most

popular categories and giving them more exposure in the SERPs;• has around 60 categories and is expanding itself;• system analyses users’ searches and identifies objects like personal names, films,

books or cars. Each object is then classified into one or more categories. Categorization helps the system discriminate between meanings when processing search queries and return results based on user intent.

Query: Napoleon

Category 2: FoodCategory 1: History

2016

Facts:

• 2005 - 2016: 52 algorithms• Moscow first, then the rest of the country, CIS and other Russian speaking countries• Game "Cities": Magadan — Nakhodka — Arzamas — Snezhinsk

Keyword Era

Backlinks Era

Era of User Behaviour & UX

Keyword Era(2007 – 2010)

Search sideYandex was focusing on better understanding of search queries and languages:

June 2007 – different ranking formula applied to one word and long tail queries.

Aug 2007 – Rodeo: new ranking formula for one word queries. Priorities in search results for these queries will have homepages.

Sep 2008 – Nakhodka:• internal pages are now in search results together with the homepages

for high volume search terms.• Yandex works on better understanding of morphology of the Russian

language: different search results for keywords in single and plural forms.

• News/information sites are now in SERP for commercial queries. Wikipedia in SERP.

Apr 2008 – documents in foreign-language are in SERP now: Yandex scrolls search engines in other languages.

May 2008 – Magadan – Yandex can read abbreviations and transliteration.

Dec 2010 – Krasnodar based on Spectrum• All websites are divided into 60 categories. Search results for queries

with high search volumes became more diverse. You could see an online store, encyclopaedia and a forum for such queries.

On Site

• Meta Tags: Title, Description, Keyword

• Keyword tag is a lower ranking factor in Yandex, but still is recommended to use

• Easy to manipulate – better ranking factors are needed!

Let’s stuff meta tags with all 10,000 keywords we want to rank for, and hope to rank for all of them – at once!

Backlinks Era(2005 - 2015)

• 2005 – Nepot (from eng. Nepotism: the practice among those with power or influence of favouring relatives or friends)• First step to battle websites that actively buy and sell links.• The sites that fall under this algorithm cannot transfer weight

to the acceptor.• Jan 2008 – Links from home pages are lowered in their value. Sites

owners starting to buy links from internal links.• Mar 2008 – Links from the homepages pointing to your site have

zero value now. Sites that were in the top because of these links are lowered in search results. Links from the internal pages are still working.

• Dec 2013 – Alexander Sadovsky, Head of Web Search at Yandex announced that they are going to stop using backlinks to rank websites for commercial search terms in the Moscow region.

• On 12th March was that Yandex silenced 50 out of 150 link-related signals, a lot of them are related to anchor text.

RuNet’s reaction on Yandex’s decision to stop using backlinks as a ranking factor

• Mar 2014 – links are not included in ranking algorithm for specific niches: Real estate, Tourism, Electronics (Moscow region only!)

• Apr 2015 – Alexander Sadovsky have decided to make a change on their previous non-link-value decision. Links will again be part of the ranking factor but not only positively! They will also give a NEGATIVE effect to the ones working with bought links.

• Sadovsky: “SEO “specialists” continued to buy links through sheer habit! That was the reason we decided to make links not only neutral or positive, but also negative – and to penalise websites for paid links.”

The owner of a medical portal that removed all paid backlinks and learned that in medical

vertical links still count.

Don’t get Fooled – Links Still Count!

Minusinsk and the end of the Backlinks EraMay 2015 – Minusinsk algorithm was rolled out. Minusinsk was introduced to eradicate black hat SEO, creating negative equity and returning the penalization of paid links for commercial queries.

This meant that all websites, all regions, and all queries became susceptible to dropping in the Yandex SERP. The top two factors for penalization include a certain percent of paid links going to a site and the absolute number of paid inbound links to the site.

• Initially, 37% of the websites notified by Yandex cleaned their paid links, while the remaining 63% were susceptible to dropping 20 positions in the SERP. In the first round of penalties, 488 websites were penalized. Between May 2015 and early February 2016, 7253 websites on the RuNet have been penalized by Yandex. 5242 of these sites (72%) removed SEO-links and avoided penalties during the same period. Currently, just over 2000 websites are still being penalized as a result of Minusinsk. (Feb 2016)

• Yandex.Webmaster blog provided screenshots from penalized and non-penalized sites’ analytics on Yandex.Metric showing the impact on websites that cleaned their paid links.

Minusinsk and the end of the Backlinks EraFunny enough, among the losers are some big Russian SEO companies:

• promo.ingate.ru

• kokoc.com

• bdbd.ru

• fabricasaitov.ru

• seo-dream.ru

The penalty affected the main domains as well as subdomains of these companies. The sites lost about 20 positions for competitive search terms.

• Other verticals that were affected are the most popular e-commerce category in Russia consumer electronics, travel, real estate and some others. Some examples of the sites that got hit by Minusinsk are:

• 1-tur.ru (travel)

• moskva.etagi.com (real estate)

• msk-stroi.com (repair and renovation services)

• nastenka.su (cleaning services)

• skif-cargo.ru (logistica services)

Minusinsk and the end of the Backlinks Era• The scheme is following: first the classifier detects all the paid links and excludes them from any further calculation in Yandex search algorithm, then all the natural links are processed.

• Penalties can be lifted from two to six months until all paid inbound links are removed and crawled again.

• The share of paid links in the Yandex database decreased by 64% in the TOP 10 results of commercial queries.

• Kate Gladkikh, the Yandex safe search analyst: “We hope that RU-net is witnessing the last days of the paid-links era and it will incentivize webmasters to spread optimization methods that will truly make the Internet better.”

• This update is meant to improve the quality of websites, encouraging them to focus more on content, design, functionality and the user interface. So, in general, websites are pushed towards the creation of a better user experience, rather than relying on black-hat tricks to increase their ranking.

Yandex: “I will find you!”

Era of User Behaviour & UX

On Site: Ads• Apr 2009 – Yandex lowers SERP positions of websites using pop-under ads.• Aug 2009 – sites using click-under ads lowered in their positions.• May 2012 – sites using pop-up ads lowered in their positions.• Mar 2014 – sites having shocking / unpleasant ads on them are lowered in search results,

including adult banners, banners featuring scenes from horror films and banners showing overweight body parts.

• Dec 2015 – the Yandex.Webmaster blog warned webmasters in Russia against using clickjack, or UI redress attack, to make users unintentionally click on buttons or links due to opaque layering. The increasingly popular practice has drawn the attention of Yandex SEO specialists who noted that it would negatively impact a site’s SERP rankings. Shortly after this announcement, webmasters’ use of clickjack dropped but soon rose again. The algorithm has discovered more than 15 thousand sites using clickjack, and about 50 services that provide these capabilities to clients ‘ sites.

On Site: Behavioral factors• Nov 2009 – Snezhinsk based on MatrixNET: thousands parameters

used for rankings – behavioural factors – one of them for the first time!

With implementation of Matrixnet, behavioural factors has become an important part of optimizing websites for Yandex. The SEO community is uncertain, which metrics Yandex looks at and where they collect the data: Yandex.Metrika and/or Yandex toolbar?Many SEOs speculate about behavioural factors, but the most common opinion is that the most important ones are:– CTR from the SERP– Average time spent on the page after the click from the SERP– Bounce rate from the page back to the SERP

• Dec 2010 – Krasnodar based on Spectrum: e-commerce site, encyclopedia and forum in search results for the same query. Priority behavioural factors (actions of the user on the page).

On Site: Behavioral factors & Site’s Usability • May 2011 – first manual penalisation of sites

“improving” their behavioural factors. Sites of famous SEO companies and their clients are lowered in search results.

• Sept 2011 – algorithm «You are a spam»: penalty for pages with low quality content (e.g.: stuffed with keywords) followed by low behavioural factors.

• Dec 2014 – thousands of sites are penalised for manipulation of their behavioural factors.

• Penalty: 6 months behind position 100 + 2-3 months in the automated queue;

• “Improvement” of behavioural factors of competitors is a frequent strategy to get rid of them.

• Services on the market: Movebo, Userator• Watch closely your IP data collected in

Yandex.Metrica to avoid it!

• Oct 2011 – Usability of a website is among Yandex’s ranking factors: sites with ads overuse lowered in search results; sites with user unfriendly navigations having difficulties to get to the top.

• Nov 2011 – new algorithm for e-commerce sites: site usability, products range, prices, service, etc. Moscow first, then the rest of the country.

On Site: Quality of the sites and AGS• July 2008 – Magadan 2.0 and content

uniqueness• Sep 2009 – AGS 17 : 1 to 10 pages are left in

search results. Algorithm is against sites with automatically generated content.

• Dec 2009 – AGS 30 : manual and automated penalties for sites which despite of having unique content are not interesting for the users.

• Nov 2013 – AGS 40: increased number of factors defining low quality of the content. MFA sites are specially affected. The main reason for the penalty is lack “additional value of the website”.

• Apr 2014 – update of AGS 40: TIC=0

• The AGS is a heavy infantry support weapon designed to operate from a tripod or mounted on an installation or vehicle.

• «АнтиГовноСайт»: Anti Sh*t Site

What is AGS?

On Site: Quality of the sites and AGSSites lacking AVW do:

• Not have a defined topic of the site

• Have the same content structure (e.g. same character number)

• Not have information about the author

• Not have contacts on the site

• Not have comments

• Not have shared content on social channels

• Have a bad navigation structure

• Overuse of ads

• Have low behavioural factors (high bounce rate etc)

Just to be good is not enough.

Your site should be unique and useful to the users.

Change the concept of your site!

Kamskaja: This is how it looks like when you are trying to optimise a website that had been

penalised for lack of additional value.

Search Side: Geotargeting• Apr 2009 Arzamas – implemented a new geotargeting feature

for Moscow, Sankt-Petersburg, Ukraine, Belarus and Kazakhstan.

• Arzamas assigns websites to a specific region automatically, although it gets it all wrong sometimes…

The known factors defining a region your site is attached to are:location-by-IP data, domain names, region you assign to the site yourself when registering it in Yandex Directory, origin of your backlinks (?)• Nov 2009 – Snezhinsk: region specific search results for 19

Russian cities.• Dec 2009 – Konakovo: region specific search results for 1250

Russian cities.• Feb 2012 – region specific search results for Yandex Images

(Russia, Ukraine, Belarus and Kazakhstan).

Search Side: Personalisation

• Aug 2011 – Reykjavik (Capital of Iceland): Search results based on user’s language settings.

• Dec 2012 – Kaliningrad: Personalisation of search results for users based on their query history.

Search Side: Mobile Search • Feb 2016 – Vladivostok: the latest search algorithm that takes into

account mobile-friendliness of a page in its ranking. • Yandex began marking mobile friendly pages for mobile search. The

criteria for mobile friendliness includes webpage content (text, images, etc) fitting a small screen, vertical scrolling, and accessibility on mobile devices (Java applets, Flash and Silverlight plug-ins are not supported by most of modern mobile browsers). Flash Video media files, for instance, cannot appear in search results on the mobile version of Yandex’s video search, Yandex.Video, which only supports the HTML5 format.

• Yandex mobile search team confirmed that the speed of a site will also still play an important factor for ranking. Yandex will indeed penalize sites that direct users to mobile apps, similar to Google. In this instance, the mobile site is not fully answering the users’ query on their website. Webmasters should make sure to properly redirect to mobile versions as it will have some impact on the Yandex mobile SERP.

• Font size matters – a new factor ranking mobile-friendly pages in Yandex: a font size of a mobile page is at least 12px;

from keywords to better user experience

from penalties to better understanding of user behavior

from static to dynamic ranking factors

from desktop to mobile search

Evolution of Yandex’s Algorithms

Sources• https://www.optimism.ru/blog/kak-sdelat-informacionnyj-startap-populyarnym-anton-nosik-o-

sekretax-prodvizheniya-novostnyx-portalov/• https://seopult.ru/• https://www.searchenginejournal.com/11-things-need-know-yandex-seo/115208/• https://yandex.com/company/technologies/matrixnet/• https://yandex.com/support/webmaster/tic/what-is-tic.xml• http://techcrunch.com/2010/12/15/russian-search-engine-yandex-gets-a-semantic-injection-2/• http://www.lumolink.com/portfolio/online-marketing-projects/yandex-search-ranking-factors-

review/• http://searchengineland.com/russian-search-engine-yandex-to-bring-back-links-to-their-ranking-

algorithm-218815• http://buzzkeep.com/yandex-linkless-algo-decyphered-123864/• https://searchenginewatch.com/sew/how-to/2405308/will-yandex-penalizing-paid-links-affect-

other-search-engines-ranking-factors• http://siteclinic.ru/blog/web-analytics/istoriya-algoritmov-yandeksa/• http://wiki.rookee.ru/Nepot/• http://optimizatorsha.ru/yandex/minusinsk-case/• http://www.stateofdigital.com/state-yandex-seo-2016/• http://www.russiansearchtips.com/2015/05/breaking-new-yandex-ranking-algorithm-and-its-first-

implications/• http://russiansearchmarketing.com/yandex-search-algorithm-minusinsk-results-from-round-one-

of-penalties/• http://www.seonews.ru/events/kto-postradal-ot-minusinska/• http://marketing.by/mnenie/belorusov-ogradyat-ot-shokiruyushchey-reklamy-v-internete/• http://siteclinic.ru/blog/filtry/yandex-o-filtre-za-nakrutku-pf/• http://www.russiansearchtips.com/2011/06/how-to-optimize-for-yandex-seo-ranking-factors/

• http://searchengineland.com/yandex-rolls-out-mobile-friendly-algorithm-in-russia-code-name-vladivistok-241609

• http://siteclinic.ru/blog/free-webinars/dobavochnaya-cennost/• http://www.russiansearchtips.com/2009/06/regional-serps-on-yandex/• http://in4wp.ru/istoriya-sozdaniya-filtrov-algoritmov-yandeksa/#17-yanvarya-2008-vosmerka-sp-1• http://www.hmx.ru/geo.html• http://apanshin.ru/blog/geozavisimost-zaprosov-v-poiskovy-h-sistemah/• http://mexboy.ru/?p=2594• http://tekseo.su/seo-instrumenty/veb-servisy/proverka-geozavisimosti-zaprosov.php• http://pgdv.ru/samostoyatelno/geozavisimost• http://www.reraitex.ru/trends/trends_16.php• http://semantica.in/blog/geozavisimye-zaprosy-v-regionalnom-prodvizhenii-xecvt.html• http://searchengines.guru/archive/index.php/t-393109.html• http://pro-wordpress.ru/poleznoe/geozavisimost-poiskovyx-zaprosov.php• http://marketingland.com/six-components-mobile-seo-2016-167064• http://www.thedrum.com/news/2016/03/10/three-search-marketing-trends-you-need-know-

googles-amp-app-seo-and-machine-learning• https://devaka.ru/articles/fake-behavior-detection• http://in4wp.ru/istoriya-sozdaniya-filtrov-algoritmov-yandeksa/#8-aprelya-2009-arzamas-anadyr• https://vc.ru/p/seo-evolution• https://moz.com/google-algorithm-change• http://isif-life.ru/web/ags-filtr-yandeksa-kak-proverit-i-ne-popast-pod-filtr.html