Robots: Txt, Meta & X - The Snog, Marry & Avoid of the Web Crawling World - Brighton SEO Sep 2017

#BrightonSEO15th September 2017

@chrisgreen87www.strategiq.co

StrategiQChris Green

@chrisgreen87http://bit.ly/snog-marry-avoid

Robots: Txt, Meta & X The Snog, Marry & Avoid of the Webcrawling World

How do we knowthe best way to manage

Googlebot’s crawl/indexing?

There are many methods(we’re spoilt for choice really)

But the two most commonly misused are Robots.txt vs Meta Robots directives

To the casual observer they’re very similar ways of doing the same thing…

To block Google

But that’s not a helpful way of thinking of them

In many

it can stop you getting the most out of your site

circumstances

I’m going to run through a framework

to help to change this thinking

I’m going to run through a framework

to help to you make the right choices

But some words of warning

This is advanced stuffOne foot wrong & you could cause serious damage

They’re not always the first-choice

These are only part of your toolkit

There are so many “ifs” & “buts”

If we can finish today

with slightly more understanding

And a different approach

then we’re onto a winner!

Time to introduce the robots

Robots.txt

Meta Robots

X-Robots

Possibly the most important SEO tools

But which do you...

Or avoid?

But what does a slightly s**t

BBC 3 show have to do with SEO?

One site’s “snog” is

One site’s “snog” is another’s “marry”

Or perhaps even “avoid”

There are lots of thoughts on how to use these

There are lots of thoughts on how to use these - many are wrong

I’m going to show you a way of simplifying

things

To pick the right tool for the job

Know the problem you’re trying to fix!

Is it a crawl problem?

Google isn’t seeing enough of your site

Or an index problem?

Google’s indexing too much of it

We fix crawl problems with Robots.txt

User-agent: *

Disallow: /*mad-spider-trap*

And we fix index problems with Meta Robots

www.domain.com/crap-page

Identifying the problem

Index problems are simple to spot

Does your site look too big?

(in Google’s eyes)

But ID’ing a crawl problem...

Can be trickier

Look for spider traps

https://www.portent.com/blog/seo/field-guide-to-spider-traps-an-seo-companion.htm

Where does this “cost” you on crawl budget?

A word on crawl budget.

It’s “a thing”

http://searchengineland.com/google-explains-crawl-budget-means-webmasters-267597

But Google doesn’t publicise a site’s crawl

budget

You can work out a version of it yourself

Thanks to Yoast for this - https://yoast.com/crawl-budget-optimization/

Look at GSC Crawl stats average pages crawled per day

Look at GSC, how big Google sees your site as

Pages / Avg crawled per day =

Crawl Score

9,781 / 1,458 = 6.7

x 6.7 more pages than are getting crawled each day

If you have 10

You have 10x the pages that Google is crawling daily

A pretty big crawl problem!

But, how big is big?

< 1,000 pages, crawl budget is less of a problem

https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html?m=1

1,000 - 10,000 is moderate

10,000+ pages… things start to get “fun”

Crawl & Index problems aren’t mutually exclusive

Index bloat at scale can hurt crawl

Crawl issues can stop or slow the repair of

index issues

Some example scenarios

eCommerce filters which are getting indexed

(badly)

www.domain.com/shop/mens/trainers/size-12/red/

User-agent: *

Disallow: /*size*Disallow: /*red*

User-agent: *

Disallow: /*size*Disallow: /*red*

Not until index issue is cleared up*

*Unless...

User-agent: *

Noindex: /*size*Disallow: /*size*Noindex: /*red*Disallow: /*red*

Google isn’t “cool” with this

https://www.seroundtable.com/google-do-not-use-noindex-in-robots-txt-20873.html

But it’s proved to work

https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/

http://ohgm.co.uk/de-index-pages-blocked-robots-txt/

Only ~0.3% of the Majestic Million use this method

Don’t be too aggressive though!

Be aware that some filtered pages can be

worth indexing

Blog taxonomies

www.domain.com/blog/blog-category/www.domain.com/blog/tags-bloody-tags/

eCommerce site without indexed filters but x6+ crawl

User-agent: *

Disallow: /*filters*

(& meta robots just in case)

Other misc pages?

Just noindex

Anything else?

Back to my original premise

Meta Robots is my“marry”

Robots.txt is my “snog”

It really can make the difference

Meta robots replaced with robots.txt disallow:

But it is easy to screwit up

What about x-robots?

For when meta robots isn’t possible…

… assuming you can edit htaccess

It’s not my “avoid” though

What is?

The lazy option!

User-agent: *

Disallow: /*pointless*Disallow: /*disallow-rules*Disallow: /*instead-of*Disallow: /*fixing-the-problem.html

www.domain.com/200000-filtered-combos

User-agent: *

Disallow: /we-should-write-better-content/

#but don’t want to prioritise

The “best choice” depends onyour limitations

Do you have all the access you need?

Or enough buy-in?

Otherwise, some workarounds are better than doing nothing

Meta Robots via GTM

https://moz.com/blog/seo-changes-using-google-tag-manager

Robots.txt when no other option

User-agent: *

Disallow: /better-than-doing-nothing/

Key takeaways

Is it a crawl or index problem?

Check what you can change

Check what you can changeAnd what you can’t…

Make the “best case” fix

Implement, crawl & check again!

Use this flowchart to help

http://bit.ly/bseo-flow

Thank you.http://bit.ly/snog-marry-avoid@chrisgreen87

Robots: Txt, Meta & X - The Snog, Marry & Avoid of the Web Crawling World - Brighton SEO Sep 2017

Marketing

Transcript of Robots: Txt, Meta & X - The Snog, Marry & Avoid of the Web Crawling World - Brighton SEO Sep 2017

Lecture 9 Evolution: Lecture 9 Evolution: Now Playing: Snog “The Human Germ”

Crawling the world

Crawling and web indexes. Today’s lecture Crawling Connectivity servers.

Crawling - Northeastern University

Creeping, crawling critters

Crawling EDGAR - unc.edu

UK HEIs and the OERu: Snog, Marry, Avoid?

Snog Marry Avoid? corporates versus activists in the public sector

Intelligent web crawling

Crawling and Flying Insects in Albertainsectsofalberta.com/.../crawling-and-flying-insects-in-alberta.pdf · Crawling and Flying Insects in Alberta ... Cersi (apendages) Ovipositor

Web Crawling & Crawler

Advanced Crawling Techniques Chapter 6. Outline Selective Crawling Focused Crawling Distributed Crawling Web Dynamics.

Brighttalk snog marry avoid 7 february 2013

Adaptive Focused Crawling

Crawling the Web

5 Benefits of Web Crawling Services Over In-house Crawling

Google crawling SEO

Crawling The Web For a Search Engine Or Why Crawling is Cool.

Crawling and Walking

Crawling HTML