Meta Robots Data: Why Do They Matter for SEO?
The so-called Meta Robots data in SEO are part of the head section of the HTML document and help search engines (especially Google). They control whether the page (URL) in question should be indexed and whether the links found on it should be followed or not.
In search engine optimisation, these tools are used for index management, which controls the discoverability of a page. It is essential for a URL to be captured (indexed) by the search engine in order to be found by searchers afterwards. Pages that are not in the index cannot be found.
Index Management Methodologies
With index management, you signal to the search engine whether a URL should be included in the index or not. In addition to the meta robots data, there are other methodologies to control indexing:
- The XML sitemap
- For international websites, the hreflang annotations
- Canonical management
- In a broader sense, also the robots.txt
- The Google Search Console
- Meta Robots Data (index/noindex & follow/nofollow)
The idea behind this is, firstly, that not all pages of a domain need to be found. For example, the shopping basket page of an online shop does not provide any value for a search query. This is a purely operational page in the checkout process that does not have any standalone content. Secondly, search engines capture pages through links. As a website operator, you do not always want a linked page to be indexed by the search engine.
Meta Robots: index/noindex
The HTML instruction for this command (HTML tag) is:
<meta name="robots" content="index" />
or
<meta name="robots" content="noindex" />
In the first case, this command indicates that the respective page should be included in the index and be findable. The second variant states exactly the opposite. As mentioned earlier, the basket page could be set to “noindex” – for which other pages does this make sense?
- All pages behind a login such as “My Account”, “My Wishlist” etc.
- The results page of the site’s own search function – this page is empty (if no search has been initiated) and thus has no value for the user. For this reason, in the worst case, such a page would only increase the bounce rate of a domain.
Since the search engine cannot trigger searches, no filled page is to be expected either.
Sometimes shop managers use filled search results pages for internal linking (e.g., when the corresponding category is missing). And that is why it is additionally important to set this page to noindex, as even filled search results do not provide any value and can worsen crawling. - Pages with session IDs
- Filter pages (depending on the case, instead of “noindex”, a canonical pointing to the unfiltered category page or, if the filter pages should not be crawled at all, exclusion via robots.txt can also be suitable; the former URL Parameters tool in Google Search Console was retired in 2022)
A recurring mistake is that de-indexing is forgotten (possibly during a relaunch). When the search engine then starts to index pages, those described above also get into the index. Often, the relevant pages are excluded via Robots.txt. However, since they are already in the index, this is pointless. At the same time, the meta data on the pages is rewritten – consequence:
- Search results (snippets) are enriched by the search engine with the note that the page is excluded.
- Nevertheless, the pages do not disappear from the index.
The problem here is that the robots.txt prevents crawling, and therefore the search engine does not pick up the new meta data because it does not crawl the page. Sounds confusing? It is, because robots.txt controls crawling, not the index.
Meta Robots: follow/nofollow
To curb link spam, for example in blog comments, Google introduced the “nofollow” link attribute in 2005. Wikipedia, which was increasingly being abused as a backlink source, has set links to external sites to “nofollow” since January 2007. This signals to search engines not to include these links in their relevance assessment.
To understand this, it is important to know that Google includes links from external sources to a domain in its relevance assessment. This led to link purchases in the early years of SEO. Google then put a stop to this by rolling out the Penguin update. So if you do not want to pass on link juice to other pages, you can also set your own links to “nofollow”.
This is done either as an attribute of the HTML hyperlink command or in the meta data:
<meta name="robots" content="follow" />
or
<meta name="robots" content="nofollow" />
In itself, this is only the “last resort” – it is actually not good style, as you consider the link you are setting to be important. Users are welcome to follow it, as it ideally represents added value in terms of content. In this case, it is only right and proper to pass on the link juice as well.
For internal links, the “nofollow” attribute makes even less sense, as internal links are never “bad” links. However, if you believe that the target page should not be indexed, then you should set it to “noindex” (see above). And for the search engine to “understand” this, it must follow the link to pick up the de-indexing information on the target page.
Do you want to improve your website’s index management and need support with technical SEO? Then find out more about our SEO services and get in touch! Do you want to learn more or have your team trained on this topic? Then check out our SEO seminars.
