Google’s John Mueller On Blocking Robots.txt Files From Being Indexed

    Google’s John Mueller recently provided some advice on how to block robots.txt and sitemap files from being indexed in search results.

    This advice was prompted by a tweet from Google’s Gary Illyes, who randomly pointed out that robots.txt can technically be indexed like any other URL. While it provides special directions for crawling, there’s nothing to stop it from being indexed.

    Here’s the full tweet from Illyes:

    “Triggered by an internal question: robots.txt from indexing point of view is just a url whose content can be indexed. It can become canonical or it can be deduped, just like any other URL.
    It only has special meaning for crawling, but there its index status doesn’t matter at all.”

    In response to his fellow Googler, Mueller said the x-robots-tag HTTP header can be used to block indexing of robots.txt or sitemap files. That wasn’t all he had to say on the matter, however, as this was arguably the key takeaway:

    “Also, if your robots.txt or sitemap file is ranking for normal queries (not site:), that’s usually a sign that your site is really bad off and should be improved instead.”

    So if you’re running into the problem where your robots.txt file is ranking in search results, blocking it using the x-robots-tag HTTP header is a good short-term solution. But if that’s happening then there are likely much larger issues to take care of in the long-term, as Mueller suggests.

    Recent Articles

    How to change your Facebook password or reset it

    Change or reset your Facebook password for your web or app account Your Facebook password is a vital piece of information if you're going to...

    This new controller will turn Samsung Galaxy Note 20 into a mobile Xbox

    MOGA XP5-X Plus will help turn your Android phone into a mobile Xbox With the Project xCloud beta bringing Xbox game streaming to Android phones and set...

    Future MacBook Pro trackpads may be larger, adjustable, and backlit

    Apple is investigating replacing the regular MacBook Pro trackpad with a virtual one that can dynamically shrink, grow, or change position as needed. The current MacBook Pro...

    How To Get ‘Fall Guys’ Free On PlayStation Plus Today As Its August Feature

    Fall Guys is smartly trying to follow the same road as Rocket League with a free month on PS Plus as part of Sony’s...

    Apple expands Apple Maps ‘Look Around’ feature to Japan

    Apple is slowly expanding the Look Around feature that was introduced last year with iOS 13. Previously available only in the United States, the...

    Latest Stories

    Stay on op - Ge the daily news in your inbox

    Do NOT follow this link or you will be banned from the site!
    Translate »