Robots.txt tells crawlers which paths they may request. A Disallow rule can silently block pages, or your whole site, from being crawled. The key trap: robots.txt controls crawling, not indexing, so blocking a page does not remove it from Google and prevents any noindex tag from working. To hide a page, allow crawling and use noindex. We audit and fix robots.txt for $49-$79.
Robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may and may not request. It is the first thing Google fetches before crawling anything else. A single Disallow rule can silently keep Google out of pages, sections, or your entire site, and because the file is invisible to normal visitors, the block often goes unnoticed until traffic drops or Search Console flags "Blocked by robots.txt." The critical thing to understand, and the source of most robots.txt disasters, is that robots.txt controls crawling, not indexing. Blocking a URL in robots.txt stops Google from fetching it, but Google can still index that URL if other pages link to it, showing it as a bare result with no description because it was never allowed to read the page. And because Google cannot crawl a blocked page, it can never see a noindex tag on that page, so blocking in robots.txt actively prevents the one signal that would properly remove a page from the index. This is why people get stuck with pages they cannot get into Google and pages they cannot get out. Getting robots.txt right means blocking only what should never be crawled, like admin and utility paths, while leaving everything you want indexed fully accessible.
Fix This Error Now →Robots.txt Blocking Google can be caused by several issues. Here are the most common.
A single line, Disallow: / under User-agent: *, tells every crawler to stay out of every path on the domain. This is the most catastrophic robots.txt mistake and the most common, because it is exactly what staging and development environments use to stay out of search. When that configuration ships to production, the whole site is walled off from Google.
Staging sites are correctly blocked from crawling so they do not compete with or duplicate the live site. The failure happens at launch: the site goes live but the blocking robots.txt, or the environment setting that generates it, is never switched off. Within days Google stops crawling and pages begin dropping out of the index.
Rules meant to hide one folder can be written too broadly and catch real content. Disallowing /blog/ to hide a single draft area, or a wildcard that matches more than intended, quietly removes whole sections from crawling. Because the rest of the site works, the missing traffic is easy to misattribute to something else.
Older robots.txt files often disallow /wp-includes/ or asset directories. Google needs to fetch CSS and JavaScript to render and understand a page, so blocking these degrades how Google sees your content and can hurt both indexing and rankings. Google explicitly recommends allowing crawlers to access the resources a page needs to render.
The single most common misunderstanding, and Google states the correction plainly: robots.txt "is not a mechanism for keeping a web page out of Google," and "a page that's disallowed in robots.txt can still be indexed if linked to from other sites." Blocking a page can leave it indexed with no snippet. Google's guidance is direct: "To keep a web page out of Google, block indexing with noindex or password-protect the page." Robots.txt is the wrong tool for hiding a page from search.
Robots.txt matching is literal and case-sensitive on the path, and a misplaced slash, a missing wildcard, or an Allow and Disallow that conflict can block far more or far less than intended. A rule that looks right can behave completely differently from what was meant, which is why the rules should always be tested rather than assumed.
If robots.txt returns a server error rather than a clean response, Google may treat the site as temporarily disallowed and pause crawling entirely to avoid breaking rules it cannot read. A robots.txt that 500s is worse than one that 404s, which Google reads as "nothing disallowed."
70-80% of our customers have WordPress sites. Here are WordPress-specific causes for this error.
Under Settings, Reading, WordPress has a checkbox to discourage search engines from indexing the site. It is turned on during development and, far too often, never turned off at launch. It makes WordPress emit a blocking signal and a noindex directive site-wide. This one checkbox is behind a huge share of "my new WordPress site will not index" cases.
WordPress serves a virtual robots.txt unless a physical file exists in the web root, in which case the physical file wins. Editing the wrong one, or having both, leads to changes that appear to do nothing because Google is reading a different file than the one being edited.
Yoast, Rank Math, and security plugins can each write or override robots.txt rules. A plugin update or a misclicked setting can add a Disallow you did not intend, so the rules need to be traced to whichever plugin is actually generating the live file.
Fetch your live robots.txt and identify exactly which file is being served, virtual, physical, or plugin-generated, so edits actually take effect
Map every Disallow and Allow rule against the URLs you want indexed to find rules that block real content, whether whole-site or overly broad
Remove or narrow the blocking rules while preserving intentional blocks for admin, cart, checkout, and internal search paths
Ensure CSS, JavaScript, and image resources are crawlable so Google can fully render your pages
On WordPress, verify the "Discourage search engines" setting is off and confirm no plugin is re-adding a block
Correct the crucial distinction: for pages that should leave the index, unblock them in robots.txt and apply a noindex tag instead
Confirm robots.txt returns a clean status and declare your sitemap with a Sitemap line in the file
Test the corrected rules against key URLs, then request re-crawling and monitor Search Console until the blocked-URL count clears
Fixed in 2 hours or your money back. We do not waste time.
No hourly billing. You know the price before we start.
Cannot fix it? You do not pay. Zero risk to you.
Our Google & SEO Issues team has fixed thousands of sites with this exact issue. 2-hour turnaround, guaranteed.
Two quick checks. In Google Search Console, the Page Indexing report lists URLs under "Blocked by robots.txt," and the URL Inspection tool tells you for any single URL whether crawling is allowed. You can also open yourdomain.com/robots.txt directly and look for Disallow rules, especially a Disallow: / which blocks everything. If you see pages you want indexed matching a Disallow rule, robots.txt is the culprit, and that is exactly what we audit and fix.
Not reliably, and this trips up almost everyone. Google says it outright: robots.txt "is not a mechanism for keeping a web page out of Google," and "a page that's disallowed in robots.txt can still be indexed if linked to from other sites." Robots.txt blocks crawling, not indexing. Google will not fetch a blocked page, but it can still index the URL if it finds links to it, showing a bare listing with no description. And because Google cannot crawl the page, it can never see a noindex tag there, so the block prevents the correct removal signal from working. Google's stated fix is to keep a page out of Google with a noindex tag or a password, not with robots.txt.
Yes, most sites should, but it should be permissive by default. Use it to keep crawlers out of areas that genuinely should never be crawled, admin, cart and checkout, internal search results, and to point to your sitemap. It should not block the content you want ranked, and it should allow the CSS and JavaScript Google needs to render pages. A good robots.txt is short and mostly gets out of Google's way. A dangerous one quietly blocks things you did not mean to.
Yes. A Disallow: / that reaches production, usually carried over from a staging setup or a leftover WordPress setting, blocks Google from crawling every page. Over the following days and weeks Google stops re-crawling and begins dropping pages it can no longer verify. Because visitors see the site working normally, the cause is often missed while traffic quietly collapses. The fix is fast once identified, which is why diagnosing robots.txt early matters so much.
Most robots.txt fixes cost $49 to $79. Removing an errant block, correcting the WordPress setting, or rewriting rules to unblock content while keeping the right areas protected is quick work. Cases that involve conflicting plugin-generated files, wildcard rules gone wrong across a large site, or pages that need to be moved from a robots.txt block to a proper noindex run toward $99. You get a firm quote before work starts.
Every technical claim on this page traces back to primary documentation and the named vendor references below.
Why your pages are not indexed, what every Search Console status means, and how to get back into Google.
Visit the SEO & Search Console Repair hub →Get robots.txt blocking google fixed today. Expert engineers. 2-hour guarantee.
Fix My Error Now →