A crawl error means Googlebot could not fetch a page, so it cannot be indexed or ranked. Google deprecated the old Crawl Errors report when it launched the new Search Console in January 2019, so crawl problems now appear in the Crawl Stats report under Settings and in the Page Indexing report. Causes include server errors, DNS failures, robots.txt blocks, redirect loops, and firewalls blocking Googlebot. We diagnose the failing layer and fix it for $49-$99.
A crawl error means Googlebot tried to fetch a page on your site and could not, which is a problem because crawling is the first step before indexing. If Google cannot reach a page, it cannot index it, and a page that is not indexed cannot appear in search results at all. Crawling and indexing are two separate stages, and a crawl error stops you at the very first one. The confusion many site owners have is that Google retired the old standalone "Crawl Errors" report back in 2019. Crawl problems now surface in two places: site-level fetch failures like DNS, server connectivity, and robots.txt fetch errors appear in the Crawl Stats report under Settings, while page-level problems like server errors, redirect errors, blocked URLs, and not-found pages appear in the Page Indexing report. So there is no single "crawl errors" screen anymore, which is exactly why people struggle to find and fix them. The underlying issue is always the same: something between Google and your content, your server, your DNS, your robots.txt, a firewall, or a redirect, is getting in the way. The fix depends entirely on which layer is failing, and diagnosing the right layer is what separates a quick fix from weeks of guessing.
Fix This Error Now →Google Crawl Errors can be caused by several issues. Here are the most common.
Your server returns a 500, 502, or 503 when Googlebot requests a page, often only under the load of crawling or at specific times. Repeated server errors make Google slow its crawl rate and eventually drop the affected pages from the index. These are easy to miss because a human visitor may never hit the error, but Googlebot does, and the server logs show it.
Google cannot resolve your domain name to an IP address, or the connection times out before your server responds. This shows in Crawl Stats as host-level failures. Causes include a misconfigured DNS record, an expired domain, an overloaded server refusing connections, or a firewall dropping Googlebot traffic. When the host itself is unreachable, no page can be crawled.
Before crawling anything, Google fetches your robots.txt. Google's own documentation describes the exact handling: if that file returns a 5xx server error or is unreachable, Google stops crawling the site for the first 12 hours, then falls back to the last cached copy for up to 30 days while it keeps retrying, and after 30 days either assumes no restrictions or, if the whole site is still unavailable, stops crawling entirely. A robots.txt that 500s is therefore worse than one that 404s, which Google reads as no restrictions. Separately, a Disallow rule blocks Googlebot from paths you may not have meant to block, which appears as blocked URLs in the Page Indexing report.
Googlebot follows redirects, but a chain of more than five hops, a loop where A redirects to B and back to A, or a redirect pointing to a broken URL all count as redirect errors. Google gives up before reaching the real content. These pile up after site migrations when multiple redirect rules are layered on top of each other.
If your server is consistently slow to respond, a high Time to First Byte, Google throttles its own crawl rate to avoid overloading you, so fewer of your pages get crawled per visit. Google states this directly in its crawl budget guidance: "If the site responds quickly for a while, the limit goes up... If the site slows down or responds with server errors, the limit goes down and Google crawls less." On large sites this means new and updated pages wait much longer to be discovered and indexed. A slow server spends crawl budget inefficiently.
Overly aggressive security software, a WAF, or a CDN rate-limiting rule can mistake Googlebot for an attacker and serve it a 403 or block it entirely. Because the block is conditional on the visitor, your site looks fine to you while Google is quietly locked out. Verifying the request is genuinely from Googlebot and allowlisting it fixes this.
Pages that return a 200 OK but show empty or error-like content are treated by Google as soft 404s, and genuinely removed pages that return a hard 404 without a redirect both show up when Google crawls URLs that no longer serve real content. A few are normal; hundreds signal a structural problem.
70-80% of our customers have WordPress sites. Here are WordPress-specific causes for this error.
Plugins like Wordfence or iThemes Security can block or challenge Googlebot if a rule is too strict or a rate limit is set too low, so Google sees a 403 or a challenge page instead of your content. Verify the crawler is real and allowlist it rather than loosening security globally.
Changing your permalink settings without flushing rewrite rules, or a corrupted .htaccess file, can make internal URLs return 404s to Googlebot. Re-saving permalinks regenerates the rules and often clears a wave of sudden crawl errors.
An old robots.txt that disallows /wp-includes/ or /wp-content/ can stop Google from fetching the CSS and JavaScript it needs to render the page, which degrades how Google understands and indexes it. Modern sites should allow Google to crawl these assets.
Pull the Crawl Stats report under Settings and the Page Indexing report to separate host-level failures (DNS, server, robots.txt) from page-level errors (5xx, redirects, blocked, not found)
Review server access logs filtered to genuine Googlebot requests to see the exact status codes Google receives, which often differ from what a browser sees
Fix server errors at the source, whether that is a crashing PHP process, an exhausted resource limit, or an intermittent host problem, then confirm a clean 200 response
Verify robots.txt returns a clean status and remove any Disallow rules blocking content you want indexed, while keeping intentional blocks for admin and utility paths
Collapse redirect chains so every old URL points in a single hop to the correct live destination, and break any redirect loops
Allowlist verified Googlebot in your firewall, security plugin, and CDN so legitimate crawling is never rate-limited or challenged
Address slow response times with caching and hosting fixes so Google can crawl more pages per visit
Use the URL Inspection tool to request re-crawling of fixed pages and monitor the reports over the following weeks to confirm the errors clear
Fixed in 2 hours or your money back. We do not waste time.
No hourly billing. You know the price before we start.
Cannot fix it? You do not pay. Zero risk to you.
Our Google & SEO Issues team has fixed thousands of sites with this exact issue. 2-hour turnaround, guaranteed.
Google retired the standalone Crawl Errors report in 2019, which is why so many people cannot find it. Crawl problems now live in two places. Site-wide fetch failures like DNS errors, server connectivity, and robots.txt fetch problems appear in the Crawl Stats report, which is under Settings in Search Console. Page-level issues like server errors, redirect errors, blocked URLs, and not-found pages appear in the Page Indexing report under the list of reasons pages are not indexed. We check both to get the full picture.
Indirectly but seriously. A crawl error does not itself apply a ranking penalty, but if Google cannot crawl a page, it cannot index it, and an unindexed page cannot rank for anything. Worse, repeated server errors and a slow, unreliable site cause Google to reduce how often and how deeply it crawls you, so new content takes longer to appear and your whole site becomes less fresh in the index. Clearing crawl errors protects the entire indexing pipeline.
Because the failure is often conditional on the visitor. A security plugin, firewall, or CDN may serve Googlebot a 403 or a challenge while showing you the real page. Or the server may only fail under the concurrent load of crawling, which a single browser visit never triggers. This is exactly why we read the server logs for actual Googlebot requests rather than trusting how the page looks in a browser.
The fix itself is usually fast. Once the underlying cause is resolved, we request re-crawling of key pages through the URL Inspection tool, which often triggers a fresh crawl within 24 to 48 hours. The reports in Search Console update on a delay, so the error counts typically drop over one to two weeks as Google re-crawls the affected URLs and confirms they now respond correctly.
Crawl budget is what Google calls the set of URLs it can and wants to crawl on your site, set by how much load your server can take and how much demand Google has for your content. Most sites never need to think about it. Google says it matters mainly for large sites, its stated thresholds are sites with more than one million unique pages that change weekly, or more than 10,000 unique pages that change daily, or any site with a large share of URLs stuck in "Discovered - currently not indexed." If you fit those, a slow server, endless low-value URLs, and crawl errors all waste budget and slow how fast your real pages get crawled. Below that scale, keeping a clean sitemap and a fast, error-free site is enough.
Most crawl error fixes cost $49 to $99 depending on the cause. A single robots.txt or redirect fix is at the low end. Diagnosing intermittent server errors, firewall blocks, or a large backlog of redirect chains across a migrated site takes more work and lands at $99 to $149. You get a firm quote before any work begins.
Every technical claim on this page traces back to primary documentation and the named vendor references below.
Why your pages are not indexed, what every Search Console status means, and how to get back into Google.
Visit the SEO & Search Console Repair hub →Get google crawl errors fixed today. Expert engineers. 2-hour guarantee.
Fix My Error Now →