Log file analysis for SEO: finding crawl waste and opportunity

Log files are where you stop guessing and start seeing what search engines do on your site.

Below is how we approach log file analysis for SEO: spotting crawl waste, finding pages that deserve more crawling, and turning a pile of server requests into a fix list.

What log file analysis is

Your server logs record requests made to your website. That includes humans, bots, uptime monitors, scrapers, your own team, and search engine crawlers.

When you analyse logs for SEO, you are usually trying to answer questions like:

Which URLs does Googlebot crawl, and how often?
How much crawl is going to redirects, 404s, parameter URLs, and duplicate paths?
Are important pages being crawled often enough, or at all?
Are bots hitting slow pages, erroring, or timing out?
Are there sections of the site that are effectively invisible to search engines?

Google defines crawl budget as the set of URLs it can and wants to crawl, shaped by crawl capacity and crawl demand. That framing matters, because it means you can influence both sides by improving site health and making the right URLs more desirable to crawl.

Google says crawl budget is mainly a constraint for very large sites (roughly a million or more unique pages changing weekly) and for sites with 10,000 or more pages that change daily. Smaller sites still get value from logs, because the same data shows errors, redirect waste and crawl traps.

Search Console’s Crawl stats report (under Settings) shows Googlebot requests by response code, file type, purpose and Googlebot type for the last 90 days. It is a useful cross check, but it covers Google only and lists sample URLs rather than every request. Logs show every request from every bot.

When log file analysis is worth doing

You do not need log files for every site. But you almost always want them if any of the following apply:

You have 10,000 plus URLs, or rapidly growing indexable pages
You run ecommerce with faceted navigation and filters
You have a publisher style archive, tags, author pages, and pagination
You have a large blog with years of content and lots of internal search
You have frequent deployments, migrations, or URL structure changes
You see spikes in crawl requests, or Search Console crawl stats look odd
You have ongoing indexation issues, despite ‘doing the basics’

If you are in one of those buckets, logs become one of the quickest ways to find out where SEO effort is being wasted.

What you need to get started

Access to logs

You typically need one of these:

Raw server logs from your hosting provider (Nginx, Apache, IIS)
CDN logs (Cloudflare, Fastly, Akamai)
Load balancer logs
A managed platform export (varies by host)

If you are not sure where the logs live, your developers or hosting provider will know. If a CDN serves cached pages, pull the CDN logs as well, because cached requests may never reach the origin server.

A timeframe

For most audits, aim for:

30 days to spot patterns
60 to 90 days for seasonality and stability
Shorter windows (7 to 14 days) if you are investigating a specific incident

Tools

You can do log analysis a few ways:

Excel or Google Sheets (fine for small sites, not fun for big ones)
Screaming Frog Log File Analyser (practical for many SEO teams)
BigQuery, SQL, and Python (great at scale)
Dedicated log platforms like Splunk, Elastic, Datadog (if the business already uses them)

That matters because log work is rarely one and done. You want a repeatable workflow you can run monthly or quarterly.

What to look for inside a log line

Every log format is slightly different, but you are normally looking for:

Timestamp
Requested URL path (and query string)
Status code (200, 301, 404, 503)
User agent (to identify Googlebot, Bingbot, etc.)
IP address (sometimes useful, but be careful with privacy)
Bytes served
Response time (if available, extremely useful)

Your first job is hygiene:

Parse the logs into a structured table
Normalise URLs (lowercase where appropriate, remove fragments, consistent trailing slashes)
Split path and query string so you can analyse parameters properly
Identify bots using user agent patterns

Step by step: a practical log file analysis workflow

Step 1: Filter to the bots you care about

Start with Googlebot, and usually Googlebot Smartphone too. You can widen later.

Plenty of bots pretend to be Googlebot. Before you make decisions based on what Googlebot crawled, verify the requests: run a reverse DNS lookup on the IP and confirm it resolves to googlebot.com or google.com, or check the IP against the Googlebot ranges Google publishes.

Step 2: Group URLs into meaningful buckets

Staring at millions of rows is pointless unless you group.

Common grouping approaches:

By directory: /blog/, /category/, /products/, /collections/
By template: product detail pages, category pages, blog posts, internal search
By parameter set: ?sort=, ?filter=, ?page=, ?q=
By status code group: 2xx, 3xx, 4xx, 5xx

Your goal is to quickly answer: where is crawl concentrated?

Step 3: Separate crawl volume from crawl value

A directory getting 40% of your crawl does not mean it is important. It might mean it is a trap.

Crawl value is about whether those crawls support indexation and revenue outcomes:

Key landing pages
Product and category pages that convert
High intent B2B service pages
Content that supports topical authority and demand capture

Step 4: Identify crawl waste patterns

This is where the fun starts. Typical crawl waste looks like:

Redirect chains and loops

If Googlebot hits:

URL A (301) to URL B (301) to URL C (200)

That is 2 wasted requests before it even reaches content. Multiply by thousands of URLs and you are burning crawl on avoidable plumbing.

What to do:

Update internal links to point directly to the final URL
Reduce chain length to 1 redirect max
Fix redirect loops immediately

404s and soft 404s

404s are not automatically ‘bad’, but frequent bot hits to 404s usually means:

Broken internal links
Old URLs still referenced externally
Bad pagination logic
Parameter URLs generating nonsense

A soft 404 is different. The URL returns a 200 status, but the page shows an error message, an empty category or an expired product. Search Console lists these as ‘Soft 404’ in the Page indexing report, and logs show how often bots keep requesting them.

What to do:

Repair internal links
Add redirects where there is a relevant replacement
If the URL should never exist, remove the source creating it

Return a real 404 or 410 when a page has no replacement
Fill or remove empty category pages rather than serving them with a 200 status

Parameter bloat and faceted crawl traps

Ecommerce and large content sites often create infinite combinations:

Sort orders
Filters
Tracking parameters
Pagination mixed with filters

What to do:

Decide which parameter combinations deserve to be indexable
Block true crawl traps via robots.txt where appropriate
Use canonical tags, internal linking rules, and noindex carefully
Keep your sitemap to URLs you want indexed

Robots.txt and noindex do different jobs. A URL blocked in robots.txt is not crawled, so Google never sees a noindex tag or canonical on it, and the URL can still be indexed from links without its content. If you want a URL out of the index with noindex, keep it crawlable until Google has processed the tag.

Duplicate URL variants

Logs often show bots crawling several versions of the same page:

HTTP and HTTPS versions
www and non-www hosts
Trailing slash and non trailing slash versions
Parameters that do not change the content

What to do:

301 every variant to one version
Point canonical tags and internal links at that version
List only that version in the XML sitemap

Internal search pages being crawled

If Googlebot is crawling:

/search?q=whatever
/?s=whatever
/site-search/results?query=whatever

You usually have a quality issue brewing. Internal search is often thin, duplicate, and infinite.

What to do:

Noindex internal search result pages
Prevent internal linking to them where possible
Block crawl in robots.txt only if you are confident you are not blocking important URLs by accident

Calendar and infinite pagination spaces

Classic trap:

/events/2024/01/
/events/2024/02/
/events/2034/09/

Bots will crawl forever if you let them.

What to do:

Limit infinite generation
Use sensible pagination and archive boundaries
Remove internal links to non valuable archive combinations

Thin tag pages and duplicate archives

Blogs often create:

/tag/seo/
/tag/seo-audit/
/category/seo/
/author/rick/

If these pages are thin and heavily crawled, that is a waste.

What to do:

Consolidate taxonomy
Noindex thin archive pages
Strengthen the handful you want ranking

Step 5: Find opportunity as well as waste

Crawl waste is only half the story. Logs also show where you are under investing.

Important pages barely crawled

If your money pages are crawled once a month while your filters are crawled 50 times a day, you have a prioritisation problem.

What to do:

Improve internal linking to priority pages
Reduce crawl noise from low value URL variants
Check that priority pages return 200, load fast and are not blocked
Include them in XML sitemaps and keep sitemaps clean

New pages not being discovered quickly

If you publish often, you want Googlebot to find new URLs quickly.

What to do:

Add ‘new content’ modules on strong pages
Link to new items from category pages
Link to new pages from your main navigation and hub pages, as well as the blog feed
Submit updated sitemaps

Orphan pages that are crawled anyway

Sometimes Googlebot crawls pages that your own site barely links to, usually because of:

Old backlinks
Past internal links that were removed
XML sitemaps still including them

To find orphan pages, compare four lists: URLs your crawler reaches through internal links, URLs in your XML sitemaps, landing pages in Search Console or analytics, and URLs in your logs. Pages that appear in the last three but not the first are orphans.

That can be an opportunity:

If the page is good, bring it back into your internal linking system
If it is outdated, redirect or consolidate it
If it is thin, improve it or remove it

Wasted crawl on slow pages

If your logs include response times, look at:

High crawl frequency URLs with slow response
Pages returning 5xx errors under load
Crawl spikes that coincide with server issues

What to do:

Fix performance bottlenecks on the most crawled templates first
Consider caching strategy improvements
Work with devs to reduce expensive queries
Use monitoring and reporting so you see it before rankings drop

AI crawlers in your logs

The same logs show which AI crawlers request your pages. The user agents worth filtering for:

GPTBot: OpenAI’s crawler for collecting model training data
OAI-SearchBot: OpenAI’s crawler for ChatGPT search results
ChatGPT-User: fetches made when a ChatGPT user’s request needs a page
ClaudeBot: Anthropic’s crawler
PerplexityBot: Perplexity’s crawler

Google-Extended will not appear in your logs. It is a robots.txt token that controls whether Google can use your content for Gemini training and grounding, and the crawling itself is done by Googlebot.

Questions to answer for each AI crawler:

Which pages does it request, and are your priority pages among them?
Which status codes does it get? CDN and bot protection settings can return 403s or challenges to AI crawlers without anyone deciding to block them.
Is it stuck in the same parameter URLs and crawl traps as Googlebot?
Does robots.txt allow or block it, and does that match what the business wants?

User agents are easy to fake, so verify these requests too. OpenAI and Perplexity publish IP ranges for their crawlers.

Most AI crawlers are widely reported to fetch raw HTML without running JavaScript, so content that only appears after rendering may not reach them. Our JavaScript SEO guide covers rendering options.

A crawl is not a citation. Logs show that a crawler fetched a page, not whether any AI answer used it. To see which answers include your brand and which sources they draw on, track the answers themselves.

Turning findings into a fix list you can ship

A good log file analysis ends with a prioritised backlog, not a 40 page deck nobody touches.

Use a simple prioritisation grid:

Impact: will this free crawl for important URLs or improve indexation quality?
Effort: is it a quick config change or a full platform rewrite?
Risk: could this accidentally block or deindex valuable pages?

Typical high impact, low effort fixes:

Update internal links to remove redirect chains
Remove sitemap URLs that redirect or 404
Noindex internal search results
Fix broken links creating 404 crawl loops
Add rules to prevent infinite URL generation

Higher effort but often worth it:

Faceted navigation strategy redesign
Parameter handling logic improvements
Consolidation of taxonomy and archives
Performance work on heavy templates

How often to run log analysis

For most UK businesses:

Quarterly is a sensible baseline
Monthly if you are large, fast moving, or ecommerce heavy
Weekly if you are in a migration, incident response, or indexation crisis

Log analysis is especially valuable after major changes:

Site migrations
New faceted systems
Large internal linking changes
Platform upgrades

Common mistakes that make log analysis useless

Looking at ‘all bots’ at once and getting lost in noise
Not grouping URLs into templates, so you cannot see patterns
Treating crawls as equal, instead of separating value from waste
Forgetting that a crawl is not an index, and an index is not a ranking
Making robots.txt changes without understanding the knock on effects
Ignoring server performance signals, especially 5xx and slow responses

A simple checklist you can use on your next run

Use this as your next pass:

Googlebot crawl volume by directory and template
Top crawled URLs, and whether they deserve it
Percentage of crawls by status code group (2xx, 3xx, 4xx, 5xx)
Top redirect sources and chain length
Top 404 URLs and where they are linked from
Parameter URL volume and the worst offenders
Crawl frequency of your top 20 priority pages
New content discovery speed (first crawl after publish)
Response time distribution for the most crawled templates
Sitemap coverage versus what Googlebot crawls
AI crawler requests and status codes by user agent