Technical SEO and Algorithmic Search

Python Log File Analysis: Writing Scripts to Track Googlebot Budgets and Isolate Internal Crawl Waste Patterns

Parašė Kinetikatech Operations Team · Atnaujinta 2026-09-29

Server logs reveal what crawlers actually do, not what Search Console reports. Googlebot's crawl budget is wasted on low-value URLs, redirect chains, and error responses. This guide writes Python scripts to parse logs, identify waste, and optimize crawl allocation.

Pagrindiniai punktai

Parse log format. Apache and Nginx logs have standard formats; extract timestamp, URL, status, and user agent.
Filter for Googlebot. Use the verified Googlebot IP ranges and user agent strings.
Calculate crawl budget. Count requests per day and identify trends over time.
Find crawl waste. Identify 404s, redirect chains, and low-value URLs consuming budget.
Analyze crawl depth. Track how deep Googlebot crawls and where it stops.
Correlate with Search Console. Compare log data with GSC indexing reports.
Generate reports. Automate daily or weekly reports on crawl efficiency.
Act on findings. Fix 404s, reduce redirect chains, and block low-value URLs.

Architecture

Logs are parsed into structured records, filtered for Googlebot, and aggregated by URL pattern, status code, and time. Reports highlight waste and trends. Actions are derived from the analysis and tracked over time.

Production configuration

Parse and filter Googlebot logs python

import re, gzip, collections

LOG_PATTERN = re.compile(
    r'(\S+) - - \[(.+?)\] "(\S+) (\S+) (\S+)" (\d+) (\d+) "([^"]*)" "([^"]*)"'
)

def parse_googlebot_log(path):
    entries = []
    with open(path) as f:
        for line in f:
            m = LOG_PATTERN.match(line)
            if m and 'Googlebot' in m.group(9):
                entries.append({
                    'ip': m.group(1), 'time': m.group(2),
                    'method': m.group(3), 'url': m.group(4),
                    'status': int(m.group(6)), 'ua': m.group(9),
                })
    return entries

Analyze crawl waste python

def analyze_waste(entries):
    waste = collections.defaultdict(int)
    for e in entries:
        if e['status'] == 404:
            waste['404'] += 1
        elif e['status'] in (301, 302):
            waste['redirect'] += 1
        elif e['status'] >= 500:
            waste['server_error'] += 1
    total = len(entries)
    for k, v in waste.items():
        print(f'{k}: {v} ({v/total:.1%})')
    return waste

Generate crawl report python

def daily_report(entries):
    by_day = collections.defaultdict(int)
    for e in entries:
        day = e['time'].split(':')[0]
        by_day[day] += 1
    for day in sorted(by_day):
        print(f'{day}: {by_day[day]} requests')
    # top URLs
    urls = collections.Counter(e['url'] for e in entries)
    print('\nTop 10 URLs:')
    for url, count in urls.most_common(10):
        print(f'  {count:5d} {url}')

Edge cases from production

Log rotation can split sessions; ensure your parser handles rotated files.
Googlebot IPs change; verify against published ranges, not just user agent.
Crawl budget varies by site authority; compare against your own baseline.
CDN logs may not include origin requests; use origin logs for accuracy.

Dažniausiai užduodami klausimai

What is crawl budget?

The number of URLs Googlebot will crawl on your site per day.

How do I reduce crawl waste?

Fix 404s, reduce redirects, and block low-value URLs with robots.txt.

Can I see Googlebot in real time?

Yes, by tailing server logs and filtering for Googlebot user agent.

How often should I analyze logs?

Weekly for trends, daily during site changes.

What is the biggest waste source?

Usually parameterized URLs and redirect chains.

Kaip Kinetikatech padeda su technical seo and algorithmic search

Jums nereikia tapti technical seo and algorithmic search ekspertu: Crawl budget, rendering, and structured data engineered for search engines and AI answer systems. Būtent tam ir skirta Svetainės spartos optimizavimas ir Serverių administravimas: paleidimas, sutvirtinimas, stebėjimas ir budintis žmogus dieną ir naktį.

Pasikalbėkite su inžinieriumi