robots.txt-regelparser
Fetch robots.txt and test whether specific URLs are allowed or blocked for any user-agent, including wildcard and path rules. Use it to debug crawler access and verify disallow rules.
Toolresultaten worden in het Engels gegenereerd.
Gerelateerde pagina's
Serverprobleem? Gehakte site? Trage website?
Kritieke incidenten binnen 30 minuten beantwoord — 24 uur per dag, 7 dagen per week.