robots.txt Rules Parser
Fetch robots.txt and test whether specific URLs are allowed or blocked for any user-agent, including wildcard and path rules. Use it to debug crawler access and verify disallow rules.
This tool runs entirely in your browser. Your inputs and files are processed locally and are never sent to or stored on Kinetikatech servers.
Related pages
Server problem? Hacked site? Slow website?
Critical issues answered within 30 minutes, 24 hours a day, 7 days a week.