agentwebsite.net
robots.txt

Robots Exclusion Standard data for agentwebsite.net

Resource Scan

Scan Details

Site Domain agentwebsite.net
Base Domain agentwebsite.net
Scan Status Ok
Last Scan2025-09-17T21:13:24+00:00
Next Scan 2025-10-17T21:13:24+00:00

Last Scan

Scanned2025-09-17T21:13:24+00:00
URL https://agentwebsite.net/robots.txt
Redirect https://www.agentwebsite.net/robots.txt
Redirect Domain www.agentwebsite.net
Redirect Base agentwebsite.net
Domain IPs 100::1, 64.77.77.215
Redirect IPs 64.77.77.215
Response IP 64.77.77.215
Found Yes
Hash f937a5856b10011d2f3eecaa7235509a72d780eb33937717647d9a6380d89a87
SimHash c93c519b2ad1

Groups

*

Rule Path
Disallow /_inc/
Disallow /images/snapshots/
Disallow /stats/
Disallow /inquiry_processor.htm
Disallow /success.htm
Disallow /design_inquiry_2.htm
Disallow /designs.htm
Disallow /terms.htm
Disallow /tou.htm
Disallow /park.htm*
Disallow /*domain%3D*

npbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

sbider

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

betabot

Rule Path
Disallow /

mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.accelobot.com)

Rule Path
Disallow /

nutchcvs/0.7.1

Rule Path
Disallow /

zeusbot

Rule Path
Disallow /

zeusbot/nutch-1.0-dev

Rule Path
Disallow /

spbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /