chapelhill.homeip.net
robots.txt

Robots Exclusion Standard data for chapelhill.homeip.net

Resource Scan

Scan Details

Site Domain chapelhill.homeip.net
Base Domain chapelhill.homeip.net
Scan Status Ok
Last Scan2024-10-03T09:19:27+00:00
Next Scan 2024-10-10T09:19:27+00:00

Last Scan

Scanned2024-10-03T09:19:27+00:00
URL http://chapelhill.homeip.net/robots.txt
Domain IPs 124.171.218.124
Response IP 124.171.218.124
Found Yes
Hash bec101de52a85e7a0b6c09258f9ab1a01bb1bc59194d4aace6de2f500ab3fb83
SimHash 203c5c70ce91

Groups

exabot

Rule Path
Disallow /

stormcrawler

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

sbider

Rule Path
Disallow /

sdcresearchlabs-testbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

oozbot

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

kyluka

Rule Path
Disallow /

httrack

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

webalta

Rule Path
Disallow /

fastcrawler

Rule Path
Disallow /

tineye-bot

Rule Path
Disallow /

citeseerxbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

boitho.com-dc

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

*

Rule Path
Disallow /doc/
Disallow /tmp/
Disallow /w/ChapelHill/
Disallow /t/
Disallow /l/
Disallow /v/
Disallow /i/

Other Records

Field Value
sitemap http://www.chapelhill.homeip.net/sitemap.xml