shopindenver.com
robots.txt

Robots Exclusion Standard data for shopindenver.com

Resource Scan

Scan Details

Site Domain shopindenver.com
Base Domain shopindenver.com
Scan Status Ok
Last Scan2024-09-05T08:59:27+00:00
Next Scan 2024-10-05T08:59:27+00:00

Last Scan

Scanned2024-09-05T08:59:27+00:00
URL http://shopindenver.com/robots.txt
Redirect http://www.shopindenver.com/robots.txt
Redirect Domain www.shopindenver.com
Redirect Base shopindenver.com
Domain IPs 144.208.82.17
Redirect IPs 144.208.82.17
Response IP 144.208.82.17
Found Yes
Hash e56326b23a5112e1357b9bf23e9ee00a22647d4bd3998a83ca6a35b3bf29912a
SimHash 4b9ed5d200e0

Groups

msnbot

Rule Path
Disallow /movie/
Disallow /Movie/

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /movie/
Disallow /Movie/

Other Records

Field Value
crawl-delay 10

naverbot

Rule Path
Disallow /

ia_archiver

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

yandex

Rule Path
Disallow /

yanga

Rule Path
Disallow /

therarestparser

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

robozilla

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

whizbang

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

jadynave

Rule Path
Disallow /

discobot

Rule Path
Disallow /

netresearchserver

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

*

Rule Path
Disallow /Categories/
Disallow /categories/
Disallow /Weather/
Disallow /weather/
Disallow /Admin/
Disallow /admin/
Disallow /Merchant/
Disallow /merchant/
Disallow /Security/
Disallow /security/
Disallow /Search/
Disallow /search/
Disallow /Naughty/
Disallow /naughty/
Disallow /mail.asp
Disallow /Review/AddReview.aspx
Disallow /movie/
Disallow /Movie/
Disallow /redirect.asp
Disallow /Email.aspx
Disallow /Map.aspx