Log In Sign Up

smokeshops.com
robots.txt

Robots Exclusion Standard data for smokeshops.com

Resource Scan

Scan Details

Site Domain smokeshops.com
Base Domain smokeshops.com
Scan Status Ok
Last Scan2024-09-27T01:51:59+00:00
Next Scan 2024-10-04T01:51:59+00:00

Last Scan

Scanned2024-09-27T01:51:59+00:00
URL https://smokeshops.com/robots.txt
Domain IPs 50.28.79.37
Response IP 50.28.79.37
Found Yes
Hash 601a763f0cbc9c71df81ea128a6c633bfb29a4815d50dc52697807f225b00c06
SimHash 123e43da8b31

Groups

*

Rule Path
Disallow

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value Comment
crawl-delay 5 1 page per 5 seconds

crazywebcrawler-spider

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bixolabs

Rule Path
Disallow /

discobot

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top

Comments

  • PARTIAL access (Spiders)
Back to top

Warnings

  • 2 invalid lines.
Back to top