majestic.com
robots.txt

Robots Exclusion Standard data for majestic.com

Resource Scan

Scan Details

Site Domain majestic.com
Base Domain majestic.com
Scan Status Ok
Last Scan2024-06-11T12:59:02+00:00
Next Scan 2024-07-11T12:59:02+00:00

Last Scan

Scanned2024-06-11T12:59:02+00:00
URL https://majestic.com/robots.txt
Domain IPs 185.17.199.7
Response IP 185.17.199.7
Found Yes
Hash 25473ba42a9c458a56400540553439f4eb930335cba5deeb5baaee195c801ec5
SimHash e508181067e1

Groups

*

Rule Path
Disallow /business/
Disallow /forbidwarn
Disallow /infectwarn

mj12bot

Rule Path
Disallow /forbidwarn
Disallow /infectwarn
Allow /business/

googlebot

Rule Path
Disallow /forbidwarn
Disallow /infectwarn
Allow /business/

bingbot

Rule Path
Disallow /forbidwarn
Disallow /infectwarn
Allow /business/

vegi bot
seostats
mauibot
techseo360

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

Comments

  • robots.txt Rev #9 20230620
  • banned bots.