tusa.com
robots.txt

Robots Exclusion Standard data for tusa.com

Resource Scan

Scan Details

Site Domain tusa.com
Base Domain tusa.com
Scan Status Ok
Last Scan2024-06-13T17:00:22+00:00
Next Scan 2024-07-13T17:00:22+00:00

Last Scan

Scanned2024-06-13T17:00:22+00:00
URL https://tusa.com/robots.txt
Domain IPs 104.21.59.180, 172.67.182.9, 2606:4700:3032::6815:3bb4, 2606:4700:3036::ac43:b609
Response IP 172.67.182.9
Found Yes
Hash 3d4550996dd67c54fa725bdfe4890507cef052c671ac1cf78a0220250417d8f7
SimHash 8f4cd6788e13

Groups

*

Rule Path
Disallow /admin

ahrefsbot
ahrefssiteaudit
amazonbot
baiduspider
barkrowler
better uptime bot
blexbot
botify
coccoc
contentking
criteobot
cxensebot
detectify
dotbot
exabot
grapeshotcrawler
hetrixtools
magpie-crawler
mauibot
petalbot
pingdom
pinterestbot
prerender
qualys
rackspace
seekportbot
seekport crawler
semrushbot
sendgrid
site24x7
siteimprove.com
slackbot
sogou
statuscake
stripe
uptimebot.org
uptimerobot
yandexbot

Rule Path
Disallow /

Comments

  • original protocol: http://www.robotstxt.org/orig.html
  • https://en.wikipedia.org/wiki/Robots.txt
  • proposed protocol: https://www.rfc-editor.org/rfc/rfc9309.html
  • https://radar.cloudflare.com/traffic/verified-bots
  • https://github.com/monperrus/crawler-user-agents/tree/master