weekly.tf
robots.txt

Robots Exclusion Standard data for weekly.tf

Resource Scan

Scan Details

Site Domain weekly.tf
Base Domain weekly.tf
Scan Status Ok
Last Scan2025-03-03T23:24:23+00:00
Next Scan 2025-04-02T23:24:23+00:00

Last Scan

Scanned2025-03-03T23:24:23+00:00
URL https://www.weekly.tf/robots.txt
Domain IPs 104.21.34.41, 172.67.196.249, 2606:4700:3032::ac43:c4f9, 2606:4700:3034::6815:2229
Response IP 172.67.196.249
Found Yes
Hash 7fc3a7906227b4e717d4e7030a6df6db047c18120283cc7e1e384d41a4780a3f
SimHash 6215d840a010

Groups

amazonbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow /nogooglebot/

*

Rule Path
Disallow /login

adsbot-google

Rule Path
Disallow /login

nutch

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /login

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

Rule Path
Disallow /login

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /login

Other Records

Field Value
crawl-delay 10

ccbot
chatgpt-user
gptbot
google-extended
anthropic-ai
omgilibot
omgili
facebookbot
bytespider

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 30

Other Records

Field Value
sitemap https://www.weekly.tf/sitemap.xml

Warnings

  • 1 invalid line.