Log In Sign Up

palsite.com
robots.txt

Robots Exclusion Standard data for palsite.com

Resource Scan

Scan Details

Site Domain palsite.com
Base Domain palsite.com
Scan Status Ok
Last Scan2026-01-04T02:17:27+00:00
Next Scan 2026-01-11T02:17:27+00:00

Last Scan

Scanned2026-01-04T02:17:27+00:00
URL https://palsite.com/robots.txt
Redirect https://www.palsite.com/robots.txt
Redirect Domain www.palsite.com
Redirect Base palsite.com
Domain IPs 212.124.193.11
Redirect IPs 104.21.9.106, 172.67.159.191, 2606:4700:3033::6815:96a, 2606:4700:3035::ac43:9fbf
Response IP 172.67.159.191
Found Yes
Hash 01d1e5094c5bb6a4dcc8c331cc1a572f851dc25cfcecaeea0f20095b1a786bfa
SimHash e686ef848dd8

Groups

*

Rule Path
Disallow /chat_page.py?*ts*

Other Records

Field Value
crawl-delay 86400

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top

Comments

  • Block all robots from old style chat_page requests
  • e.g. chat_page.py?r=1&cp=U-Matic&ts=992639580&pl=1;
  • User-agent: Baiduspider just this one is causing problems, but may as well block all
  • Attempt to slow down crawls to one per day
Back to top