Log In Sign Up

samayapost.com
robots.txt

Robots Exclusion Standard data for samayapost.com

Resource Scan

Scan Details

Site Domain samayapost.com
Base Domain samayapost.com
Scan Status Ok
Last Scan2025-12-20T15:51:14+00:00
Next Scan 2026-01-19T15:51:14+00:00

Last Scan

Scanned2025-12-20T15:51:14+00:00
URL https://samayapost.com/robots.txt
Domain IPs 104.21.96.106, 172.67.176.220, 2606:4700:3032::ac43:b0dc, 2606:4700:3037::6815:606a
Response IP 172.67.176.220
Found Yes
Hash 6e34e52392572da25d4a8ffd1cc0d98e8e9dc5fd49f2fc56e77ac8d564646edb
SimHash c23ece4223ba

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /xmlrpc.php
Allow /wp-admin/admin-ajax.php

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

python-requests

Rule Path
Disallow /

python

Rule Path
Disallow /

curl

Rule Path
Disallow /
Back to top

Comments

  • Slow down some aggressive bots (polite ones will respect this)
  • Block shady scrapers that do respect robots.txt (some do)
Back to top