Log In Sign Up

transweaver.com
robots.txt

Robots Exclusion Standard data for transweaver.com

Resource Scan

Scan Details

Site Domain transweaver.com
Base Domain transweaver.com
Scan Status Ok
Last Scan2025-12-10T12:11:51+00:00
Next Scan 2025-12-17T12:11:51+00:00

Last Scan

Scanned2025-12-10T12:11:51+00:00
URL https://transweaver.com/robots.txt
Domain IPs 104.21.69.90, 172.67.206.193, 2606:4700:3032::6815:455a, 2606:4700:3037::ac43:cec1
Response IP 104.21.69.90
Found Yes
Hash d5359495554492de598908d9d4c702dfb9981e85f4c5ee0233020c98e638aad6
SimHash 40a8cc37b7ea

Groups

proximic

Rule Path
Disallow /

ttd-content

Rule Path
Disallow /

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexmobilebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Block the most aggressive bots
  • Limit Googlebot's activity instead of blocking
  • Slow down AhrefsBot, Bingbot, and Yandex bots
Back to top