Log In Sign Up

truthout.com
robots.txt

Robots Exclusion Standard data for truthout.com

Resource Scan

Scan Details

Site Domain truthout.com
Base Domain truthout.com
Scan Status Ok
Last Scan2025-08-25T04:46:00+00:00
Next Scan 2025-09-24T04:46:00+00:00

Last Scan

Scanned2025-08-25T04:46:00+00:00
URL https://truthout.com/robots.txt
Redirect https://truthout.org/robots.txt
Redirect Domain truthout.org
Redirect Base truthout.org
Domain IPs 104.21.18.231, 172.67.183.224, 2606:4700:3030::6815:12e7, 2606:4700:3034::ac43:b7e0
Redirect IPs 172.66.135.25, 172.66.139.249, 2606:4700:10::ac42:8719, 2606:4700:10::ac42:8bf9
Response IP 172.66.139.249
Found Yes
Hash 8600bdc2f58f7291444635a7cf02cdeb316515e3110463417183ef7683e0d0a3
SimHash 41a4cf40d4c0

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 0

googlebot-news

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 0
Back to top

Other Records

Field Value
sitemap https://truthout.org/sitemap_index.xml
sitemap https://truthout.org/sitemap_index.xml
sitemap https://truthout.org/news-sitemap.xml
sitemap https://truthout.org/news-sitemap.xml
Back to top

Comments

  • Default crawl delay for unknown bots
  • https://developers.cloudflare.com/fundamentals/get-started/reference/cdn-cgi-endpoint/
  • Googlebot crawl delay unset
  • Googlebot-news crawl delay unset
  • Bingbot crawl delay unset
Back to top