Log In Sign Up

truth-out.org
robots.txt

Robots Exclusion Standard data for truth-out.org

Resource Scan

Scan Details

Site Domain truth-out.org
Base Domain truth-out.org
Scan Status Ok
Last Scan2025-09-19T07:04:16+00:00
Next Scan 2025-10-19T07:04:16+00:00

Last Scan

Scanned2025-09-19T07:04:16+00:00
URL https://truth-out.org/robots.txt
Redirect https://truthout.org/robots.txt
Redirect Domain truthout.org
Redirect Base truthout.org
Domain IPs 104.21.89.88, 172.67.157.99, 2606:4700:3030::6815:5958, 2606:4700:3034::ac43:9d63
Redirect IPs 172.66.135.25, 172.66.139.249, 2606:4700:10::ac42:8719, 2606:4700:10::ac42:8bf9
Response IP 172.66.135.25
Found Yes
Hash 8600bdc2f58f7291444635a7cf02cdeb316515e3110463417183ef7683e0d0a3
SimHash 41a4cf40d4c0

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 0

googlebot-news

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Disallow /wp-admin/
Disallow /cdn-cgi/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 0
Back to top

Other Records

Field Value
sitemap https://truthout.org/sitemap_index.xml
sitemap https://truthout.org/sitemap_index.xml
sitemap https://truthout.org/news-sitemap.xml
sitemap https://truthout.org/news-sitemap.xml
Back to top

Comments

  • Default crawl delay for unknown bots
  • https://developers.cloudflare.com/fundamentals/get-started/reference/cdn-cgi-endpoint/
  • Googlebot crawl delay unset
  • Googlebot-news crawl delay unset
  • Bingbot crawl delay unset
Back to top