tjnews.remembering.ca
robots.txt

Robots Exclusion Standard data for tjnews.remembering.ca

Resource Scan

Scan Details

Site Domain tjnews.remembering.ca
Base Domain remembering.ca
Scan Status Ok
Last Scan2025-09-04T00:00:46+00:00
Next Scan 2025-10-04T00:00:46+00:00

Last Scan

Scanned2025-09-04T00:00:46+00:00
URL https://tjnews.remembering.ca/robots.txt
Domain IPs 34.210.198.174, 35.81.130.154, 35.81.61.120
Response IP 35.81.130.154
Found Yes
Hash 0bbfc27e82c743a4edddcf96603b8951549a433bb3185010d341d53161c3004b
SimHash ed212f30c3ca

Groups

*

Rule Path
Disallow /*?*search_type*

*

Rule Path
Disallow /search*

*

Rule Path
Disallow /*?*ap_search_*

*

Rule Path
Disallow /ajax/post_form/

*

Rule Path
Disallow /admin/

*

Rule Path
Disallow /*-admin/

*

Rule Path
Disallow /manage-*/

*

Rule Path
Disallow /create-*/

*

Rule Path
Disallow /edit-*/

*

Rule Path
Disallow /claim-*/

*

Rule Path
Disallow /init-tribute-store/*

*

Rule Path
Disallow /page-fragment/*

Comments

  • Stop the crawlers from hitting all the permutations of the search filters
  • Stop the crawlers from searching
  • Disallow bots from posting
  • Disallow bots from content update
  • Disallow bots from indexing header/footer fragments