Log In Sign Up

wcipeg.com
robots.txt

Robots Exclusion Standard data for wcipeg.com

Resource Scan

Scan Details

Site Domain wcipeg.com
Base Domain wcipeg.com
Scan Status Ok
Last Scan2025-04-06T20:53:32+00:00
Next Scan 2025-05-06T20:53:32+00:00

Last Scan

Scanned2025-04-06T20:53:32+00:00
URL https://wcipeg.com/robots.txt
Domain IPs 104.21.55.101, 172.67.147.117, 2606:4700:3030::ac43:9375, 2606:4700:3032::6815:3765
Response IP 104.21.55.101
Found Yes
Hash 1f6828311bbd1d409c07773c034ed26f5bae25ea67300214bcbe85fc81478f85
SimHash a50b6c00cc73

Groups

*

Rule Path
Disallow /submissions/
Back to top

Comments

  • Crawling submissions pages is detrimental to the quality of search results,
  • since people probably do not want to see them cluttering up the results page
  • (e.g., /submissions/bbi5291,ccc03s4, /submissions/bbi5291,ccc03s5, ...)
  • It's nothing to me if you ignore this advice, though.
Back to top