thecollector.com
robots.txt

Robots Exclusion Standard data for thecollector.com

Resource Scan

Scan Details

Site Domain thecollector.com
Base Domain thecollector.com
Scan Status Ok
Last Scan2024-11-13T21:28:27+00:00
Next Scan 2024-11-20T21:28:27+00:00

Last Scan

Scanned2024-11-13T21:28:27+00:00
URL https://thecollector.com/robots.txt
Redirect https://www.thecollector.com/robots.txt
Redirect Domain www.thecollector.com
Redirect Base thecollector.com
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.123, 76.76.21.61
Response IP 76.76.21.93
Found Yes
Hash 0e99c7d634238c3c2edfa26935d5c241460a55de77233b00cfae0fd3419978f7
SimHash e50d5c022733

Groups

*

Rule Path
Disallow /thecollector.vercel.app/

proximic

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

criteobot/0.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://www.thecollector.com/sitemap/

Warnings

  • `nofollow` is not a known field.
  • `noindex` is not a known field.