commodity.com
robots.txt

Robots Exclusion Standard data for commodity.com

Resource Scan

Scan Details

Site Domain commodity.com
Base Domain commodity.com
Scan Status Ok
Last Scan2024-11-08T16:51:53+00:00
Next Scan 2024-11-15T16:51:53+00:00

Last Scan

Scanned2024-11-08T16:51:53+00:00
URL https://commodity.com/robots.txt
Domain IPs 104.26.14.131, 104.26.15.131, 172.67.68.192, 2606:4700:20::681a:e83, 2606:4700:20::681a:f83, 2606:4700:20::ac43:44c0
Response IP 104.26.14.131
Found Yes
Hash be6a1e7cd6907db46bf3f31206a9bafc2008d3b2de33da50c16c745745392b6f
SimHash e944d800eb12

Groups

*

Rule Path
Disallow /go/

gptbot

Rule Path
Disallow /

firecrawlagent

Rule Path
Disallow /