naturalgasintel.com
robots.txt

Robots Exclusion Standard data for naturalgasintel.com

Resource Scan

Scan Details

Site Domain naturalgasintel.com
Base Domain naturalgasintel.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-20T13:47:51+00:00
Next Scan 2024-11-19T13:47:51+00:00

Last Successful Scan

Scanned2024-08-29T13:46:46+00:00
URL https://naturalgasintel.com/robots.txt
Domain IPs 3.134.22.182, 3.139.154.220, 3.16.71.73
Response IP 3.139.154.220
Found Yes
Hash 17015a24c2f1b4c9a4e467eac0d84f73316b3800f710a0072c9365f6095dfb11
SimHash 5814d944a943

Groups

*

Rule Path
Disallow /search/

chatgpt-user
ccbot
gptbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://naturalgasintel.com/sitemap.xml

Comments

  • Index sitemap
  • Disallow LLM/AI crawlers