naturalgasintel.com
robots.txt

Robots Exclusion Standard data for naturalgasintel.com

Resource Scan

Scan Details

Site Domain naturalgasintel.com
Base Domain naturalgasintel.com
Scan Status Ok
Last Scan2024-06-30T13:46:32+00:00
Next Scan 2024-07-30T13:46:32+00:00

Last Scan

Scanned2024-06-30T13:46:32+00:00
URL https://naturalgasintel.com/robots.txt
Domain IPs 18.116.123.254, 3.14.215.58, 3.140.222.184
Response IP 18.116.123.254
Found Yes
Hash 17015a24c2f1b4c9a4e467eac0d84f73316b3800f710a0072c9365f6095dfb11
SimHash 5814d944a943

Groups

*

Rule Path
Disallow /search/

chatgpt-user
ccbot
gptbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://naturalgasintel.com/sitemap.xml

Comments

  • Index sitemap
  • Disallow LLM/AI crawlers