gtanet.com
robots.txt

Robots Exclusion Standard data for gtanet.com

Resource Scan

Scan Details

Site Domain gtanet.com
Base Domain gtanet.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-11T00:49:14+00:00
Next Scan 2024-10-11T00:49:14+00:00

Last Successful Scan

Scanned2024-08-12T16:05:24+00:00
URL https://gtanet.com/robots.txt
Domain IPs 104.21.1.122, 172.67.129.54, 2606:4700:3033::6815:17a, 2606:4700:3036::ac43:8136
Response IP 104.21.1.122
Found Yes
Hash eaa94bc8af7329c7d3bab7c2b5084c542a2cd7d64132d58661ec8ac2168d6fbf
SimHash 223c1e504171

Groups

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 5

Other Records

Field Value
sitemap https://gtanet.com/sitemap_index.xml

Comments

  • ByeGPT
  • Adios Google Bard/Gemini AI as well
  • Does not impact normal Google crawlers
  • See: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
  • Weird SEO bots which just hog everything
  • Bots who respect this please be nicer
  • Crawl every 5 seconds and not any faster