starfield.ai
robots.txt

Robots Exclusion Standard data for starfield.ai

Resource Scan

Scan Details

Site Domain starfield.ai
Base Domain starfield.ai
Scan Status Ok
Last Scan2024-05-12T23:33:42+00:00
Next Scan 2024-06-11T23:33:42+00:00

Last Scan

Scanned2024-05-12T23:33:42+00:00
URL https://starfield.ai/robots.txt
Domain IPs 209.133.192.154, 2604:4500:0:388::2
Response IP 209.133.192.154
Found Yes
Hash 8beeadcd066e5f74db7b436b539413ba5bf5bba8c77ae1a6538429098db42572
SimHash 46364d50c619

Groups

bytespider
baiduspider
yisouspider
petalbot
ahrefsbot
yandexbot
yandeximageresizer
seznambot
coccocbot-web
zoombot
seekrbot
blexbot
the knowledge ai
mojeekbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 1

Warnings

  • 1 invalid line.