Log In Sign Up

aviator.nyc
robots.txt

Robots Exclusion Standard data for aviator.nyc

Resource Scan

Scan Details

Site Domain aviator.nyc
Base Domain aviator.nyc
Scan Status Ok
Last Scan2025-09-23T08:46:10+00:00
Next Scan 2025-09-30T08:46:10+00:00

Last Scan

Scanned2025-09-23T08:46:10+00:00
URL https://aviator.nyc/robots.txt
Domain IPs 35.215.115.191
Response IP 35.215.115.191
Found Yes
Hash 70e5f382396932bdf40eb3f6c52f969f5108ee30f2f3c9a870a85e342ed323e2
SimHash 00441050c70f

Groups

*

Rule Path
Allow /
Allow /llms.txt
Allow /llms-full.txt
Allow /docs/
Allow /*.md$

Other Records

Field Value
crawl-delay 3

gptbot

Rule Path
Allow /llms.txt

claudebot

Rule Path
Allow /llms.txt

anthropicai

Rule Path
Allow /llms.txt
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /*?utm_*
Disallow /*?fbclid
Back to top

Other Records

Field Value
sitemap https://www.aviator.nyc/sitemap.xml
Back to top

Comments

  • AI + LLM Optimization
  • LLM-specific agents (future-proofed)
  • Non-critical paths
  • Legal/data usage policy
  • By accessing or ingesting this site, agents agree to: https://www.aviator.nyc/terms-of-service/
Back to top