1729.com
robots.txt

Robots Exclusion Standard data for 1729.com

Resource Scan

Scan Details

Site Domain 1729.com
Base Domain 1729.com
Scan Status Ok
Last Scan2025-11-29T01:01:34+00:00
Next Scan 2025-12-29T01:01:34+00:00

Last Scan

Scanned2025-11-29T01:01:34+00:00
URL https://1729.com/robots.txt
Redirect https://thenetworkstate.com/robots.txt
Redirect Domain thenetworkstate.com
Redirect Base thenetworkstate.com
Domain IPs 104.21.16.180, 172.67.215.62, 2606:4700:3035::6815:10b4, 2606:4700:3035::ac43:d73e
Redirect IPs 104.26.0.45, 104.26.1.45, 172.67.75.157, 2606:4700:20::681a:12d, 2606:4700:20::681a:2d, 2606:4700:20::ac43:4b9d
Response IP 172.67.75.157
Found Yes
Hash 904b6d16f000f038f98c7f61d2947ae3104867295720001a2550a345aebcc35a
SimHash 5b08dd53f701

Groups

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

*

Rule Path
Disallow /api/
Disallow /_next/
Disallow /static/
Disallow /__nextjs/
Allow /*/

gptbot

Rule Path
Disallow /api/
Disallow /admin/

chatgpt-user

Rule Path
Disallow /api/
Disallow /admin/

ccbot

Rule Path
Disallow /api/
Disallow /admin/

anthropic-ai

Rule Path
Disallow /api/
Disallow /admin/

claude-web

Rule Path
Disallow /api/
Disallow /admin/

ahrefsbot

Rule Path
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

sitelockspider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://thenetworkstate.com/sitemap.xml

Comments

  • robots.txt for thenetworkstate.com
  • Last updated: 2025-08-27
  • Default rule for all bots
  • Sitemaps
  • Crawl-delay (in seconds) - be respectful to smaller sites
  • Disallow private/admin areas
  • Allow search engines to index public profiles
  • Specific bot rules - AI/LLM crawlers
  • SEO/Marketing bots we want to limit
  • Bad bots to block completely