Log In Sign Up

datalyzer.com
robots.txt

Robots Exclusion Standard data for datalyzer.com

Resource Scan

Scan Details

Site Domain datalyzer.com
Base Domain datalyzer.com
Scan Status Ok
Last Scan2025-12-11T21:34:33+00:00
Next Scan 2026-01-10T21:34:33+00:00

Last Scan

Scanned2025-12-11T21:34:33+00:00
URL https://datalyzer.com/robots.txt
Domain IPs 45.82.188.62
Response IP 45.82.188.62
Found Yes
Hash 281ef69df1710f5303d207dc2945bfa4530fa5b7f52fc96977e8a405fdd4351b
SimHash 78184b4184e3

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Allow /*.pdf$

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

ccbot

Rule Path
Allow /

applebot

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

amazonbot

Rule Path
Allow /

amazonbot-image

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://datalyzer.com/sitemap_index.xml
sitemap https://datalyzer.com/page-sitemap.xml
sitemap https://datalyzer.com/sfwd-courses-sitemap.xml
sitemap https://datalyzer.com/sfwd-lessons-sitemap.xml
Back to top

Comments

  • robots.txt voor GEO (generative engine optimization) + SEO
  • Laat crawlers je content en PDF's gebruiken in AI-overzichten en antwoorden
  • basis
  • geef populaire (AI) crawlers expliciet toegang
  • optioneel: een milde crawl-delay voor niet-Google bots
  • (Google negeert dit; veel andere bots respecteren het wel)
  • sitemaps
Back to top