Log In Sign Up

bdweb.fr
robots.txt

Robots Exclusion Standard data for bdweb.fr

Resource Scan

Scan Details

Site Domain bdweb.fr
Base Domain bdweb.fr
Scan Status Ok
Last Scan2026-02-20T19:20:21+00:00
Next Scan 2026-03-22T19:20:21+00:00

Last Scan

Scanned2026-02-20T19:20:21+00:00
URL https://bdweb.fr/robots.txt
Redirect https://www.bdweb.fr/robots.txt
Redirect Domain www.bdweb.fr
Redirect Base bdweb.fr
Domain IPs 104.26.10.168, 104.26.11.168, 172.67.74.50, 2606:4700:20::681a:aa8, 2606:4700:20::681a:ba8, 2606:4700:20::ac43:4a32
Redirect IPs 104.26.10.168, 104.26.11.168, 172.67.74.50, 2606:4700:20::681a:aa8, 2606:4700:20::681a:ba8, 2606:4700:20::ac43:4a32
Response IP 104.26.11.168
Found Yes
Hash 5b48c002865e5071fa578faeb313ecd1a20a054de3aa4689b46569122b83cdda
SimHash 515091126eb2

Groups

*

Rule Path
Disallow /*.asp
Disallow /*.asp/
Disallow /*/zoeken
Disallow /*/zoeken?*
Disallow /*/recherche
Disallow /*/recherche?*

Other Records

Field Value
crawl-delay 2

chatgpt-user

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-web

Rule Path
Allow /

applebot-extended

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.bdweb.fr/sitemap_index_bdwebFR.xml
Back to top

Comments

  • Set crawl delay to 10 seconds for all crawlers
  • === Op vraag van cnip (9 feb 2026) ===
  • Blokkeer oude .asp URL-paden (crawl trap)
  • Blokkeer interne zoekresultaten
  • --- AI ZOEK-BOTS (sturen verkeer naar de site) ---
  • OpenAI / ChatGPT
  • Google Gemini / AI Overviews
  • Perplexity
  • Anthropic / Claude
  • Apple Intelligence / Siri
Back to top