Log In Sign Up

caps.be
robots.txt

Robots Exclusion Standard data for caps.be

Resource Scan

Scan Details

Site Domain caps.be
Base Domain caps.be
Scan Status Ok
Last Scan2025-08-29T20:16:23+00:00
Next Scan 2025-09-12T20:16:23+00:00

Last Scan

Scanned2025-08-29T20:16:23+00:00
URL https://caps.be/robots.txt
Domain IPs 2a04:3544:1000:1510:3cc8:64ff:fefa:5871, 94.237.125.171
Response IP 94.237.125.171
Found Yes
Hash 0ff1f8f88f96fdc60f81fc43c767304b2944cf4ca01b9e75bf44da69374685d6
SimHash 415c19627f80

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/

Other Records

Field Value
crawl-delay 20

amazonbot
semrushbot
ahrefsbot
barkrowler
blexbot
bw/1.1
bytespider
censysinspect
dalvik/2.1.0
dataforseobot
dataprovider
dotbot
expanse
foregenix
gptbot
imagesiftbot
internet-measurement
ioncrawl
java
mj12bot
mozlila
orbbot
petalbot
python-requests
scrapy
wp_is_mobile
awariobot
claudebot
zoominfobot
yandexbot
seznambot
coccocbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://caps.be/nl/sitemaps-1-sitemap.xml
sitemap https://caps.be/fr/sitemaps-1-sitemap.xml
Back to top

Comments

  • robots.txt for https://caps.be/
  • live - don't allow web crawlers to index cpresources/ or vendor/
  • Disallow ChatGPT bot, as there's no benefit to allowing it to index your site
Back to top