Log In Sign Up

digitalpulse.be
robots.txt

Robots Exclusion Standard data for digitalpulse.be

Resource Scan

Scan Details

Site Domain digitalpulse.be
Base Domain digitalpulse.be
Scan Status Ok
Last Scan2025-09-21T05:12:04+00:00
Next Scan 2025-10-05T05:12:04+00:00

Last Scan

Scanned2025-09-21T05:12:04+00:00
URL https://digitalpulse.be/robots.txt
Domain IPs 2a04:3544:1000:1510:3cc8:64ff:fefa:5871, 94.237.125.171
Response IP 94.237.125.171
Found Yes
Hash 9b02734bd24bf8289d101e1fea9d5f79dbb3ca76b78ea5202cb210b5bfc91a25
SimHash e1181922fdb0

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/

Other Records

Field Value
crawl-delay 20

amazonbot
semrushbot
ahrefsbot
barkrowler
blexbot
bw/1.1
bytespider
censysinspect
dalvik/2.1.0
dataforseobot
dataprovider
dotbot
expanse
foregenix
imagesiftbot
internet-measurement
ioncrawl
java
mj12bot
mozlila
orbbot
petalbot
python-requests
scrapy
wp_is_mobile
awariobot
zoominfobot
yandexbot
seznambot
coccocbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://digitalpulse.be/nl/sitemaps-1-sitemap.xml
Back to top

Comments

  • robots.txt for https://digitalpulse.be/
  • live - don't allow web crawlers to index cpresources/ or vendor/
  • Disallow ChatGPT bot, as there's no benefit to allowing it to index your site
  • Disallow Google Bard and Vertex AI bots, as there's no benefit to allowing it to index your site
  • Disallow Perplexity bot, as there's no benefit to allowing it to index your site
Back to top