dvcustoms.nl
robots.txt
Robots Exclusion Standard data for dvcustoms.nl
Resource Scan
Scan Details
| Site Domain | dvcustoms.nl |
| Base Domain | dvcustoms.nl |
| Scan Status | Ok |
| Last Scan | 2026-01-21T01:03:33+00:00 |
| Next Scan | 2026-02-04T01:03:33+00:00 |
Last Scan
| Scanned | 2026-01-21T01:03:33+00:00 |
| URL | https://dvcustoms.nl/robots.txt |
| Domain IPs | 45.152.249.178 |
| Response IP | 45.152.249.178 |
| Found | Yes |
| Hash | 486a53bcbe0cf5f22454732ae217a8fb91c9032a8cbae784e257b461728fde93 |
| SimHash | 523a4f41f61a |
Groups
adsbot-google
ahrefsbot
aliyun
aliyunsecbot
amazonbot
anthropic-ai
applebot
baiduspider
barkrowler
blexbot
bytespider
ccbot
chatgpt-user
claudebot
crawlbot
dataforseobot
dotbot
duckduckgo
exabot
facebookexternalhit
facebookbot
facebot
feedfetcher-google
freshbot
googlebot-image
googleother
gptbot
grapeshot
ias-va
mediapartners-google
meta-externalagent
mj12bot
newsai
petalbot
pinterestbot
proximic
scrapy
sebot-wa
semrushbot
sentibot
seositecheckup
serankingbacklinksbot
sogou
telegrambot
tiktokspider
trendictionbot
twitterbot
verity
yandexbot
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /tmp/ |
| Disallow | /cache/ |
| Disallow | /wp-admin/ |
| Disallow | /wp-includes/ |
| Disallow | /wp-login.php |
| Disallow | /private/ |
| Disallow | /admin/ |
| Disallow | /includes/ |
| Disallow | /config/ |
| Allow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 300 |
*
| Rule | Path |
|---|---|
| Disallow | /cgi-bin/ |
| Disallow | /tmp/ |
| Disallow | /cache/ |
| Disallow | /wp-admin/ |
| Disallow | /wp-includes/ |
| Disallow | /wp-login.php |
| Disallow | /private/ |
| Disallow | /admin/ |
| Disallow | /includes/ |
| Disallow | /config/ |
| Allow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 60 |
Comments