verzekeringvergelijk.nl
robots.txt
Robots Exclusion Standard data for verzekeringvergelijk.nl
Resource Scan
Scan Details
| Site Domain | verzekeringvergelijk.nl |
| Base Domain | verzekeringvergelijk.nl |
| Scan Status | Ok |
| Last Scan | 2024-09-20T02:38:45+00:00 |
| Next Scan | 2024-10-20T02:38:45+00:00 |
Last Scan
| Scanned | 2024-09-20T02:38:45+00:00 |
| URL | https://verzekeringvergelijk.nl/robots.txt |
| Domain IPs | 149.210.224.180, 2a01:7c8:aaba:177::1 |
| Response IP | 149.210.224.180 |
| Found | Yes |
| Hash | fa292bf519f47fa4df7338bd64c3686fb5911b3ac5253697a74ce8594b3a9c73 |
| SimHash | f29ce54488b0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /?s= |
| Disallow | /page/*/?s= |
| Disallow | /search/ |
| Disallow | /wp-json/ |
| Disallow | /?rest_route= |
ccbot
gptbot
chatgpt-user
google-extended
amazonbot
facebookbot
anthropic-ai
claude-web
cohere-ai
omgilibot
omgili
perplexitybot
youbot
bytespider
diffbot
applebot-extended
nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler
semrushbot
yandex
rogerbot
ezooms
webcrawler
bubing
proximic
trendictionbot
vagabondo
bilbo
linkdexbot
exabot
slurp
niki-bot
compspybot
siteauditbot
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ct
semrushbot-bm
splitsignalbot
semrushbot-coub
beetlebot
mail.ru_bot
istellabot
plukkie
melodibot
spbot
queryseekerspider
buddhabot
imrbot
acoonbot
yacybot
gufybot
nesotebot
sogou web spider
seokicks-robot
solomonobot
wbsearchbot
noyabot
flappybot
siteexplorer
crawler4j
nutch
berlin-fu-cow
python-requests*
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.verzekeringvergelijk.nl/sitemap_index.xml |
Warnings
- 2 invalid lines.