ptmd.nl
robots.txt

Robots Exclusion Standard data for ptmd.nl

Resource Scan

Scan Details

Site Domain ptmd.nl
Base Domain ptmd.nl
Scan Status Ok
Last Scan2024-09-17T14:11:47+00:00
Next Scan 2024-10-17T14:11:47+00:00

Last Scan

Scanned2024-09-17T14:11:47+00:00
URL https://ptmd.nl/robots.txt
Redirect https://www.ptmd.nl/robots.txt
Redirect Domain www.ptmd.nl
Redirect Base ptmd.nl
Domain IPs 194.61.65.247
Redirect IPs 194.61.65.247
Response IP 194.61.65.247
Found Yes
Hash f4fb14f09a3b47b1744d31ab955f31a3468751a4c6455ccd1ce88a1a101c653a
SimHash 2cb39ac446f5

Groups

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
adidxbot
applebot
applenewsbot
bingbot
bingpreview
bublupbot
ccbot
cliqzbot
daumoa
dazoobot
deusu
duckduckbot
duckduckgo-favicons-bot
euripbot
exploratodo
facebot
findxbot
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
istellabot
lycos
mediapartners-google
mojeekbot
msnbot
msnbot-media
orangebot
pinterest
plukkie
qwantify
seznambot
slurp
teoma
twitterbot
wotbox
yacybot
yeti
yioopbot

Rule Path
Disallow /store
Disallow /stores/
Disallow /winkels/
Disallow /shops/
Disallow /magasins/
Disallow /tiendas/
Disallow /negozi/
Disallow /sklepy/
Disallow /stores-category
Disallow /stores-category/

*

Rule Path
Disallow /

Comments

  • WHITELIST
  • google.com landing page quality checks
  • google.com app resource fetcher
  • bing ads bot
  • apple.com search engine
  • bing.com international search engine
  • bublup.com suggestion/search engine
  • commoncrawl.org open repository of web crawl data
  • cliqz.com german in-product search engine
  • daum.net korean search engine
  • dazoo.fr french search engine
  • deusu.de german search engine
  • duckduckgo.com international privacy search engine
  • eurip.com european search engine
  • exploratodo.com latin search engine
  • facebook.com social network
  • findx.com european search engine
  • google.com international search engine
  • istella.it italian search engine
  • lycos.com & hotbot.com international search engine
  • google.com adsense bot
  • mojeek.com search engine
  • bing.com international search engine
  • orange.com international search engine
  • pinterest.com social networtk
  • botje.nl dutch search engine
  • qwant.com french search engine
  • seznam.cz czech search engine
  • yahoo.com international search engine
  • ask.com international search engine
  • twitter.com bot
  • wotbox.com international search engine
  • yacy.net p2p search software
  • search.naver.com south korean search engine
  • yioop.com international search engine
  • crawling rule(s) for above bots
  • BLACKLIST
  • disallow all other bots