Log In Sign Up

arbo.nu
robots.txt

Robots Exclusion Standard data for arbo.nu

Resource Scan

Scan Details

Site Domain arbo.nu
Base Domain arbo.nu
Scan Status Ok
Last Scan2024-10-26T13:43:40+00:00
Next Scan 2024-11-09T13:43:40+00:00

Last Scan

Scanned2024-10-26T13:43:40+00:00
URL https://arbo.nu/robots.txt
Domain IPs 2a02:2968::1c00:efff:fe00:e79, 62.84.245.221
Response IP 62.84.245.221
Found Yes
Hash 3bedeabcf6bef4c40bff87995a1cbf54ea0c33c7a392120b5feef26e2a5106be
SimHash be36f65b81f8

Groups

*

Rule Path
Disallow /

googlebot

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

adsbot-google

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

googlebot-image

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

googlebot-video

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

mediapartners-google

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

bingbot

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

duckduckbot

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

w3c-checklink

Rule Path
Allow /

morningscore bot

Rule Path
Allow /

pinterestbot

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

ia_archiver

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

archive.org_bot

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

facebookexternalhit

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

linkedinbot

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

whatsapp

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

twitterbot

Rule Path Comment
Allow / -
Disallow /wp-json/ -
Disallow /?rest_route= -
Disallow /wp-admin/ -
Disallow /wp-content/cache/ -
Disallow /wp-content/plugins/ -
Disallow /xmlrpc.php -
Disallow /wp-includes/ -
Disallow /*add-to-cart%3D* -
Disallow /feed/ -
Disallow */feed -
Disallow */feed$ -
Disallow /feed/$ -
Disallow /comments/feed -
Disallow /?feed= -
Disallow /wp-feed -
Disallow /wp-admin/ block access to admin section
Disallow /wp-login.php block access to admin section
Disallow /search/* block access to internal search result pages
Disallow /*?s=* block access to internal search result pages
Disallow /?p=* block access to pages for which permalinks fails
Disallow /%26p%3D* block access to pages for which permalinks fails
Disallow *%26preview%3D* block access to preview pages
Disallow /?lang=* block access to language translations
Disallow /tag/ block access to tag pages
Disallow /author/ block access to author pages
Disallow /404-error/ block access to 404 page
Disallow /cart/ -
Disallow /winkelmand/ -
Disallow /checkout/ -
Disallow /afrekenen/ -
Disallow /my-account/ -
Disallow /mijn-account/ -
Allow /wp-includes/css/ -
Allow /wp-includes/js/ -
Allow /wp-admin/admin-ajax.php -

ahrefsbot
applebot
amazonbot
amazon-kendra
babya discoverer
barkrowler
baiduspider
blekkobot
blexbot
builtwith
bytespider
cazoodlebot
ccbot
claudebot
datadog/synthetics
dataprovider
discobot
dotbot/1.0
exabot
ezooms robot
geedoproductsearch
gigabot
go-http-client
imagesiftbot
magpie-crawler
mj12bot
mojeekbot
msnbot
neevabot
netestate ne crawler
node/simplecrawler
nuclei
perl lwp
petalbot
qwantbot
riddler
semrushbot
seokicks
seokicks-robot
seranking seochecker
serpstatbot
seznambot
sistrix crawler
sitecheckerbotcrawler
sogou web spider
sogou inst spider
slurp
taboolabot
turnitin robot
uptimerobot/2.0
velenpublicwebcrawler
wikido
wiseguys robot
yandex
yisouspider
zoominfobot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://arbo.nu/sitemaps.xml
Back to top

Comments

  • Make the internet greener
  • Reduce your carbon footprint
  • Reduce data centre waste
  • Reduce the number of search engines and bots from all over the world
  • Only allow search engines that matters
  • Questions, want to be allowed? Contact me: info@ivanit.nl
  • Our default for all search engines is to not allow crawling:
  • Below are the crawlers that are allowed to crawl this site.
  • Below that list, you will find paths that are blocked, even for them,
  • and then paths within those blocked paths that are allowed.
  • Google Bot
  • Google adsbot
  • Google Image
  • Google Video
  • Google AdSense
  • Bing
  • Duck Duck Go search
  • W3C link checker
  • Morningscore bot
  • Pinterest bot
  • Internet Archive (archive.org) WayBack Machine
  • Internet Archive (archive.org) WayBack Machine
  • Facebook
  • Linkedin
  • WhatsApp
  • Twitter
  • Block webcrawlers individually because stuborn or not listening to first directive or bad bots or not helpful for me or my clients
  • Sitemap
Back to top

Warnings

  • 2 invalid lines.
Back to top