Log In Sign Up

autojunk.nl
robots.txt

Robots Exclusion Standard data for autojunk.nl

Resource Scan

Scan Details

Site Domain autojunk.nl
Base Domain autojunk.nl
Scan Status Ok
Last Scan2024-10-30T20:25:00+00:00
Next Scan 2024-11-06T20:25:00+00:00

Last Scan

Scanned2024-10-30T20:25:00+00:00
URL https://autojunk.nl/robots.txt
Domain IPs 172.66.40.140, 172.66.43.116, 2606:4700:3108::ac42:288c, 2606:4700:3108::ac42:2b74
Response IP 172.66.40.140
Found Yes
Hash 8fdefaf46ebd7094c1acd56ecdd2a8e09b22d1a68a8eb0521d1ccf68aeb29950
SimHash ae5e4c301ff4

Groups

*

Rule Path
Disallow /wp-json/

googlebot

Rule Path
Disallow /wp-content/plugins/*
Disallow /tag/*/feed
Disallow /random
Disallow /abcount2.php
Disallow /*?m=yes
Disallow /*?s=*
Disallow /wp-login.php*
Disallow /index.php*
Disallow /gallery*
Disallow /image-gallery*

*

Rule Path
Disallow /?username=

Other Records

Field Value
crawl-delay 1

mj12bot
yandeximages
nuclei
wikido
semrushbot
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.autoblog.nl/sitemap_index.xml
Back to top

Comments

  • Global rules for http://www.autoblog.nl and netwerk!
  • ------------------------------------
  • Sitemap
  • ------------------------------------
  • https://tickets.quanza.net/Ticket/Display.html?id=28420#txn-808974
  • Ban bots that don't benefit the site
  • ------------------------------------
Back to top