Log In Sign Up

autowiki.nl
robots.txt

Robots Exclusion Standard data for autowiki.nl

Resource Scan

Scan Details

Site Domain autowiki.nl
Base Domain autowiki.nl
Scan Status Ok
Last Scan2024-11-15T03:25:49+00:00
Next Scan 2024-11-22T03:25:49+00:00

Last Scan

Scanned2024-11-15T03:25:49+00:00
URL https://autowiki.nl/robots.txt
Domain IPs 104.21.27.179, 172.67.143.108, 2606:4700:3031::ac43:8f6c, 2606:4700:3037::6815:1bb3
Response IP 104.21.27.179
Found Yes
Hash 8fdefaf46ebd7094c1acd56ecdd2a8e09b22d1a68a8eb0521d1ccf68aeb29950
SimHash ae5e4c301ff4

Groups

*

Rule Path
Disallow /wp-json/

googlebot

Rule Path
Disallow /wp-content/plugins/*
Disallow /tag/*/feed
Disallow /random
Disallow /abcount2.php
Disallow /*?m=yes
Disallow /*?s=*
Disallow /wp-login.php*
Disallow /index.php*
Disallow /gallery*
Disallow /image-gallery*

*

Rule Path
Disallow /?username=

Other Records

Field Value
crawl-delay 1

mj12bot
yandeximages
nuclei
wikido
semrushbot
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.autoblog.nl/sitemap_index.xml
Back to top

Comments

  • Global rules for http://www.autoblog.nl and netwerk!
  • ------------------------------------
  • Sitemap
  • ------------------------------------
  • https://tickets.quanza.net/Ticket/Display.html?id=28420#txn-808974
  • Ban bots that don't benefit the site
  • ------------------------------------
Back to top