Log In Sign Up

brickfever.nl
robots.txt

Robots Exclusion Standard data for brickfever.nl

Resource Scan

Scan Details

Site Domain brickfever.nl
Base Domain brickfever.nl
Scan Status Ok
Last Scan2024-06-07T20:23:54+00:00
Next Scan 2024-07-07T20:23:54+00:00

Last Scan

Scanned2024-06-07T20:23:54+00:00
URL https://brickfever.nl/robots.txt
Domain IPs 104.26.12.136, 104.26.13.136, 172.67.72.204, 2606:4700:20::681a:c88, 2606:4700:20::681a:d88, 2606:4700:20::ac43:48cc
Response IP 104.26.13.136
Found Yes
Hash 577467d669e4f1a90e7fc00bd84723c118bfc5a19de7b481b7995eb65bd100ff
SimHash df15d4203db0

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-json/
Disallow /wp-includes/
Allow /wp-includes/css/
Allow /wp-includes/js/
Allow /wp-includes/images/
Disallow /trackback/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /*blackhole
Disallow /?blackhole
Disallow /404$
Disallow /?s=*
Disallow /search/*

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler
semrushbot
semrushbot-sa
semrushbot
blexbot
mauibot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://brickfever.nl/sitemap.xml
Back to top

Comments

  • Global rules
  • -----------------
  • We're experimenting with blocking search results to prevent search result spam
  • Sitemap
  • -----------------
  • Ban bots that don't benefit us.
  • --------------------------------
Back to top