Log In Sign Up

nicolasexton.co.uk
robots.txt

Robots Exclusion Standard data for nicolasexton.co.uk

Resource Scan

Scan Details

Site Domain nicolasexton.co.uk
Base Domain nicolasexton.co.uk
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-10-04T12:25:05+00:00
Next Scan 2025-11-03T12:25:05+00:00

Last Successful Scan

Scanned2025-08-13T05:36:16+00:00
URL https://nicolasexton.co.uk/robots.txt
Redirect https://www.nicolasexton.co.uk/robots.txt
Redirect Domain www.nicolasexton.co.uk
Redirect Base nicolasexton.co.uk
Domain IPs 217.160.147.12
Redirect IPs 217.160.147.12
Response IP 217.160.147.12
Found Yes
Hash 3c333411e0cd0099364d358804cf8cfbfc2a4ef77a8e4161d2243022e91e33e9
SimHash 515b9d01c550

Groups

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

perl lwp

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

turnitin robot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

worldwebheritage.org

Rule Path
Disallow /

worldwebheritage.org/1.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60
Back to top

Comments

  • Disallow following crawlers (might also be blocked by fail2ban or similar)
  • Block MJ12bot as it is just noise
  • Block Ahrefs
  • Block Sogou
  • Block SEOkicks
  • Block BlexBot
  • Block SISTRIX
  • Block Uptime robot
  • Block Ezooms Robot
  • Block Perl LWP
  • Block BlexBot
  • Block netEstate NE Crawler (+http://www.website-datenbank.de/)
  • Block WiseGuys Robot
  • Block Turnitin Robot
  • Block Heritrix
  • Block pricepi
  • Block other bots (though remember they might not repect robots.txt)
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
Back to top

Warnings

  • 4 invalid lines.
Back to top