lekkeslaap.co.za
robots.txt

Robots Exclusion Standard data for lekkeslaap.co.za

Resource Scan

Scan Details

Site Domain lekkeslaap.co.za
Base Domain lekkeslaap.co.za
Scan Status Ok
Last Scan2024-11-18T16:33:27+00:00
Next Scan 2024-12-02T16:33:27+00:00

Last Scan

Scanned2024-11-18T16:33:27+00:00
URL https://lekkeslaap.co.za/robots.txt
Redirect https://www.lekkeslaap.co.za/robots.txt
Redirect Domain www.lekkeslaap.co.za
Redirect Base lekkeslaap.co.za
Domain IPs 197.189.255.50, 41.203.0.58
Redirect IPs 197.189.255.50, 41.203.0.58
Response IP 41.203.0.58
Found Yes
Hash bdb45148a53ac6d522413689675c4c546d1e27b0f9b4e1d6d5b40c32033c9de3
SimHash a1953840ccf5

Groups

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

*

Rule Path
Disallow /suppliers
Disallow /users
Disallow /besprekings
Disallow /bookings
Disallow /favourites
Disallow /gunstelinge
Disallow /scripts
Disallow /login/processor/
Disallow /uitteken
Disallow /teken-af
Disallow /profile
Disallow /profile/rewards
Disallow /social
Disallow /nuusbriewe
Disallow /cdn-cgi
Allow /cdn-cgi/image/

Other Records

Field Value
sitemap https://www.lekkeslaap.co.za/sitemap.xml?token=FCA8ED9F943243976907823A480D0C91A4CA427A75FE5C4C44DE1A577321616E

Comments

  • Some exceptions:
  • Explicitly allow Cloudflare image resize service

Warnings

  • `visit-time` is not a known field.