distances.co.za
robots.txt

Robots Exclusion Standard data for distances.co.za

Resource Scan

Scan Details

Site Domain distances.co.za
Base Domain distances.co.za
Scan Status Ok
Last Scan2024-05-17T06:59:19+00:00
Next Scan 2024-05-24T06:59:19+00:00

Last Scan

Scanned2024-05-17T06:59:19+00:00
URL https://distances.co.za/robots.txt
Domain IPs 104.21.55.144, 172.67.149.34, 2606:4700:3033::ac43:9522, 2606:4700:3037::6815:3790
Response IP 172.67.149.34
Found Yes
Hash 3d5d16b2ff65abcab6a4f81e6b3a7e23d0eebbdbf1cc5738a66fd0342aae02ad
SimHash e35ed440ee99

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /