rain.co.za
robots.txt

Robots Exclusion Standard data for rain.co.za

Resource Scan

Scan Details

Site Domain rain.co.za
Base Domain rain.co.za
Scan Status Ok
Last Scan2024-09-22T18:31:50+00:00
Next Scan 2024-10-06T18:31:50+00:00

Last Scan

Scanned2024-09-22T18:31:50+00:00
URL https://rain.co.za/robots.txt
Redirect https://www.rain.co.za/robots.txt
Redirect Domain www.rain.co.za
Redirect Base rain.co.za
Domain IPs 41.73.51.11
Redirect IPs 104.18.34.244, 172.64.153.12, 2606:4700:4400::6812:22f4, 2606:4700:4400::ac40:990c
Response IP 172.64.153.12
Found Yes
Hash fc9ae8feb3784ddf1457885296b2303f0409e140ca36a4d2ab89e0a3a142ec6b
SimHash 591d51c0d4d9

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

bingpreview

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

msnbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 60

facebookexternalhit

Rule Path
Allow /

applebot

Rule Path
Allow /

applewebkit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

yandex

Rule Path
Allow /

yandexbot

Rule Path
Allow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.rain.co.za/sitemap.xml

Warnings

  • 2 invalid lines.