Log In Sign Up

waltons.co.za
robots.txt

Robots Exclusion Standard data for waltons.co.za

Resource Scan

Scan Details

Site Domain waltons.co.za
Base Domain waltons.co.za
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-28T06:14:57+00:00
Next Scan 2024-12-27T06:14:57+00:00

Last Successful Scan

Scanned2024-08-06T21:57:03+00:00
URL https://waltons.co.za/robots.txt
Redirect https://www.waltons.co.za/robots.txt
Redirect Domain www.waltons.co.za
Redirect Base waltons.co.za
Domain IPs 160.79.116.33
Redirect IPs 108.157.254.15, 108.157.254.28, 108.157.254.79, 108.157.254.90, 2600:9000:2753:1600:3:ac0:740:93a1, 2600:9000:2753:2200:3:ac0:740:93a1, 2600:9000:2753:5800:3:ac0:740:93a1, 2600:9000:2753:6e00:3:ac0:740:93a1, 2600:9000:2753:7a00:3:ac0:740:93a1, 2600:9000:2753:8c00:3:ac0:740:93a1, 2600:9000:2753:b800:3:ac0:740:93a1, 2600:9000:2753:e400:3:ac0:740:93a1
Response IP 108.157.254.28
Found Yes
Hash b8734f693a847d7135200b15492080296a4a83e059d48b894a9d7b1b176bf585
SimHash 0a765d124699

Groups

*

Rule Path
Disallow /
Disallow /*?
Disallow /*?*
Disallow /products/browse/*?*
Allow /account/login
Allow /account/register

linkedinbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Allow /
Disallow *gclid%3D*
Disallow *gb%3D*
Disallow /*?
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*
Disallow /*?*

Other Records

Field Value
crawl-delay 10

blackboard safeassign

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Disallow *gclid%3D*
Disallow *gb%3D*
Disallow /*?
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*
Disallow /*?*

Other Records

Field Value
crawl-delay 10

googlebot-image

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

applebot

Rule Path
Allow /
Disallow /*?
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 20

bingbot

Rule Path
Allow /
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 40

facebot

Rule Path
Allow /
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 30

duckduckbot

Rule Path
Allow /
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 30

mediapartners-google

Rule Path
Allow /
Disallow /*?
Disallow /*?*
Disallow *gclid%3D*
Disallow *gb%3D*
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.waltons.co.za/sitemap.xml
Back to top