Log In Sign Up

waltons.co.za
robots.txt

Robots Exclusion Standard data for waltons.co.za

Resource Scan

Scan Details

Site Domain waltons.co.za
Base Domain waltons.co.za
Scan Status Ok
Last Scan2024-06-07T21:56:58+00:00
Next Scan 2024-07-07T21:56:58+00:00

Last Scan

Scanned2024-06-07T21:56:58+00:00
URL https://waltons.co.za/robots.txt
Redirect https://www.waltons.co.za/robots.txt
Redirect Domain www.waltons.co.za
Redirect Base waltons.co.za
Domain IPs 160.79.116.33
Redirect IPs 108.157.254.15, 108.157.254.28, 108.157.254.79, 108.157.254.90, 2600:9000:2753:400:3:ac0:740:93a1, 2600:9000:2753:5800:3:ac0:740:93a1, 2600:9000:2753:7400:3:ac0:740:93a1, 2600:9000:2753:8800:3:ac0:740:93a1, 2600:9000:2753:b600:3:ac0:740:93a1, 2600:9000:2753:cc00:3:ac0:740:93a1, 2600:9000:2753:de00:3:ac0:740:93a1, 2600:9000:2753:e000:3:ac0:740:93a1
Response IP 108.157.254.15
Found Yes
Hash b8734f693a847d7135200b15492080296a4a83e059d48b894a9d7b1b176bf585
SimHash 0a765d124699

Groups

*

Rule Path
Disallow /
Disallow /*?
Disallow /*?*
Disallow /products/browse/*?*
Allow /account/login
Allow /account/register

linkedinbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Allow /
Disallow *gclid%3D*
Disallow *gb%3D*
Disallow /*?
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*
Disallow /*?*

Other Records

Field Value
crawl-delay 10

blackboard safeassign

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Disallow *gclid%3D*
Disallow *gb%3D*
Disallow /*?
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*
Disallow /*?*

Other Records

Field Value
crawl-delay 10

googlebot-image

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

applebot

Rule Path
Allow /
Disallow /*?
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 20

bingbot

Rule Path
Allow /
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 40

facebot

Rule Path
Allow /
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /profile/
Disallow /scripts/
Disallow /styles/
Disallow /images/
Disallow /fonts/
Disallow /ckeditor/
Disallow /backoffice/
Disallow */javascript
Disallow /javascript
Disallow /products/splash
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 30

duckduckbot

Rule Path
Allow /
Disallow /*?*
Disallow /error/
Disallow /platformhealthcheck/
Disallow /publish/
Disallow /checkout/
Disallow /cart/
Disallow /account/
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 30

mediapartners-google

Rule Path
Allow /
Disallow /*?
Disallow /*?*
Disallow *gclid%3D*
Disallow *gb%3D*
Disallow /products/browse/*?*
Disallow /products/browse?
Disallow /products/browse/?
Disallow /products/browse/?categoryids=*
Disallow /products/browse?categoryids=*
Disallow /products/browse?manufacturerids=*
Disallow /products/browse/?manufacturerids=*
Disallow /products/browse/?all=true&categoryids=*&labelsids=*
Disallow /products/browse?all=true&categoryids=*&labelsids=*
Disallow /products/browse/?categoryids=*&labelsids=*
Disallow /manufacturerids%3D*

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.waltons.co.za/sitemap.xml
Back to top