lancia.co.za
robots.txt

Robots Exclusion Standard data for lancia.co.za

Resource Scan

Scan Details

Site Domain lancia.co.za
Base Domain lancia.co.za
Scan Status Ok
Last Scan2024-09-17T00:24:21+00:00
Next Scan 2024-10-17T00:24:21+00:00

Last Scan

Scanned2024-09-17T00:24:21+00:00
URL https://lancia.co.za/robots.txt
Domain IPs 45.56.88.234
Response IP 45.56.88.234
Found Yes
Hash 66361cee1489496b4f5f1f255a136ec72ca2269372b592d44b5e9d62e3022a86
SimHash 2640d9fae493

Groups

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

gptbot

Rule Path
Disallow /article/

petalbot

Rule Path
Disallow /article

Other Records

Field Value
crawl-delay 500

ninjabot

Rule Path
Disallow /article/

dotbot

Rule Path
Disallow /article/

semrushbot

Rule Path
Disallow /article/

dotbot

Rule Path
Disallow /article/

Other Records

Field Value
crawl-delay 500

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 500

msnbot-news

Rule Path
Disallow /article/

Other Records

Field Value
crawl-delay 500

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 500

msnbot-products

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 500

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 500

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 500

serpastatbot

Rule Path
Disallow /article/

Other Records

Field Value
crawl-delay 500

claudebot

Rule Path
Disallow /article/

Other Records

Field Value
crawl-delay 500

*

Rule Path
Disallow /phppgadmin/
Disallow /phppgadmin5/
Disallow /eper/
Disallow /admin/
Disallow /contact/.svn/
Disallow /contact/stats/
Disallow /history/.svn/
Disallow /iframes/.svn/
Disallow /lancia_reviews/
Disallow /model/.svn/
Disallow /offer/.svn/
Disallow /popups/
Disallow /restricted/
Disallow /return/.svn/
Disallow /search/.svn/
Disallow /service/.svn/
Disallow /stories/.svn/
Disallow /vcs/
Disallow /wpad/
Disallow /.svn/