shopti.pl
robots.txt

Robots Exclusion Standard data for shopti.pl

Resource Scan

Scan Details

Site Domain shopti.pl
Base Domain shopti.pl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-08T03:00:59+00:00
Next Scan 2025-12-07T03:00:59+00:00

Last Successful Scan

Scanned2022-10-14T03:27:03+00:00
URL https://shopti.pl/robots.txt
Redirect https://www.shopti.pl/robots.txt
Redirect Domain www.shopti.pl
Redirect Base shopti.pl
Response IP 172.67.221.65, 104.21.24.252
Found Yes
Hash c7c3dc1b572fbdf3a0ec015da1bdbec30dc56e3dc0d2ee43708734e4cfed4dd4
SimHash 0a18d955d271

Groups

*

Rule Path
Allow /*.css
Allow /*.js
Disallow /product/click-eshop/
Disallow /eshop-click/

spbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

msnbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 16

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 18

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

spiderling

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

bubing

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

kingkevinbot

Rule Path
Disallow /
Disallow /admin/
Disallow /auth/

Other Records

Field Value
sitemap https://www.shopti.pl/sitemap_index.xml