gwiazdor.pl
robots.txt

Robots Exclusion Standard data for gwiazdor.pl

Resource Scan

Scan Details

Site Domain gwiazdor.pl
Base Domain gwiazdor.pl
Scan Status Ok
Last Scan2024-06-30T12:31:50+00:00
Next Scan 2024-07-07T12:31:50+00:00

Last Scan

Scanned2024-06-30T12:31:50+00:00
URL https://gwiazdor.pl/robots.txt
Redirect https://www.gwiazdor.pl/robots.txt
Redirect Domain www.gwiazdor.pl
Redirect Base gwiazdor.pl
Domain IPs 104.21.20.10, 172.67.190.195, 2606:4700:3032::6815:140a, 2606:4700:3033::ac43:bec3
Redirect IPs 104.21.20.10, 172.67.190.195, 2606:4700:3032::6815:140a, 2606:4700:3033::ac43:bec3
Response IP 172.67.190.195
Found Yes
Hash 031c43a8a35ad3a77f86cb1a6c1f8f55dbf5b400f5f942816e3c7f812e9594a5
SimHash 7b58d8f3d035

Groups

mediapartners-google

Rule Path
Allow /

*

Rule Path
Disallow /*/zglos-problem.html
Disallow /strona/*/get-comments-*
Disallow /strona/*/get-comments
Disallow /uzytkownik/*/comments-
Disallow /uzytkownik/*/shoutbox-
Disallow /szukaj/

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

xenu

Rule Path
Disallow /

synthesio

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

wget

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

spbot

Rule Path
Disallow /