Log In Sign Up

astra-g.pl
robots.txt

Robots Exclusion Standard data for astra-g.pl

Resource Scan

Scan Details

Site Domain astra-g.pl
Base Domain astra-g.pl
Scan Status Ok
Last Scan2025-11-21T12:02:41+00:00
Next Scan 2025-11-28T12:02:41+00:00

Last Scan

Scanned2025-11-21T12:02:41+00:00
URL https://astra-g.pl/robots.txt
Redirect https://www.astra-g.pl/robots.txt
Redirect Domain www.astra-g.pl
Redirect Base astra-g.pl
Domain IPs 185.110.51.100
Redirect IPs 185.110.51.100
Response IP 185.110.51.100
Found Yes
Hash 73213d7ecf49e976794bc48145fdc24d4ade7ec84d7bf85b9e12b3140b7d212d
SimHash bc184d4beca0

Groups

urlmetriken

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

fast

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

linguee

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

neticle.hu

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

semrushbot-seoab

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120
Back to top

Comments

  • advertising-related bots:
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Misbehaving: requests much too fast:
  • Webmeup.com
Back to top