cluegroup.pl
robots.txt

Robots Exclusion Standard data for cluegroup.pl

Resource Scan

Scan Details

Site Domain cluegroup.pl
Base Domain cluegroup.pl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-05-18T08:55:08+00:00
Next Scan 2025-07-17T08:55:08+00:00

Last Successful Scan

Scanned2025-02-25T07:02:32+00:00
URL https://cluegroup.pl/robots.txt
Domain IPs 157.90.182.9
Response IP 157.90.182.9
Found Yes
Hash adf8bd2eb5f27c3213cbd92eb871d1c1642c83f45a115d99c85c0d35f0393831
SimHash c344586a63b0

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/
Allow /wp-includes/js/
Allow /wp-includes/images/
Disallow /trackback/
Disallow /wp-login.php
Disallow /wp-register.php