dsl-bulgaria.com
robots.txt

Robots Exclusion Standard data for dsl-bulgaria.com

Resource Scan

Scan Details

Site Domain dsl-bulgaria.com
Base Domain dsl-bulgaria.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-02-19T02:16:16+00:00
Next Scan 2025-05-20T02:16:16+00:00

Last Successful Scan

Scanned2023-01-28T19:09:29+00:00
URL https://dsl-bulgaria.com/robots.txt
Redirect https://www.dsl-bulgaria.com/robots.txt
Redirect Domain www.dsl-bulgaria.com
Redirect Base dsl-bulgaria.com
Domain IPs 104.21.79.37, 172.67.141.138, 2606:4700:3035::ac43:8d8a, 2606:4700:3037::6815:4f25
Redirect IPs 104.21.79.37, 172.67.141.138, 2606:4700:3035::ac43:8d8a, 2606:4700:3037::6815:4f25
Response IP 104.21.79.37
Found Yes
Hash 58890524a9a01895469225e3bd0befbdff8b85ecfeec525cd1722bc2ab859d58
SimHash 600098020f93

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
sitemap https://www.dsl-bulgaria.com/wp-sitemap.xml