caffebook.it
robots.txt

Robots Exclusion Standard data for caffebook.it

Resource Scan

Scan Details

Site Domain caffebook.it
Base Domain caffebook.it
Scan Status Ok
Last Scan2025-04-02T16:49:53+00:00
Next Scan 2025-04-09T16:49:53+00:00

Last Scan

Scanned2025-04-02T16:49:53+00:00
URL https://caffebook.it/robots.txt
Domain IPs 104.21.36.131, 172.67.194.109, 2606:4700:3032::6815:2483, 2606:4700:3035::ac43:c26d
Response IP 104.21.36.131
Found Yes
Hash 1714affeb078c23bb6cb975b45566fd4ba1f4e2bf60f57ea925a6be89fbe7227
SimHash 6a014a0283b3

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Disallow /wp-content/plugins/
Disallow /wp-admin/

Other Records

Field Value
sitemap https://caffebook.it/sitemap_index.xml