deadsea.com
robots.txt

Robots Exclusion Standard data for deadsea.com

Resource Scan

Scan Details

Site Domain deadsea.com
Base Domain deadsea.com
Scan Status Ok
Last Scan2024-05-17T08:16:59+00:00
Next Scan 2024-05-24T08:16:59+00:00

Last Scan

Scanned2024-05-17T08:16:59+00:00
URL https://deadsea.com/robots.txt
Domain IPs 104.21.43.17, 172.67.215.217, 2606:4700:3031::6815:2b11, 2606:4700:3031::ac43:d7d9
Response IP 104.21.43.17
Found Yes
Hash f4f95115880982b0219118e47446de03fafc6bb4db318c15ab17ff8dd55c5f5c
SimHash 61085e3243a6

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 600

Warnings

  • `site` is not a known field.