davidzwirner.com
robots.txt

Robots Exclusion Standard data for davidzwirner.com

Resource Scan

Scan Details

Site Domain davidzwirner.com
Base Domain davidzwirner.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-02T09:57:47+00:00
Next Scan 2024-12-01T09:57:47+00:00

Last Successful Scan

Scanned2024-04-13T07:32:57+00:00
URL https://davidzwirner.com/robots.txt
Redirect https://www.davidzwirner.com/robots.txt
Redirect Domain www.davidzwirner.com
Redirect Base davidzwirner.com
Domain IPs 18.155.68.39, 18.155.68.52, 18.155.68.61, 18.155.68.90
Redirect IPs 18.155.68.39, 18.155.68.52, 18.155.68.61, 18.155.68.90
Response IP 18.155.68.90
Found Yes
Hash 9ba39cf5810556688b7571cc7b59cd56e7a7dcf3d24f49924f01bd94d5eac3fd
SimHash 415edc70c631

Groups

*

Rule Path
Allow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider+

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /api/*

Other Records

Field Value
sitemap https://www.davidzwirner.com/sitemap.xml