physiciansweekly.com
robots.txt

Robots Exclusion Standard data for physiciansweekly.com

Resource Scan

Scan Details

Site Domain physiciansweekly.com
Base Domain physiciansweekly.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-10T22:48:04+00:00
Next Scan 2024-08-08T22:48:04+00:00

Last Successful Scan

Scanned2023-07-16T22:27:28+00:00
URL https://physiciansweekly.com/robots.txt
Domain IPs 172.66.41.43, 172.66.42.213, 2606:4700:3108::ac42:292b, 2606:4700:3108::ac42:2ad5
Response IP 172.66.41.43
Found Yes
Hash 7013b023cb70e2423980b4ec594950d196540bb9225c7de620576845ada7d055
SimHash 61014b1a4693

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /index.php
Disallow /xmlrpc.php
Disallow /wp-content/plugins/
Disallow /survey/
Disallow /oncology-tablet-survey/

mediapartners-google*

Rule Path
Allow /

googlebot-image

Rule Path
Allow /wp-content/uploads/

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

Other Records

Field Value
sitemap http://www.physiciansweekly.com/sitemap.xml