portlandmaine.gov
robots.txt

Robots Exclusion Standard data for portlandmaine.gov

Resource Scan

Scan Details

Site Domain portlandmaine.gov
Base Domain portlandmaine.gov
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-07-27T21:21:23+00:00
Next Scan 2024-10-25T21:21:23+00:00

Last Successful Scan

Scanned2022-06-04T21:48:49+00:00
URL https://portlandmaine.gov/robots.txt
Response IP 208.90.191.71
Found Yes
Hash 6771bf6465ddd6e600a7866b5e41825007c4a20433b378e56746547c40d205c5
SimHash 01199cd020e1

Groups

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

*

Rule Path
Disallow /activedit
Disallow /admin
Disallow /common/admin/
Disallow /OJA
Disallow /support
Disallow /currenteventsview.asp
Disallow /search.asp
Disallow /currenteventsview.aspx
Disallow /search.aspx
Disallow /currentevents.aspx
Disallow /Support
Disallow /CurrentEventsView.asp
Disallow /Search.asp
Disallow /CurrentEventsView.aspx
Disallow /Search.aspx
Disallow /Search
Disallow /CurrentEvents.aspx
Disallow /Currentevents.aspx
Disallow /map.aspx
Disallow /map.asp
Disallow /Map.aspx
Disallow /Map.asp
Disallow /RSS.aspx

siteimprove

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

siteimprovebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

siteimprovebot-crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

Other Records

Field Value
sitemap /sitemap.xml