cdn.scottsdaleaz.gov
robots.txt

Robots Exclusion Standard data for cdn.scottsdaleaz.gov

Resource Scan

Scan Details

Site Domain cdn.scottsdaleaz.gov
Base Domain scottsdaleaz.gov
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-05T18:02:27+00:00
Next Scan 2024-12-04T18:02:27+00:00

Last Successful Scan

Scanned2022-09-27T02:53:06+00:00
URL http://cdn.scottsdaleaz.gov/robots.txt
Response IP 13.107.246.59, 13.107.213.59
Found Yes
Hash cfa2679f32c6027e52f2c5e9bffc27295a1e07feb3233a92d6b8afab8bfd3396
SimHash 29109375d681

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /tmp/
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.gif$
Allow /*.png$
Allow /*.webp$

dmca.com page protection crawling service

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

yahoo pipes 1.0

Rule Path
Allow /

yahoo! slurp

Rule Path
Allow /

yandexbot

Rule Path
Allow /

yandeximages

Rule Path
Allow /

yandexnews

Rule Path
Allow /

yandexwebmaster

Rule Path
Allow /

yandexpagechecker

Rule Path
Allow /

zyborg

Rule Path
Allow /

exabot

Rule Path
Allow /

facebot

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

ahrefbot
ahref
archive.org_bot

Rule Path
Allow /

architextspider

Rule Path
Allow /

feedfetcher-google

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

Other Records

Field Value
sitemap http://cdn.scottsdaleaz.gov/sitemap.xml

Warnings

  • 2 invalid lines.
  • `disalow` is not a known field.