mountsinai.org
robots.txt

Robots Exclusion Standard data for mountsinai.org

Resource Scan

Scan Details

Site Domain mountsinai.org
Base Domain mountsinai.org
Scan Status Ok
Last Scan2024-08-29T11:19:50+00:00
Next Scan 2024-09-28T11:19:50+00:00

Last Scan

Scanned2024-08-29T11:19:50+00:00
URL https://mountsinai.org/robots.txt
Redirect https://www.mountsinai.org/robots.txt
Redirect Domain www.mountsinai.org
Redirect Base mountsinai.org
Domain IPs 146.203.138.44
Redirect IPs 23.210.108.159
Response IP 173.223.95.13
Found Yes
Hash c2a67da08cd6414c21d17450c9d19a2b34648086be477085cba1b1481b727fda
SimHash f92cdc70ce9b

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /tmp/
Disallow /vgn-ext/templating
Disallow /*lp/
Disallow /insurance/
Disallow /about/preparedness/coronavirus/staff-resources*
Disallow /about/covid19/staff-resources*

alexabot

Rule Path
Disallow /about-us/newsroom/

Other Records

Field Value
crawl-delay 5

ccbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

blexbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

findxbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

rogerbot

Rule Path
Disallow /about-us/newsroom/

Other Records

Field Value
crawl-delay 10

yeti

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

semrushbot-sa

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

semrushbot-desktop

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

opensiteexplorer.org

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

sufog.com

Rule Path
Disallow /

mj12

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

ads.mopub.com

Rule Path
Disallow /

mj12bot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.mountsinai.org/files/MSHealth/Assets/sitemaps/care-sitemap.xml
sitemap https://www.mountsinai.org/files/MSHealth/Assets/sitemaps/healthlibrary-sitemap.xml
sitemap https://www.mountsinai.org/files/MSHealth/Assets/sitemaps/locations-sitemap.xml
sitemap https://www.mountsinai.org/files/MSHealth/Assets/sitemaps/doctors-sitemap.xml