Log In Sign Up

www.acces.nysed.gov
robots.txt

Robots Exclusion Standard data for www.acces.nysed.gov

Resource Scan

Scan Details

Site Domain www.acces.nysed.gov
Base Domain nysed.gov
Scan Status Ok
Last Scan2025-03-03T18:34:55+00:00
Next Scan 2025-04-02T18:34:55+00:00

Last Scan

Scanned2025-03-03T18:34:55+00:00
URL https://www.acces.nysed.gov/robots.txt
Domain IPs 52.7.171.100
Response IP 52.7.171.100
Found Yes
Hash 8e61597e50020daa4d07598f841556a0bb2a3e2c69f373054073ebde563a0e5d
SimHash 0ab49f09c754

Groups

alexa

Product Comment
alexa Amazon's user agent
Rule Path Comment
Disallow / disallow everything

alexabot

Product Comment
alexabot Amazon's user agent
Rule Path Comment
Disallow / disallow everything

alexatoolbar

Product Comment
alexatoolbar Amazon's user agent
Rule Path Comment
Disallow / disallow everything

aranhabot

Product Comment
aranhabot Amazon's user agent
Rule Path Comment
Disallow / disallow everything

amazon-qbusiness

Product Comment
amazon-qbusiness Amazon Q Web Crawler
Rule Path Comment
Disallow / disallow everything

amazon cloudfront

Product Comment
amazon cloudfront Amazon's user agent
Rule Path Comment
Disallow / disallow everything

amazonbot

Product Comment
amazonbot Amazon's user agent
Rule Path Comment
Disallow / disallow everything

amzn_assoc

Product Comment
amzn_assoc Amazon's user agent
Rule Path Comment
Disallow / disallow everything

amazon-kendra-web-crawler-*

Product Comment
amazon-kendra-web-crawler-* all customers of Amazon Kendra's web crawler
Rule Path Comment
Disallow / disallow everything

amazon-kendra

Product Comment
amazon-kendra all customers of Amazon Kendra's web crawler
Rule Path Comment
Disallow / disallow everything

amazonadbot

Product Comment
amazonadbot Amazon's user agent
Rule Path Comment
Disallow / disallow everything

amazonproductdiscoverybot

Product Comment
amazonproductdiscoverybot Amazon's user agent
Rule Path Comment
Disallow / disallow everything

aws*

Product Comment
aws* Amazon's user agent
Rule Path Comment
Disallow / disallow everything

aws-internal

Product Comment
aws-internal Amazon's user agent
Rule Path Comment
Disallow / disallow everything

aws*

Product Comment
aws* Amazon's user agent
Rule Path Comment
Disallow / disallow everything

echo

Product Comment
echo Amazon's user agent
Rule Path Comment
Disallow / disallow everything

echo*

Product Comment
echo* Amazon's user agent
Rule Path Comment
Disallow / disallow everything

storebot-google

Product Comment
storebot-google Google's user agent
Rule Path Comment
Disallow / disallow everything

googlebot-image

Product Comment
googlebot-image Google's user agent
Rule Path Comment
Disallow / disallow everything

googlebot-news

Product Comment
googlebot-news Google's user agent
Rule Path Comment
Disallow / disallow everything

googlebot-news

Product Comment
googlebot-news Google's user agent
Rule Path Comment
Disallow / disallow everything

googlebot-video

Product Comment
googlebot-video Google's user agent
Rule Path Comment
Disallow / disallow everything

adsbot-google-mobile

Product Comment
adsbot-google-mobile Google's user agent
Rule Path Comment
Disallow / disallow everything

mediapartners-google

Product Comment
mediapartners-google Google's user agent
Rule Path Comment
Disallow / disallow everything

feedfetcher-google

Product Comment
feedfetcher-google Google's user agent
Rule Path Comment
Disallow / disallow everything

googleproducer

Product Comment
googleproducer Google's user agent
Rule Path Comment
Disallow / disallow everything

googleother

Product Comment
googleother Google's user agent
Rule Path Comment
Disallow / disallow everything

google-extended

Product Comment
google-extended Google's user agent
Rule Path Comment
Disallow / disallow everything

apis-google

Product Comment
apis-google Google's user agent
Rule Path Comment
Disallow / disallow everything

*

Rule Path
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /COMS/
Disallow /stateaid/
Disallow /cafe/
Disallow /heds/
Disallow /dpls/
Disallow /tapeusage/
Disallow /cn/
Disallow /fm/
Disallow /fmis/
Disallow /stateaid/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /homepage-slider/*
Disallow /node/4695

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap http://www.nysed.gov/sitemap.xml
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
Back to top

Warnings

  • 2 invalid lines.
Back to top