ncsl.org
robots.txt

Robots Exclusion Standard data for ncsl.org

Resource Scan

Scan Details

Site Domain ncsl.org
Base Domain ncsl.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2024-09-21T20:02:08+00:00
Next Scan 2024-10-21T20:02:08+00:00

Last Successful Scan

Scanned2024-07-31T17:37:41+00:00
URL https://ncsl.org/robots.txt
Domain IPs 162.213.217.181
Response IP 162.213.217.181
Found Yes
Hash 2b66069fa0286f38b70a7b0935b9448ef9cea1237c729fcde0b65b04f1d32619
SimHash 721d7d63a482

Groups

*

Rule Path
Disallow /admin/
Disallow /App_GlobalResources/
Disallow /bin/
Disallow /Components/
Disallow /contest/
Disallow /controls/
Disallow /HttpModules/
Disallow /images/
Disallow /statefed/
Disallow /public/
Disallow /Install/
Disallow /Providers/
Disallow /template/
Disallow /ffis/

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

awariorssbot
awariosmartbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.ncsl.org/sitemap.aspx

Comments

  • Begin robots.txt file
  • /-----------------------------------------------\
  • | In single portal/domain situations, uncomment the sitmap line and enter domain name
  • \-----------------------------------------------/
  • End of robots.txt file