ilsos.gov
robots.txt

Robots Exclusion Standard data for ilsos.gov

Resource Scan

Scan Details

Site Domain ilsos.gov
Base Domain ilsos.gov
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-22T21:29:36+00:00
Next Scan 2024-08-20T21:29:36+00:00

Last Successful Scan

Scanned2023-01-01T22:30:34+00:00
URL https://ilsos.gov/robots.txt
Domain IPs 12.3.98.151
Response IP 12.3.98.151
Found Yes
Hash 9866fdd00f64b25e43ac6bd9d5bc378393de4226ddf9acf141d2482c057da858
SimHash 707cd71165e7

Groups

*

Rule Path
Disallow /

swiftbot

Rule Path
Allow /
Disallow /pert/
Disallow /search

applebot

Rule Path
Allow /
Disallow /pert/
Disallow /search

archive.org_bot

Rule Path
Allow /
Disallow /pert/
Disallow /search

archive-it

Rule Path
Allow /
Disallow /pert/
Disallow /search

ia_archiver

Rule Path
Allow
Disallow /pert/
Disallow /search

bingbot

Rule Path
Allow /
Disallow /pert/
Disallow /search

duckduckbot

Rule Path
Allow /
Disallow /pert/
Disallow /search

facebookexternalhit/1.1

Rule Path
Allow /
Disallow /pert/
Disallow /search

googlebot

Rule Path
Allow /
Disallow /pert/
Disallow /search

googlebot-image

Rule Path
Allow /
Disallow /pert/
Disallow /search

googlebot-mobile

Rule Path
Allow /
Disallow /pert/
Disallow /search

googlebot-news

Rule Path
Allow /
Disallow /pert/
Disallow /search

heritrix

Rule Path
Allow /
Disallow /pert/
Disallow /search

hubspot crawler

Rule Path
Allow /
Disallow /pert/
Disallow /search

msnbot

Rule Path
Allow /
Disallow /pert/
Disallow /search

pinterestbot

Rule Path
Allow /
Disallow /pert/
Disallow /search

slurp

Rule Path
Allow
Disallow /pert/
Disallow /search

special_archiver

Rule Path
Allow /
Disallow /pert/
Disallow /search

twitterbot

Rule Path
Allow /
Disallow /pert/
Disallow /search

yahoo!

Rule Path
Allow /
Disallow /pert/
Disallow /search

yahoo-mmcrawler

Rule Path
Allow
Disallow /pert/
Disallow /search

yahoo-blogs/v3.9

Rule Path
Allow
Disallow /pert/
Disallow /search

bitlybot

Rule Path
Disallow /

brandverity/1.0

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chrome-lighthouse

Rule Path
Disallow /

dataprovider.com

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

diffeobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

episerver link checker

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

genieo

Rule Path
Disallow /

http://lookseek.com/seeker/

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

lightspeedsystemscrawler

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

monsidobot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

proximic

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

t-h-u-n-d-e-r-s-t-o-n-e

Rule Path
Disallow /

tineye

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yioopbot

Rule Path
Disallow /

Comments

  • Default for search engines is disallow
  • Updated 20220928
  • Local services at ilsos.gov
  • Allowed Trusted Search Engines
  • Revoked Known Search Engines: Unecessary