ilsos.net
robots.txt

Robots Exclusion Standard data for ilsos.net

Archived Snapshots

Resource Scan

Scan Details

Site Domain	ilsos.net
Base Domain	ilsos.net
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-11-15T06:19:49+00:00
Next Scan	2025-02-13T06:19:49+00:00

Last Successful Scan

Scanned	2022-12-31T09:44:45+00:00
URL	https://ilsos.net/robots.txt
Domain IPs	12.3.98.151
Response IP	12.3.98.151
Found	Yes
Hash	9866fdd00f64b25e43ac6bd9d5bc378393de4226ddf9acf141d2482c057da858
SimHash	707cd71165e7

Groups

*

Rule	Path
Disallow	/

Rule

Path

Disallow

swiftbot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

applebot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

archive.org_bot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

archive-it

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

ia_archiver

Rule	Path
Allow
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

bingbot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

duckduckbot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

facebookexternalhit/1.1

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

googlebot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

googlebot-image

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

googlebot-mobile

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

googlebot-news

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

heritrix

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

hubspot crawler

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

msnbot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

pinterestbot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

slurp

Rule	Path
Allow
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

special_archiver

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

twitterbot

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

yahoo!

Rule	Path
Allow	/
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

yahoo-mmcrawler

Rule	Path
Allow
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

yahoo-blogs/v3.9

Rule	Path
Allow
Disallow	/pert/
Disallow	/search

Rule

Path

Allow

Disallow

/pert/

Disallow

/search

bitlybot

Rule	Path
Disallow	/

Rule

Path

Disallow

brandverity/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

chrome-lighthouse

Rule	Path
Disallow	/

Rule

Path

Disallow

dataprovider.com

Rule	Path
Disallow	/

Rule

Path

Disallow

deepcrawl

Rule	Path
Disallow	/

Rule

Path

Disallow

diffeobot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

episerver link checker

Rule	Path
Disallow	/

Rule

Path

Disallow

flipboardproxy

Rule	Path
Disallow	/

Rule

Path

Disallow

genieo

Rule	Path
Disallow	/

Rule

Path

Disallow

http://lookseek.com/seeker/

Rule	Path
Disallow	/

Rule

Path

Disallow

ips-agent

Rule	Path
Disallow	/

Rule

Path

Disallow

lightspeedsystemscrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

linkdexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

monsidobot

Rule	Path
Disallow	/

Rule

Path

Disallow

panscient.com

Rule	Path
Disallow	/

Rule

Path

Disallow

proximic

Rule	Path
Disallow	/

Rule

Path

Disallow

screaming frog seo spider

Rule	Path
Disallow	/

Rule

Path

Disallow

semanticscholarbot

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

surdotlybot

Rule	Path
Disallow	/

Rule

Path

Disallow

t-h-u-n-d-e-r-s-t-o-n-e

Rule	Path
Disallow	/

Rule

Path

Disallow

tineye

Rule	Path
Disallow	/

Rule

Path

Disallow

yacybot

Rule

Path

Disallow

yioopbot

Rule

Path

Disallow

Comments

Default for search engines is disallow
Updated 20220928
Local services at ilsos.gov
Allowed Trusted Search Engines
Revoked Known Search Engines: Unecessary

ilsos.netrobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

*

swiftbot

applebot

archive.org_bot

archive-it

ia_archiver

bingbot

duckduckbot

facebookexternalhit/1.1

googlebot

googlebot-image

googlebot-mobile

googlebot-news

heritrix

hubspot crawler

msnbot

pinterestbot

slurp

special_archiver

twitterbot

yahoo!

yahoo-mmcrawler

yahoo-blogs/v3.9

bitlybot

brandverity/1.0

ccbot

chrome-lighthouse

dataprovider.com

deepcrawl

diffeobot

dotbot

episerver link checker

flipboardproxy

genieo

http://lookseek.com/seeker/

ips-agent

lightspeedsystemscrawler

linkdexbot

mj12bot

monsidobot

panscient.com

proximic

screaming frog seo spider

semanticscholarbot

semrushbot

surdotlybot

t-h-u-n-d-e-r-s-t-o-n-e

tineye

yacybot

yioopbot

Comments

ilsos.net
robots.txt