helmet.finna.fi
robots.txt

Robots Exclusion Standard data for helmet.finna.fi

Resource Scan

Scan Details

Site Domain helmet.finna.fi
Base Domain finna.fi
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-05-29T20:16:57+00:00
Next Scan 2025-07-28T20:16:57+00:00

Last Successful Scan

Scanned2025-03-24T20:15:16+00:00
URL https://helmet.finna.fi/robots.txt
Domain IPs 193.166.187.203
Response IP 193.166.187.203
Found Yes
Hash d5e20650ac1a19d42f26091903302336aad63cde6c591acc64bc741d5bd9a598
SimHash 7d8258f98b91

Groups

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claudebot/1.0

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

omgili

Rule Path
Disallow /

academicbotrtu

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

test-bot

Rule Path
Disallow /

*

Rule Path
Disallow /AJAX/
Disallow /*/AJAX/
Disallow /Search/FacetList
Disallow /*/Search/FacetList
Disallow /QRCode/
Disallow /*/QRCode/
Disallow /*/api/
Disallow /*/*/api/
Disallow /MetaLib/
Disallow /*/MetaLib/
Disallow /Primo/
Disallow /*/Primo/
Disallow /PrimoRecord/
Disallow /*/PrimoRecord/
Disallow /Record/*/Holdings
Disallow /*/Record/*/Holdings
Disallow /SearchApi/
Disallow /*/SearchApi/
Disallow /EDS/
Disallow /*/EDS/
Disallow /EdsRecord/
Disallow /*/EdsRecord/
Disallow /Summon/
Disallow /*/Summon/
Disallow /SummonRecord/
Disallow /*/SummonRecord/
Disallow /Record/DownloadFile
Disallow /Record/
Disallow /*/Record/
Disallow /AJAX/
Disallow */AJAX/
Disallow /Blender/
Disallow */Blender/
Disallow /Browse/
Disallow */Browse/
Disallow /EDS/
Disallow */EDS/
Disallow /EDSRecord/
Disallow */EDSRecord/
Disallow /L1/
Disallow */L1/
Disallow /MyResearch/
Disallow */MyResearch/
Disallow /Primo/
Disallow */Primo/
Disallow /PrimoRecord/
Disallow */PrimoRecord/
Disallow /QRCode/
Disallow */QRCode/
Disallow /Search/
Disallow */Search/
Disallow /Search2/
Disallow */Search2/
Disallow /Summon/
Disallow */Summon/
Disallow /SummonRecord/
Disallow */SummonRecord/

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://helmet.finna.fi/sitemapIndex.xml.gz

Comments

  • Take into account both the default view and any non-default views
  • (hence two versions of each url)
  • Block QRCodes:
  • Block any api version:
  • Allow indexing of record pages by commenting out the following two lines: