Log In Sign Up

ospar.org
robots.txt

Robots Exclusion Standard data for ospar.org

Resource Scan

Scan Details

Site Domain ospar.org
Base Domain ospar.org
Scan Status Ok
Last Scan2026-01-25T18:55:35+00:00
Next Scan 2026-02-24T18:55:35+00:00

Last Scan

Scanned2026-01-25T18:55:35+00:00
URL https://ospar.org/robots.txt
Redirect https://www.ospar.org/robots.txt
Redirect Domain www.ospar.org
Redirect Base ospar.org
Domain IPs 104.21.21.248, 172.67.201.117, 2606:4700:3031::6815:15f8, 2606:4700:3034::ac43:c975
Redirect IPs 104.21.21.248, 172.67.201.117, 2606:4700:3031::6815:15f8, 2606:4700:3034::ac43:c975
Response IP 172.67.201.117
Found Yes
Hash 803903c32e511af29fe6433a57b54165666175771477c1dfc296e23a98a750cc
SimHash 432fd642c3a7

Groups

*

Rule Path
Disallow /search
Disallow /*search?

Other Records

Field Value
crawl-delay 5

teleport

Rule Path
Disallow /

webzip

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

ernie-bot

Rule Path
Disallow /

blowfish/1.0

Rule Path
Disallow /

grub

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

titan

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

bullseye/1.0

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

sogou

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

szukacz/1.4

Rule Path
Disallow /

botalot

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

crescent

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

foobot

Rule Path
Disallow /

harvest/1.5

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

moget

Rule Path
Disallow /

netants

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

radiation retriever

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

repomonkey bait & tackle/v1.01

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

wget

Rule Path
Disallow /

zeus

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ospar.org/sitemap.xml
Back to top

Comments

  • =======================
  • General User-agent Rule
  • =======================
  • ========================
  • Block Known Scraper Bots
  • ========================
  • ====================
  • Block Malicious Bots
  • ====================
  • =============================
  • Block High-Risk Regional Bots
  • =============================
  • ==============================
  • Block Known Data Scraping Bots
  • ==============================
  • =======
  • Sitemap
  • =======
Back to top

Warnings

  • 2 invalid lines.
Back to top