Log In Sign Up

hondanews.eu
robots.txt

Robots Exclusion Standard data for hondanews.eu

Resource Scan

Scan Details

Site Domain hondanews.eu
Base Domain hondanews.eu
Scan Status Ok
Last Scan2024-09-11T12:25:47+00:00
Next Scan 2024-10-11T12:25:47+00:00

Last Scan

Scanned2024-09-11T12:25:47+00:00
URL https://hondanews.eu/robots.txt
Domain IPs 3.98.252.51, 35.183.247.62
Response IP 3.98.252.51
Found Yes
Hash c3913c43f355a8df7d706fd2c6d6a3bf19805784708a410d4ea96f7490593daa
SimHash f0731948c330

Groups

ahrefsbot
ezooms
sistrix
mj12bot
megaindex.ru
megaindex.com
petalbot

Rule Path
Disallow /

ccbot
claudebot
claude-web
chatgpt-user
gptbot
google-extended
applebot-extended
anthropic-ai
omgilibot
omgili
facebookbot
diffbot
bytespider
imagesiftbot
perplexitybot
cohere-ai

Rule Path
Disallow /

*

Rule Path
Disallow /*/*/account/*
Disallow /*/*/search/
Disallow /*/*/print/
Disallow /*/*/download/*
Disallow /*/*/basket/*
Disallow /*/enhanced/
Disallow /Content/Compiled/
Disallow /Content/JQueryUI/
Disallow /Content/Flash/
Disallow /Content/Fonts/
Disallow /Scripts/
Disallow /Style/

Other Records

Field Value
crawl-delay 2
Back to top

Comments

  • AI Data Scrapers
  • ----------------
  • This list of bots based on https://darkvisitors.com/ and https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
  • Info on the different bots is possible at https://darkvisitors.com/
Back to top