Log In Sign Up

boatflow.jp
robots.txt

Robots Exclusion Standard data for boatflow.jp

Resource Scan

Scan Details

Site Domain boatflow.jp
Base Domain boatflow.jp
Scan Status Ok
Last Scan2026-01-16T20:28:25+00:00
Next Scan 2026-01-23T20:28:25+00:00

Last Scan

Scanned2026-01-16T20:28:25+00:00
URL https://boatflow.jp/robots.txt
Domain IPs 104.21.48.161, 172.67.154.85, 2606:4700:3033::ac43:9a55, 2606:4700:3037::6815:30a1
Response IP 172.67.154.85
Found Yes
Hash d90568b963b270e28410b3593047aa1f217490cd10e446b0dbf347cd56ea3aa2
SimHash 40181e37a5b3

Groups

googlebot
bingbot
slurp
duckduckbot
baiduspider
yandexbot
facebookexternalhit
twitterbot
linkedinbot
whatsapp
applebot

Rule Path
Disallow /admin
Disallow /sidekiq
Disallow /users/sign_in
Disallow /users/sign_up
Disallow /users/password
Allow /

Other Records

Field Value
crawl-delay 1

httrack
wget
curl
scrapy
python-requests
libwww-perl
mechanize
go-http-client
okhttp
apache-httpclient
node-fetch
axios
postman
insomnia
nikto
sqlmap
nmap
masscan
zap
burp
w3af
skipfish
dirb
dirbuster
gobuster
wfuzz
ffuf
hydra
medusa
metasploit
nessus
openvas
nuclei
sqlninja
havij
pangolin
bbqsql
bsqlbf
mole
paros
webscarab
webinspect
appscan
acunetix
qualys
rapid7
veracode
checkmarx
fortify
sonarqube
snyk

Rule Path
Disallow /

*

Rule Path
Disallow /admin/
Disallow /sidekiq/
Disallow /users/sign_in
Disallow /users/sign_up
Disallow /users/password
Disallow /search?*
Disallow /*?*
Disallow /tmp/
Disallow /cache/
Disallow /log/
Disallow /*.json
Disallow /*.xml$
Disallow /*_xmlhttprequest
Disallow /*ajax*
Disallow /api/

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap http://cdn41.s3.amazonaws.com/sitemaps/sitemap.xml.gz
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • Allow good bots
  • Block known scrapers and bad bots
  • Default rules for other bots
Back to top

Warnings

  • `host` is not a known field.
Back to top