Log In Sign Up

bantenhits.com
robots.txt

Robots Exclusion Standard data for bantenhits.com

Resource Scan

Scan Details

Site Domain bantenhits.com
Base Domain bantenhits.com
Scan Status Ok
Last Scan2024-11-11T05:50:04+00:00
Next Scan 2024-11-18T05:50:04+00:00

Last Scan

Scanned2024-11-11T05:50:04+00:00
URL https://bantenhits.com/robots.txt
Domain IPs 104.26.2.186, 104.26.3.186, 172.67.68.36, 2606:4700:20::681a:2ba, 2606:4700:20::681a:3ba, 2606:4700:20::ac43:4424
Response IP 104.26.2.186
Found Yes
Hash 075e51a7a33a541a3a67f8d1a57d3c729362d414a368a08f03ead176ce59e56e
SimHash 4b5dfcc8ea83

Groups

mediapartners-google
*

Rule Path
Disallow /wp-admin/

googlebot

Rule Path
Allow /

petalbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /wp-admin

arachni

Rule Path
Disallow /

acunetix

Rule Path
Disallow /

w3af.org

Rule Path
Disallow /

webinspect

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

nikto

Rule Path
Disallow /

nmap

Rule Path
Disallow /

openvas

Rule Path
Disallow /

nessus

Rule Path
Disallow /

masscan

Rule Path
Disallow /

zap

Rule Path
Disallow /

xenu

Rule Path
Disallow /

sqlninja

Rule Path
Disallow /

havij

Rule Path
Disallow /

grabber

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

httrack

Rule Path
Disallow /

paros

Rule Path
Disallow /

pangolin

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

scanbot

Rule Path
Disallow /

sucuri

Rule Path
Disallow /

detectify

Rule Path
Disallow /

nexpose

Rule Path
Disallow /

webshag

Rule Path
Disallow /

n-stalker

Rule Path
Disallow /

sitesucker

Rule Path
Disallow /

dnseye

Rule Path
Disallow /

rat scanner

Rule Path
Disallow /

wapiti

Rule Path
Disallow /

awesomium

Rule Path
Disallow /

w3m

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

smuggler

Rule Path
Disallow /

metasploit

Rule Path
Disallow /

appscan

Rule Path
Disallow /

wget

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

wpscan

Rule Path
Disallow /

wpscan

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshotcrawler

No rules defined. All paths allowed.

queryseekerspider

Rule Path
Disallow /

tineye

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

barkrowler

No rules defined. All paths allowed.

Back to top

Other Records

Field Value
sitemap https://bantenhits.com/feed
sitemap https://bantenhits.com/sitemap_index.xml
Back to top

Comments

  • Crawl-delay: 10
  • Block known web scanners and crawlers
  • Block WPScan
Back to top

Warnings

  • 3 invalid lines.
Back to top