villanueva.edu
robots.txt
Robots Exclusion Standard data for villanueva.edu
Resource Scan
Scan Details
| Site Domain | villanueva.edu |
| Base Domain | villanueva.edu |
| Scan Status | Ok |
| Last Scan | 2024-10-28T06:18:38+00:00 |
| Next Scan | 2024-11-27T06:18:38+00:00 |
Last Scan
| Scanned | 2024-10-28T06:18:38+00:00 |
| URL | https://villanueva.edu/robots.txt |
| Domain IPs | 89.17.193.183 |
| Response IP | 89.17.193.183 |
| Found | Yes |
| Hash | e77e63477118695bf9425eb5aca3cb1d6e3845d4c90e01b13c8f3d19d77643d4 |
| SimHash | bbdf74c089e6 |
Groups
*
| Rule | Path |
|---|---|
| Disallow |
addthis.com
admantx
ahrefsbot
bdcbot
bender
bixocrawler
bl.uk_lddc_bot
blexbot
bubing
cliqzbot
cncdialer
crawler4j
crystalsemanticsbot
cyberalert
digext
discobot
discoverybot
dloader
dloader(naverrobot)
doc
dotbot
download ninja
dts agent
exabot
ezooms
fairshare
fetch
flamingo_searchengine
genieo
gigabot
grub-client
heritrix
heritrix/3.3.0
httrack
ia_archiver
integromedb
istellabot
jikespider
jyxobot
k2spider
kimengi
kimengi/nineconnections.com
larbin
lexxebot/1.0
libwww
linko
livelapbot
magpie-crawler
maxthon
metauri
microsoft.url.control
mj12bot
moreover
moreoverbot
msiecrawler
nabot
naverbot
nerdbynature.bot
netestate ne crawler
netseer crawler
newscan
nextgensearchbot
npbot
nutch
offline explorer
omgilibot
orthogaffe
piplbot
pixray-seeker
proximic
psbot
queryseekerspider
rogerbot
seokicks
seokicks-robot
sitebot
sitebot/0.1
sitecheck.internetseer.com
sitesnagger
slurp
sogou
sosospider
spbot
spinn3r
teleport
teleportpro
trendictionbot
trovitbot
turnitinbot
ubicrawler
umbot-ln
unisterbot
universalfeedparser
wbsearchbot
webcopier
webreaper
webstripper
webzip
wesee:search
wget
wotbot
wotbox
xenu
yasni
zao
zealbot
zyborg
| Rule | Path |
|---|---|
| Disallow | / |
Warnings
- 1 invalid line.
Comments