lgbtru.com
robots.txt
Robots Exclusion Standard data for lgbtru.com
Resource Scan
Scan Details
| Site Domain | lgbtru.com |
| Base Domain | lgbtru.com |
| Scan Status | Ok |
| Last Scan | 2024-09-27T00:45:09+00:00 |
| Next Scan | 2024-10-27T00:45:09+00:00 |
Last Scan
| Scanned | 2024-09-27T00:45:09+00:00 |
| URL | https://lgbtru.com/robots.txt |
| Domain IPs | 142.132.196.168, 145.239.67.120, 172.104.232.45 |
| Response IP | 145.239.67.120 |
| Found | Yes |
| Hash | 2df0a39e2042a543c413fc26d9fbda9e087aa9df1198bf5de805adc623c36b10 |
| SimHash | 9369b4fb4e0b |
Groups
websauger
webfetch
surfbot
larbin
download demon
image stripper
grafula
ahrefsbot
webleacher
smartdownload
eyenetie
pcbrowser
jetcar
web sucker
mj12bot
webauto
zeus
webwhacker
leechftp
express webpictures
webcopier
wwwoffle
xaldon webspider
ahrefsbot/5.1
dotbot
emailwolf
flashget
voideye
go-ahead-got-it
superbot
midown tool
blackwidow
wget
indy library
image sucker
superhttp
teleport pro
papa foto
webzip
netspider
bot [email="craftbot@yahoo.com"]mailto:craftbot@yahoo.com[/email]
httrack
semrushbot/1.1~bl
webgo is
extractorpro
webstripper
getright
eirgrabber
interget
mister pix
exabot
website quester
getweb!
hmview
custo
nearsite
disco
ecatch
offline explorer
mass downloader
web image collector
joc web spider
realdownload
gigabot
takeout
linkpadbot
webreaper
sitesnagger
rogerbot
net vampire
netzip
reget
internet ninja
grabnet
navroad
pavuk
website extractor
netants
octopus
pagegrabber
emailsiphon
chinaclaw
semrushbot
widow
go!zilla
offline navigator
| Rule | Path |
|---|---|
| Disallow | / |
*
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |