socalsubies.com
robots.txt
Robots Exclusion Standard data for socalsubies.com
Resource Scan
Scan Details
| Site Domain | socalsubies.com |
| Base Domain | socalsubies.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-04-18T21:06:42+00:00 |
| Next Scan | 2024-07-17T21:06:42+00:00 |
Last Successful Scan
| Scanned | 2023-09-22T12:18:55+00:00 |
| URL | https://socalsubies.com/robots.txt |
| Redirect | https://www.socalsubies.com/robots.txt |
| Redirect Domain | www.socalsubies.com |
| Redirect Base | socalsubies.com |
| Domain IPs | 104.16.248.51, 104.16.249.51 |
| Redirect IPs | 104.17.97.5, 104.17.98.5, 2606:4700::6811:6105, 2606:4700::6811:6205 |
| Response IP | 104.17.98.5 |
| Found | Yes |
| Hash | 6c06f38a5ad1b7edb1d05da271e12d5ea68dae93cf73be7a1603d5f6208ada09 |
| SimHash | f20f6647b45c |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /images/ |
| Disallow | /forums/images/ |
| Disallow | /movies/ |
| Disallow | /photos/ |
| Disallow | /graphics/ |
| Disallow | /uploads/ |
| Disallow | /forums/showthread.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 120 |
bingbot
| Rule | Path |
|---|---|
| Disallow | /images/ |
| Disallow | /forums/images/ |
| Disallow | /movies/ |
| Disallow | /photos/ |
| Disallow | /graphics/ |
| Disallow | /uploads/ |
| Disallow | /forums/showthread.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 4 |
slurp
yahoo! slurp
msnbot
msnbot-media
twitterbot
baiduspider
tweetedtimes bot
tweetmemebot
paperlibot
bomborabot
wf search
heritrix
expo9
netseer
addthis.com
qwantify
feed
sogou
zumbot
feedspot
acebookexternalhit
facebookexternalhit
storygizebot
scrapy
germcrawler
netvibes
vagabondo
genieo
um-ic
typhoeus
newspaper
weborama-fetcher
scooperbot
byo-2
simplepie
grouphigh
feedparser
universalfeedparser
pulsepoint
steeler
garlikcrawler
flipboard
flipboardrss
moreover
mercury
feedburner
yahoo
omgili
bloglovin
yahoocachesystem
mtalk
everyonesocialbot
ezinearticleslinkscanner
dlvr.it
pfblogs
aol
toutiaospider
photon
picofeed
um-fc
digg
synaptive
yisouspider
byo-4
feedbin
twingly
tiny
www-mechanize
commafeed
tkbot
tineye-bot
qwantify
exabot
coccoc
blexbot
umbot-ln
queryseekerspider
yandex
yandexbot
yandexmobilebot
yandeximageresizer
yandeximages
sogou spider
sogou web spider
baiduspider
youdaobot
mj12bot
spbot
seznambot
ahrefsbot
linkdex
linkdexbot
semrushbot
semrushbot-sa
cfnetwork
sputnikbot
turnitinbot
proximic
grapeshot
viglink
clickagy
yak
deusu
taboolabot
pinterestbot
psbot
plukkie
getintent
facebot
linkdexbot
vegi
istellabot
leikibot
claritybot
piplbot
feedspotbot
moatbot
yeti
contxbot
seokicks-robot
rogerbot
discordbot
tapatalk
mail.ru_bot
coccocbot
hybridbot
slack-imgproxy
twitterbot
adbeat_bot
kraken
mojeekbot
calculon
mcsbot
avkzarabotok
searchmetricsbot
uptimebot
yacybot
applebot
gigabot
g2reader-bot
seoterritory
ahrefsbot
feedbot
findxbot
wotbox
msnbot
smtbot
ocarinabot
dragonbot
zombiebot
metabot.ru
rssmicro
_zbot
uptimerobot
uipbot
zoominfobot
flamingo_searchengine
bot-pge.chlooe.com
girafabot
multiviewbot
safednsbot
gnowitnewsbot
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
uptimerobot
uipbot
zoominfobot
flamingo_searchengine
bot-pge.chlooe.com
girafabot
multiviewbot
safednsbot
gnowitnewsbot
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
cliqzbot
dotbot
exabot
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
Warnings
- 1 invalid line.