takethislife.com
robots.txt
Robots Exclusion Standard data for takethislife.com
Resource Scan
Scan Details
| Site Domain | takethislife.com |
| Base Domain | takethislife.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2024-10-15T12:05:04+00:00 |
| Next Scan | 2025-01-13T12:05:04+00:00 |
Last Successful Scan
| Scanned | 2023-09-21T23:54:04+00:00 |
| URL | https://takethislife.com/robots.txt |
| Redirect | https://www.takethislife.com/robots.txt |
| Redirect Domain | www.takethislife.com |
| Redirect Base | takethislife.com |
| Domain IPs | 104.16.248.51, 104.16.249.51 |
| Redirect IPs | 104.17.97.5, 104.17.98.5, 2606:4700::6811:6105, 2606:4700::6811:6205 |
| Response IP | 104.17.98.5 |
| Found | Yes |
| Hash | 62fd5574b0d2ba4c742c6d2720543dc4122f849711ac88aec369f8f06b96ea9b |
| SimHash | 72af4747661c |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /admincp/ |
| Disallow | /ajax.php |
| Disallow | /clientscript/ |
| Disallow | /cpstyles/ |
| Disallow | /images/ |
| Disallow | /includes/ |
| Disallow | /install/ |
| Disallow | /modcp/ |
| Disallow | /subscriptions/ |
| Disallow | /customavatars/ |
| Disallow | /customprofilepics/ |
| Disallow | /announcement.php |
| Disallow | /attachment.php |
| Disallow | /calendar.php |
| Disallow | /cron.php |
| Disallow | /editpost.php |
| Disallow | /faq.php |
| Disallow | /frm_attach |
| Disallow | /image.php |
| Disallow | /inlinemod.php |
| Disallow | /joinrequests.php |
| Disallow | /login.php |
| Disallow | /memberlist.php |
| Disallow | /misc.php |
| Disallow | /moderator.php |
| Disallow | /newattachment.php |
| Disallow | /newreply.php |
| Disallow | /newthread.php |
| Disallow | /online.php |
| Disallow | /payment_gateway.php |
| Disallow | /payments.php |
| Disallow | /poll.php |
| Disallow | /postings.php |
| Disallow | /private.php |
| Disallow | /profile.php |
| Disallow | /register.php |
| Disallow | /report.php |
| Disallow | /reputation.php |
| Disallow | /search.php |
| Disallow | /sendmessage.php |
| Disallow | /showgroups.php |
| Disallow | /showpost.php |
| Disallow | /subscription.php |
| Disallow | /usercp.php |
| Disallow | /threadrate.php |
| Disallow | /usernote.php |
bingbot
| Rule | Path |
|---|---|
| Disallow | /install/ |
| Disallow | /customavatars/ |
| Disallow | /admincp/ |
| Disallow | /clientscript/ |
| Disallow | /cpstyles/ |
| Disallow | /customprofilepics/ |
| Disallow | /modcp/ |
| Disallow | /ajax.php |
| Disallow | /attachment.php |
| Disallow | /calendar.php |
| Disallow | /cron.php |
| Disallow | /editpost.php |
| Disallow | /global.php |
| Disallow | /image.php |
| Disallow | /inlinemod.php |
| Disallow | /joinrequests.php |
| Disallow | /login.php |
| Disallow | /members/ |
| Disallow | /members.php |
| Disallow | /misc.php |
| Disallow | /moderator.php |
| Disallow | /newattachment.php |
| Disallow | /newreply.php |
| Disallow | /newthread.php |
| Disallow | /online.php |
| Disallow | /postings.php |
| Disallow | /printthread.php |
| Disallow | /private.php |
| Disallow | /register.php |
| Disallow | /report.php |
| Disallow | /reputation.php |
| Disallow | /search.php |
| Disallow | /sendmessage.php |
| Disallow | /showgroups.php |
| Disallow | /showpost.php |
| Disallow | /subscription.php |
| Disallow | /threadrate.php |
| Disallow | /usercp.php |
| Disallow | /usernote.php |
| Disallow | /customgroupicons/ |
| Disallow | /signaturepics/ |
| Disallow | /showthread.php |
| Disallow | /member.php |
| Disallow | /forumdisplay.php |
| Disallow | /blog.php |
| Disallow | /activity.php |
| Disallow | /index.php |
| Disallow | /itrader.php |
| Disallow | /garage.php |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 4 |
slurp
yahoo! slurp
msnbot
msnbot-media
twitterbot
baiduspider
tweetedtimes bot
tweetmemebot
paperlibot
netseer
bomborabot
wf search
heritrix
expo9
netseer
addthis.com
qwantify
feed
sogou
zumbot
feedspot
acebookexternalhit
facebookexternalhit
storygizebot
scrapy
germcrawler
netvibes
vagabondo
genieo
um-ic
typhoeus
newspaper
weborama-fetcher
scooperbot
byo-2
simplepie
grouphigh
feedparser
universalfeedparser
pulsepoint
steeler
garlikcrawler
flipboard
flipboardrss
moreover
mercury
feedburner
yahoo
omgili
bloglovin
yahoocachesystem
mtalk
everyonesocialbot
ezinearticleslinkscanner
dlvr.it
pfblogs
aol
toutiaospider
photon
picofeed
um-fc
digg
synaptive
yisouspider
byo-4
feedbin
twingly
tiny
www-mechanize
commafeed
tkbot
tineye-bot
qwantify
exabot
coccoc
blexbot
umbot-ln
queryseekerspider
yandex
yandexbot
yandexmobilebot
yandeximageresizer
yandeximages
sogou spider
sogou web spider
baiduspider
youdaobot
mj12bot
spbot
seznambot
ahrefsbot
linkdex
linkdexbot
semrushbot
semrushbot-sa
cfnetwork
sputnikbot
turnitinbot
proximic
grapeshot
viglink
clickagy
yak
deusu
taboolabot
pinterestbot
psbot
plukkie
getintent
facebot
linkdexbot
vegi
istellabot
leikibot
claritybot
piplbot
feedspotbot
moatbot
yeti
contxbot
seokicks-robot
rogerbot
discordbot
tapatalk
mail.ru_bot
coccocbot
hybridbot
slack-imgproxy
twitterbot
adbeat_bot
kraken
mojeekbot
calculon
mcsbot
avkzarabotok
searchmetricsbot
uptimebot
yacybot
applebot
gigabot
g2reader-bot
seoterritory
ahrefsbot
feedbot
findxbot
wotbox
msnbot
smtbot
ocarinabot
dragonbot
zombiebot
metabot.ru
rssmicro
_zbot
uptimerobot
uipbot
zoominfobot
flamingo_searchengine
bot-pge.chlooe.com
girafabot
multiviewbot
safednsbot
gnowitnewsbot
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
uptimerobot
uipbot
zoominfobot
flamingo_searchengine
bot-pge.chlooe.com
girafabot
multiviewbot
safednsbot
gnowitnewsbot
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
treato-bot
pvblcbot
laserlikebot
trendiction
mediatoolkitbot
charliebot
bubing
superfeedr
focusbot
sottopop
maxpointcrawler
nutch-1.10
wonderbot
zarabotok--doma
cliqzbot
dotbot
exabot
moatbot
tapatalk
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
Warnings
- 1 invalid line.
Comments