Log In Sign Up

thedyrt.com
robots.txt

Robots Exclusion Standard data for thedyrt.com

Resource Scan

Scan Details

Site Domain thedyrt.com
Base Domain thedyrt.com
Scan Status Ok
Last Scan2024-09-28T17:25:31+00:00
Next Scan 2024-10-05T17:25:31+00:00

Last Scan

Scanned2024-09-28T17:25:31+00:00
URL https://thedyrt.com/robots.txt
Domain IPs 199.232.194.132, 199.232.198.132, 2a04:4e42:4c::644, 2a04:4e42:4d::644
Response IP 199.232.194.132
Found Yes
Hash 7050fd5cf3a23edf9bd10d8f1741b04fbbf55829ab60e7078129527dda5a60f0
SimHash a315c252cdb3

Groups

butterfly

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ginxbot

Rule Path
Disallow /

metauri

Rule Path
Disallow /

postrank

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

thingfetcher

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

tagoobot

Rule Path
Disallow /

mxbot

Rule Path
Disallow /

twitturls

Rule Path
Disallow /

radian6

Rule Path
Disallow /

viralheat

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

twingly recon

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

purebot

Rule Path
Disallow /

twitturly

Rule Path
Disallow /

oneriot

Rule Path
Disallow /

feedtrace

Rule Path
Disallow /

fairshare

Rule Path
Disallow /

njuicebot

Rule Path
Disallow /

semiocast

Rule Path
Disallow /

typhoeus

Rule Path
Disallow /

my6sense

Rule Path
Disallow /

twitmatic

Rule Path
Disallow /

blogpulselive

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

tweetedtimes bot

Rule Path
Disallow /

nhn corp

Rule Path
Disallow /

bender

Rule Path
Disallow /

proximic

Rule Path
Disallow /

netseer

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

birubot

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

botw spider

Rule Path
Disallow /

summify

Rule Path
Disallow /

tanner

Rule Path
Disallow /

gosospider

Rule Path
Disallow /

expo9

Rule Path
Disallow /

nutch

Rule Path
Disallow /

crowsnest

Rule Path
Disallow /

docomo

Rule Path
Disallow /

percbotspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

knowaboutbot

Rule Path
Disallow /

atomic_email_hunter

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /join
Disallow /login
Disallow /sign-out
Disallow /oauth-authorize
Disallow /password
Disallow /find-to-review
Disallow /manager
Disallow /dump-stations
Disallow /water-stations
Disallow /list
Disallow /*/edit$
Disallow /*/edit/$
Back to top

Other Records

Field Value
sitemap https://thedyrt.com/sitemaps/sitemap_index.xml.gz
sitemap https://thedyrt.com/magazine/sitemap_index.xml
sitemap https://thedyrt.com/forums/sitemap.xml
Back to top