dslreports.com
robots.txt

Robots Exclusion Standard data for dslreports.com

Resource Scan

Scan Details

Site Domain dslreports.com
Base Domain dslreports.com
Scan Status Ok
Last Scan2024-09-05T12:24:58+00:00
Next Scan 2024-10-05T12:24:58+00:00

Last Scan

Scanned2024-09-05T12:24:58+00:00
URL https://dslreports.com/robots.txt
Redirect http://www.dslreports.com/robots.txt
Redirect Domain www.dslreports.com
Redirect Base dslreports.com
Domain IPs 64.91.255.98
Redirect IPs 64.91.255.98
Response IP 64.91.255.98
Found Yes
Hash 3c98110f77ee5074791afcaffd9f69fbdaa419626b72a383ef710bbaa912d1b3
SimHash 9a581a50b870

Groups

*

Rule Path
Disallow /*?*stheme=*
Disallow /uidpop
Disallow /attribute
Disallow /url
Disallow /front/shutdown.html
Disallow /pics/dimaging/
Disallow /showpic/
Disallow /gmaps/service/
Disallow /gmaps/dslr/
Disallow /who/
Disallow /linequality/nil
Disallow /tp
Disallow /autocomplete/
Disallow /*ajh%3D1
Disallow /postlist/
Disallow /revisions
Disallow /topicpts
Disallow /forums/fav0
Disallow /forums/fav3
Disallow /gs/
Disallow /trackback/
Disallow /whois/
Disallow /tweakr/
Disallow /forgot/
Disallow /login/
Disallow /redir/
Disallow /dsl_diary/
Disallow /chatbox
Disallow /abcdef/
Disallow /mktplace/
Disallow /prequal_watch/
Disallow /prequal/
Disallow /mailshot/
Disallow /useremail/
Disallow /monitored/
Disallow /metashare/
Disallow /speak/
Disallow /allreviews
Disallow /pics/resize
Disallow /t1plus
Disallow /nav/
Disallow /nsearch
Disallow /instant
Disallow /i/
Disallow /ATTENTION/
Disallow /secret/
Disallow /comap/
Disallow /vprof/
Disallow /bug/
Disallow /sitestats/
Disallow /quality/nil
Disallow /uidpop/
Disallow /speedtest/post
Disallow /diag_stest/record
Disallow /*poll%3D*
Disallow /check-login
Disallow /rpxtoken

exabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

arabybot

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

Other Records

Field Value
sitemap http://www.dslreports.com/sitemap.xml

Comments

  • robots.txt for http://www.dslreports.com