Log In Sign Up

deskdemon.com
robots.txt

Robots Exclusion Standard data for deskdemon.com

Resource Scan

Scan Details

Site Domain deskdemon.com
Base Domain deskdemon.com
Scan Status Ok
Last Scan2024-09-30T21:12:02+00:00
Next Scan 2024-10-30T21:12:02+00:00

Last Scan

Scanned2024-09-30T21:12:02+00:00
URL https://deskdemon.com/robots.txt
Domain IPs 91.209.244.86
Response IP 91.209.244.86
Found Yes
Hash 1f38cc3489e2afcddd43ae0e13cb985ee8771b192593c7c930ad9857002d6bf4
SimHash 9435a12088c3

Groups

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; .net clr 1.1.4322; .net clr 2.0.50727; .net clr 3.0.04506.30; media center pc 6.0; ms-rtc lm 8)

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; slcc2; .net clr 1.1.4322; .net clr 2.0.50727; .net clr 3.0.04506.648; media center pc 6.0)

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

ezooms/1.0; ezooms.bot@gmail.com

Rule Path
Disallow /

hailoobot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yeti

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /templates
Disallow /wwwthreads
Disallow /upb
Disallow /upbpnp
Disallow /mail-old
Disallow /staff
Disallow /who
Disallow /phpMyAdmin
Disallow /my
Disallow /what
Disallow /us
Disallow /raj
Disallow /pvl
Disallow /stats
Disallow /drg/upb
Disallow /sales/admin
Disallow /venues/admin
Disallow /venues/sales
Disallow /training
Disallow /lifestyle/admin
Disallow /basma
Disallow /mek
Disallow /btshow/admin
Disallow /edesk/admin
Disallow /dyk/admin
Disallow /lifestyle/admin
Disallow /sale
Disallow /virtual_assistants/admin
Disallow /us
Disallow /nl
Disallow /jobs
Disallow /ddb
Disallow /duk
Disallow /fkb
Disallow /iqps
Disallow /shefsec
Disallow /jane
Disallow /susan
Disallow /download
Disallow /adserv
Disallow /adserv2
Disallow /clickscounter
Disallow /crm
Disallow /poll
Disallow /ck
Disallow /dfp
Disallow /pages/uk/edesk/details
Disallow /pages/uk/templates/search
Disallow /pages/uk/venues/search
Disallow /pages/uk/jobdesk/result
Disallow /pages/us/venues/search
Disallow /pages/us/venues/details
Disallow /pages/uk/venues/details
Disallow /dnet/twitter_timeline_public.php
Disallow /dnet///twitter_timeline_public.php
Disallow /dnet////twitter_timeline_public.php
Disallow /dnet/////twitter_timeline_public.php
Disallow /dnet//////twitter_timeline_public.php
Disallow /pages/uk/jobdesk/jobseekers
Disallow /pages/btu/index
Disallow /community/forums/showflat.php
Disallow /dnet//browse_listings.php
Disallow /dnet///browse_listings.php
Disallow /dnet////browse_listings.php
Disallow /dnet/////browse_listings.php
Disallow /dnet//////browse_listings.php
Disallow /dnet/profile.php
Disallow /dnet//profile.php
Disallow /dnet///profile.php
Disallow /dnet////profile.php
Disallow /dnet/////profile.php
Disallow /dnet//////profile.php
Disallow /dnet///////profile.php
Disallow /dnet//link.php
Disallow /dnet///link.php
Disallow /dnet////link.php
Disallow /dnet/////link.php
Disallow /dnet//////link.php
Disallow /dnet//browse_links.php
Disallow /dnet///browse_links.php
Disallow /dnet////browse_links.php
Disallow /dnet/////browse_links.php
Disallow /dnet//////browse_links.php
Disallow /dnet//browse_vids.php
Disallow /dnet///browse_vids.php
Disallow /dnet////browse_vids.php
Disallow /dnet/////browse_vids.php
Disallow /dnet//////browse_vids.php
Disallow /dnet//vid.php
Disallow /dnet///vid.php
Disallow /dnet////vid.php
Disallow /dnet/////vid.php
Disallow /dnet//////vid.php
Disallow /dnet//browse_businesses.php
Disallow /dnet///browse_businesses.php
Disallow /dnet////browse_businesses.php
Disallow /dnet/////browse_businesses.php
Disallow /dnet//////browse_businesses.php
Disallow /dnet//userpage.php
Disallow /dnet///userpage.php
Disallow /dnet////userpage.php
Disallow /dnet/////userpage.php
Disallow /dnet//////userpage.php
Disallow /dnet//browse_pages.php
Disallow /dnet///browse_pages.php
Disallow /dnet////browse_pages.php
Disallow /dnet/////browse_pages.php
Disallow /dnet//////browse_pages.php
Disallow /dnet//twitter_timeline_public.php

*

Rule Path
Disallow /dnet/index.php?Deskdemon=*&type=rss%3Baction=.xml

Other Records

Field Value
crawl-delay 8
Back to top

Comments

  • This is for every user agent.
  • block Chinese Search Engine Baidu, raj 06/09/2012
  • Disallow stuff we don't want to have coming up in search engines.
  • Kev.
  • Somehow this wrong link crept in :-/ - Kev, 2011/12/27
  • Added 2011/06/08 by Kev after crawlers seemed to mess us up :-/
  • Increased (3>8) 2011/12/27 by Kev after crawlers continued to mess us up :-/
  • User-Agent line added 2019/02/21 because it looks like Bingbot doesn't respect
  • it otherwise.
  • Disallow pages with session ID's in them for /dnet/ that gets search bots into
  • a loop, Kev, 2019/11/16.
Back to top

Warnings

  • 1 invalid line.
Back to top