communityguide.com.au
robots.txt
Robots Exclusion Standard data for communityguide.com.au
Resource Scan
Scan Details
Site Domain | communityguide.com.au |
Base Domain | communityguide.com.au |
Scan Status | Ok |
Last Scan | 2024-09-21T06:39:33+00:00 |
Next Scan | 2024-10-21T06:39:33+00:00 |
Last Scan
Scanned | 2024-09-21T06:39:33+00:00 |
URL | http://communityguide.com.au/robots.txt |
Domain IPs | 76.12.71.179 |
Response IP | 76.12.71.179 |
Found | Yes |
Hash | 2839abb148550a9f9178ca027167c9116478c69350592a2d5d874cdd557ae2e9 |
SimHash | 316c500c42b2 |
Groups
*
No rules defined. All paths allowed.
Other Records
Field | Value |
---|---|
crawl-delay | 30 |
nextgensearchbot+1+(for+information+visit+http://www.zoominfo.com/about/misc/nextgensearchbot.aspx)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/robots/2.0;++http://go.mail.ru/help/robots)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/2.0;++http://go.mail.ru/help/robots)
Rule | Path |
---|---|
Disallow | / |
dataparksearch+(+http://www.dataparksearch.org/bot)+@+http://inet-sochi.ru/gday
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+archive.org_bot;+wayback+machine+live+record;++http://archive.org/details/archive.org_bot)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+archive.org_bot++http://www.archive.org/details/archive.org_bot)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+baiduspider/2.0;++http://www.baidu.com/search/spider.html)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+domainappender+/1.0;++http://www.profound.net/domainappender)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+dotbot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+grapeshotcrawler/2.0;++http://www.grapeshot.co.uk/crawler.php)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/2.0;++http://go.mail.ru/help/robots)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/robots/2.0;++http://go.mail.ru/help/robots)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+msie+10.0;+windows+nt+6.1;+trident/6.0)+linkcheck+by+siteimprove.com
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+msie+9.0;+windows+nt+6.1;+trident/5.0);+360spider(compatible;+haosouspider;+http://www.haosou.com/help/help_3_2.html)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(iphone;+cpu+iphone+os+6_0+like+mac+os+x)+applewebkit/536.26+(khtml,+like+gecko)+++++++++++++++++version/6.0+mobile/10a5376e+safari/8536.25+(compatible;+smtbot/1.0;++http://www.similartech.com/smtbot)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(windows+nt+6.1)+(compatible;+smtbot/1.0;++http://www.similartech.com/smtbot)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(windows+nt+6.1)+applewebkit/537.1+(khtml,+like+gecko)+chrome/21.0.1180.89+safari/537.1;+360spider(compatible;+haosouspider;+http://www.haosou.com/help/help_3_2.html)
Rule | Path |
---|---|
Disallow | / |
safesearch+microdata+crawler+(https://safesearch.avira.com,+safesearch-abuse@avira.com)
Rule | Path |
---|---|
Disallow | / |
sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm
Product | Comment |
---|---|
sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm | 07) |
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/img/2.0;++http://go.mail.ru/help/robots)
Rule | Path |
---|---|
Disallow | / |
curl/7.19.7+(x86_64-unknown-linux-gnu)+libcurl/7.19.7+nss/3.12.7.0+zlib/1.2.3+libidn/1.18+libssh2/1.2.2
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(macintosh;+intel+mac+os+x+10_10_1)+applewebkit/600.2.5+(khtml,+like+gecko)+version/8.0.2+safari/600.2.5+(applebot/0.1;++http://www.apple.com/go/applebot)
Rule | Path |
---|---|
Disallow | / |
mozilla/5.0+(compatible;+mj12bot/v1.4.5;+http://www.majestic12.co.uk/bot.php?+)
Rule | Path |
---|---|
Disallow | / |
*
Rule | Path |
---|---|
Disallow | /cgadmin/ |
Disallow | /cgrsadmin/ |
Disallow | /siteadmin/ |
Disallow | /syteadmin/ |
Warnings
- 6 invalid lines.
Comments