Log In Sign Up

websyte.com.au
robots.txt

Robots Exclusion Standard data for websyte.com.au

Resource Scan

Scan Details

Site Domain websyte.com.au
Base Domain websyte.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-03-23T06:30:37+00:00
Next Scan 2024-06-21T06:30:37+00:00

Last Successful Scan

Scanned2023-12-01T06:28:51+00:00
URL http://websyte.com.au/robots.txt
Domain IPs 76.12.71.174
Response IP 76.12.71.174
Found Yes
Hash d1c60ee533a00f1b41db32ac098a66fa8431c7271bf7e3f64c93a7571656b497
SimHash 316c500c42b2

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

nextgensearchbot+1+(for+information+visit+http://www.zoominfo.com/about/misc/nextgensearchbot.aspx)

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

http://www.zoominfo.com

Rule Path
Disallow /

nextgensearchbot.aspx

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru/

Rule Path
Disallow /

zoominformation+bot

Rule Path
Disallow /

mozilla/5.0+(compatible;+megaindex.ru/2.0;++http://megaindex.com/crawler)

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru_bot/robots/2.0

Rule Path
Disallow /

go.mail.ru/

Rule Path
Disallow /

mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/robots/2.0;++http://go.mail.ru/help/robots)

Rule Path
Disallow /

mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/2.0;++http://go.mail.ru/help/robots)

Rule Path
Disallow /

awooo

Rule Path
Disallow /

compatible;baiduspider/2.0;++http://www.baidu.com/search/spider.html

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

www.baidu.com/search/spider.html

Rule Path
Disallow /

curl/7.29.0

Rule Path
Disallow /

curl/7.43.0

Rule Path
Disallow /

crawler/js-crawler

Rule Path
Disallow /

js-crawler

Rule Path
Disallow /

www.dataparksearch.org/bot

Rule Path
Disallow /

dataparksearch+(+http://www.dataparksearch.org/bot)+@+http://inet-sochi.ru/gday

Rule Path
Disallow /

dataparksearch

Rule Path
Disallow /

em7/contentverification/5.1+http://em7.sciencelogic.com

Rule Path
Disallow /

contentverification/5.1

Rule Path
Disallow /

http://em7.sciencelogic.com

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

internetseer.com

Rule Path
Disallow /

java/1.6.0_21

Rule Path
Disallow /

ltx71+-+(http://ltx71.com/)

Rule Path
Disallow /

java/1.4.1_04

Rule Path
Disallow /

java/1.6.0_04

Rule Path
Disallow /

http://ltx71.com/

Rule Path
Disallow /

luis@cybo.com

Rule Path
Disallow /

lwp::simple/6.13+libwww-perl/6.13

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

nutch-1.10

Rule Path
Disallow /

maxpoint.crawler+at+maxpointinteractive+dot+com

Rule Path
Disallow /

maxpoint.crawler

Rule Path
Disallow /

maxpoint.crawler@maxpointinteractive.com

Rule Path
Disallow /

mozilla/5.0+(compatible;+ahrefsbot/5.1;++http://ahrefs.com/robot/)

Rule Path
Disallow /

mozilla/5.0+(compatible;+cliqzbot/1.0++http://cliqz.com/company/cliqzbot)

Rule Path
Disallow /

cliqzbot/1.0

Rule Path
Disallow /

http://cliqz.com

Rule Path
Disallow /

ahrefsbot/5.1

Rule Path
Disallow /

http://ahrefs.com/robot/

Rule Path
Disallow /

mozilla/5.0+(compatible;+archive.org_bot;+wayback+machine+live+record;++http://archive.org/details/archive.org_bot)

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

http://archive.org/details/archive.org_bot

Rule Path
Disallow /

mozilla/5.0+(compatible;+archive.org_bot++http://www.archive.org/details/archive.org_bot)

Rule Path
Disallow /

mozilla/5.0+(compatible;+baiduspider/2.0;++http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

http://www.baidu.com/search/spider.html

Rule Path
Disallow /

www.baidu.com

Rule Path
Disallow /

mozilla/5.0+(compatible;+dataprovider/6.92;++https://www.dataprovider.com/)

Rule Path
Disallow /

dataprovider/6.92

Rule Path
Disallow /

https://www.dataprovider.com/

Rule Path
Disallow /

mozilla/5.0+(compatible;+deusu/5.0.2;++https://deusu.de/robot.html)

Rule Path
Disallow /

deusu/5.0.2

Rule Path
Disallow /

https://deusu.de/robot.html

Rule Path
Disallow /

mozilla/5.0+(compatible;+domainappender+/1.0;++http://www.profound.net/domainappender)

Rule Path
Disallow /

mozilla/5.0+(compatible;+linkdexbot/2.2;++http://www.linkdex.com/bots/)

Rule Path
Disallow /

linkdexbot/2.2

Rule Path
Disallow /

http://www.linkdex.com/bots/

Rule Path
Disallow /

domainappender+/1.0

Rule Path
Disallow /

http://www.profound.net/domainappender

Rule Path
Disallow /

mozilla/5.0+(compatible;+dotbot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

http://www.opensiteexplorer.org/dotbot

Rule Path
Disallow /

mozilla/5.0+(compatible;+duckduckgo-favicons-bot/1.0;++http://duckduckgo.com)

Rule Path
Disallow /

duckduckgo

Rule Path
Disallow /

http://duckduckgo.com

Rule Path
Disallow /

mozilla/5.0+(compatible;+genieo/1.0+http://www.genieo.com/webfilter.html)

Rule Path
Disallow /

genieo/1.0

Rule Path
Disallow /

www.genieo.com/webfilter.html

Rule Path
Disallow /

mozilla/5.0+(compatible;+grapeshotcrawler/2.0;++http://www.grapeshot.co.uk/crawler.php)

Rule Path
Disallow /

grapeshotcrawler/2.0

Rule Path
Disallow /

http://www.grapeshot.co.uk/crawler.php

Rule Path
Disallow /

mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/2.0;++http://go.mail.ru/help/robots)

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

http://go.mail.ru/help/robots

Rule Path
Disallow /

mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/robots/2.0;++http://go.mail.ru/help/robots)

Rule Path
Disallow /

mail.ru_bot/robots/2.0

Rule Path
Disallow /

mozilla/5.0+(compatible;+megaindex.ru/2.0;++http://megaindex.com/crawler)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

http://megaindex.com/crawler

Rule Path
Disallow /

mozilla/5.0+(compatible;+mojeekbot/0.6;++https://www.mojeek.com/bot.html)

Rule Path
Disallow /

mojeekbot/0.6

Rule Path
Disallow /

https://www.mojeek.com/bot.html

Rule Path
Disallow /

mozilla/5.0+(compatible;+msie+10.0;+windows+nt+6.1;+trident/6.0)+linkcheck+by+siteimprove.com

Rule Path
Disallow /

linkcheck+by+siteimprove.com

Rule Path
Disallow /

siteimprove.com

Rule Path
Disallow /

mozilla/5.0+(compatible;+msie+9.0;+windows+nt+6.1;+trident/5.0);+360spider(compatible;+haosouspider;+http://www.haosou.com/help/help_3_2.html)

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

http://www.haosou.com

Rule Path
Disallow /

mozilla/5.0+(compatible;+orangebot/2.0;+support.orangebot@orange.com)

Rule Path
Disallow /

orangebot/2.0

Rule Path
Disallow /

support.orangebot@orange.com

Rule Path
Disallow /

orange.com

Rule Path
Disallow /

mozilla/5.0+(compatible;+spbot/5.0.2;++http://openlinkprofiler.org/bot+)

Rule Path
Disallow /

spbot/5.0.2

Rule Path
Disallow /

http://openlinkprofiler.org/bot

Rule Path
Disallow /

mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)

Rule Path
Disallow /

proximic

Rule Path
Disallow /

www.proximic.com/info/spider.php

Rule Path
Disallow /

www.proximic.com

Rule Path
Disallow /

mozilla/5.0+(compatible;+surdotlybot/1.0;++http://sur.ly/bot.html)

Rule Path
Disallow /

surdotlybot/1.0

Rule Path
Disallow /

http://sur.ly/bot.htm

Rule Path
Disallow /

mozilla/5.0+(compatible;+yandexbot/3.0;++http://yandex.com/bots)

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

http://yandex.com/bots

Rule Path
Disallow /

mozilla/5.0+(compatible;+yoozbot-2.2;+http://yooz.ir;+info@yooz.ir)

Rule Path
Disallow /

yoozbot-2.2

Rule Path
Disallow /

http://yooz.ir;+info@yooz.ir

Rule Path
Disallow /

yooz.ir

Rule Path
Disallow /

mozilla/5.0+(iphone;+cpu+iphone+os+6_0+like+mac+os+x)+applewebkit/536.26+(khtml,+like+gecko)+++++++++++++++++version/6.0+mobile/10a5376e+safari/8536.25+(compatible;+smtbot/1.0;++http://www.similartech.com/smtbot)

Rule Path
Disallow /

smtbot/1.0

Rule Path
Disallow /

http://www.similartech.com/smtbot

Rule Path
Disallow /

mozilla/5.0+(windows+nt+6.1)+(compatible;+smtbot/1.0;++http://www.similartech.com/smtbot)

Rule Path
Disallow /

mozilla/5.0+(windows+nt+6.1)+applewebkit/537.1+(khtml,+like+gecko)+chrome/21.0.1180.89+safari/537.1;+360spider(compatible;+haosouspider;+http://www.haosou.com/help/help_3_2.html)

Rule Path
Disallow /

nrlcorpusbuilder/nutch-1.9

Rule Path
Disallow /

nutch-1.9

Rule Path
Disallow /

nutch-1.4/nutch-1.4

Rule Path
Disallow /

nutch-1.4

Rule Path
Disallow /

wotbox/2.01+(+http://www.wotbox.com/bot/)

Rule Path
Disallow /

wotbox/2.01

Rule Path
Disallow /

http://www.wotbox.com/bot/

Rule Path
Disallow /

safesearch+microdata+crawler+(https://safesearch.avira.com,+safesearch-abuse@avira.com)

Rule Path
Disallow /

safesearch.avira.com

Rule Path
Disallow /

safesearch+microdata+crawler

Rule Path
Disallow /

screaming+frog+seo+spider/5.1

Rule Path
Disallow /

spider/5.1

Rule Path
Disallow /

screaming+frog+seo

Rule Path
Disallow /

slack-imgproxy+1.136+(+https://api.slack.com/robots)

Rule Path
Disallow /

slack-imgproxy+1.136

Rule Path
Disallow /

https://api.slack.com/robots

Rule Path
Disallow /

sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou+web+spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

sogou+web+spider/4.0

Rule Path
Disallow /

www.sogou.com

Rule Path
Disallow /

spiderman

Rule Path
Disallow /

uxcrawlerbot

Rule Path
Disallow /

xenu+link+sleuth/1.3.8

Rule Path
Disallow /

xenu+link

Rule Path
Disallow /

sleuth/1.3.8

Rule Path
Disallow /

xsocks+v0.1

Rule Path
Disallow /

hetzner-rz-nbg-net

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

depspid

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

tutorgigbot

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

e-societyrobot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

surveybot_ignoreip

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

baidumobaider

Rule Path
Disallow /

baiduimagespider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

semalt.com

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mozilla/5.0+(compatible;+linux+x86_64;+mail.ru_bot/img/2.0;++http://go.mail.ru/help/robots)

Rule Path
Disallow /

mail.ru_bot/img/2.0

Rule Path
Disallow /

http://go.mail.ru/help/robots

Rule Path
Disallow /

curl/7.19.7+(x86_64-unknown-linux-gnu)+libcurl/7.19.7+nss/3.12.7.0+zlib/1.2.3+libidn/1.18+libssh2/1.2.2

Rule Path
Disallow /

libssh2/1.2.2

Rule Path
Disallow /

libidn/1.18

Rule Path
Disallow /

zlib/1.2.3

Rule Path
Disallow /

libcurl/7.19.7

Rule Path
Disallow /

mozilla/5.0+(macintosh;+intel+mac+os+x+10_10_1)+applewebkit/600.2.5+(khtml,+like+gecko)+version/8.0.2+safari/600.2.5+(applebot/0.1;++http://www.apple.com/go/applebot)

Rule Path
Disallow /

applebot/0.1

Rule Path
Disallow /

www.apple.com/go/applebot

Rule Path
Disallow /

safednsbot+(https://www.safedns.com/searchbot)

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

safedns.com/searchbot

Rule Path
Disallow /

mozilla/5.0+(compatible;+mj12bot/v1.4.5;+http://www.majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.4.5

Rule Path
Disallow /

majestic12.co.uk/bot.php

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

*

Rule Path
Disallow /cgadmin/
Disallow /cgrsadmin/
Disallow /siteadmin/
Disallow /syteadmin/
Back to top

Comments

  • robots.txt file for http://www.communityguide.com.au
  • http://moz.com/products
  • http://openlinkprofiler.org/bot
  • http://www.opensiteexplorer.org/dotbot
  • http://moz.com/researchtools/ose/dotbot
  • https://megaindex.com/crawler
  • http://www.moz.com/dp/rogerbot
  • http://http://www.semrush.com
Back to top

Warnings

  • 6 invalid lines.
Back to top