Log In Sign Up

cikolatasepeti.com
robots.txt

Robots Exclusion Standard data for cikolatasepeti.com

Resource Scan

Scan Details

Site Domain cikolatasepeti.com
Base Domain cikolatasepeti.com
Scan Status Ok
Last Scan2024-10-10T23:52:01+00:00
Next Scan 2024-11-09T23:52:01+00:00

Last Scan

Scanned2024-10-10T23:52:01+00:00
URL https://cikolatasepeti.com/robots.txt
Redirect https://www.cikolatasepeti.com/robots.txt
Redirect Domain www.cikolatasepeti.com
Redirect Base cikolatasepeti.com
Domain IPs 178.157.14.187
Redirect IPs 178.157.14.187
Response IP 178.157.14.187
Found Yes
Hash aa40cd8515e75df00aa8b592af757687a1aec9f425330d96c132dcb4ce883aae
SimHash 529ac8d95479

Groups

baidu

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow
Allow /*

duggmirror

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

birubot

Rule Path
Disallow /

bixolabs

Rule Path
Disallow /

botonparade

Rule Path
Disallow /

discobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

eurobot

Rule Path
Disallow /

unwindfetchor

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

iccrawler - icjobs

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

obot

Rule Path
Disallow /

oneriot

Rule Path
Disallow /

ruby

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

speedy

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

twenga2.com

Rule Path
Disallow /

twenga.com

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.cikolatasepeti.com/sitemap.xml
sitemap https://www.cikolatasepeti.com/image-sitemap.xml
Back to top

Comments

  • Baiduspider (http://www.baidu.com/search/spider.htm)
  • Internet Archiver Wayback Machine
  • BacklinkCrawler (http://www.backlinktest.com/crawler.html)
  • Google AdSense
  • digg mirror
  • Ahrefs.com (http://ahrefs.com/robot/)
  • Birubot [Bot]
  • Bixo Labs (http://bixolabs.com)
  • BotOnParade (www.bots-on-para.de/bot.html)
  • discobot (http://discoveryengine.com/discobot.html)
  • Exabot/3.0 (http://www.exabot.com/go/robot)
  • EUROBOT/1.1 (HTTP://EUROBOT.AYELL.EU)
  • Gnip [Bot] (http://www.gnip.com)
  • Huawei Symantec (http://www.huaweisymantec.com/en/IRL/spider/)
  • iCjobs Stellenangebote Jobs (www.icjobs.de/bot.htm)
  • Majestic-12 : DSearch : MJ12bot (http://www.majestic12.co.uk/bot.php)
  • MojeekBot/0.2 (http://www.mojeek.com/bot.html)
  • NerdByNature (http://www.nerdbynature.net/bot)
  • oBot/2.3.1 (http://filterdb.iss.net/crawler/)
  • OneRiot [Bot] (http://www.oneriot.com)
  • Ruby
  • ScoutJet (http://www.scoutjet.com/)
  • SEOkicks-Robot (http://www.seokicks.de/robot.html)
  • Sosospider (http://help.soso.com/webspider.htm)
  • Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
  • Spinn3r (http://spinn3r.com/robot)
  • TwengaBot (http://www.twenga.com/bot.html)
  • VOILABOT BETA 1.2 (SUPPORT.VOILABOT@ORANGE-FTGROUP.COM)
Back to top

Warnings

  • `host` is not a known field.
Back to top