ker.com
robots.txt

Robots Exclusion Standard data for ker.com

Resource Scan

Scan Details

Site Domain ker.com
Base Domain ker.com
Scan Status Ok
Last Scan2026-02-12T16:59:29+00:00
Next Scan 2026-03-14T16:59:29+00:00

Last Scan

Scanned2026-02-12T16:59:29+00:00
URL https://ker.com/robots.txt
Domain IPs 104.26.2.239, 104.26.3.239, 172.67.71.204, 2606:4700:20::681a:2ef, 2606:4700:20::681a:3ef, 2606:4700:20::ac43:47cc
Response IP 104.26.2.239
Found Yes
Hash c373b01b6319bee3e5d353c6f53dac295dbf20c9ba4f09d0fd0eaf1cdfe44986
SimHash 6af63ac082fc

Groups

*

Rule Path
Disallow /wp-admin
Disallow /wp-login.php
Disallow /wp-includes/*php
Disallow /wp-content/plugins
Disallow /wp-content/cache
Disallow */comment*
Disallow /partner/*
Disallow /openx/*
Disallow /sites/*
Disallow /debug-focus-articles/

Other Records

Field Value
crawl-delay 5

googlebot-image

Rule Path
Disallow /images
Disallow /wp-content/uploads

msnbot-media

Rule Path
Disallow /

psbot

Rule Path
Disallow /

yahoo-mmcrawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot/2~bl

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

baiduspider/2.0;+http://www.baidu.com/search/spider.html

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

mozilla/5.0(compatible; baiduspider/2.0;+http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandex.com

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

Comments

  • See htaccess instead HB
  • Disallow: *.pdf
  • Baiduspider
  • Yandex
  • Block BlexBot
  • Block MegaIndex.ru
  • Block Ahrefs
  • Block MJ12bot as it is just noise

Warnings

  • 6 invalid lines.