Log In Sign Up

centerblog.net
robots.txt

Robots Exclusion Standard data for centerblog.net

Resource Scan

Scan Details

Site Domain centerblog.net
Base Domain centerblog.net
Scan Status Ok
Last Scan2024-10-31T02:19:02+00:00
Next Scan 2024-11-07T02:19:02+00:00

Last Scan

Scanned2024-10-31T02:19:02+00:00
URL http://centerblog.net/robots.txt
Redirect http://www.centerblog.net/robots.txt
Redirect Domain www.centerblog.net
Redirect Base centerblog.net
Domain IPs 87.98.133.18
Redirect IPs 178.32.1.148
Response IP 178.32.1.148
Found Yes
Hash f512516a747af9fc5b444cbb6e5e843a092b9ac3c7bd4e1e940d0273e4436c80
SimHash 311c5107ea48

Groups

*

Rule Path
Disallow /gerer/abonnement/inscription/

dataforseobot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

twiceler.

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

httrack 3.0

Rule Path
Disallow /

yandex bot

Rule Path
Disallow /

yandexsomething

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

radian6_default

Rule Path
Disallow /

radian6

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow

dotbot

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

spbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

eccp/1.0 (search@eniro.com)

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image
mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)
mozilla/5.0 (compatible; baiduspider/3.0; +http://www.baidu.com/search/spider.html)
mozilla/5.0 (compatible; baiduspider/4.0; +http://www.baidu.com/search/spider.html)
mozilla/5.0 (compatible; baiduspider/5.0; +http://www.baidu.com/search/spider.html)
baiduspider/2.0
baiduspider/3.0
baiduspider/4.0
baiduspider/5.0

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

gsa-crawler (enterprise; t4-knhh62cdkc2w3; gsa_manage@nikon-sys.co.jp)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot/2.0; +http://go.mail.ru/help/robots

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.3

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

semrushbot-seoab

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /
Back to top

Comments

  • Block Searchmetrics Bot
  • User-agent: SearchmetricsBot
  • Disallow: /
  • Block Eniro
  • Block Baidu
  • Block SoGou
  • Block Youdao
  • Block Nikon JP Crawler
  • Block MegaIndex.ru
Back to top