Log In Sign Up

chelsea-news.net
robots.txt

Robots Exclusion Standard data for chelsea-news.net

Resource Scan

Scan Details

Site Domain chelsea-news.net
Base Domain chelsea-news.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2024-04-10T04:03:24+00:00
Next Scan 2024-07-09T04:03:24+00:00

Last Successful Scan

Scanned2022-10-02T10:01:39+00:00
URL https://chelsea-news.net/robots.txt
Redirect http://www.kicknews.com/robots.txt
Redirect Domain www.kicknews.com
Redirect Base kicknews.com
Response IP 104.21.2.161, 172.67.129.96
Found Yes
Hash 7014bd55936923ad0b699cda87a67fb3c1b643ee39f6b5f18dff2fd4be80237a
SimHash ee97d336c19f

Groups

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /account/login/
Disallow /account/join/
Disallow /account/
Disallow /news/search
Disallow /news/search?*
Disallow /kick/
Disallow /link/
Allow /story/
Allow /news/
Allow /news?page*
Allow /news/latest
Allow /news/latest?page=*
Allow /news/popular
Allow /news/popular?page=*
Allow /sitemapxml?date=*
Allow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

spinn3r

Rule Path
Disallow /

yahoo pipes 1.0

Rule Path
Disallow /

moget
ichiro

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

kscrawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemapxml
Back to top

Comments

  • note that manual allows override the broader disallows specified earlier
  • for "/*?", refer to http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
  • this disallows indexing of any URL with a querystring
  • beware, the sections below WILL NOT INHERIT from the above!
  • http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
  • disallow adsense bot, as we no longer do adsense.
  • User-agent: Mediapartners-Google
  • Disallow: /
  • Yahoo bot is evil.
  • Disallow: /
  • Spinn3r is also evil.
  • Yahoo Pipes is for feeds not web pages.
  • Asian search engines we don't need to be indexed by
  • User-agent: Yandex
  • Disallow: /
  • Disallow: /
  • KSCrawler - we don't need help from you
  • MJ12bot - we don't need help from you
  • this technically isn't valid, since for some godforsaken reason
  • sitemap paths must be ABSOLUTE and not relative.
Back to top