Log In Sign Up

ksitetv.com
robots.txt

Robots Exclusion Standard data for ksitetv.com

Resource Scan

Scan Details

Site Domain ksitetv.com
Base Domain ksitetv.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-08T01:55:45+00:00
Next Scan 2025-01-06T01:55:45+00:00

Last Successful Scan

Scanned2023-09-13T14:05:35+00:00
URL https://ksitetv.com/robots.txt
Redirect https://www.ksitetv.com/robots.txt
Redirect Domain www.ksitetv.com
Redirect Base ksitetv.com
Domain IPs 104.26.8.218, 104.26.9.218, 172.67.69.135, 2606:4700:20::681a:8da, 2606:4700:20::681a:9da, 2606:4700:20::ac43:4587
Redirect IPs 104.26.8.218, 104.26.9.218, 172.67.69.135, 2606:4700:20::681a:8da, 2606:4700:20::681a:9da, 2606:4700:20::ac43:4587
Response IP 104.26.8.218
Found Yes
Hash 520ed0e32d3751d3b943adc73700de56554ed15503e5522326e28a1ba338e326
SimHash 68945fa1e555

Groups

mauibot

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

yahoo! slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

magpie-crawler

Rule Path
Disallow /

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 20

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Comments

  • ****************************************************************************
  • robots.txt
  • : Robots, spiders, and search engines use this file to detmine which
  • content they should *not* crawl while indexing your website.
  • : This system is called "The Robots Exclusion Standard."
  • : It is strongly encouraged to use a robots.txt validator to check
  • for valid syntax before any robots read it!
  • Examples:
  • Instruct all robots to stay out of the admin area.
  • : User-agent: *
  • : Disallow: /admin/
  • Restrict Google and MSN from indexing your images.
  • : User-agent: Googlebot
  • : Disallow: /images/
  • : User-agent: MSNBot
  • : Disallow: /images/
  • ****************************************************************************
  • Biduspider
  • Yandex
Back to top