Log In Sign Up

kutubee.com
robots.txt

Robots Exclusion Standard data for kutubee.com

Resource Scan

Scan Details

Site Domain kutubee.com
Base Domain kutubee.com
Scan Status Ok
Last Scan2025-11-04T12:55:54+00:00
Next Scan 2025-11-18T12:55:54+00:00

Last Scan

Scanned2025-11-04T12:55:54+00:00
URL https://kutubee.com/robots.txt
Domain IPs 18.198.155.135
Response IP 18.198.155.135
Found Yes
Hash 109accaeb6ea332e6781074206285f190a69c12df780e77e7341a57370a40ad5
SimHash 736e49d262a8

Groups

*

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /wp-json/
Disallow /feed/
Disallow /?feed=
Disallow /?s=
Disallow /search/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /wp-content/cache/
Disallow /cgi-bin/
Disallow /*?s=
Disallow /*/trackback/
Disallow /*/feed/
Disallow /*/comments/
Disallow /author/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /xmlrpc.php
Disallow /.git/
Disallow /.env
Disallow /.htaccess
Disallow /readme.html
Disallow /license.txt
Disallow /*?p=
Disallow /*%26p%3D
Disallow /refer/
Disallow /go/
Disallow /recommend/
Disallow /wp-content/uploads/wpo-plugins-tables-list.json

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

builtwithcrawler

Rule Path
Disallow /

whatwebbot

Rule Path
Disallow /

wappalyzer

Rule Path
Disallow /

whatcms

Rule Path
Disallow /

webmeup

Rule Path
Disallow /

technoratibot

Rule Path
Disallow /

w3c-checklink

Rule Path
Disallow /

w3c-checklink

Rule Path
Disallow /

w3c_validator

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

archive.today

Rule Path
Disallow /

archivebox

Rule Path
Disallow /

wget

Rule Path
Disallow /

curl

Rule Path
Disallow /

httrack

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linqiascrapebot

Rule Path
Disallow /

webcachearchive

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yandexvideo

Rule Path
Disallow /

yandexmedia

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sogou spider2

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

tkspider

Rule Path
Disallow /

headlesschrome

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

piplbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://kutubee.com/sitemap_index.xml
Back to top

Comments

  • WordPress robots.txt - Enhanced Security Configuration
  • Last updated: January 2025
  • Global Crawl Rate Control
  • Sitemaps
  • Default Behavior for Good Bots
  • Block AI and ML Training Bots
  • Block SEO Tools & Analytics Bots
  • Block Site Analysis & Technology Detection Tools
  • Block Web Archive & Caching Tools
  • Block International Search Engines
  • Block Monitoring & Scraping Tools
Back to top

Warnings

  • 1 invalid line.
Back to top