webkla.com
robots.txt

Robots Exclusion Standard data for webkla.com

Resource Scan

Scan Details

Site Domain webkla.com
Base Domain webkla.com
Scan Status Ok
Last Scan2025-05-16T02:13:34+00:00
Next Scan 2025-06-15T02:13:34+00:00

Last Scan

Scanned2025-05-16T02:13:34+00:00
URL https://webkla.com/robots.txt
Domain IPs 2a02:4780:15:1e3d:1b2:e46b:1427:e8ee, 84.32.84.107
Response IP 77.37.66.192
Found Yes
Hash 1e439e6c18ad37ade1328e0e9425647a114670fa230ae2bea264b11749d74566
SimHash 48bf5d03e4a2

Groups

*

Rule Path
Disallow */feed/
Disallow /feed/$
Disallow /*?
Disallow /*?=
Disallow /search/
Disallow *?s=*
Disallow /tag/
Disallow /page/
Disallow /page/*/?s=
Disallow /*?page=
Disallow /amp/
Disallow /*?amp=
Disallow /rss/
Disallow /category/feed/
Disallow /tag/feed/
Disallow /*?feed=
Disallow /wp-json/
Disallow /?rest_route=
Disallow /*.htm$
Disallow /*.html$
Disallow /wp-login.php?*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Allow /*.webp$
Allow /*.svg$

googlebot

Rule Path
Disallow */feed/
Disallow /feed/$
Disallow /*?
Disallow /*?=
Disallow /search/
Disallow *?s=*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

googlebot-mobile

Rule Path
Disallow */feed/
Disallow /feed/$
Disallow /*?
Disallow /*?=
Disallow /search/
Disallow *?s=*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

bingbot

Rule Path
Disallow */feed/
Disallow /feed/$
Disallow /*?
Disallow /*?=
Disallow /search/
Disallow *?s=*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

oai searchbot

Rule Path
Disallow */feed/
Disallow /feed/$
Disallow /*?
Disallow /*?=
Disallow /search/
Disallow *?s=*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

gptbot

Rule Path
Disallow */feed/
Disallow /feed/$
Disallow /*?
Disallow /*?=
Disallow /search/
Disallow *?s=*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

chatgpt-user

Rule Path
Disallow */feed/
Disallow /feed/$
Disallow /*?
Disallow /*?=
Disallow /search/
Disallow *?s=*
Allow /*.js
Allow /*.css
Allow /*.jpg
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$

oncrawl

Rule Path
Disallow /

alexabot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

crawlbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

facebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

linkedinbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

pinterestbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

instagram

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.webkla.com/sitemap.xml
sitemap https://www.webkla.com/post-sitemap.xml

Comments

  • START BLOCK
  • ---------------------------
  • Allow social media crawlers