Log In Sign Up

kolappan.com
robots.txt

Robots Exclusion Standard data for kolappan.com

Resource Scan

Scan Details

Site Domain kolappan.com
Base Domain kolappan.com
Scan Status Ok
Last Scan2025-04-08T11:19:41+00:00
Next Scan 2025-04-22T11:19:41+00:00

Last Scan

Scanned2025-04-08T11:19:41+00:00
URL https://kolappan.com/robots.txt
Domain IPs 104.21.47.115, 172.67.147.124, 2606:4700:3030::ac43:937c, 2606:4700:3035::6815:2f73
Response IP 104.21.47.115
Found Yes
Hash c507c10751fa38eda0ca8b1ff2cd91d17a61ffd1711e371ac07a9f677214fabb
SimHash 715c8352e6b1

Groups

*

Rule Path
Allow /
Disallow /tags/

Other Records

Field Value
crawl-delay 30

amazonbot
amazonadbot
friendlycrawler

Rule Path
Disallow /

google-extended
googlebot-image
googlebot-news
googlebot-video
storebot-google
googleother
googleother-image
googleother-video
adsbot-google
adsbot-google-mobile
mediapartners-google

Rule Path
Disallow /

applebot-extended
gptbot
anthropic-ai
bytespider
ccbot
claude-web
claudebot
cohere-ai
diffbot
facebookbot
imagesiftbot
omigilibot
omigili
perplexitybot
scoop.it

Rule Path
Disallow /

adidxbot
criteobot
grapeshot
proximic
taboolabot

Rule Path
Disallow /

addthis.com
ahrefsbot
awariobot
awariosmartbot
awariorssbot
barkrowler
botify
blexbot
dataforseo
dotbot
embedly
gingercrawler
piplbot
semrushbot
webreaper

Rule Path
Disallow /

baiduspider
baiduspider-image
baiduspider-news
baiduspider-video

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

sogou blog
sogou inst spider
sogou news spider
sogou orion spider
sogou spider2
sogou web spider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://kolappan.com/sitemap.xml
Back to top

Comments

  • Amazon bots
  • Google Bot
  • Ref: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
  • AI & LLMs
  • Ad bots
  • SEO & Analytics
  • chinese search engines
  • baidu.com
  • so.com
  • jike.com / chinaso.com
  • sogou.com
  • soso.com
Back to top

Warnings

  • 4 invalid lines.
Back to top