kr.techbriefly.com
robots.txt

Robots Exclusion Standard data for kr.techbriefly.com

Resource Scan

Scan Details

Site Domain kr.techbriefly.com
Base Domain techbriefly.com
Scan Status Ok
Last Scan2025-10-18T14:56:57+00:00
Next Scan 2025-11-17T14:56:57+00:00

Last Scan

Scanned2025-10-18T14:56:57+00:00
URL https://kr.techbriefly.com/robots.txt
Domain IPs 104.21.4.156, 172.67.132.58, 2606:4700:3034::6815:49c, 2606:4700:3035::ac43:843a
Response IP 172.67.132.58
Found Yes
Hash a83911d9fdf0a8b39e59e4b9b6cb849ab44b5956803a61524df867e2ce4cdf77
SimHash 531f4843e2a0

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /login/
Disallow /private/
Disallow *?s=*
Disallow /search/
Disallow /*.pdf$

Other Records

Field Value
crawl-delay 20

googlebot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

gptbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

claude-user

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

bingbot

Rule Path
Allow /

applebot

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

facebookbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

bytespider

Rule Path
Allow /

petalbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

googleother

Rule Path
Allow /

mistralai-user

Rule Path
Allow /

duckassistbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

Other Records

Field Value
sitemap https://techbriefly.com/sitemap.xml