Log In Sign Up

biographya.com
robots.txt

Robots Exclusion Standard data for biographya.com

Resource Scan

Scan Details

Site Domain biographya.com
Base Domain biographya.com
Scan Status Ok
Last Scan2026-01-22T06:58:30+00:00
Next Scan 2026-01-29T06:58:30+00:00

Last Scan

Scanned2026-01-22T06:58:30+00:00
URL https://biographya.com/robots.txt
Domain IPs 104.21.65.235, 172.67.194.68, 2606:4700:3033::ac43:c244, 2606:4700:3037::6815:41eb
Response IP 104.21.65.235
Found Yes
Hash a18191f83476950337b0124a50746dd3714f1cce7550ffe6ba60c4c061154a7e
SimHash 792b595127f2

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /adblocker
Disallow /?s=
Disallow /search/
Disallow /readme.html
Disallow /wp-content/cache/
Disallow /wp-content/plugins/

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

huggingface

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

newsnow

Rule Path
Disallow /

news-please

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

googlebot-news

Rule Path
Disallow /sponsored/
Back to top

Other Records

Field Value
sitemap https://biographya.com/sitemap_index.xml
sitemap https://biographya.com/post-sitemap.xml
sitemap https://biographya.com/category-sitemap.xml
sitemap https://biographya.com/post_tag-sitemap.xml
sitemap https://biographya.com/page-sitemap.xml
Back to top

Comments

  • ------------------------------------------
  • robots.txt for https://biographya.com/
  • Last Updated: 25 Oct 2025
  • Optimized for GEO & Security
  • ------------------------------------------
  • ------------------------------------------
  • Core & Functional Rules
  • ------------------------------------------
  • Optional: caching/plugins
  • ------------------------------------------
  • AI & Data Scraper Bots (Blocked)
  • ------------------------------------------
  • ------------------------------------------
  • Search Engine & Social Bots (Allowed)
  • ------------------------------------------
  • Googlebot-News exceptions
  • ------------------------------------------
  • Sitemaps
  • ------------------------------------------
Back to top