bytestechnolab.com
robots.txt

Robots Exclusion Standard data for bytestechnolab.com

Resource Scan

Scan Details

Site Domain bytestechnolab.com
Base Domain bytestechnolab.com
Scan Status Ok
Last Scan2026-01-31T10:59:08+00:00
Next Scan 2026-02-07T10:59:08+00:00

Last Scan

Scanned2026-01-31T10:59:08+00:00
URL https://bytestechnolab.com/robots.txt
Domain IPs 104.21.76.228, 172.67.201.237, 2606:4700:3030::6815:4ce4, 2606:4700:3032::ac43:c9ed
Response IP 172.67.201.237
Found Yes
Hash 5b7322d59189e1a4158b5a8b6bc0de5029156aef22bc814ab4bfdf0e0058f4ca
SimHash 7a1302cbe6a2

Groups

*

Rule Path
Allow /
Disallow /wp-content/plugins/
Disallow /wp-admin/
Disallow /demo/
Disallow /taskmanager/
Disallow /wp-content/uploads/wp-import-export-lite/
Disallow /blog/page/
Disallow /blog/category/
Disallow /cdn-cgi/
Disallow /au/?page_id=
Disallow /au/blog/category/

chatgpt-user

Rule Path
Allow /

gptbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

youbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

omgilibot

Rule Path
Allow /

omgili

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

duckassistbot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claudebot

Rule Path
Allow /

google-extended

Rule Path
Allow /

ccbot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

gemini

Rule Path
Allow /

bingbot

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /
Allow /llms.txt
Allow /au/llms.txt

Other Records

Field Value
sitemap https://www.bytestechnolab.com/sitemap_index.xml
sitemap https://www.bytestechnolab.com/uk/sitemap_index.xml
sitemap https://www.bytestechnolab.com/au/sitemap_index.xml
sitemap https://www.bytestechnolab.com/sa/sitemap_index.xml

Comments

  • --- AI/LLM CRAWLER ---
  • ---BOTS RESTRICTIONS---
  • --- LLMS File ---