Log In Sign Up

petico.my
robots.txt

Robots Exclusion Standard data for petico.my

Resource Scan

Scan Details

Site Domain petico.my
Base Domain petico.my
Scan Status Ok
Last Scan2026-02-07T12:19:52+00:00
Next Scan 2026-03-09T12:19:52+00:00

Last Scan

Scanned2026-02-07T12:19:52+00:00
URL https://petico.my/robots.txt
Domain IPs 104.26.2.126, 104.26.3.126, 172.67.72.101, 2606:4700:20::681a:27e, 2606:4700:20::681a:37e, 2606:4700:20::ac43:4865
Response IP 172.67.72.101
Found Yes
Hash 02ea63e0b4ddc792bbd32ff091d60901d298b68ae4b5a54de84b09944cd3f410
SimHash 0804c992e731

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

bingbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

meta-external-agent

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /*.php$
Disallow /*.json$
Disallow /*.zip$
Disallow /classes/
Disallow /config/
Disallow /ignore/
Disallow /logs/
Disallow /payment_session/
Disallow /session_storage/
Disallow /pages/
Back to top

Other Records

Field Value
sitemap https://petico.my/sitemap.xml
Back to top

Comments

  • NOTICE: By accessing this website (petico.my), you agree to abide by the following content policy.
  • (a) If content-signal = yes, you may collect content for the corresponding purpose.
  • (b) If content-signal = no, you may not collect content for the corresponding purpose.
  • (c) If no content-signal is provided, permission is neither granted nor restricted.
  • Content Signal Definitions:
  • search = building a search index and serving results
  • ai-input = using content for real-time AI queries or summaries
  • ai-train = training or fine-tuning AI models
  • Any restrictions expressed via content signals are reserved rights under EU Directive 2019/790.
  • =====================================================
  • BEGIN: Content & AI Policy
  • =====================================================
  • =====================================================
  • =====================================================
  • BEGIN: Approved Search Engine Crawlers
  • =====================================================
  • =====================================================
  • =====================================================
  • BEGIN: AI & Data Bot Management
  • =====================================================
  • Allow GPTBot for live AI query referencing (not for dataset training)
  • Block known AI scrapers and data-collection bots
  • =====================================================
  • =====================================================
  • BEGIN: General Crawl Rules
  • =====================================================
  • Block sensitive or backend file types
  • Block internal or system directories (adjust if paths differ on your server)
  • Sitemap
  • =====================================================
Back to top

Warnings

  • `content-signal` is not a known field.
Back to top