callaghan-finepaintings.com
robots.txt

Robots Exclusion Standard data for callaghan-finepaintings.com

Resource Scan

Scan Details

Site Domain callaghan-finepaintings.com
Base Domain callaghan-finepaintings.com
Scan Status Ok
Last Scan2025-09-27T11:59:11+00:00
Next Scan 2025-10-27T11:59:11+00:00

Last Scan

Scanned2025-09-27T11:59:11+00:00
URL https://callaghan-finepaintings.com/robots.txt
Domain IPs 104.26.2.165, 104.26.3.165, 172.67.68.93, 2606:4700:20::681a:2a5, 2606:4700:20::681a:3a5, 2606:4700:20::ac43:445d
Response IP 104.26.2.165
Found Yes
Hash 27ad39e1b14ae5b90e3c45a7159bf3fe366dc95f866e2987125b71862a9a5e68
SimHash 7396d8c1c0c6

Groups

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /wishlist/
Disallow /admin/
Disallow /catalogsearch/
Disallow /review/product/
Disallow /sendfriend/
Disallow /enable-cookies/
Disallow /LICENSE.txt
Disallow /LICENSE.html
Disallow /directory/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /tag/
Disallow /review/

ai2bot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

amazon-kendra

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

brightbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

duckassistbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

iaskspider/2.0

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

linerbot

Rule Path
Disallow /

machinelearningforpeacebot

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

nicecrawler

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity-user

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

qualifiedbot

Rule Path
Disallow /

scoop.it

Rule Path
Disallow /

seekr

Rule Path
Disallow /

semrushbot-ocob

Rule Path
Disallow /

sidetrade indexer bot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /

youbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.callaghan-finepaintings.com/pub/media/sitemap/sitemap.xml

Comments

  • Crawler Setup
  • Disallow: /skin/
  • Disallow: /js/
  • Disallow: /pub/
  • XML Sitemap
  • Cloudflare Managed Robots.txt to block AI related bots.