Log In Sign Up

chrisdixonstudios.com
robots.txt

Robots Exclusion Standard data for chrisdixonstudios.com

Resource Scan

Scan Details

Site Domain chrisdixonstudios.com
Base Domain chrisdixonstudios.com
Scan Status Ok
Last Scan2025-11-17T23:47:18+00:00
Next Scan 2025-12-17T23:47:18+00:00

Last Scan

Scanned2025-11-17T23:47:18+00:00
URL https://chrisdixonstudios.com/robots.txt
Redirect https://www.chrisdixonstudios.com/robots.txt
Redirect Domain www.chrisdixonstudios.com
Redirect Base chrisdixonstudios.com
Domain IPs 209.133.196.165
Redirect IPs 209.133.196.165
Response IP 209.133.196.165
Found Yes
Hash 4ed0acca8761e58b7a24dbb59f34ee81598add3ca66afa74433b9a7b49666909
SimHash 65265a7066f2

Groups

brave-bot
duckduckbot
mojeekbot
qwantify
startpage
searxbot

Product Comment
qwantify Qwant crawler
startpage Startpage proxy
searxbot Searx instances
Rule Path Comment
Allow / -
Allow /cdsgallery/*.html$ -
Allow /artgallery/*.html$ -
Allow /category/*.html$ -
Allow /images/cdsgallery/ -
Allow /images/artgallery/ -
Allow /specials/ For privacy-focused deal listings
Disallow /private/ -
Disallow /customer/ Blocks profile crawling

Other Records

Field Value
crawl-delay 2

qwantify
startpage
searxbot

Rule Path Comment
Allow /api/ Allows product API endpoints if available

Other Records

Field Value Comment
crawl-delay 3 Slower delay for distributed instances

cloudflare-alwaysonline

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

amazonbot
fastly-googlebot

Product Comment
amazonbot AWS CloudFront
fastly-googlebot Fastly CDN
Rule Path
Disallow /staging/
Disallow /dev/
Allow /*.css$
Allow /*.js$

yandex
baiduspider
sogou

Product Comment
yandex Russian crawler
baiduspider Chinese crawler
sogou Chinese crawler
Rule Path
Disallow /

gptbot
claude-web
anthropic-ai
facebookbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

ddos-guard
project-25499
masscan-ng

Product Comment
project-25499 Known malicious botnet
Rule Path Comment
Disallow / -
Disallow /*.php$ Block all PHP except index
Allow /index.php$ -
Disallow /*.sql$ -
Disallow /*.env$ -
Disallow /*.git/ -
Disallow /*.well-known/ Blocks security cert scans
Disallow /cdn-cgi/ Cloudflare exploit path

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

googlebot
bingbot

Rule Path Comment
Allow /*.css$ -
Allow /*.js$ -
Allow /cdsgallery/*.html$ E-commerce product pages
Allow /artgallery/*.html$ E-commerce product pages
Allow /category/*.html$ Product categories
Allow /cdsgallery/ Product images
Allow /images/artgallery/ Product images

Other Records

Field Value
crawl-delay 2

semrushbot
semrushbot-sa
ahrefsbot
mj12bot
dotbot
mauibot
blexbot
extlinksbot
zoominfobot
barkrowler
ccbot

Rule Path
Disallow /
Disallow /admin/
Disallow /includes/
Disallow /tmp/
Disallow /cache/
Disallow /config/
Disallow /logs/
Disallow /private/
Disallow /New_Folder*/
Disallow /configuration.php
Disallow /install.php
Disallow /phpmyadmin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /aCopy_of_gallery/
Disallow /aSubGallery/
Disallow /subimages/
Disallow /subsubimages/
Disallow /backup/
Disallow /blog/
Disallow /cgi-bin/
Disallow /email/
Disallow /gallery/
Disallow /gallery2/
Disallow /giftshop2/
Disallow /ip24u/
Disallow /ip24uazhl-iponu/
Disallow /iponu/
Disallow /LL-DecoyDucks/
Disallow /lo/
Disallow /rate_cgi.php
Disallow /rate/
Disallow /TESTER/
Disallow /apoliticallycorrect/
Disallow /ZZ*/
Allow /public_images/*.jpg$
Allow /public_images/*.png$
Allow /public_images/*.gif$
Allow /fonts/
Back to top

Other Records

Field Value
sitemap https://www.chrisdixonstudios.com/sitemap.xml
sitemap https://www.chrisdixonstudios.com/sitemap.html
sitemap https://www.chrisdixonstudios.com/image.xml
Back to top

Comments

  • robots.txt for http://www.chrisdixonstudios.com/
  • Enterprise-Grade Configuration | Last updated: 2025-06-30
  • PRIVACY-FOCUSED SEARCH ENGINES
  • PRIVACY SEARCH SPECIAL RULES
  • Qwant (French privacy search)
  • Startpage (Google proxy)
  • Searx (open-source metasearch)
  • CDN CONFIGURATION
  • Cloudflare (adjust patterns to your CDN)
  • GEO-BLOCKING
  • Block known hostile regions (adjust per analytics)
  • AI/SCRAPER MITIGATION
  • DDoS PROTECTION
  • ADVANCED SECURITY
  • ENTERPRISE CRAWL CONTROL
  • SEARCH ENGINE DIRECTIVES
  • User-agent: Googlebot
  • Sitemap: https://www.chrisdixonstudios.com/sitemap.xml
  • http://www.sitemaps.org/protocol.php
  • User-agent: Mediapartners-Google
  • User-agent: A1 Sitemap Generator
  • User-agent: miggibot
  • Search engine crawl control
  • AGGRESSIVE CRAWLER BLOCKING
  • CONTENT CONTROL
  • System directories
  • Application files
  • Custom directories
  • Explicit allows
  • SITEMAP REFERENCES
  • Sitemap: https://www.chrisdixonstudios.com/sitemap-index.xml
Back to top

Warnings

  • `host` is not a known field.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top