topkitchengadget.com
robots.txt

Robots Exclusion Standard data for topkitchengadget.com

Resource Scan

Scan Details

Site Domain topkitchengadget.com
Base Domain topkitchengadget.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2026-03-05T04:20:27+00:00
Next Scan 2026-06-03T04:20:27+00:00

Last Successful Scan

Scanned2025-11-04T21:19:57+00:00
URL https://topkitchengadget.com/robots.txt
Domain IPs 104.21.20.137, 172.67.192.240, 2606:4700:3030::6815:1489, 2606:4700:3033::ac43:c0f0
Response IP 172.67.192.240
Found Yes
Hash bc21d3500c31d260345c009b1f60d834ad55b566fb119ffd59b61d73bb9ea006
SimHash 536051524433

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-json/
Disallow /xmlrpc.php
Disallow /trackback/
Disallow /feed/
Disallow /readme.html
Disallow /license.txt
Disallow /wp-content/uploads/wc-logs/
Disallow /wp-content/uploads/woocommerce_transient_files/
Disallow /wp-content/uploads/woocommerce_uploads/
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Allow /wp-content/themes/flatsome/assets/

oai-searchbot

Rule Path
Disallow

google-extended

Rule Path
Disallow

perplexitybot

Rule Path
Disallow

gptbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://topkitchengadget.com/sitemap_index.xml

Comments

  • Recommended robots.txt for WordPress
  • Last Updated: 2025-07-07
  • Allow theme assets for proper page rendering
  • --- AI Crawler Rules ---
  • Allow OpenAI's Search Bot (Powers ChatGPT Discovery)
  • Allow Google's Generative AI (Gemini & AI Overviews)
  • Allow Perplexity AI
  • Block OpenAI's general model training bot
  • Block Anthropic's model training bot
  • Block Common Crawl's data scraper