Log In Sign Up

simonandsimon.co.uk
robots.txt

Robots Exclusion Standard data for simonandsimon.co.uk

Resource Scan

Scan Details

Site Domain simonandsimon.co.uk
Base Domain simonandsimon.co.uk
Scan Status Ok
Last Scan2026-02-26T17:54:42+00:00
Next Scan 2026-03-28T17:54:42+00:00

Last Scan

Scanned2026-02-26T17:54:42+00:00
URL https://simonandsimon.co.uk/robots.txt
Redirect https://www.simonandsimon.co.uk/robots.txt
Redirect Domain www.simonandsimon.co.uk
Redirect Base simonandsimon.co.uk
Domain IPs 35.214.63.138
Redirect IPs 35.214.63.138
Response IP 35.214.63.138
Found Yes
Hash 927fe41b2fa1499e8613b68f75373d47b851f95d5fd614af69d5b82125eebbcb
SimHash 593cd453269a

Groups

*

Rule Path
Allow /
Disallow /*?
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/
Disallow /xmlrpc.php
Disallow /wp-content/cache/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /cgi-bin/
Disallow /private/
Disallow /tmp/
Disallow /cart/
Disallow /checkout/
Disallow /search/
Disallow /trackback/
Disallow /comments/
Disallow /author/

semrushbot

Rule Path
Allow /

semrushbot-sa

Rule Path
Allow /

semrushbot-ba

Rule Path
Allow /

semrushbot-si

Rule Path
Allow /

semrushbot-swa

Rule Path
Allow /

semrushbot-ct

Rule Path
Allow /

semrushbot-bm

Rule Path
Allow /

splitsignalbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

rogerbot

Rule Path
Allow /

mj12bot

Rule Path
Allow /

gptbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.simonandsimon.co.uk/sitemap_index.xml
Back to top

Comments

  • WORDPRESS + RANK MATH OPTIMIZED ROBOTS.TXT
  • --- WordPress Internal Protection ---
  • --- Block low-value URLs ---
  • ALLOW ALL SEO + MARKETING CRAWLERS
  • Semrush SEO crawler (fully allowed)
  • Ahrefs SEO crawler (fully allowed)
  • Moz SEO crawler
  • Majestic SEO crawler
  • ALLOW AI CRAWLERS (GPT, CLAUDE, PERPLEXITY)
  • OpenAI GPTBot
  • Anthropic ClaudeBot
  • Perplexity AI Crawler
  • Amazon GPT-style Amazonbot
  • Apple’s AI/Siri crawler
  • Facebook / Meta AI scraper
  • ALLOW SEARCH ENGINE BOTS
  • BLOCK ONLY HARMFUL / ABUSIVE BOTS
  • RANKMATH SITEMAP
Back to top