Log In Sign Up

webtreeonline.com
robots.txt

Robots Exclusion Standard data for webtreeonline.com

Resource Scan

Scan Details

Site Domain webtreeonline.com
Base Domain webtreeonline.com
Scan Status Ok
Last Scan2025-11-03T17:38:49+00:00
Next Scan 2025-12-03T17:38:49+00:00

Last Scan

Scanned2025-11-03T17:38:49+00:00
URL https://webtreeonline.com/robots.txt
Domain IPs 104.21.95.159, 172.67.145.233, 2606:4700:3036::ac43:91e9, 2606:4700:3037::6815:5f9f
Response IP 104.21.95.159
Found Yes
Hash da91ef874a0fa3288a38eb385ecb8918d2d41d42fd05eeedbbb2e7d411e60e28
SimHash 70389e02c732

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

oai-searchbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

google-extended

Rule Path
Allow /

ccbot

Rule Path
Allow /

facebookbot

Rule Path
Allow /

applebot

Rule Path
Allow /

bytespider

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.webtreeonline.com/sitemap_index.xml
Back to top

Comments

  • Default WordPress rules
  • Allow all major chatbot and AI crawlers
  • OpenAI (Search + Training)
  • Anthropic (Claude)
  • Perplexity
  • Google AI (Bard/Gemini)
  • Common AI research crawler
  • Facebook / Meta AI
  • AppleBot (used in Apple AI features & Siri)
  • Bytedance / TikTok AI
Back to top