Log In Sign Up

teflinstitute.com
robots.txt

Robots Exclusion Standard data for teflinstitute.com

Resource Scan

Scan Details

Site Domain teflinstitute.com
Base Domain teflinstitute.com
Scan Status Ok
Last Scan2025-09-22T01:00:20+00:00
Next Scan 2025-10-22T01:00:20+00:00

Last Scan

Scanned2025-09-22T01:00:20+00:00
URL https://teflinstitute.com/robots.txt
Domain IPs 104.21.84.32, 172.67.185.206, 2606:4700:3031::ac43:b9ce, 2606:4700:3037::6815:5420
Response IP 104.21.84.32
Found Yes
Hash 6198749e04a49e5020cd4ed911a0598518d5bb558be78c38a8d52ee2e5653f19
SimHash f3994260a4ba

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-content/uploads/wc-logs/
Disallow /wp-content/uploads/woocommerce_transient_files/
Disallow /wp-content/uploads/woocommerce_uploads/
Disallow /?s=
Disallow /page/*/?s=
Disallow /search/

mozilla/5.0 applewebkit/537.36 (khtml, like gecko); compatible; oai-searchbot/1.0; +https://openai.com/searchbot
mozilla/5.0 applewebkit/537.36 (khtml, like gecko); compatible; chatgpt-user/1.0; +https://openai.com/bot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

bingbot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

oai-searchbot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

mistralai-user/1.0

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

chatgpt-user
chatgpt-user/2.0

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

gptbot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

anthropic-ai

Product Comment
anthropic-ai bulk model training
Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

claudebot
claude-web

Product Comment
claudebot chat citation fetch
claude-web web-focused crawl
Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

perplexitybot

Product Comment
perplexitybot index builder
Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

perplexity-user

Product Comment
perplexity-user human-triggered visit
Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

google-extended

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

amazonbot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

applebot
applebot-extended

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

facebookbot
meta-externalagent

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

linkedinbot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

bytespider

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

duckassistbot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

cohere-ai

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/

timpibot
youbot

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Back to top

Other Records

Field Value
sitemap https://teflinstitute.com/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • OPENAI
  • Search (shows my webpages as links inside ChatGPT search).
  • Mistral AI
  • User-driven browsing from ChatGPT and Custom GPTs. Acts after a human click.
  • Model-training crawler. Opt-out here if I don’t want content in GPT-4o or GPT-5.
  • ANTHROPIC (Claude)
  • PERPLEXITY
  • GOOGLE (Gemini)
  • AMAZON
  • APPLE
  • META
  • LINKEDIN
  • BYTEDANCE
  • DUCKDUCKGO
  • COHERE
  • EMERGING SEARCH START-UPS
  • ---------------------------
  • END YOAST BLOCK
Back to top