Log In Sign Up

totl.ai
robots.txt

Robots Exclusion Standard data for totl.ai

Resource Scan

Scan Details

Site Domain totl.ai
Base Domain totl.ai
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-10-25T15:29:44+00:00
Next Scan 2026-01-23T15:29:44+00:00

Last Successful Scan

Scanned2025-06-21T00:46:19+00:00
URL https://totl.ai/robots.txt
Domain IPs 217.21.84.252
Response IP 217.21.84.252
Found Yes
Hash 9c373c33b91a02b4a6fcc237bd9346a52b06b019eb0c6b90c8621e597db516ca
SimHash 6810d621c0a3

Groups

*
googlebot
googlebot-mobile
googlebot-image
googlebot-news
googlebot-video

Rule Path
Allow /
Disallow /category
Allow /wp-admin/admin-ajax.php
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Disallow /*?=
Disallow /author/
Disallow /work_genre/
Disallow /plus-mega-menu
Disallow */page/
Disallow /refund-cancellation-policy
Disallow /basecamp
Disallow /career/
Disallow */deep-crawl/?unapproved=
Disallow */?utm_source=

chatgpt

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

chatgpt-user

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

oai-search

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

bard

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

gemini

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

oai-searchbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

openai

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

generativeai

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

predictiveai

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Allow /
Disallow /category
Allow /wp-admin/admin-ajax.php
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Disallow /*?=
Disallow /author/
Disallow /work_genre/
Disallow /plus-mega-menu
Disallow */page/
Disallow /refund-cancellation-policy
Disallow /career/
Disallow /basecamp
Disallow */deep-crawl/?unapproved=
Disallow */?utm_source=

Other Records

Field Value
crawl-delay 15

baiduspider

Rule Path
Allow /
Disallow /category
Allow /wp-admin/admin-ajax.php
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Disallow /*?=
Disallow /author/
Disallow /work_genre/
Disallow /plus-mega-menu
Disallow */page/
Disallow /refund-cancellation-policy
Disallow /career/
Disallow /basecamp
Disallow */deep-crawl/?unapproved=
Disallow */?utm_source=

Other Records

Field Value
crawl-delay 30

yandexbot

Rule Path
Allow /
Disallow /category
Allow /wp-admin/admin-ajax.php
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Disallow /*?=
Disallow /author/
Disallow /work_genre/
Disallow /plus-mega-menu
Disallow */page/
Disallow /refund-cancellation-policy
Disallow /career/
Disallow /basecamp
Disallow */deep-crawl/?unapproved=
Disallow */?utm_source=

Other Records

Field Value
crawl-delay 28

duckduckbot

Rule Path
Allow /
Disallow /category
Allow /wp-admin/admin-ajax.php
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Disallow /*?=
Disallow /author/
Disallow /work_genre/
Disallow /plus-mega-menu
Disallow */page/
Disallow /refund-cancellation-policy
Disallow /career/
Disallow /basecamp
Disallow */deep-crawl/?unapproved=
Disallow */?utm_source=

Other Records

Field Value
crawl-delay 29

slurp

Rule Path
Allow /
Disallow /category
Allow /wp-admin/admin-ajax.php
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Disallow /*?=
Disallow /author/
Disallow /work_genre/
Disallow /plus-mega-menu
Disallow */page/
Disallow /refund-cancellation-policy
Disallow /career/
Disallow /basecamp
Disallow */deep-crawl/?unapproved=
Disallow */?utm_source=

Other Records

Field Value
crawl-delay 20

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

facebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 30

linkedinbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

pinterestbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

instagram

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

msnbot

Rule Path
Allow /
Disallow /category
Allow /wp-admin/admin-ajax.php
Allow /*.js$
Allow /*.css$
Allow /*.jpg$
Allow /*.png$
Allow /*.jpeg$
Allow /*.gif$
Disallow /*?=
Disallow /author/
Disallow /work_genre/
Disallow /plus-mega-menu
Disallow */page/
Disallow /refund-cancellation-policy
Disallow /career/
Disallow /basecamp
Disallow */deep-crawl/?unapproved=
Disallow */?utm_source=

Other Records

Field Value
crawl-delay 24

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 17

alexacrawler

Rule Path
Allow /

Other Records

Field Value
crawl-delay 40
Back to top

Other Records

Field Value
sitemap https://totl.ai/sitemap_index.xml
Back to top

Comments

  • Allow AI and generative bots
  • Allow all other search engines
  • Allow social media crawlers
  • Allow Apple and Amazon bots
Back to top