Log In Sign Up

isurflive.com
robots.txt

Robots Exclusion Standard data for isurflive.com

Resource Scan

Scan Details

Site Domain isurflive.com
Base Domain isurflive.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2026-02-15T22:25:03+00:00
Next Scan 2026-05-16T22:25:03+00:00

Last Successful Scan

Scanned2025-10-18T19:18:34+00:00
URL https://isurflive.com/robots.txt
Domain IPs 104.21.58.201, 172.67.164.90, 2606:4700:3033::6815:3ac9, 2606:4700:3037::ac43:a45a
Response IP 104.21.58.201
Found Yes
Hash 0c364688e1bae13451522b610a277e08117b1b91ae1ce501bf9ac8cff70010a4
SimHash 6917dc906490

Groups

*

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Allow /_next/
Disallow /404
Disallow /500
Disallow /maintenance
Disallow /admin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /private/
Disallow /tmp/
Disallow /cache/
Disallow /api/
Disallow *?s=*
Disallow *?search=*
Disallow */search?*
Disallow */filter?*
Disallow *?sort=*
Disallow *?page=*
Disallow */cart
Disallow */checkout
Disallow */login
Disallow */register

googlebot

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Allow /_next/
Disallow /404
Disallow /500
Disallow /maintenance
Disallow /admin/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /private/
Disallow /api/
Disallow *?s=*
Disallow */cart
Disallow */checkout
Disallow */login

googlebot-news

Rule Path
Allow /
Allow /news/
Allow /blog/
Disallow /admin/
Disallow /private/
Disallow /api/

googlebot-image

Rule Path
Allow /
Allow /images/
Disallow /admin/
Disallow /private/
Disallow /api/

googlebot-mobile

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

googlebot-video

Rule Path
Allow /
Allow /videos/
Disallow /admin/
Disallow /private/
Disallow /api/

bingbot

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Allow /_next/
Disallow /404
Disallow /500
Disallow /maintenance
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow *?s=*
Disallow */cart
Disallow */checkout
Disallow */login

Other Records

Field Value
crawl-delay 1

bingbot-mobile

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow *?s=*
Disallow */cart
Disallow */checkout
Disallow */login

Other Records

Field Value
crawl-delay 1

yandexbot

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow *?s=*
Disallow */cart
Disallow */checkout
Disallow */login

Other Records

Field Value
crawl-delay 1

yandeximages

Rule Path
Allow /
Allow /images/
Disallow /admin/
Disallow /private/
Disallow /api/

yandexnews

Rule Path
Allow /
Allow /news/
Allow /blog/
Disallow /admin/
Disallow /private/
Disallow /api/

seznambot

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow *?s=*
Disallow */cart
Disallow */checkout
Disallow */login

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow *?s=*
Disallow */cart
Disallow */checkout
Disallow */login

Other Records

Field Value
crawl-delay 2

duckduckbot

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow *?s=*

Other Records

Field Value
crawl-delay 1

yeti

Rule Path
Allow /
Allow /about
Allow /faq
Allow /discord
Allow /status
Disallow /admin/
Disallow /private/
Disallow /api/
Disallow *?s=*

Other Records

Field Value
crawl-delay 2

mediapartners-google

Rule Path
Allow /
Disallow /api/

adsbot-google

Rule Path
Allow /
Disallow /api/

adsbot-google-mobile

Rule Path
Allow /
Disallow /api/

facebookexternalhit

Rule Path
Allow /
Disallow /api/

twitterbot

Rule Path
Allow /
Disallow /api/

linkedinbot

Rule Path
Allow /
Disallow /api/

whatsapp

Rule Path
Allow /
Disallow /api/

telegrambot

Rule Path
Allow /
Disallow /api/

applebot

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /api/

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

majesticseo

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

gptbot

Rule Path
Allow /
Disallow /api/

chatgpt-user

Rule Path
Allow /
Disallow /api/

ccbot

Rule Path
Allow /
Disallow /api/

anthropic-ai

Rule Path
Allow /
Disallow /api/

claudebot

Rule Path
Allow /
Disallow /api/

claude-web

Rule Path
Allow /
Disallow /api/

perplexity

Rule Path
Allow /
Disallow /api/

perplexitybot

Rule Path
Allow /
Disallow /api/

bytespider

Rule Path
Allow /
Disallow /api/

diffbot

Rule Path
Allow /
Disallow /api/

friendlycrawler

Rule Path
Allow /
Disallow /api/
Back to top

Other Records

Field Value
sitemap https://isurflive.com/sitemap.xml
Back to top

Comments

  • Comprehensive robots.txt file for all major search engines
  • Last updated: 2025
  • Default rules for all crawlers
  • Google Search Engine
  • Google News
  • Google Images
  • Google Mobile
  • Google Video
  • Microsoft Bing (IndexNow supporter)
  • Bing Mobile
  • Yahoo/Verizon Media (Slurp)
  • Yandex (IndexNow supporter, major in Russia/Eastern Europe)
  • Yandex Images
  • Yandex News
  • Seznam (IndexNow supporter, major in Czech Republic)
  • Baidu (Major in China)
  • DuckDuckGo
  • Naver (Major in South Korea)
  • Google Ads crawlers
  • Social media crawlers (allow for social sharing)
  • Apple (Applebot)
  • Block problematic/aggressive crawlers
  • AI Training Bots - ALLOW for training data but block APIs
  • Note: Generic bad bots blocked at Cloudflare level
  • Sitemap location
Back to top