Log In Sign Up

umva.us
robots.txt

Robots Exclusion Standard data for umva.us

Resource Scan

Scan Details

Site Domain umva.us
Base Domain umva.us
Scan Status Ok
Last Scan2025-11-24T21:41:25+00:00
Next Scan 2025-12-01T21:41:25+00:00

Last Scan

Scanned2025-11-24T21:41:25+00:00
URL https://umva.us/robots.txt
Domain IPs 104.21.95.170, 172.67.146.57, 2606:4700:3031::ac43:9239, 2606:4700:3037::6815:5faa
Response IP 172.67.146.57
Found Yes
Hash 9787de11a5f0e72e5f7acc40fd613910368420cf2a6b52a30f2db1e60788e515
SimHash 6398f8916450

Groups

*

Rule Path
Allow /
Disallow /admin
Disallow /admin/
Disallow /merchant/login
Disallow /merchant/register
Disallow /agent/login
Disallow /agent/register
Disallow /user/login
Disallow /user/register
Disallow /user/dashboard
Disallow /user/password
Disallow /password/
Disallow /logout
Disallow /verify/
Disallow /otp-verification
Disallow /api/
Disallow /cron
Disallow /webhook
Disallow /ipn/
Disallow /temp/
Disallow /storage/
Disallow /vendor/
Disallow /core/
Disallow /backups/
Disallow /*.json$
Disallow /*.xml$
Disallow /*?*session
Disallow /*?*token
Disallow /*?*api_key
Disallow /search?
Disallow /*?page=
Disallow /*%26page%3D
Allow /api/documentation
Allow /api/reference
Allow /contact
Allow /about
Allow /tips/
Allow /assets/
Allow /sitemap.xml
Allow /robots.txt

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /
Disallow /admin/
Disallow /user/
Disallow /merchant/
Disallow /agent/
Disallow /api/

Other Records

Field Value
crawl-delay 0.5

googlebot-image

Rule Path
Allow /assets/images/
Disallow /assets/images/user/
Disallow /assets/images/merchant/
Disallow /assets/images/agent/

googlebot-mobile

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0.5

bingbot

Rule Path
Allow /
Disallow /admin/
Disallow /user/

Other Records

Field Value
crawl-delay 1

msnbot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 2

yandexbot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 2

facebookexternalhit

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0

twitterbot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0

linkedinbot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0

pinterest

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 1

whatsapp

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0

telegrambot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0

slackbot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0

discordbot

Rule Path
Allow /
Disallow /admin/

Other Records

Field Value
crawl-delay 0

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

wget

Rule Path
Disallow /

curl

Rule Path
Disallow /

httrack

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

*

Rule Path
Disallow /*/
Back to top

Other Records

Field Value
sitemap https://umva.net/sitemap.xml
Back to top

Comments

  • UMVA Super App Platform - Robots.txt
  • Enhanced SEO Configuration
  • Last Updated: 2025-10-19
  • Default Crawlers
  • Allow Important Pages
  • Google Crawlers
  • Bing Crawler
  • Yahoo Crawler
  • DuckDuckGo
  • Baidu (Chinese Search Engine)
  • Yandex (Russian Search Engine)
  • Facebook Crawler (for Open Graph)
  • Twitter Crawler (for Twitter Cards)
  • LinkedIn Crawler
  • Pinterest Crawler
  • WhatsApp Crawler (for link previews)
  • Telegram Crawler (for link previews)
  • Slack Crawler (for link previews)
  • Discord Crawler (for link previews)
  • Block Bad Bots
  • Block AI Scrapers (Optional - uncomment if needed)
  • User-agent: GPTBot
  • Disallow: /
  • User-agent: ChatGPT-User
  • Disallow: /
  • User-agent: CCBot
  • Disallow: /
  • User-agent: anthropic-ai
  • Disallow: /
  • User-agent: Claude-Web
  • Disallow: /
  • Block Download Bots
  • Sitemap Location
  • Additional Security
  • Prevent directory listing
  • Host (Optional - specify preferred domain)
  • Host: https://umva.net
Back to top