vastdata.com
robots.txt

Robots Exclusion Standard data for vastdata.com

Resource Scan

Scan Details

Site Domain vastdata.com
Base Domain vastdata.com
Scan Status Ok
Last Scan2025-08-04T15:49:19+00:00
Next Scan 2025-09-03T15:49:19+00:00

Last Scan

Scanned2025-08-04T15:49:19+00:00
URL https://vastdata.com/robots.txt
Redirect https://www.vastdata.com/robots.txt
Redirect Domain www.vastdata.com
Redirect Base vastdata.com
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.21
Response IP 76.76.21.21
Found Yes
Hash eb57efd8bdd9d8f06d0c2f659c2ce21ced1d4054e017bfda65109f12b5b20898
SimHash 71215a44acb3

Groups

*

Rule Path
Disallow /_nuxt/img/
Disallow /admin/
Disallow /login
Disallow /register
Disallow /404
Disallow /500
Disallow /nomoretiers
Disallow /search
Disallow /events/newsevent
Disallow /resources/forms/
Disallow /vast-cash
Disallow /cosmos-confirmation
Disallow /deal-confirmation
Disallow /demo-confirmation
Disallow /entry-confirmation
Disallow /event-confirmation
Disallow /german-event-confirmation
Disallow /live-demo-confirmation
Disallow /lunch-confirmation
Disallow /meeting-confirmation
Disallow /partner-confirmation
Disallow /partner-sko-confirmation
Disallow /thank-you
Disallow /thank-you/
Disallow /trade-show-confirmation
Disallow /vastronaut-confirmation
Disallow /*?query=
Disallow /*?page=
Disallow /*?sort=
Disallow /*?filter=
Disallow /ja/
Allow /ja/contact
Allow /ja/demo
Allow /ja/personalized-demo
Allow /ja/platform/dataengine
Allow /ja/platform/database
Allow /ja/platform/dataspace
Allow /ja/platform/datastore
Allow /ja/platform/overview
Allow /ja/support-services-terms
Allow /ja/end-user-services-and-license-agreement
Allow /ja/solutions
Allow /ja/whitepaper
Allow /ja$

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

omigilibot

Rule Path
Disallow /

omigili

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.vastdata.com/sitemap.xml

Comments

  • Main robots rules
  • Block Nuxt image assets
  • Block admin and utility pages
  • Block confirmation pages
  • Block search results, pagination, sorting, and filtering
  • Block all Japanese pages by default
  • Explicitly allow specific Japanese pages
  • Special handling for social media crawlers
  • Block AI crawlers
  • Sitemap