Complete Guide to VAST Orbit
From data ingestion to machine learning at scale
Welcome to the VAST Orbit User Guide! This comprehensive tutorial series takes you from basics to advanced topics, teaching you how to leverage VAST Data Platform for data science at scale.
**What You'll Learn:**
- Connect to VAST DataBase and query data across multiple sources
- Perform in-database analytics without moving data
- Build hybrid ML workflows with sklearn/Spark
- Create interactive visualizations with federated queries
- Deploy production-ready data pipelines
.. tip::
**New Users**: Start with **Introduction** to understand VAST Orbit fundamentals, then progress through each section sequentially. Each guide includes hands-on examples you can run immediately.
____
Learning Path
-------------
Follow this structured path to master VAST Orbit. Total time: ~4.5 hours.
.. grid:: 1 1 2 2
:class-container: feature-tiles
:gutter: 3
.. grid-item::
.. card:: |i-docs| 1. Introduction
:link: user_guide.introduction
:link-type: ref
:text-align: center
:class-card: custom-card-8
:class-footer: user_guide_footer
⏱️ **36 minutes**
Master VAST Orbit fundamentals including **VastFrame** (pandas-like API) and **VastColumn** operations. Learn how to connect to VAST DataBase and execute your first queries.
**Key Topics:**
- Connection setup and authentication
- VastFrame creation and basic operations
- Column selection and filtering
- Lazy evaluation and query optimization
:bdg-primary:`VastFrame` :bdg-primary:`VastColumn` :bdg-primary:`Connections`
+++
Start Learning →
.. grid-item::
.. card:: |i-files| 2. Data Ingestion
:link: user_guide.data_ingestion
:link-type: ref
:text-align: center
:class-card: custom-card-8
:class-footer: user_guide_footer
⏱️ **20 minutes**
Load data from multiple sources into VAST DataBase. Query files directly (Parquet, CSV, JSON) without loading. Access external databases via VAST DataBase's federated queries.
**Key Topics:**
- Loading from VAST tables
- Querying files in S3/DataStore
- Importing from PostgreSQL, MongoDB
- Streaming data from Kafka
:bdg-primary:`ETL` :bdg-primary:`Data Loading` :bdg-primary:`Federated Queries`
+++
Start Learning →
.. grid-item::
.. card:: |i-explore| 3. Data Exploration
:link: user_guide.data_exploration
:link-type: ref
:text-align: center
:class-card: custom-card-8
:class-footer: user_guide_footer
⏱️ **34 minutes**
Discover patterns and insights through interactive visualizations. Create charts that execute in-database with automatic sampling for large datasets.
**Key Topics:**
- Descriptive statistics and profiling
- Interactive plotting (Plotly)
- Correlation analysis
- Distribution visualization
:bdg-primary:`Visualization` :bdg-primary:`EDA` :bdg-primary:`Charts`
+++
Start Learning →
.. grid-item::
.. card:: |i-prep| 4. Data Preparation
:link: user_guide.data_preparation
:link-type: ref
:text-align: center
:class-card: custom-card-8
:class-footer: user_guide_footer
⏱️ **52 minutes**
Transform raw data into analysis-ready datasets. All operations execute in-database for maximum performance on large datasets.
**Key Topics:**
- Data cleaning and missing value handling
- Feature engineering and transformation
- Aggregation and pivoting
- Window functions and time-series ops
:bdg-primary:`Data Cleaning` :bdg-primary:`Feature Engineering` :bdg-primary:`Transformations`
+++
Start Learning →
.. grid-item::
.. card:: |i-ml| 5. Machine Learning
:link: user_guide.machine_learning
:link-type: ref
:text-align: center
:class-card: custom-card-8
:class-footer: user_guide_footer
⏱️ **38 minutes**
Build hybrid ML workflows: train models with sklearn/Spark, deploy for blazing-fast in-database inference. Scale to billions of rows without data movement.
**Key Topics:**
- Training with sklearn and Spark
- In-database inference at scale
- Model evaluation and metrics
- Hyperparameter tuning
:bdg-primary:`Machine Learning` :bdg-primary:`sklearn` :bdg-primary:`Spark MLlib`
+++
Start Learning →
.. grid-item::
.. card:: |i-start| 6. Full Stack
:link: user_guide.full_stack
:link-type: ref
:text-align: center
:class-card: custom-card-8
:class-footer: user_guide_footer
⏱️ **87 minutes**
Advanced topics for production deployments. User-defined functions, geospatial analysis, streaming pipelines, and integration with the broader data ecosystem.
**Key Topics:**
- Geospatial analytics with Geopandas
- Real-time streaming with Kafka
- Production deployment patterns
:bdg-primary:`Geopandas` :bdg-primary:`Production` :bdg-primary:`Advanced`
+++
Start Learning →
____
Quick Navigation
----------------
**By Topic:**
.. grid:: 2 2 3 3
:class-container: feature-tiles
:gutter: 2
.. grid-item::
**Data Access**
- :ref:`user_guide.introduction` - Connections
- :ref:`user_guide.data_ingestion` - Loading data
.. grid-item::
**Analysis**
- :ref:`user_guide.data_exploration` - Visualization
- :ref:`user_guide.data_preparation` - Transformations
.. grid-item::
**Advanced**
- :ref:`user_guide.machine_learning` - ML workflows
- :ref:`user_guide.full_stack` - Production patterns
**Prerequisites:**
- Basic Python knowledge (pandas familiarity helpful)
- Access to a VAST Cluster (4.5+)
- Python 3.12+ installed
- Familiarity with SQL concepts (helpful but not required)
**Format:**
Each guide includes:
- |check| **Hands-on Examples** - Copy-paste code that runs immediately
- |check| **Best Practices** - Production-ready patterns
- |check| **Common Pitfalls** - What to avoid and why
- |check| **Performance Tips** - Optimize queries for VAST
____
Example: Your First VAST Orbit Query
------------------------------------
Here's a taste of what you'll learn:
.. code-block:: python
import vastorbit as vo
# 1. Connect to VAST
vo.new_connection({
'host': 'vast-cluster.example.com',
'port': 8080,
'catalog': 'vast_catalog'
})
# 2. Load data (from VAST table or S3 file)
customers = vo.VastFrame('customers')
transactions = vo.VastFrame('datalake.transactions')
# 3. Analyze with pandas-like syntax (executes in VAST!)
result = customers.join(transactions, on='customer_id')
summary = result.groupby(
'region',
['sum(revenue)', 'count(customer_count)']
)
# 4. Visualize
summary.bar(columns=['region', 'revenue'])
**What makes this special?**
- |database| All operations execute **in-database** - no data movement
- |globe| **Federated query** - join VAST table + S3 file seamlessly
- |chart| **Smart sampling** - visualizations work on billions of rows
- |zap| **VAST SQL Engine** - coming soon for 10-100x performance boost
.. seealso::
- :ref:`getting_started` - Installation and setup
- :ref:`api` - Complete API reference
- :ref:`examples` - More code examples
- :ref:`chart_gallery` - Visualization gallery
.. note::
**Questions?** Join our community at **`vastsupport.slack.com