User Guide¶

Complete Guide to VAST Orbit

From data ingestion to machine learning at scale

Welcome to the VAST Orbit User Guide! This comprehensive tutorial series takes you from basics to advanced topics, teaching you how to leverage VAST AI OS for data science at scale.

What You’ll Learn:

Connect to VAST DataBase and query data across multiple sources
Perform in-database analytics without moving data
Build hybrid ML workflows with sklearn/Spark
Create interactive visualizations with federated queries
Deploy production-ready data pipelines

Tip

New Users: Start with Introduction to understand VAST Orbit fundamentals, then progress through each section sequentially. Each guide includes hands-on examples you can run immediately.

Learning Path¶

Follow this structured path to master VAST Orbit. Total time: ~4.5 hours.

1. Introduction

⏱️ 36 minutes

Master VAST Orbit fundamentals including VastFrame (pandas-like API) and VastColumn operations. Learn how to connect to VAST DataBase and execute your first queries.

Key Topics:

Connection setup and authentication
VastFrame creation and basic operations
Column selection and filtering
Lazy evaluation and query optimization

VastFrame VastColumn Connections

Start Learning →

Introduction

2. Data Ingestion

⏱️ 20 minutes

Load data from multiple sources into VAST DataBase. Query files directly (Parquet, CSV, JSON) without loading. Access external databases via VAST DataBase’s federated queries.

Key Topics:

Loading from VAST tables
Querying files in S3/DataStore
Importing from PostgreSQL, MongoDB
Streaming data from Kafka

ETL Data Loading Federated Queries

Start Learning →

Data Ingestion

3. Data Exploration

⏱️ 34 minutes

Discover patterns and insights through interactive visualizations. Create charts that execute in-database with automatic sampling for large datasets.

Key Topics:

Descriptive statistics and profiling
Interactive plotting (Plotly)
Correlation analysis
Distribution visualization

Visualization EDA Charts

Start Learning →

Data Exploration

4. Data Preparation

⏱️ 52 minutes

Transform raw data into analysis-ready datasets. All operations execute in-database for maximum performance on large datasets.

Key Topics:

Data cleaning and missing value handling
Feature engineering and transformation
Aggregation and pivoting
Window functions and time-series ops

Data Cleaning Feature Engineering Transformations

Start Learning →

Data Preparation

5. Machine Learning

⏱️ 38 minutes

Build hybrid ML workflows: train models with sklearn/Spark, deploy for blazing-fast in-database inference. Scale to billions of rows without data movement.

Key Topics:

Training with sklearn and Spark
In-database inference at scale
Model evaluation and metrics
Hyperparameter tuning

Machine Learning sklearn Spark MLlib

Start Learning →

Machine Learning

6. Full Stack

⏱️ 87 minutes

Advanced topics for production deployments. User-defined functions, geospatial analysis, streaming pipelines, and integration with the broader data ecosystem.

Key Topics:

Geospatial analytics with Geopandas
Real-time streaming with Kafka
Production deployment patterns

Geopandas Production Advanced

Start Learning →

Full Stack

Quick Navigation¶

By Topic:

Data Access

Introduction - Connections
Data Ingestion - Loading data

Analysis

Data Exploration - Visualization
Data Preparation - Transformations

Advanced

Machine Learning - ML workflows
Full Stack - Production patterns

Prerequisites:

Basic Python knowledge (pandas familiarity helpful)
Access to a VAST Cluster (4.5+)
Python 3.12+ installed
Familiarity with SQL concepts (helpful but not required)

Format:

Each guide includes:

Hands-on Examples - Copy-paste code that runs immediately
Best Practices - Production-ready patterns
Common Pitfalls - What to avoid and why
Performance Tips - Optimize queries for VAST

Example: Your First VAST Orbit Query¶

Here’s a taste of what you’ll learn:

import vastorbit as vo

# 1. Connect to VAST
vo.new_connection({
    'host': 'vast-cluster.example.com',
    'port': 8080,
    'catalog': 'vast_catalog'
})

# 2. Load data (from VAST table or S3 file)
customers = vo.VastFrame('customers')
transactions = vo.VastFrame('datalake.transactions')

# 3. Analyze with pandas-like syntax (executes in VAST!)
result = customers.join(transactions, on='customer_id')
summary = result.groupby(
    'region',
    ['sum(revenue)', 'count(customer_count)']
)

# 4. Visualize
summary.bar(columns=['region', 'revenue'])

What makes this special?

All operations execute in-database - no data movement
Federated query - join VAST table + S3 file seamlessly
Smart sampling - visualizations work on billions of rows
VAST SQL Engine - coming soon for 10-100x performance boost

Continue Learning¶

Ready to start? Begin with the Introduction guide:

Start with Introduction →

Or jump to a specific topic that interests you!