The VastFrame¶

Master the core object for in-database analytics with VAST Orbit.

Overview¶

The VastFrame is the core object of VAST Orbit. It enables Python-based data manipulation without moving data from the VAST DataBase to local memory.

Key benefits:

In-database processing – All operations execute in VAST’s Trino engine
Minimal memory usage – Only metadata is stored in Python
Parallel execution – Leverage VAST’s distributed query engine
Lazy evaluation – Operations are optimized before execution

VastFrames behave like SQL views, formulating operations as queries that execute directly in the database.

Creating VastFrames¶

Load the Titanic dataset:

from vastorbit.datasets import load_titanic

load_titanic()

	123 pclass Integer 100%	123 survived Integer 100%	Abc name Varchar(164) 100%	Abc sex Varchar(20) 100%	123 age Double 79%	123 sibsp Integer 100%	Abc ticket Varchar(36) 100%	123 fare Double 99%	Abc cabin Varchar(30) 22%	Abc embarked Varchar(20) 99%	Abc boat Varchar(100) 37%	123 body Integer 9%	Abc home.dest Varchar(100) 56%
1	3	1	McCormack, Mr. Thomas Joseph	male	[null]	0	367228	7.75	[null]	Q	[null]	[null]	[null]
2	3	1	McCoy, Miss. Agnes	female	[null]	2	367226	23.25	[null]	Q	16	[null]	[null]
3	3	1	McCoy, Miss. Alicia	female	[null]	2	367226	23.25	[null]	Q	16	[null]	[null]
4	3	1	McCoy, Mr. Bernard	male	[null]	2	367226	23.25	[null]	Q	16	[null]	[null]
5	3	1	McDermott, Miss. Brigdet Delia	female	[null]	0	330932	7.7875	[null]	Q	13	[null]	[null]
6	3	0	McEvoy, Mr. Michael	male	[null]	0	36568	15.5	[null]	Q	[null]	[null]	[null]
7	3	1	McGovern, Miss. Mary	female	[null]	0	330931	7.8792	[null]	Q	13	[null]	[null]
8	3	1	McGowan, Miss. Anna "Annie"	female	15.0	0	330923	8.0292	[null]	Q	[null]	[null]	[null]
9	3	0	McGowan, Miss. Katherine	female	35.0	0	9232	7.75	[null]	Q	[null]	[null]	[null]
10	3	0	McMahon, Mr. Martin	male	[null]	0	370372	7.75	[null]	Q	[null]	[null]	[null]
11	3	0	McNamee, Mr. Neal	male	24.0	1	376566	16.1	[null]	S	[null]	[null]	[null]
12	3	0	McNamee, Mrs. Neal (Eileen O'Leary)	female	19.0	1	376566	16.1	[null]	S	[null]	53	[null]
13	3	0	McNeill, Miss. Bridget	female	[null]	0	370368	7.75	[null]	Q	[null]	[null]	[null]
14	3	0	Meanwell, Miss. (Marion Ogden)	female	[null]	0	SOTON/O.Q. 392087	8.05	[null]	S	[null]	[null]	[null]
15	3	0	Meek, Mrs. Thomas (Annie Louise Rowley)	female	[null]	0	343095	8.05	[null]	S	[null]	[null]	[null]
16	3	0	Meo, Mr. Alfonzo	male	55.5	0	A.5. 11206	8.05	[null]	S	[null]	201	[null]
17	3	0	Mernagh, Mr. Robert	male	[null]	0	368703	7.75	[null]	Q	[null]	[null]	[null]
18	3	1	Midtsjo, Mr. Karl Albert	male	21.0	0	345501	7.775	[null]	S	15	[null]	[null]
19	3	0	Miles, Mr. Frank	male	[null]	0	359306	8.05	[null]	S	[null]	[null]	[null]
20	3	0	Mineff, Mr. Ivan	male	24.0	0	349233	7.8958	[null]	S	[null]	[null]	[null]

From an existing table:

import vastorbit as vo

vo.VastFrame("titanic")

From a SQL query:

vo.VastFrame("SELECT pclass, AVG(survived) AS survived FROM titanic GROUP BY 1")

	123 pclass Integer 100%	123 survived Double 100%
1	3	0.2552891396332863
2	2	0.4296028880866426
3	1	0.6191950464396285

For more examples, see VastFrame.

In-Database vs In-Memory¶

The following examples demonstrate performance advantages of in-database processing.

Note

These examples show the process without actual computation due to dataset size. When executed, in-database processing is significantly faster.

Load data into VAST:

vo.read_csv(
    "expedia.csv",
    schema="default",
    parse_nrows=20000000,
)

Create VastFrame (in-database):

import time

start_time = time.time()
expedia = vo.VastFrame("expedia")
print(f"Elapsed time: {time.time() - start_time:.2f}s")

All 4GB of data remains in VAST—no in-memory loading required.

Compare with pandas (in-memory):

Warning

Avoid running this on machines with less than 2GB RAM.

import pandas as pd

start_time = time.time()
expedia_df = pd.read_csv("expedia.csv")
print(f"Elapsed time: {time.time() - start_time:.2f}s")

Loading into pandas takes orders of magnitude longer and consumes significant memory.

Compute correlation matrix (pandas):

columns_to_drop = ["date_time", "srch_ci", "srch_co"]
expedia_df = expedia_df.drop(columns_to_drop, axis=1)

start_time = time.time()
expedia_df.corr()
print(f"Elapsed time: {time.time() - start_time:.2f}s")

Compute correlation matrix (VastFrame):

# Remove non-numeric columns
expedia.drop(columns=["date_time", "srch_ci", "srch_co"])

start_time = time.time()
expedia.corr(show=False)
print(f"Elapsed time: {time.time() - start_time:.2f}s")

VAST Orbit caches computed aggregations for instant retrieval:

Note

Disable caching with: vo.set_option("cache", False)

start_time = time.time()
expedia.corr(show=False)
print(f"Elapsed time: {time.time() - start_time:.2f}s")  # Nearly instant

Memory Usage¶

pandas DataFrame:

expedia_df.info()

Memory usage equals original file size (~4GB).

VastFrame:

VastFrame uses only ~37KB! By storing data in VAST and only tracking metadata in Python, memory usage is minimized.

Tip

In-database processing eliminates the need for downsampling, preserving all data insights.

VastFrame Structure¶

VastFrames are composed of VastColumn objects.

View all columns:

expedia.get_columns()

Access a column:

Note

VAST Orbit caches aggregations to avoid recomputation.

expedia["is_booking"].describe()

	value
name	"is_booking"
dtype	integer
unique	2.0
count	149814.0
0	138566
1	11248

View column catalog:

Each VastColumn maintains a catalog of user modifications:

expedia["is_booking"]._catalog

Enable SQL code generation:

vo.set_option("sql_on", True)
expedia["cnt"].describe()

-- Computing aggregations
SELECT
    APPROX_DISTINCT("cnt")
FROM (
    SELECT * FROM "expedia"
) AS VASTORBIT_SUBTABLE
LIMIT 1;

	value
name	"cnt"
dtype	integer
unique	30.0
count	149814.0
mean	1.4938523769474148
std	1.2299667612009215
min	1.0
approx_25%	1.0
approx_50%	1.0
approx_75%	2.0
max	47.0

Enable query timing:

vo.set_option("sql_on", False)
expedia = vo.VastFrame("expedia")
vo.set_option("time_on", True)
expedia.corr()

Cached results are instant:

import time

start_time = time.time()
expedia.corr()
print(f"Elapsed time: {time.time() - start_time:.2f}s")

Disable options:

vo.set_option("sql_on", False)
vo.set_option("time_on", False)

Query Relations¶

View current relation:

print(expedia.current_relation())

After modifications:

expedia["orig_destination_distance"].fillna(method="avg")
expedia["is_package"].drop()
print(expedia.current_relation())

Notice the SQL reflects the changes: is_package removed and COALESCE added for imputation.

VastFrame Attributes¶

VastFrames have two attribute types:

Virtual Columns – VastColumn objects
Main attributes – Stored in _vars dictionary

Warning

Never modify _vars manually.

expedia._vars

Data Types¶

VAST Orbit recognizes four main data types:

int – Treated as categorical when low cardinality, otherwise numeric
real – Numeric data types
date – Date/timestamp types
text – Categorical data types

View data types:

expedia.dtypes()

	dtype
"date_time"	timestamp(3)
"site_name"	integer
"posa_continent"	integer
"user_location_country"	integer
"user_location_region"	integer
"user_location_city"	integer
"orig_destination_distance"	real
"user_id"	integer
"is_mobile"	integer
"channel"	integer
"srch_ci"	date
"srch_co"	date
"srch_adults_cnt"	integer
"srch_children_cnt"	integer
"srch_rm_cnt"	integer
"srch_destination_id"	integer
"srch_destination_type_id"	integer
"is_booking"	integer
"cnt"	integer
"hotel_continent"	integer
"hotel_country"	integer
"hotel_market"	integer
"hotel_cluster"	integer

Convert data types:

expedia["hotel_market"].astype("varchar")
expedia["hotel_market"].ctype()

View column category:

expedia["hotel_market"].category()

Saving and Loading¶

Save current state:

expedia.save()
expedia.filter("is_booking = 1")

	📅 date_time Timestamp(3) 100%	123 site_name Integer 100%	123 posa_continent Integer 100%	123 user_location_country Integer 100%	123 user_location_region Integer 100%	123 user_location_city Integer 100%	123 orig_destination_distance Real 100%	123 user_id Integer 100%	123 is_mobile Integer 100%	123 channel Integer 100%	📅 srch_ci Date 100%	📅 srch_co Date 100%	123 srch_adults_cnt Integer 100%	123 srch_children_cnt Integer 100%	123 srch_rm_cnt Integer 100%	123 srch_destination_id Integer 100%	123 srch_destination_type_id Integer 100%	123 is_booking Integer 100%	123 cnt Integer 100%	123 hotel_continent Integer 100%	123 hotel_country Integer 100%	Abc hotel_market Varchar 100%	123 hotel_cluster Integer 100%
1	2013-07-01 12:17:38	24	2	3	63	1210	2081.977416385884	2900	0	0	2013-07-07	2013-07-08	1	0	1	19548	6	1	1	3	106	107	18
2	2013-07-01 15:00:55	24	2	3	50	5703	2081.977416385884	1310	1	1	2013-07-07	2013-07-18	1	0	1	8811	1	1	1	2	198	391	10
3	2013-07-01 22:35:36	2	3	66	174	46432	69.9807	1272	0	0	2013-07-15	2013-07-16	2	0	1	24677	6	1	1	2	50	354	28
4	2013-07-02 12:57:06	28	1	55	25	4780	2081.977416385884	2473	0	5	2013-07-08	2013-07-12	2	1	1	11713	3	1	1	6	68	295	82
5	2013-07-02 13:23:34	34	3	205	354	52284	200.1696	251	1	9	2013-07-14	2013-07-15	2	0	1	27000	6	1	1	2	50	574	42
6	2013-07-02 15:50:43	13	1	46	172	8425	494.7855	2883	0	5	2013-10-16	2013-10-22	2	1	1	8746	1	1	2	6	105	29	8
7	2013-07-02 21:55:58	8	4	77	871	36643	9186.6001	1996	0	9	2013-09-08	2013-09-10	2	0	1	44045	3	1	1	2	50	701	83
8	2013-07-03 11:32:22	2	3	66	260	36620	477.2572	1626	0	9	2013-08-20	2013-08-23	2	0	1	25734	1	1	1	2	50	730	17
9	2013-07-03 15:29:26	24	2	3	40	35699	2081.977416385884	1363	0	9	2013-08-29	2013-09-02	2	0	2	5401	6	1	1	3	182	83	15
10	2013-07-03 19:34:45	24	2	3	50	5703	2081.977416385884	2461	0	1	2013-07-11	2013-07-12	1	0	1	12452	5	1	1	2	50	656	48
11	2013-07-04 11:46:55	24	2	3	50	5703	2081.977416385884	1192	0	0	2013-07-05	2013-07-07	1	0	1	19969	6	1	1	3	99	88	81
12	2013-07-04 13:23:44	23	1	1	380	5919	264.5345	1004	0	5	2013-07-13	2013-07-17	2	2	1	23999	6	1	1	6	105	1826	89
13	2013-07-04 20:04:37	24	2	3	64	3169	2081.977416385884	3511	0	1	2013-10-04	2013-10-07	1	0	1	51654	6	1	1	3	130	91	32
14	2013-07-05 09:38:22	24	2	3	50	5703	2081.977416385884	2961	0	0	2013-08-30	2013-08-31	1	0	1	12251	6	1	1	6	204	27	5
15	2013-07-05 10:20:45	24	2	3	50	5703	2081.977416385884	2961	0	0	2013-09-03	2013-09-04	1	0	1	22616	6	1	1	6	204	1452	33
16	2013-07-05 10:52:27	24	2	3	50	5703	2081.977416385884	1492	0	9	2013-09-11	2013-09-12	2	0	1	20332	1	1	1	3	126	264	85
17	2013-07-05 14:02:31	30	4	195	597	51774	2081.977416385884	2661	0	9	2013-09-16	2013-09-18	2	0	1	8279	1	1	1	2	50	1230	55
18	2014-10-15 11:30:29	24	2	3	50	5703	2081.977416385884	2545	0	5	2014-10-21	2014-10-23	1	0	1	43130	1	1	2	0	63	1001	11
19	2014-10-15 12:32:46	37	1	69	596	24410	87.3674	1224	0	9	2014-11-01	2014-11-02	2	0	1	23507	6	1	1	6	70	19	98
20	2014-10-15 12:57:48	2	3	66	174	16634	24.5499	3941	0	9	2014-11-02	2014-11-03	2	1	1	24603	1	1	1	2	50	1042	91

Restore previous state:

expedia = expedia.load()
print(expedia.shape())

Exporting to Database¶

VastFrame modifications don’t affect the underlying database. To persist changes, save to a new table.

Check storage requirements:

expedia.expected_store_usage(unit="Gb")

	expected_size (Gb)	max_size (Gb)	type
"date_time"	7.450580596923828e-09	0.0011162012815475464	timestamp(3)
"site_name"	0.00023080874234437943	0.0011162012815475464	integer
"posa_continent"	0.0001395251601934433	0.0011162012815475464	integer
"user_location_country"	0.0002450430765748024	0.0011162012815475464	integer
"user_location_region"	0.0003590201959013939	0.0011162012815475464	integer
"user_location_city"	0.0006500827148556709	0.0011162012815475464	integer
"orig_destination_distance"	7.450580596923828e-09	0.0011162012815475464	real
"user_id"	0.0005371104925870895	0.0011162012815475464	integer
"is_mobile"	0.0001395251601934433	0.0011162012815475464	integer
"channel"	0.00013953540474176407	0.0011162012815475464	integer
"srch_ci"	7.450580596923828e-09	0.0011138394474983215	date
"srch_co"	7.450580596923828e-09	0.0011138394474983215	date
"srch_adults_cnt"	0.0001395251601934433	0.0011162012815475464	integer
"srch_children_cnt"	0.0001395251601934433	0.0011162012815475464	integer
"srch_rm_cnt"	0.0001395251601934433	0.0011162012815475464	integer
"srch_destination_id"	0.0006121760234236717	0.0011162012815475464	integer
"srch_destination_type_id"	0.0001395251601934433	0.0011162012815475464	integer
"is_booking"	0.0001395251601934433	0.0011162012815475464	integer
"cnt"	0.0001399517059326172	0.0011162012815475464	integer
"hotel_continent"	0.0001395251601934433	0.0011162012815475464	integer
"hotel_country"	0.00033083464950323105	0.0011162012815475464	integer
"hotel_market"	0.00040635280311107635	0.011162012815475464	varchar
"hotel_cluster"	0.0002643503248691559	0.0011162012815475464	integer
separator	0.003209078684449196	0.003209078684449196
header	3.4831464290618896e-07	3.4831464290618896e-07
rawsize	0.008240924216806889	0.03892314434051514

Save to database:

expedia.to_db(
    "expedia_clean",
    relation_type="table",
)

Tip

VastFrames behave like views—they’re lightweight representations of database queries. Use to_db() only when you need to materialize results.