Loading...

VastFrame

Main data structure for working with VAST DataBase - pandas-like API for data at scale.

Core Classes

VastFrame - DataFrame-like object for VAST DataBase tables

class vastorbit.VastFrame(input_relation: str | list | dict | DataFrame | ndarray | TableSample, usecols: Annotated[str | list[str], 'STRING representing one column or a list of columns'] | None = None, schema: str | None = None, _empty: bool = False, _is_sql_magic: int = 0, _clean_query: bool = True)

An object that records all user modifications, allowing users to manipulate the relation without mutating the underlying data in VAST. When changes are made, the VastFrame queries the VAST DataBase, which aggregates and returns the final result. The VastFrame creates, for each column of the relation, a Virtual Column (VastColumn) that stores the column alias an all user transformations.

Parameters:

  • input_relation (str | TableSample | pandas.DataFrame | list | numpy.ndarray | dict, optional) – If the input_relation is of type str, it must represent the relation (view, table, or temporary table) used to create the object. To get a specific schema relation, your string must include both the relation and schema: 'schema.relation' or '"schema"."relation"'. Alternatively, you can use the ‘schema’ parameter, in which case the input_relation must exclude the schema name. It can also be the SQL query used to create the VastFrame. If it is a pandas.DataFrame, a temporary local table is created. Otherwise, the VastFrame is created using the generated SQL code of multiple UNIONs.

  • usecols (SQLColumns, optional) – When input_relation is not an array-like type: List of columns used to create the object. As VAST is a columnar DB, including less columns makes the process faster. Do not hesitate to exclude useless columns. Otherwise: List of column names.

  • schema (str, optional) – The schema of the relation. Specifying a schema allows you to specify a table within a particular schema, or to specify a schema and relation name that contain period ‘.’ characters. If specified, the input_relation cannot include a schema.

Variables:

VastColumns (VastColumn) – Each VastColumn of the VastFrame is accessible by specifying its name between brackets. For example, to access the VastColumn “myVC”: VastFrame["myVC"].

Examples

In this example, we will look at some of the ways how we can create a VastFrame.

  • From dictionary

  • From numpy.array

  • From pandas.DataFrame

  • From SQL Query

  • From a table

After that we will also look at the mathematical operators that are available:

  • pandas-Like

  • SQL-Like

Lastly, we will look at some examples of applications of functions that be applied directly on the VastFrame.

Let’s begin by importing vastorbit.

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

Dictionary

This is the most direct way to create a VastFrame:

vdf = vo.VastFrame(
    {
        "cats": ["A", "B", "C"],
        "reps": [2, 4, 8],
    },
)
Abc
cats
Varchar(1)
100%
123
reps
Integer
100%
1B4
2C8
3A2

Numpy Array

We can also use a numpy.array:

import numpy as np

vdf = vo.VastFrame(
    np.array(
        [
            [1, 2, 3],
            [4, 5, 6],
            [7, 8, 9],
        ],
    ),
    usecols = [
        "col_A",
        "col_B",
        "col_C",
    ],
)
123
col_A
Integer
100%
123
col_B
Integer
100%
123
col_C
Integer
100%
1789
2456
3123

pandas DataFrame

We can also use a pandas.DataFrame object:

# Import pandas library
import pandas as pd

# Create the data dictionary
data = {
    'Name': ['John', 'Ali', 'Pheona'],
    'Age': [25, 30, 22],
    'City': ['New York', 'Gaza', 'Los Angeles'],
}

# Create the pandas DataFrame object
df = pd.DataFrame(data)

# Create a VastFrame
vdf = vo.VastFrame(df)
Abc
name
Varchar(50)
100%
123
age
Integer
100%
Abc
city
Varchar(50)
100%
1John25New York
2Ali30Gaza
3Pheona22Los Angeles

SQL Query

We can also use a SQL Query:

# Write a SQL Query to fetch three rows from the Titanic table
sql_query = "SELECT age, sex FROM default.titanic LIMIT 3;"

# Create a VastFrame
vdf = vo.VastFrame(sql_query)
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
Abc
sex
Varchar(20)
100%
123
age
Double
79%
123
sibsp
Integer
100%
123
parch
Integer
100%
Abc
ticket
Varchar(36)
100%
123
fare
Double
99%
Abc
cabin
Varchar(30)
22%
Abc
embarked
Varchar(20)
99%
Abc
boat
Varchar(100)
37%
123
body
Integer
9%
Abc
home.dest
Varchar(100)
56%
111Allen, Miss. Elisabeth Waltonfemale29.00024160211.3375B5S2[null]St Louis, MO
211Allison, Master. Hudson Trevormale0.9212113781151.55C22 C26S11[null]Montreal, PQ / Chesterville, ON
310Allison, Miss. Helen Lorainefemale2.012113781151.55C22 C26S[null][null]Montreal, PQ / Chesterville, ON
410Allison, Mr. Hudson Joshua Creightonmale30.012113781151.55C22 C26S[null]135Montreal, PQ / Chesterville, ON
510Allison, Mrs. Hudson J C (Bessie Waldo Daniels)female25.012113781151.55C22 C26S[null][null]Montreal, PQ / Chesterville, ON
611Anderson, Mr. Harrymale48.0001995226.55E12S3[null]New York, NY
711Andrews, Miss. Kornelia Theodosiafemale63.0101350277.9583D7S10[null]Hudson, NY
810Andrews, Mr. Thomas Jrmale39.0001120500.0A36S[null][null]Belfast, NI
911Appleton, Mrs. Edward Dale (Charlotte Lamson)female53.0201176951.4792C101SD[null]Bayside, Queens, NY
1010Artagaveytia, Mr. Ramonmale71.000PC 1760949.5042[null]C[null]22Montevideo, Uruguay
1110Astor, Col. John Jacobmale47.010PC 17757227.525C62 C64C[null]124New York, NY
1211Astor, Mrs. John Jacob (Madeleine Talmadge Force)female18.010PC 17757227.525C62 C64C4[null]New York, NY
1311Aubart, Mme. Leontine Paulinefemale24.000PC 1747769.3B35C9[null]Paris, France
1411Barber, Miss. Ellen "Nellie"female26.0001987778.85[null]S6[null][null]
1511Barkworth, Mr. Algernon Henry Wilsonmale80.0002704230.0A23SB[null]Hessle, Yorks
1610Baumann, Mr. John Dmale[null]00PC 1731825.925[null]S[null][null]New York, NY
1710Baxter, Mr. Quigg Edmondmale24.001PC 17558247.5208B58 B60C[null][null]Montreal, PQ
1811Baxter, Mrs. James (Helene DeLaudeniere Chaput)female50.001PC 17558247.5208B58 B60C6[null]Montreal, PQ
1911Bazzani, Miss. Albinafemale32.0001181376.2917D15C8[null][null]
2010Beattie, Mr. Thomsonmale36.0001305075.2417C6CA[null]Winnipeg, MN

Table

A table can also be directly ingested:

# Create a VastFrame from the titanic table in default schema
vdf = vo.VastFrame("titanic")
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
Abc
sex
Varchar(20)
100%
123
age
Double
79%
123
sibsp
Integer
100%
123
parch
Integer
100%
Abc
ticket
Varchar(36)
100%
123
fare
Double
99%
Abc
cabin
Varchar(30)
22%
Abc
embarked
Varchar(20)
99%
Abc
boat
Varchar(100)
37%
123
body
Integer
9%
Abc
home.dest
Varchar(100)
56%
111Allen, Miss. Elisabeth Waltonfemale29.00024160211.3375B5S2[null]St Louis, MO
211Allison, Master. Hudson Trevormale0.9212113781151.55C22 C26S11[null]Montreal, PQ / Chesterville, ON
310Allison, Miss. Helen Lorainefemale2.012113781151.55C22 C26S[null][null]Montreal, PQ / Chesterville, ON
410Allison, Mr. Hudson Joshua Creightonmale30.012113781151.55C22 C26S[null]135Montreal, PQ / Chesterville, ON
510Allison, Mrs. Hudson J C (Bessie Waldo Daniels)female25.012113781151.55C22 C26S[null][null]Montreal, PQ / Chesterville, ON
611Anderson, Mr. Harrymale48.0001995226.55E12S3[null]New York, NY
711Andrews, Miss. Kornelia Theodosiafemale63.0101350277.9583D7S10[null]Hudson, NY
810Andrews, Mr. Thomas Jrmale39.0001120500.0A36S[null][null]Belfast, NI
911Appleton, Mrs. Edward Dale (Charlotte Lamson)female53.0201176951.4792C101SD[null]Bayside, Queens, NY
1010Artagaveytia, Mr. Ramonmale71.000PC 1760949.5042[null]C[null]22Montevideo, Uruguay
1110Astor, Col. John Jacobmale47.010PC 17757227.525C62 C64C[null]124New York, NY
1211Astor, Mrs. John Jacob (Madeleine Talmadge Force)female18.010PC 17757227.525C62 C64C4[null]New York, NY
1311Aubart, Mme. Leontine Paulinefemale24.000PC 1747769.3B35C9[null]Paris, France
1411Barber, Miss. Ellen "Nellie"female26.0001987778.85[null]S6[null][null]
1511Barkworth, Mr. Algernon Henry Wilsonmale80.0002704230.0A23SB[null]Hessle, Yorks
1610Baumann, Mr. John Dmale[null]00PC 1731825.925[null]S[null][null]New York, NY
1710Baxter, Mr. Quigg Edmondmale24.001PC 17558247.5208B58 B60C[null][null]Montreal, PQ
1811Baxter, Mrs. James (Helene DeLaudeniere Chaput)female50.001PC 17558247.5208B58 B60C6[null]Montreal, PQ
1911Bazzani, Miss. Albinafemale32.0001181376.2917D15C8[null][null]
2010Beattie, Mr. Thomsonmale36.0001305075.2417C6CA[null]Winnipeg, MN

Mathematical Operators

We can use all the common mathematical operators on the VastFrame.

pandas-Like

First let us re-create a simple VastFrame:

vdf = vo.VastFrame(
    {
        "cats": ["A", "B", "C"],
        "reps": [2, 4, 8],
    },
)

In order to search for a specific string value of a specific column:

result = vdf[vdf["cats"] == "A"]
Abc
cats
Varchar(1)
100%
123
reps
Integer
100%
1C8
2B4

Similarly we can perform a mathematical operations as well for numerical columns:

result = vdf[vdf["reps"] > 2]
Abc
cats
Varchar(1)
100%
123
reps
Integer
100%
1C8
2B4

Both operators could also be combined:

result = vdf[vdf["reps"] > 2][vdf["cats"] == "C"]
Abc
cats
Varchar(1)
100%
123
reps
Integer
100%
1C8

We can also perform mathematical calculations on the elements inside the VastFrame quite conveniently:

vdf["new"] = abs(vdf["reps"] * 4 - 100)
Abc
cats
Varchar(1)
100%
123
reps
Integer
100%
123
new
Integer
100%
1C868
2B484
3A292

SQL-Like

SQL queries can be directly applied on the VastFrame using StringSQL. This adds a new level of flexibility to the VastFrame. StringSQL allows the user to generate formatted SQL queries in a string form. Since any SQL query in string format can be passed to the VastFrame, you can seamlessly pass the output of StringSQL directly to the VastFrame.

# Create the SQL Query using StringSQL
sql_query = vo.StringSQL("reps > 2")

# Get the output as a VastFrame
result = vdf[sql_query]
countmeanstdminapprox_25%approx_50%approx_75%max
"pclass"1309.02.2948815889992360.83783601897012721.02.03.03.03.0
"survived"1309.00.38197097020626430.48605517086648320.00.00.01.01.0
"age"1046.029.88113766730401414.4134932112713270.1720.9053026791508328.176473812090138.5731092436974880.0
"sibsp"1309.00.49885408708938121.04165839059610190.00.00.01.08.0
"parch"1309.00.38502673796791440.8655602753495150.00.00.00.09.0
"fare"1308.033.2954792813456551.7586682391741850.07.92011339130434814.51723166666666631.24888996860731512.3292
"body"121.0160.809917355371997.696921996003081.073.0157.0255.0328.0

Note

Have a look at StringSQL for more details.

Another example of a slightly advanced SQL Query could be:

# Create the SQL Query using StringSQL
sql_query = vo.StringSQL("reps BETWEEN 3 AND 8 AND cats = 'B'")

# Get the output as a VastFrame
result = vdf[sql_query]
Abc
cats
Varchar(1)
100%
123
reps
Integer
100%
123
new
Integer
100%
1B484

Direct Functions

There are many methods that can be directly used by VastFrame. Let us look at how conveiently we can call them. Here is an example of the VastFrame.describe() method:

# Import the dataset
from vastorbit.datasets import load_titanic

# Create VastFrame
vdf = load_titanic()

# Summarize the VastFrame
vdf.describe()
countmeanstdminapprox_25%approx_50%approx_75%max
"pclass"1309.02.2948815889992360.83783601897012721.02.03.03.03.0
"survived"1309.00.38197097020626430.48605517086648320.00.00.01.01.0
"age"1046.029.88113766730401414.4134932112713270.1720.9053026791508328.176473812090138.5731092436974880.0
"sibsp"1309.00.49885408708938121.04165839059610190.00.00.01.08.0
"parch"1309.00.38502673796791440.8655602753495150.00.00.00.09.0
"fare"1308.033.2954792813456551.7586682391741850.07.92011339130434814.51723166666666631.24888996860731512.3292
"body"121.0160.809917355371997.696921996003081.073.0157.0255.0328.0

Note

Explore VastFrame and VastColumn different methods to see more examples.

See also

VastColumn : Columns of VastFrame object.

VastColumn - Column operations and transformations

class vastorbit.VastColumn(alias: str, transformations: list | None = None, parent: VastFrame | None = None, catalog: dict | None = None)

Python object that stores all user transformations. If the VastFrame represents the entire relation, a VastColumn can be seen as one column of that relation. Through its abstractions, VastColumn simplify several processes.

Parameters:

  • alias (str) – VastColumn alias.

  • transformations (list, optional) – List of the different transformations. Each transformation must be similar to the following: (function, type, category)

  • parent (VastFrame, optional) – Parent of the VastColumn. One VastFrame can have multiple children VastColumn, whereas one VastColumn can only have one parent.

  • catalog (dict, optional) – Catalog where each key corresponds to an aggregation. VastColumn will memorize the already computed aggregations to increase performance. The catalog is updated when the parent VastFrame is modified.

Variables:

  • str (transformations,) – VastColumn alias.

  • dict (catalog,) – Catalog of pre-computed aggregations.

  • VastFrame (parent,) – Parent of the VastColumn.

  • str – List of the different transformations.

Examples

Let’s begin by importing vastorbit.

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

Let’s create a VastFrame with two VastColumn:

vdf = vo.VastFrame(
    {
        "cats": ["A", "B", "C"],
        "reps": [2, 4, 8],
    },
)
Abc
cats
Varchar(1)
100%
123
reps
Integer
100%
1B4
2C8
3A2

“cats” and “reps” are VastColumn objects. They can be accessed the same way as a dictionary or a pandas.DataFrame. They represent the columns of the entire relation.

For example, the following code will access the VastColumn “cats”:

vdf["cats"]

Note

VastColumn are columns inside a VastFrame; they have their own methods but cannot exist without a parent VastFrame. Please refer to VastFrame to see an entire example.

See also

VastFrame : Main vastorbit dataset object.

Visualization

Create interactive charts with Plotly or Matplotlib backends.

General Plots

bar(columns[, method, of, max_cardinality, ...])

Draws the bar chart of the input VastColumns based on an aggregation.

barh(columns[, method, of, max_cardinality, ...])

Draws the horizontal bar chart of the input VastColumns based on an aggregation.

boxplot([columns, q, max_nb_fliers, whis, chart])

Draws the Box Plot of the input VastColumns.

contour(columns, func[, nbins, chart])

Draws the contour plot of the input function using two input VastColumns.

density([columns, nbins, chart])

Draws the VastColumns Density Plot using histogram approximation.

heatmap(columns[, method, of, h, chart])

Draws the Heatmap of the two input VastColumns.

hexbin(columns[, method, of, bbox, img, chart])

Draws the Hexbin of the input VastColumns based on an aggregation.

hist(columns[, method, of, h, chart])

Draws the histograms of the input VastColumns based on an aggregation.

outliers_plot(columns[, threshold, ...])

Draws the global outliers plot of one or two columns based on their ZSCORE.

pie(columns[, method, of, max_cardinality, ...])

Draws the nested pie chart of the input VastColumns.

pivot_table(columns[, method, of, ...])

Draws the pivot table of one or two columns based on an aggregation.

plot(ts[, columns, start_date, end_date, ...])

Draws the time series.

scatter(columns[, by, size, cmap_col, ...])

Draws the scatter plot of the input VastColumns.

scatter_matrix([columns, max_nb_points])

Draws the scatter matrix of the VastFrame.

pivot_table_chi2(response[, columns, nbins, ...])

Returns the chi-square term using the pivot table of the response VastColumn against the input VastColumn.

range_plot(columns, ts[, q, start_date, ...])

Draws the range plot of the input VastColumns.

bar([method, of, max_cardinality, nbins, h, ...])

Draws the bar chart of the VastColumn based on an aggregation.

barh([method, of, max_cardinality, nbins, ...])

Draws the horizontal bar chart of the VastColumn based on an aggregation.

candlestick(ts[, method, q, start_date, ...])

Draws the Time Series of the VastColumn.

boxplot([by, q, h, max_cardinality, ...])

Draws the box plot of the VastColumn.

density([by, nbins, chart])

Draws the VastColumn Density Plot using histogram approximation.

hist([by, method, of, h, h_by, ...])

Draws the histogram of the input VastColumn based on an aggregation.

pie([method, of, max_cardinality, h, kind, ...])

Draws the pie chart of the VastColumn based on an aggregation.

plot(ts[, by, start_date, end_date, kind, chart])

Draws the Time Series of the VastColumn.

range_plot(ts[, q, start_date, end_date, ...])

Draws the range plot of the VastColumn.

spider([by, method, of, max_cardinality, h, ...])

Draws the spider plot of the input VastColumn based on an aggregation.

Animated Plots

animated_bar(ts, columns[, by, start_date, ...])

Draws the animated bar chart (bar race).

animated_pie(ts, columns[, by, start_date, ...])

Draws the animated pie chart.

animated_plot(ts[, columns, by, start_date, ...])

Draws the animated line plot.

animated_scatter(ts, columns[, by, ...])

Draws the animated scatter plot.

Plotting Backend Classes

Advanced: Internal Plotting Classes

ACFPlot(*args, **kwargs)

BarChart(*args, **kwargs)

BarChart2D(*args, **kwargs)

BoxPlot(*args, **kwargs)

CandleStick(*args, **kwargs)

ChampionChallengerPlot(*args, **kwargs)

ContourPlot(*args, **kwargs)

CutoffCurve(*args, **kwargs)

DensityPlot(*args, **kwargs)

ElbowCurve(*args, **kwargs)

HeatMap(*args, **kwargs)

Histogram(*args, **kwargs)

HorizontalBarChart(*args, **kwargs)

HorizontalBarChart2D(*args, **kwargs)

ImportanceBarChart(*args, **kwargs)

LiftChart(*args, **kwargs)

LinePlot(*args, **kwargs)

LogisticRegressionPlot(*args, **kwargs)

LOFPlot(*args, **kwargs)

MultiDensityPlot(*args, **kwargs)

MultiLinePlot(*args, **kwargs)

NestedPieChart(*args, **kwargs)

OutliersPlot(*args, **kwargs)

PCACirclePlot(*args, **kwargs)

PieChart(*args, **kwargs)

PlotlyBase(*args, **kwargs)

Plotly Base Class.

PRCCurve(*args, **kwargs)

RangeCurve(*args, **kwargs)

RegressionPlot(*args, **kwargs)

RegressionTreePlot(*args, **kwargs)

ROCCurve(*args, **kwargs)

ScatterPlot(*args, **kwargs)

SpiderChart(*args, **kwargs)

StepwisePlot(*args, **kwargs)

SVMClassifierPlot(*args, **kwargs)

TSPlot(*args, **kwargs)

VoronoiPlot(*args, **kwargs)

ACFPlot(*args, **kwargs)

ACFPACFPlot(*args, **kwargs)

AnimatedBarChart(*args, **kwargs)

AnimatedBase(*args, **kwargs)

AnimatedBubblePlot(*args, **kwargs)

AnimatedLinePlot(*args, **kwargs)

AnimatedPieChart(*args, **kwargs)

BarChart(*args, **kwargs)

BarChart2D(*args, **kwargs)

BoxPlot(*args, **kwargs)

CandleStick(*args, **kwargs)

ChampionChallengerPlot(*args, **kwargs)

ContourPlot(*args, **kwargs)

CutoffCurve(*args, **kwargs)

DensityPlot(*args, **kwargs)

DensityPlot2D(*args, **kwargs)

ElbowCurve(*args, **kwargs)

HeatMap(*args, **kwargs)

Histogram(*args, **kwargs)

HorizontalBarChart(*args, **kwargs)

HorizontalBarChart2D(*args, **kwargs)

ImportanceBarChart(*args, **kwargs)

LiftChart(*args, **kwargs)

LinePlot(*args, **kwargs)

LogisticRegressionPlot(*args, **kwargs)

LOFPlot(*args, **kwargs)

MatplotlibBase(*args, **kwargs)

MultiDensityPlot(*args, **kwargs)

MultiLinePlot(*args, **kwargs)

NestedPieChart(*args, **kwargs)

OutliersPlot(*args, **kwargs)

PCACirclePlot(*args, **kwargs)

PieChart(*args, **kwargs)

PRCCurve(*args, **kwargs)

RangeCurve(*args, **kwargs)

RegressionPlot(*args, **kwargs)

RegressionTreePlot(*args, **kwargs)

ROCCurve(*args, **kwargs)

ScatterMatrix(*args, **kwargs)

ScatterPlot(*args, **kwargs)

SpiderChart(*args, **kwargs)

StepwisePlot(*args, **kwargs)

SVMClassifierPlot(*args, **kwargs)

TSPlot(*args, **kwargs)

VoronoiPlot(*args, **kwargs)

Descriptive Statistics

In-database aggregations and statistical summaries.

aad([columns])

Utilizes the aad (Average Absolute Deviation) aggregation method to analyze the VastColumn.

aggregate(func[, columns, ncols_block, ...])

Aggregates the VastFrame using the input functions.

all(columns, **agg_kwargs)

Applies the BOOL_AND aggregation method to the VastFrame.

any(columns, **agg_kwargs)

Uses the BOOL_OR aggregation method in the VastFrame.

avg([columns])

This operation aggregates the VastFrame using the AVG aggregation, which calculates the average value for the selected column or columns.

count([columns])

This operation aggregates the VastFrame using the COUNT aggregation, providing the count of non-missing values for specified columns.

count_percent([columns, sort_result, desc])

Performs aggregation on the VastFrame using a list of aggregate functions, including count and percent.

describe([method, columns, unique, ...])

This function aggregates the VastFrame using multiple statistical aggregations such as minimum (min), maximum (max), median, cardinality (unique), and other relevant statistics.

duplicated([columns, count, limit])

This function returns a list or set of values that occur more than once within the dataset.

kurtosis([columns])

Calculates the kurtosis of the VastFrame to obtain a measure of the data's peakedness or tailness.

mad([columns])

Utilizes the mad (Median Absolute Deviation) aggregation method with the VastFrame.

max([columns])

Aggregates the VastFrame by applying the MAX aggregation, which calculates the maximum value, for the specified columns.

median([columns, approx])

Aggregates the VastFrame using the MEDIAN or APPROX_MEDIAN aggregation, which calculates the median value for the specified columns.

min([columns])

Aggregates the VastFrame by applying the MIN aggregation, which calculates the minimum value, for the specified columns.

nunique([columns, approx])

When aggregating the VastFrame using nunique (cardinality), vastorbit employs the COUNT DISTINCT function to determine the number of unique values in a particular column.

product([columns])

Aggregates the VastFrame by applying the product aggregation function.

quantile(q[, columns, approx])

Aggregates the VastFrame using specified quantile.

score(y_true, y_score, metric)

Computes the score using the input columns and the input metric.

sem([columns])

Leverages the sem (Standard Error of the Mean) aggregation technique to perform analysis and aggregation on the VastFrame.

skewness([columns])

Utilizes the skewness aggregation method to analyze and aggregate the VastFrame.

std([columns])

Aggregates the VastFrame using STDDEV aggregation (Standard Deviation), providing insights into the spread or variability of data for the selected columns.

sum([columns])

Aggregates the VastFrame using SUM aggregation, which computes the total sum of values for the specified columns, providing a cumulative view of numerical data.

var([columns])

Aggregates the VastFrame using VAR aggregation (Variance), providing insights into the spread or variability of data for the selected columns.

aad()

Utilizes the aad (Average Absolute Deviation) aggregation method to analyze the VastColumn.

aggregate(func)

Aggregates the VastFrame using the input functions.

avg()

This operation aggregates the VastFrame using the AVG aggregation, which calculates the average value for the input column.

count()

This operation aggregates the VastFrame using the COUNT aggregation, providing the count of non-missing values for the input column.

describe([method, max_cardinality, numcol])

This function aggregates the VastColumn using multiple statistical aggregations such as minimum (min), maximum (max), median, cardinality (unique), and other relevant statistics.

distinct(**kwargs)

This function returns the distinct categories or unique values within a VastColumn.

kurtosis()

Calculates the kurtosis of the VastColumn to obtain a measure of the data's peakedness or tailness.

mad()

Utilizes the mad (Median Absolute Deviation) aggregation method with the VastFrame.

max()

Aggregates the VastFrame by applying the 'MAX' aggregation, which calculates the maximum value, for the input column.

median([approx])

Aggregates the VastFrame using the MEDIAN or APPROX_MEDIAN aggregation, which calculates the median value for the specified columns.

min()

Aggregates the VastFrame by applying the MIN aggregation, which calculates the minimum value, for the input column.

mode([dropna, n])

This function returns the nth most frequently occurring element in the VastColumn.

nlargest([n])

Returns the n largest VastColumn elements.

nsmallest([n])

Returns the n smallest elements in the VastColumn.

nunique([approx])

When aggregating the VastFrame using nunique (cardinality), vastorbit employs the COUNT DISTINCT function to determine the number of unique values in particular columns.

product()

Aggregates the VastColumn by applying the product aggregation function.

quantile(q[, approx])

Aggregates the VastColumn using a specified quantile.

sem()

Leverages the sem (Standard Error of the Mean) aggregation technique to perform analysis and aggregation on the VastColumn.

skewness()

Utilizes the skewness aggregation method to analyze and aggregate the VastColumn.

std()

Aggregates the VastFrame using STDDEV aggregation (Standard Deviation), providing insights into the spread or variability of data for the input column.

sum()

Aggregates the VastFrame using SUM aggregation, which computes the total sum of values for the specified columns, providing a cumulative view of numerical data.

topk([k, dropna])

This function returns the k most frequently occurring elements in a column, along with their distribution expressed as percentages.

value_counts([k])

This function returns the k most frequently occurring elements in a column, along with information about how often they occur.

var()

Aggregates the VastFrame using VAR aggregation (Variance), providing insights into the spread or variability of data for the input column.

Correlation & Dependencies

General Correlation

acf(column, ts[, by, p, unit, method, ...])

Calculates the correlations between the specified VastColumn and its various time lags.

corr([columns, method, mround, focus, show, ...])

Calculates the Correlation Matrix for the VastFrame.

corr_pvalue(column1, column2[, method])

Computes the Correlation Coefficient between two input VastColumns, along with its associated p-value.

cov([columns, focus, show, chart])

Computes the covariance matrix of the VastFrame.

iv_woe(y[, columns, nbins, show, chart])

Calculates the Information Value (IV) Table, a powerful tool for assessing the predictive capability of an independent variable concerning a dependent variable.

pacf(column, ts[, by, p, unit, method, ...])

Computes the partial autocorrelations of the specified VastColumn.

regr([columns, method, show, chart])

Calculates the regression matrix for the given VastFrame.

iv_woe(y[, nbins])

Calculates the Information Value (IV) / Weight Of Evidence (WOE) Table.

Time Series Analysis

acf(column, ts[, by, p, unit, method, ...])

Calculates the correlations between the specified VastColumn and its various time lags.

pacf(column, ts[, by, p, unit, method, ...])

Computes the partial autocorrelations of the specified VastColumn.

Data Preprocessing

Encoding

case_when(name, *args)

Creates a new feature by evaluating on provided conditions.

one_hot_encode([columns, max_cardinality, ...])

Encodes the VastColumns using the One Hot Encoding algorithm.

cut(breaks[, labels, include_lowest, right])

Discretizes the VastColumn using the input list.

decode(*args)

Encodes the VastColumn using a user-defined encoding.

discretize([method, h, nbins, k, ...])

Discretizes the VastColumn using the input method.

label_encode()

Encodes the VastColumn using a bijection from the different categories to [0, n - 1] (n being the VastColumn cardinality).

mean_encode(response)

Encodes the VastColumn using the average of the response partitioned by the different VastColumn categories.

one_hot_encode([prefix, prefix_sep, ...])

Encodes the VastColumn with the One-Hot Encoding algorithm.

Missing Values

dropna([columns])

Filters the specified VastColumns in a VastFrame for missing values.

fillna([val, method, numeric_only])

Fills missing elements in VastColumn using specific rules.

interpolate(ts, rule[, method, by])

Computes a regular time interval VastFrame by interpolating the missing values using different techniques.

dropna()

Filters the VastFrame where the VastColumn is missing.

fillna([val, method, expr, by, order_by])

Fills missing elements in the VastColumn with a user-specified rule.

Duplicate Values

drop_duplicates([columns])

Filters the duplicates using a partition by the input VastColumns.

Normalization & Outliers

outliers([columns, name, threshold, robust])

Adds a new VastColumn labeled with 0 or 1, where 1 indicates that the record is a global outlier.

scale([columns, method])

Scales the input VastColumns using the input method.

clip([lower, upper])

Clips the VastColumn by transforming the values less than the lower bound to the lower bound value and the values higher than the upper bound to the upper bound value.

fill_outliers([method, threshold, ...])

Fills the VastColumns outliers using the input method.

normalize([method, by, return_trans])

Scales the input VastColumns using the input method.

Data Type Conversion

astype(dtype)

Converts the VastColumns to the input types.

bool_to_int()

Converts all booleans VastColumns to integers.

astype(dtype)

Converts the VastColumn to the input type.

Formatting

format_colnames(*args[, columns, ...])

Method used to format the input columns by using the VastFrame columns' names.

get_match_index(x, col_list[, str_check])

Returns the matching index.

is_colname_in(column)

Method used to check if the input column name is used by the VastFrame.

merge_similar_names(skip_word)

Merges columns with similar names.

astype(dtype)

Converts the VastColumn to the input type.

rename(new_name[, inplace])

Renames the VastColumn.

Train/Test Split

train_test_split([test_size, order_by, ...])

Creates two VastFrames (train/test), which can be used to evaluate a model.

Working with Weights

add_duplicates(weight[, use_gcd])

Duplicates the VastFrame using the input weight.

Complete Disjunctive Table

cdt([columns, max_cardinality, nbins, tcdt, ...])

Returns the complete disjunctive table of the VastFrame.

Feature Engineering

Analytic Functions

analytic(func[, columns, by, order_by, ...])

Adds a new VastColumn to the VastFrame by using an advanced analytical function on one or two specific VastColumns.

interpolate(ts, rule[, method, by])

Computes a regular time interval VastFrame by interpolating the missing values using different techniques.

sessionize(ts[, by, session_threshold, name])

Adds a new VastColumn to the VastFrame that corresponds to sessions (user activity during a specific time).

Custom Feature Creation

case_when(name, *args)

Creates a new feature by evaluating on provided conditions.

eval(name, expr)

Evaluates a customized expression.

Feature Transformations

abs([columns])

Applies the absolute value function to all input VastColumns.

apply(func)

Applies each function of the dictionary to the input VastColumns.

applymap(func[, numeric_only])

Applies a function to all VastColumns.

polynomial_comb([columns, r])

Returns a VastFrame containing the different product combinations of the input VastColumn.

swap(column1, column2)

Swap the two input VastColumns.

abs()

Applies the absolute value function to the input VastColumn.

add(x)

Adds the input element to the VastColumn.

apply(func[, copy_name])

Applies a function to the VastColumn.

apply_fun(func[, x])

Applies a default function to the VastColumn.

date_part(field)

Extracts a specific TS field from the VastColumn (only if the VastColumn type is date like).

div(x)

Divides the VastColumn by the input element.

mul(x)

Multiplies the VastColumn by the input element.

round(n)

Rounds the VastColumn by keeping only the input number of digits after the decimal point.

slice(length[, unit, start])

Slices and transforms the VastColumn using a time series rule.

sub(x)

Subtracts the input element from the VastColumn.

Moving Windows

cummax(column[, by, order_by, name])

Adds a new VastColumn to the VastFrame by computing the cumulative maximum of the input VastColumn.

cummin(column[, by, order_by, name])

Adds a new VastColumn to the VastFrame by computing the cumulative minimum of the input VastColumn.

cumprod(column[, by, order_by, name])

Adds a new VastColumn to the VastFrame by computing the cumulative product of the input VastColumn.

cumsum(column[, by, order_by, name])

Adds a new VastColumn to the VastFrame by computing the cumulative sum of the input VastColumn.

rolling(func, window, columns[, by, ...])

Adds a new VastColumn to the VastFrame by using an advanced analytical window function on one or two specific VastColumn.

Text Operations

regexp(column, pattern[, method, position, ...])

Computes a new VastColumn based on regular expressions.

str_contains(pat)

Verifies if the regular expression is in each of the VastColumn records.

str_count(pat)

Computes the number of matches for the regular expression in each record of the VastColumn.

str_extract(pat)

Extracts the regular expression in each record of the VastColumn.

str_replace(to_replace[, value])

Replaces the regular expression matches in each of the VastColumn record by an input value.

str_slice(start, step)

Slices the VastColumn.

Binary Operators

add(x)

Adds the input element to the VastColumn.

div(x)

Divides the VastColumn by the input element.

mul(x)

Multiplies the VastColumn by the input element.

sub(x)

Subtracts the input element from the VastColumn.

Feature Selection

chaid(response, columns[, nbins, method])

Returns a CHAID (Chi-square Automatic Interaction Detector) tree.

chaid_columns([columns, max_cardinality])

Function used to simplify the code.

Join, Sort & Transform

append(input_relation[, expr1, expr2, union_all])

Merges the VastFrame with another VastFrame or an input relation, and returns a new VastFrame.

copy()

Returns a deep copy of the VastFrame.

groupby(columns[, expr, rollup, having])

This method facilitates the aggregation of the VastFrame by grouping its elements based on one or more specified criteria.

join(input_relation[, on, how, expr1, ...])

Joins the VastFrame with another one or an input_relation.

narrow(index[, columns, col_name, val_name])

Returns the Narrow Table of the VastFrame using the input VastColumns.

pivot(index, columns, values[, aggr, prefix])

Returns the Pivot of the VastFrame using the input aggregation.

recommend(unique_id, item_id[, method, ...])

Recommend items based on the Collaborative Filtering (CF) technique.

sort(columns)

Sorts the VastFrame using the input VastColumn.

add_copy(name)

Adds a copy VastColumn to the parent VastFrame.

Filter & Sample

Sample

sample([n, x, method, by])

Downsamples the input VastFrame.

Balance

balance(column[, method, x, order_by])

Balances the dataset using the input method.

Filter Columns

drop([columns])

Drops the input VastColumns from the VastFrame.

select(columns)

Returns a copy of the VastFrame with only the selected VastColumn.

drop([add_history])

Drops the VastColumn from the VastFrame.

drop_outliers([threshold, use_threshold, alpha])

Drops outliers in the VastColumn.

Filter Records

at_time(ts, time)

Filters the VastFrame by only keeping the records at the input time.

between(column[, start, end, inplace])

Filters the VastFrame by only keeping the records between two input elements.

between_time(ts[, start_time, end_time, inplace])

Filters the VastFrame by only keeping the records between two input times.

filter([conditions])

Filters the VastFrame using the input expressions.

first(ts, offset)

Filters the VastFrame by only keeping the first records.

isin(val)

Checks whether specific records are in the VastFrame and returns the new VastFrame of the search.

last(ts, offset)

Filters the VastFrame by only keeping the last records.

isin(val, *args)

Checks whether specific records are in the VastColumn and returns the new VastFrame of the search.

Serialization & Export

General Formats

to_csv([path, sep, na_rep, quotechar, ...])

Creates a CSV file or folder of CSV files of the current VastFrame relation.

to_json([path, usecols, order_by, n_files])

Creates a JSON file or folder of JSON files of the current VastFrame relation.

In-Memory Objects

to_numpy()

Converts the VastFrame to a numpy.array.

to_pandas()

Converts the VastFrame to a pandas.DataFrame.

to_list()

Converts the VastFrame to a Python list.

to_geopandas(geometry)

Converts the VastFrame to a Geopandas DataFrame.

Databases

to_db(name[, usecols, relation_type, ...])

Saves the VastFrame current relation to the VAST database.

Binary Formats

to_pickle(name)

Saves the VastFrame to a Python pickle file.

Utilities & Information

Information

catcol([max_cardinality])

Returns the VastFrame categorical VastColumns.

current_relation([reindent, split])

Returns the current VastFrame relation.

datecol()

Returns a list of the VastColumns of type date in the VastFrame.

dtypes()

Returns the different VastColumns types.

empty()

Returns True if the VastFrame is empty.

explain([analyze, verbose])

Provides information on how Trino is computing the current VastFrame relation.

get_columns([exclude_columns])

Returns the VastFrame VastColumns.

head([limit])

Returns the VastFrame head.

idisplay()

This method displays the interactive table.

iloc([limit, offset, columns])

Returns a part of the VastFrame (delimited by an offset and a limit).

info()

Displays information about the different VastFrame transformations.

memory_usage()

Returns the VastFrame memory usage.

expected_store_usage([unit])

Returns the VastFrame expected store usage.

numcol([exclude_columns])

Returns a list of names of the numerical VastColumns in the VastFrame.

shape()

Returns the number of rows and columns of the VastFrame.

tail([limit])

Returns the tail of the VastFrame.

category()

Returns the category of the VastColumn.

ctype()

Returns the VastColumn DB type.

dtype()

Returns the VastColumn DB type.

get_len()

Returns a new VastColumn that represents the length of each element.

head([limit])

Returns the head of the VastColumn.

iloc([limit, offset])

Returns a part of the VastColumn (delimited by an offset and a limit).

isarray()

Returns True if the VastColumn is an array, False otherwise.

isbool()

Returns True if the VastColumn is boolean, False otherwise.

isdate()

Returns True if the VastColumn category is date, False otherwise.

isnum()

Returns True if the VastColumn is numerical, False otherwise.

memory_usage()

Returns the VastColumn memory usage.

store_usage()

Returns the VastColumn expected store usage (unit: b).

tail([limit])

Returns the tail of the VastColumn.

Management

del_catalog()

Deletes the current VastFrame catalog.

load([offset])

Loads a previous structure of the VastFrame.

save()

Saves the current structure of the VastFrame.