vastorbit.VastFrame.describe¶

VastFrame.describe(method: Literal['numerical', 'categorical', 'statistics', 'length', 'range', 'all', 'auto'] = 'auto', columns: Annotated[str | list[str], 'STRING representing one column or a list of columns'] | None = None, unique: bool = False, ncols_block: int = 20, processes: int = 1) → TableSample¶

This function aggregates the VastFrame using multiple statistical aggregations such as minimum (min), maximum (max), median, cardinality (unique), and other relevant statistics. The specific aggregations applied depend on the data types of the VastColumns. For example, numeric columns are aggregated with numerical aggregations (min, median, max…), while categorical columns are aggregated using categorical ones (cardinality, mode…). This versatile function provides valuable insights into the dataset’s statistical properties and can be customized to meet specific analytical requirements.

Note

This function can offer faster performance compared to the VastFrame.aggregate() method, as it leverages specialized and optimized backend functions.

Parameters:

method (str, optional) –
The describe method.
- all:
  Aggregates all statistics for all VastColumns. The exact method depends on the VastColumn type (numerical dtype: numerical; timestamp dtype: range; categorical dtype: length)
- auto:
  Sets the method to numerical if at least one VastColumn of the VastFrame is numerical, categorical otherwise.
- categorical:
  Uses only categorical aggregations.
- length:
  Aggregates the VastFrame using numerical aggregation on the length of all selected VastColumns.
- numerical:
  Uses only numerical descriptive statistics, which are computed faster than the aggregate method.
- range:
  Aggregates the VastFrame using multiple statistical aggregations - min, max, range…
- statistics:
  Aggregates the VastFrame using multiple statistical aggregations - kurtosis, skewness, min, max…
columns (SQLColumns, optional) – List of the VastColumns names. If empty, the VastColumns are selected depending on the parameter method.
unique (bool, optional) – If set to True, computes the cardinality of each element.
ncols_block (int, optional) – Number of columns used per query. Setting this parameter divides what would otherwise be one large query into many smaller queries called “blocks”, whose size is determined by the ncols_block parmeter.
processes (int, optional) – Number of child processes to create. Setting this with the ncols_block parameter lets you parallelize a single query into many smaller queries, where each child process creates its own connection to the database and sends one query. This can improve query performance, but consumes more resources. If processes is set to 1, the queries are sent iteratively from a single process.

Returns:

result.

Return type:

TableSample

Examples

For this example, we will use the following dataset:

import vastorbit as vo

data = vo.VastFrame(
    {
        "x": [1, 2, 4, 9, 10, 15, 20, 22],
        "y": [1, 2, 1, 2, 1, 1, 2, 1],
        "z": [10, 12, 2, 1, 9, 8, 1, 3],
        "c": ['A', 'A', 'A', 'A', 'B', 'B', 'C', 'D'],
    }
)

The describe method provides you with a variety of statistical methods.

The numerical parameter allows for the computation of numerical aggregations.

data.describe(
    columns = ["x", "y", "z"],
    method = "numerical",
)

	count	mean	std	min	approx_25%	approx_50%	approx_75%	max
"x"	8.0	10.375	8.016723591521485	1.0	4.0	10.0	20.0	22.0
"y"	8.0	1.375	0.5175491695067657	1.0	1.0	1.0	2.0	2.0
"z"	8.0	5.75	4.464142854857069	1.0	2.0	8.0	10.0	12.0

Rows: 1-3 | Columns: 9

The categorical parameter allows for the computation of categorical aggregations.

data.describe(
    columns = ["x", "y", "z", "c"],
    method = "categorical",
)

	dtype	count	top	top_percent
"x"	integer	8	1	12.5
"y"	integer	8	1	62.5
"z"	integer	8	1	25.0
"c"	varchar(1)	8	A	50.0

Rows: 1-4 | Columns: 5

The all parameter allows for the computation of both categorical and numerical aggregations.

data.describe(
    columns = ["x", "y", "z", "c"],
    method = "all",
)

	"x"	"y"	"z"	"c"
dtype	integer	integer	integer	varchar(1)
percent	100.0	100.0	100.0	100.0
count	8	8	8	8
top	10	1	1	A
top_percent	12.5	62.5	25.0	50.0
avg	10.375	1.375	5.75	1.0
stddev	8.016723591521485	0.5175491695067657	4.464142854857069	0.0
min	1	1	1	1
approx_25%	4	1	2	1
approx_50%	10	1	8	1
approx_75%	20	2	10	1
max	22	2	12	1
range	21	1	11	0
empty	[null]	[null]	[null]	0

Rows: 1-14 | Columns: 5

Note

Many other methods are available, and their cost in terms of computation can vary.

Note

All the calculations are pushed to the database.