vastorbit.VastColumn.fill_outliers¶

VastColumn.fill_outliers(method: Literal['winsorize', 'null', 'mean'] = 'winsorize', threshold: Annotated[int | float | Decimal, 'Python Numbers'] = 4.0, use_threshold: bool = True, alpha: Annotated[int | float | Decimal, 'Python Numbers'] = 0.05) → VastFrame¶

Fills the VastColumns outliers using the input method.

Parameters:

method (str, optional) –
Method used to fill the VastColumn outliers.
- mean:
  Replaces the upper and lower outliers by their respective average.
- null:
  Replaces the outliers by the NULL value.
- winsorize:
  If ‘use_threshold’ is set to False, clips the VastColumn using quantile(alpha) as lower bound and quantile(1-alpha) as upper bound; otherwise uses the lower and upper ZScores.
threshold (PythonNumber, optional) – Uses the Gaussian distribution to define the outliers. After normalizing the data (Z-Score), if the absolute value of the record is greater than the threshold, it will be considered as an outlier.
use_threshold (bool, optional) – Uses the threshold instead of the ‘alpha’ parameter.
alpha (PythonNumber, optional) – Number representing the outliers threshold. Values less than quantile(alpha) or greater than quantile(1-alpha) are filled.

Returns:

self._parent

Return type:

VastFrame

Examples

We import vastorbit:

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

For this example, we will use a dummy data that has one outlier:

vdf = vo.VastFrame({"vals": [20, 10, 0, -20, 10, 20, 1200]})

	123 vals Integer
1	1200
2	20
3	10
4	10
5	-20
6	20
7	0

Rows: 1-7 | Column: vals | Type: integer

Note

vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.

We can see that there are some extreme values in the data. We may need to remove those values. For this we can use the fill_outliers function.

vdf["vals"].fill_outliers(method = "null", threshold = 1)

	123 vals Integer
1	10
2	20
3	-20
4	[null]
5	10
6	0
7	20

Rows: 1-7 | Column: vals | Type: integer

Note

We can use either the alpha parameter or the z-score threshold parameter. By default it uses the threshold.