vastorbit.VastFrame.drop_duplicates¶

VastFrame.drop_duplicates(columns: Annotated[str | list[str], 'STRING representing one column or a list of columns'] | None = None) → VastFrame¶

Filters the duplicates using a partition by the input VastColumns.

Warning

Dropping duplicates will make the VastFrame structure heavier. It is recommended that you check the current structure using the current_relation method and save it using the to_db method, using the parameters inplace = True and relation_type = table.

Parameters:: columns (SQLColumns, optional) – List of the VastColumns names. If empty, all VastColumns are selected.
Returns:: self
Return type:: VastFrame

Examples

We import vastorbit:

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

For this example, we will use a dummy dataset with three columns:

vdf = vo.VastFrame(
    {
        "col1": [1, 2, 3, 1],
        "col2": [3, 3, 1, 3],
        "col":['a', 'b', 'v', 'a'],
    }
)

	123 col1 Integer	123 col2 Integer	Abc col Varchar(1)
1	2	3	b
2	3	1	v
3	1	3	a
4	1	3	a

Rows: 1-4 | Columns: 3

In the above dataset, notice that the first and last entries are identical i.e. duplicates.

Note

vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.

Using drop_duplicates we can take out any duplicates:

vdf.drop_duplicates()

	123 col1 Integer	123 col2 Integer	Abc col Varchar(1)
1	3	1	v
2	2	3	b
3	1	3	a

Rows: 3 | Columns: 3