vastorbit.VastFrame.merge_similar_names¶

VastFrame.merge_similar_names(skip_word: str | list[str]) → VastFrame¶

Merges columns with similar names. The function generates a COALESCE statement that merges the columns into a single column that excludes the input words. Note that the order of the variables in the COALESCE statement is based on the order of the ‘get_columns’ method.

Parameters:: skip_word (str | list, optional) – List of words to exclude from the provided column names. For example, if two columns are named ‘age.information.phone’ and ‘age.phone’ AND skip_word is set to ['.information'], then the two columns are merged together with the following COALESCE statement: COALESCE("age.phone", "age.information.phone") AS "age.phone"
Returns:: An object containing the merged element.
Return type:: VastFrame

Examples

Let’s begin by importing vastorbit.

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

For this example, let’s generate a dataset which has two columns that are duplicates with slight change in spelling and some missing values:

vdf = vo.VastFrame(
    {
        "user.id": [12, None, 13],
        "id": [12, 11, None],
    }
)

	123 user.id Integer	123 id Integer
1	[null]	11
2	13	[null]
3	12	12

Rows: 1-3 | Columns: 2

In order to remove the redundant column, we can combine them using merge_similar_names:

vdf.merge_similar_names(skip_word = "user.")

	123 id Integer
1	12
2	13
3	11

Rows: 1-3 | Column: id | Type: integer

Note

This function is particularly useful when flattening highly nested JSON files. Such files may contain redundant features and inconsistencies. The function is designed to merge these features, ensuring consistent information.