vastorbit.VastColumn.normalize

VastColumn.normalize(method: Literal['zscore', 'robust_zscore', 'minmax'] = 'zscore', by: Annotated[str | list[str], 'STRING representing one column or a list of columns'] | None = None, return_trans: bool = False) → VastFrame

Scales the input VastColumns using the input method.

Parameters:

• method (str, optional) –

  Method used to scale the data.

  • zscore:

    Normalization using the Z-Score.

    \[Z_{score}(x) = (x - x_{avg}) / x_{std}\]

  • robust_zscore:

    Normalization using the Robust Z-Score.

    \[Z_{rscore}(x) = (x - x_{med}) / (1.4826 * x_{mad})\]

  • minmax:

    Normalization using the MinMax.

    \[Z_{minmax}(x) = (x - x_{min}) / (x_{max} - x_{min})\]

• by (SQLColumns, optional) – VastColumns used in the partition.

• return_trans (bool, optimal) – If set to True, the method returns the transformation used instead of the parent VastFrame. This parameter is used for testing purposes.

Returns:

self._parent

Return type:

VastFrame

Examples

We import vastorbit:

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

For this example, we will use the Titanic dataset.

import vastorbit.datasets as vod

data = vod.load_titanic()

	123 pclass Integer	123 survived Integer	Abc name Varchar(164)	Abc sex Varchar(20)	123 age Double	123 sibsp Integer	123 parch Integer	Abc ticket Varchar(36)	123 fare Double	Abc cabin Varchar(30)	Abc embarked Varchar(20)	Abc boat Varchar(100)	123 body Integer	Abc home.dest Varchar(100)
1	3	1	McCormack, Mr. Thomas Joseph	male	[null]	0	0	367228	7.75	[null]	Q	[null]	[null]	[null]
2	3	1	McCoy, Miss. Agnes	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
3	3	1	McCoy, Miss. Alicia	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
4	3	1	McCoy, Mr. Bernard	male	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
5	3	1	McDermott, Miss. Brigdet Delia	female	[null]	0	0	330932	7.7875	[null]	Q	13	[null]	[null]
6	3	0	McEvoy, Mr. Michael	male	[null]	0	0	36568	15.5	[null]	Q	[null]	[null]	[null]
7	3	1	McGovern, Miss. Mary	female	[null]	0	0	330931	7.8792	[null]	Q	13	[null]	[null]
8	3	1	McGowan, Miss. Anna "Annie"	female	15.0	0	0	330923	8.0292	[null]	Q	[null]	[null]	[null]
9	3	0	McGowan, Miss. Katherine	female	35.0	0	0	9232	7.75	[null]	Q	[null]	[null]	[null]
10	3	0	McMahon, Mr. Martin	male	[null]	0	0	370372	7.75	[null]	Q	[null]	[null]	[null]
11	3	0	McNamee, Mr. Neal	male	24.0	1	0	376566	16.1	[null]	S	[null]	[null]	[null]
12	3	0	McNamee, Mrs. Neal (Eileen O'Leary)	female	19.0	1	0	376566	16.1	[null]	S	[null]	53	[null]
13	3	0	McNeill, Miss. Bridget	female	[null]	0	0	370368	7.75	[null]	Q	[null]	[null]	[null]
14	3	0	Meanwell, Miss. (Marion Ogden)	female	[null]	0	0	SOTON/O.Q. 392087	8.05	[null]	S	[null]	[null]	[null]
15	3	0	Meek, Mrs. Thomas (Annie Louise Rowley)	female	[null]	0	0	343095	8.05	[null]	S	[null]	[null]	[null]
16	3	0	Meo, Mr. Alfonzo	male	55.5	0	0	A.5. 11206	8.05	[null]	S	[null]	201	[null]
17	3	0	Mernagh, Mr. Robert	male	[null]	0	0	368703	7.75	[null]	Q	[null]	[null]	[null]
18	3	1	Midtsjo, Mr. Karl Albert	male	21.0	0	0	345501	7.775	[null]	S	15	[null]	[null]
19	3	0	Miles, Mr. Frank	male	[null]	0	0	359306	8.05	[null]	S	[null]	[null]	[null]
20	3	0	Mineff, Mr. Ivan	male	24.0	0	0	349233	7.8958	[null]	S	[null]	[null]	[null]
21	3	0	Minkoff, Mr. Lazar	male	21.0	0	0	349211	7.8958	[null]	S	[null]	[null]	[null]
22	3	0	Mionoff, Mr. Stoytcho	male	28.0	0	0	349207	7.8958	[null]	S	[null]	[null]	[null]
23	3	0	Mitkoff, Mr. Mito	male	[null]	0	0	349221	7.8958	[null]	S	[null]	[null]	[null]
24	3	1	Mockler, Miss. Helen Mary "Ellie"	female	[null]	0	0	330980	7.8792	[null]	Q	16	[null]	[null]
25	3	0	Moen, Mr. Sigurd Hansen	male	25.0	0	0	348123	7.65	F G73	S	[null]	309	[null]
26	3	1	Moor, Master. Meier	male	6.0	0	1	392096	12.475	E121	S	14	[null]	[null]
27	3	1	Moor, Mrs. (Beila)	female	27.0	0	1	392096	12.475	E121	S	14	[null]	[null]
28	3	0	Moore, Mr. Leonard Charles	male	[null]	0	0	A4. 54510	8.05	[null]	S	[null]	[null]	[null]
29	3	1	Moran, Miss. Bertha	female	[null]	1	0	371110	24.15	[null]	Q	16	[null]	[null]
30	3	0	Moran, Mr. Daniel J	male	[null]	1	0	371110	24.15	[null]	Q	[null]	[null]	[null]
31	3	0	Moran, Mr. James	male	[null]	0	0	330877	8.4583	[null]	Q	[null]	[null]	[null]
32	3	0	Morley, Mr. William	male	34.0	0	0	364506	8.05	[null]	S	[null]	[null]	[null]
33	3	0	Morrow, Mr. Thomas Rowan	male	[null]	0	0	372622	7.75	[null]	Q	[null]	[null]	[null]
34	3	1	Moss, Mr. Albert Johan	male	[null]	0	0	312991	7.775	[null]	S	B	[null]	[null]
35	3	1	Moubarek, Master. Gerios	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
36	3	1	Moubarek, Master. Halim Gonios ("William George")	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
37	3	1	Moubarek, Mrs. George (Omine "Amenia" Alexander)	female	[null]	0	2	2661	15.2458	[null]	C	C	[null]	[null]
38	3	1	Moussa, Mrs. (Mantoura Boulos)	female	[null]	0	0	2626	7.2292	[null]	C	[null]	[null]	[null]
39	3	0	Moutal, Mr. Rahamin Haim	male	[null]	0	0	374746	8.05	[null]	S	[null]	[null]	[null]
40	3	1	Mullens, Miss. Katherine "Katie"	female	[null]	0	0	35852	7.7333	[null]	Q	16	[null]	[null]
41	3	1	Mulvihill, Miss. Bertha E	female	24.0	0	0	382653	7.75	[null]	Q	15	[null]	[null]
42	3	0	Murdlin, Mr. Joseph	male	[null]	0	0	A./5. 3235	8.05	[null]	S	[null]	[null]	[null]
43	3	1	Murphy, Miss. Katherine "Kate"	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
44	3	1	Murphy, Miss. Margaret Jane	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
45	3	1	Murphy, Miss. Nora	female	[null]	0	0	36568	15.5	[null]	Q	16	[null]	[null]
46	3	0	Myhrman, Mr. Pehr Fabian Oliver Malkolm	male	18.0	0	0	347078	7.75	[null]	S	[null]	[null]	[null]
47	3	0	Naidenoff, Mr. Penko	male	22.0	0	0	349206	7.8958	[null]	S	[null]	[null]	[null]
48	3	1	Najib, Miss. Adele Kiamie "Jane"	female	15.0	0	0	2667	7.225	[null]	C	C	[null]	[null]
49	3	1	Nakid, Miss. Maria ("Mary")	female	1.0	0	2	2653	15.7417	[null]	C	C	[null]	[null]
50	3	1	Nakid, Mr. Sahid	male	20.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
51	3	1	Nakid, Mrs. Said (Waika "Mary" Mowad)	female	19.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
52	3	0	Nancarrow, Mr. William Henry	male	33.0	0	0	A./5. 3338	8.05	[null]	S	[null]	[null]	[null]
53	3	0	Nankoff, Mr. Minko	male	[null]	0	0	349218	7.8958	[null]	S	[null]	[null]	[null]
54	3	0	Nasr, Mr. Mustafa	male	[null]	0	0	2652	7.2292	[null]	C	[null]	[null]	[null]
55	3	0	Naughton, Miss. Hannah	female	[null]	0	0	365237	7.75	[null]	Q	[null]	[null]	[null]
56	3	0	Nenkoff, Mr. Christo	male	[null]	0	0	349234	7.8958	[null]	S	[null]	[null]	[null]
57	3	1	Nicola-Yarred, Master. Elias	male	12.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
58	3	1	Nicola-Yarred, Miss. Jamila	female	14.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
59	3	0	Nieminen, Miss. Manta Josefina	female	29.0	0	0	3101297	7.925	[null]	S	[null]	[null]	[null]
60	3	0	Niklasson, Mr. Samuel	male	28.0	0	0	363611	8.05	[null]	S	[null]	[null]	[null]
61	3	1	Nilsson, Miss. Berta Olivia	female	18.0	0	0	347066	7.775	[null]	S	D	[null]	[null]
62	3	1	Nilsson, Miss. Helmina Josefina	female	26.0	0	0	347470	7.8542	[null]	S	13	[null]	[null]
63	3	0	Nilsson, Mr. August Ferdinand	male	21.0	0	0	350410	7.8542	[null]	S	[null]	[null]	[null]
64	3	0	Nirva, Mr. Iisakki Antino Aijo	male	41.0	0	0	SOTON/O2 3101272	7.125	[null]	S	[null]	[null]	Finland Sudbury, ON
65	3	1	Niskanen, Mr. Juha	male	39.0	0	0	STON/O 2. 3101289	7.925	[null]	S	9	[null]	[null]
66	3	0	Nosworthy, Mr. Richard Cater	male	21.0	0	0	A/4. 39886	7.8	[null]	S	[null]	[null]	[null]
67	3	0	Novel, Mr. Mansouer	male	28.5	0	0	2697	7.2292	[null]	C	[null]	181	[null]
68	3	1	Nysten, Miss. Anna Sofia	female	22.0	0	0	347081	7.75	[null]	S	13	[null]	[null]
69	3	0	Nysveen, Mr. Johan Hansen	male	61.0	0	0	345364	6.2375	[null]	S	[null]	[null]	[null]
70	3	0	O'Brien, Mr. Thomas	male	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
71	3	0	O'Brien, Mr. Timothy	male	[null]	0	0	330979	7.8292	[null]	Q	[null]	[null]	[null]
72	3	1	O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)	female	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
73	3	0	O'Connell, Mr. Patrick D	male	[null]	0	0	334912	7.7333	[null]	Q	[null]	[null]	[null]
74	3	0	O'Connor, Mr. Maurice	male	[null]	0	0	371060	7.75	[null]	Q	[null]	[null]	[null]
75	3	0	O'Connor, Mr. Patrick	male	[null]	0	0	366713	7.75	[null]	Q	[null]	[null]	[null]
76	3	0	Odahl, Mr. Nils Martin	male	23.0	0	0	7267	9.225	[null]	S	[null]	[null]	[null]
77	3	0	O'Donoghue, Ms. Bridget	female	[null]	0	0	364856	7.75	[null]	Q	[null]	[null]	[null]
78	3	1	O'Driscoll, Miss. Bridget	female	[null]	0	0	14311	7.75	[null]	Q	D	[null]	[null]
79	3	1	O'Dwyer, Miss. Ellen "Nellie"	female	[null]	0	0	330959	7.8792	[null]	Q	[null]	[null]	[null]
80	3	1	Ohman, Miss. Velin	female	22.0	0	0	347085	7.775	[null]	S	C	[null]	[null]
81	3	1	O'Keefe, Mr. Patrick	male	[null]	0	0	368402	7.75	[null]	Q	B	[null]	[null]
82	3	1	O'Leary, Miss. Hanora "Norah"	female	[null]	0	0	330919	7.8292	[null]	Q	13	[null]	[null]
83	3	1	Olsen, Master. Artur Karl	male	9.0	0	1	C 17368	3.1708	[null]	S	13	[null]	[null]
84	3	0	Olsen, Mr. Henry Margido	male	28.0	0	0	C 4001	22.525	[null]	S	[null]	173	[null]
85	3	0	Olsen, Mr. Karl Siegwart Andreas	male	42.0	0	1	4579	8.4042	[null]	S	[null]	[null]	[null]
86	3	0	Olsen, Mr. Ole Martin	male	[null]	0	0	Fa 265302	7.3125	[null]	S	[null]	[null]	[null]
87	3	0	Olsson, Miss. Elina	female	31.0	0	0	350407	7.8542	[null]	S	[null]	[null]	[null]
88	3	0	Olsson, Mr. Nils Johan Goransson	male	28.0	0	0	347464	7.8542	[null]	S	[null]	[null]	[null]
89	3	1	Olsson, Mr. Oscar Wilhelm	male	32.0	0	0	347079	7.775	[null]	S	A	[null]	[null]
90	3	0	Olsvigen, Mr. Thor Anderson	male	20.0	0	0	6563	9.225	[null]	S	[null]	89	Oslo, Norway Cameron, WI
91	3	0	Oreskovic, Miss. Jelka	female	23.0	0	0	315085	8.6625	[null]	S	[null]	[null]	[null]
92	3	0	Oreskovic, Miss. Marija	female	20.0	0	0	315096	8.6625	[null]	S	[null]	[null]	[null]
93	3	0	Oreskovic, Mr. Luka	male	20.0	0	0	315094	8.6625	[null]	S	[null]	[null]	[null]
94	3	0	Osen, Mr. Olaf Elon	male	16.0	0	0	7534	9.2167	[null]	S	[null]	[null]	[null]
95	3	1	Osman, Mrs. Mara	female	31.0	0	0	349244	8.6833	[null]	S	[null]	[null]	[null]
96	3	0	O'Sullivan, Miss. Bridget Mary	female	[null]	0	0	330909	7.6292	[null]	Q	[null]	[null]	[null]
97	3	0	Palsson, Master. Gosta Leonard	male	2.0	3	1	349909	21.075	[null]	S	[null]	4	[null]
98	3	0	Palsson, Master. Paul Folke	male	6.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
99	3	0	Palsson, Miss. Stina Viola	female	3.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
100	3	0	Palsson, Miss. Torborg Danira	female	8.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]

Rows: 1-100 | Columns: 14

Note

vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.

Let’s look at the “fare” and “age” of the passengers.

data[["age", "fare"]]

	123 age Double	123 fare Double
1	[null]	7.75
2	[null]	23.25
3	[null]	23.25
4	[null]	23.25
5	[null]	7.7875
6	[null]	15.5
7	[null]	7.8792
8	15.0	8.0292
9	35.0	7.75
10	[null]	7.75
11	24.0	16.1
12	19.0	16.1
13	[null]	7.75
14	[null]	8.05
15	[null]	8.05
16	55.5	8.05
17	[null]	7.75
18	21.0	7.775
19	[null]	8.05
20	24.0	7.8958
21	21.0	7.8958
22	28.0	7.8958
23	[null]	7.8958
24	[null]	7.8792
25	25.0	7.65
26	6.0	12.475
27	27.0	12.475
28	[null]	8.05
29	[null]	24.15
30	[null]	24.15
31	[null]	8.4583
32	34.0	8.05
33	[null]	7.75
34	[null]	7.775
35	[null]	15.2458
36	[null]	15.2458
37	[null]	15.2458
38	[null]	7.2292
39	[null]	8.05
40	[null]	7.7333
41	24.0	7.75
42	[null]	8.05
43	[null]	15.5
44	[null]	15.5
45	[null]	15.5
46	18.0	7.75
47	22.0	7.8958
48	15.0	7.225
49	1.0	15.7417
50	20.0	15.7417
51	19.0	15.7417
52	33.0	8.05
53	[null]	7.8958
54	[null]	7.2292
55	[null]	7.75
56	[null]	7.8958
57	12.0	11.2417
58	14.0	11.2417
59	29.0	7.925
60	28.0	8.05
61	18.0	7.775
62	26.0	7.8542
63	21.0	7.8542
64	41.0	7.125
65	39.0	7.925
66	21.0	7.8
67	28.5	7.2292
68	22.0	7.75
69	61.0	6.2375
70	[null]	15.5
71	[null]	7.8292
72	[null]	15.5
73	[null]	7.7333
74	[null]	7.75
75	[null]	7.75
76	23.0	9.225
77	[null]	7.75
78	[null]	7.75
79	[null]	7.8792
80	22.0	7.775
81	[null]	7.75
82	[null]	7.8292
83	9.0	3.1708
84	28.0	22.525
85	42.0	8.4042
86	[null]	7.3125
87	31.0	7.8542
88	28.0	7.8542
89	32.0	7.775
90	20.0	9.225
91	23.0	8.6625
92	20.0	8.6625
93	20.0	8.6625
94	16.0	9.2167
95	31.0	8.6833
96	[null]	7.6292
97	2.0	21.075
98	6.0	21.075
99	3.0	21.075
100	8.0	21.075

Rows: 1-100 | Columns: 2

Note

You can observe that “age” and “fare” features lie in different numerical intervals so it’s probably a good idea to normalize them.

Let’s use the VastColumn.scale() method to normalize the data.

data["age"].scale(method = "minmax")
data["fare"].scale(method = "minmax")
data[["age", "fare"]]

	123 age Double	123 fare Double
1	[null]	0.015126992566498259
2	[null]	0.04538097769949478
3	[null]	0.04538097769949478
4	[null]	0.04538097769949478
5	[null]	0.015200187691820024
6	[null]	0.030253985132996517
7	[null]	0.01537917417160685
8	0.18576976074157586	0.015671954672893913
9	0.4363021420518602	0.015126992566498259
10	[null]	0.015126992566498259
11	0.2985093323312038	0.031425107138144774
12	0.2358762370036327	0.031425107138144774
13	[null]	0.015126992566498259
14	[null]	0.015712553569072387
15	[null]	0.015712553569072387
16	0.6930978328949017	0.015712553569072387
17	[null]	0.015126992566498259
18	0.26092947513466114	0.015175789316712771
19	[null]	0.015712553569072387
20	0.2985093323312038	0.015411575213749284
21	0.26092947513466114	0.015411575213749284
22	0.3486158085932607	0.015411575213749284
23	[null]	0.015411575213749284
24	[null]	0.01537917417160685
25	0.311035951396718	0.014931805565640218
26	0.0730301891519479	0.024349578357040744
27	0.33608918952774647	0.024349578357040744
28	[null]	0.015712553569072387
29	[null]	0.04713766070721715
30	[null]	0.04713766070721715
31	[null]	0.01650950209357577
32	0.42377552298634596	0.015712553569072387
33	[null]	0.015126992566498259
34	[null]	0.015175789316712771
35	[null]	0.029757819776815374
36	[null]	0.029757819776815374
37	[null]	0.029757819776815374
38	[null]	0.014110458666029575
39	[null]	0.015712553569072387
40	[null]	0.015094396337354966
41	0.2985093323312038	0.015126992566498259
42	[null]	0.015712553569072387
43	[null]	0.030253985132996517
44	[null]	0.030253985132996517
45	[null]	0.030253985132996517
46	0.22334961793811847	0.015126992566498259
47	0.27345609420017536	0.015411575213749284
48	0.18576976074157586	0.014102260811993537
49	0.0103970938243768	0.030725752114070404
50	0.24840285606914692	0.030725752114070404
51	0.2358762370036327	0.030725752114070404
52	0.41124890392083174	0.015712553569072387
53	[null]	0.015411575213749284
54	[null]	0.014110458666029575
55	[null]	0.015126992566498259
56	[null]	0.015411575213749284
57	0.1481899035450332	0.021942337075458514
58	0.17324314167606164	0.021942337075458514
59	0.36114242765877486	0.015468569817999833
60	0.3486158085932607	0.015712553569072387
61	0.22334961793811847	0.015175789316712771
62	0.32356257046223225	0.015330377421392339
63	0.26092947513466114	0.015330377421392339
64	0.5114618564449455	0.013907073811135496
65	0.4864086183139171	0.015468569817999833
66	0.26092947513466114	0.01522458606692728
67	0.3548791181260178	0.014110458666029575
68	0.27345609420017536	0.015126992566498259
69	0.7619942377552299	0.012174789178520372
70	[null]	0.030253985132996517
71	[null]	0.015281580671177828
72	[null]	0.030253985132996517
73	[null]	0.015094396337354966
74	[null]	0.015126992566498259
75	[null]	0.015126992566498259
76	0.2859827132656896	0.01800600082915438
77	[null]	0.015126992566498259
78	[null]	0.015126992566498259
79	[null]	0.01537917417160685
80	0.27345609420017536	0.015175789316712771
81	[null]	0.015126992566498259
82	[null]	0.015281580671177828
83	0.11061004634849055	0.006188989423206797
84	0.3486158085932607	0.04396587194327397
85	0.5239884755104597	0.01640390592611157
86	[null]	0.014273049437744325
87	0.3861956657898033	0.015330377421392339
88	0.3486158085932607	0.015330377421392339
89	0.3987222848553175	0.015175789316712771
90	0.24840285606914692	0.01800600082915438
91	0.2859827132656896	0.016908073949327893
92	0.24840285606914692	0.016908073949327893
93	0.24840285606914692	0.016908073949327893
94	0.19829637980709008	0.01798980030808316
95	0.3861956657898033	0.016948672845506364
96	[null]	0.014891206669461744
97	0.02292371288989102	0.04113566043083236
98	0.0730301891519479	0.04113566043083236
99	0.035450331955405236	0.04113566043083236
100	0.09808342728297632	0.04113566043083236

Rows: 1-100 | Columns: 2

Note

You can observe that both “age” and “fare” features now scale in [0,1] interval.