vastorbit.VastColumn.discretize¶
- VastColumn.discretize(method: Literal['auto', 'same_width', 'same_freq', 'topk'] = 'auto', h: Annotated[int | float | Decimal, 'Python Numbers'] = 0, nbins: int = -1, k: int = 6, new_category: str = 'Others', return_enum_trans: bool = False) VastFrame¶
Discretizes the VastColumn using the input method.
- Parameters:
method (str, optional) –
The method used to discretize the VastColumn:
- auto:
Uses method ‘same_width’ for numerical VastColumns, casts the other types to varchar.
- same_freq:
Computes bins with the same number of elements.
- same_width:
Computes regular width bins.
- topk:
Keeps the topk most frequent categories and merge the other into one unique category.
h (PythonNumber, optional) – The interval size used to convert the VastColumn. If this parameter is equal to 0, an optimised interval is computed.
nbins (int, optional) – Number of bins used for the discretization (must be > 1)
k (int, optional) – The integer k of the ‘topk’ method.
new_category (str, optional) – The name of the merging category when using the ‘topk’ method.
return_enum_trans (bool, optional) – Returns the transformation instead of the VastFrame parent, and does not apply the transformation. This parameter is useful for testing the look of the final transformation.
- Returns:
self._parent
- Return type:
Examples
We import
vastorbit:import vastorbit as vo
Hint
By assigning an alias to
vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions fromvastorbitare used as intended without interfering with functions from other libraries.For this example, we will use the Titanic dataset.
import vastorbit.datasets as vod data = vod.load_titanic()
123pclassInteger123survivedIntegerAbcnameVarchar(164)AbcsexVarchar(20)123ageDouble123sibspInteger123parchIntegerAbcticketVarchar(36)123fareDoubleAbccabinVarchar(30)AbcembarkedVarchar(20)AbcboatVarchar(100)123bodyIntegerAbchome.destVarchar(100)1 3 1 McCormack, Mr. Thomas Joseph male [null] 0 0 367228 7.75 [null] Q [null] [null] [null] 2 3 1 McCoy, Miss. Agnes female [null] 2 0 367226 23.25 [null] Q 16 [null] [null] 3 3 1 McCoy, Miss. Alicia female [null] 2 0 367226 23.25 [null] Q 16 [null] [null] 4 3 1 McCoy, Mr. Bernard male [null] 2 0 367226 23.25 [null] Q 16 [null] [null] 5 3 1 McDermott, Miss. Brigdet Delia female [null] 0 0 330932 7.7875 [null] Q 13 [null] [null] 6 3 0 McEvoy, Mr. Michael male [null] 0 0 36568 15.5 [null] Q [null] [null] [null] 7 3 1 McGovern, Miss. Mary female [null] 0 0 330931 7.8792 [null] Q 13 [null] [null] 8 3 1 McGowan, Miss. Anna "Annie" female 15.0 0 0 330923 8.0292 [null] Q [null] [null] [null] 9 3 0 McGowan, Miss. Katherine female 35.0 0 0 9232 7.75 [null] Q [null] [null] [null] 10 3 0 McMahon, Mr. Martin male [null] 0 0 370372 7.75 [null] Q [null] [null] [null] 11 3 0 McNamee, Mr. Neal male 24.0 1 0 376566 16.1 [null] S [null] [null] [null] 12 3 0 McNamee, Mrs. Neal (Eileen O'Leary) female 19.0 1 0 376566 16.1 [null] S [null] 53 [null] 13 3 0 McNeill, Miss. Bridget female [null] 0 0 370368 7.75 [null] Q [null] [null] [null] 14 3 0 Meanwell, Miss. (Marion Ogden) female [null] 0 0 SOTON/O.Q. 392087 8.05 [null] S [null] [null] [null] 15 3 0 Meek, Mrs. Thomas (Annie Louise Rowley) female [null] 0 0 343095 8.05 [null] S [null] [null] [null] 16 3 0 Meo, Mr. Alfonzo male 55.5 0 0 A.5. 11206 8.05 [null] S [null] 201 [null] 17 3 0 Mernagh, Mr. Robert male [null] 0 0 368703 7.75 [null] Q [null] [null] [null] 18 3 1 Midtsjo, Mr. Karl Albert male 21.0 0 0 345501 7.775 [null] S 15 [null] [null] 19 3 0 Miles, Mr. Frank male [null] 0 0 359306 8.05 [null] S [null] [null] [null] 20 3 0 Mineff, Mr. Ivan male 24.0 0 0 349233 7.8958 [null] S [null] [null] [null] 21 3 0 Minkoff, Mr. Lazar male 21.0 0 0 349211 7.8958 [null] S [null] [null] [null] 22 3 0 Mionoff, Mr. Stoytcho male 28.0 0 0 349207 7.8958 [null] S [null] [null] [null] 23 3 0 Mitkoff, Mr. Mito male [null] 0 0 349221 7.8958 [null] S [null] [null] [null] 24 3 1 Mockler, Miss. Helen Mary "Ellie" female [null] 0 0 330980 7.8792 [null] Q 16 [null] [null] 25 3 0 Moen, Mr. Sigurd Hansen male 25.0 0 0 348123 7.65 F G73 S [null] 309 [null] 26 3 1 Moor, Master. Meier male 6.0 0 1 392096 12.475 E121 S 14 [null] [null] 27 3 1 Moor, Mrs. (Beila) female 27.0 0 1 392096 12.475 E121 S 14 [null] [null] 28 3 0 Moore, Mr. Leonard Charles male [null] 0 0 A4. 54510 8.05 [null] S [null] [null] [null] 29 3 1 Moran, Miss. Bertha female [null] 1 0 371110 24.15 [null] Q 16 [null] [null] 30 3 0 Moran, Mr. Daniel J male [null] 1 0 371110 24.15 [null] Q [null] [null] [null] 31 3 0 Moran, Mr. James male [null] 0 0 330877 8.4583 [null] Q [null] [null] [null] 32 3 0 Morley, Mr. William male 34.0 0 0 364506 8.05 [null] S [null] [null] [null] 33 3 0 Morrow, Mr. Thomas Rowan male [null] 0 0 372622 7.75 [null] Q [null] [null] [null] 34 3 1 Moss, Mr. Albert Johan male [null] 0 0 312991 7.775 [null] S B [null] [null] 35 3 1 Moubarek, Master. Gerios male [null] 1 1 2661 15.2458 [null] C C [null] [null] 36 3 1 Moubarek, Master. Halim Gonios ("William George") male [null] 1 1 2661 15.2458 [null] C C [null] [null] 37 3 1 Moubarek, Mrs. George (Omine "Amenia" Alexander) female [null] 0 2 2661 15.2458 [null] C C [null] [null] 38 3 1 Moussa, Mrs. (Mantoura Boulos) female [null] 0 0 2626 7.2292 [null] C [null] [null] [null] 39 3 0 Moutal, Mr. Rahamin Haim male [null] 0 0 374746 8.05 [null] S [null] [null] [null] 40 3 1 Mullens, Miss. Katherine "Katie" female [null] 0 0 35852 7.7333 [null] Q 16 [null] [null] 41 3 1 Mulvihill, Miss. Bertha E female 24.0 0 0 382653 7.75 [null] Q 15 [null] [null] 42 3 0 Murdlin, Mr. Joseph male [null] 0 0 A./5. 3235 8.05 [null] S [null] [null] [null] 43 3 1 Murphy, Miss. Katherine "Kate" female [null] 1 0 367230 15.5 [null] Q 16 [null] [null] 44 3 1 Murphy, Miss. Margaret Jane female [null] 1 0 367230 15.5 [null] Q 16 [null] [null] 45 3 1 Murphy, Miss. Nora female [null] 0 0 36568 15.5 [null] Q 16 [null] [null] 46 3 0 Myhrman, Mr. Pehr Fabian Oliver Malkolm male 18.0 0 0 347078 7.75 [null] S [null] [null] [null] 47 3 0 Naidenoff, Mr. Penko male 22.0 0 0 349206 7.8958 [null] S [null] [null] [null] 48 3 1 Najib, Miss. Adele Kiamie "Jane" female 15.0 0 0 2667 7.225 [null] C C [null] [null] 49 3 1 Nakid, Miss. Maria ("Mary") female 1.0 0 2 2653 15.7417 [null] C C [null] [null] 50 3 1 Nakid, Mr. Sahid male 20.0 1 1 2653 15.7417 [null] C C [null] [null] 51 3 1 Nakid, Mrs. Said (Waika "Mary" Mowad) female 19.0 1 1 2653 15.7417 [null] C C [null] [null] 52 3 0 Nancarrow, Mr. William Henry male 33.0 0 0 A./5. 3338 8.05 [null] S [null] [null] [null] 53 3 0 Nankoff, Mr. Minko male [null] 0 0 349218 7.8958 [null] S [null] [null] [null] 54 3 0 Nasr, Mr. Mustafa male [null] 0 0 2652 7.2292 [null] C [null] [null] [null] 55 3 0 Naughton, Miss. Hannah female [null] 0 0 365237 7.75 [null] Q [null] [null] [null] 56 3 0 Nenkoff, Mr. Christo male [null] 0 0 349234 7.8958 [null] S [null] [null] [null] 57 3 1 Nicola-Yarred, Master. Elias male 12.0 1 0 2651 11.2417 [null] C C [null] [null] 58 3 1 Nicola-Yarred, Miss. Jamila female 14.0 1 0 2651 11.2417 [null] C C [null] [null] 59 3 0 Nieminen, Miss. Manta Josefina female 29.0 0 0 3101297 7.925 [null] S [null] [null] [null] 60 3 0 Niklasson, Mr. Samuel male 28.0 0 0 363611 8.05 [null] S [null] [null] [null] 61 3 1 Nilsson, Miss. Berta Olivia female 18.0 0 0 347066 7.775 [null] S D [null] [null] 62 3 1 Nilsson, Miss. Helmina Josefina female 26.0 0 0 347470 7.8542 [null] S 13 [null] [null] 63 3 0 Nilsson, Mr. August Ferdinand male 21.0 0 0 350410 7.8542 [null] S [null] [null] [null] 64 3 0 Nirva, Mr. Iisakki Antino Aijo male 41.0 0 0 SOTON/O2 3101272 7.125 [null] S [null] [null] Finland Sudbury, ON 65 3 1 Niskanen, Mr. Juha male 39.0 0 0 STON/O 2. 3101289 7.925 [null] S 9 [null] [null] 66 3 0 Nosworthy, Mr. Richard Cater male 21.0 0 0 A/4. 39886 7.8 [null] S [null] [null] [null] 67 3 0 Novel, Mr. Mansouer male 28.5 0 0 2697 7.2292 [null] C [null] 181 [null] 68 3 1 Nysten, Miss. Anna Sofia female 22.0 0 0 347081 7.75 [null] S 13 [null] [null] 69 3 0 Nysveen, Mr. Johan Hansen male 61.0 0 0 345364 6.2375 [null] S [null] [null] [null] 70 3 0 O'Brien, Mr. Thomas male [null] 1 0 370365 15.5 [null] Q [null] [null] [null] 71 3 0 O'Brien, Mr. Timothy male [null] 0 0 330979 7.8292 [null] Q [null] [null] [null] 72 3 1 O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey) female [null] 1 0 370365 15.5 [null] Q [null] [null] [null] 73 3 0 O'Connell, Mr. Patrick D male [null] 0 0 334912 7.7333 [null] Q [null] [null] [null] 74 3 0 O'Connor, Mr. Maurice male [null] 0 0 371060 7.75 [null] Q [null] [null] [null] 75 3 0 O'Connor, Mr. Patrick male [null] 0 0 366713 7.75 [null] Q [null] [null] [null] 76 3 0 Odahl, Mr. Nils Martin male 23.0 0 0 7267 9.225 [null] S [null] [null] [null] 77 3 0 O'Donoghue, Ms. Bridget female [null] 0 0 364856 7.75 [null] Q [null] [null] [null] 78 3 1 O'Driscoll, Miss. Bridget female [null] 0 0 14311 7.75 [null] Q D [null] [null] 79 3 1 O'Dwyer, Miss. Ellen "Nellie" female [null] 0 0 330959 7.8792 [null] Q [null] [null] [null] 80 3 1 Ohman, Miss. Velin female 22.0 0 0 347085 7.775 [null] S C [null] [null] 81 3 1 O'Keefe, Mr. Patrick male [null] 0 0 368402 7.75 [null] Q B [null] [null] 82 3 1 O'Leary, Miss. Hanora "Norah" female [null] 0 0 330919 7.8292 [null] Q 13 [null] [null] 83 3 1 Olsen, Master. Artur Karl male 9.0 0 1 C 17368 3.1708 [null] S 13 [null] [null] 84 3 0 Olsen, Mr. Henry Margido male 28.0 0 0 C 4001 22.525 [null] S [null] 173 [null] 85 3 0 Olsen, Mr. Karl Siegwart Andreas male 42.0 0 1 4579 8.4042 [null] S [null] [null] [null] 86 3 0 Olsen, Mr. Ole Martin male [null] 0 0 Fa 265302 7.3125 [null] S [null] [null] [null] 87 3 0 Olsson, Miss. Elina female 31.0 0 0 350407 7.8542 [null] S [null] [null] [null] 88 3 0 Olsson, Mr. Nils Johan Goransson male 28.0 0 0 347464 7.8542 [null] S [null] [null] [null] 89 3 1 Olsson, Mr. Oscar Wilhelm male 32.0 0 0 347079 7.775 [null] S A [null] [null] 90 3 0 Olsvigen, Mr. Thor Anderson male 20.0 0 0 6563 9.225 [null] S [null] 89 Oslo, Norway Cameron, WI 91 3 0 Oreskovic, Miss. Jelka female 23.0 0 0 315085 8.6625 [null] S [null] [null] [null] 92 3 0 Oreskovic, Miss. Marija female 20.0 0 0 315096 8.6625 [null] S [null] [null] [null] 93 3 0 Oreskovic, Mr. Luka male 20.0 0 0 315094 8.6625 [null] S [null] [null] [null] 94 3 0 Osen, Mr. Olaf Elon male 16.0 0 0 7534 9.2167 [null] S [null] [null] [null] 95 3 1 Osman, Mrs. Mara female 31.0 0 0 349244 8.6833 [null] S [null] [null] [null] 96 3 0 O'Sullivan, Miss. Bridget Mary female [null] 0 0 330909 7.6292 [null] Q [null] [null] [null] 97 3 0 Palsson, Master. Gosta Leonard male 2.0 3 1 349909 21.075 [null] S [null] 4 [null] 98 3 0 Palsson, Master. Paul Folke male 6.0 3 1 349909 21.075 [null] S [null] [null] [null] 99 3 0 Palsson, Miss. Stina Viola female 3.0 3 1 349909 21.075 [null] S [null] [null] [null] 100 3 0 Palsson, Miss. Torborg Danira female 8.0 3 1 349909 21.075 [null] S [null] [null] [null] Rows: 1-100 | Columns: 14Note
vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.
Let’s look at “age” VastColumn
data["age"]
123ageDouble1 [null] 2 [null] 3 [null] 4 [null] 5 [null] 6 [null] 7 [null] 8 15.0 9 35.0 10 [null] 11 24.0 12 19.0 13 [null] 14 [null] 15 [null] 16 55.5 17 [null] 18 21.0 19 [null] 20 24.0 21 21.0 22 28.0 23 [null] 24 [null] 25 25.0 26 6.0 27 27.0 28 [null] 29 [null] 30 [null] 31 [null] 32 34.0 33 [null] 34 [null] 35 [null] 36 [null] 37 [null] 38 [null] 39 [null] 40 [null] 41 24.0 42 [null] 43 [null] 44 [null] 45 [null] 46 18.0 47 22.0 48 15.0 49 1.0 50 20.0 51 19.0 52 33.0 53 [null] 54 [null] 55 [null] 56 [null] 57 12.0 58 14.0 59 29.0 60 28.0 61 18.0 62 26.0 63 21.0 64 41.0 65 39.0 66 21.0 67 28.5 68 22.0 69 61.0 70 [null] 71 [null] 72 [null] 73 [null] 74 [null] 75 [null] 76 23.0 77 [null] 78 [null] 79 [null] 80 22.0 81 [null] 82 [null] 83 9.0 84 28.0 85 42.0 86 [null] 87 31.0 88 28.0 89 32.0 90 20.0 91 23.0 92 20.0 93 20.0 94 16.0 95 31.0 96 [null] 97 2.0 98 6.0 99 3.0 100 8.0 Rows: 1-100 of 1309 | Column: age | Type: doubleLet’s look at the distribution of age.
data["age"].bar()
Let’s discretize “age” using the same bar width.
data["age"].discretize(method = "same_width", h = 10) data["age"]
AbcageVarchar1 [null] 2 [null] 3 [null] 4 [null] 5 [null] 6 [null] 7 [null] 8 [1.0E1;2.0E1] 9 [3.0E1;4.0E1] 10 [null] 11 [2.0E1;3.0E1] 12 [1.0E1;2.0E1] 13 [null] 14 [null] 15 [null] 16 [5.0E1;6.0E1] 17 [null] 18 [2.0E1;3.0E1] 19 [null] 20 [2.0E1;3.0E1] 21 [2.0E1;3.0E1] 22 [2.0E1;3.0E1] 23 [null] 24 [null] 25 [2.0E1;3.0E1] 26 [0E0;1.0E1] 27 [2.0E1;3.0E1] 28 [null] 29 [null] 30 [null] 31 [null] 32 [3.0E1;4.0E1] 33 [null] 34 [null] 35 [null] 36 [null] 37 [null] 38 [null] 39 [null] 40 [null] 41 [2.0E1;3.0E1] 42 [null] 43 [null] 44 [null] 45 [null] 46 [1.0E1;2.0E1] 47 [2.0E1;3.0E1] 48 [1.0E1;2.0E1] 49 [0E0;1.0E1] 50 [2.0E1;3.0E1] 51 [1.0E1;2.0E1] 52 [3.0E1;4.0E1] 53 [null] 54 [null] 55 [null] 56 [null] 57 [1.0E1;2.0E1] 58 [1.0E1;2.0E1] 59 [2.0E1;3.0E1] 60 [2.0E1;3.0E1] 61 [1.0E1;2.0E1] 62 [2.0E1;3.0E1] 63 [2.0E1;3.0E1] 64 [4.0E1;5.0E1] 65 [3.0E1;4.0E1] 66 [2.0E1;3.0E1] 67 [2.0E1;3.0E1] 68 [2.0E1;3.0E1] 69 [6.0E1;7.0E1] 70 [null] 71 [null] 72 [null] 73 [null] 74 [null] 75 [null] 76 [2.0E1;3.0E1] 77 [null] 78 [null] 79 [null] 80 [2.0E1;3.0E1] 81 [null] 82 [null] 83 [0E0;1.0E1] 84 [2.0E1;3.0E1] 85 [4.0E1;5.0E1] 86 [null] 87 [3.0E1;4.0E1] 88 [2.0E1;3.0E1] 89 [3.0E1;4.0E1] 90 [2.0E1;3.0E1] 91 [2.0E1;3.0E1] 92 [2.0E1;3.0E1] 93 [2.0E1;3.0E1] 94 [1.0E1;2.0E1] 95 [3.0E1;4.0E1] 96 [null] 97 [0E0;1.0E1] 98 [0E0;1.0E1] 99 [0E0;1.0E1] 100 [0E0;1.0E1] Rows: 1-100 of 1309 | Column: age | Type: varcharLet’s look at the distribution of age again.
data["age"].bar()
Let’s discretize “age” using the same frequency per bin.
data = vod.load_titanic() # Reloading the dataset data["age"].discretize(method = "same_freq", nbins = 5) data["age"]
AbcageVarchar(11)1 [null] 2 [null] 3 [null] 4 [null] 5 [null] 6 [null] 7 [null] 8 [0.17;19.0] 9 [31.0;42.0] 10 [null] 11 [19.0;25.0] 12 [0.17;19.0] 13 [null] 14 [null] 15 [null] 16 [42.0;80.0] 17 [null] 18 [19.0;25.0] 19 [null] 20 [19.0;25.0] 21 [19.0;25.0] 22 [25.0;31.0] 23 [null] 24 [null] 25 [19.0;25.0] 26 [0.17;19.0] 27 [25.0;31.0] 28 [null] 29 [null] 30 [null] 31 [null] 32 [31.0;42.0] 33 [null] 34 [null] 35 [null] 36 [null] 37 [null] 38 [null] 39 [null] 40 [null] 41 [19.0;25.0] 42 [null] 43 [null] 44 [null] 45 [null] 46 [0.17;19.0] 47 [19.0;25.0] 48 [0.17;19.0] 49 [0.17;19.0] 50 [19.0;25.0] 51 [0.17;19.0] 52 [31.0;42.0] 53 [null] 54 [null] 55 [null] 56 [null] 57 [0.17;19.0] 58 [0.17;19.0] 59 [25.0;31.0] 60 [25.0;31.0] 61 [0.17;19.0] 62 [25.0;31.0] 63 [19.0;25.0] 64 [31.0;42.0] 65 [31.0;42.0] 66 [19.0;25.0] 67 [25.0;31.0] 68 [19.0;25.0] 69 [42.0;80.0] 70 [null] 71 [null] 72 [null] 73 [null] 74 [null] 75 [null] 76 [19.0;25.0] 77 [null] 78 [null] 79 [null] 80 [19.0;25.0] 81 [null] 82 [null] 83 [0.17;19.0] 84 [25.0;31.0] 85 [31.0;42.0] 86 [null] 87 [25.0;31.0] 88 [25.0;31.0] 89 [31.0;42.0] 90 [19.0;25.0] 91 [19.0;25.0] 92 [19.0;25.0] 93 [19.0;25.0] 94 [0.17;19.0] 95 [25.0;31.0] 96 [null] 97 [0.17;19.0] 98 [0.17;19.0] 99 [0.17;19.0] 100 [0.17;19.0] Rows: 1-100 of 1309 | Column: age | Type: varchar(11)Let’s look at the distribution of age again.
data["age"].bar()
See also
VastFrame.decode(): User Defined Encoding.VastFrame.label_encode(): Label Encoding.VastFrame.mean_encode(): Mean Encoding.VastFrame.one_hot_encode(): One Hot Encoding.