Loading...

vastorbit.VastFrame.fillna

VastFrame.fillna(val: dict | None = None, method: dict | None = None, numeric_only: bool = False) VastFrame

Fills missing elements in VastColumn using specific rules.

Parameters:
  • val (dict, optional) – Dictionary of values. The dictionary must be similar to the following: {"column1": val1 ..., "columnk": valk}. Each key of the dictionary must be a VastColumn . The missing values of the input VastColumn are replaced by the input value.

  • method (dict, optional) –

    Method used to impute the missing values.

    • auto:

      Mean for the numerical and Mode for the categorical VastColumn.

    • mean:

      Average.

    • median:

      Median.

    • mode:

      Mode (most occurent element).

    • 0ifnull:

      0 when the VastColumn is None, 1 otherwise.

    More Methods are available in the VastColumn.fillna() method.

  • numeric_only (bool, optional) – If parameters ‘val’ and ‘method’ are empty and ‘numeric_only’ is set to True, all numerical VastColumn are imputed by their average. If set to False, all categorical VastColumn are also imputed by their mode.

Returns:

self

Return type:

VastFrame

Examples

We import vastorbit:

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

For this example, we will use the Titanic dataset.

from vastorbit.datasets import load_titanic

data = load_titanic()
123
pclass
Integer
123
survived
Integer
Abc
name
Varchar(164)
Abc
sex
Varchar(20)
123
age
Double
123
sibsp
Integer
123
parch
Integer
Abc
ticket
Varchar(36)
123
fare
Double
Abc
cabin
Varchar(30)
Abc
embarked
Varchar(20)
Abc
boat
Varchar(100)
123
body
Integer
Abc
home.dest
Varchar(100)
131McCormack, Mr. Thomas Josephmale[null]003672287.75[null]Q[null][null][null]
231McCoy, Miss. Agnesfemale[null]2036722623.25[null]Q16[null][null]
331McCoy, Miss. Aliciafemale[null]2036722623.25[null]Q16[null][null]
431McCoy, Mr. Bernardmale[null]2036722623.25[null]Q16[null][null]
531McDermott, Miss. Brigdet Deliafemale[null]003309327.7875[null]Q13[null][null]
630McEvoy, Mr. Michaelmale[null]003656815.5[null]Q[null][null][null]
731McGovern, Miss. Maryfemale[null]003309317.8792[null]Q13[null][null]
831McGowan, Miss. Anna "Annie"female15.0003309238.0292[null]Q[null][null][null]
930McGowan, Miss. Katherinefemale35.00092327.75[null]Q[null][null][null]
1030McMahon, Mr. Martinmale[null]003703727.75[null]Q[null][null][null]
1130McNamee, Mr. Nealmale24.01037656616.1[null]S[null][null][null]
1230McNamee, Mrs. Neal (Eileen O'Leary)female19.01037656616.1[null]S[null]53[null]
1330McNeill, Miss. Bridgetfemale[null]003703687.75[null]Q[null][null][null]
1430Meanwell, Miss. (Marion Ogden)female[null]00SOTON/O.Q. 3920878.05[null]S[null][null][null]
1530Meek, Mrs. Thomas (Annie Louise Rowley)female[null]003430958.05[null]S[null][null][null]
1630Meo, Mr. Alfonzomale55.500A.5. 112068.05[null]S[null]201[null]
1730Mernagh, Mr. Robertmale[null]003687037.75[null]Q[null][null][null]
1831Midtsjo, Mr. Karl Albertmale21.0003455017.775[null]S15[null][null]
1930Miles, Mr. Frankmale[null]003593068.05[null]S[null][null][null]
2030Mineff, Mr. Ivanmale24.0003492337.8958[null]S[null][null][null]
2130Minkoff, Mr. Lazarmale21.0003492117.8958[null]S[null][null][null]
2230Mionoff, Mr. Stoytchomale28.0003492077.8958[null]S[null][null][null]
2330Mitkoff, Mr. Mitomale[null]003492217.8958[null]S[null][null][null]
2431Mockler, Miss. Helen Mary "Ellie"female[null]003309807.8792[null]Q16[null][null]
2530Moen, Mr. Sigurd Hansenmale25.0003481237.65F G73S[null]309[null]
2631Moor, Master. Meiermale6.00139209612.475E121S14[null][null]
2731Moor, Mrs. (Beila)female27.00139209612.475E121S14[null][null]
2830Moore, Mr. Leonard Charlesmale[null]00A4. 545108.05[null]S[null][null][null]
2931Moran, Miss. Berthafemale[null]1037111024.15[null]Q16[null][null]
3030Moran, Mr. Daniel Jmale[null]1037111024.15[null]Q[null][null][null]
3130Moran, Mr. Jamesmale[null]003308778.4583[null]Q[null][null][null]
3230Morley, Mr. Williammale34.0003645068.05[null]S[null][null][null]
3330Morrow, Mr. Thomas Rowanmale[null]003726227.75[null]Q[null][null][null]
3431Moss, Mr. Albert Johanmale[null]003129917.775[null]SB[null][null]
3531Moubarek, Master. Geriosmale[null]11266115.2458[null]CC[null][null]
3631Moubarek, Master. Halim Gonios ("William George")male[null]11266115.2458[null]CC[null][null]
3731Moubarek, Mrs. George (Omine "Amenia" Alexander)female[null]02266115.2458[null]CC[null][null]
3831Moussa, Mrs. (Mantoura Boulos)female[null]0026267.2292[null]C[null][null][null]
3930Moutal, Mr. Rahamin Haimmale[null]003747468.05[null]S[null][null][null]
4031Mullens, Miss. Katherine "Katie"female[null]00358527.7333[null]Q16[null][null]
4131Mulvihill, Miss. Bertha Efemale24.0003826537.75[null]Q15[null][null]
4230Murdlin, Mr. Josephmale[null]00A./5. 32358.05[null]S[null][null][null]
4331Murphy, Miss. Katherine "Kate"female[null]1036723015.5[null]Q16[null][null]
4431Murphy, Miss. Margaret Janefemale[null]1036723015.5[null]Q16[null][null]
4531Murphy, Miss. Norafemale[null]003656815.5[null]Q16[null][null]
4630Myhrman, Mr. Pehr Fabian Oliver Malkolmmale18.0003470787.75[null]S[null][null][null]
4730Naidenoff, Mr. Penkomale22.0003492067.8958[null]S[null][null][null]
4831Najib, Miss. Adele Kiamie "Jane"female15.00026677.225[null]CC[null][null]
4931Nakid, Miss. Maria ("Mary")female1.002265315.7417[null]CC[null][null]
5031Nakid, Mr. Sahidmale20.011265315.7417[null]CC[null][null]
5131Nakid, Mrs. Said (Waika "Mary" Mowad)female19.011265315.7417[null]CC[null][null]
5230Nancarrow, Mr. William Henrymale33.000A./5. 33388.05[null]S[null][null][null]
5330Nankoff, Mr. Minkomale[null]003492187.8958[null]S[null][null][null]
5430Nasr, Mr. Mustafamale[null]0026527.2292[null]C[null][null][null]
5530Naughton, Miss. Hannahfemale[null]003652377.75[null]Q[null][null][null]
5630Nenkoff, Mr. Christomale[null]003492347.8958[null]S[null][null][null]
5731Nicola-Yarred, Master. Eliasmale12.010265111.2417[null]CC[null][null]
5831Nicola-Yarred, Miss. Jamilafemale14.010265111.2417[null]CC[null][null]
5930Nieminen, Miss. Manta Josefinafemale29.00031012977.925[null]S[null][null][null]
6030Niklasson, Mr. Samuelmale28.0003636118.05[null]S[null][null][null]
6131Nilsson, Miss. Berta Oliviafemale18.0003470667.775[null]SD[null][null]
6231Nilsson, Miss. Helmina Josefinafemale26.0003474707.8542[null]S13[null][null]
6330Nilsson, Mr. August Ferdinandmale21.0003504107.8542[null]S[null][null][null]
6430Nirva, Mr. Iisakki Antino Aijomale41.000SOTON/O2 31012727.125[null]S[null][null]Finland Sudbury, ON
6531Niskanen, Mr. Juhamale39.000STON/O 2. 31012897.925[null]S9[null][null]
6630Nosworthy, Mr. Richard Catermale21.000A/4. 398867.8[null]S[null][null][null]
6730Novel, Mr. Mansouermale28.50026977.2292[null]C[null]181[null]
6831Nysten, Miss. Anna Sofiafemale22.0003470817.75[null]S13[null][null]
6930Nysveen, Mr. Johan Hansenmale61.0003453646.2375[null]S[null][null][null]
7030O'Brien, Mr. Thomasmale[null]1037036515.5[null]Q[null][null][null]
7130O'Brien, Mr. Timothymale[null]003309797.8292[null]Q[null][null][null]
7231O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)female[null]1037036515.5[null]Q[null][null][null]
7330O'Connell, Mr. Patrick Dmale[null]003349127.7333[null]Q[null][null][null]
7430O'Connor, Mr. Mauricemale[null]003710607.75[null]Q[null][null][null]
7530O'Connor, Mr. Patrickmale[null]003667137.75[null]Q[null][null][null]
7630Odahl, Mr. Nils Martinmale23.00072679.225[null]S[null][null][null]
7730O'Donoghue, Ms. Bridgetfemale[null]003648567.75[null]Q[null][null][null]
7831O'Driscoll, Miss. Bridgetfemale[null]00143117.75[null]QD[null][null]
7931O'Dwyer, Miss. Ellen "Nellie"female[null]003309597.8792[null]Q[null][null][null]
8031Ohman, Miss. Velinfemale22.0003470857.775[null]SC[null][null]
8131O'Keefe, Mr. Patrickmale[null]003684027.75[null]QB[null][null]
8231O'Leary, Miss. Hanora "Norah"female[null]003309197.8292[null]Q13[null][null]
8331Olsen, Master. Artur Karlmale9.001C 173683.1708[null]S13[null][null]
8430Olsen, Mr. Henry Margidomale28.000C 400122.525[null]S[null]173[null]
8530Olsen, Mr. Karl Siegwart Andreasmale42.00145798.4042[null]S[null][null][null]
8630Olsen, Mr. Ole Martinmale[null]00Fa 2653027.3125[null]S[null][null][null]
8730Olsson, Miss. Elinafemale31.0003504077.8542[null]S[null][null][null]
8830Olsson, Mr. Nils Johan Goranssonmale28.0003474647.8542[null]S[null][null][null]
8931Olsson, Mr. Oscar Wilhelmmale32.0003470797.775[null]SA[null][null]
9030Olsvigen, Mr. Thor Andersonmale20.00065639.225[null]S[null]89Oslo, Norway Cameron, WI
9130Oreskovic, Miss. Jelkafemale23.0003150858.6625[null]S[null][null][null]
9230Oreskovic, Miss. Marijafemale20.0003150968.6625[null]S[null][null][null]
9330Oreskovic, Mr. Lukamale20.0003150948.6625[null]S[null][null][null]
9430Osen, Mr. Olaf Elonmale16.00075349.2167[null]S[null][null][null]
9531Osman, Mrs. Marafemale31.0003492448.6833[null]S[null][null][null]
9630O'Sullivan, Miss. Bridget Maryfemale[null]003309097.6292[null]Q[null][null][null]
9730Palsson, Master. Gosta Leonardmale2.03134990921.075[null]S[null]4[null]
9830Palsson, Master. Paul Folkemale6.03134990921.075[null]S[null][null][null]
9930Palsson, Miss. Stina Violafemale3.03134990921.075[null]S[null][null][null]
10030Palsson, Miss. Torborg Danirafemale8.03134990921.075[null]S[null][null][null]
Rows: 1-100 | Columns: 14

Note

vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.

We can see the count of each column to check if any column has missing values.

data.count()
count
"pclass"1309.0
"survived"1309.0
"name"1309.0
"sex"1309.0
"age"1046.0
"sibsp"1309.0
"parch"1309.0
"ticket"1309.0
"fare"1308.0
"cabin"295.0
"embarked"1307.0
"boat"486.0
"body"121.0
"home.dest"745.0
Rows: 1-14 | Columns: 2

From the above table, we can see that the count of boats is less than 1234. This suggests that it is missing some values.

Now we can use the fillna method to fill those values. Let’s use a custom function to fill these values.

data.fillna(
    val = {"boat": "No boat"},
    method = {
        "age": "mean",
        "embarked": "mode",
        "fare": "median",
    }
)
123
pclass
Integer
123
survived
Integer
Abc
name
Varchar(164)
Abc
sex
Varchar(20)
123
age
Double
123
sibsp
Integer
123
parch
Integer
Abc
ticket
Varchar(36)
123
fare
Double
Abc
cabin
Varchar(30)
Abc
embarked
Varchar(20)
Abc
boat
Varchar(100)
123
body
Integer
Abc
home.dest
Varchar(100)
131McCormack, Mr. Thomas Josephmale29.881137667304017003672287.75[null]QNo boat[null][null]
231McCoy, Miss. Agnesfemale29.8811376673040172036722623.25[null]Q16[null][null]
331McCoy, Miss. Aliciafemale29.8811376673040172036722623.25[null]Q16[null][null]
431McCoy, Mr. Bernardmale29.8811376673040172036722623.25[null]Q16[null][null]
531McDermott, Miss. Brigdet Deliafemale29.881137667304017003309327.7875[null]Q13[null][null]
630McEvoy, Mr. Michaelmale29.881137667304017003656815.5[null]QNo boat[null][null]
731McGovern, Miss. Maryfemale29.881137667304017003309317.8792[null]Q13[null][null]
831McGowan, Miss. Anna "Annie"female15.0003309238.0292[null]QNo boat[null][null]
930McGowan, Miss. Katherinefemale35.00092327.75[null]QNo boat[null][null]
1030McMahon, Mr. Martinmale29.881137667304017003703727.75[null]QNo boat[null][null]
1130McNamee, Mr. Nealmale24.01037656616.1[null]SNo boat[null][null]
1230McNamee, Mrs. Neal (Eileen O'Leary)female19.01037656616.1[null]SNo boat53[null]
1330McNeill, Miss. Bridgetfemale29.881137667304017003703687.75[null]QNo boat[null][null]
1430Meanwell, Miss. (Marion Ogden)female29.88113766730401700SOTON/O.Q. 3920878.05[null]SNo boat[null][null]
1530Meek, Mrs. Thomas (Annie Louise Rowley)female29.881137667304017003430958.05[null]SNo boat[null][null]
1630Meo, Mr. Alfonzomale55.500A.5. 112068.05[null]SNo boat201[null]
1730Mernagh, Mr. Robertmale29.881137667304017003687037.75[null]QNo boat[null][null]
1831Midtsjo, Mr. Karl Albertmale21.0003455017.775[null]S15[null][null]
1930Miles, Mr. Frankmale29.881137667304017003593068.05[null]SNo boat[null][null]
2030Mineff, Mr. Ivanmale24.0003492337.8958[null]SNo boat[null][null]
2130Minkoff, Mr. Lazarmale21.0003492117.8958[null]SNo boat[null][null]
2230Mionoff, Mr. Stoytchomale28.0003492077.8958[null]SNo boat[null][null]
2330Mitkoff, Mr. Mitomale29.881137667304017003492217.8958[null]SNo boat[null][null]
2431Mockler, Miss. Helen Mary "Ellie"female29.881137667304017003309807.8792[null]Q16[null][null]
2530Moen, Mr. Sigurd Hansenmale25.0003481237.65F G73SNo boat309[null]
2631Moor, Master. Meiermale6.00139209612.475E121S14[null][null]
2731Moor, Mrs. (Beila)female27.00139209612.475E121S14[null][null]
2830Moore, Mr. Leonard Charlesmale29.88113766730401700A4. 545108.05[null]SNo boat[null][null]
2931Moran, Miss. Berthafemale29.8811376673040171037111024.15[null]Q16[null][null]
3030Moran, Mr. Daniel Jmale29.8811376673040171037111024.15[null]QNo boat[null][null]
3130Moran, Mr. Jamesmale29.881137667304017003308778.4583[null]QNo boat[null][null]
3230Morley, Mr. Williammale34.0003645068.05[null]SNo boat[null][null]
3330Morrow, Mr. Thomas Rowanmale29.881137667304017003726227.75[null]QNo boat[null][null]
3431Moss, Mr. Albert Johanmale29.881137667304017003129917.775[null]SB[null][null]
3531Moubarek, Master. Geriosmale29.88113766730401711266115.2458[null]CC[null][null]
3631Moubarek, Master. Halim Gonios ("William George")male29.88113766730401711266115.2458[null]CC[null][null]
3731Moubarek, Mrs. George (Omine "Amenia" Alexander)female29.88113766730401702266115.2458[null]CC[null][null]
3831Moussa, Mrs. (Mantoura Boulos)female29.8811376673040170026267.2292[null]CNo boat[null][null]
3930Moutal, Mr. Rahamin Haimmale29.881137667304017003747468.05[null]SNo boat[null][null]
4031Mullens, Miss. Katherine "Katie"female29.88113766730401700358527.7333[null]Q16[null][null]
4131Mulvihill, Miss. Bertha Efemale24.0003826537.75[null]Q15[null][null]
4230Murdlin, Mr. Josephmale29.88113766730401700A./5. 32358.05[null]SNo boat[null][null]
4331Murphy, Miss. Katherine "Kate"female29.8811376673040171036723015.5[null]Q16[null][null]
4431Murphy, Miss. Margaret Janefemale29.8811376673040171036723015.5[null]Q16[null][null]
4531Murphy, Miss. Norafemale29.881137667304017003656815.5[null]Q16[null][null]
4630Myhrman, Mr. Pehr Fabian Oliver Malkolmmale18.0003470787.75[null]SNo boat[null][null]
4730Naidenoff, Mr. Penkomale22.0003492067.8958[null]SNo boat[null][null]
4831Najib, Miss. Adele Kiamie "Jane"female15.00026677.225[null]CC[null][null]
4931Nakid, Miss. Maria ("Mary")female1.002265315.7417[null]CC[null][null]
5031Nakid, Mr. Sahidmale20.011265315.7417[null]CC[null][null]
5131Nakid, Mrs. Said (Waika "Mary" Mowad)female19.011265315.7417[null]CC[null][null]
5230Nancarrow, Mr. William Henrymale33.000A./5. 33388.05[null]SNo boat[null][null]
5330Nankoff, Mr. Minkomale29.881137667304017003492187.8958[null]SNo boat[null][null]
5430Nasr, Mr. Mustafamale29.8811376673040170026527.2292[null]CNo boat[null][null]
5530Naughton, Miss. Hannahfemale29.881137667304017003652377.75[null]QNo boat[null][null]
5630Nenkoff, Mr. Christomale29.881137667304017003492347.8958[null]SNo boat[null][null]
5731Nicola-Yarred, Master. Eliasmale12.010265111.2417[null]CC[null][null]
5831Nicola-Yarred, Miss. Jamilafemale14.010265111.2417[null]CC[null][null]
5930Nieminen, Miss. Manta Josefinafemale29.00031012977.925[null]SNo boat[null][null]
6030Niklasson, Mr. Samuelmale28.0003636118.05[null]SNo boat[null][null]
6131Nilsson, Miss. Berta Oliviafemale18.0003470667.775[null]SD[null][null]
6231Nilsson, Miss. Helmina Josefinafemale26.0003474707.8542[null]S13[null][null]
6330Nilsson, Mr. August Ferdinandmale21.0003504107.8542[null]SNo boat[null][null]
6430Nirva, Mr. Iisakki Antino Aijomale41.000SOTON/O2 31012727.125[null]SNo boat[null]Finland Sudbury, ON
6531Niskanen, Mr. Juhamale39.000STON/O 2. 31012897.925[null]S9[null][null]
6630Nosworthy, Mr. Richard Catermale21.000A/4. 398867.8[null]SNo boat[null][null]
6730Novel, Mr. Mansouermale28.50026977.2292[null]CNo boat181[null]
6831Nysten, Miss. Anna Sofiafemale22.0003470817.75[null]S13[null][null]
6930Nysveen, Mr. Johan Hansenmale61.0003453646.2375[null]SNo boat[null][null]
7030O'Brien, Mr. Thomasmale29.8811376673040171037036515.5[null]QNo boat[null][null]
7130O'Brien, Mr. Timothymale29.881137667304017003309797.8292[null]QNo boat[null][null]
7231O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)female29.8811376673040171037036515.5[null]QNo boat[null][null]
7330O'Connell, Mr. Patrick Dmale29.881137667304017003349127.7333[null]QNo boat[null][null]
7430O'Connor, Mr. Mauricemale29.881137667304017003710607.75[null]QNo boat[null][null]
7530O'Connor, Mr. Patrickmale29.881137667304017003667137.75[null]QNo boat[null][null]
7630Odahl, Mr. Nils Martinmale23.00072679.225[null]SNo boat[null][null]
7730O'Donoghue, Ms. Bridgetfemale29.881137667304017003648567.75[null]QNo boat[null][null]
7831O'Driscoll, Miss. Bridgetfemale29.88113766730401700143117.75[null]QD[null][null]
7931O'Dwyer, Miss. Ellen "Nellie"female29.881137667304017003309597.8792[null]QNo boat[null][null]
8031Ohman, Miss. Velinfemale22.0003470857.775[null]SC[null][null]
8131O'Keefe, Mr. Patrickmale29.881137667304017003684027.75[null]QB[null][null]
8231O'Leary, Miss. Hanora "Norah"female29.881137667304017003309197.8292[null]Q13[null][null]
8331Olsen, Master. Artur Karlmale9.001C 173683.1708[null]S13[null][null]
8430Olsen, Mr. Henry Margidomale28.000C 400122.525[null]SNo boat173[null]
8530Olsen, Mr. Karl Siegwart Andreasmale42.00145798.4042[null]SNo boat[null][null]
8630Olsen, Mr. Ole Martinmale29.88113766730401700Fa 2653027.3125[null]SNo boat[null][null]
8730Olsson, Miss. Elinafemale31.0003504077.8542[null]SNo boat[null][null]
8830Olsson, Mr. Nils Johan Goranssonmale28.0003474647.8542[null]SNo boat[null][null]
8931Olsson, Mr. Oscar Wilhelmmale32.0003470797.775[null]SA[null][null]
9030Olsvigen, Mr. Thor Andersonmale20.00065639.225[null]SNo boat89Oslo, Norway Cameron, WI
9130Oreskovic, Miss. Jelkafemale23.0003150858.6625[null]SNo boat[null][null]
9230Oreskovic, Miss. Marijafemale20.0003150968.6625[null]SNo boat[null][null]
9330Oreskovic, Mr. Lukamale20.0003150948.6625[null]SNo boat[null][null]
9430Osen, Mr. Olaf Elonmale16.00075349.2167[null]SNo boat[null][null]
9531Osman, Mrs. Marafemale31.0003492448.6833[null]SNo boat[null][null]
9630O'Sullivan, Miss. Bridget Maryfemale29.881137667304017003309097.6292[null]QNo boat[null][null]
9730Palsson, Master. Gosta Leonardmale2.03134990921.075[null]SNo boat4[null]
9830Palsson, Master. Paul Folkemale6.03134990921.075[null]SNo boat[null][null]
9930Palsson, Miss. Stina Violafemale3.03134990921.075[null]SNo boat[null][null]
10030Palsson, Miss. Torborg Danirafemale8.03134990921.075[null]SNo boat[null][null]
Rows: 1-100 of 1309 | Columns: 14

See also

VastFrame.interpolate() : Fill missing values by interpolating.
VastColumn.fill_outliers() : Fill the outliers using the input method.