Loading...

vastorbit.VastFrame.dropna

VastFrame.dropna(columns: Annotated[str | list[str], 'STRING representing one column or a list of columns'] | None = None) VastFrame

Filters the specified VastColumns in a VastFrame for missing values.

Parameters:

columns (SQLColumns, optional) – List of the VastColumns names. If empty, all VastColumns are selected.

Returns:

self

Return type:

VastFrame

Examples

We import vastorbit:

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

For this example, we will use the Titanic dataset:

from vastorbit.datasets import load_titanic

vdf = load_titanic()
123
pclass
Integer
123
survived
Integer
Abc
name
Varchar(164)
Abc
sex
Varchar(20)
123
age
Double
123
sibsp
Integer
123
parch
Integer
Abc
ticket
Varchar(36)
123
fare
Double
Abc
cabin
Varchar(30)
Abc
embarked
Varchar(20)
Abc
boat
Varchar(100)
123
body
Integer
Abc
home.dest
Varchar(100)
131McCormack, Mr. Thomas Josephmale[null]003672287.75[null]Q[null][null][null]
231McCoy, Miss. Agnesfemale[null]2036722623.25[null]Q16[null][null]
331McCoy, Miss. Aliciafemale[null]2036722623.25[null]Q16[null][null]
431McCoy, Mr. Bernardmale[null]2036722623.25[null]Q16[null][null]
531McDermott, Miss. Brigdet Deliafemale[null]003309327.7875[null]Q13[null][null]
630McEvoy, Mr. Michaelmale[null]003656815.5[null]Q[null][null][null]
731McGovern, Miss. Maryfemale[null]003309317.8792[null]Q13[null][null]
831McGowan, Miss. Anna "Annie"female15.0003309238.0292[null]Q[null][null][null]
930McGowan, Miss. Katherinefemale35.00092327.75[null]Q[null][null][null]
1030McMahon, Mr. Martinmale[null]003703727.75[null]Q[null][null][null]
1130McNamee, Mr. Nealmale24.01037656616.1[null]S[null][null][null]
1230McNamee, Mrs. Neal (Eileen O'Leary)female19.01037656616.1[null]S[null]53[null]
1330McNeill, Miss. Bridgetfemale[null]003703687.75[null]Q[null][null][null]
1430Meanwell, Miss. (Marion Ogden)female[null]00SOTON/O.Q. 3920878.05[null]S[null][null][null]
1530Meek, Mrs. Thomas (Annie Louise Rowley)female[null]003430958.05[null]S[null][null][null]
1630Meo, Mr. Alfonzomale55.500A.5. 112068.05[null]S[null]201[null]
1730Mernagh, Mr. Robertmale[null]003687037.75[null]Q[null][null][null]
1831Midtsjo, Mr. Karl Albertmale21.0003455017.775[null]S15[null][null]
1930Miles, Mr. Frankmale[null]003593068.05[null]S[null][null][null]
2030Mineff, Mr. Ivanmale24.0003492337.8958[null]S[null][null][null]
2130Minkoff, Mr. Lazarmale21.0003492117.8958[null]S[null][null][null]
2230Mionoff, Mr. Stoytchomale28.0003492077.8958[null]S[null][null][null]
2330Mitkoff, Mr. Mitomale[null]003492217.8958[null]S[null][null][null]
2431Mockler, Miss. Helen Mary "Ellie"female[null]003309807.8792[null]Q16[null][null]
2530Moen, Mr. Sigurd Hansenmale25.0003481237.65F G73S[null]309[null]
2631Moor, Master. Meiermale6.00139209612.475E121S14[null][null]
2731Moor, Mrs. (Beila)female27.00139209612.475E121S14[null][null]
2830Moore, Mr. Leonard Charlesmale[null]00A4. 545108.05[null]S[null][null][null]
2931Moran, Miss. Berthafemale[null]1037111024.15[null]Q16[null][null]
3030Moran, Mr. Daniel Jmale[null]1037111024.15[null]Q[null][null][null]
3130Moran, Mr. Jamesmale[null]003308778.4583[null]Q[null][null][null]
3230Morley, Mr. Williammale34.0003645068.05[null]S[null][null][null]
3330Morrow, Mr. Thomas Rowanmale[null]003726227.75[null]Q[null][null][null]
3431Moss, Mr. Albert Johanmale[null]003129917.775[null]SB[null][null]
3531Moubarek, Master. Geriosmale[null]11266115.2458[null]CC[null][null]
3631Moubarek, Master. Halim Gonios ("William George")male[null]11266115.2458[null]CC[null][null]
3731Moubarek, Mrs. George (Omine "Amenia" Alexander)female[null]02266115.2458[null]CC[null][null]
3831Moussa, Mrs. (Mantoura Boulos)female[null]0026267.2292[null]C[null][null][null]
3930Moutal, Mr. Rahamin Haimmale[null]003747468.05[null]S[null][null][null]
4031Mullens, Miss. Katherine "Katie"female[null]00358527.7333[null]Q16[null][null]
4131Mulvihill, Miss. Bertha Efemale24.0003826537.75[null]Q15[null][null]
4230Murdlin, Mr. Josephmale[null]00A./5. 32358.05[null]S[null][null][null]
4331Murphy, Miss. Katherine "Kate"female[null]1036723015.5[null]Q16[null][null]
4431Murphy, Miss. Margaret Janefemale[null]1036723015.5[null]Q16[null][null]
4531Murphy, Miss. Norafemale[null]003656815.5[null]Q16[null][null]
4630Myhrman, Mr. Pehr Fabian Oliver Malkolmmale18.0003470787.75[null]S[null][null][null]
4730Naidenoff, Mr. Penkomale22.0003492067.8958[null]S[null][null][null]
4831Najib, Miss. Adele Kiamie "Jane"female15.00026677.225[null]CC[null][null]
4931Nakid, Miss. Maria ("Mary")female1.002265315.7417[null]CC[null][null]
5031Nakid, Mr. Sahidmale20.011265315.7417[null]CC[null][null]
5131Nakid, Mrs. Said (Waika "Mary" Mowad)female19.011265315.7417[null]CC[null][null]
5230Nancarrow, Mr. William Henrymale33.000A./5. 33388.05[null]S[null][null][null]
5330Nankoff, Mr. Minkomale[null]003492187.8958[null]S[null][null][null]
5430Nasr, Mr. Mustafamale[null]0026527.2292[null]C[null][null][null]
5530Naughton, Miss. Hannahfemale[null]003652377.75[null]Q[null][null][null]
5630Nenkoff, Mr. Christomale[null]003492347.8958[null]S[null][null][null]
5731Nicola-Yarred, Master. Eliasmale12.010265111.2417[null]CC[null][null]
5831Nicola-Yarred, Miss. Jamilafemale14.010265111.2417[null]CC[null][null]
5930Nieminen, Miss. Manta Josefinafemale29.00031012977.925[null]S[null][null][null]
6030Niklasson, Mr. Samuelmale28.0003636118.05[null]S[null][null][null]
6131Nilsson, Miss. Berta Oliviafemale18.0003470667.775[null]SD[null][null]
6231Nilsson, Miss. Helmina Josefinafemale26.0003474707.8542[null]S13[null][null]
6330Nilsson, Mr. August Ferdinandmale21.0003504107.8542[null]S[null][null][null]
6430Nirva, Mr. Iisakki Antino Aijomale41.000SOTON/O2 31012727.125[null]S[null][null]Finland Sudbury, ON
6531Niskanen, Mr. Juhamale39.000STON/O 2. 31012897.925[null]S9[null][null]
6630Nosworthy, Mr. Richard Catermale21.000A/4. 398867.8[null]S[null][null][null]
6730Novel, Mr. Mansouermale28.50026977.2292[null]C[null]181[null]
6831Nysten, Miss. Anna Sofiafemale22.0003470817.75[null]S13[null][null]
6930Nysveen, Mr. Johan Hansenmale61.0003453646.2375[null]S[null][null][null]
7030O'Brien, Mr. Thomasmale[null]1037036515.5[null]Q[null][null][null]
7130O'Brien, Mr. Timothymale[null]003309797.8292[null]Q[null][null][null]
7231O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)female[null]1037036515.5[null]Q[null][null][null]
7330O'Connell, Mr. Patrick Dmale[null]003349127.7333[null]Q[null][null][null]
7430O'Connor, Mr. Mauricemale[null]003710607.75[null]Q[null][null][null]
7530O'Connor, Mr. Patrickmale[null]003667137.75[null]Q[null][null][null]
7630Odahl, Mr. Nils Martinmale23.00072679.225[null]S[null][null][null]
7730O'Donoghue, Ms. Bridgetfemale[null]003648567.75[null]Q[null][null][null]
7831O'Driscoll, Miss. Bridgetfemale[null]00143117.75[null]QD[null][null]
7931O'Dwyer, Miss. Ellen "Nellie"female[null]003309597.8792[null]Q[null][null][null]
8031Ohman, Miss. Velinfemale22.0003470857.775[null]SC[null][null]
8131O'Keefe, Mr. Patrickmale[null]003684027.75[null]QB[null][null]
8231O'Leary, Miss. Hanora "Norah"female[null]003309197.8292[null]Q13[null][null]
8331Olsen, Master. Artur Karlmale9.001C 173683.1708[null]S13[null][null]
8430Olsen, Mr. Henry Margidomale28.000C 400122.525[null]S[null]173[null]
8530Olsen, Mr. Karl Siegwart Andreasmale42.00145798.4042[null]S[null][null][null]
8630Olsen, Mr. Ole Martinmale[null]00Fa 2653027.3125[null]S[null][null][null]
8730Olsson, Miss. Elinafemale31.0003504077.8542[null]S[null][null][null]
8830Olsson, Mr. Nils Johan Goranssonmale28.0003474647.8542[null]S[null][null][null]
8931Olsson, Mr. Oscar Wilhelmmale32.0003470797.775[null]SA[null][null]
9030Olsvigen, Mr. Thor Andersonmale20.00065639.225[null]S[null]89Oslo, Norway Cameron, WI
9130Oreskovic, Miss. Jelkafemale23.0003150858.6625[null]S[null][null][null]
9230Oreskovic, Miss. Marijafemale20.0003150968.6625[null]S[null][null][null]
9330Oreskovic, Mr. Lukamale20.0003150948.6625[null]S[null][null][null]
9430Osen, Mr. Olaf Elonmale16.00075349.2167[null]S[null][null][null]
9531Osman, Mrs. Marafemale31.0003492448.6833[null]S[null][null][null]
9630O'Sullivan, Miss. Bridget Maryfemale[null]003309097.6292[null]Q[null][null][null]
9730Palsson, Master. Gosta Leonardmale2.03134990921.075[null]S[null]4[null]
9830Palsson, Master. Paul Folkemale6.03134990921.075[null]S[null][null][null]
9930Palsson, Miss. Stina Violafemale3.03134990921.075[null]S[null][null][null]
10030Palsson, Miss. Torborg Danirafemale8.03134990921.075[null]S[null][null][null]
Rows: 1-100 | Columns: 14

In the above dataset, notice that the first and last entries are identical i.e. duplicates.

Note

vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.

We can see the count of each column to check if any column has missing values.

vdf.count()
count
"pclass"1309.0
"survived"1309.0
"name"1309.0
"sex"1309.0
"age"1046.0
"sibsp"1309.0
"parch"1309.0
"ticket"1309.0
"fare"1308.0
"cabin"295.0
"embarked"1307.0
"boat"486.0
"body"121.0
"home.dest"745.0
Rows: 1-14 | Columns: 2

From the above table, we can see that there are multiple columns with missing/NA values.

Using dropna, we can select which columns do we want the dataset to filter by:

vdf.dropna(columns = ["fare", "embarked", "age"])
123
pclass
Integer
123
survived
Integer
Abc
name
Varchar(164)
Abc
sex
Varchar(20)
123
age
Double
123
sibsp
Integer
123
parch
Integer
Abc
ticket
Varchar(36)
123
fare
Double
Abc
cabin
Varchar(30)
Abc
embarked
Varchar(20)
Abc
boat
Varchar(100)
123
body
Integer
Abc
home.dest
Varchar(100)
131McGowan, Miss. Anna "Annie"female15.0003309238.0292[null]Q[null][null][null]
230McGowan, Miss. Katherinefemale35.00092327.75[null]Q[null][null][null]
330McNamee, Mr. Nealmale24.01037656616.1[null]S[null][null][null]
430McNamee, Mrs. Neal (Eileen O'Leary)female19.01037656616.1[null]S[null]53[null]
530Meo, Mr. Alfonzomale55.500A.5. 112068.05[null]S[null]201[null]
631Midtsjo, Mr. Karl Albertmale21.0003455017.775[null]S15[null][null]
730Mineff, Mr. Ivanmale24.0003492337.8958[null]S[null][null][null]
830Minkoff, Mr. Lazarmale21.0003492117.8958[null]S[null][null][null]
930Mionoff, Mr. Stoytchomale28.0003492077.8958[null]S[null][null][null]
1030Moen, Mr. Sigurd Hansenmale25.0003481237.65F G73S[null]309[null]
1131Moor, Master. Meiermale6.00139209612.475E121S14[null][null]
1231Moor, Mrs. (Beila)female27.00139209612.475E121S14[null][null]
1330Morley, Mr. Williammale34.0003645068.05[null]S[null][null][null]
1431Mulvihill, Miss. Bertha Efemale24.0003826537.75[null]Q15[null][null]
1530Myhrman, Mr. Pehr Fabian Oliver Malkolmmale18.0003470787.75[null]S[null][null][null]
1630Naidenoff, Mr. Penkomale22.0003492067.8958[null]S[null][null][null]
1731Najib, Miss. Adele Kiamie "Jane"female15.00026677.225[null]CC[null][null]
1831Nakid, Miss. Maria ("Mary")female1.002265315.7417[null]CC[null][null]
1931Nakid, Mr. Sahidmale20.011265315.7417[null]CC[null][null]
2031Nakid, Mrs. Said (Waika "Mary" Mowad)female19.011265315.7417[null]CC[null][null]
2130Nancarrow, Mr. William Henrymale33.000A./5. 33388.05[null]S[null][null][null]
2231Nicola-Yarred, Master. Eliasmale12.010265111.2417[null]CC[null][null]
2331Nicola-Yarred, Miss. Jamilafemale14.010265111.2417[null]CC[null][null]
2430Nieminen, Miss. Manta Josefinafemale29.00031012977.925[null]S[null][null][null]
2530Niklasson, Mr. Samuelmale28.0003636118.05[null]S[null][null][null]
2631Nilsson, Miss. Berta Oliviafemale18.0003470667.775[null]SD[null][null]
2731Nilsson, Miss. Helmina Josefinafemale26.0003474707.8542[null]S13[null][null]
2830Nilsson, Mr. August Ferdinandmale21.0003504107.8542[null]S[null][null][null]
2930Nirva, Mr. Iisakki Antino Aijomale41.000SOTON/O2 31012727.125[null]S[null][null]Finland Sudbury, ON
3031Niskanen, Mr. Juhamale39.000STON/O 2. 31012897.925[null]S9[null][null]
3130Nosworthy, Mr. Richard Catermale21.000A/4. 398867.8[null]S[null][null][null]
3230Novel, Mr. Mansouermale28.50026977.2292[null]C[null]181[null]
3331Nysten, Miss. Anna Sofiafemale22.0003470817.75[null]S13[null][null]
3430Nysveen, Mr. Johan Hansenmale61.0003453646.2375[null]S[null][null][null]
3530Odahl, Mr. Nils Martinmale23.00072679.225[null]S[null][null][null]
3631Ohman, Miss. Velinfemale22.0003470857.775[null]SC[null][null]
3731Olsen, Master. Artur Karlmale9.001C 173683.1708[null]S13[null][null]
3830Olsen, Mr. Henry Margidomale28.000C 400122.525[null]S[null]173[null]
3930Olsen, Mr. Karl Siegwart Andreasmale42.00145798.4042[null]S[null][null][null]
4030Olsson, Miss. Elinafemale31.0003504077.8542[null]S[null][null][null]
4130Olsson, Mr. Nils Johan Goranssonmale28.0003474647.8542[null]S[null][null][null]
4231Olsson, Mr. Oscar Wilhelmmale32.0003470797.775[null]SA[null][null]
4330Olsvigen, Mr. Thor Andersonmale20.00065639.225[null]S[null]89Oslo, Norway Cameron, WI
4430Oreskovic, Miss. Jelkafemale23.0003150858.6625[null]S[null][null][null]
4530Oreskovic, Miss. Marijafemale20.0003150968.6625[null]S[null][null][null]
4630Oreskovic, Mr. Lukamale20.0003150948.6625[null]S[null][null][null]
4730Osen, Mr. Olaf Elonmale16.00075349.2167[null]S[null][null][null]
4831Osman, Mrs. Marafemale31.0003492448.6833[null]S[null][null][null]
4930Palsson, Master. Gosta Leonardmale2.03134990921.075[null]S[null]4[null]
5030Palsson, Master. Paul Folkemale6.03134990921.075[null]S[null][null][null]
5130Palsson, Miss. Stina Violafemale3.03134990921.075[null]S[null][null][null]
5230Palsson, Miss. Torborg Danirafemale8.03134990921.075[null]S[null][null][null]
5330Palsson, Mrs. Nils (Alma Cornelia Berglund)female29.00434990921.075[null]S[null]206[null]
5430Panula, Master. Eino Viljamimale1.041310129539.6875[null]S[null][null][null]
5530Panula, Master. Juha Niilomale7.041310129539.6875[null]S[null][null][null]
5630Panula, Master. Urho Abrahammale2.041310129539.6875[null]S[null][null][null]
5730Panula, Mr. Ernesti Arvidmale16.041310129539.6875[null]S[null][null][null]
5830Panula, Mr. Jaako Arnoldmale14.041310129539.6875[null]S[null][null][null]
5930Panula, Mrs. Juha (Maria Emilia Ojala)female41.005310129539.6875[null]S[null][null][null]
6030Pasic, Mr. Jakobmale21.0003150978.6625[null]S[null][null][null]
6130Patchett, Mr. Georgemale19.00035858514.5[null]S[null][null][null]
6230Pavlovic, Mr. Stefomale32.0003492427.8958[null]S[null][null][null]
6330Peacock, Master. Alfred Edwardmale0.7511SOTON/O.Q. 310131513.775[null]S[null][null][null]
6430Peacock, Miss. Treasteallfemale3.011SOTON/O.Q. 310131513.775[null]S[null][null][null]
6530Peacock, Mrs. Benjamin (Edith Nile)female26.002SOTON/O.Q. 310131513.775[null]S[null][null][null]
6630Pekoniemi, Mr. Edvardmale21.000STON/O 2. 31012947.925[null]S[null][null][null]
6730Peltomaki, Mr. Nikolai Johannesmale25.000STON/O 2. 31012917.925[null]S[null][null][null]
6830Perkin, Mr. John Henrymale22.000A/5 211747.25[null]S[null][null][null]
6931Persson, Mr. Ernst Ulrikmale25.0103470837.775[null]S15[null][null]
7030Petersen, Mr. Mariusmale24.0003424418.05[null]S[null][null][null]
7130Petranec, Miss. Matildafemale28.0003492457.8958[null]S[null][null][null]
7230Petroff, Mr. Nedeliomale19.0003492127.8958[null]S[null][null][null]
7330Petterson, Mr. Johan Emilmale25.0103470767.775[null]S[null][null][null]
7430Pettersson, Miss. Ellen Nataliafemale18.0003470877.775[null]S[null][null][null]
7531Pickard, Mr. Berk (Berk Trembisky)male32.000SOTON/O.Q. 3920788.05E10S9[null][null]
7630Pokrnic, Mr. Matemale17.0003150958.6625[null]S[null][null][null]
7730Pokrnic, Mr. Tomemale24.0003150928.6625[null]S[null][null][null]
7830Rekic, Mr. Tidomale38.0003492497.8958[null]S[null][null][null]
7930Reynolds, Mr. Harold Jmale21.0003426848.05[null]S[null][null][null]
8030Rice, Master. Albertmale10.04138265229.125[null]Q[null][null][null]
8130Rice, Master. Arthurmale4.04138265229.125[null]Q[null][null][null]
8230Rice, Master. Ericmale7.04138265229.125[null]Q[null][null][null]
8330Rice, Master. Eugenemale2.04138265229.125[null]Q[null][null][null]
8430Rice, Master. George Hughmale8.04138265229.125[null]Q[null][null][null]
8530Rice, Mrs. William (Margaret Norton)female39.00538265229.125[null]Q[null]327[null]
8630Riihivouri, Miss. Susanna Juhantytar "Sanni"female22.000310129539.6875[null]S[null][null][null]
8730Rintamaki, Mr. Mattimale35.000STON/O 2. 31012737.125[null]S[null][null][null]
8830Robins, Mr. Alexander Amale50.010A/5. 333714.5[null]S[null]119[null]
8930Robins, Mrs. Alexander A (Grace Charity Laury)female47.010A/5. 333714.5[null]S[null]7[null]
9030Rosblom, Miss. Salli Helenafemale2.01137012920.2125[null]S[null][null][null]
9130Rosblom, Mr. Viktor Richardmale18.01137012920.2125[null]S[null][null][null]
9230Rosblom, Mrs. Viktor (Helena Wilhelmina)female41.00237012920.2125[null]S[null][null][null]
9330Rouse, Mr. Richard Henrymale50.000A/5 35948.05[null]S[null][null][null]
9430Rush, Mr. Alfred George Johnmale16.000A/4. 205898.05[null]S[null][null][null]
9530Saad, Mr. Khalilmale25.00026727.225[null]C[null][null][null]
9630Saether, Mr. Simon Sivertsenmale38.500SOTON/O.Q. 31012627.25[null]S[null]32[null]
9730Sage, Master. William Henrymale14.582CA. 234369.55[null]S[null]67[null]
9830Salander, Mr. Karl Johanmale24.00072669.325[null]S[null][null][null]
9931Salkjelsvik, Miss. Anna Kristinefemale21.0003431207.65[null]SC[null][null]
10030Salonen, Mr. Johan Wernermale39.00031012967.925[null]S[null][null][null]
Rows: 1-100 of 1043 | Columns: 14

Now again, if we look at the count, we will notice that the total count has decreased.

vdf.count()
count
"pclass"1043.0
"survived"1043.0
"name"1043.0
"sex"1043.0
"age"1043.0
"sibsp"1043.0
"parch"1043.0
"ticket"1043.0
"fare"1043.0
"cabin"270.0
"embarked"1043.0
"boat"415.0
"body"119.0
"home.dest"684.0
Rows: 1-14 | Columns: 2

See also

VastFrame.balance() : Balances the VastFrame.
VastFrame.drop() : Drops the VastFrame input columns.