Missing Values

Missing values occur when no data value is stored for the variable in an observation and are most often represented with a NULL or None. Not handling them can lead to unexpected results (for example, some ML algorithms can’t handle missing values at all) and worse, it can lead to incorrect conclusions.

There are 3 main types of missing values:

• MCAR (Missing Completely at Random): The events that lead to any particular data-item being missing occur entirely at random. For example, in IOT, we can lose sensory data in transmission.

• MAR (Missing {Conditionally} at Random): Missing data doesn’t happen at random and is instead related to some of the observed data. For example, some students may have not answered to some specific questions of a test because they were absent during the relevant lesson.

• MNAR (Missing not at Random): The value of the variable that’s missing is related to the reason it’s missing. For example, if someone didn’t subscribe to a loyalty program, we can leave the cell empty.

Different types of missing values tend to suggest different methods for imputing them. For example, when dealing with MCAR values, you can use mathematical aggregations to impute the missing values. For MNAR values, we can simply create another category. MAR values, however, we’ll need to do some more investigation before deciding how to impute the data.

To see how to handle missing values in vastorbit, we’ll use the well-known titanic dataset.

from vastorbit.datasets import load_titanic

titanic = load_titanic()
titanic.head(100)

	123 pclass Integer 100%	123 survived Integer 100%	Abc name Varchar(164) 100%	Abc sex Varchar(20) 100%	123 age Double 79%	123 sibsp Integer 100%	123 parch Integer 100%	Abc ticket Varchar(36) 100%	123 fare Double 99%	Abc cabin Varchar(30) 22%	Abc embarked Varchar(20) 99%	Abc boat Varchar(100) 37%	123 body Integer 9%	Abc home.dest Varchar(100) 56%
1	3	1	McCormack, Mr. Thomas Joseph	male	[null]	0	0	367228	7.75	[null]	Q	[null]	[null]	[null]
2	3	1	McCoy, Miss. Agnes	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
3	3	1	McCoy, Miss. Alicia	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
4	3	1	McCoy, Mr. Bernard	male	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
5	3	1	McDermott, Miss. Brigdet Delia	female	[null]	0	0	330932	7.7875	[null]	Q	13	[null]	[null]
6	3	0	McEvoy, Mr. Michael	male	[null]	0	0	36568	15.5	[null]	Q	[null]	[null]	[null]
7	3	1	McGovern, Miss. Mary	female	[null]	0	0	330931	7.8792	[null]	Q	13	[null]	[null]
8	3	1	McGowan, Miss. Anna "Annie"	female	15.0	0	0	330923	8.0292	[null]	Q	[null]	[null]	[null]
9	3	0	McGowan, Miss. Katherine	female	35.0	0	0	9232	7.75	[null]	Q	[null]	[null]	[null]
10	3	0	McMahon, Mr. Martin	male	[null]	0	0	370372	7.75	[null]	Q	[null]	[null]	[null]
11	3	0	McNamee, Mr. Neal	male	24.0	1	0	376566	16.1	[null]	S	[null]	[null]	[null]
12	3	0	McNamee, Mrs. Neal (Eileen O'Leary)	female	19.0	1	0	376566	16.1	[null]	S	[null]	53	[null]
13	3	0	McNeill, Miss. Bridget	female	[null]	0	0	370368	7.75	[null]	Q	[null]	[null]	[null]
14	3	0	Meanwell, Miss. (Marion Ogden)	female	[null]	0	0	SOTON/O.Q. 392087	8.05	[null]	S	[null]	[null]	[null]
15	3	0	Meek, Mrs. Thomas (Annie Louise Rowley)	female	[null]	0	0	343095	8.05	[null]	S	[null]	[null]	[null]
16	3	0	Meo, Mr. Alfonzo	male	55.5	0	0	A.5. 11206	8.05	[null]	S	[null]	201	[null]
17	3	0	Mernagh, Mr. Robert	male	[null]	0	0	368703	7.75	[null]	Q	[null]	[null]	[null]
18	3	1	Midtsjo, Mr. Karl Albert	male	21.0	0	0	345501	7.775	[null]	S	15	[null]	[null]
19	3	0	Miles, Mr. Frank	male	[null]	0	0	359306	8.05	[null]	S	[null]	[null]	[null]
20	3	0	Mineff, Mr. Ivan	male	24.0	0	0	349233	7.8958	[null]	S	[null]	[null]	[null]
21	3	0	Minkoff, Mr. Lazar	male	21.0	0	0	349211	7.8958	[null]	S	[null]	[null]	[null]
22	3	0	Mionoff, Mr. Stoytcho	male	28.0	0	0	349207	7.8958	[null]	S	[null]	[null]	[null]
23	3	0	Mitkoff, Mr. Mito	male	[null]	0	0	349221	7.8958	[null]	S	[null]	[null]	[null]
24	3	1	Mockler, Miss. Helen Mary "Ellie"	female	[null]	0	0	330980	7.8792	[null]	Q	16	[null]	[null]
25	3	0	Moen, Mr. Sigurd Hansen	male	25.0	0	0	348123	7.65	F G73	S	[null]	309	[null]
26	3	1	Moor, Master. Meier	male	6.0	0	1	392096	12.475	E121	S	14	[null]	[null]
27	3	1	Moor, Mrs. (Beila)	female	27.0	0	1	392096	12.475	E121	S	14	[null]	[null]
28	3	0	Moore, Mr. Leonard Charles	male	[null]	0	0	A4. 54510	8.05	[null]	S	[null]	[null]	[null]
29	3	1	Moran, Miss. Bertha	female	[null]	1	0	371110	24.15	[null]	Q	16	[null]	[null]
30	3	0	Moran, Mr. Daniel J	male	[null]	1	0	371110	24.15	[null]	Q	[null]	[null]	[null]
31	3	0	Moran, Mr. James	male	[null]	0	0	330877	8.4583	[null]	Q	[null]	[null]	[null]
32	3	0	Morley, Mr. William	male	34.0	0	0	364506	8.05	[null]	S	[null]	[null]	[null]
33	3	0	Morrow, Mr. Thomas Rowan	male	[null]	0	0	372622	7.75	[null]	Q	[null]	[null]	[null]
34	3	1	Moss, Mr. Albert Johan	male	[null]	0	0	312991	7.775	[null]	S	B	[null]	[null]
35	3	1	Moubarek, Master. Gerios	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
36	3	1	Moubarek, Master. Halim Gonios ("William George")	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
37	3	1	Moubarek, Mrs. George (Omine "Amenia" Alexander)	female	[null]	0	2	2661	15.2458	[null]	C	C	[null]	[null]
38	3	1	Moussa, Mrs. (Mantoura Boulos)	female	[null]	0	0	2626	7.2292	[null]	C	[null]	[null]	[null]
39	3	0	Moutal, Mr. Rahamin Haim	male	[null]	0	0	374746	8.05	[null]	S	[null]	[null]	[null]
40	3	1	Mullens, Miss. Katherine "Katie"	female	[null]	0	0	35852	7.7333	[null]	Q	16	[null]	[null]
41	3	1	Mulvihill, Miss. Bertha E	female	24.0	0	0	382653	7.75	[null]	Q	15	[null]	[null]
42	3	0	Murdlin, Mr. Joseph	male	[null]	0	0	A./5. 3235	8.05	[null]	S	[null]	[null]	[null]
43	3	1	Murphy, Miss. Katherine "Kate"	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
44	3	1	Murphy, Miss. Margaret Jane	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
45	3	1	Murphy, Miss. Nora	female	[null]	0	0	36568	15.5	[null]	Q	16	[null]	[null]
46	3	0	Myhrman, Mr. Pehr Fabian Oliver Malkolm	male	18.0	0	0	347078	7.75	[null]	S	[null]	[null]	[null]
47	3	0	Naidenoff, Mr. Penko	male	22.0	0	0	349206	7.8958	[null]	S	[null]	[null]	[null]
48	3	1	Najib, Miss. Adele Kiamie "Jane"	female	15.0	0	0	2667	7.225	[null]	C	C	[null]	[null]
49	3	1	Nakid, Miss. Maria ("Mary")	female	1.0	0	2	2653	15.7417	[null]	C	C	[null]	[null]
50	3	1	Nakid, Mr. Sahid	male	20.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
51	3	1	Nakid, Mrs. Said (Waika "Mary" Mowad)	female	19.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
52	3	0	Nancarrow, Mr. William Henry	male	33.0	0	0	A./5. 3338	8.05	[null]	S	[null]	[null]	[null]
53	3	0	Nankoff, Mr. Minko	male	[null]	0	0	349218	7.8958	[null]	S	[null]	[null]	[null]
54	3	0	Nasr, Mr. Mustafa	male	[null]	0	0	2652	7.2292	[null]	C	[null]	[null]	[null]
55	3	0	Naughton, Miss. Hannah	female	[null]	0	0	365237	7.75	[null]	Q	[null]	[null]	[null]
56	3	0	Nenkoff, Mr. Christo	male	[null]	0	0	349234	7.8958	[null]	S	[null]	[null]	[null]
57	3	1	Nicola-Yarred, Master. Elias	male	12.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
58	3	1	Nicola-Yarred, Miss. Jamila	female	14.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
59	3	0	Nieminen, Miss. Manta Josefina	female	29.0	0	0	3101297	7.925	[null]	S	[null]	[null]	[null]
60	3	0	Niklasson, Mr. Samuel	male	28.0	0	0	363611	8.05	[null]	S	[null]	[null]	[null]
61	3	1	Nilsson, Miss. Berta Olivia	female	18.0	0	0	347066	7.775	[null]	S	D	[null]	[null]
62	3	1	Nilsson, Miss. Helmina Josefina	female	26.0	0	0	347470	7.8542	[null]	S	13	[null]	[null]
63	3	0	Nilsson, Mr. August Ferdinand	male	21.0	0	0	350410	7.8542	[null]	S	[null]	[null]	[null]
64	3	0	Nirva, Mr. Iisakki Antino Aijo	male	41.0	0	0	SOTON/O2 3101272	7.125	[null]	S	[null]	[null]	Finland Sudbury, ON
65	3	1	Niskanen, Mr. Juha	male	39.0	0	0	STON/O 2. 3101289	7.925	[null]	S	9	[null]	[null]
66	3	0	Nosworthy, Mr. Richard Cater	male	21.0	0	0	A/4. 39886	7.8	[null]	S	[null]	[null]	[null]
67	3	0	Novel, Mr. Mansouer	male	28.5	0	0	2697	7.2292	[null]	C	[null]	181	[null]
68	3	1	Nysten, Miss. Anna Sofia	female	22.0	0	0	347081	7.75	[null]	S	13	[null]	[null]
69	3	0	Nysveen, Mr. Johan Hansen	male	61.0	0	0	345364	6.2375	[null]	S	[null]	[null]	[null]
70	3	0	O'Brien, Mr. Thomas	male	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
71	3	0	O'Brien, Mr. Timothy	male	[null]	0	0	330979	7.8292	[null]	Q	[null]	[null]	[null]
72	3	1	O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)	female	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
73	3	0	O'Connell, Mr. Patrick D	male	[null]	0	0	334912	7.7333	[null]	Q	[null]	[null]	[null]
74	3	0	O'Connor, Mr. Maurice	male	[null]	0	0	371060	7.75	[null]	Q	[null]	[null]	[null]
75	3	0	O'Connor, Mr. Patrick	male	[null]	0	0	366713	7.75	[null]	Q	[null]	[null]	[null]
76	3	0	Odahl, Mr. Nils Martin	male	23.0	0	0	7267	9.225	[null]	S	[null]	[null]	[null]
77	3	0	O'Donoghue, Ms. Bridget	female	[null]	0	0	364856	7.75	[null]	Q	[null]	[null]	[null]
78	3	1	O'Driscoll, Miss. Bridget	female	[null]	0	0	14311	7.75	[null]	Q	D	[null]	[null]
79	3	1	O'Dwyer, Miss. Ellen "Nellie"	female	[null]	0	0	330959	7.8792	[null]	Q	[null]	[null]	[null]
80	3	1	Ohman, Miss. Velin	female	22.0	0	0	347085	7.775	[null]	S	C	[null]	[null]
81	3	1	O'Keefe, Mr. Patrick	male	[null]	0	0	368402	7.75	[null]	Q	B	[null]	[null]
82	3	1	O'Leary, Miss. Hanora "Norah"	female	[null]	0	0	330919	7.8292	[null]	Q	13	[null]	[null]
83	3	1	Olsen, Master. Artur Karl	male	9.0	0	1	C 17368	3.1708	[null]	S	13	[null]	[null]
84	3	0	Olsen, Mr. Henry Margido	male	28.0	0	0	C 4001	22.525	[null]	S	[null]	173	[null]
85	3	0	Olsen, Mr. Karl Siegwart Andreas	male	42.0	0	1	4579	8.4042	[null]	S	[null]	[null]	[null]
86	3	0	Olsen, Mr. Ole Martin	male	[null]	0	0	Fa 265302	7.3125	[null]	S	[null]	[null]	[null]
87	3	0	Olsson, Miss. Elina	female	31.0	0	0	350407	7.8542	[null]	S	[null]	[null]	[null]
88	3	0	Olsson, Mr. Nils Johan Goransson	male	28.0	0	0	347464	7.8542	[null]	S	[null]	[null]	[null]
89	3	1	Olsson, Mr. Oscar Wilhelm	male	32.0	0	0	347079	7.775	[null]	S	A	[null]	[null]
90	3	0	Olsvigen, Mr. Thor Anderson	male	20.0	0	0	6563	9.225	[null]	S	[null]	89	Oslo, Norway Cameron, WI
91	3	0	Oreskovic, Miss. Jelka	female	23.0	0	0	315085	8.6625	[null]	S	[null]	[null]	[null]
92	3	0	Oreskovic, Miss. Marija	female	20.0	0	0	315096	8.6625	[null]	S	[null]	[null]	[null]
93	3	0	Oreskovic, Mr. Luka	male	20.0	0	0	315094	8.6625	[null]	S	[null]	[null]	[null]
94	3	0	Osen, Mr. Olaf Elon	male	16.0	0	0	7534	9.2167	[null]	S	[null]	[null]	[null]
95	3	1	Osman, Mrs. Mara	female	31.0	0	0	349244	8.6833	[null]	S	[null]	[null]	[null]
96	3	0	O'Sullivan, Miss. Bridget Mary	female	[null]	0	0	330909	7.6292	[null]	Q	[null]	[null]	[null]
97	3	0	Palsson, Master. Gosta Leonard	male	2.0	3	1	349909	21.075	[null]	S	[null]	4	[null]
98	3	0	Palsson, Master. Paul Folke	male	6.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
99	3	0	Palsson, Miss. Stina Viola	female	3.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
100	3	0	Palsson, Miss. Torborg Danira	female	8.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]

We can examine the missing values with the count() method.

titanic.count_percent()

	count	percent
"pclass"	1309.0	100.0
"survived"	1309.0	100.0
"name"	1309.0	100.0
"sex"	1309.0	100.0
"sibsp"	1309.0	100.0
"parch"	1309.0	100.0
"ticket"	1309.0	100.0
"fare"	1308.0	99.924
"embarked"	1307.0	99.847
"age"	1046.0	79.908
"home.dest"	745.0	56.914
"boat"	486.0	37.128
"cabin"	295.0	22.536
"body"	121.0	9.244

The missing values for boat are MNAR; missing values simply indicate that the passengers didn’t pay for a lifeboat. We can replace all the missing values with a new category No Lifeboat using the fillna() method.

titanic["boat"].fillna("No Lifeboat")
titanic["boat"]

	Abc boat Varchar(100)
1	No Lifeboat
2	16
3	16
4	16
5	13
6	No Lifeboat
7	13
8	No Lifeboat
9	No Lifeboat
10	No Lifeboat
11	No Lifeboat
12	No Lifeboat
13	No Lifeboat
14	No Lifeboat
15	No Lifeboat
16	No Lifeboat
17	No Lifeboat
18	15
19	No Lifeboat
20	No Lifeboat

Missing values for age seem to be MCAR, so the best way to impute them is with mathematical aggregations. Let’s impute the age using the average age of passengers of the same sex and class.

titanic["age"].fillna(
    method = "avg",
    by = ["pclass", "sex"],
)
titanic["age"]

	123 age Double
1	29.0
2	2.0
3	25.0
4	63.0
5	53.0
6	18.0
7	24.0
8	26.0
9	50.0
10	32.0
11	47.0
12	42.0
13	29.0
14	19.0
15	35.0
16	30.0
17	58.0
18	45.0
19	22.0
20	44.0

The features embarked and fare have a couple missing values. Instead of using a technique to impute them, we can just drop them with the dropna() method.

titanic["fare"].dropna()
titanic["embarked"].dropna()

	123 pclass Integer 100%	123 survived Integer 100%	Abc name Varchar(164) 100%	Abc sex Varchar(20) 100%	123 age Double 100%	123 sibsp Integer 100%	123 parch Integer 100%	Abc ticket Varchar(36) 100%	123 fare Double 100%	Abc cabin Varchar(30) 22%	Abc embarked Varchar(20) 100%	Abc boat Varchar(100) 100%	123 body Integer 9%	Abc home.dest Varchar(100) 56%
1	2	1	Abelson, Mrs. Samuel (Hannah Wizosky)	female	28.0	1	0	P/PP 3381	24.0	[null]	C	10	[null]	Russia New York, NY
2	2	1	Angle, Mrs. William A (Florence "Mary" Agnes Hughes)	female	36.0	1	0	226875	26.0	[null]	S	11	[null]	Warwick, England
3	2	1	Ball, Mrs. (Ada E Hall)	female	36.0	0	0	28551	13.0	D	S	10	[null]	Bristol, Avon / Jacksonville, FL
4	2	1	Beane, Mrs. Edward (Ethel Clarke)	female	19.0	1	0	2908	26.0	[null]	S	13	[null]	Norwich / New York, NY
5	2	1	Becker, Miss. Marion Louise	female	4.0	2	1	230136	39.0	F4	S	11	[null]	Guntur, India / Benton Harbour, MI
6	2	1	Becker, Miss. Ruth Elizabeth	female	12.0	2	1	230136	39.0	F4	S	13	[null]	Guntur, India / Benton Harbour, MI
7	2	1	Becker, Mrs. Allen Oliver (Nellie E Baumgardner)	female	36.0	0	3	230136	39.0	F4	S	11	[null]	Guntur, India / Benton Harbour, MI
8	2	1	Bentham, Miss. Lilian W	female	19.0	0	0	28404	13.0	[null]	S	12	[null]	Rochester, NY
9	2	1	Brown, Miss. Amelia "Mildred"	female	24.0	0	0	248733	13.0	F33	S	11	[null]	London / Montreal, PQ
10	2	1	Brown, Miss. Edith Eileen	female	15.0	0	2	29750	39.0	[null]	S	14	[null]	Cape Town, South Africa / Seattle, WA
11	2	1	Brown, Mrs. Thomas William Solomon (Elizabeth Catherine Ford)	female	40.0	1	1	29750	39.0	[null]	S	14	[null]	Cape Town, South Africa / Seattle, WA
12	2	1	Bryhl, Miss. Dagmar Jenny Ingeborg	female	20.0	1	0	236853	26.0	[null]	S	12	[null]	Skara, Sweden / Rockford, IL
13	2	1	Buss, Miss. Kate	female	36.0	0	0	27849	13.0	[null]	S	9	[null]	Sittingbourne, England / San Diego, CA
14	2	1	Bystrom, Mrs. (Karolina)	female	42.0	0	0	236852	13.0	[null]	S	No Lifeboat	[null]	New York, NY
15	2	1	Caldwell, Mrs. Albert Francis (Sylvia Mae Harbaugh)	female	22.0	1	1	248738	29.0	[null]	S	13	[null]	Bangkok, Thailand / Roseville, IL
16	2	1	Cameron, Miss. Clear Annie	female	35.0	0	0	F.C.C. 13528	21.0	[null]	S	14	[null]	Mamaroneck, NY
17	2	0	Carter, Mrs. Ernest Courtenay (Lilian Hughes)	female	44.0	1	0	244252	26.0	[null]	S	No Lifeboat	[null]	London
18	2	0	Chapman, Mrs. John Henry (Sara Elizabeth Lawry)	female	29.0	1	0	SC/AH 29037	26.0	[null]	S	No Lifeboat	[null]	Cornwall / Spokane, WA
19	2	1	Christy, Miss. Julie Rachel	female	25.0	1	1	237789	30.0	[null]	S	12	[null]	London
20	2	1	Christy, Mrs. (Alice Frances)	female	45.0	0	2	237789	30.0	[null]	S	12	[null]	London

The fillna() method offers many options. Let’s use the help() function to view its parameters.

help(titanic["embarked"].fillna)

print(titanic.current_relation())

Depending on the circumstances, we’ll need to investigate to find the most suitable solution.

In conclusion, before imputing missing data, you have to understand why it might be missing and how it relates to the rest of your dataset.