Outliers

Outliers are data points that differ significantly from the rest of the data. While some outliers can reveal some important information (machine failure, systems fraud…), they can also be simple errors.

Some machine learning algorithms are sensitive to outliers. In fact, they can destroy the final predictions because of how much bias they add to the data, and handling outliers in our data is one of the most important parts of the data preparation.

Outliers consist of three main types:

• Global Outliers: Values far outside the entirety of their source dataset.

• Contextual Outliers: Values deviate significantly from the rest of the data points in the same context.

• Collective Outliers: Values that aren’t global or contextual outliers, but as a collection deviate significantly from the entire dataset.

Global outliers are often the most critical type and can add a significant amount of bias into the data. Fortunately, we can easily identify these outliers by computing the Z-Score.

Let’s look at some examples using the Heart Disease dataset. This dataset contains information on patients who are likely to have heart-related complications.

import vastorbit as vo

heart = vo.read_csv("heart.csv")
heart.head(100)

	123 age Integer 100%	123 sex Integer 100%	123 cp Integer 100%	123 trestbps Integer 100%	123 chol Integer 100%	123 fbs Integer 100%	123 restecg Integer 100%	123 thalach Integer 100%	123 exang Integer 100%	123 oldpeak Double 100%	123 slope Integer 100%	123 ca Integer 100%	123 thal Integer 100%	123 target Integer 100%
1	67	1	0	120	229	0	0	129	1	2.6	1	2	3	0
2	63	1	3	145	233	1	0	150	0	2.3	0	0	1	1
3	63	0	0	124	197	0	1	136	1	0.0	1	0	2	0
4	52	1	0	112	230	0	1	160	0	0.0	2	1	2	0
5	58	0	0	130	197	0	1	131	0	0.6	1	0	2	1
6	53	1	0	142	226	0	0	111	1	0.0	2	0	3	1
7	57	1	0	150	276	0	0	112	1	0.6	1	1	1	0
8	44	1	2	130	233	0	1	179	1	0.4	2	0	2	1
9	51	1	2	94	227	0	1	154	1	0.0	2	1	3	1
10	54	0	2	110	214	0	1	158	0	1.6	1	0	2	1
11	40	1	0	110	167	0	0	114	1	2.0	1	0	3	0
12	57	1	1	124	261	0	1	141	0	0.3	2	0	3	0
13	62	0	0	140	268	0	0	160	0	3.6	0	2	2	0
14	53	1	0	140	203	1	0	155	1	3.1	0	0	3	0
15	62	1	1	128	208	1	0	140	0	0.0	2	0	2	1
16	58	1	2	105	240	0	0	154	1	0.6	1	0	3	1
17	70	1	1	156	245	0	0	143	0	0.0	2	0	2	1
18	45	1	0	115	260	0	0	185	0	0.0	2	0	2	1
19	42	1	3	148	244	0	0	178	0	0.8	2	2	2	1
20	58	0	0	170	225	1	0	146	1	2.8	1	2	1	0
21	61	1	0	140	207	0	0	138	1	1.9	2	1	3	0
22	62	0	0	140	268	0	0	160	0	3.6	0	2	2	0
23	60	1	0	130	253	0	1	144	1	1.4	2	1	3	0
24	54	1	0	140	239	0	1	160	0	1.2	2	0	2	1
25	61	1	0	138	166	0	0	125	1	3.6	1	1	2	0
26	63	0	2	135	252	0	0	172	0	0.0	2	0	2	1
27	42	1	2	130	180	0	1	150	0	0.0	2	0	2	1
28	57	1	2	128	229	0	0	150	0	0.4	1	1	3	0
29	44	1	2	130	233	0	1	179	1	0.4	2	0	2	1
30	54	1	0	124	266	0	0	109	1	2.2	1	1	3	0
31	51	1	2	100	222	0	1	143	1	1.2	1	0	2	1
32	58	1	1	125	220	0	1	144	0	0.4	1	4	3	1
33	68	1	2	118	277	0	1	151	0	1.0	2	1	3	1
34	55	1	0	140	217	0	1	111	1	5.6	0	0	3	0
35	42	1	0	136	315	0	1	125	1	1.8	1	0	1	0
36	49	1	2	118	149	0	0	126	0	0.8	2	3	2	0
37	53	0	0	138	234	0	0	160	0	0.0	2	0	2	1
38	52	1	2	172	199	1	1	162	0	0.5	2	0	3	1
39	51	1	3	125	213	0	0	125	1	1.4	2	1	2	1
40	51	1	0	140	261	0	0	186	1	0.0	2	0	2	1
41	70	1	0	145	174	0	1	125	1	2.6	0	0	3	0
42	35	0	0	138	183	0	1	182	0	1.4	2	0	2	1
43	58	1	2	112	230	0	0	165	0	2.5	1	1	3	0
44	59	1	3	160	273	0	0	125	0	0.0	2	0	2	0
45	60	1	0	140	293	0	0	170	0	1.2	1	2	3	0
46	56	1	0	132	184	0	0	105	1	2.1	1	1	1	0
47	35	0	0	138	183	0	1	182	0	1.4	2	0	2	1
48	61	1	0	138	166	0	0	125	1	3.6	1	1	2	0
49	58	0	3	150	283	1	0	162	0	1.0	2	0	2	1
50	52	1	0	128	255	0	1	161	1	0.0	2	1	3	0
51	58	1	1	120	284	0	0	160	0	1.8	1	0	2	0
52	37	1	2	130	250	0	1	187	0	3.5	0	0	2	1
53	52	1	0	128	255	0	1	161	1	0.0	2	1	3	0
54	67	1	0	120	229	0	0	129	1	2.6	1	2	3	0
55	65	1	3	138	282	1	0	174	0	1.4	1	1	2	0
56	46	1	1	101	197	1	1	156	0	0.0	2	0	3	1
57	68	0	2	120	211	0	0	115	0	1.5	1	0	2	1
58	43	1	0	115	303	0	1	181	0	1.2	1	0	2	1
59	68	0	2	120	211	0	0	115	0	1.5	1	0	2	1
60	51	1	0	140	299	0	1	173	1	1.6	2	0	3	0
61	52	1	0	112	230	0	1	160	0	0.0	2	1	2	0
62	64	1	2	140	335	0	1	158	0	0.0	2	0	2	0
63	59	1	3	170	288	0	0	159	0	0.2	1	0	3	0
64	52	1	0	125	212	0	1	168	0	1.0	2	2	3	0
65	59	1	3	160	273	0	0	125	0	0.0	2	0	2	0
66	60	0	3	150	240	0	1	171	0	0.9	2	0	2	1
67	41	1	2	112	250	0	1	179	0	0.0	2	0	2	1
68	41	1	1	110	235	0	1	153	0	0.0	2	0	2	1
69	56	1	1	120	240	0	1	169	0	0.0	0	0	2	1
70	56	1	1	120	236	0	1	178	0	0.8	2	0	2	1
71	48	0	2	130	275	0	1	139	0	0.2	2	0	2	1
72	39	1	2	140	321	0	0	182	0	0.0	2	0	2	1
73	64	1	3	170	227	0	0	155	0	0.6	1	0	3	1
74	57	1	0	140	192	0	1	148	0	0.4	1	0	1	1
75	59	1	3	160	273	0	0	125	0	0.0	2	0	2	0
76	60	1	0	130	206	0	0	132	1	2.4	1	2	3	0
77	61	1	0	140	207	0	0	138	1	1.9	2	1	3	0
78	43	0	2	122	213	0	1	165	0	0.2	1	0	2	1
79	54	1	0	120	188	0	1	113	0	1.4	1	1	3	0
80	59	1	0	138	271	0	0	182	0	0.0	2	0	2	1
81	57	1	0	132	207	0	1	168	1	0.0	2	0	3	1
82	57	1	1	154	232	0	0	164	0	0.0	2	1	2	0
83	57	1	0	130	131	0	1	115	1	1.2	1	1	3	0
84	48	1	0	124	274	0	0	166	0	0.5	1	0	3	0
85	70	1	0	145	174	0	1	125	1	2.6	0	0	3	0
86	57	1	0	165	289	1	0	124	0	1.0	1	3	3	0
87	61	1	0	120	260	0	1	140	1	3.6	1	1	3	0
88	57	1	0	110	201	0	1	126	1	1.5	1	0	1	1
89	60	0	0	150	258	0	0	157	0	2.6	1	2	3	0
90	63	0	0	150	407	0	0	154	0	4.0	1	3	3	0
91	55	0	0	128	205	0	2	130	1	2.0	1	1	3	0
92	64	0	0	180	325	0	1	154	1	0.0	2	0	2	1
93	54	1	0	110	239	0	1	126	1	2.8	1	1	3	0
94	52	1	0	128	204	1	1	156	1	1.0	1	0	0	0
95	51	1	0	140	299	0	1	173	1	1.6	2	0	3	0
96	62	0	2	130	263	0	1	97	0	1.2	1	1	3	0
97	59	1	3	178	270	0	0	145	0	4.2	0	0	3	1
98	52	1	1	134	201	0	1	158	0	0.8	2	1	2	1
99	42	0	0	102	265	0	0	122	0	0.6	1	0	2	1
100	59	1	0	135	234	0	1	161	0	0.5	1	0	3	1

Let’s focus on a patient’s maximum heart rate (thalach) and the cholesterol (chol) to identify some outliers.

heart.scatter(["thalach", "chol"])

We can see some outliers of the distribution: people with high cholesterol and others with a very low heart rate. Let’s compute the global outliers using the outliers() method.

heart.outliers(["thalach", "chol"], "global_outliers")
heart.scatter(["thalach", "chol"], by = "global_outliers")

It is also possible to draw an outlier plot using the outliers_plot() method.

heart.outliers_plot(["thalach", "chol"],)

We’ve detected some global outliers in the distribution and we can impute these with the fill_outliers() method.

Generally, you can identify global outliers with the Z-Score. We’ll consider a Z-Score greater than 3 indicates that the datapoint is an outlier. Some less precise techniques consider the data points belonging in the first and last alpha-quantile as outliers. You’re free to choose either of these strategies when filling outliers.

heart["thalach"].fill_outliers(
    use_threshold = True,
    threshold = 3.0,
    method = "winsorize",
)
heart["chol"].fill_outliers(
    use_threshold = True,
    threshold = 3.0,
    method = "winsorize",
)
heart.scatter(
    ["thalach", "chol"],
    by = "global_outliers",
)

Other techniques like DBSCAN or local outlier factor (LOF) can be to used to check other data points for outliers.

from vastorbit.machine_learning.vast import DBSCAN

model = DBSCAN(eps = 20, min_samples = 10)
model.fit(heart, ["thalach", "chol"])
model.plot()

heart_dbscan = model.predict()
heart_dbscan["outliers_dbscan"] = "CAST((dbscan_clusters = -1) AS integer)"
heart_dbscan.scatter(
    ["thalach", "chol"],
    by = "outliers_dbscan",
)

While DBSCAN identifies outliers when computing the clusters, LOF computes an outlier score. Generally, a LOF Score greater than 1.5 indicates an outlier.

from vastorbit.machine_learning.vast import LocalOutlierFactor

model = LocalOutlierFactor()
model.fit(heart, ["thalach", "chol",])
model.plot()

heart_lof = model.predict()
heart_lof["outliers"] = "(CASE WHEN lof_score > 1.5 THEN 1 ELSE 0 END)"
heart_lof.scatter(
    ["thalach", "chol"],
    by = "outliers",
)

We have many other techniques like the KMeans clustering for finding outliers, but the most important method is using the Z-Score. After identifying outliers, we just have to decide how to impute the missing values. We’ll focus on missing values in the next lesson.