Correlation and Dependency

Finding links between variables is a very important task. The main purpose of data science is to find relationships between variables, and to understand how these relationships can help us make better decisions.

Machine learning models are also sensitive to the number of variables and how they relate and affect each other, so finding correlations and dependencies can help us make better use of our machine learning algorithms.

Let’s use the Telco Churn dataset to understand how we can find links between different variables in vastorbit.

import vastorbit as vo

churn = vo.read_csv("customers.csv")
churn.head(100)

	Abc customerid Varchar(50) 100%	Abc gender Varchar(50) 100%	123 seniorcitizen Integer 100%	Abc partner Varchar(50) 100%	Abc dependents Varchar(50) 100%	123 tenure Integer 100%	Abc phoneservice Varchar(50) 100%	Abc multiplelines Varchar(50) 100%	Abc internetservice Varchar(50) 100%	Abc onlinesecurity Varchar(50) 100%	Abc onlinebackup Varchar(50) 100%	Abc deviceprotection Varchar(50) 100%	Abc techsupport Varchar(50) 100%	Abc streamingtv Varchar(50) 100%	Abc streamingmovies Varchar(50) 100%	Abc contract Varchar(50) 100%	Abc paperlessbilling Varchar(50) 100%	Abc paymentmethod Varchar(50) 100%	123 monthlycharges Double 100%	123 totalcharges Double 99%	Abc churn Varchar(50) 100%
1	5141-ZUVBH	Female	0	No	Yes	9	Yes	No	Fiber optic	Yes	No	Yes	Yes	No	Yes	Month-to-month	No	Bank transfer (automatic)	93.0	870.25	No
2	1089-HDMKP	Male	1	No	Yes	11	Yes	No	DSL	No	Yes	No	Yes	No	No	Month-to-month	Yes	Electronic check	54.55	601.25	No
3	7623-HKYRK	Male	0	No	Yes	6	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	No	Mailed check	19.7	111.65	No
4	0310-SUCIN	Female	0	Yes	No	71	Yes	No	DSL	Yes	Yes	Yes	Yes	Yes	Yes	One year	Yes	Bank transfer (automatic)	84.8	6046.1	No
5	5197-PYEPU	Female	0	Yes	Yes	42	Yes	Yes	Fiber optic	Yes	Yes	No	No	Yes	No	One year	Yes	Credit card (automatic)	94.45	3923.8	No
6	2929-ERCFZ	Female	0	Yes	Yes	8	Yes	No	Fiber optic	No	No	Yes	No	Yes	Yes	Month-to-month	Yes	Electronic check	94.2	777.3	Yes
7	7548-SEPYI	Female	0	No	No	5	Yes	No	Fiber optic	No	Yes	No	No	Yes	Yes	Month-to-month	Yes	Electronic check	96.25	512.45	Yes
8	8835-VSDSE	Female	0	Yes	No	2	Yes	No	Fiber optic	No	No	No	No	No	No	Month-to-month	Yes	Electronic check	70.7	141.45	Yes
9	6619-RPLQZ	Female	0	Yes	Yes	45	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	No	Mailed check	20.85	892.15	No
10	3275-RHRNE	Male	0	Yes	Yes	28	No	No phone service	DSL	No	Yes	Yes	Yes	Yes	Yes	One year	Yes	Credit card (automatic)	60.0	1682.05	No
11	3503-TYDAY	Female	0	Yes	No	43	Yes	Yes	Fiber optic	No	Yes	No	No	No	No	Month-to-month	No	Electronic check	80.45	3398.9	No
12	6901-GOGZG	Male	0	No	Yes	60	Yes	Yes	DSL	Yes	No	Yes	Yes	Yes	Yes	Two year	Yes	Electronic check	84.95	4984.85	No
13	1623-NLDOT	Female	0	Yes	No	42	No	No phone service	DSL	No	Yes	No	Yes	No	No	One year	No	Mailed check	33.55	1445.3	Yes
14	7021-XSNYE	Male	0	Yes	Yes	7	No	No phone service	DSL	Yes	No	Yes	Yes	No	Yes	Two year	No	Credit card (automatic)	49.65	305.55	No
15	9621-OUPYD	Female	0	Yes	No	25	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	Yes	Bank transfer (automatic)	20.2	507.9	No
16	9898-KZQDZ	Female	1	Yes	Yes	40	Yes	Yes	Fiber optic	No	No	No	No	Yes	Yes	Month-to-month	Yes	Credit card (automatic)	94.55	3640.45	Yes
17	8982-NHAVY	Male	0	No	No	27	Yes	Yes	Fiber optic	No	No	Yes	No	Yes	Yes	One year	Yes	Bank transfer (automatic)	100.5	2673.45	No
18	4307-KWMXE	Male	0	No	No	10	No	No phone service	DSL	No	Yes	No	Yes	No	No	Month-to-month	No	Electronic check	35.75	389.8	No
19	0141-YEAYS	Female	1	No	No	27	Yes	Yes	Fiber optic	No	Yes	Yes	No	No	No	Month-to-month	Yes	Bank transfer (automatic)	86.45	2401.05	No
20	2450-ZKEED	Female	0	No	No	11	Yes	No	DSL	No	No	Yes	Yes	No	No	One year	No	Bank transfer (automatic)	53.8	651.55	No
21	3694-DELSO	Male	0	Yes	Yes	4	No	No phone service	DSL	Yes	No	No	No	Yes	No	Month-to-month	No	Credit card (automatic)	38.55	156.1	No
22	3893-JRNFS	Male	0	No	No	68	No	No phone service	DSL	No	No	Yes	No	No	Yes	Month-to-month	Yes	Bank transfer (automatic)	39.9	2796.35	No
23	9603-OAIHC	Male	1	No	No	1	Yes	No	Fiber optic	No	No	No	No	No	No	Month-to-month	Yes	Electronic check	70.05	70.05	No
24	1133-KXCGE	Female	0	No	No	18	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	No	Mailed check	20.1	407.05	No
25	5236-PERKL	Female	0	No	No	57	Yes	Yes	Fiber optic	Yes	Yes	Yes	No	Yes	Yes	Two year	Yes	Bank transfer (automatic)	112.95	6465.0	Yes
26	0142-GVYSN	Male	0	No	No	26	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	Yes	Electronic check	20.3	511.25	No
27	1049-FYSYG	Female	0	Yes	No	17	No	No phone service	DSL	No	Yes	Yes	No	No	No	Month-to-month	Yes	Credit card (automatic)	35.65	646.05	No
28	7854-FOKSF	Male	0	No	No	1	No	No phone service	DSL	No	No	No	No	No	Yes	Month-to-month	No	Electronic check	35.9	35.9	Yes
29	2519-FAKOD	Male	0	No	Yes	38	Yes	Yes	Fiber optic	No	No	Yes	No	Yes	Yes	Month-to-month	Yes	Bank transfer (automatic)	99.25	3777.15	Yes
30	8406-LNMHF	Male	1	Yes	No	59	Yes	No	Fiber optic	Yes	Yes	Yes	No	No	No	One year	Yes	Credit card (automatic)	82.95	4903.15	No
31	1821-BUCWY	Male	0	No	No	30	Yes	No	DSL	Yes	No	Yes	No	No	No	Two year	Yes	Mailed check	55.65	1653.85	No
32	8263-JQAIK	Male	1	No	No	2	No	No phone service	DSL	No	No	No	No	No	No	Month-to-month	Yes	Mailed check	24.45	47.5	Yes
33	0023-UYUPN	Female	1	Yes	No	50	Yes	Yes	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	No	Electronic check	25.2	1306.3	No
34	8314-DPQHL	Male	0	No	No	9	No	No phone service	DSL	Yes	No	Yes	Yes	No	Yes	One year	No	Mailed check	50.8	463.6	No
35	1465-WCZVT	Female	0	Yes	Yes	3	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	No	Mailed check	19.65	60.65	No
36	9481-SFCQY	Female	0	No	Yes	14	Yes	No	DSL	No	No	Yes	No	Yes	No	Month-to-month	Yes	Credit card (automatic)	59.8	824.85	No
37	6360-SVNWV	Female	1	No	No	31	Yes	No	Fiber optic	Yes	No	No	No	No	No	Month-to-month	Yes	Credit card (automatic)	73.55	2094.65	No
38	0567-GGCAC	Female	0	No	No	7	Yes	No	DSL	Yes	Yes	Yes	No	No	No	Month-to-month	No	Electronic check	61.4	438.9	No
39	7089-IVVAZ	Female	0	No	No	8	Yes	Yes	Fiber optic	Yes	No	No	Yes	Yes	Yes	Month-to-month	Yes	Bank transfer (automatic)	103.35	847.3	Yes
40	8884-MRNSU	Male	0	Yes	Yes	17	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	No	Bank transfer (automatic)	19.9	329.75	No
41	2171-UDMFD	Male	0	Yes	Yes	32	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	Yes	Credit card (automatic)	19.45	674.55	No
42	9050-IKDZA	Female	1	No	No	2	Yes	No	Fiber optic	No	No	No	No	Yes	No	Month-to-month	No	Mailed check	81.5	162.55	No
43	2205-YMZZJ	Male	1	No	No	7	Yes	Yes	Fiber optic	No	No	No	No	Yes	No	Month-to-month	Yes	Electronic check	84.8	546.95	Yes
44	9802-CAQUT	Female	0	Yes	Yes	72	Yes	Yes	Fiber optic	Yes	Yes	Yes	No	Yes	Yes	Two year	No	Credit card (automatic)	109.55	7887.25	No
45	1254-IZEYF	Female	1	No	No	31	Yes	Yes	Fiber optic	No	Yes	No	No	Yes	Yes	Month-to-month	Yes	Electronic check	99.95	3186.65	Yes
46	0187-WZNAB	Female	0	Yes	Yes	27	Yes	No	Fiber optic	Yes	No	No	No	No	No	Month-to-month	No	Mailed check	74.4	1972.35	No
47	9492-TOKRI	Female	0	No	No	18	Yes	Yes	Fiber optic	No	Yes	No	No	Yes	No	Month-to-month	Yes	Credit card (automatic)	90.0	1527.35	Yes
48	1475-VWVDO	Male	0	No	No	7	Yes	No	Fiber optic	No	No	Yes	No	No	No	Month-to-month	Yes	Electronic check	74.9	490.55	No
49	9221-OTIVJ	Female	1	No	No	14	Yes	Yes	Fiber optic	No	Yes	Yes	No	Yes	Yes	Month-to-month	Yes	Electronic check	104.85	1531.4	Yes
50	1357-MVDOZ	Male	0	Yes	Yes	11	Yes	No	DSL	No	No	Yes	No	Yes	No	Month-to-month	Yes	Bank transfer (automatic)	59.65	683.25	No
51	7602-MVRMB	Female	0	Yes	Yes	72	Yes	Yes	Fiber optic	Yes	No	Yes	Yes	Yes	Yes	Two year	Yes	Bank transfer (automatic)	110.45	8058.85	No
52	3325-FUYCG	Male	0	Yes	Yes	28	Yes	Yes	Fiber optic	Yes	Yes	No	No	Yes	Yes	One year	No	Electronic check	106.1	2847.4	Yes
53	5908-QMGOE	Male	1	No	No	15	Yes	Yes	Fiber optic	No	No	No	No	No	No	Month-to-month	Yes	Credit card (automatic)	74.2	1133.9	Yes
54	1197-BVMVG	Female	1	No	No	4	Yes	Yes	Fiber optic	No	No	No	No	No	No	Month-to-month	Yes	Electronic check	74.45	294.45	No
55	5406-KGRMX	Female	0	No	No	71	Yes	Yes	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	No	Electronic check	24.55	1719.15	No
56	8481-YYXWG	Female	0	No	No	5	Yes	No	Fiber optic	No	Yes	No	Yes	No	Yes	Month-to-month	Yes	Electronic check	89.35	461.7	Yes
57	5968-HYJRZ	Male	0	Yes	Yes	47	Yes	Yes	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	No	Mailed check	24.55	1160.45	No
58	5198-EFNBM	Male	1	Yes	No	57	Yes	No	Fiber optic	No	Yes	Yes	No	Yes	No	One year	Yes	Electronic check	90.65	5199.8	No
59	7516-GMHUV	Male	1	Yes	No	50	Yes	Yes	Fiber optic	Yes	No	Yes	No	Yes	Yes	Two year	Yes	Credit card (automatic)	105.05	5163.3	No
60	7140-ADSMJ	Male	0	No	No	8	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	Yes	Credit card (automatic)	20.45	162.3	No
61	2230-XTUWL	Female	0	Yes	Yes	48	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	No	Mailed check	19.55	883.35	No
62	7706-YLMQA	Female	0	No	No	70	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	No	Bank transfer (automatic)	19.7	1341.5	No
63	2585-KTFRE	Male	0	No	Yes	1	Yes	No	DSL	Yes	Yes	No	Yes	No	Yes	Month-to-month	Yes	Bank transfer (automatic)	70.45	70.45	No
64	7994-UYIVZ	Male	0	Yes	No	8	Yes	Yes	Fiber optic	No	No	No	No	No	Yes	Month-to-month	Yes	Bank transfer (automatic)	85.65	659.45	No
65	2609-IAICY	Female	0	No	No	1	Yes	Yes	Fiber optic	No	No	No	No	No	No	Month-to-month	Yes	Electronic check	77.15	77.15	Yes
66	1740-CSDJP	Male	0	No	No	1	No	No phone service	DSL	No	No	No	No	No	Yes	Month-to-month	Yes	Bank transfer (automatic)	35.25	35.25	Yes
67	7717-BICXI	Male	0	Yes	Yes	60	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	No	Mailed check	20.55	1205.05	No
68	6559-RAKOZ	Male	0	Yes	Yes	49	Yes	Yes	Fiber optic	No	No	No	Yes	Yes	Yes	One year	Yes	Electronic check	97.95	4917.9	No
69	2636-OHFMN	Male	0	Yes	No	4	No	No phone service	DSL	No	Yes	No	No	Yes	Yes	Month-to-month	Yes	Electronic check	48.55	201.0	Yes
70	4716-MRVEN	Female	0	No	No	29	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	No	Mailed check	20.0	599.3	No
71	2143-LJULT	Female	0	Yes	No	67	Yes	Yes	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	Yes	Credit card (automatic)	25.25	1733.15	No
72	1323-OOEPC	Female	0	Yes	No	53	Yes	Yes	Fiber optic	No	Yes	No	No	Yes	Yes	Month-to-month	Yes	Credit card (automatic)	98.4	5149.5	Yes
73	3200-MNQTF	Male	0	Yes	No	67	Yes	Yes	DSL	Yes	Yes	Yes	Yes	No	No	Two year	No	Credit card (automatic)	70.9	4677.1	No
74	6164-HXUGH	Female	0	Yes	Yes	6	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	No	Mailed check	19.85	119.3	No
75	5630-IXDXV	Female	0	No	No	47	Yes	Yes	Fiber optic	No	Yes	Yes	No	Yes	Yes	Month-to-month	Yes	Electronic check	106.35	4849.1	No
76	0320-DWVTU	Female	0	No	No	53	Yes	No	Fiber optic	Yes	Yes	Yes	Yes	No	Yes	Two year	No	Mailed check	99.5	5424.25	No
77	9135-MGVPY	Male	0	Yes	No	69	Yes	No	DSL	Yes	Yes	Yes	Yes	Yes	Yes	Two year	Yes	Mailed check	84.7	5878.9	No
78	1212-GLHMD	Male	0	No	No	3	Yes	No	Fiber optic	Yes	No	No	No	No	Yes	Month-to-month	Yes	Mailed check	86.05	244.85	No
79	7878-JGDKK	Male	0	No	No	4	Yes	No	DSL	No	No	No	No	No	No	Month-to-month	No	Mailed check	44.55	220.75	No
80	1088-AUUZZ	Male	0	Yes	Yes	56	Yes	Yes	DSL	Yes	Yes	No	Yes	No	Yes	Two year	Yes	Credit card (automatic)	75.85	4261.2	No
81	0397-GZBBC	Male	1	Yes	No	59	Yes	Yes	Fiber optic	No	No	No	No	Yes	Yes	Month-to-month	Yes	Bank transfer (automatic)	93.85	5574.75	Yes
82	6614-YWYSC	Male	1	Yes	No	61	Yes	Yes	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	No	Bank transfer (automatic)	25.0	1501.75	No
83	9588-OZDMQ	Female	0	Yes	No	2	Yes	No	DSL	No	No	No	No	No	No	Month-to-month	Yes	Electronic check	45.0	89.75	No
84	7641-EUYET	Male	1	Yes	Yes	46	Yes	Yes	Fiber optic	Yes	Yes	Yes	No	Yes	No	Month-to-month	Yes	Electronic check	100.7	4541.2	Yes
85	6476-EPYZR	Male	0	Yes	Yes	12	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	No	Mailed check	20.5	255.5	No
86	9921-ZVRHG	Female	0	No	No	14	Yes	Yes	Fiber optic	No	No	Yes	No	No	No	Month-to-month	Yes	Electronic check	80.45	1072.0	Yes
87	1174-FGIFN	Female	0	Yes	Yes	28	Yes	Yes	Fiber optic	No	No	No	Yes	No	Yes	One year	Yes	Electronic check	90.45	2509.25	No
88	6620-HVDUJ	Male	0	No	No	24	Yes	No	DSL	Yes	Yes	No	Yes	No	No	Month-to-month	No	Bank transfer (automatic)	60.45	1440.75	No
89	4701-MLJPN	Male	0	No	No	31	Yes	No	DSL	No	No	Yes	Yes	No	No	Month-to-month	Yes	Electronic check	55.25	1715.65	Yes
90	1032-MAELW	Female	0	Yes	Yes	68	Yes	Yes	DSL	Yes	Yes	Yes	Yes	No	Yes	One year	Yes	Electronic check	78.45	5333.35	No
91	7641-TQFHN	Male	0	No	Yes	39	Yes	Yes	Fiber optic	No	No	No	Yes	Yes	Yes	Two year	No	Mailed check	100.55	3895.35	No
92	1552-TKMXS	Female	0	Yes	No	42	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	No	Credit card (automatic)	20.35	869.9	No
93	9206-GVPEQ	Male	0	Yes	No	13	No	No phone service	DSL	Yes	No	No	Yes	Yes	Yes	Month-to-month	Yes	Electronic check	54.45	706.85	Yes
94	8622-ZLFKO	Female	0	Yes	No	6	Yes	Yes	Fiber optic	No	Yes	No	No	No	Yes	Month-to-month	Yes	Electronic check	90.75	512.25	No
95	1596-BBVTG	Male	0	No	No	35	Yes	Yes	Fiber optic	No	No	No	No	No	No	Month-to-month	No	Credit card (automatic)	75.35	2636.05	Yes
96	6188-UXBBR	Female	0	Yes	No	38	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	Yes	Credit card (automatic)	20.25	814.75	No
97	2333-KWEWW	Male	0	No	No	18	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Two year	No	Credit card (automatic)	20.05	388.6	No
98	5702-SKUOB	Female	0	Yes	No	4	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	No	Mailed check	19.6	93.45	No
99	1134-YWTYF	Male	0	Yes	No	27	Yes	No	DSL	No	Yes	No	Yes	No	No	Month-to-month	Yes	Electronic check	53.8	1389.85	No
100	6061-GWWAV	Male	0	No	Yes	41	Yes	No	DSL	Yes	Yes	Yes	No	Yes	No	One year	No	Mailed check	70.2	2894.55	No

The Pearson correlation coefficient is a very common correlation function. In this case, it helped us to find linear links between the variables. Having a strong Pearson relationship means that the two input variables are linearly correlated.

churn.corr(method = "pearson")

We can see that tenure is well-correlated to the TotalCharges, which makes sense.

churn.scatter(["tenure", "TotalCharges"])

churn.corr(["tenure", "TotalCharges"], method = "pearson")

Note, however, that having a low Pearson relationship imply that the variables aren’t correlated. For example, let’s compute the Pearson correlation coefficient between tenure and TotalCharges to the power of 20.

churn["TotalCharges^20"] = churn["TotalCharges"] ** 20
churn.corr(["tenure", "TotalCharges^20"], method = "pearson")

We know that the tenure and TotalCharges are strongly linearly correlated. However we can notice that the correlation between the tenure and TotalCharges to the power of 20 is not very high. Indeed, the Pearson correlation coefficient is not robust for monotonic relationships, but rank-based correlations are. Knowing this, we’ll calculate the Spearman’s rank correlation coefficient instead.

churn.corr(method = "spearman", show = False)

	"seniorcitizen"	"tenure"	"monthlycharges"	"totalcharges"	"TotalCharges^20"
"seniorcitizen"	1.0	0.01907679	0.22109252	0.10579534	0.10579534
"tenure"	0.01907679	1.0	0.27634224	0.8831034	0.8831034
"monthlycharges"	0.22109252	0.27634224	1.0	0.6339584	0.6339584
"totalcharges"	0.10579534	0.8831034	0.6339584	1.0	1.0
"TotalCharges^20"	0.10579534	0.8831034	0.6339584	1.0	1.0

churn.corr(method = "spearman")

The Spearman’s rank correlation coefficient determines the monotonic relationships between the variables.

churn.corr(["tenure", "TotalCharges^20"], method = "spearman")

We can notice that Spearman’s rank correlation coefficient stays the same if one of the variables can be expressed using a monotonic function on the other. The same applies to Kendall rank correlation coefficient.

churn.corr(method = "kendall")

Notice that the Kendall rank correlation coefficient will also detect the monotonic relationship.

churn.corr(["tenure", "TotalCharges^20"], method = "kendall")

However, the Kendall rank correlation coefficient is very computationally expensive, so we’ll generally use Pearson and Spearman when dealing with correlations between numerical variables.

Binary features are considered numerical, but this isn’t technically accurate. Since binary variables can only take two values, calculating correlations between a binary and numerical variable can lead to misleading results. To account for this, we’ll want to use the Biserial Point method to calculate the Point-Biserial correlation coefficient. This powerful method will help us understand the link between a binary variable and a numerical variable.

churn.corr(method = "biserial")

Lastly, we’ll look at the relationship between categorical columns. In this case, the Cramer's V method is very efficient. Since there is no position in the Euclidean space for those variables, the Cramer's V coefficients cannot be negative (which is a sign of an opposite relationship) and they will range in the interval [0,1].

churn.corr(method = "cramer")

Sometimes, we just need to look at the correlation between a response and other variables. The parameter focus will isolate and show us the specified correlation vector.

churn.corr(method = "cramer", focus = "Churn")

Sometimes a correlation coefficient can lead to incorrect assumptions, so we should always look at the coefficient p-value.

churn.corr_pvalue("Churn", "customerID", method = "cramer")

We can see that churning correlates to the type of contract (monthly, yearly, etc.) which makes sense: you would expect that different types of contracts differ in flexibility for the customer, and particularly restrictive contracts may make churning more likely.

The type of internet service also seems to correlate with churning. Let’s split the different categories to binaries to understand which services can influence the global churning rate.

churn["InternetService"].one_hot_encode()
churn.corr(
    method = "spearman",
    focus = "Churn",
    columns = [
        "InternetService_DSL",
        "InternetService_Fiber_optic",
    ],
)

We can see that the Fiber Optic option in particular seems to be directly linked to a customer’s likelihood to churn. Let’s compute some aggregations to find a causal relationship.

churn["contract"].one_hot_encode()
churn.groupby(
    [
        "InternetService_Fiber_optic",
    ],
    [
        "AVG(tenure) AS tenure",
        "AVG(totalcharges) AS totalcharges",
        'AVG("contract_month-to-month") AS "contract_month-to-month"',
        'AVG("monthlycharges") AS "monthlycharges"',
    ],
)

	123 internetservice_Fiber_optic Integer 100%	123 tenure Double 100%	123 totalcharges Double 100%	123 contract_month-to-month Double 100%	123 monthlycharges Double 100%
1	1	32.91795865633075	3205.3045704134365	0.6873385012919897	91.50012919896639
2	0	31.942234608563467	1558.0654852642276	0.44261464403344314	43.788244236128705

It seems that users with the Fiber Optic option tend more to churn not because of the option itself, but probably because of the type of contracts and the monthly charges the users are paying to get it. Be careful when dealing with identifying correlations! Remember: correlation doesn’t imply causation!

Another important type of correlation is the autocorrelation. Let’s use the amazon dataset to understand it.

from vastorbit.datasets import load_amazon

amazon = load_amazon()
amazon.head(100)

	📅 date Date 100%	Abc state Varchar(32) 100%	123 number Integer 100%
1	2013-09-01	MATO GROSSO DO SUL	5576
2	2013-10-01	MATO GROSSO DO SUL	1986
3	2013-11-01	MATO GROSSO DO SUL	1064
4	2013-12-01	MATO GROSSO DO SUL	447
5	2013-01-01	MINAS GERAIS	75
6	2013-02-01	MINAS GERAIS	143
7	2013-03-01	MINAS GERAIS	110
8	2013-04-01	MINAS GERAIS	61
9	2013-05-01	MINAS GERAIS	150
10	2013-06-01	MINAS GERAIS	144
11	2013-07-01	MINAS GERAIS	462
12	2013-08-01	MINAS GERAIS	911
13	2013-09-01	MINAS GERAIS	1656
14	2013-10-01	MINAS GERAIS	1221
15	2013-11-01	MINAS GERAIS	277
16	2013-12-01	MINAS GERAIS	98
17	2013-01-01	PARANÁ	28
18	2013-02-01	PARANÁ	9
19	2013-03-01	PARANÁ	18
20	2013-04-01	PARANÁ	72
21	2013-05-01	PARANÁ	85
22	2013-06-01	PARANÁ	52
23	2013-07-01	PARANÁ	168
24	2013-08-01	PARANÁ	481
25	2013-09-01	PARANÁ	476
26	2013-10-01	PARANÁ	233
27	2013-11-01	PARANÁ	126
28	2013-12-01	PARANÁ	155
29	2013-01-01	PARAÍBA	38
30	2013-02-01	PARAÍBA	9
31	2013-03-01	PARAÍBA	3
32	2013-04-01	PARAÍBA	6
33	2013-05-01	PARAÍBA	3
34	2013-06-01	PARAÍBA	3
35	2013-07-01	PARAÍBA	1
36	2013-08-01	PARAÍBA	3
37	2013-09-01	PARAÍBA	25
38	2013-10-01	PARAÍBA	58
39	2013-11-01	PARAÍBA	95
40	2013-12-01	PARAÍBA	78
41	2013-01-01	PARÁ	167
42	2013-02-01	PARÁ	30
43	2013-03-01	PARÁ	19
44	2013-04-01	PARÁ	20
45	2013-05-01	PARÁ	42
46	2013-06-01	PARÁ	173
47	2013-07-01	PARÁ	447
48	2013-08-01	PARÁ	2161
49	2013-09-01	PARÁ	3360
50	2013-10-01	PARÁ	4921
51	2013-11-01	PARÁ	3688
52	2013-12-01	PARÁ	5514
53	2013-01-01	PERNAMBUCO	55
54	2013-02-01	PERNAMBUCO	29
55	2013-03-01	PERNAMBUCO	40
56	2013-04-01	PERNAMBUCO	11
57	2013-05-01	PERNAMBUCO	9
58	2013-06-01	PERNAMBUCO	12
59	2013-07-01	PERNAMBUCO	4
60	2013-08-01	PERNAMBUCO	14
61	2013-09-01	PERNAMBUCO	143
62	2013-10-01	PERNAMBUCO	161
63	2013-11-01	PERNAMBUCO	110
64	2013-12-01	PERNAMBUCO	141
65	2013-01-01	PIAUÍ	72
66	2013-02-01	PIAUÍ	68
67	2013-03-01	PIAUÍ	14
68	2013-04-01	PIAUÍ	34
69	2013-05-01	PIAUÍ	40
70	2013-06-01	PIAUÍ	263
71	2013-07-01	PIAUÍ	361
72	2013-08-01	PIAUÍ	1015
73	2013-09-01	PIAUÍ	2311
74	2013-10-01	PIAUÍ	1269
75	2013-11-01	PIAUÍ	640
76	2013-12-01	PIAUÍ	468
77	2013-01-01	RIO DE JANEIRO	9
78	2013-02-01	RIO DE JANEIRO	12
79	2013-03-01	RIO DE JANEIRO	9
80	2013-04-01	RIO DE JANEIRO	9
81	2013-05-01	RIO DE JANEIRO	16
82	2013-06-01	RIO DE JANEIRO	17
83	2013-07-01	RIO DE JANEIRO	58
84	2013-08-01	RIO DE JANEIRO	74
85	2013-09-01	RIO DE JANEIRO	117
86	2013-10-01	RIO DE JANEIRO	44
87	2013-11-01	RIO DE JANEIRO	28
88	2013-12-01	RIO DE JANEIRO	12
89	2013-01-01	RIO GRANDE DO NORTE	25
90	2013-02-01	RIO GRANDE DO NORTE	24
91	2013-03-01	RIO GRANDE DO NORTE	6
92	2013-04-01	RIO GRANDE DO NORTE	7
93	2013-05-01	RIO GRANDE DO NORTE	2
94	2013-06-01	RIO GRANDE DO NORTE	1
95	2013-07-01	RIO GRANDE DO NORTE	0
96	2013-08-01	RIO GRANDE DO NORTE	2
97	2013-09-01	RIO GRANDE DO NORTE	7
98	2013-10-01	RIO GRANDE DO NORTE	71
99	2013-11-01	RIO GRANDE DO NORTE	77
100	2013-12-01	RIO GRANDE DO NORTE	46

Our goal is to predict the number of forest fires in Brazil. To do this, we can draw an autocorrelation plot and a partial autocorrelation plot.

amazon.acf(
    column = "number",
    ts = "date",
    by = ["state"],
    p = 24,
    method = "pearson",
)

amazon.pacf(
    column = "number",
    ts = "date",
    by = ["state"],
    p = 8,
)

We can see the seasonality forest fires.

It’s mathematically impossible to build the perfect correlation function, but we still have several powerful functions at our disposal for finding relationships in all kinds of datasets.