Telco Churn

This example uses the Telco Churn dataset to predict which Telco user is likely to churn; that is, customers that will likely stop using Telco.

• Churn: customers that left within the last month.

• Services: services of each customer (phone, multiple lines, internet, online security, online backup, device protection, tech support, and streaming TV and movies).

• Customer account information: how long they’ve been a customer, contract, payment method, paperless billing, monthly charges, and total charges.

• Customer demographics: gender, age range, and if they have partners and dependents.

We will follow the data science cycle (Data Exploration - Data Preparation - Data Modeling - Model Evaluation - Model Deployment) to solve this problem.

Initialization

This example uses the following version of vastorbit:

import vastorbit as vo

vo.__version__

Connect to VAST. This example uses an existing connection called VASTDSN . For details on how to create a connection, see the Connection tutorial. You can skip the below cell if you already have an established connection.

vo.connect("VASTDSN")

Let’s create a VastFrame of the dataset. The dataset is available here.

churn = vo.read_csv("customers.csv")

Let’s take a look at the first few entries in the dataset.

churn.head(10)

	Abc customerid Varchar(50) 100%	Abc gender Varchar(50) 100%	123 seniorcitizen Integer 100%	Abc partner Varchar(50) 100%	Abc dependents Varchar(50) 100%	123 tenure Integer 100%	Abc phoneservice Varchar(50) 100%	Abc multiplelines Varchar(50) 100%	Abc internetservice Varchar(50) 100%	Abc onlinesecurity Varchar(50) 100%	Abc onlinebackup Varchar(50) 100%	Abc deviceprotection Varchar(50) 100%	Abc techsupport Varchar(50) 100%	Abc streamingtv Varchar(50) 100%	Abc streamingmovies Varchar(50) 100%	Abc contract Varchar(50) 100%	Abc paperlessbilling Varchar(50) 100%	Abc paymentmethod Varchar(50) 100%	123 monthlycharges Double 100%	123 totalcharges Double 99%	Abc churn Varchar(50) 100%
1	0895-DQHEW	Male	0	Yes	No	54	Yes	Yes	Fiber optic	No	Yes	Yes	No	Yes	Yes	Month-to-month	Yes	Electronic check	104.3	5278.15	Yes
2	4717-GHADL	Female	0	No	No	50	Yes	Yes	DSL	No	Yes	No	Yes	Yes	Yes	Two year	Yes	Mailed check	79.6	4024.2	No
3	5501-TVMGM	Male	0	No	No	1	Yes	No	DSL	No	No	No	No	No	Yes	Month-to-month	Yes	Electronic check	55.25	55.25	No
4	5879-HMFFH	Female	0	Yes	No	72	Yes	Yes	DSL	Yes	Yes	Yes	Yes	Yes	Yes	Two year	No	Bank transfer (automatic)	88.05	6520.8	No
5	6772-WFQRD	Male	0	No	Yes	40	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	One year	Yes	Bank transfer (automatic)	20.4	854.9	No
6	3810-DVDQQ	Female	0	Yes	Yes	72	Yes	Yes	Fiber optic	Yes	Yes	Yes	Yes	Yes	Yes	Two year	Yes	Bank transfer (automatic)	117.6	8308.9	No
7	6972-SNKKW	Female	0	No	No	6	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	No	Mailed check	20.0	109.2	No
8	3694-GLTJM	Female	0	No	No	5	Yes	No	No	No internet service	No internet service	No internet service	No internet service	No internet service	No internet service	Month-to-month	Yes	Credit card (automatic)	19.65	92.05	No
9	8550-XSXUQ	Male	0	Yes	No	48	Yes	Yes	DSL	No	Yes	Yes	No	No	Yes	One year	Yes	Credit card (automatic)	70.55	3420.5	No
10	8149-RSOUN	Female	0	No	No	1	Yes	Yes	Fiber optic	No	No	No	No	Yes	Yes	Month-to-month	Yes	Electronic check	93.85	93.85	Yes

Data Exploration and Preparation

Let’s examine our data.

churn.describe(method = "categorical", unique = True)

	dtype	count	top	top_percent	unique
"customerid"	varchar(50)	7043	0440-QEXBZ	0.014	7043.0
"gender"	varchar(50)	7043	Male	50.476	2.0
"seniorcitizen"	integer	7043	0	83.785	2.0
"partner"	varchar(50)	7043	No	51.697	2.0
"dependents"	varchar(50)	7043	No	70.041	2.0
"tenure"	integer	7043	1	8.704	73.0
"phoneservice"	varchar(50)	7043	Yes	90.317	2.0
"multiplelines"	varchar(50)	7043	No	48.133	3.0
"internetservice"	varchar(50)	7043	Fiber optic	43.959	3.0
"onlinesecurity"	varchar(50)	7043	No	49.666	3.0
"onlinebackup"	varchar(50)	7043	No	43.845	3.0
"deviceprotection"	varchar(50)	7043	No	43.944	3.0
"techsupport"	varchar(50)	7043	No	49.311	3.0
"streamingtv"	varchar(50)	7043	No	39.898	3.0
"streamingmovies"	varchar(50)	7043	No	39.543	3.0
"contract"	varchar(50)	7043	Month-to-month	55.019	3.0
"paperlessbilling"	varchar(50)	7043	Yes	59.222	2.0
"paymentmethod"	varchar(50)	7043	Electronic check	33.579	4.0
"monthlycharges"	double	7043	20.05	0.866	1585.0
"totalcharges"	double	7032	[null]	0.156	6530.0
"churn"	varchar(50)	7043	No	73.463	2.0

Several variables are categorical, and since they all have low cardinalities, we can compute their dummies. We can also convert all booleans to numeric and fill the missing values.

for column in [
    "DeviceProtection",
    "MultipleLines",
    "PaperlessBilling",
    "Churn",
    "TechSupport",
    "Partner",
    "StreamingTV",
    "OnlineBackup",
    "Dependents",
    "OnlineSecurity",
    "PhoneService",
    "StreamingMovies",
]:
    churn[column].decode("Yes", 1, 0)
churn.one_hot_encode().drop(
    [
        "customerID",
        "gender",
        "Contract",
        "PaymentMethod",
        "InternetService",
    ],
)
churn.fillna()

	123 seniorcitizen Integer 100%	123 partner Integer 100%	123 dependents Integer 100%	123 tenure Integer 100%	123 phoneservice Integer 100%	123 multiplelines Integer 100%	123 onlinesecurity Integer 100%	123 onlinebackup Integer 100%	123 deviceprotection Integer 100%	123 techsupport Integer 100%	123 streamingtv Integer 100%	123 streamingmovies Integer 100%	123 paperlessbilling Integer 100%	123 monthlycharges Double 100%	123 totalcharges Real 100%	123 churn Integer 100%	123 gender_Female Bool 100%	123 internetservice_DSL Bool 100%	123 internetservice_Fiber_optic Bool 100%	123 contract_Month-to-month Bool 100%	123 contract_One_year Bool 100%	123 paymentmethod_Bank_transfer_(automatic) Bool 100%	123 paymentmethod_Credit_card_(automatic) Bool 100%	123 paymentmethod_Electronic_check Bool 100%
1	0	1	0	54	1	1	0	1	1	0	1	1	1	104.3	5278.15	1	0	0	1	1	0	0	0	1
2	0	0	0	50	1	1	0	1	0	1	1	1	1	79.6	4024.2	0	1	1	0	0	0	0	0	0
3	0	0	0	1	1	0	0	0	0	0	0	1	1	55.25	55.25	0	0	1	0	1	0	0	0	1
4	0	1	0	72	1	1	1	1	1	1	1	1	0	88.05	6520.8	0	1	1	0	0	0	1	0	0
5	0	0	1	40	1	0	0	0	0	0	0	0	1	20.4	854.9	0	0	0	0	0	1	1	0	0
6	0	1	1	72	1	1	1	1	1	1	1	1	1	117.6	8308.9	0	1	0	1	0	0	1	0	0
7	0	0	0	6	1	0	0	0	0	0	0	0	0	20.0	109.2	0	1	0	0	1	0	0	0	0
8	0	0	0	5	1	0	0	0	0	0	0	0	1	19.65	92.05	0	1	0	0	1	0	0	1	0
9	0	1	0	48	1	1	0	1	1	0	0	1	1	70.55	3420.5	0	0	1	0	0	1	0	1	0
10	0	0	0	1	1	1	0	0	0	0	1	1	1	93.85	93.85	1	1	0	1	1	0	0	0	1
11	0	1	1	64	1	0	1	0	0	1	1	0	0	65.8	4068.0	0	1	1	0	0	1	0	0	0
12	0	0	0	17	1	0	0	0	0	0	0	0	0	20.05	337.9	0	1	0	0	0	1	0	1	0
13	0	1	0	40	1	0	0	1	1	0	0	0	1	80.0	3168.75	0	0	0	1	1	0	0	0	1
14	0	1	1	41	0	0	1	0	1	0	0	0	0	35.4	1412.4	0	1	1	0	0	1	0	1	0
15	1	0	0	51	1	1	0	1	0	0	0	0	1	79.6	3974.7	0	0	0	1	0	1	1	0	0
16	0	1	0	41	1	1	0	1	0	0	0	0	1	80.25	3439.0	0	0	0	1	1	0	0	0	1
17	0	1	1	1	1	0	0	0	1	0	0	0	0	50.45	50.45	1	1	1	0	1	0	0	0	0
18	0	0	0	2	1	0	0	0	0	0	0	0	1	20.45	42.45	0	1	0	0	1	0	0	0	0
19	0	0	0	68	1	0	0	1	1	0	0	0	0	79.6	5461.45	0	0	0	1	0	1	0	1	0
20	0	1	1	24	1	1	0	0	0	0	0	0	0	24.7	571.75	0	1	0	0	0	0	0	0	0

Let’s compute the correlations between the different variables and the response column.

churn.corr(focus = "Churn")

Many features have a strong correlation with the Churn variable. For example, the customers that have a Month to Month contract are more likely to churn. Having this type of contract gives customers a lot of flexibility and allows them to leave at any time. New customers are also likely to churn.

# No lock-in = Churn
churn.barh(["Contract_Month-to-month", "tenure"], method = "avg", of = "Churn", height = 500)

The following scatter plot shows that providing better tariff plans can prevent churning. Indeed, customers having high total charges are more likely to churn even if they’ve been with the company for a long time.

churn.scatter(["TotalCharges", "tenure"], by = "Churn")

Let’s move on to machine learning.

---

Machine Learning

LogisticRegression is a very powerful algorithm and we can use it to detect churns. Let’s split our VastFrame into training and testing set to evaluate our model.

train, test = churn.train_test_split(
    test_size = 0.2,
    random_state = 0,
)

Let’s train and evaluate our model.

from vastorbit.machine_learning.vast import LogisticRegression

model = LogisticRegression(
    max_iter = 3000,
)
model.fit(
    train,
    churn.get_columns(exclude_columns = ["churn"]),
    "churn",
    test,
)
model.classification_report()

	value
auc	0.840838494406836
prc_auc	0.46069025204961966
accuracy	0.8162422573984859
log_loss	0.4404031247622047
precision	0.6948640483383686
recall	0.5808080808080808
f1_score	0.6327372764786795
mcc	0.5151626244476125
informedness	0.4852546276387337
markedness	0.5469139592118089
csi	0.46277665995975853

The model is excellent! Let’s run some machine learning on the entire dataset and compute the importance of each feature.

model.drop()
model.fit(
    churn,
    churn.get_columns(exclude_columns = ["churn"]),
    "churn",
)
model.features_importance()

Based on our model, most churning customers are at least one of the following:

• Paying higher bills

• New Telco customers

• Have a monthly contract

Notice that customers have a Fiber Optic option are also likely to churn. Let’s check if this relationship is causal by computing some aggregations.

import vastorbit.sql.functions as fun

# Is Fiber optic a Bad Option? - vastorbit
churn.groupby(
    ["InternetService_Fiber_optic"],
    [
        fun.avg(churn["tenure"])._as("tenure"),
        fun.avg(churn["totalcharges"])._as("totalcharges"),
        fun.avg(churn["contract_month-to-month"])._as("contract_month_to_month"),
        fun.avg(churn["monthlycharges"])._as("monthlycharges"),
    ]
)

	123 internetservice_Fiber_optic Integer 100%	123 tenure Double 100%	123 totalcharges Double 100%	123 contract_month_to_month Double 100%	123 monthlycharges Double 100%
1	1	32.91795865633075	3205.304570413437	0.6873385012919897	91.50012919896639
2	0	31.942234608563467	1560.0866619835977	0.44261464403344314	43.78824423612872

It seems like the Fiber Optic option in and of itself doesn’t lead to churning, but customers that have this option tend to churn because their contract puts them into one of the three categories we listed before: they’re paying more.

To retain these customers, we’ll need to make some changes to what types of contracts we offer.

We’ll use a lift chart to help us identify which of our customers are likely to churn.

model.lift_chart()

By targeting less than 10% of the entire distribution, our predictions will be more than two times more accurate than the other 90%.

Conclusion

We’ve solved our problem in a pandas-like way, all without ever loading data into memory!