Spam

This example uses the spam dataset to detect SMS spam.

• v1: the SMS type (spam or ham).

• v2: SMS content.

We will follow the data science cycle (Data Exploration - Data Preparation - Data Modeling - Model Evaluation - Model Deployment) to solve this problem.

Initialization

This example uses the following version of vastorbit:

import vastorbit as vo

vo.__version__

Connect to VAST. This example uses an existing connection called VASTDSN . For details on how to create a connection, see the Connection tutorial. You can skip the below cell if you already have an established connection.

vo.connect("VASTDSN")

Let’s create a VastFrame of the dataset. The dataset is available here.

spam = vo.read_csv("spam.csv")

Let’s take a look at the first few entries in the dataset.

spam

	Abc type Varchar(50) 100%	Abc content Varchar(1010) 100%
1	ham	K.then any other special?
2	ham	Carlos is taking his sweet time as usual so let me know when you and patty are done/want to smoke and I'll tell him to haul ass
3	ham	Ok pa. Nothing problem:-)
4	ham	Have you heard about that job? I'm going to that wildlife talk again tonight if u want2come. Its that2worzels and a wizzle or whatever it is?!
5	ham	God picked up a flower and dippeditinaDEW, lovingly touched itwhichturnedinto u, and the he gifted tomeandsaid,THIS FRIEND IS 4U
6	ham	When you came to hostel.
7	ham	Ok no prob... I'll come after lunch then...
8	ham	Jus telling u dat i'll b leaving 4 shanghai on 21st instead so we'll haf more time 2 meet up cya...
9	ham	Are your freezing ? Are you home yet ? Will you remember to kiss your mom in the morning? Do you love me ? Do you think of me ? Are you missing me yet ?
10	ham	You all ready for * big day tomorrow?
11	ham	I'll probably be around mu a lot
12	ham	645
13	spam	RT-KIng Pro Video Club>> Need help? info@ringtoneking.co.uk or call 08701237397 You must be 16+ Club credits redeemable at www.ringtoneking.co.uk! Enjoy!
14	ham	Thnx dude. u guys out 2nite?
15	ham	Me sef dey laugh you. Meanwhile how's my darling anjie!
16	ham	Mm i had my food da from out
17	ham	K, makes sense, btw carlos is being difficult so you guys are gonna smoke while I go pick up the second batch and get gas
18	ham	Did u download the fring app?
19	ham	The 2 oz guy is being kinda flaky but one friend is interested in picking up $ <#> worth tonight if possible
20	ham	Friends that u can stay on fb chat with

Data Exploration and Preparation

Our dataset relies on text analysis. First, we should create some features. For example, we can use the SMS length and label encoding on the ‘type’ to get a dummy (1 if the message is a SPAM, 0 otherwise). We should also convert the message content to lowercase to simplify our analysis.

import vastorbit.sql.functions as fun

spam["length"] = fun.length(spam["content"])
spam["text_index"] = fun.row_number()._over(by = "content") # index for tfidf
spam["content"].apply("LOWER({})")
spam["type"].decode('spam', 1, 0)

	123 type Integer 100%	Abc content Varchar(1010) 100%	123 length Bigint 100%	123 text_index Bigint 100%
1	1	i don't know u and u don't know me. send chat to 86688 now and let's find each other! only 150p/msg rcvd. hg/suite342/2lands/row/w1j6hl ldn. 18 years or over.	158	1
2	0	lol please do. actually send a pic of yourself right now. i wanna see. pose with a comb and hair dryer or something.	116	1
3	1	valentines day special! win over å£1000 in our quiz and take your partner on the trip of a lifetime! send go to 83600 now. 150p/msg rcvd. custcare:08718720201	158	1
4	0	back in brum! thanks for putting us up and keeping us all and happy. see you soon	82	1
5	0	rose needs water, season needs change, poet needs imagination..my phone needs ur sms and i need ur lovely frndship forever....	126	1
6	0	* am on my way	14	1
7	0	aiyah e rain like quite big leh. if drizzling i can at least run home.	70	1
8	0	the monthly amount is not that terrible and you will not pay anything till 6months after finishing school.	106	1
9	0	package all your programs well	30	1
10	0	oh fine, i'll be by tonight	27	1
11	0	i'll reach in ard 20 mins ok...	31	1
12	0	i also thk too fast... xy suggest one not me. u dun wan it's ok. going 2 rain leh where got gd.	95	1
13	0	i dont want to hear anything	28	1
14	1	knock knock txt whose there to 80082 to enter r weekly draw 4 a å£250 gift voucher 4 a store of yr choice. t&cs www.tkls.com age16 to stoptxtstopå£1.50/week	156	1
15	0	hey pple...$700 or $900 for 5 nights...excellent location wif breakfast hamper!!!	81	1
16	0	feel like trying kadeem again? :v	33	1
17	0	that's cool, i'll come by like <#> ish	45	1
18	0	i am 6 ft. we will be a good combination!	41	1
19	0	i'm okay. chasing the dream. what's good. what are you doing next.	66	1
20	1	urgent! we are trying to contact u. todays draw shows that you have won a å£800 prize guaranteed. call 09050003091 from land line. claim c52. valid 12hrs only	158	1

Let’s compute some statistics using the length of the message.

spam['type'].describe(
    method = 'cat_stats',
    numcol = 'length',
)

	count	percent	mean	std	min	approx_10%	approx_25%	approx_50%	approx_75%	approx_90%	max
1	747	13.406317300789663	138.8661311914324	29.183082366438917	13	99	132	149	157	161	224
0	4825	86.59368269921033	71.02362694300518	58.02227532043164	2	24	33	52	92	144	910

Note

Spam tends to be longer than a normal message. First, let’s create a view with just spam. Then, we’ll use the TfidfVectorizer to create a dictionary and identify keywords.

spams = spam.search(spam["type"] == 1)

from vastorbit.machine_learning.vast import TfidfVectorizer

dict_spams = TfidfVectorizer()
dict_spams.fit(spams, index = "text_index", x = "content")
dict_spams = dict_spams.transform(spams, index = "text_index", x = "content")
dict_spams

	123 row_id Bigint 100%	Abc word Varchar(511565) 100%	123 tf Double 100%	123 idf Double 100%	123 tfidf Double 100%
1	3	find	1.0	1.2231435513142097	0.06915497637343532
2	3	rcvd	1.0	1.2231435513142097	0.06915497637343532
3	3	5000	1.0	1.0	0.05653872458317062
4	3	16	1.0	1.2231435513142097	0.06915497637343532
5	3	bonus	1.0	1.2231435513142097	0.06915497637343532
6	3	86688	2.0	1.2231435513142097	0.13830995274687063
7	3	lets	1.0	1.2231435513142097	0.06915497637343532
8	3	only	1.0	1.2231435513142097	0.06915497637343532
9	3	no1	1.0	1.2231435513142097	0.06915497637343532
10	3	tone	1.0	1.2231435513142097	0.06915497637343532
11	3	36504	1.0	1.2231435513142097	0.06915497637343532
12	3	75000	1.0	1.2231435513142097	0.06915497637343532
13	3	text	1.0	1.2231435513142097	0.06915497637343532
14	3	now	2.0	1.2231435513142097	0.13830995274687063
15	3	dont	2.0	1.2231435513142097	0.13830995274687063
16	3	freephone	1.0	1.0	0.05653872458317062
17	3	camera	2.0	1.2231435513142097	0.13830995274687063
18	3	to	4.0	1.2231435513142097	0.27661990549374127
19	3	welcome	1.0	1.2231435513142097	0.06915497637343532
20	3	still	1.0	1.2231435513142097	0.06915497637343532

Let’s add the most occurent words in our VastFrame and compute the correlation vector.

for word in dict_spams.groupby("word", "SUM(tf) AS cnt").sort({"cnt": "desc"}).head(200).values["word"]:
    if word not in ['content', 'length', 'type'] : # because there is already a column called content, length and type
        try:
            spam.regexp(
                name = word,
                pattern = word,
                method = "count",
                column = "content",
            )
        except:
            pass
spam.corr(focus = "type")

Let’s just keep the first 100-most correlated features and merge the numbers together.

words = spam.corr(focus = "type", show = False)
spam.drop(columns = words["index"][101:])

for word in words["index"][1:101]:
    if any(char.isdigit() for char in word):
        spam[word].drop()

spam.regexp(
    column = "content",
    pattern = "([0-9])+",
    method = "count",
    name = "nb_numbers",
)

	123 type Integer 100%	Abc content Varchar(1010) 100%	123 length Bigint 100%	123 to Bigint 100%	123 a Bigint 100%	123 call Bigint 100%	123 your Bigint 100%	123 free Bigint 100%	123 now Bigint 100%	123 or Bigint 100%	123 txt Bigint 100%	123 u Bigint 100%	123 on Bigint 100%	123 ur Bigint 100%	123 from Bigint 100%	123 mobile Bigint 100%	123 text Bigint 100%	123 claim Bigint 100%	123 stop Bigint 100%	123 reply Bigint 100%	123 prize Bigint 100%	123 our Bigint 100%	123 won Bigint 100%	123 new Bigint 100%	123 nokia Bigint 100%	...	123 selected Bigint 100%	123 network Bigint 100%	123 mob Bigint 100%	123 offer Bigint 100%	123 weekly Bigint 100%	123 valid Bigint 100%	123 cost Bigint 100%	123 collect Bigint 100%	123 attempt Bigint 100%	123 r Bigint 100%	123 bonus Bigint 100%	123 sae Bigint 100%	123 vouchers Bigint 100%	123 rate Bigint 100%	123 c Bigint 100%	123 await Bigint 100%	123 unsubscribe Bigint 100%	123 land Bigint 100%	123 poly Bigint 100%	123 pobox Bigint 100%	123 winner Bigint 100%	123 top Bigint 100%	123 wk Bigint 100%	123 voucher Bigint 100%	123 nb_numbers Bigint 100%
1	0	k.then any other special?	25	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0
2	0	carlos is taking his sweet time as usual so let me know when you and patty are done/want to smoke and i'll tell him to haul ass	127	2	11	0	0	0	1	0	0	4	1	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	2	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0
3	0	ok pa. nothing problem:-)	25	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
4	0	have you heard about that job? i'm going to that wildlife talk again tonight if u want2come. its that2worzels and a wizzle or whatever it is?!	143	2	13	0	0	0	0	2	0	3	1	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	4	0	0	0	0	1	0	0	0	0	0	0	0	0	0	2
5	0	god picked up a flower and dippeditinadew, lovingly touched itwhichturnedinto u, and the he gifted tomeandsaid,this friend is 4u	128	3	6	0	0	0	0	0	0	5	0	1	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	3	0	0	0	0	3	0	0	0	0	0	0	0	0	0	1
6	0	when you came to hostel.	24	1	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0
7	0	ok no prob... i'll come after lunch then...	43	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	2	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0
8	0	jus telling u dat i'll b leaving 4 shanghai on 21st instead so we'll haf more time 2 meet up cya...	99	0	7	0	0	0	0	1	0	3	1	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1	0	0	0	0	0	0	0	0	0	3
9	0	are your freezing ? are you home yet ? will you remember to kiss your mom in the morning? do you love me ? do you think of me ? are you missing me yet ?	152	1	3	0	2	1	0	1	0	7	0	2	0	0	0	0	0	0	0	2	0	0	0	...	0	0	0	0	0	0	0	0	0	9	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
10	0	you all ready for * big day tomorrow?	37	1	3	0	0	0	0	2	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	4	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
11	0	i'll probably be around mu a lot	32	0	3	0	0	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
12	0	645	3	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1
13	1	rt-king pro video club>> need help? info@ringtoneking.co.uk or call 08701237397 you must be 16+ club credits redeemable at www.ringtoneking.co.uk! enjoy!	153	2	3	1	0	0	0	1	0	6	2	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	7	0	0	0	0	6	0	0	0	0	0	0	0	0	0	2
14	0	thnx dude. u guys out 2nite?	28	0	0	0	0	0	0	0	0	4	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1
15	0	me sef dey laugh you. meanwhile how's my darling anjie!	55	0	4	0	0	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
16	0	mm i had my food da from out	28	0	2	0	0	0	0	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
17	0	k, makes sense, btw carlos is being difficult so you guys are gonna smoke while i go pick up the second batch and get gas	121	0	7	0	0	0	0	0	0	4	2	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	2	0	0	0	0	5	0	0	0	0	0	0	0	0	0	0
18	0	did u download the fring app?	29	0	2	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
19	0	the 2 oz guy is being kinda flaky but one friend is interested in picking up $ <#> worth tonight if possible	115	1	2	0	0	0	0	1	0	3	2	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	3	0	0	0	0	1	0	0	0	0	0	0	0	0	0	1
20	0	friends that u can stay on fb chat with	39	0	4	0	0	0	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	...	0	0	0	0	0	0	0	0	0	1	0	0	0	0	2	0	0	0	0	0	0	0	0	0	0

Let’s narrow down our keyword list to words of more than two characters.

columns = spam.get_columns()
for word in columns:
    if len(word.replace('"', '')) <= 2:
        spam[word].drop()

Compute the correlation vector again using the response column.

spam.corr(focus = "type")

We have enough correlated features with our response to create a fantastic model.

---

Machine Learning

The LogisticRegression classifier is a powerful and performant algorithm for text analytics and binary classification. Before using it on our data, let’s use a cross-validation to test the efficiency of our model.

from vastorbit.machine_learning.vast import LogisticRegression

model = LogisticRegression(max_iter=1000)

from vastorbit.machine_learning.model_selection import cross_validate

cross_validate(
    model,
    spam,
    spam.get_columns(exclude_columns = ["type", "content"]),
    "type",
    cv = 5,
)

	auc	prc_auc	accuracy	log_loss	precision	recall	f1_score	mcc	informedness	markedness	csi	time
1-fold	0.9865825592258715	0.09704331287879608	0.982695810564663	0.062448879087365405	0.9758064516129032	0.8832116788321168	0.9272030651340997	0.9188798087206884	0.8800899306531367	0.9593793468901106	0.8642857142857143	0.45777010917663574
2-fold	0.984904372903835	0.1021524260447777	0.983274647887324	0.06478735830732793	0.9794520547945206	0.89937106918239	0.9377049180327869	0.9291919331545831	0.8963004448221032	0.9632904386329044	0.8827160493827161	0.3983190059661865
3-fold	0.9754224693124929	0.10910655354307175	0.9778761061946902	0.07327044938266461	0.9624060150375939	0.8648648648648649	0.911032028469751	0.9001131639560654	0.8597732151703639	0.9423458344959692	0.8366013071895425	0.33335399627685547
4-fold	0.9788645841964287	0.11173743409192372	0.975914362176628	0.07687763582266288	0.9552238805970149	0.8590604026845637	0.9045936395759717	0.8924976324421157	0.852887563178391	0.933947284852334	0.8258064516129032	0.4583089351654053
5-fold	0.9896015825505708	0.10570858057301646	0.9748201438848921	0.07510317460759292	0.9655172413793104	0.8588957055214724	0.9090909090909091	0.8965981002667766	0.8536270016226315	0.941732339621296	0.8333333333333334	0.35788416862487793
avg	0.9830751136378397	0.10514966142631714	0.9789162141416394	0.07049749944152275	0.9676811286842686	0.8730807442170816	0.9179249120607038	0.9074561277080457	0.8685356310893253	0.9481390488985229	0.8485485711608419	0.4011272430419922
std	0.005187189679032966	0.005178116283671962	0.003468487426698121	0.00577916320467179	0.00885740183517589	0.015868856266096746	0.012494855256784303	0.014131653714179682	0.017019376690604143	0.011242084623790847	0.021477983247294215	0.0508905155321698

We have an excellent model! Let’s learn from the data.

model.fit(
    spam,
    spam.get_columns(exclude_columns = ["type", "content"]),
    "type",
)
model.confusion_matrix()

Our model can reliably identify spam.

Conclusion

We’ve solved our problem in a pandas-like way, all without ever loading data into memory!