Loading...

Titanic

This example uses the titanic dataset to predict the survival of passengers on the Titanic.

We will follow the data science cycle (Data Exploration - Data Preparation - Data Modeling - Model Evaluation - Model Deployment) to solve this problem.

Initialization

This example uses the following version of vastorbit:

import vastorbit as vo

vo.__version__

Connect to VAST. This example uses an existing connection called VASTDSN . For details on how to create a connection, see the Connection tutorial.

You can skip the below cell if you already have an established connection.

vo.connect("VASTDSN")

Let’s create a VastFrame of the dataset.

from vastorbit.datasets import load_titanic

titanic = load_titanic()
titanic.head(5)
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
Abc
sex
Varchar(20)
100%
123
age
Double
79%
123
sibsp
Integer
100%
123
parch
Integer
100%
Abc
ticket
Varchar(36)
100%
123
fare
Double
99%
Abc
cabin
Varchar(30)
22%
Abc
embarked
Varchar(20)
99%
Abc
boat
Varchar(100)
37%
123
body
Integer
9%
Abc
home.dest
Varchar(100)
56%
111Allen, Miss. Elisabeth Waltonfemale29.00024160211.3375B5S2[null]St Louis, MO
211Allison, Master. Hudson Trevormale0.9212113781151.55C22 C26S11[null]Montreal, PQ / Chesterville, ON
310Allison, Miss. Helen Lorainefemale2.012113781151.55C22 C26S[null][null]Montreal, PQ / Chesterville, ON
410Allison, Mr. Hudson Joshua Creightonmale30.012113781151.55C22 C26S[null]135Montreal, PQ / Chesterville, ON
510Allison, Mrs. Hudson J C (Bessie Waldo Daniels)female25.012113781151.55C22 C26S[null][null]Montreal, PQ / Chesterville, ON

Data Exploration and Preparation

Let’s explore the data by displaying descriptive statistics of all the columns.

titanic.describe(method = "categorical", unique = True)
dtypecounttoptop_percentunique
"pclass"integer1309354.1633.0
"survived"integer1309061.8032.0
"name"varchar(164)1309Connolly, Miss. Kate0.1531307.0
"sex"varchar(20)1309male64.42.0
"age"double1046[null]20.09298.0
"sibsp"integer1309068.0677.0
"parch"integer1309076.5478.0
"ticket"varchar(36)1309CA. 23430.84929.0
"fare"double13088.054.584281.0
"cabin"varchar(30)295[null]77.464186.0
"embarked"varchar(20)1307S69.8243.0
"boat"varchar(100)486[null]62.87227.0
"body"integer121[null]90.756121.0
"home.dest"varchar(100)745[null]43.086369.0

The columns body (passenger ID), home.dest (passenger origin/destination), embarked (origin port) and ticket (ticket ID) shouldn’t influence survival, so we can ignore these.

Let’s focus our analysis on the columns name and cabin. We’ll begin with the passengers’ names.

from vastorbit.machine_learning.vast import TfidfVectorizer

model = TfidfVectorizer()
model.fit(titanic, "ticket", x="Name")
model.transform(titanic, "ticket", x="Name")["word"].topk()
countpercent
mr69614.926
miss2184.675
mrs1964.203
william711.523
john581.244
master420.901
henry400.858
james350.751
charles330.708
thomas310.665
george300.643
mary280.6
joseph270.579
elizabeth250.536
edward250.536
johan210.45
anna180.386
maria180.386
arthur170.365
frederick170.365
robert170.365
alfred170.365
alexander150.322
richard150.322
samuel150.322
karl130.279
jr130.279
j130.279
margaret130.279
patrick120.257
alice120.257
frank110.236
harry110.236
annie110.236
albert110.236
h100.214
helen100.214
ellen100.214
ernest100.214
david90.193
martin90.193
kate90.193
edith90.193
leonard90.193
peter90.193
elias90.193
dr80.172
walter80.172
e80.172
francis80.172
catherine80.172
rev80.172
august80.172
bertha70.15
nils70.15
louise70.15
victor70.15
florence70.15
daniel70.15
de60.129
johansson60.129
brown60.129
c60.129
y60.129
ivan60.129
bridget60.129
l60.129
a60.129
marie60.129
gustafsson60.129
andersson60.129
delia60.129
ada60.129
olsen60.129
augusta60.129
gustaf60.129
ida60.129
katherine60.129
ernst60.129
sofia60.129
benjamin50.107
marion50.107
emily50.107
philip50.107
davies50.107
howard50.107
hansen50.107
douglas50.107
smith50.107
williams50.107
edvin50.107
anne50.107
carl50.107
andrew50.107
emil50.107
emma50.107
edvard50.107
bertram50.107
michael50.107
kelly50.107
nellie50.107
hanna50.107
jane50.107
abraham50.107
ali40.086
katie40.086
karlsson40.086
reginald40.086
g40.086
elisabeth40.086
elsie40.086
julia40.086
martha40.086
johannes40.086
hugh40.086
col40.086
white40.086
dorothy40.086
herbert40.086
hocking40.086
sidney40.086
cacic40.086
van40.086
lillian40.086
may40.086
olof40.086
svensson40.086
stanley40.086
tannous40.086
harris40.086
frances40.086
constance40.086
paul40.086
hans40.086
jakob40.086
hannah40.086
leo40.086
wilhelm40.086
eugene40.086
nora40.086
ethel40.086
agnes40.086
percy40.086
clifford40.086
fleming30.064
miriam30.064
d30.064
hall30.064
ruth30.064
emilia30.064
rosalie30.064
helene30.064
rene30.064
ward30.064
allen30.064
gordon30.064
julius30.064
amy30.064
hughes30.064
edwin30.064
maggie30.064
jessie30.064
ii30.064
betros30.064
mabel30.064
hanora30.064
irene30.064
davis30.064
w30.064
charlotte30.064
daly30.064
rogers30.064
jacob30.064
larsson30.064
percival30.064
sigrid30.064
eleanor30.064
vander30.064
khalil30.064
jean30.064
bradley30.064
lily30.064
augustus30.064
johanna30.064
carlsson30.064
persson30.064
ann30.064
hays30.064
emilio30.064
caroline30.064
gladys30.064
eliza30.064
solomon30.064
julian30.064
adolf30.064
giles30.064
ware30.064
matilda30.064
gilbert30.064
r30.064
watson30.064
jose30.064
morgan30.064
gertrude30.064
boulos30.064
madeleine30.064
cor30.064
jensen30.064
oreskovic30.064
nilsson30.064
edgar30.064
johnson30.064
viktor30.064
wright30.064
graham30.064
wilson30.064
oscar30.064
jussila30.064
erik30.064
bengtsson30.064
olsson30.064
oskar30.064
keane30.064
barbara30.064
simon30.064
flynn30.064
stephen30.064
ford30.064
anderson30.064
elin30.064
norman30.064
amelia30.064
harold30.064
grace20.043
thamine20.043
youssef20.043
edmund20.043
romaine20.043
frederic20.043
fox20.043
mccarthy20.043
harper20.043
forbes20.043
denis20.043
assad20.043
canavan20.043
petroff20.043
zakarian20.043
anders20.043
jefferys20.043
isidor20.043
mathilde20.043
austin20.043
calic20.043
antoinette20.043
walton20.043
neal20.043
archibald20.043
thorne20.043
duane20.043
marjorie20.043
henrik20.043
clara20.043
lester20.043
ralph20.043
thomson20.043
morris20.043
ilmakangas20.043
brogren20.043
konrad20.043
washington20.043
oconnor20.043
arvid20.043
sophia20.043
noel20.043
christian20.043
scott20.043
manuel20.043
fredrik20.043
christy20.043
frauenthal20.043
cook20.043
juho20.043
meyer20.043
antti20.043
moore20.043
asplund20.043
josefina20.043
pauline20.043
carr20.043
gillespie20.043
virginia20.043
murphy20.043
goldsmith20.043
lucien20.043
parker20.043
oliver20.043
lawrence20.043
mark20.043
edmond20.043
m20.043
taylor20.043
mahon20.043
lalio20.043
rosa20.043
laura20.043
hugo20.043
emanuel20.043
lauritz20.043
spencer20.043
thompson20.043
lewis20.043
malkolm20.043
duff20.043
gerios20.043
warren20.043
bowen20.043
needs20.043
hunt20.043
aaron20.043
webber20.043
abelseth20.043
roger20.043
arne20.043
campbell20.043
hart20.043
tobin20.043
kink20.043
mathias20.043
arnold20.043
lane20.043
anthony20.043
bloomfield20.043
buckley20.043
ms20.043
phillips20.043
vande20.043
gretchen20.043
braund20.043
marija20.043
long20.043
eva20.043
andreas20.043
sara20.043
eileen20.043
dennis20.043
miller20.043
gerda20.043
andersen20.043
connolly20.043
coleff20.043
gerious20.043
hagland20.043
franz20.043
yousseff20.043
sarah20.043
hoyt20.043
wiklund20.043
fischer20.043
assaf20.043
godfrey20.043
marshall20.043
joel20.043
burns20.043
saad20.043
planke20.043
einar20.043
harald20.043
duran20.043
pokrnic20.043
crosby20.043
gifford20.043
jeannie20.043
attalah20.043
mlle20.043
kristina20.043
theodor20.043
winnie20.043
sarkis20.043
louis20.043
hubert20.043
jenny20.043
thayer20.043
daisy20.043
bonnell20.043
hilda20.043
clarence20.043
stanton20.043
lucy20.043
carrie20.043
carter20.043
mcgowan20.043
ryan20.043
jacques20.043
monypeny20.043
more20.043
jonsson20.043
lindqvist20.043
klasen20.043
clarke20.043
vivian20.043
minko20.043
luka20.043
thornton20.043
franklin20.043
maurice20.043
vilhelm20.043
oleary20.043
nikolai20.043
phyllis20.043
kiernan20.043
hedwig20.043
jennie20.043
andrews20.043
hjalmar20.043
leslie20.043
peder20.043
eino20.043
rowe20.043
rudolf20.043
hulda20.043
vera20.043
alfons20.043
lamson20.043
herman20.043
amin20.043
natalia20.043
juha20.043
matti20.043
stone20.043
bessie20.043
owen20.043
olaf20.043
nelson20.043
louisa20.043
stuart20.043
winifred20.043
adele20.043
chapman20.043
eugenie20.043
ingeborg20.043
bourke20.043
richards20.043
sigvard20.043
foley20.043
major20.043
moran20.043
razi20.043
alexandra20.043
harbeck20.043
mauritz20.043
matthew20.043
gertrud20.043
charlotta20.043
lilian20.043
berglund20.043
morley20.043
susan20.043
obrien20.043
timothy20.043
cruyssen10.021
goodwin10.021
warburton10.021
kristine10.021
farthing10.021
countess10.021
alhomaki10.021
nelle10.021
leander10.021
swift10.021
urho10.021
bowenur10.021
paust10.021
mohamed10.021
fosdick10.021
farred10.021
renouf10.021
luise10.021
malake10.021
osman10.021
dooley10.021
sutherland10.021
anton10.021
omont10.021
riordan10.021
ragnhild10.021
willey10.021
fannie10.021
kent10.021
lehmann10.021
hosono10.021
niklasson10.021
carver10.021
bazzani10.021
gilnagh10.021
kantor10.021
aime10.021
turcin10.021
maenpaa10.021
mapriededer10.021
kalle10.021
engelhart10.021
susanna10.021
buss10.021
beard10.021
dibden10.021
sivic10.021
karvin10.021
karoliina10.021
andreasson10.021
markun10.021
danbom10.021
philippe10.021
torborg10.021
ennis10.021
paula10.021
seman10.021
silverthorne10.021
markoff10.021
mitto10.021
pears10.021
shorney10.021
stevenson10.021
milton10.021
gavey10.021
jorgensen10.021
malachard10.021
cassem10.021
mcdowell10.021
moutal10.021
goldenberg10.021
clemmer10.021
carmichael10.021
ivar10.021
edwy10.021
bostandyeff10.021
dakic10.021
henriksson10.021
ibrahim10.021
selman10.021
livija10.021
francisco10.021
barber10.021
encarnacion10.021
sagesser10.021
albina10.021
aldworth10.021
lines10.021
andre10.021
jarvis10.021
fernand10.021
beatrice10.021
hodges10.021
peel10.021
demetrios10.021
roussel10.021
jovan10.021
ahlin10.021
mulvihill10.021
fernandeo10.021
icard10.021
gallagher10.021
newell10.021
jalsevac10.021
jerwan10.021
tenglin10.021
svend10.021
gunnar10.021
mccoy10.021
vendel10.021
linhart10.021
wizosky10.021
joshua10.021
israel10.021
assi10.021
badt10.021
mamee10.021
oconnell10.021
theodosia10.021
dulles10.021
frolicherstehli10.021
hamalainen10.021
elise10.021
leopold10.021
byles10.021
zahie10.021
cotterill10.021
fridtjof10.021
hold10.021
matinoff10.021
otter10.021
shellard10.021
raffull10.021
baxter10.021
stanlick10.021
beattie10.021
mangan10.021
murray10.021
vasil10.021
lucile10.021
husein10.021
compton10.021
wilfred10.021
evans10.021
ilario10.021
hippach10.021
ridsdale10.021
adolphe10.021
mirium10.021
s10.021
pedersen10.021
louch10.021
trevor10.021
reynaldo10.021
hawksford10.021
faunthorpe10.021
lithman10.021
gaskell10.021
perkin10.021
padro10.021
nassr10.021
lyyli10.021
isham10.021
abbott10.021
fiske10.021
ester10.021
eliina10.021
lovisa10.021
vilhelmina10.021
holm10.021
ogden10.021
zenni10.021
ellie10.021
rice10.021
stefo10.021
roth10.021
cumings10.021
butt10.021
gee10.021
fenton10.021
greenfield10.021
fynney10.021
sobey10.021
mallet10.021
barry10.021
brobeck10.021
hatfield10.021
persdotter10.021
mansour10.021
ristiu10.021
saether10.021
shadrach10.021
smyth10.021
marcella10.021
vanden10.021
horgan10.021
jacobsohn10.021
laury10.021
lingane10.021
todor10.021
violet10.021
sunderland10.021
reeves10.021
bissette10.021
sigfrid10.021
heinsheimer10.021
gosta10.021
mayne10.021
jackson10.021
edgardo10.021
ervin10.021
danoff10.021
florentina10.021
rowan10.021
givard10.021
whabee10.021
kirkland10.021
halaut10.021
natsch10.021
kinkheilmann10.021
juozas10.021
bengt10.021
rossmore10.021
viktoria10.021
cohen10.021
ole10.021
devaney10.021
pearce10.021
keefe10.021
walle10.021
mineff10.021
berthe10.021
achilles10.021
young10.021
brito10.021
carlo10.021
aks10.021
doling10.021
palsson10.021
jamila10.021
mme10.021
darwis10.021
birnbaum10.021
wittevrongel10.021
berriman10.021
bucknell10.021
mangiavacchi10.021
nicholas10.021
pinsky10.021
light10.021
jeffery10.021
laitinen10.021
chronopoulos10.021
mitkoff10.021
estanslas10.021
rasmussen10.021
rosenbaum10.021
willer10.021
of10.021
bowerman10.021
elmer10.021
conover10.021
magnin10.021
castellana10.021
kassem10.021
bystrom10.021
fang10.021
mellors10.021
lindell10.021
stokes10.021
stoytcho10.021
heikkinen10.021
nedelio10.021
siegwart10.021
zabour10.021
marsh10.021
ross10.021
colvin10.021
munger10.021
minkoff10.021
sinkkonen10.021
beauchamp10.021
kilgannon10.021
manent10.021
stina10.021
addie10.021
lena10.021
doyle10.021
melkebeke10.021
lam10.021
cresson10.021
helmina10.021
oliva10.021
hungerford10.021
stewart10.021
oberst10.021
riihivouri10.021
archie10.021
jennings10.021
rojj10.021
aline10.021
eliezer10.021
josefine10.021
ignjac10.021
guillaume10.021
moor10.021
hagardon10.021
hakan10.021
loraine10.021
wallach10.021
siegel10.021
frasar10.021
olai10.021
foo10.021
olga10.021
ingvald10.021
jan10.021
sdycoff10.021
stjepan10.021
govaert10.021
potter10.021
sallie10.021
holmes10.021
ashby10.021
liudevit10.021
charity10.021
hilding10.021
cornelius10.021
fared10.021
snyder10.021
gill10.021
selena10.021
asim10.021
lafargue10.021
theodore10.021
dibo10.021
tyrell10.021
houssein10.021
markland10.021
len10.021
rolmane10.021
brigdet10.021
fahlstrom10.021
berk10.021
gronnestad10.021
shine10.021
sirota10.021
nestor10.021
roderick10.021
zimmerman10.021
ryerson10.021
serepeca10.021
sebastiano10.021
swane10.021
braf10.021
banoura10.021
berg10.021
cerin10.021
hee10.021
nosworthy10.021
payne10.021
everett10.021
aronsson10.021
pietari10.021
lindblom10.021
elida10.021
lulic10.021
ortin10.021
danira10.021
parham10.021
spinner10.021
troutt10.021
dodge10.021
heininen10.021
heath10.021
adrian10.021
denzil10.021
ramell10.021
jeremiah10.021
joan10.021
marinko10.021
constanzia10.021
mionoff10.021
candee10.021
dutton10.021
kenyon10.021
lucille10.021
barrett10.021
don10.021
adola10.021
simonne10.021
beila10.021
pernot10.021
malvina10.021
conlon10.021
holland10.021
karaic10.021
hellstrom10.021
lundahl10.021
pennington10.021
abi10.021
homer10.021
chevre10.021
wayland10.021
molson10.021
levy10.021
rothschild10.021
mckane10.021
max10.021
peruschitz10.021
laleff10.021
emir10.021
lundstrom10.021
katriina10.021
holverson10.021
roland10.021
pillsbury10.021
allum10.021
augustsson10.021
carla10.021
mito10.021
leinonen10.021
osen10.021
touma10.021
elon10.021
howell10.021
werner10.021
caram10.021
harknett10.021
luigi10.021
lovell10.021
nancarrow10.021
mccormack10.021
martinez10.021
theobald10.021
deacon10.021
myna10.021
t10.021
lindstrom10.021
ekstrom10.021
clear10.021
brewe10.021
jonas10.021
chambers10.021
kvillner10.021
eustis10.021
masselmani10.021
stehli10.021
dolly10.021
baron10.021
yasbeck10.021
chip10.021
annan10.021
halvorsen10.021
blun10.021
hipkins10.021
schmidt10.021
antonine10.021
villiers10.021
roscoe10.021
sinai10.021
hood10.021
baccos10.021
balkic10.021
birger10.021
healy10.021
albin10.021
turpin10.021
creighton10.021
barah10.021
funk10.021
daher10.021
ambrose10.021
mardirosian10.021
inglis10.021
strouse10.021
ylio10.021
ponsonby10.021
rommetvedt10.021
spedden10.021
saundercock10.021
abbing10.021
nakli10.021
nakid10.021
victorine10.021
cyriel10.021
woolner10.021
kornelia10.021
price10.021
pedro10.021
lemercier10.021
sylfven10.021
qurban10.021
abrahamsson10.021
holthen10.021
loch10.021
jonkoff10.021
morrow10.021
adelia10.021
sage10.021
ulrik10.021
chaput10.021
genevieve10.021
najib10.021
sir10.021
patchett10.021
parr10.021
sap10.021
ponesell10.021
orsen10.021
stanko10.021
blanche10.021
stranden10.021
doharr10.021
tornquist10.021
maisner10.021
borland10.021
margareth10.021
cameron10.021
sadlier10.021
toomey10.021
servando10.021
edwart10.021
mccrae10.021
milan10.021
rizk10.021
suzette10.021
catharina10.021
shutes10.021
amelie10.021
jozef10.021
marta10.021
mcdermott10.021
lemore10.021
niilo10.021
michel10.021
peltomaki10.021
arnoldfranchi10.021
salonen10.021
shedid10.021
fortune10.021
hakkarainen10.021
forby10.021
reed10.021
lizzie10.021
allison10.021
hiltunen10.021
davidson10.021
moraweck10.021
fry10.021
berta10.021
ovies10.021
jelka10.021
franchi10.021
lesurer10.021
davison10.021
ball10.021
hirvonen10.021
bartol10.021
moussa10.021
lindahl10.021
millet10.021
tido10.021
stengel10.021
phillippe10.021
adahl10.021
dick10.021
linus10.021
nielsine10.021
bullen10.021
redjo10.021
strilic10.021
ling10.021
philemon10.021
sahid10.021
widegren10.021
nasr10.021
keeping10.021
christo10.021
lindeberglind10.021
sheerlinck10.021
abelson10.021
lionel10.021
denkoff10.021
thelma10.021
odwyer10.021
rheims10.021
barkworth10.021
mandelbaum10.021
nina10.021
matthews10.021
damsgaard10.021
mudd10.021
ileen10.021
nesson10.021
ellis10.021
eklund10.021
eberhard10.021
moubarek10.021
salkjelsvik10.021
olivia10.021
corey10.021
ocana10.021
makinen10.021
juliet10.021
hudson10.021
johannesson10.021
hassab10.021
petar10.021
moses10.021
colbert10.021
lemberopolous10.021
odahl10.021
marguerite10.021
panula10.021
behr10.021
sanni10.021
gibson10.021
flegenheim10.021
philipp10.021
b10.021
joaquim10.021
longley10.021
nenkoff10.021
farquarson10.021
petersen10.021
gus10.021
talmadge10.021
katavelas10.021
austen10.021
aurora10.021
reuchlin10.021
bernard10.021
margaretta10.021
sadowitz10.021
robins10.021
storey10.021
aubart10.021
thorneycroft10.021
corse10.021
irving10.021
harrison10.021
klaber10.021
bechstein10.021
adams10.021
weir10.021
ilia10.021
botsford10.021
vartanian10.021
miles10.021
rebecca10.021
nicolayarred10.021
gerald10.021
toufik10.021
giglio10.021
cosmo10.021
hilliard10.021
foreman10.021
posse10.021
eitemiller10.021
denbury10.021
francoise10.021
trout10.021
leah10.021
johann10.021
dai10.021
mernagh10.021
burke10.021
ohman10.021
agda10.021
simmons10.021
naidenoff10.021
p10.021
cornelia10.021
newsom10.021
harbaugh10.021
parrish10.021
masabumi10.021
vincenz10.021
madigan10.021
portage10.021
nile10.021
quigg10.021
winfield10.021
loring10.021
metcalfe10.021
jonkheer10.021
rosen10.021
coleridge10.021
mantoura10.021
wills10.021
mahala10.021
madsen10.021
truelove10.021
rush10.021
pickard10.021
vestrom10.021
leroy10.021
rintamaki10.021
adelaide10.021
emelia10.021
farrell10.021
stead10.021
lobb10.021
garside10.021
peacock10.021
greenberg10.021
ramon10.021
salomon10.021
birkhardt10.021
evelyn10.021
bryhl10.021
marthe10.021
kristensen10.021
sigurd10.021
milligan10.021
pavlovic10.021
norris10.021
skoog10.021
collett10.021
lahtinen10.021
mack10.021
todoroff10.021
rugg10.021
abisaab10.021
sharp10.021
wilkes10.021
ayoub10.021
parsons10.021
appleton10.021
slow10.021
borebank10.021
carrau10.021
kalvik10.021
saalfeld10.021
saade10.021
parkes10.021
iv10.021
haas10.021
maguire10.021
leeni10.021
smart10.021
polk10.021
vidaver10.021
larned10.021
imanita10.021
corning10.021
agatha10.021
atkinson10.021
meek10.021
baumgardner10.021
jaako10.021
helga10.021
milling10.021
christiana10.021
baclini10.021
straus10.021
mirko10.021
wyckoff10.021
ossian10.021
sedgwick10.021
nicolai10.021
drazenoic10.021
mcnamee10.021
nysveen10.021
maioni10.021
laventall10.021
edwina10.021
hallace10.021
delalic10.021
der10.021
fuller10.021
hale10.021
chibnall10.021
hildur10.021
silven10.021
shawah10.021
yrois10.021
kraeff10.021
mockler10.021
radeff10.021
mile10.021
strandberg10.021
beckwith10.021
wazli10.021
colley10.021
cardeza10.021
harriet10.021
lulu10.021
kimber10.021
thuillard10.021
ahmed10.021
knight10.021
badman10.021
castello10.021
aina10.021
easu10.021
mcevoy10.021
erna10.021
halstead10.021
latifa10.021
boulton10.021
thelander10.021
oakley10.021
jansson10.021
troupiansky10.021
warner10.021
vere10.021
laver10.021
harmer10.021
wennerstrom10.021
vassilios10.021
duquemin10.021
impe10.021
alexanteri10.021
futrelle10.021
jacobsen10.021
brines10.021
hanne10.021
karnes10.021
chaffee10.021
wheeler10.021
sleeper10.021
ilmari10.021
weisz10.021
wendla10.021
bing10.021
ivanoff10.021
linehan10.021
salander10.021
cordelia10.021
pomeroy10.021
marin10.021
said10.021
baptist10.021
ojala10.021
janko10.021
sivola10.021
cassebeer10.021
amanda10.021
toogood10.021
wirz10.021
gustave10.021
brayton10.021
lewy10.021
fletcher10.021
jovo10.021
del10.021
dimic10.021
andree10.021
lishin10.021
nasser10.021
lievens10.021
rouse10.021
billiard10.021
clinch10.021
treasteall10.021
chang10.021
bird10.021
robina10.021
borie10.021
tyler10.021
watt10.021
paulner10.021
plotcharsky10.021
somerton10.021
bidois10.021
stoytcheff10.021
crispin10.021
olive10.021
zebley10.021
andersenjensen10.021
neville10.021
alicia10.021
risien10.021
tillie10.021
quincy10.021
collander10.021
emerentia10.021
portaluppi10.021
grabowska10.021
worth10.021
margareta10.021
saiide10.021
gonios10.021
ferdinand10.021
brandeis10.021
peters10.021
goldschmidt10.021
bernhardina10.021
jego10.021
bateman10.021
blackwell10.021
albimona10.021
frans10.021
backstrom10.021
clyde10.021
lahoud10.021
hitchcock10.021
gracie10.021
williamslambert10.021
fellows10.021
angheloff10.021
escott10.021
jermyn10.021
andy10.021
pastcho10.021
lundin10.021
wood10.021
jones10.021
dahlberg10.021
leila10.021
honora10.021
hastings10.021
halim10.021
olaus10.021
velin10.021
dantcheff10.021
helena10.021
samaan10.021
crafton10.021
edmondus10.021
ismay10.021
margaretha10.021
barron10.021
porter10.021
dart10.021
kingcome10.021
n10.021
nieminen10.021
shawneene10.021
torber10.021
giuseppe10.021
charters10.021
valtcho10.021
culumovic10.021
kristo10.021
rahamin10.021
midtsjo10.021
barton10.021
mustafa10.021
doolina10.021
petco10.021
humblen10.021
morrison10.021
sleiman10.021
clinton10.021
peju10.021
guentcho10.021
lofqvist10.021
edvardsson10.021
smiljanic10.021
lang10.021
mock10.021
rowley10.021
mellinger10.021
novel10.021
thure10.021
tome10.021
juhantytar10.021
earnshaw10.021
cherry10.021
pede10.021
lingrey10.021
sultana10.021
ruby10.021
lennon10.021
pekka10.021
mara10.021
rodriguez10.021
youseff10.021
perreault10.021
delaudeniere10.021
alexenia10.021
webster10.021
lutie10.021
dona10.021
millvina10.021
hull10.021
haim10.021
laina10.021
goransson10.021
pasic10.021
staneff10.021
pekoniemi10.021
daniels10.021
waddington10.021
kreuchen10.021
antino10.021
karolina10.021
odriscoll10.021
karun10.021
brady10.021
eugenia10.021
navratil10.021
knud10.021
slemen10.021
vandemoortele10.021
kallio10.021
simoniusblumer10.021
lazar10.021
ilett10.021
marius10.021
claire10.021
mcneill10.021
walcroft10.021
nankoff10.021
sven10.021
camille10.021
jeso10.021
wilkinson10.021
zillah10.021
gideon10.021
soto10.021
alfonzo10.021
konstantia10.021
willard10.021
farnham10.021
gottfrid10.021
f10.021
goncalves10.021
karen10.021
palmquist10.021
iisakki10.021
heilmann10.021
alma10.021
mullens10.021
trembisky10.021
pehr10.021
blyler10.021
nysten10.021
georgette10.021
penasco10.021
celia10.021
caldwell10.021
minnie10.021
serafino10.021
vovk10.021
sibley10.021
francatelli10.021
sincock10.021
hogeboom10.021
sam10.021
reiersen10.021
slabenoff10.021
cahoone10.021
haxtun10.021
pettersson10.021
josefa10.021
churchill10.021
aloisia10.021
cavendish10.021
kanio10.021
marechal10.021
capt10.021
stephenson10.021
fermina10.021
maidment10.021
the10.021
hoef10.021
davids10.021
leyson10.021
guest10.021
antoni10.021
strom10.021
edwiga10.021
robbins10.021
schabert10.021
gates10.021
cunningham10.021
lamb10.021
ulrika10.021
messemaeker10.021
steen10.021
astor10.021
perez10.021
wick10.021
banfield10.021
floyd10.021
trevaskis10.021
domingos10.021
dolck10.021
penko10.021
torfa10.021
geiger10.021
roberta10.021
bailey10.021
maybery10.021
pulbaum10.021
waika10.021
bjornstromsteffansson10.021
apostolos10.021
coutts10.021
branko10.021
sivertsen10.021
selini10.021
harder10.021
blank10.021
chaudanson10.021
emilie10.021
akar10.021
uscher10.021
turja10.021
gregg10.021
towner10.021
vallejo10.021
elna10.021
bishop10.021
thorpe10.021
welles10.021
selma10.021
eiriik10.021
elina10.021
longstreth10.021
milley10.021
celotti10.021
yoto10.021
nikola10.021
mowad10.021
adolfina10.021
wardle10.021
eric10.021
wilhelmina10.021
hammad10.021
beesley10.021
bentham10.021
courtenay10.021
claus10.021
johanson10.021
viljami10.021
scanlan10.021
baptiste10.021
sylvia10.021
von10.021
maude10.021
mulder10.021
mannion10.021
kimball10.021
minahan10.021
mcmillan10.021
pieta10.021
viola10.021
achille10.021
windelov10.021
boeson10.021
manda10.021
evan10.021
sutehall10.021
cleaver10.021
solvang10.021
wenzel10.021
dunton10.021
green10.021
manta10.021
thorsten10.021
tikkanen10.021
waelens10.021
artagaveytia10.021
partner10.021
rachel10.021
francesco10.021
bracken10.021
wheadon10.021
nourelain10.021
dintcheff10.021
okeefe10.021
hickman10.021
jenkin10.021
drachstedt10.021
phoebe10.021
folke10.021
nicholson10.021
tucker10.021
dika10.021
mackay10.021
amenia10.021
lady10.021
hoffman10.021
christine10.021
naughton10.021
woolf10.021
algernon10.021
baimbrigge10.021
juliette10.021
mathilda10.021
meo10.021
aijo10.021
stankovic10.021
griggs10.021
guggenheim10.021
sloper10.021
lawry10.021
quick10.021
milne10.021
ernesti10.021
mate10.021
orian10.021
frolicher10.021
natalie10.021
kurt10.021
dahl10.021
demetri10.021
mcgough10.021
eden10.021
hegarty10.021
klas10.021
chisholm10.021
crothers10.021
hill10.021
honkanen10.021
willie10.021
velde10.021
briggs10.021
taussig10.021
bone10.021
oxenham10.021
garfirth10.021
erland10.021
petterson10.021
spector10.021
axel10.021
coelho10.021
satio10.021
georgetta10.021
thorilda10.021
moss10.021
cervin10.021
yousif10.021
calderhead10.021
glynn10.021
raihed10.021
harvey10.021
gurshon10.021
lindsey10.021
silvey10.021
carbines10.021
curnow10.021
nathan10.021
eugen10.021
montgomery10.021
sternin10.021
murdlin10.021
pentcho10.021
mccaffry10.021
walker10.021
ebba10.021
mariana10.021
gheorgheff10.021
gilinski10.021
hassan10.021
omine10.021
fabian10.021
thor10.021
angle10.021
alden10.021
nicholls10.021
birkeland10.021
crease10.021
bernt10.021
lockyer10.021
saks10.021
gale10.021
emile10.021
wonnacott10.021
filip10.021
brocklebank10.021
connaghton10.021
pelsmaeker10.021
mcgovern10.021
niskanen10.021
enander10.021
hewlett10.021
mitchell10.021
bjorklund10.021
sandstrom10.021
weller10.021
marvin10.021
rood10.021
butler10.021
tate10.021
laroche10.021
henriette10.021
osullivan10.021
sjoblom10.021
tomlin10.021
viljo10.021
fardon10.021
hampe10.021
cornell10.021
madill10.021
seward10.021
chehab10.021
soholt10.021
melville10.021
dyeredwards10.021
baird10.021
dennick10.021
arman10.021
harrington10.021
wells10.021
treanor10.021
mussey10.021
maxfield10.021
beane10.021
argenia10.021
elsbury10.021
johnston10.021
mari10.021
rosblom10.021
maxmillian10.021
fillbrook10.021
bennett10.021
josef10.021
monsen10.021
dowdell10.021
frankie10.021
k10.021
baumann10.021
corbett10.021
assam10.021
choong10.021
veal10.021
iris10.021
pierre10.021
eloise10.021
elkins10.021
pain10.021
dean10.021
olsvigen10.021
force10.021
willingham10.021
drew10.021
nye10.021
nicola10.021
myhrman10.021
norah10.021
turkula10.021
dickinson10.021
irwin10.021
grice10.021
esther10.021
shelley10.021
dyker10.021
donald10.021
amalie10.021
ringhini10.021
hyman10.021
larssonrondberg10.021
petranec10.021
allis10.021
leon10.021
landergren10.021
lyntakoff10.021
mansouer10.021
ostby10.021
rosenshine10.021
niels10.021
shaughnessy10.021
valentine10.021
cribb10.021
johannesenbratthammer10.021
meanwell10.021
rut10.021
hileni10.021
becker10.021
judith10.021
danielsen10.021
ilieff10.021
stahelinmaeglin10.021
montvila10.021
nassef10.021
felix10.021
corn10.021
catavelas10.021
neshan10.021
sawyer10.021
junkins10.021
sante10.021
lottie10.021
sophie10.021
lee10.021
neto10.021
reynolds10.021
weart10.021
case10.021
peuchen10.021
genovesi10.021
achem10.021
lefebre10.021
ingersoll10.021
dorking10.021
margido10.021
norton10.021
stella10.021
margit10.021
leader10.021
mildred10.021
julie10.021
collyer10.021
stanio10.021
moen10.021
artur10.021
slocovski10.021
waldo10.021
cairns10.021
sjostedt10.021
beavan10.021
finck10.021
hendekovic10.021
gervasius10.021
peduzzi10.021
salli10.021
fisher10.021
uruchurtu10.021
downton10.021
frost10.021
christmann10.021
finoli10.021
isaac10.021
haven10.021
chamberlain10.021
slayter10.021
wilhelms10.021
hedman10.021
manca10.021
lurette10.021
manley10.021
pallas10.021
algot10.021
joackim10.021
fatima10.021
ingvar10.021
odonoghue10.021
endres10.021
margaritha10.021
ella10.021
widener10.021
mccrie10.021
mcmahon10.021
kiamie10.021
telma10.021
west10.021
connors10.021
sutton10.021
dagmar10.021
asuncion10.021
pengelly10.021
abrahim10.021
drapkin10.021
cater10.021
dale10.021
leontine10.021
drake10.021
roebling10.021
nirva10.021
rekic10.021
flora10.021
leitch10.021
nourney10.021
grga10.021
cann10.021
juul10.021
sirayanian10.021
clark10.021
mcdougald10.021
satode10.021
mae10.021
myles10.021
hayden10.021
alfrida10.021
meier10.021
sundman10.021
bruce10.021
wearne10.021
rothes10.021
cummins10.021
marian10.021
maybelle10.021
v10.021
sweet10.021
jardin10.021
kennedy10.021
georges10.021
carlcharles10.021
christopher10.021
coxon10.021
krekorian10.021
fahim10.021
wiseman10.021
head10.021

Passengers’ titles might come in handy. We can extract these from their names.

Let’s move on to the cabins.

model = TfidfVectorizer()
model.fit(titanic, "ticket", x="cabin")
model.transform(titanic, "ticket", x="cabin")["word"].topk()
countpercent
f83.433
d41.717
f3341.717
e10131.288
e2420.858
c7820.858
d1720.858
e4620.858
b5120.858
g620.858
b5520.858
c10620.858
g7320.858
e2520.858
c5220.858
g6320.858
b5320.858
c10120.858
f220.858
c12420.858
c9510.429
b10210.429
e6810.429
e4510.429
e5710.429
b3010.429
a1810.429
c14810.429
b2210.429
d4010.429
a2410.429
e4410.429
b1910.429
b4510.429
c12810.429
e810.429
d2610.429
c3010.429
d5010.429
t10.429
d1210.429
d3810.429
d3010.429
d4610.429
c6810.429
d4310.429
e1210.429
b9410.429
c3210.429
d4810.429
b5610.429
c4510.429
c8710.429
a2610.429
c12510.429
b9810.429
b2410.429
c8510.429
c11610.429
b3910.429
a3210.429
e6010.429
d3710.429
d610.429
e5010.429
c4610.429
c10410.429
b1810.429
b5010.429
c6210.429
b4910.429
d3410.429
b7910.429
c5310.429
e5210.429
c8310.429
b8410.429
b3510.429
b5710.429
b10110.429
a3110.429
c210.429
c5710.429
c9010.429
b1010.429
c8610.429
e6710.429
b1110.429
e4910.429
a1410.429
b8010.429
c11810.429
e6310.429
c9310.429
c11010.429
f410.429
c8210.429
e3610.429
e1710.429
e3310.429
c3110.429
a1910.429
e5810.429
e6910.429
d2010.429
d3610.429
b3810.429
b7710.429
a2310.429
c4910.429
d2210.429
b2610.429
b410.429
c2710.429
a510.429
a2110.429
b5410.429
d5610.429
b6310.429
c710.429
e4110.429
c4710.429
b7110.429
b9610.429
d4510.429
c9110.429
e1010.429
d2110.429
c9910.429
a1010.429
d910.429
a710.429
c10510.429
c9710.429
c11110.429
e3110.429
a2010.429
c9210.429
b2010.429
c8010.429
c5010.429
c5110.429
c7010.429
e3810.429
b510.429
a3410.429
b6010.429
c5410.429
d1510.429
b6910.429
d710.429
a1610.429
c2510.429
b6610.429
f3810.429
c2310.429
c12610.429
d1910.429
b2810.429
c2810.429
a1110.429
a2910.429
a610.429
d3310.429
b8610.429
c13210.429
c13010.429
c2610.429
d1110.429
b3610.429
c6510.429
d1010.429
e12110.429
b7310.429
e7710.429
c12310.429
b4210.429
b7810.429
c3910.429
c2210.429
b8210.429
b5210.429
c5510.429
b310.429
a910.429
b4110.429
b6110.429
e3410.429
e4010.429
e3910.429
c8910.429
d4910.429
d2810.429
c6410.429
d4710.429
c610.429
b5910.429
c10310.429
b5810.429
b3710.429
a3610.429
d3510.429

Here, we have the cabin IDs, the letter of which represents a certain position on the boat. Let’s see how often each cabin occurs in the dataset.

model = TfidfVectorizer()
model.fit(titanic, "ticket", x="cabin")
model.transform(titanic, "ticket", x="cabin")["word"].str_slice(1, 2).groupby(
    columns = ["word"], expr = ["SUM(tf) AS occurences"]
).sort({"occurences": "desc"})
Abc
word
Varchar(495)
100%
123
occurences
Double
100%
1c114.0
2b96.0
3d48.0
4e45.0
5a22.0
6f21.0
7g9.0
8t1.0

While NULL values for “boat” clearly represent passengers who have a dedicated “lifeboat”, we can’t be so sure about NULL values for “cabin”. We can guess that these might represent passengers without a cabin. If this is the case, then these are missing values not at random (MNAR).

We’ll revisit this problem later. For now, let’s drop the columns that don’t affect survival and then encode the rest.

titanic.drop(["body", "home.dest", "embarked", "ticket"])
titanic["cabin"].str_slice(1, 2)["name"].str_extract(
        ' ([A-Za-z]+)\\.')["boat"].fillna(
        method = "0ifnull"
)["cabin"].fillna("No Cabin")["name"].one_hot_encode()
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
Abc
sex
Varchar(20)
100%
123
age
Double
79%
123
sibsp
Integer
100%
123
parch
Integer
100%
123
fare
Double
99%
Abc
cabin
Varchar(30)
100%
123
boat
Bool
100%
123
name__Capt.
Bool
100%
123
name__Col.
Bool
100%
123
name__Countess.
Bool
100%
123
name__Don.
Bool
100%
123
name__Dona.
Bool
100%
123
name__Dr.
Bool
100%
123
name__Jonkheer.
Bool
100%
123
name__Lady.
Bool
100%
123
name__Major.
Bool
100%
123
name__Master.
Bool
100%
123
name__Miss.
Bool
100%
123
name__Mlle.
Bool
100%
123
name__Mme.
Bool
100%
123
name__Mr.
Bool
100%
123
name__Mrs.
Bool
100%
123
name__Ms.
Bool
100%
123
name__Rev.
Bool
100%
111 Miss.female29.000211.3375B100000000001000000
211 Master.male0.9212151.55C100000000010000000
310 Miss.female2.012151.55C000000000001000000
410 Mr.male30.012151.55C000000000000001000
510 Mrs.female25.012151.55C000000000000000100
611 Mr.male48.00026.55E100000000000001000
711 Miss.female63.01077.9583D100000000001000000
810 Mr.male39.0000.0A000000000000001000
911 Mrs.female53.02051.4792C100000000000000100
1010 Mr.male71.00049.5042No Cabin000000000000001000
1110 Col.male47.010227.525C001000000000000000
1211 Mrs.female18.010227.525C100000000000000100
1311 Mme.female24.00069.3B100000000000010000
1411 Miss.female26.00078.85No Cabin100000000001000000
1511 Mr.male80.00030.0A100000000000001000
1610 Mr.male[null]0025.925No Cabin000000000000001000
1710 Mr.male24.001247.5208B000000000000001000
1811 Mrs.female50.001247.5208B100000000000000100
1911 Miss.female32.00076.2917D100000000001000000
2010 Mr.male36.00075.2417C100000000000001000

Looking at our data now, we can see that some first class passengers have a NULL value for their cabin, so we can safely say that our assumption about the meaning of a NULL value of “cabin” turned out to be incorrect. This means that the “cabin” column has far too many missing values at random (MAR). We’ll have to drop it.

titanic["cabin"].drop()
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
Abc
sex
Varchar(20)
100%
123
age
Double
79%
123
sibsp
Integer
100%
123
parch
Integer
100%
123
fare
Double
99%
123
boat
Bool
100%
123
name__Capt.
Bool
100%
123
name__Col.
Bool
100%
123
name__Countess.
Bool
100%
123
name__Don.
Bool
100%
123
name__Dona.
Bool
100%
123
name__Dr.
Bool
100%
123
name__Jonkheer.
Bool
100%
123
name__Lady.
Bool
100%
123
name__Major.
Bool
100%
123
name__Master.
Bool
100%
123
name__Miss.
Bool
100%
123
name__Mlle.
Bool
100%
123
name__Mme.
Bool
100%
123
name__Mr.
Bool
100%
123
name__Mrs.
Bool
100%
123
name__Ms.
Bool
100%
123
name__Rev.
Bool
100%
111 Miss.female29.000211.3375100000000001000000
211 Master.male0.9212151.55100000000010000000
310 Miss.female2.012151.55000000000001000000
410 Mr.male30.012151.55000000000000001000
510 Mrs.female25.012151.55000000000000000100
611 Mr.male48.00026.55100000000000001000
711 Miss.female63.01077.9583100000000001000000
810 Mr.male39.0000.0000000000000001000
911 Mrs.female53.02051.4792100000000000000100
1010 Mr.male71.00049.5042000000000000001000
1110 Col.male47.010227.525001000000000000000
1211 Mrs.female18.010227.525100000000000000100
1311 Mme.female24.00069.3100000000000010000
1411 Miss.female26.00078.85100000000001000000
1511 Mr.male80.00030.0100000000000001000
1610 Mr.male[null]0025.925000000000000001000
1710 Mr.male24.001247.5208000000000000001000
1811 Mrs.female50.001247.5208100000000000000100
1911 Miss.female32.00076.2917100000000001000000
2010 Mr.male36.00075.2417100000000000001000

Let’s look at descriptive statistics of the entire VastFrame.

titanic.describe(method = "all")
"pclass""survived""age""sibsp""parch""fare""boat""name__Capt.""name__Col.""name__Countess.""name__Don.""name__Dona.""name__Dr.""name__Jonkheer.""name__Lady.""name__Major.""name__Master.""name__Miss.""name__Mlle.""name__Mme.""name__Mr.""name__Mrs.""name__Ms.""name__Rev.""name""sex"
dtypeintegerintegerdoubleintegerintegerdoubleboolboolboolboolboolboolboolboolboolboolboolboolboolboolboolboolboolboolvarchar(164)varchar(20)
percent10010079.90810010099.924100100100100100100100100100100100100100100100100100100100100
count1309130910461309130913081309.01309130913091309130913091309130913091309130913091309130913091309130913091309
top30[null]008.05000000000000001000 Mr.male
top_percent54.16361.80320.09268.06776.5474.58462.87299.92499.69499.92499.92499.92499.38999.92499.92499.84795.3480.13899.84799.92457.8384.9599.84799.38957.8364.4
avg2.2948815889992360.381970970206264329.8811376673040140.49885408708938120.385026737967914433.295479281345650.37127578304048890.00076394194041252860.00305576776165011450.00076394194041252860.00076394194041252860.00076394194041252860.0061115355233002290.00076394194041252860.00076394194041252860.00152788388082505730.046600458365164250.198624904507257450.00152788388082505730.00076394194041252860.57830404889228420.150496562261268150.00152788388082505730.0061115355233002294.7669977081741794.711993888464477
stddev0.83783601897012720.486055170866483214.4134932112713271.04165839059610190.86556027534951551.7586682391741850.48333067286175090.0276394996411391150.055215569543335830.0276394996411391220.0276394996411391150.0276394996411391220.077966842496727240.0276394996411391220.027639499641139120.039073210437363970.210862093939001270.399117456085774260.0390732104373639750.0276394996411391150.4940191491389880.35769412859151850.0390732104373639960.077966842496727221.09486370376008720.9579945668826566
min100.17000.000000000000000000044
approx_25%2020.90530267915083007.92011339130434800000000000000000044
approx_50%3028.17647381209010014.51723166666666600000000000000100044
approx_75%3138.573109243697481031.2488899686073110000000000000100066
max3180.089512.3292111111111111111111106
range2179.8389512.329211111111111111111162
empty[null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null][null]00

Descriptive statistics can give us valuable insights into our data. Notice, for example, that the column “fare” has many outliers (The maximum of 512.33 is much greater than the 9th decile of 79.13). Most passengers traveled in 3rd class (median of pclass = 3).

The “sibsp” column represents the number of siblings for each passenger, while the “parch” column represents the number of parents and children. We can use these to create a new feature: “family_size”.

titanic["family_size"] = titanic["parch"] + titanic["sibsp"] + 1

Let’s move on to outliers. We have several tools for locating outliers (LocalOutlierFactor, DBSCAN, KMeans…), but we’ll just use winsorization in this example. Again, “fare” has many outliers, so we’ll start there.

titanic["fare"].fill_outliers(
    method = "winsorize",
    alpha = 0.03,
)
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
Abc
sex
Varchar(20)
100%
123
age
Double
79%
123
sibsp
Integer
100%
123
parch
Integer
100%
123
fare
Double
99%
123
boat
Bool
100%
123
name__Capt.
Bool
100%
123
name__Col.
Bool
100%
123
name__Countess.
Bool
100%
123
name__Don.
Bool
100%
123
name__Dona.
Bool
100%
123
name__Dr.
Bool
100%
123
name__Jonkheer.
Bool
100%
123
name__Lady.
Bool
100%
123
name__Major.
Bool
100%
123
name__Master.
Bool
100%
123
name__Miss.
Bool
100%
123
name__Mlle.
Bool
100%
123
name__Mme.
Bool
100%
123
name__Mr.
Bool
100%
123
name__Mrs.
Bool
100%
123
name__Ms.
Bool
100%
123
name__Rev.
Bool
100%
123
family_size
Integer
100%
111 Miss.female29.000211.33751000000000010000001
211 Master.male0.9212151.551000000000100000004
310 Miss.female2.012151.550000000000010000004
410 Mr.male30.012151.550000000000000010004
510 Mrs.female25.012151.550000000000000001004
611 Mr.male48.00026.551000000000000010001
711 Miss.female63.01077.95831000000000010000002
810 Mr.male39.0000.00000000000000010001
911 Mrs.female53.02051.47921000000000000001003
1010 Mr.male71.00049.50420000000000000010001
1110 Col.male47.010227.5250010000000000000002
1211 Mrs.female18.010227.5251000000000000001002
1311 Mme.female24.00069.31000000000000100001
1411 Miss.female26.00078.851000000000010000001
1511 Mr.male80.00030.01000000000000010001
1610 Mr.male[null]0025.9250000000000000010001
1710 Mr.male24.001240.33015223804240000000000000010002
1811 Mrs.female50.001240.33015223804241000000000000001002
1911 Miss.female32.00076.29171000000000010000001
2010 Mr.male36.00075.24171000000000000010001

Let’s encode the column sex so we can use it with numerical methods.

titanic["sex"].label_encode()
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
123
sex
Integer
100%
123
age
Double
79%
123
sibsp
Integer
100%
123
parch
Integer
100%
123
fare
Double
99%
123
boat
Bool
100%
123
name__Capt.
Bool
100%
123
name__Col.
Bool
100%
123
name__Countess.
Bool
100%
123
name__Don.
Bool
100%
123
name__Dona.
Bool
100%
123
name__Dr.
Bool
100%
123
name__Jonkheer.
Bool
100%
123
name__Lady.
Bool
100%
123
name__Major.
Bool
100%
123
name__Master.
Bool
100%
123
name__Miss.
Bool
100%
123
name__Mlle.
Bool
100%
123
name__Mme.
Bool
100%
123
name__Mr.
Bool
100%
123
name__Mrs.
Bool
100%
123
name__Ms.
Bool
100%
123
name__Rev.
Bool
100%
123
family_size
Integer
100%
111 Miss.029.000211.33751000000000010000001
211 Master.10.9212151.551000000000100000004
310 Miss.02.012151.550000000000010000004
410 Mr.130.012151.550000000000000010004
510 Mrs.025.012151.550000000000000001004
611 Mr.148.00026.551000000000000010001
711 Miss.063.01077.95831000000000010000002
810 Mr.139.0000.00000000000000010001
911 Mrs.053.02051.47921000000000000001003
1010 Mr.171.00049.50420000000000000010001
1110 Col.147.010227.5250010000000000000002
1211 Mrs.018.010227.5251000000000000001002
1311 Mme.024.00069.31000000000000100001
1411 Miss.026.00078.851000000000010000001
1511 Mr.180.00030.01000000000000010001
1610 Mr.1[null]0025.9250000000000000010001
1710 Mr.124.001240.33015223804240000000000000010002
1811 Mrs.050.001240.33015223804241000000000000001002
1911 Miss.032.00076.29171000000000010000001
2010 Mr.136.00075.24171000000000000010001

The column age has too many missing values and since most machine learning algorithms can’t handle missing values, we need to impute our data. Let’s fill the missing values using the average age of the passengers that have the same “pclass” and “sex”.

titanic["age"].fillna(method = "mean", by = ["pclass", "sex"])
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
123
sex
Integer
100%
123
age
Real
100%
123
sibsp
Integer
100%
123
parch
Integer
100%
123
fare
Double
99%
123
boat
Bool
100%
123
name__Capt.
Bool
100%
123
name__Col.
Bool
100%
123
name__Countess.
Bool
100%
123
name__Don.
Bool
100%
123
name__Dona.
Bool
100%
123
name__Dr.
Bool
100%
123
name__Jonkheer.
Bool
100%
123
name__Lady.
Bool
100%
123
name__Major.
Bool
100%
123
name__Master.
Bool
100%
123
name__Miss.
Bool
100%
123
name__Mlle.
Bool
100%
123
name__Mme.
Bool
100%
123
name__Mr.
Bool
100%
123
name__Mrs.
Bool
100%
123
name__Ms.
Bool
100%
123
name__Rev.
Bool
100%
123
family_size
Integer
100%
111 Miss.029.000211.33751000000000010000001
210 Miss.02.012151.550000000000010000004
310 Mrs.025.012151.550000000000000001004
411 Miss.063.01077.95831000000000010000002
511 Mrs.053.02051.47921000000000000001003
611 Mrs.018.010227.5251000000000000001002
711 Mme.024.00069.31000000000000100001
811 Miss.026.00078.851000000000010000001
911 Mrs.050.001240.33015223804241000000000000001002
1011 Miss.032.00076.29171000000000010000001
1111 Mrs.047.01152.55421000000000000001003
1211 Miss.042.000227.5251000000000010000001
1311 Miss.029.000221.77921000000000010000001
1411 Mrs.019.01091.07921000000000000001002
1511 Miss.035.000135.63331000000000010000001
1611 Miss.030.000164.86671000000000010000001
1711 Miss.058.00026.551000000000010000001
1811 Miss.045.000240.33015223804241000000000010000001
1911 Miss.022.00155.01000000000010000002
2011 Mrs.044.00027.72081000000000000001001

Let’s draw the correlation matrix to see the links between variables.

titanic.corr(method = "spearman")

Fare correlates strongly with family size. This is about what you would expect: a larger family means more tickets, and more tickets means a greater fare.

Survival correlates strongly with whether or not a passenger has a lifeboat (the boat variable). Still, to increase the generality of our model, we should avoid predictions based on just one variable. Let’s split the study into two use cases:

  • Passengers with a lifeboat

  • Passengers without a lifeboat

Before we move on: we did a lot of work to clean up this data, but we haven’t saved anything to our VAST DataBase! Let’s look at the modifications we’ve made to the VastFrame.

print(titanic.current_relation())

Let see what’s happening when we aggregate and turn on SQL generation.

vo.set_option("sql_on", True)
titanic_avg = titanic.avg()

vastorbit dynamically generates SQL code whenever you make modifications to your data. To avoid recomputation, it also stores previous aggregations. If we filter anything in our data, it will update the catalog with our modifications.

vo.set_option("sql_on", False)
print(titanic.info())

Let’s move on to modeling our data. Save the VastFrame to your VAST DataBase.

from vastorbit.sql import drop

# Titanic Boat
drop("titanic_boat", method = "view")
titanic_boat = titanic.search(titanic["boat"] == 1).to_db("titanic_boat", relation_type = "view", inplace = True)

# Titanic No Boat
drop("titanic_no_boat", method = "view")
titanic_no_boat = titanic.search(titanic["boat"] == 0).to_db("titanic_no_boat", relation_type = "view", inplace = True)

Machine Learning

Passengers with a lifeboat

First, let’s look at the number of survivors.

titanic_boat["survived"].describe()
value
name"survived"
dtypeinteger
unique2.0
count486.0
1477
09

We have nine deaths. Let’s try to understand why these passengers died.

titanic_boat.search(titanic_boat["survived"] == 0).head(10)
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
123
sex
Integer
100%
123
age
Double
100%
123
sibsp
Integer
100%
123
parch
Integer
100%
123
fare
Double
100%
123
boat
Integer
100%
123
name__Capt.
Integer
100%
123
name__Col.
Integer
100%
123
name__Countess.
Integer
100%
123
name__Don.
Integer
100%
123
name__Dona.
Integer
100%
123
name__Dr.
Integer
100%
123
name__Jonkheer.
Integer
100%
123
name__Lady.
Integer
100%
123
name__Major.
Integer
100%
123
name__Master.
Integer
100%
123
name__Miss.
Integer
100%
123
name__Mlle.
Integer
100%
123
name__Mme.
Integer
100%
123
name__Mr.
Integer
100%
123
name__Mrs.
Integer
100%
123
name__Ms.
Integer
100%
123
name__Rev.
Integer
100%
123
family_size
Integer
100%
130 Mr.127.01014.45421000000000000010002
230 Mr.132.01015.851000000000000010002
330 Mr.125.0007.251000000000000010001
430 Mr.125.962263610315187007.251000000000000010001
530 Mr.136.01015.551000000000000010002
610 Mr.136.00075.24171000000000000010001
710 Mr.141.029271523178810030.69581000000000000010001
830 Mrs.030.01015.551000000000000001002
920 Mr.134.01021.01000000000000010002

Apart from a decent amount of these passengers being third-class passengers, it doesn’t seem like there are any clear predictors here for their deaths. Making a model from this would be unhelpful.

Passengers without a lifeboat

Let’s move on to passengers without a lifeboat.

titanic_no_boat["survived"].describe()
value
name"survived"
dtypeinteger
unique2.0
count823.0
0800
123

Only 23 survived. Let’s find out why.

titanic_no_boat.search(titanic_boat["survived"] == 1).head(20)
123
pclass
Integer
100%
123
survived
Integer
100%
Abc
name
Varchar(164)
100%
123
sex
Integer
100%
123
age
Double
100%
123
sibsp
Integer
100%
123
parch
Integer
100%
123
fare
Double
100%
123
boat
Integer
100%
123
name__Capt.
Integer
100%
123
name__Col.
Integer
100%
123
name__Countess.
Integer
100%
123
name__Don.
Integer
100%
123
name__Dona.
Integer
100%
123
name__Dr.
Integer
100%
123
name__Jonkheer.
Integer
100%
123
name__Lady.
Integer
100%
123
name__Major.
Integer
100%
123
name__Master.
Integer
100%
123
name__Miss.
Integer
100%
123
name__Mlle.
Integer
100%
123
name__Mme.
Integer
100%
123
name__Mr.
Integer
100%
123
name__Mrs.
Integer
100%
123
name__Ms.
Integer
100%
123
name__Rev.
Integer
100%
123
family_size
Integer
100%
131 Miss.015.0008.02920000000000010000001
231 Mrs.022.185328947368422007.22920000000000000001001
331 Mrs.022.1853289473684221015.50000000000000001002
431 Miss.022.185328947368422007.87920000000000010000001
531 Mrs.031.0008.68330000000000000001001
631 Miss.022.185328947368422007.77920000000000010000001
731 Mrs.047.0107.00000000000000001002
831 Mrs.015.01014.45420000000000000001002
931 Mrs.033.03015.850000000000000001004
1031 Miss.023.0008.050000000000010000001
1131 Miss.026.0007.9250000000000010000001
1231 Miss.027.0007.9250000000000010000001
1331 Mr.125.962263610315187007.750000000000000010001
1431 Mr.125.962263610315187007.750000000000000010001
1521 Mrs.042.00013.00000000000000001001
1621 Miss.018.00123.00000000000010000002
1721 Mrs.034.00123.00000000000000001002
1821 Miss.017.00010.50000000000010000001
1921 Mrs.042.01026.00000000000000001002
2011 Miss.058.000146.52080000000000010000001

Most survivors seem to be women. Let’s build a model with this in mind.

One of our predictors is categorical: the passenger title. Some of these predictors are corrleated, so it’d be best to work with a non-linear classifier that can handle that. In this case, a random forest classifier seems to be perfect. Let’s evaluate it with a cross-validation.

from vastorbit.machine_learning.vast import RandomForestClassifier
from vastorbit.machine_learning.model_selection import cross_validate

predictors = titanic.get_columns(exclude_columns = ["survived", "name", "fare", "boat"])
response = "survived"
model = RandomForestClassifier(
    n_estimators = 7
)
cross_validate(model, titanic_no_boat, predictors, response)
aucprc_aucaccuracylog_lossprecisionrecallf1_scoremccinformednessmarkednesscsitime
1-fold0.00.00.0107526881720430120.017393775712155140.00.00.00.0-0.9892473118279570.00.00.31421613693237305
2-fold0.00.00.017730496453900710.041191186275547960.00.00.00.0-0.98226950354609930.00.00.3452432155609131
3-fold0.00.00.00358422939068100360.028092614683797260.00.00.00.0-0.9964157706093190.00.00.34694600105285645
avg0.00.00.0106891380055415750.0288925255571667870.00.00.00.0-0.98931086199445850.00.00.33546845118204754
std0.00.00.0057753641688197440.009731703531402610.00.00.00.00.0057753641688197170.00.00.015043725498981978

This dataset is pretty unbalanced so we’ll use an AUC to evaluate it. Looking at our table, our model has an average AUC of more than 0.8, so our model is quite good.

We can now build a model with the entire dataset.

model.fit(titanic_no_boat, predictors, response)

Let’s look at the importance of each feature.

model.features_importance()

As expected, the passenger’s age and title are the most important predictors of survival.

Conclusion

We’ve solved our problem in a pandas-like way, all without ever loading data into memory!