vastorbit.VastFrame.train_test_split

VastFrame.train_test_split(test_size: float = 0.33, order_by: None | str | list | dict = None, random_state: int = None) → tuple[VastFrame, VastFrame]

Creates two VastFrames (train/test), which can be used to evaluate a model. The intersection between the train and test set is empty only if you specify a unique order_by.

Parameters:

• test_size (float, optional) – Proportion of the test set compared to the training set.

• order_by (str | dict | list, optional) – List of the VastColumn used to sort the data, using asc order or a dictionary of all sorting methods. For example, to sort by “column1” ASC and “column2” DESC, write: {"column1": "asc", "column2": "desc"} Without this parameter, the seeded random number used to split the data into train and test cannot guarantee that no collision will occur. Using this parameter avoids the possibility of collisions.

• random_state (int, optional) – Integer used to seed the randomness.

Returns:

(train VastFrame, test VastFrame)

Return type:

tuple

Examples

For this example, we will use the Titanic dataset.

import vastorbit.datasets as vod

data = vod.load_titanic()

	123 pclass Integer	123 survived Integer	Abc name Varchar(164)	Abc sex Varchar(20)	123 age Double	123 sibsp Integer	123 parch Integer	Abc ticket Varchar(36)	123 fare Double	Abc cabin Varchar(30)	Abc embarked Varchar(20)	Abc boat Varchar(100)	123 body Integer	Abc home.dest Varchar(100)
1	3	1	McCormack, Mr. Thomas Joseph	male	[null]	0	0	367228	7.75	[null]	Q	[null]	[null]	[null]
2	3	1	McCoy, Miss. Agnes	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
3	3	1	McCoy, Miss. Alicia	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
4	3	1	McCoy, Mr. Bernard	male	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
5	3	1	McDermott, Miss. Brigdet Delia	female	[null]	0	0	330932	7.7875	[null]	Q	13	[null]	[null]
6	3	0	McEvoy, Mr. Michael	male	[null]	0	0	36568	15.5	[null]	Q	[null]	[null]	[null]
7	3	1	McGovern, Miss. Mary	female	[null]	0	0	330931	7.8792	[null]	Q	13	[null]	[null]
8	3	1	McGowan, Miss. Anna "Annie"	female	15.0	0	0	330923	8.0292	[null]	Q	[null]	[null]	[null]
9	3	0	McGowan, Miss. Katherine	female	35.0	0	0	9232	7.75	[null]	Q	[null]	[null]	[null]
10	3	0	McMahon, Mr. Martin	male	[null]	0	0	370372	7.75	[null]	Q	[null]	[null]	[null]
11	3	0	McNamee, Mr. Neal	male	24.0	1	0	376566	16.1	[null]	S	[null]	[null]	[null]
12	3	0	McNamee, Mrs. Neal (Eileen O'Leary)	female	19.0	1	0	376566	16.1	[null]	S	[null]	53	[null]
13	3	0	McNeill, Miss. Bridget	female	[null]	0	0	370368	7.75	[null]	Q	[null]	[null]	[null]
14	3	0	Meanwell, Miss. (Marion Ogden)	female	[null]	0	0	SOTON/O.Q. 392087	8.05	[null]	S	[null]	[null]	[null]
15	3	0	Meek, Mrs. Thomas (Annie Louise Rowley)	female	[null]	0	0	343095	8.05	[null]	S	[null]	[null]	[null]
16	3	0	Meo, Mr. Alfonzo	male	55.5	0	0	A.5. 11206	8.05	[null]	S	[null]	201	[null]
17	3	0	Mernagh, Mr. Robert	male	[null]	0	0	368703	7.75	[null]	Q	[null]	[null]	[null]
18	3	1	Midtsjo, Mr. Karl Albert	male	21.0	0	0	345501	7.775	[null]	S	15	[null]	[null]
19	3	0	Miles, Mr. Frank	male	[null]	0	0	359306	8.05	[null]	S	[null]	[null]	[null]
20	3	0	Mineff, Mr. Ivan	male	24.0	0	0	349233	7.8958	[null]	S	[null]	[null]	[null]
21	3	0	Minkoff, Mr. Lazar	male	21.0	0	0	349211	7.8958	[null]	S	[null]	[null]	[null]
22	3	0	Mionoff, Mr. Stoytcho	male	28.0	0	0	349207	7.8958	[null]	S	[null]	[null]	[null]
23	3	0	Mitkoff, Mr. Mito	male	[null]	0	0	349221	7.8958	[null]	S	[null]	[null]	[null]
24	3	1	Mockler, Miss. Helen Mary "Ellie"	female	[null]	0	0	330980	7.8792	[null]	Q	16	[null]	[null]
25	3	0	Moen, Mr. Sigurd Hansen	male	25.0	0	0	348123	7.65	F G73	S	[null]	309	[null]
26	3	1	Moor, Master. Meier	male	6.0	0	1	392096	12.475	E121	S	14	[null]	[null]
27	3	1	Moor, Mrs. (Beila)	female	27.0	0	1	392096	12.475	E121	S	14	[null]	[null]
28	3	0	Moore, Mr. Leonard Charles	male	[null]	0	0	A4. 54510	8.05	[null]	S	[null]	[null]	[null]
29	3	1	Moran, Miss. Bertha	female	[null]	1	0	371110	24.15	[null]	Q	16	[null]	[null]
30	3	0	Moran, Mr. Daniel J	male	[null]	1	0	371110	24.15	[null]	Q	[null]	[null]	[null]
31	3	0	Moran, Mr. James	male	[null]	0	0	330877	8.4583	[null]	Q	[null]	[null]	[null]
32	3	0	Morley, Mr. William	male	34.0	0	0	364506	8.05	[null]	S	[null]	[null]	[null]
33	3	0	Morrow, Mr. Thomas Rowan	male	[null]	0	0	372622	7.75	[null]	Q	[null]	[null]	[null]
34	3	1	Moss, Mr. Albert Johan	male	[null]	0	0	312991	7.775	[null]	S	B	[null]	[null]
35	3	1	Moubarek, Master. Gerios	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
36	3	1	Moubarek, Master. Halim Gonios ("William George")	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
37	3	1	Moubarek, Mrs. George (Omine "Amenia" Alexander)	female	[null]	0	2	2661	15.2458	[null]	C	C	[null]	[null]
38	3	1	Moussa, Mrs. (Mantoura Boulos)	female	[null]	0	0	2626	7.2292	[null]	C	[null]	[null]	[null]
39	3	0	Moutal, Mr. Rahamin Haim	male	[null]	0	0	374746	8.05	[null]	S	[null]	[null]	[null]
40	3	1	Mullens, Miss. Katherine "Katie"	female	[null]	0	0	35852	7.7333	[null]	Q	16	[null]	[null]
41	3	1	Mulvihill, Miss. Bertha E	female	24.0	0	0	382653	7.75	[null]	Q	15	[null]	[null]
42	3	0	Murdlin, Mr. Joseph	male	[null]	0	0	A./5. 3235	8.05	[null]	S	[null]	[null]	[null]
43	3	1	Murphy, Miss. Katherine "Kate"	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
44	3	1	Murphy, Miss. Margaret Jane	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
45	3	1	Murphy, Miss. Nora	female	[null]	0	0	36568	15.5	[null]	Q	16	[null]	[null]
46	3	0	Myhrman, Mr. Pehr Fabian Oliver Malkolm	male	18.0	0	0	347078	7.75	[null]	S	[null]	[null]	[null]
47	3	0	Naidenoff, Mr. Penko	male	22.0	0	0	349206	7.8958	[null]	S	[null]	[null]	[null]
48	3	1	Najib, Miss. Adele Kiamie "Jane"	female	15.0	0	0	2667	7.225	[null]	C	C	[null]	[null]
49	3	1	Nakid, Miss. Maria ("Mary")	female	1.0	0	2	2653	15.7417	[null]	C	C	[null]	[null]
50	3	1	Nakid, Mr. Sahid	male	20.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
51	3	1	Nakid, Mrs. Said (Waika "Mary" Mowad)	female	19.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
52	3	0	Nancarrow, Mr. William Henry	male	33.0	0	0	A./5. 3338	8.05	[null]	S	[null]	[null]	[null]
53	3	0	Nankoff, Mr. Minko	male	[null]	0	0	349218	7.8958	[null]	S	[null]	[null]	[null]
54	3	0	Nasr, Mr. Mustafa	male	[null]	0	0	2652	7.2292	[null]	C	[null]	[null]	[null]
55	3	0	Naughton, Miss. Hannah	female	[null]	0	0	365237	7.75	[null]	Q	[null]	[null]	[null]
56	3	0	Nenkoff, Mr. Christo	male	[null]	0	0	349234	7.8958	[null]	S	[null]	[null]	[null]
57	3	1	Nicola-Yarred, Master. Elias	male	12.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
58	3	1	Nicola-Yarred, Miss. Jamila	female	14.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
59	3	0	Nieminen, Miss. Manta Josefina	female	29.0	0	0	3101297	7.925	[null]	S	[null]	[null]	[null]
60	3	0	Niklasson, Mr. Samuel	male	28.0	0	0	363611	8.05	[null]	S	[null]	[null]	[null]
61	3	1	Nilsson, Miss. Berta Olivia	female	18.0	0	0	347066	7.775	[null]	S	D	[null]	[null]
62	3	1	Nilsson, Miss. Helmina Josefina	female	26.0	0	0	347470	7.8542	[null]	S	13	[null]	[null]
63	3	0	Nilsson, Mr. August Ferdinand	male	21.0	0	0	350410	7.8542	[null]	S	[null]	[null]	[null]
64	3	0	Nirva, Mr. Iisakki Antino Aijo	male	41.0	0	0	SOTON/O2 3101272	7.125	[null]	S	[null]	[null]	Finland Sudbury, ON
65	3	1	Niskanen, Mr. Juha	male	39.0	0	0	STON/O 2. 3101289	7.925	[null]	S	9	[null]	[null]
66	3	0	Nosworthy, Mr. Richard Cater	male	21.0	0	0	A/4. 39886	7.8	[null]	S	[null]	[null]	[null]
67	3	0	Novel, Mr. Mansouer	male	28.5	0	0	2697	7.2292	[null]	C	[null]	181	[null]
68	3	1	Nysten, Miss. Anna Sofia	female	22.0	0	0	347081	7.75	[null]	S	13	[null]	[null]
69	3	0	Nysveen, Mr. Johan Hansen	male	61.0	0	0	345364	6.2375	[null]	S	[null]	[null]	[null]
70	3	0	O'Brien, Mr. Thomas	male	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
71	3	0	O'Brien, Mr. Timothy	male	[null]	0	0	330979	7.8292	[null]	Q	[null]	[null]	[null]
72	3	1	O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)	female	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
73	3	0	O'Connell, Mr. Patrick D	male	[null]	0	0	334912	7.7333	[null]	Q	[null]	[null]	[null]
74	3	0	O'Connor, Mr. Maurice	male	[null]	0	0	371060	7.75	[null]	Q	[null]	[null]	[null]
75	3	0	O'Connor, Mr. Patrick	male	[null]	0	0	366713	7.75	[null]	Q	[null]	[null]	[null]
76	3	0	Odahl, Mr. Nils Martin	male	23.0	0	0	7267	9.225	[null]	S	[null]	[null]	[null]
77	3	0	O'Donoghue, Ms. Bridget	female	[null]	0	0	364856	7.75	[null]	Q	[null]	[null]	[null]
78	3	1	O'Driscoll, Miss. Bridget	female	[null]	0	0	14311	7.75	[null]	Q	D	[null]	[null]
79	3	1	O'Dwyer, Miss. Ellen "Nellie"	female	[null]	0	0	330959	7.8792	[null]	Q	[null]	[null]	[null]
80	3	1	Ohman, Miss. Velin	female	22.0	0	0	347085	7.775	[null]	S	C	[null]	[null]
81	3	1	O'Keefe, Mr. Patrick	male	[null]	0	0	368402	7.75	[null]	Q	B	[null]	[null]
82	3	1	O'Leary, Miss. Hanora "Norah"	female	[null]	0	0	330919	7.8292	[null]	Q	13	[null]	[null]
83	3	1	Olsen, Master. Artur Karl	male	9.0	0	1	C 17368	3.1708	[null]	S	13	[null]	[null]
84	3	0	Olsen, Mr. Henry Margido	male	28.0	0	0	C 4001	22.525	[null]	S	[null]	173	[null]
85	3	0	Olsen, Mr. Karl Siegwart Andreas	male	42.0	0	1	4579	8.4042	[null]	S	[null]	[null]	[null]
86	3	0	Olsen, Mr. Ole Martin	male	[null]	0	0	Fa 265302	7.3125	[null]	S	[null]	[null]	[null]
87	3	0	Olsson, Miss. Elina	female	31.0	0	0	350407	7.8542	[null]	S	[null]	[null]	[null]
88	3	0	Olsson, Mr. Nils Johan Goransson	male	28.0	0	0	347464	7.8542	[null]	S	[null]	[null]	[null]
89	3	1	Olsson, Mr. Oscar Wilhelm	male	32.0	0	0	347079	7.775	[null]	S	A	[null]	[null]
90	3	0	Olsvigen, Mr. Thor Anderson	male	20.0	0	0	6563	9.225	[null]	S	[null]	89	Oslo, Norway Cameron, WI
91	3	0	Oreskovic, Miss. Jelka	female	23.0	0	0	315085	8.6625	[null]	S	[null]	[null]	[null]
92	3	0	Oreskovic, Miss. Marija	female	20.0	0	0	315096	8.6625	[null]	S	[null]	[null]	[null]
93	3	0	Oreskovic, Mr. Luka	male	20.0	0	0	315094	8.6625	[null]	S	[null]	[null]	[null]
94	3	0	Osen, Mr. Olaf Elon	male	16.0	0	0	7534	9.2167	[null]	S	[null]	[null]	[null]
95	3	1	Osman, Mrs. Mara	female	31.0	0	0	349244	8.6833	[null]	S	[null]	[null]	[null]
96	3	0	O'Sullivan, Miss. Bridget Mary	female	[null]	0	0	330909	7.6292	[null]	Q	[null]	[null]	[null]
97	3	0	Palsson, Master. Gosta Leonard	male	2.0	3	1	349909	21.075	[null]	S	[null]	4	[null]
98	3	0	Palsson, Master. Paul Folke	male	6.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
99	3	0	Palsson, Miss. Stina Viola	female	3.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
100	3	0	Palsson, Miss. Torborg Danira	female	8.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]

Rows: 1-100 | Columns: 14

Note

vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.

Let us check the size of the dataset:

data.shape()

Now, we can easily divide the dataset into training and testing subsets using the VastFrame.train_test_split() method.

train, test = data.train_test_split(test_size = 0.2)

We can confirm the size of the train set:

train.shape()

And test set:

test.shape()

Warning

This function utilizes seeded random values to ensure reproducibility, requiring an index to prevent collisions. It is highly recommended to save the results as tables using the VastFrame.to_db() method to expedite the process and ensure consistent results.

See also

VastFrame.sample() : Downsamples the input VastFrame.