Features Engineering

While preparing our data, we need to think constantly about the most suitable features we can use to achieve our overall goals.

Features engineering makes use of many techniques - too many to go over in this short lesson. We’ll focus on the most popular ones.

Customized Features Engineering

To build a customized feature, you can use the eval() method of the VastFrame. Let’s look at an example with the well-known titanic dataset.

import vastorbit as vo
from vastorbit.datasets import load_titanic

titanic = load_titanic()
titanic.head(100)

	123 pclass Integer 100%	123 survived Integer 100%	Abc name Varchar(164) 100%	Abc sex Varchar(20) 100%	123 age Double 79%	123 sibsp Integer 100%	123 parch Integer 100%	Abc ticket Varchar(36) 100%	123 fare Double 99%	Abc cabin Varchar(30) 22%	Abc embarked Varchar(20) 99%	Abc boat Varchar(100) 37%	123 body Integer 9%	Abc home.dest Varchar(100) 56%
1	3	1	McCormack, Mr. Thomas Joseph	male	[null]	0	0	367228	7.75	[null]	Q	[null]	[null]	[null]
2	3	1	McCoy, Miss. Agnes	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
3	3	1	McCoy, Miss. Alicia	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
4	3	1	McCoy, Mr. Bernard	male	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
5	3	1	McDermott, Miss. Brigdet Delia	female	[null]	0	0	330932	7.7875	[null]	Q	13	[null]	[null]
6	3	0	McEvoy, Mr. Michael	male	[null]	0	0	36568	15.5	[null]	Q	[null]	[null]	[null]
7	3	1	McGovern, Miss. Mary	female	[null]	0	0	330931	7.8792	[null]	Q	13	[null]	[null]
8	3	1	McGowan, Miss. Anna "Annie"	female	15.0	0	0	330923	8.0292	[null]	Q	[null]	[null]	[null]
9	3	0	McGowan, Miss. Katherine	female	35.0	0	0	9232	7.75	[null]	Q	[null]	[null]	[null]
10	3	0	McMahon, Mr. Martin	male	[null]	0	0	370372	7.75	[null]	Q	[null]	[null]	[null]
11	3	0	McNamee, Mr. Neal	male	24.0	1	0	376566	16.1	[null]	S	[null]	[null]	[null]
12	3	0	McNamee, Mrs. Neal (Eileen O'Leary)	female	19.0	1	0	376566	16.1	[null]	S	[null]	53	[null]
13	3	0	McNeill, Miss. Bridget	female	[null]	0	0	370368	7.75	[null]	Q	[null]	[null]	[null]
14	3	0	Meanwell, Miss. (Marion Ogden)	female	[null]	0	0	SOTON/O.Q. 392087	8.05	[null]	S	[null]	[null]	[null]
15	3	0	Meek, Mrs. Thomas (Annie Louise Rowley)	female	[null]	0	0	343095	8.05	[null]	S	[null]	[null]	[null]
16	3	0	Meo, Mr. Alfonzo	male	55.5	0	0	A.5. 11206	8.05	[null]	S	[null]	201	[null]
17	3	0	Mernagh, Mr. Robert	male	[null]	0	0	368703	7.75	[null]	Q	[null]	[null]	[null]
18	3	1	Midtsjo, Mr. Karl Albert	male	21.0	0	0	345501	7.775	[null]	S	15	[null]	[null]
19	3	0	Miles, Mr. Frank	male	[null]	0	0	359306	8.05	[null]	S	[null]	[null]	[null]
20	3	0	Mineff, Mr. Ivan	male	24.0	0	0	349233	7.8958	[null]	S	[null]	[null]	[null]
21	3	0	Minkoff, Mr. Lazar	male	21.0	0	0	349211	7.8958	[null]	S	[null]	[null]	[null]
22	3	0	Mionoff, Mr. Stoytcho	male	28.0	0	0	349207	7.8958	[null]	S	[null]	[null]	[null]
23	3	0	Mitkoff, Mr. Mito	male	[null]	0	0	349221	7.8958	[null]	S	[null]	[null]	[null]
24	3	1	Mockler, Miss. Helen Mary "Ellie"	female	[null]	0	0	330980	7.8792	[null]	Q	16	[null]	[null]
25	3	0	Moen, Mr. Sigurd Hansen	male	25.0	0	0	348123	7.65	F G73	S	[null]	309	[null]
26	3	1	Moor, Master. Meier	male	6.0	0	1	392096	12.475	E121	S	14	[null]	[null]
27	3	1	Moor, Mrs. (Beila)	female	27.0	0	1	392096	12.475	E121	S	14	[null]	[null]
28	3	0	Moore, Mr. Leonard Charles	male	[null]	0	0	A4. 54510	8.05	[null]	S	[null]	[null]	[null]
29	3	1	Moran, Miss. Bertha	female	[null]	1	0	371110	24.15	[null]	Q	16	[null]	[null]
30	3	0	Moran, Mr. Daniel J	male	[null]	1	0	371110	24.15	[null]	Q	[null]	[null]	[null]
31	3	0	Moran, Mr. James	male	[null]	0	0	330877	8.4583	[null]	Q	[null]	[null]	[null]
32	3	0	Morley, Mr. William	male	34.0	0	0	364506	8.05	[null]	S	[null]	[null]	[null]
33	3	0	Morrow, Mr. Thomas Rowan	male	[null]	0	0	372622	7.75	[null]	Q	[null]	[null]	[null]
34	3	1	Moss, Mr. Albert Johan	male	[null]	0	0	312991	7.775	[null]	S	B	[null]	[null]
35	3	1	Moubarek, Master. Gerios	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
36	3	1	Moubarek, Master. Halim Gonios ("William George")	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
37	3	1	Moubarek, Mrs. George (Omine "Amenia" Alexander)	female	[null]	0	2	2661	15.2458	[null]	C	C	[null]	[null]
38	3	1	Moussa, Mrs. (Mantoura Boulos)	female	[null]	0	0	2626	7.2292	[null]	C	[null]	[null]	[null]
39	3	0	Moutal, Mr. Rahamin Haim	male	[null]	0	0	374746	8.05	[null]	S	[null]	[null]	[null]
40	3	1	Mullens, Miss. Katherine "Katie"	female	[null]	0	0	35852	7.7333	[null]	Q	16	[null]	[null]
41	3	1	Mulvihill, Miss. Bertha E	female	24.0	0	0	382653	7.75	[null]	Q	15	[null]	[null]
42	3	0	Murdlin, Mr. Joseph	male	[null]	0	0	A./5. 3235	8.05	[null]	S	[null]	[null]	[null]
43	3	1	Murphy, Miss. Katherine "Kate"	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
44	3	1	Murphy, Miss. Margaret Jane	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
45	3	1	Murphy, Miss. Nora	female	[null]	0	0	36568	15.5	[null]	Q	16	[null]	[null]
46	3	0	Myhrman, Mr. Pehr Fabian Oliver Malkolm	male	18.0	0	0	347078	7.75	[null]	S	[null]	[null]	[null]
47	3	0	Naidenoff, Mr. Penko	male	22.0	0	0	349206	7.8958	[null]	S	[null]	[null]	[null]
48	3	1	Najib, Miss. Adele Kiamie "Jane"	female	15.0	0	0	2667	7.225	[null]	C	C	[null]	[null]
49	3	1	Nakid, Miss. Maria ("Mary")	female	1.0	0	2	2653	15.7417	[null]	C	C	[null]	[null]
50	3	1	Nakid, Mr. Sahid	male	20.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
51	3	1	Nakid, Mrs. Said (Waika "Mary" Mowad)	female	19.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
52	3	0	Nancarrow, Mr. William Henry	male	33.0	0	0	A./5. 3338	8.05	[null]	S	[null]	[null]	[null]
53	3	0	Nankoff, Mr. Minko	male	[null]	0	0	349218	7.8958	[null]	S	[null]	[null]	[null]
54	3	0	Nasr, Mr. Mustafa	male	[null]	0	0	2652	7.2292	[null]	C	[null]	[null]	[null]
55	3	0	Naughton, Miss. Hannah	female	[null]	0	0	365237	7.75	[null]	Q	[null]	[null]	[null]
56	3	0	Nenkoff, Mr. Christo	male	[null]	0	0	349234	7.8958	[null]	S	[null]	[null]	[null]
57	3	1	Nicola-Yarred, Master. Elias	male	12.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
58	3	1	Nicola-Yarred, Miss. Jamila	female	14.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
59	3	0	Nieminen, Miss. Manta Josefina	female	29.0	0	0	3101297	7.925	[null]	S	[null]	[null]	[null]
60	3	0	Niklasson, Mr. Samuel	male	28.0	0	0	363611	8.05	[null]	S	[null]	[null]	[null]
61	3	1	Nilsson, Miss. Berta Olivia	female	18.0	0	0	347066	7.775	[null]	S	D	[null]	[null]
62	3	1	Nilsson, Miss. Helmina Josefina	female	26.0	0	0	347470	7.8542	[null]	S	13	[null]	[null]
63	3	0	Nilsson, Mr. August Ferdinand	male	21.0	0	0	350410	7.8542	[null]	S	[null]	[null]	[null]
64	3	0	Nirva, Mr. Iisakki Antino Aijo	male	41.0	0	0	SOTON/O2 3101272	7.125	[null]	S	[null]	[null]	Finland Sudbury, ON
65	3	1	Niskanen, Mr. Juha	male	39.0	0	0	STON/O 2. 3101289	7.925	[null]	S	9	[null]	[null]
66	3	0	Nosworthy, Mr. Richard Cater	male	21.0	0	0	A/4. 39886	7.8	[null]	S	[null]	[null]	[null]
67	3	0	Novel, Mr. Mansouer	male	28.5	0	0	2697	7.2292	[null]	C	[null]	181	[null]
68	3	1	Nysten, Miss. Anna Sofia	female	22.0	0	0	347081	7.75	[null]	S	13	[null]	[null]
69	3	0	Nysveen, Mr. Johan Hansen	male	61.0	0	0	345364	6.2375	[null]	S	[null]	[null]	[null]
70	3	0	O'Brien, Mr. Thomas	male	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
71	3	0	O'Brien, Mr. Timothy	male	[null]	0	0	330979	7.8292	[null]	Q	[null]	[null]	[null]
72	3	1	O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)	female	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
73	3	0	O'Connell, Mr. Patrick D	male	[null]	0	0	334912	7.7333	[null]	Q	[null]	[null]	[null]
74	3	0	O'Connor, Mr. Maurice	male	[null]	0	0	371060	7.75	[null]	Q	[null]	[null]	[null]
75	3	0	O'Connor, Mr. Patrick	male	[null]	0	0	366713	7.75	[null]	Q	[null]	[null]	[null]
76	3	0	Odahl, Mr. Nils Martin	male	23.0	0	0	7267	9.225	[null]	S	[null]	[null]	[null]
77	3	0	O'Donoghue, Ms. Bridget	female	[null]	0	0	364856	7.75	[null]	Q	[null]	[null]	[null]
78	3	1	O'Driscoll, Miss. Bridget	female	[null]	0	0	14311	7.75	[null]	Q	D	[null]	[null]
79	3	1	O'Dwyer, Miss. Ellen "Nellie"	female	[null]	0	0	330959	7.8792	[null]	Q	[null]	[null]	[null]
80	3	1	Ohman, Miss. Velin	female	22.0	0	0	347085	7.775	[null]	S	C	[null]	[null]
81	3	1	O'Keefe, Mr. Patrick	male	[null]	0	0	368402	7.75	[null]	Q	B	[null]	[null]
82	3	1	O'Leary, Miss. Hanora "Norah"	female	[null]	0	0	330919	7.8292	[null]	Q	13	[null]	[null]
83	3	1	Olsen, Master. Artur Karl	male	9.0	0	1	C 17368	3.1708	[null]	S	13	[null]	[null]
84	3	0	Olsen, Mr. Henry Margido	male	28.0	0	0	C 4001	22.525	[null]	S	[null]	173	[null]
85	3	0	Olsen, Mr. Karl Siegwart Andreas	male	42.0	0	1	4579	8.4042	[null]	S	[null]	[null]	[null]
86	3	0	Olsen, Mr. Ole Martin	male	[null]	0	0	Fa 265302	7.3125	[null]	S	[null]	[null]	[null]
87	3	0	Olsson, Miss. Elina	female	31.0	0	0	350407	7.8542	[null]	S	[null]	[null]	[null]
88	3	0	Olsson, Mr. Nils Johan Goransson	male	28.0	0	0	347464	7.8542	[null]	S	[null]	[null]	[null]
89	3	1	Olsson, Mr. Oscar Wilhelm	male	32.0	0	0	347079	7.775	[null]	S	A	[null]	[null]
90	3	0	Olsvigen, Mr. Thor Anderson	male	20.0	0	0	6563	9.225	[null]	S	[null]	89	Oslo, Norway Cameron, WI
91	3	0	Oreskovic, Miss. Jelka	female	23.0	0	0	315085	8.6625	[null]	S	[null]	[null]	[null]
92	3	0	Oreskovic, Miss. Marija	female	20.0	0	0	315096	8.6625	[null]	S	[null]	[null]	[null]
93	3	0	Oreskovic, Mr. Luka	male	20.0	0	0	315094	8.6625	[null]	S	[null]	[null]	[null]
94	3	0	Osen, Mr. Olaf Elon	male	16.0	0	0	7534	9.2167	[null]	S	[null]	[null]	[null]
95	3	1	Osman, Mrs. Mara	female	31.0	0	0	349244	8.6833	[null]	S	[null]	[null]	[null]
96	3	0	O'Sullivan, Miss. Bridget Mary	female	[null]	0	0	330909	7.6292	[null]	Q	[null]	[null]	[null]
97	3	0	Palsson, Master. Gosta Leonard	male	2.0	3	1	349909	21.075	[null]	S	[null]	4	[null]
98	3	0	Palsson, Master. Paul Folke	male	6.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
99	3	0	Palsson, Miss. Stina Viola	female	3.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
100	3	0	Palsson, Miss. Torborg Danira	female	8.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]

The feature parch corresponds to the number of parents and children on-board. The feature sibsp corresponds to the number of siblings and spouses on-board. We can create the feature family_size which is equal to parch + sibsp + 1.

titanic["family_size"] = titanic["parch"] + titanic["sibsp"] + 1
titanic.select(["parch", "sibsp", "family_size"])

	123 parch Integer 100%	123 sibsp Integer 100%	123 family_size Integer 100%
1	0	0	1
2	0	2	3
3	0	2	3
4	0	2	3
5	0	0	1
6	0	0	1
7	0	0	1
8	0	0	1
9	0	0	1
10	0	0	1
11	0	1	2
12	0	1	2
13	0	0	1
14	0	0	1
15	0	0	1
16	0	0	1
17	0	0	1
18	0	0	1
19	0	0	1
20	0	0	1

When using the eval() method, you can enter any SQL expression and vastorbit will evaluate it!

Regular Expressions

To compute features using regular expressions, we’ll use the regexp() method.

help(vo.VastFrame.regexp)

Consider the following example: notice that passenger names include their title.

titanic["name"]

	Abc name Varchar(164)
1	McCormack, Mr. Thomas Joseph
2	McCoy, Miss. Agnes
3	McCoy, Miss. Alicia
4	McCoy, Mr. Bernard
5	McDermott, Miss. Brigdet Delia
6	McEvoy, Mr. Michael
7	McGovern, Miss. Mary
8	McGowan, Miss. Anna "Annie"
9	McGowan, Miss. Katherine
10	McMahon, Mr. Martin
11	McNamee, Mr. Neal
12	McNamee, Mrs. Neal (Eileen O'Leary)
13	McNeill, Miss. Bridget
14	Meanwell, Miss. (Marion Ogden)
15	Meek, Mrs. Thomas (Annie Louise Rowley)
16	Meo, Mr. Alfonzo
17	Mernagh, Mr. Robert
18	Midtsjo, Mr. Karl Albert
19	Miles, Mr. Frank
20	Mineff, Mr. Ivan

Let’s extract the title using regular expressions.

titanic.regexp(
    column = "name",
    name = "title",
    pattern = " ([A-Za-z])+\\.",
    method = "substr",
)
titanic.select(["name", "title"])

	Abc name Varchar(164) 100%	Abc title Varchar(164) 100%
1	McCormack, Mr. Thomas Joseph	Mr.
2	McCoy, Miss. Agnes	Miss.
3	McCoy, Miss. Alicia	Miss.
4	McCoy, Mr. Bernard	Mr.
5	McDermott, Miss. Brigdet Delia	Miss.
6	McEvoy, Mr. Michael	Mr.
7	McGovern, Miss. Mary	Miss.
8	McGowan, Miss. Anna "Annie"	Miss.
9	McGowan, Miss. Katherine	Miss.
10	McMahon, Mr. Martin	Mr.
11	McNamee, Mr. Neal	Mr.
12	McNamee, Mrs. Neal (Eileen O'Leary)	Mrs.
13	McNeill, Miss. Bridget	Miss.
14	Meanwell, Miss. (Marion Ogden)	Miss.
15	Meek, Mrs. Thomas (Annie Louise Rowley)	Mrs.
16	Meo, Mr. Alfonzo	Mr.
17	Mernagh, Mr. Robert	Mr.
18	Midtsjo, Mr. Karl Albert	Mr.
19	Miles, Mr. Frank	Mr.
20	Mineff, Mr. Ivan	Mr.

Advanced Analytical Functions

The analytic() method contains the many advanced analytical functions in vastorbit.

help(vo.VastFrame.analytic)

To demonstrate some of these techniques, let’s use the amazon dataset and perform some computations.

from vastorbit.datasets import load_amazon

amazon = load_amazon()
amazon.head(100)

	📅 date Date 100%	Abc state Varchar(32) 100%	123 number Integer 100%
1	2007-01-01	ESPÍRITO SANTO	0
2	2007-02-01	ESPÍRITO SANTO	3
3	2007-03-01	ESPÍRITO SANTO	9
4	2007-04-01	ESPÍRITO SANTO	2
5	2007-05-01	ESPÍRITO SANTO	1
6	2007-06-01	ESPÍRITO SANTO	4
7	2007-07-01	ESPÍRITO SANTO	13
8	2007-08-01	ESPÍRITO SANTO	82
9	2007-09-01	ESPÍRITO SANTO	63
10	2007-10-01	ESPÍRITO SANTO	79
11	2007-11-01	ESPÍRITO SANTO	112
12	2007-12-01	ESPÍRITO SANTO	14
13	2007-01-01	GOIÁS	28
14	2007-02-01	GOIÁS	13
15	2007-03-01	GOIÁS	49
16	2007-04-01	GOIÁS	17
17	2007-05-01	GOIÁS	94
18	2007-06-01	GOIÁS	186
19	2007-07-01	GOIÁS	399
20	2007-08-01	GOIÁS	2382
21	2007-09-01	GOIÁS	3745
22	2007-10-01	GOIÁS	1195
23	2007-11-01	GOIÁS	79
24	2007-12-01	GOIÁS	16
25	2007-01-01	MARANHÃO	428
26	2007-02-01	MARANHÃO	13
27	2007-03-01	MARANHÃO	52
28	2007-04-01	MARANHÃO	5
29	2007-05-01	MARANHÃO	73
30	2007-06-01	MARANHÃO	515
31	2007-07-01	MARANHÃO	1205
32	2007-08-01	MARANHÃO	8409
33	2007-09-01	MARANHÃO	8524
34	2007-10-01	MARANHÃO	4772
35	2007-11-01	MARANHÃO	2297
36	2007-12-01	MARANHÃO	800
37	2007-01-01	MATO GROSSO	19
38	2007-02-01	MATO GROSSO	111
39	2007-03-01	MATO GROSSO	137
40	2007-04-01	MATO GROSSO	46
41	2007-05-01	MATO GROSSO	62
42	2007-06-01	MATO GROSSO	141
43	2007-07-01	MATO GROSSO	197
44	2007-08-01	MATO GROSSO	1823
45	2007-09-01	MATO GROSSO	4446
46	2007-10-01	MATO GROSSO	668
47	2007-11-01	MATO GROSSO	82
48	2007-12-01	MATO GROSSO	15
49	2007-01-01	MATO GROSSO DO SUL	476
50	2007-02-01	MATO GROSSO DO SUL	213
51	2007-03-01	MATO GROSSO DO SUL	476
52	2007-04-01	MATO GROSSO DO SUL	79
53	2007-05-01	MATO GROSSO DO SUL	359
54	2007-06-01	MATO GROSSO DO SUL	1339
55	2007-07-01	MATO GROSSO DO SUL	1790
56	2007-08-01	MATO GROSSO DO SUL	14453
57	2007-09-01	MATO GROSSO DO SUL	25963
58	2007-10-01	MATO GROSSO DO SUL	4890
59	2007-11-01	MATO GROSSO DO SUL	299
60	2007-12-01	MATO GROSSO DO SUL	81
61	2007-01-01	MINAS GERAIS	91
62	2007-02-01	MINAS GERAIS	71
63	2007-03-01	MINAS GERAIS	155
64	2007-04-01	MINAS GERAIS	35
65	2007-05-01	MINAS GERAIS	124
66	2007-06-01	MINAS GERAIS	184
67	2007-07-01	MINAS GERAIS	446
68	2007-08-01	MINAS GERAIS	2975
69	2007-09-01	MINAS GERAIS	5088
70	2007-10-01	MINAS GERAIS	3467
71	2007-11-01	MINAS GERAIS	1305
72	2007-12-01	MINAS GERAIS	95
73	2007-01-01	PARANÁ	11
74	2007-02-01	PARANÁ	32
75	2007-03-01	PARANÁ	54
76	2007-04-01	PARANÁ	14
77	2007-05-01	PARANÁ	14
78	2007-06-01	PARANÁ	57
79	2007-07-01	PARANÁ	60
80	2007-08-01	PARANÁ	580
81	2007-09-01	PARANÁ	960
82	2007-10-01	PARANÁ	194
83	2007-11-01	PARANÁ	59
84	2007-12-01	PARANÁ	22
85	2007-01-01	PARAÍBA	69
86	2007-02-01	PARAÍBA	7
87	2007-03-01	PARAÍBA	1
88	2007-04-01	PARAÍBA	0
89	2007-05-01	PARAÍBA	0
90	2007-06-01	PARAÍBA	0
91	2007-07-01	PARAÍBA	1
92	2007-08-01	PARAÍBA	8
93	2007-09-01	PARAÍBA	22
94	2007-10-01	PARAÍBA	111
95	2007-11-01	PARAÍBA	256
96	2007-12-01	PARAÍBA	133
97	2007-01-01	PARÁ	467
98	2007-02-01	PARÁ	23
99	2007-03-01	PARÁ	6
100	2007-04-01	PARÁ	0

For each state, let’s compute the previous number of forest fires.

amazon.analytic(
    name = "previous_number",
    func = "lag",
    columns = "number",
    by = "state",
    order_by = {"date": "asc"},
)

	📅 date Date 100%	Abc state Varchar(32) 100%	123 number Integer 100%	123 previous_number Integer 99%
1	1998-01-01	TOCANTINS	0	[null]
2	1998-02-01	TOCANTINS	0	0
3	1998-03-01	TOCANTINS	0	0
4	1998-04-01	TOCANTINS	0	0
5	1998-05-01	TOCANTINS	0	0
6	1998-06-01	TOCANTINS	252	0
7	1998-07-01	TOCANTINS	640	252
8	1998-08-01	TOCANTINS	3747	640
9	1998-09-01	TOCANTINS	5149	3747
10	1998-10-01	TOCANTINS	1738	5149
11	1998-11-01	TOCANTINS	1	1738
12	1998-12-01	TOCANTINS	9	1
13	1999-01-01	TOCANTINS	36	9
14	1999-02-01	TOCANTINS	1	36
15	1999-03-01	TOCANTINS	1	1
16	1999-04-01	TOCANTINS	9	1
17	1999-05-01	TOCANTINS	24	9
18	1999-06-01	TOCANTINS	113	24
19	1999-07-01	TOCANTINS	373	113
20	1999-08-01	TOCANTINS	1284	373

Moving Windows

Moving windows are powerful features. Moving windows are managed by the rolling() method in vastorbit.

help(vo.VastFrame.rolling)

Let’s look at forest fires for each state three months preceding two months following the examined period.

amazon.rolling(
    name = "number_3mp_2mf",
    func = "sum",
    window = ("- 3 months", "2 months"),
    columns = "number",
    by = "state",
    order_by = {"date": "asc"},
)

	📅 date Date 100%	Abc state Varchar(32) 100%	123 number Integer 100%	123 previous_number Integer 99%	123 number_3mp_2mf Bigint 100%
1	1998-01-01	MINAS GERAIS	0	[null]	0
2	1998-02-01	MINAS GERAIS	0	0	0
3	1998-03-01	MINAS GERAIS	0	0	0
4	1998-04-01	MINAS GERAIS	0	0	70
5	1998-05-01	MINAS GERAIS	0	0	302
6	1998-06-01	MINAS GERAIS	70	0	1177
7	1998-07-01	MINAS GERAIS	232	70	3158
8	1998-08-01	MINAS GERAIS	875	232	4251
9	1998-09-01	MINAS GERAIS	1981	875	4283
10	1998-10-01	MINAS GERAIS	1093	1981	4234
11	1998-11-01	MINAS GERAIS	32	1093	4038
12	1998-12-01	MINAS GERAIS	21	32	3275
13	1999-01-01	MINAS GERAIS	36	21	1307
14	1999-02-01	MINAS GERAIS	112	36	241
15	1999-03-01	MINAS GERAIS	13	112	260
16	1999-04-01	MINAS GERAIS	27	13	371
17	1999-05-01	MINAS GERAIS	51	27	653
18	1999-06-01	MINAS GERAIS	132	51	1731
19	1999-07-01	MINAS GERAIS	318	132	4404
20	1999-08-01	MINAS GERAIS	1190	318	5740

Moving windows give us infinite possibilities for creating new features.

After we’ve finished preparing our data, our next task is to create a machine learning model.