Amazon¶

This example uses the amazon dataset to predict the number of forest fires in Brazil.

date: Date of the record
number: Number of forest fires
state: State in Brazil

We’ll follow the data science cycle (Data Exploration - Data Preparation - Data Modeling - Model Evaluation - Model Deployment) to solve this problem, and we’ll do it without ever loading our data into memory.

Initialization¶

This example uses the following version of vastorbit:

import vastorbit as vo

vo.__version__

Connect to VAST. This example uses an existing connection called VASTDSN. For details on how to create a connection, see the Connection tutorial. You can skip the below cell if you already have an established connection.

vo.connect("VASTDSN")

Let’s create a VastFrame of the dataset.

from vastorbit.datasets import load_amazon

amazon = load_amazon()
amazon.head(5)

	📅 date Date 100%	Abc state Varchar(32) 100%
1	1998-01-01	ACRE
2	1998-02-01	ACRE
3	1998-03-01	ACRE
4	1998-04-01	ACRE
5	1998-05-01	ACRE

Data Exploration and Preparation¶

We can explore our data by displaying descriptive statistics of all the columns.

amazon.describe(method = "categorical", unique = True)

	dtype	count	top	top_percent	unique
"date"	date	6454	2017-01-01	0.434	239.0
"state"	varchar(32)	6454	ALAGOAS	3.719	27.0
"number"	integer	6454	0	9.56	1480.0

Using the describe() method, we can see that our data ranges from the beginning of 1998 to the end of 2017.

amazon["date"].describe()

	value
name	"date"
dtype	date
count	6454
min	1998-01-01
max	2017-11-01

Brazil has dry and rainy seasons. Knowing this, we would expect that the frequency of forest fires vary between seasons. Let’s confirm our hypothesis using an autocorrelation plot with 48 lags (4 years).

amazon.acf(
    column = "number",
    ts = "date",
    by = ["state"],
    p = 48,
)

The process is not stationary. Let’s use a Dickey-Fuller test to confirm our hypothesis.

from vastorbit.machine_learning.model_selection.statistical_tests import adfuller

adfuller(
    amazon,
    ts = "date",
    column = "number",
    by = ["state"],
    p = 48,
)

	value
ADF Test Statistic	-1.7878804788753853
p_value	0.07385486605728553
# Lags used	48
# Observations Used	5131
Critical Value (1%)	-3.43
Critical Value (2.5%)	-3.12
Critical Value (5%)	-2.86
Critical Value (10%)	-2.57
Stationarity (alpha = 1%)	False

The effects of each season seem pretty clear. We can see this graphically using the cumulative sum of the number of forest fires partitioned by states. If our hypothesis is correct, we should see staircase functions.

amazon.cumsum(
    "number",
    by = ["state"],
    order_by = ["date"],
    name = "cum_sum",
)
amazon["cum_sum"].plot(
    ts = "date",
    by = "state",
)

We can clearly observe the seasonality within each state, which contributes to an overall global seasonality. Let’s plot the total number of forest fires to illustrate this more clearly.

import vastorbit.sql.functions as fun

amazon = amazon.groupby(
    ["date"],
    [
        fun.sum(amazon["number"])._as("number"),
    ],
)
amazon["number"].plot(ts = "date")

Although it would be preferable to use seasonal decomposition and predict the residuals, let’s build an AR model on the data.

Machine Learning¶

Since the seasonality occurs monthly, we set p = 12. Let’s proceed with building the model.

from vastorbit.machine_learning.vast import AR

model = AR(
    p = 12,
    missing = "drop",
)
model.fit(
    amazon,
    y = "number",
    ts = "date",
)
model.regression_report(start = 50)

	value
explained_variance	0.7818945763658622
max_error	31495.699489466093
median_absolute_error	2126.9172
mean_absolute_error	5800.795592801477
mean_squared_error	120263045.71588834
root_mean_squared_error	10966.450917041864
r2	0.7334789259027081
r2_adj	0.7001637916405467
aic	192.90906235061817
bic	190.6570896794634

Our model is quite good. Let’s look at our predictions.

model.plot(
    vdf = amazon,
    ts = "date",
    y = "number",
    npredictions = 40,
    method = "auto",
)

The plot shows that our model has successfully captured the seasonality present in the data. However, to improve the model, we should remove the seasonality and focus on predicting the residuals directly. The current model is not entirely stable and requires further adjustments.

Conclusion¶

We’ve solved our problem in a pandas-like way, all without ever loading data into memory!