vastorbit.machine_learning.model_selection.statistical_tests.tsa.durbin_watson

vastorbit.machine_learning.model_selection.statistical_tests.tsa.durbin_watson(input_relation: Annotated[str | VastFrame, ''], eps: str, ts: str, by: Annotated[str | list[str], 'STRING representing one column or a list of columns'] | None = None) → float

Durbin Watson test (residuals autocorrelation).

Parameters:

• input_relation (SQLRelation) – Input relation.

• eps (str) – Input residual VastColumn.

• ts (str) – VastColumn used as timeline to order the data. It can be a numerical or date-like type (date, datetime, timestamp…) VastColumn.

• by (SQLColumns, optional) – VastColumns used in the partition.

Returns:

Durbin Watson statistic.

Return type:

float

Examples

Initialization

Let’s try this test on a dummy dataset that has the following elements:

• A value of interest that has noise related to time

• Time-stamp data

Before we begin we can import the necessary libraries:

import vastorbit as vo
import numpy as np

Data

Now we can create the dummy dataset:

# Initialization
N = 50 # Number of Rows
days = list(range(N))
y_val = [2 * x + np.random.normal(scale = 4 * x * x) for x in days]

# VastFrame
vdf = vo.VastFrame(
    {
        "day": days,
        "y1": y_val,
    }
)

Model Fitting

Next, we can fit a Linear Model. To do that we need to first import the model and intialize:

from vastorbit.machine_learning.vast.linear_model import LinearRegression

model = LinearRegression()

Next we can fit the model:

model.fit(vdf, X = "day", y = "y1")

We can create a column in the VastFrame that has the predictions:

model.predict(vdf, X = "day", name = "y_pred")

	123 day Integer	123 y1 Decimal(20, 15)	123 y_pred Decimal(29, 16)
1	14	-1786.3496401756818	24.95078885364371
2	32	-3392.6819214570687	-11.621391402404718
3	17	-422.5358663558964	18.855425477635638
4	26	-3010.3693930783616	0.5693353496114244
5	45	-18647.41757952407	-38.034632698439694
6	22	324.41524071051145	8.696486517622187
7	8	213.45229399100617	37.141515605659855
8	35	-7585.859000880063	-17.716754778412792
9	36	-1444.9615939848813	-19.74854257041548
10	42	1108.3970083845666	-31.939269322431624
11	27	3706.6768336821206	-1.4624524423912661
12	33	3512.2281729342913	-13.653179194407409
13	43	4264.268936826552	-33.971057114434316
14	29	-1257.1959336519483	-5.526028026396648
15	7	242.1657106813575	39.17330339766254
16	41	7505.971897571791	-29.907481530428935
17	44	-1454.8836658618152	-36.002844906437005
18	46	16906.09270757043	-40.06642049044239
19	9	-180.49042409760543	35.109727813657166
20	19	-860.077161287886	14.791849893630259
21	28	-879.3176730588347	-3.4942402343939567
22	25	652.6135785510917	2.601123141614115
23	6	-34.227932360927774	41.20509118966524
24	39	1740.5175715195064	-25.843905946423554
25	49	2653.6882423137154	-46.161783866450456
26	23	1640.144774843036	6.6646987256194965
27	30	5556.850333818485	-7.557815818399338
28	11	288.0590927489465	31.046152229651785
29	24	-365.3976850031337	4.632910933616806
30	47	-6963.54718764456	-42.09820828244508
31	2	20.19638047680634	49.332242357676
32	34	-2182.7779910863865	-15.6849669864101
33	48	-5370.178926362465	-44.12999607444777
34	18	-1677.5309593739482	16.82363768563295
35	38	5251.8320231696935	-23.81211815442086
36	31	2255.7910254483477	-9.589603610402028
37	20	168.96381120531984	12.760062101627568
38	37	2545.763654427294	-21.780330362418173
39	40	-4338.557601209795	-27.875693738426243
40	21	2290.392387914692	10.728274309624878
41	5	-82.42640531487099	43.23687898166793
42	16	1633.591040520693	20.88721326963833
43	1	6.313868374527412	51.36403014967869
44	15	-719.3929999871947	22.919001061641023
45	3	-16.449171022659606	47.300454565673306
46	13	-1259.3400506314322	26.982576645646404
47	4	-103.6853053998591	45.26866677367062
48	10	-130.44419858634137	33.07794002165448
49	12	-141.43946840631904	29.014364437649093
50	0	0.0	53.39581794168138

Rows: 1-50 | Columns: 3

Then we can calculate the residuals i.e. eps:

vdf["eps"] = vdf["y1"] - vdf["y_pred"]

We can plot the residuals to see the trend:

vdf.scatter(["day", "eps"])

Test

Now we can apply the Durbin Watson Test:

from vastorbit.machine_learning.model_selection.statistical_tests import durbin_watson

durbin_watson(input_relation = vdf, ts = "day", eps = "eps")

We can see that the Durbin-Watson statistic is not equal to 2. This shows the presence of autocorrelation.

Note

The Durbin-Watson statistic values can be interpretted as such:

Approximately 2: No significant autocorrelation.

Less than 2: Positive autocorrelation (residuals are correlated positively with their lagged values).

Greater than 2: Negative autocorrelation (residuals are correlated negatively with their lagged values).