vastorbit.machine_learning.model_selection.stepwise

vastorbit.machine_learning.model_selection.stepwise(estimator: VASTModel, input_relation: Annotated[str | VastFrame, ''], X: Annotated[str | list[str], 'STRING representing one column or a list of columns'], y: str, criterion: Literal['aic', 'bic'] = 'bic', direction: Literal['forward', 'backward'] = 'backward', max_steps: int = 100, criterion_threshold: int = 3, drop_final_estimator: bool = True, x_order: Literal['pearson', 'spearman', 'random', 'none', None] = 'pearson', print_info: bool = True, show: bool = True, chart: PlottingBase | TableSample | Axes | mFigure | Figure | None = None, **style_kwargs) → TableSample

Uses the Stepwise algorithm to find the most suitable number of features when fitting the estimator.

Parameters:

• estimator (VASTModel) – VAST estimator with a fit method. It must be a Binary Classifier or a Regressor.

• input_relation (SQLRelation) – Relation used to train the model.

• X (SQLColumns) – List of the predictor columns.

• y (str) – Response Column.

• criterion (str, optional) –

  Criterion used to evaluate the model.

  • aic : Akaike’s Information Criterion

  • bic : Bayesian Information Criterion

• direction (str, optional) – Method for starting the stepwise search, either ‘backward’ or ‘forward’.

• max_steps (int, optional) – The maximum number of steps to be considered.

• criterion_threshold (int, optional) – Threshold used when comparing the models criterions. If the difference is less than the threshold, then the current ‘best’ model is changed.

• drop_final_estimator (bool, optional) – If set to True, the final estimator is dropped.

• x_order (str, optional) –

  Method for preprocessing X before using the stepwise algorithm.

  • pearson:

    X is ordered based on the Pearson’s correlation coefficient.

  • spearman:

    X is ordered based on the Spearman’s correlation coefficient.

  • random:

    Shuffles the vector X before applying the stepwise algorithm.

  • none:

    Does not change the order of X.

• print_info (bool, optional) – If set to True, prints the model information at each step.

• show (bool, optional) – If set to True, the stepwise graphic is drawn.

• chart (PlottingObject, optional) – The chart object to plot on.

• **style_kwargs – Any optional parameter to pass to the Plotting functions.

Returns:

result of the stepwise.

Return type:

TableSample

Examples

Let us use a dataset which has a variety of predictors and one value of interest. The Titanic dataset is a good example.

import vastorbit.datasets as vod

data = vod.load_titanic().fillna()

	123 pclass Integer	123 survived Integer	Abc name Varchar(164)	Abc sex Varchar(20)	123 age Double	123 sibsp Integer	123 parch Integer	Abc ticket Varchar(36)	123 fare Double	Abc cabin Varchar(30)	Abc embarked Varchar(20)	Abc boat Varchar(100)	123 body Integer	Abc home.dest Varchar(100)
1	3	1	McCormack, Mr. Thomas Joseph	male	[null]	0	0	367228	7.75	[null]	Q	[null]	[null]	[null]
2	3	1	McCoy, Miss. Agnes	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
3	3	1	McCoy, Miss. Alicia	female	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
4	3	1	McCoy, Mr. Bernard	male	[null]	2	0	367226	23.25	[null]	Q	16	[null]	[null]
5	3	1	McDermott, Miss. Brigdet Delia	female	[null]	0	0	330932	7.7875	[null]	Q	13	[null]	[null]
6	3	0	McEvoy, Mr. Michael	male	[null]	0	0	36568	15.5	[null]	Q	[null]	[null]	[null]
7	3	1	McGovern, Miss. Mary	female	[null]	0	0	330931	7.8792	[null]	Q	13	[null]	[null]
8	3	1	McGowan, Miss. Anna "Annie"	female	15.0	0	0	330923	8.0292	[null]	Q	[null]	[null]	[null]
9	3	0	McGowan, Miss. Katherine	female	35.0	0	0	9232	7.75	[null]	Q	[null]	[null]	[null]
10	3	0	McMahon, Mr. Martin	male	[null]	0	0	370372	7.75	[null]	Q	[null]	[null]	[null]
11	3	0	McNamee, Mr. Neal	male	24.0	1	0	376566	16.1	[null]	S	[null]	[null]	[null]
12	3	0	McNamee, Mrs. Neal (Eileen O'Leary)	female	19.0	1	0	376566	16.1	[null]	S	[null]	53	[null]
13	3	0	McNeill, Miss. Bridget	female	[null]	0	0	370368	7.75	[null]	Q	[null]	[null]	[null]
14	3	0	Meanwell, Miss. (Marion Ogden)	female	[null]	0	0	SOTON/O.Q. 392087	8.05	[null]	S	[null]	[null]	[null]
15	3	0	Meek, Mrs. Thomas (Annie Louise Rowley)	female	[null]	0	0	343095	8.05	[null]	S	[null]	[null]	[null]
16	3	0	Meo, Mr. Alfonzo	male	55.5	0	0	A.5. 11206	8.05	[null]	S	[null]	201	[null]
17	3	0	Mernagh, Mr. Robert	male	[null]	0	0	368703	7.75	[null]	Q	[null]	[null]	[null]
18	3	1	Midtsjo, Mr. Karl Albert	male	21.0	0	0	345501	7.775	[null]	S	15	[null]	[null]
19	3	0	Miles, Mr. Frank	male	[null]	0	0	359306	8.05	[null]	S	[null]	[null]	[null]
20	3	0	Mineff, Mr. Ivan	male	24.0	0	0	349233	7.8958	[null]	S	[null]	[null]	[null]
21	3	0	Minkoff, Mr. Lazar	male	21.0	0	0	349211	7.8958	[null]	S	[null]	[null]	[null]
22	3	0	Mionoff, Mr. Stoytcho	male	28.0	0	0	349207	7.8958	[null]	S	[null]	[null]	[null]
23	3	0	Mitkoff, Mr. Mito	male	[null]	0	0	349221	7.8958	[null]	S	[null]	[null]	[null]
24	3	1	Mockler, Miss. Helen Mary "Ellie"	female	[null]	0	0	330980	7.8792	[null]	Q	16	[null]	[null]
25	3	0	Moen, Mr. Sigurd Hansen	male	25.0	0	0	348123	7.65	F G73	S	[null]	309	[null]
26	3	1	Moor, Master. Meier	male	6.0	0	1	392096	12.475	E121	S	14	[null]	[null]
27	3	1	Moor, Mrs. (Beila)	female	27.0	0	1	392096	12.475	E121	S	14	[null]	[null]
28	3	0	Moore, Mr. Leonard Charles	male	[null]	0	0	A4. 54510	8.05	[null]	S	[null]	[null]	[null]
29	3	1	Moran, Miss. Bertha	female	[null]	1	0	371110	24.15	[null]	Q	16	[null]	[null]
30	3	0	Moran, Mr. Daniel J	male	[null]	1	0	371110	24.15	[null]	Q	[null]	[null]	[null]
31	3	0	Moran, Mr. James	male	[null]	0	0	330877	8.4583	[null]	Q	[null]	[null]	[null]
32	3	0	Morley, Mr. William	male	34.0	0	0	364506	8.05	[null]	S	[null]	[null]	[null]
33	3	0	Morrow, Mr. Thomas Rowan	male	[null]	0	0	372622	7.75	[null]	Q	[null]	[null]	[null]
34	3	1	Moss, Mr. Albert Johan	male	[null]	0	0	312991	7.775	[null]	S	B	[null]	[null]
35	3	1	Moubarek, Master. Gerios	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
36	3	1	Moubarek, Master. Halim Gonios ("William George")	male	[null]	1	1	2661	15.2458	[null]	C	C	[null]	[null]
37	3	1	Moubarek, Mrs. George (Omine "Amenia" Alexander)	female	[null]	0	2	2661	15.2458	[null]	C	C	[null]	[null]
38	3	1	Moussa, Mrs. (Mantoura Boulos)	female	[null]	0	0	2626	7.2292	[null]	C	[null]	[null]	[null]
39	3	0	Moutal, Mr. Rahamin Haim	male	[null]	0	0	374746	8.05	[null]	S	[null]	[null]	[null]
40	3	1	Mullens, Miss. Katherine "Katie"	female	[null]	0	0	35852	7.7333	[null]	Q	16	[null]	[null]
41	3	1	Mulvihill, Miss. Bertha E	female	24.0	0	0	382653	7.75	[null]	Q	15	[null]	[null]
42	3	0	Murdlin, Mr. Joseph	male	[null]	0	0	A./5. 3235	8.05	[null]	S	[null]	[null]	[null]
43	3	1	Murphy, Miss. Katherine "Kate"	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
44	3	1	Murphy, Miss. Margaret Jane	female	[null]	1	0	367230	15.5	[null]	Q	16	[null]	[null]
45	3	1	Murphy, Miss. Nora	female	[null]	0	0	36568	15.5	[null]	Q	16	[null]	[null]
46	3	0	Myhrman, Mr. Pehr Fabian Oliver Malkolm	male	18.0	0	0	347078	7.75	[null]	S	[null]	[null]	[null]
47	3	0	Naidenoff, Mr. Penko	male	22.0	0	0	349206	7.8958	[null]	S	[null]	[null]	[null]
48	3	1	Najib, Miss. Adele Kiamie "Jane"	female	15.0	0	0	2667	7.225	[null]	C	C	[null]	[null]
49	3	1	Nakid, Miss. Maria ("Mary")	female	1.0	0	2	2653	15.7417	[null]	C	C	[null]	[null]
50	3	1	Nakid, Mr. Sahid	male	20.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
51	3	1	Nakid, Mrs. Said (Waika "Mary" Mowad)	female	19.0	1	1	2653	15.7417	[null]	C	C	[null]	[null]
52	3	0	Nancarrow, Mr. William Henry	male	33.0	0	0	A./5. 3338	8.05	[null]	S	[null]	[null]	[null]
53	3	0	Nankoff, Mr. Minko	male	[null]	0	0	349218	7.8958	[null]	S	[null]	[null]	[null]
54	3	0	Nasr, Mr. Mustafa	male	[null]	0	0	2652	7.2292	[null]	C	[null]	[null]	[null]
55	3	0	Naughton, Miss. Hannah	female	[null]	0	0	365237	7.75	[null]	Q	[null]	[null]	[null]
56	3	0	Nenkoff, Mr. Christo	male	[null]	0	0	349234	7.8958	[null]	S	[null]	[null]	[null]
57	3	1	Nicola-Yarred, Master. Elias	male	12.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
58	3	1	Nicola-Yarred, Miss. Jamila	female	14.0	1	0	2651	11.2417	[null]	C	C	[null]	[null]
59	3	0	Nieminen, Miss. Manta Josefina	female	29.0	0	0	3101297	7.925	[null]	S	[null]	[null]	[null]
60	3	0	Niklasson, Mr. Samuel	male	28.0	0	0	363611	8.05	[null]	S	[null]	[null]	[null]
61	3	1	Nilsson, Miss. Berta Olivia	female	18.0	0	0	347066	7.775	[null]	S	D	[null]	[null]
62	3	1	Nilsson, Miss. Helmina Josefina	female	26.0	0	0	347470	7.8542	[null]	S	13	[null]	[null]
63	3	0	Nilsson, Mr. August Ferdinand	male	21.0	0	0	350410	7.8542	[null]	S	[null]	[null]	[null]
64	3	0	Nirva, Mr. Iisakki Antino Aijo	male	41.0	0	0	SOTON/O2 3101272	7.125	[null]	S	[null]	[null]	Finland Sudbury, ON
65	3	1	Niskanen, Mr. Juha	male	39.0	0	0	STON/O 2. 3101289	7.925	[null]	S	9	[null]	[null]
66	3	0	Nosworthy, Mr. Richard Cater	male	21.0	0	0	A/4. 39886	7.8	[null]	S	[null]	[null]	[null]
67	3	0	Novel, Mr. Mansouer	male	28.5	0	0	2697	7.2292	[null]	C	[null]	181	[null]
68	3	1	Nysten, Miss. Anna Sofia	female	22.0	0	0	347081	7.75	[null]	S	13	[null]	[null]
69	3	0	Nysveen, Mr. Johan Hansen	male	61.0	0	0	345364	6.2375	[null]	S	[null]	[null]	[null]
70	3	0	O'Brien, Mr. Thomas	male	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
71	3	0	O'Brien, Mr. Timothy	male	[null]	0	0	330979	7.8292	[null]	Q	[null]	[null]	[null]
72	3	1	O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey)	female	[null]	1	0	370365	15.5	[null]	Q	[null]	[null]	[null]
73	3	0	O'Connell, Mr. Patrick D	male	[null]	0	0	334912	7.7333	[null]	Q	[null]	[null]	[null]
74	3	0	O'Connor, Mr. Maurice	male	[null]	0	0	371060	7.75	[null]	Q	[null]	[null]	[null]
75	3	0	O'Connor, Mr. Patrick	male	[null]	0	0	366713	7.75	[null]	Q	[null]	[null]	[null]
76	3	0	Odahl, Mr. Nils Martin	male	23.0	0	0	7267	9.225	[null]	S	[null]	[null]	[null]
77	3	0	O'Donoghue, Ms. Bridget	female	[null]	0	0	364856	7.75	[null]	Q	[null]	[null]	[null]
78	3	1	O'Driscoll, Miss. Bridget	female	[null]	0	0	14311	7.75	[null]	Q	D	[null]	[null]
79	3	1	O'Dwyer, Miss. Ellen "Nellie"	female	[null]	0	0	330959	7.8792	[null]	Q	[null]	[null]	[null]
80	3	1	Ohman, Miss. Velin	female	22.0	0	0	347085	7.775	[null]	S	C	[null]	[null]
81	3	1	O'Keefe, Mr. Patrick	male	[null]	0	0	368402	7.75	[null]	Q	B	[null]	[null]
82	3	1	O'Leary, Miss. Hanora "Norah"	female	[null]	0	0	330919	7.8292	[null]	Q	13	[null]	[null]
83	3	1	Olsen, Master. Artur Karl	male	9.0	0	1	C 17368	3.1708	[null]	S	13	[null]	[null]
84	3	0	Olsen, Mr. Henry Margido	male	28.0	0	0	C 4001	22.525	[null]	S	[null]	173	[null]
85	3	0	Olsen, Mr. Karl Siegwart Andreas	male	42.0	0	1	4579	8.4042	[null]	S	[null]	[null]	[null]
86	3	0	Olsen, Mr. Ole Martin	male	[null]	0	0	Fa 265302	7.3125	[null]	S	[null]	[null]	[null]
87	3	0	Olsson, Miss. Elina	female	31.0	0	0	350407	7.8542	[null]	S	[null]	[null]	[null]
88	3	0	Olsson, Mr. Nils Johan Goransson	male	28.0	0	0	347464	7.8542	[null]	S	[null]	[null]	[null]
89	3	1	Olsson, Mr. Oscar Wilhelm	male	32.0	0	0	347079	7.775	[null]	S	A	[null]	[null]
90	3	0	Olsvigen, Mr. Thor Anderson	male	20.0	0	0	6563	9.225	[null]	S	[null]	89	Oslo, Norway Cameron, WI
91	3	0	Oreskovic, Miss. Jelka	female	23.0	0	0	315085	8.6625	[null]	S	[null]	[null]	[null]
92	3	0	Oreskovic, Miss. Marija	female	20.0	0	0	315096	8.6625	[null]	S	[null]	[null]	[null]
93	3	0	Oreskovic, Mr. Luka	male	20.0	0	0	315094	8.6625	[null]	S	[null]	[null]	[null]
94	3	0	Osen, Mr. Olaf Elon	male	16.0	0	0	7534	9.2167	[null]	S	[null]	[null]	[null]
95	3	1	Osman, Mrs. Mara	female	31.0	0	0	349244	8.6833	[null]	S	[null]	[null]	[null]
96	3	0	O'Sullivan, Miss. Bridget Mary	female	[null]	0	0	330909	7.6292	[null]	Q	[null]	[null]	[null]
97	3	0	Palsson, Master. Gosta Leonard	male	2.0	3	1	349909	21.075	[null]	S	[null]	4	[null]
98	3	0	Palsson, Master. Paul Folke	male	6.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
99	3	0	Palsson, Miss. Stina Viola	female	3.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]
100	3	0	Palsson, Miss. Torborg Danira	female	8.0	3	1	349909	21.075	[null]	S	[null]	[null]	[null]

Rows: 1-100 | Columns: 14

Note

vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.

Next, we can initialize a Logistic Regression model:

from vastorbit.machine_learning.vast import LogisticRegression

model = LogisticRegression()

Now we can conveniently use the stepwise function to do eith forward or backward stepwise feature selection.

from vastorbit.machine_learning.model_selection import stepwise

result = stepwise(
    model,
    input_relation = data,
    X = ["age", "fare", "parch", "pclass",],
    y = "survived",
    direction = "backward"
)
result

	features	bic	change	variable	importance
0	['age', 'parch', 'fare', 'pclass']	-2049.972057802635	[null]	[null]	0.0
1	['parch', 'fare', 'pclass']	-2013.8555142766222	+	"age"	17.155486643619533
2	['age', 'fare', 'pclass']	-2053.3133671271116	-	"parch"	0.0
3	['age', 'pclass']	-2051.087131958318	-	"fare"	1.0574696240322146
4	['age']	-1878.905171388761	+	"pclass"	81.78704373234825

Rows: 1-5 | Columns: 6

We can also plot the feature selection process by:

result.step_wise_

Plotting the feature importance is also pretty intuitive:

result.importance_

Note

For a complete guide on stepwise plots, please look at Stepwise Plot.