vastorbit.machine_learning.model_selection.randomized_features_search_cv¶
- vastorbit.machine_learning.model_selection.randomized_features_search_cv(estimator: VASTModel, input_relation: Annotated[str | VastFrame, ''], X: Annotated[str | list[str], 'STRING representing one column or a list of columns'], y: str, metric: str = 'auto', cv: int = 3, average: Literal['binary', 'micro', 'macro', 'weighted'] = 'weighted', pos_label: Annotated[bool | float | str | timedelta | datetime, 'Python Scalar'] | None = None, cutoff: Annotated[int | float | Decimal, 'Python Numbers'] = -1, training_score: bool = True, comb_limit: int = 100, skip_error: bool = True, print_info: bool = True, **kwargs) TableSample¶
Computes the k-fold grid search of an estimator using different feature combinations. It can be used to find the set of variables that will optimize the model.
- Parameters:
estimator (VASTModel) – VAST estimator with a fit method.
input_relation (SQLRelation) – Relation used to train the model.
X (SQLColumns) – List of the predictor columns.
y (str) – Response Column.
metric (str, optional) –
Metric used for the model evaluation.
- auto:
logloss for classification & rmse for regression.
For Classification:
- accuracy:
Accuracy.
\[Accuracy = \frac{TP + TN}{TP + TN + FP + FN}\]
- auc:
Area Under the Curve (ROC).
\[AUC = \int_{0}^{1} TPR(FPR) \, dFPR\]
- ba:
Balanced Accuracy.
\[BA = \frac{TPR + TNR}{2}\]
- bm:
Informedness
\[BM = TPR + TNR - 1\]
- csi:
Critical Success Index
\[index = \frac{TP}{TP + FN + FP}\]
- f1:
F1 Score .. math:
F_1 Score = 2 \times
Recall} (rac{Precision times Recall}{Precision +) –
- fdr:
False Discovery Rate
\[FDR = 1 - PPV\]
- fm:
Fowlkes-Mallows index
\[FM = \sqrt{PPV * TPR}\]
- fnr:
False Negative Rate
\[FNR = \frac{FN}{FN + TP}\]
- for:
False Omission Rate
\[FOR = 1 - NPV\]
- fpr:
False Positive Rate
\[FPR = \frac{FP}{FP + TN}\]
- logloss:
Log Loss
\[Loss = -\frac{1}{N} \sum_{i=1}^{N} \left( y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right)\]
- lr+:
Positive Likelihood Ratio.
\[LR+ = \frac{TPR}{FPR}\]
- lr-:
Negative Likelihood Ratio.
\[LR- = \frac{FNR}{TNR}\]
- dor:
Diagnostic Odds Ratio.
\[DOR = \frac{TP \times TN}{FP \times FN}\]
- mcc:
Matthews Correlation Coefficient
- mk:
Markedness
\[MK = PPV + NPV - 1\]
- npv:
Negative Predictive Value
\[NPV = \frac{TN}{TN + FN}\]
- prc_auc:
Area Under the Curve (PRC)
\[AUC = \int_{0}^{1} Precision(Recall) \, dRecall\]
- precision:
Precision
\[TP / (TP + FP)\]
- pt:
Prevalence Threshold.
\[\frac{\sqrt{FPR}}{\sqrt{TPR} + \sqrt{FPR}}\]
- recall:
Recall.
\[TP / (TP + FN)\]
- specificity:
Specificity.
\[TN / (TN + FP)\]
For Regression:
- max:
Max Error.
\[ME = \max_{i=1}^{n} \left| y_i - \hat{y}_i \right|\]
- mae:
Mean Absolute Error.
\[MAE = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right|\]
- median:
Median Absolute Error.
\[MedAE = \text{median}_{i=1}^{n} \left| y_i - \hat{y}_i \right|\]
- mse:
Mean Squared Error.
\[MSE = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2\]
- msle:
Mean Squared Log Error.
\[MSLE = \frac{1}{n} \sum_{i=1}^{n} (\log(1 + y_i) - \log(1 + \hat{y}_i))^2\]
- r2:
R squared coefficient.
\[R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}\]
- r2a:
R2 adjusted
\[\text{Adjusted } R^2 = 1 - \frac{(1 - R^2)(n - 1)}{n - k - 1}\]
- var:
Explained Variance.
\[VAR = 1 - \frac{Var(y - \hat{y})}{Var(y)}\]
- rmse:
Root-mean-squared error
\[RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}\]
cv (int, optional) – Number of folds.
average (str, optional) –
The method used to compute the final score for multiclass-classification.
- binary:
considers one of the classes as positive and use the binary confusion matrix to compute the score.
- micro:
positive and negative values globally.
- macro:
average of the score of each class.
- weighted:
weighted average of the score of each class.
pos_label (PythonScalar, optional) – The main class to be considered as positive (classification only).
cutoff (float, optional) – The model cutoff (classification only).
training_score (bool, optional) – If set to True, the training score is computed with the validation score.
comb_limit (int, optional) – Maximum number of features combinations used to train the model.
skip_error (bool, optional) – If set to True and an error occurs, the error is displayed but not raised.
print_info (bool, optional) – If set to True, prints the model information at each step.
- Returns:
result of the randomized features search.
- Return type:
Examples
Let us use a dataset which has a variety of predictors and one value of interest. The Titanic dataset is a good example.
import vastorbit.datasets as vod data = vod.load_titanic().fillna()
123pclassInteger123survivedIntegerAbcnameVarchar(164)AbcsexVarchar(20)123ageDouble123sibspInteger123parchIntegerAbcticketVarchar(36)123fareDoubleAbccabinVarchar(30)AbcembarkedVarchar(20)AbcboatVarchar(100)123bodyIntegerAbchome.destVarchar(100)1 3 1 McCormack, Mr. Thomas Joseph male [null] 0 0 367228 7.75 [null] Q [null] [null] [null] 2 3 1 McCoy, Miss. Agnes female [null] 2 0 367226 23.25 [null] Q 16 [null] [null] 3 3 1 McCoy, Miss. Alicia female [null] 2 0 367226 23.25 [null] Q 16 [null] [null] 4 3 1 McCoy, Mr. Bernard male [null] 2 0 367226 23.25 [null] Q 16 [null] [null] 5 3 1 McDermott, Miss. Brigdet Delia female [null] 0 0 330932 7.7875 [null] Q 13 [null] [null] 6 3 0 McEvoy, Mr. Michael male [null] 0 0 36568 15.5 [null] Q [null] [null] [null] 7 3 1 McGovern, Miss. Mary female [null] 0 0 330931 7.8792 [null] Q 13 [null] [null] 8 3 1 McGowan, Miss. Anna "Annie" female 15.0 0 0 330923 8.0292 [null] Q [null] [null] [null] 9 3 0 McGowan, Miss. Katherine female 35.0 0 0 9232 7.75 [null] Q [null] [null] [null] 10 3 0 McMahon, Mr. Martin male [null] 0 0 370372 7.75 [null] Q [null] [null] [null] 11 3 0 McNamee, Mr. Neal male 24.0 1 0 376566 16.1 [null] S [null] [null] [null] 12 3 0 McNamee, Mrs. Neal (Eileen O'Leary) female 19.0 1 0 376566 16.1 [null] S [null] 53 [null] 13 3 0 McNeill, Miss. Bridget female [null] 0 0 370368 7.75 [null] Q [null] [null] [null] 14 3 0 Meanwell, Miss. (Marion Ogden) female [null] 0 0 SOTON/O.Q. 392087 8.05 [null] S [null] [null] [null] 15 3 0 Meek, Mrs. Thomas (Annie Louise Rowley) female [null] 0 0 343095 8.05 [null] S [null] [null] [null] 16 3 0 Meo, Mr. Alfonzo male 55.5 0 0 A.5. 11206 8.05 [null] S [null] 201 [null] 17 3 0 Mernagh, Mr. Robert male [null] 0 0 368703 7.75 [null] Q [null] [null] [null] 18 3 1 Midtsjo, Mr. Karl Albert male 21.0 0 0 345501 7.775 [null] S 15 [null] [null] 19 3 0 Miles, Mr. Frank male [null] 0 0 359306 8.05 [null] S [null] [null] [null] 20 3 0 Mineff, Mr. Ivan male 24.0 0 0 349233 7.8958 [null] S [null] [null] [null] 21 3 0 Minkoff, Mr. Lazar male 21.0 0 0 349211 7.8958 [null] S [null] [null] [null] 22 3 0 Mionoff, Mr. Stoytcho male 28.0 0 0 349207 7.8958 [null] S [null] [null] [null] 23 3 0 Mitkoff, Mr. Mito male [null] 0 0 349221 7.8958 [null] S [null] [null] [null] 24 3 1 Mockler, Miss. Helen Mary "Ellie" female [null] 0 0 330980 7.8792 [null] Q 16 [null] [null] 25 3 0 Moen, Mr. Sigurd Hansen male 25.0 0 0 348123 7.65 F G73 S [null] 309 [null] 26 3 1 Moor, Master. Meier male 6.0 0 1 392096 12.475 E121 S 14 [null] [null] 27 3 1 Moor, Mrs. (Beila) female 27.0 0 1 392096 12.475 E121 S 14 [null] [null] 28 3 0 Moore, Mr. Leonard Charles male [null] 0 0 A4. 54510 8.05 [null] S [null] [null] [null] 29 3 1 Moran, Miss. Bertha female [null] 1 0 371110 24.15 [null] Q 16 [null] [null] 30 3 0 Moran, Mr. Daniel J male [null] 1 0 371110 24.15 [null] Q [null] [null] [null] 31 3 0 Moran, Mr. James male [null] 0 0 330877 8.4583 [null] Q [null] [null] [null] 32 3 0 Morley, Mr. William male 34.0 0 0 364506 8.05 [null] S [null] [null] [null] 33 3 0 Morrow, Mr. Thomas Rowan male [null] 0 0 372622 7.75 [null] Q [null] [null] [null] 34 3 1 Moss, Mr. Albert Johan male [null] 0 0 312991 7.775 [null] S B [null] [null] 35 3 1 Moubarek, Master. Gerios male [null] 1 1 2661 15.2458 [null] C C [null] [null] 36 3 1 Moubarek, Master. Halim Gonios ("William George") male [null] 1 1 2661 15.2458 [null] C C [null] [null] 37 3 1 Moubarek, Mrs. George (Omine "Amenia" Alexander) female [null] 0 2 2661 15.2458 [null] C C [null] [null] 38 3 1 Moussa, Mrs. (Mantoura Boulos) female [null] 0 0 2626 7.2292 [null] C [null] [null] [null] 39 3 0 Moutal, Mr. Rahamin Haim male [null] 0 0 374746 8.05 [null] S [null] [null] [null] 40 3 1 Mullens, Miss. Katherine "Katie" female [null] 0 0 35852 7.7333 [null] Q 16 [null] [null] 41 3 1 Mulvihill, Miss. Bertha E female 24.0 0 0 382653 7.75 [null] Q 15 [null] [null] 42 3 0 Murdlin, Mr. Joseph male [null] 0 0 A./5. 3235 8.05 [null] S [null] [null] [null] 43 3 1 Murphy, Miss. Katherine "Kate" female [null] 1 0 367230 15.5 [null] Q 16 [null] [null] 44 3 1 Murphy, Miss. Margaret Jane female [null] 1 0 367230 15.5 [null] Q 16 [null] [null] 45 3 1 Murphy, Miss. Nora female [null] 0 0 36568 15.5 [null] Q 16 [null] [null] 46 3 0 Myhrman, Mr. Pehr Fabian Oliver Malkolm male 18.0 0 0 347078 7.75 [null] S [null] [null] [null] 47 3 0 Naidenoff, Mr. Penko male 22.0 0 0 349206 7.8958 [null] S [null] [null] [null] 48 3 1 Najib, Miss. Adele Kiamie "Jane" female 15.0 0 0 2667 7.225 [null] C C [null] [null] 49 3 1 Nakid, Miss. Maria ("Mary") female 1.0 0 2 2653 15.7417 [null] C C [null] [null] 50 3 1 Nakid, Mr. Sahid male 20.0 1 1 2653 15.7417 [null] C C [null] [null] 51 3 1 Nakid, Mrs. Said (Waika "Mary" Mowad) female 19.0 1 1 2653 15.7417 [null] C C [null] [null] 52 3 0 Nancarrow, Mr. William Henry male 33.0 0 0 A./5. 3338 8.05 [null] S [null] [null] [null] 53 3 0 Nankoff, Mr. Minko male [null] 0 0 349218 7.8958 [null] S [null] [null] [null] 54 3 0 Nasr, Mr. Mustafa male [null] 0 0 2652 7.2292 [null] C [null] [null] [null] 55 3 0 Naughton, Miss. Hannah female [null] 0 0 365237 7.75 [null] Q [null] [null] [null] 56 3 0 Nenkoff, Mr. Christo male [null] 0 0 349234 7.8958 [null] S [null] [null] [null] 57 3 1 Nicola-Yarred, Master. Elias male 12.0 1 0 2651 11.2417 [null] C C [null] [null] 58 3 1 Nicola-Yarred, Miss. Jamila female 14.0 1 0 2651 11.2417 [null] C C [null] [null] 59 3 0 Nieminen, Miss. Manta Josefina female 29.0 0 0 3101297 7.925 [null] S [null] [null] [null] 60 3 0 Niklasson, Mr. Samuel male 28.0 0 0 363611 8.05 [null] S [null] [null] [null] 61 3 1 Nilsson, Miss. Berta Olivia female 18.0 0 0 347066 7.775 [null] S D [null] [null] 62 3 1 Nilsson, Miss. Helmina Josefina female 26.0 0 0 347470 7.8542 [null] S 13 [null] [null] 63 3 0 Nilsson, Mr. August Ferdinand male 21.0 0 0 350410 7.8542 [null] S [null] [null] [null] 64 3 0 Nirva, Mr. Iisakki Antino Aijo male 41.0 0 0 SOTON/O2 3101272 7.125 [null] S [null] [null] Finland Sudbury, ON 65 3 1 Niskanen, Mr. Juha male 39.0 0 0 STON/O 2. 3101289 7.925 [null] S 9 [null] [null] 66 3 0 Nosworthy, Mr. Richard Cater male 21.0 0 0 A/4. 39886 7.8 [null] S [null] [null] [null] 67 3 0 Novel, Mr. Mansouer male 28.5 0 0 2697 7.2292 [null] C [null] 181 [null] 68 3 1 Nysten, Miss. Anna Sofia female 22.0 0 0 347081 7.75 [null] S 13 [null] [null] 69 3 0 Nysveen, Mr. Johan Hansen male 61.0 0 0 345364 6.2375 [null] S [null] [null] [null] 70 3 0 O'Brien, Mr. Thomas male [null] 1 0 370365 15.5 [null] Q [null] [null] [null] 71 3 0 O'Brien, Mr. Timothy male [null] 0 0 330979 7.8292 [null] Q [null] [null] [null] 72 3 1 O'Brien, Mrs. Thomas (Johanna "Hannah" Godfrey) female [null] 1 0 370365 15.5 [null] Q [null] [null] [null] 73 3 0 O'Connell, Mr. Patrick D male [null] 0 0 334912 7.7333 [null] Q [null] [null] [null] 74 3 0 O'Connor, Mr. Maurice male [null] 0 0 371060 7.75 [null] Q [null] [null] [null] 75 3 0 O'Connor, Mr. Patrick male [null] 0 0 366713 7.75 [null] Q [null] [null] [null] 76 3 0 Odahl, Mr. Nils Martin male 23.0 0 0 7267 9.225 [null] S [null] [null] [null] 77 3 0 O'Donoghue, Ms. Bridget female [null] 0 0 364856 7.75 [null] Q [null] [null] [null] 78 3 1 O'Driscoll, Miss. Bridget female [null] 0 0 14311 7.75 [null] Q D [null] [null] 79 3 1 O'Dwyer, Miss. Ellen "Nellie" female [null] 0 0 330959 7.8792 [null] Q [null] [null] [null] 80 3 1 Ohman, Miss. Velin female 22.0 0 0 347085 7.775 [null] S C [null] [null] 81 3 1 O'Keefe, Mr. Patrick male [null] 0 0 368402 7.75 [null] Q B [null] [null] 82 3 1 O'Leary, Miss. Hanora "Norah" female [null] 0 0 330919 7.8292 [null] Q 13 [null] [null] 83 3 1 Olsen, Master. Artur Karl male 9.0 0 1 C 17368 3.1708 [null] S 13 [null] [null] 84 3 0 Olsen, Mr. Henry Margido male 28.0 0 0 C 4001 22.525 [null] S [null] 173 [null] 85 3 0 Olsen, Mr. Karl Siegwart Andreas male 42.0 0 1 4579 8.4042 [null] S [null] [null] [null] 86 3 0 Olsen, Mr. Ole Martin male [null] 0 0 Fa 265302 7.3125 [null] S [null] [null] [null] 87 3 0 Olsson, Miss. Elina female 31.0 0 0 350407 7.8542 [null] S [null] [null] [null] 88 3 0 Olsson, Mr. Nils Johan Goransson male 28.0 0 0 347464 7.8542 [null] S [null] [null] [null] 89 3 1 Olsson, Mr. Oscar Wilhelm male 32.0 0 0 347079 7.775 [null] S A [null] [null] 90 3 0 Olsvigen, Mr. Thor Anderson male 20.0 0 0 6563 9.225 [null] S [null] 89 Oslo, Norway Cameron, WI 91 3 0 Oreskovic, Miss. Jelka female 23.0 0 0 315085 8.6625 [null] S [null] [null] [null] 92 3 0 Oreskovic, Miss. Marija female 20.0 0 0 315096 8.6625 [null] S [null] [null] [null] 93 3 0 Oreskovic, Mr. Luka male 20.0 0 0 315094 8.6625 [null] S [null] [null] [null] 94 3 0 Osen, Mr. Olaf Elon male 16.0 0 0 7534 9.2167 [null] S [null] [null] [null] 95 3 1 Osman, Mrs. Mara female 31.0 0 0 349244 8.6833 [null] S [null] [null] [null] 96 3 0 O'Sullivan, Miss. Bridget Mary female [null] 0 0 330909 7.6292 [null] Q [null] [null] [null] 97 3 0 Palsson, Master. Gosta Leonard male 2.0 3 1 349909 21.075 [null] S [null] 4 [null] 98 3 0 Palsson, Master. Paul Folke male 6.0 3 1 349909 21.075 [null] S [null] [null] [null] 99 3 0 Palsson, Miss. Stina Viola female 3.0 3 1 349909 21.075 [null] S [null] [null] [null] 100 3 0 Palsson, Miss. Torborg Danira female 8.0 3 1 349909 21.075 [null] S [null] [null] [null] Rows: 1-100 | Columns: 14Note
vastorbit offers a wide range of sample datasets that are ideal for training and testing purposes. You can explore the full list of available datasets in the Datasets, which provides detailed information on each dataset and how to use them effectively. These datasets are invaluable resources for honing your data analysis and machine learning skills within the vastorbit environment.
Next, we can initialize a Logistic Regression model:
from vastorbit.machine_learning.vast import LogisticRegression model = LogisticRegression()
Now we can conveniently use the
randomized_features_search_cvfunction to do either forward or backward randomized features search feature selection.from vastorbit.machine_learning.model_selection import randomized_features_search_cv result = randomized_features_search_cv( model, input_relation = data, X = ["age", "fare", "parch", "pclass",], y = "survived", cv = 3, )
features avg_score avg_train_score avg_time score_std score_train_std 1 ['age', 'fare', 'pclass'] 0.5951383070379649 0.6005456226398698 0.2524605592091878 0.010140508708026427 0.007020772263870089 2 ['age', 'fare', 'parch', 'pclass'] 0.5972288691341497 0.5945456522208784 0.24932026863098145 0.010035594799307055 0.0027791774660392006 3 ['age', 'parch', 'pclass'] 0.6005790873086783 0.5942421465527983 0.29668164253234863 0.020504029748668304 0.012937304223568428 4 ['fare', 'pclass'] 0.601021994067112 0.6111357717153129 0.25335367520650226 0.010318736539551239 0.00267764561875376 5 ['age', 'pclass'] 0.6025715301353866 0.5954391395671392 0.28185057640075684 0.019419345625037426 0.012425727407952952 6 ['pclass'] 0.6118862713572045 0.6147682544427279 0.24561897913614908 0.008182735855034295 0.007643021912371926 7 ['fare'] 0.6222120908974028 0.6375910507878493 0.26256434122721356 0.0026209090011147245 0.0007541277603727561 8 ['age', 'fare'] 0.6230115943272432 0.625583722749565 0.26221545537312824 0.005753235727958881 0.004666188569476611 9 ['fare', 'parch', 'pclass'] 0.6261802321748634 0.6071497981562518 0.25786423683166504 0.017216933467369595 0.007281240704743727 10 ['parch', 'pclass'] 0.6263340870534121 0.6126810698875609 0.2359894116719564 0.011492227462347243 0.003282883626266529 11 ['fare', 'parch'] 0.6282968388930178 0.6363383432646602 0.22934937477111816 0.01036344224097955 0.00436119680007173 12 ['age', 'fare', 'parch'] 0.6463533697540366 0.6186094428714405 0.2728509108225505 0.006011217306006416 0.0026136970977846323 13 ['age', 'parch'] 0.6620862153313993 0.6653017441046818 0.2477727731068929 0.004986671725996179 0.0025152467204894503 14 ['parch'] 0.6678147708044598 0.6607362817567642 0.25994078318277997 0.00491268013497973 0.001491224758091951 15 ['age'] 0.6705398195627638 0.6615562318497739 0.28521013259887695 0.0047928939902736 0.002492124724531832 Rows: 1-15 | Columns: 6Note
The models are arranged in ascending order of
avg_score.