vastorbit.machine_learning.metrics.classification_report¶

Computes a classification report using multiple metrics (AUC, accuracy, PRC AUC, F1…). In the case of multiclass classification, it considers each category as positive and switches to the next one during the computation.

Parameters:

y_true (str) – Response column.
y_score (str) – Prediction.
input_relation (SQLRelation) – Relation to use for scoring. This relation can be a view, table, or a customized relation (if an alias is used at the end of the relation). For example: (SELECT … FROM …) x
metrics –
List of the metrics used to compute the final report.
- accuracy:
  Accuracy.
  
  \[Accuracy = \frac{TP + TN}{TP + TN + FP + FN}\]
- aic:
  Akaike’s Information Criterion
  
  \[AIC = 2k - 2\ln(\hat{L})\]
- auc:
  Area Under the Curve (ROC).
  
  \[AUC = \int_{0}^{1} TPR(FPR) \, dFPR\]
- ba:
  Balanced Accuracy.
  
  \[BA = \frac{TPR + TNR}{2}\]
- best_cutoff:
  Cutoff which optimised the ROC Curve prediction.
- bic:
  Bayesian Information Criterion
  
  \[BIC = -2\ln(\hat{L}) + k \ln(n)\]
- bm:
  Informedness
  
  \[BM = TPR + TNR - 1\]
- csi:
  Critical Success Index
  
  \[index = \frac{TP}{TP + FN + FP}\]
- f1:
  F1 Score
  
  \[F_1 Score = 2 \times \frac{Precision \times Recall}{Precision + Recall}\]
- fdr:
  False Discovery Rate
  
  \[FDR = 1 - PPV\]
- fm:
  Fowlkes-Mallows index
  
  \[FM = \sqrt{PPV * TPR}\]
- fnr:
  False Negative Rate
  
  \[FNR = \frac{FN}{FN + TP}\]
- for:
  False Omission Rate
  
  \[FOR = 1 - NPV\]
- fpr:
  False Positive Rate
  
  \[FPR = \frac{FP}{FP + TN}\]
- logloss:
  Log Loss.
  
  \[Loss = -\frac{1}{N} \sum_{i=1}^{N} \left( y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right)\]

labels: ArrayLike, optional: List of the response column categories to use.
cutoff: PythonNumber, optional: Cutoff for which the tested category will be accepted as prediction.
nbins: int, optional: [Used to compute ROC AUC, PRC AUC and the best cutoff] An integer value that determines the number of decision boundaries. Decision boundaries are set at equally spaced intervals between 0 and 1, inclusive. Greater values for nbins give more precise estimations of the AUC, but can potentially decrease performance. The maximum value is 9999. If negative, the maximum value is used.
estimator: object, optional: Estimator used to compute the classification report.

Returns:: report.
Return type:: TableSample

Examples

We should first import vastorbit.

import vastorbit as vo

Binary Classification¶

Let’s create a small dataset that has:

true value
probability of the true value
predicted value

data = vo.VastFrame(
    {
        "y_true": [1, 1, 0, 0, 1],
        "y_prob": [0.8, 0.2, 0.1, 0.6, 0.8],
        "y_pred": [1, 0, 0, 1, 1]
    },
)

Next, we import the metric:

from vastorbit.machine_learning.metrics import classification_report

Now we can conveniently calculate the score:

classification_report(
    y_true  = "y_true",
    y_score = ["y_prob", "y_pred"],
    input_relation = data,
)

Important

In binary classification, y_score should be a list of two column names:

Probability of true value
Prediction value

In the case of multi-class, y_score, is the list of two elements:

list of column names for class probabilities for each class
Prediction value

Note

For multi-class classification, we can select the average method for averaging from the following options: - binary - micro - macro - scores - weighted

It is also possible to directly compute the score from the VastFrame:

data.score(
    y_true  = "y_true",
    y_score = ["y_prob", "y_pred"],
    metric  = "classification_report",
)

Note

vastorbit uses simple SQL queries to compute various metrics. You can use the set_option() function with the sql_on parameter to enable SQL generation and examine the generated queries.

Multi-class Classification¶

Let’s create a small dataset that has:

true value with more than two classes
probability of each class
predicted value

data = vo.VastFrame(
    {
        "y_true": [1, 2, 0, 0, 1],
        "y_prob_0": [0.1, 0.1, 0.1, 0.1, 0.1],
        "y_prob_1": [0.8, 0.6, 0.4, 0.6, 0.2],
        "y_prob_2": [0.1, 0.3, 0.5, 0.3, 0.7],
        "y_pred": [1, 2, 0, 1, 1],
    },
)

Next, we import the metric:

from vastorbit.machine_learning.metrics import classification_report

Now we can conveniently calculate the score:

classification_report(
    y_true  = "y_true",
    y_score =[["y_prob_0","y_prob_1","y_prob_1"], "y_pred"],
    labels = [0,1,2],
    input_relation = data,
)