vastorbit.machine_learning.vast.feature_extraction.text.TfidfVectorizer

class vastorbit.machine_learning.vast.feature_extraction.text.TfidfVectorizer(name: str = None, overwrite_model: bool = False, lowercase: bool = True, vocabulary: Annotated[list | ndarray, 'Array Like Structure'] | None = None, max_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, min_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, norm: Literal['l1', 'l2', None] = 'l2', smooth_idf: bool = True, compute_vocabulary: bool = True)

[Beta Version] Create tfidf representation of documents.

The formula that is used to compute the tf-idf for a term t of a document d in a document set is

\[tf-idf(t, d) = tf(t, d) * idf(t),\]

and if smooth_idf = False, the idf is computed as

\[idf(t) = log [ n / df(t) ] + 1,\]

where n is the total number of documents in the document set and df(t) is the document frequency of t; the document frequency is the number of documents in the document set that contain the term t. The effect of adding “1” to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored.

If smooth_idf=True (the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions:

\[idf(t) = log [ (1 + n) / (1 + df(t)) ] + 1.\]

Parameters:

• name (str, optional) – Name of the model.

• overwrite_model (bool, optional) – If set to True, training a model with the same name as an existing model overwrites the existing model.

• lowercase (bool, optional) – Converts all the elements to lowercase before processing.

• vocabulary (list, optional) – A list of string elements to be regarded as the primary vocabulary.

• max_df (PythonNumber, optional) – While constructing the vocabulary, exclude terms with a document frequency surpassing the specified threshold, essentially treating them as corpus-specific stop words. If the value is a float within the range [0.0, 1.0], it denotes a proportion of documents; if an integer, it signifies absolute counts. Note that this parameter is disregarded if a custom vocabulary is provided.

• min_df (PythonNumber, optional) – When constructing the vocabulary, omit terms with a document frequency below the specified threshold, often referred to as the cut-off in literature. If the value is a float within the range [0.0, 1.0], it denotes a proportion of documents; if an integer, it signifies absolute counts. It’s important to note that this parameter is disregarded if a custom vocabulary is provided.

• norm (str, optional) –

  The tfidf values of each document will have unit norm, either:

  • l2:

    Sum of squares of vector elements is 1.

  • l1:

    Sum of absolute values of vector elements is 1.

  • None:

    No normalization.

• smooth_idf (bool, optional) – Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions.

• compute_vocabulary (bool, optional) – If set to true, the vocabulary is computed, making the operation more resource-intensive.

Variables:

• created (Many attributes are)

• phase. (during the fitting)

• vocabulary_ (ArrayLike) – The ultimate vocabulary. If empty, it implies that all words are utilized, and the user opted not to compute a specific vocabulary.

• fixed_vocabulary_ (bool) – Boolean indicating whether a vocabulary was supplied by the user.

• idf_ (VastFrame) – The IDF table which is computed based on the relation used for the fitting process.

• tf_ (VastFrame) – The TF table which is computed based on the relation used for the fitting process.

• stop_words_ (ArrayLike) –

  Terms are excluded under the following conditions:

  • They appear in an excessive number of documents

  (controlled by max_df).

  • They appear in an insufficient number of documents

  (controlled by min_df).

  This functionality is only applicable when no specific vocabulary is provided and compute_vocabulary is set to True.

• n_document_ (int) – Total number of document. This functionality is only applicable when no specific vocabulary is provided and compute_vocabulary is set to True.

• note:: (..) – All attributes can be accessed using the get_attributes() method.

Examples

We import vastorbit:

import vastorbit as vo

Hint

By assigning an alias to vastorbit, we mitigate the risk of code collisions with other libraries. This precaution is necessary because vastorbit uses commonly known function names like “average” and “median”, which can potentially lead to naming conflicts. The use of an alias ensures that the functions from vastorbit are used as intended without interfering with functions from other libraries.

For this example, let’s generate some text.

documents = [
    "Natural language processing is a field of study in artificial intelligence.",
    "TF-IDF stands for Term Frequency-Inverse Document Frequency.",
    "Machine learning algorithms can be applied to text data for classification.",
    "The 20 Newsgroups dataset is a collection of text documents used for text classification.",
    "Clustering is a technique used to group similar documents together.",
    "Python is a popular programming language for natural language processing tasks.",
    "TF-IDF is a technique widely used in information retrieval.",
    "An algorithm is a set of instructions designed to perform a specific task.",
    "Data preprocessing is an important step in preparing data for machine learning.",
]

Next, we can insert this text into a VastFrame:

data = vo.VastFrame(
    {
        "id": (list(range(1,len(documents)+1))),
        "docs": documents,
    }
)

Then we can initialize the object and fit the model, to learn the idf weigths.

from vastorbit.machine_learning.vast.feature_extraction.text import TfidfVectorizer

model = TfidfVectorizer(name = "test_idf")
model.fit(
    input_relation = data,
    index = "id",
    x = "docs",
)

We apply the transform function to obtain the idf representation.

model.transform(
    vdf = data,
    index = "id",
    x = "docs",
)

	123 row_id Integer	Abc word Varchar(4049)	123 tf Double	123 idf Double	123 tfidf Double
1	6	python	1.0	2.6094379124341005	0.3313965871579751
2	6	is	1.0	1.2231435513142097	0.1553382805462904
3	6	a	1.0	1.3566749439387324	0.17229666364609753
4	6	popular	1.0	2.6094379124341005	0.3313965871579751
5	6	programming	1.0	2.6094379124341005	0.3313965871579751
6	6	language	2.0	2.203972804325936	0.559805667007647
7	6	for	1.0	1.5108256237659907	0.19187368019798118
8	6	natural	1.0	2.203972804325936	0.2799028335038235
9	6	processing	1.0	2.203972804325936	0.2799028335038235
10	6	tasks	1.0	2.6094379124341005	0.3313965871579751
11	1	natural	1.0	2.203972804325936	0.30420571107707617
12	1	language	1.0	2.203972804325936	0.30420571107707617
13	1	processing	1.0	2.203972804325936	0.30420571107707617
14	1	is	1.0	1.2231435513142097	0.16882570104610647
15	1	a	1.0	1.3566749439387324	0.18725651478606037
16	1	field	1.0	2.6094379124341005	0.3601704676688489
17	1	of	1.0	1.916290731874155	0.2644980843574777
18	1	study	1.0	2.6094379124341005	0.3601704676688489
19	1	in	1.0	1.916290731874155	0.2644980843574777
20	1	artificial	1.0	2.6094379124341005	0.3601704676688489
21	1	intelligence	1.0	2.6094379124341005	0.3601704676688489
22	7	tfidf	1.0	2.203972804325936	0.3449460916969199
23	7	is	1.0	1.2231435513142097	0.19143547814291995
24	7	a	1.0	1.3566749439387324	0.21233461624220487
25	7	technique	1.0	2.203972804325936	0.3449460916969199
26	7	widely	1.0	2.6094379124341005	0.4084058603868319
27	7	used	1.0	1.916290731874155	0.2999206692648759
28	7	in	1.0	1.916290731874155	0.2999206692648759
29	7	information	1.0	2.6094379124341005	0.4084058603868319
30	7	retrieval	1.0	2.6094379124341005	0.4084058603868319
31	8	an	1.0	2.203972804325936	0.2658587251660463
32	8	algorithm	1.0	2.6094379124341005	0.31476878273543557
33	8	is	1.0	1.2231435513142097	0.14754419138439442
34	8	a	2.0	1.3566749439387324	0.3273033772034955
35	8	set	1.0	2.6094379124341005	0.31476878273543557
36	8	of	1.0	1.916290731874155	0.231156487059915
37	8	instructions	1.0	2.6094379124341005	0.31476878273543557
38	8	designed	1.0	2.6094379124341005	0.31476878273543557
39	8	to	1.0	1.916290731874155	0.231156487059915
40	8	perform	1.0	2.6094379124341005	0.31476878273543557
41	8	specific	1.0	2.6094379124341005	0.31476878273543557
42	8	task	1.0	2.6094379124341005	0.31476878273543557
43	9	data	2.0	2.203972804325936	0.5318504429864452
44	9	preprocessing	1.0	2.6094379124341005	0.31484751239890096
45	9	is	1.0	1.2231435513142097	0.1475810949948256
46	9	an	1.0	2.203972804325936	0.2659252214932226
47	9	important	1.0	2.6094379124341005	0.31484751239890096
48	9	step	1.0	2.6094379124341005	0.31484751239890096
49	9	in	1.0	1.916290731874155	0.2312143036968633
50	9	preparing	1.0	2.6094379124341005	0.31484751239890096
51	9	for	1.0	1.5108256237659907	0.18229201279118498
52	9	machine	1.0	2.203972804325936	0.2659252214932226
53	9	learning	1.0	2.203972804325936	0.2659252214932226
54	3	machine	1.0	2.203972804325936	0.2907046445125331
55	3	learning	1.0	2.203972804325936	0.2907046445125331
56	3	algorithms	1.0	2.6094379124341005	0.344185608471555
57	3	can	1.0	2.6094379124341005	0.344185608471555
58	3	be	1.0	2.6094379124341005	0.344185608471555
59	3	applied	1.0	2.6094379124341005	0.344185608471555
60	3	to	1.0	1.916290731874155	0.25275929671124686
61	3	text	1.0	2.203972804325936	0.2907046445125331
62	3	data	1.0	2.203972804325936	0.2907046445125331
63	3	for	1.0	1.5108256237659907	0.19927833275222503
64	3	classification	1.0	2.203972804325936	0.2907046445125331
65	2	tfidf	1.0	2.203972804325936	0.3434249460827697
66	2	stands	1.0	2.6094379124341005	0.40660486945440955
67	2	for	1.0	1.5108256237659907	0.2354181536922318
68	2	term	1.0	2.6094379124341005	0.40660486945440955
69	2	frequencyinverse	1.0	2.6094379124341005	0.40660486945440955
70	2	document	1.0	2.6094379124341005	0.40660486945440955
71	2	frequency	1.0	2.6094379124341005	0.40660486945440955
72	4	the	1.0	2.6094379124341005	0.2990194911597364
73	4	20	1.0	2.6094379124341005	0.2990194911597364
74	4	newsgroups	1.0	2.6094379124341005	0.2990194911597364
75	4	dataset	1.0	2.6094379124341005	0.2990194911597364
76	4	is	1.0	1.2231435513142097	0.14016189486115033
77	4	a	1.0	1.3566749439387324	0.1554634618714917
78	4	collection	1.0	2.6094379124341005	0.2990194911597364
79	4	of	1.0	1.916290731874155	0.21959069301044337
80	4	text	2.0	2.203972804325936	0.5051132455301
81	4	documents	1.0	2.203972804325936	0.25255662276505
82	4	used	1.0	1.916290731874155	0.21959069301044337
83	4	for	1.0	1.5108256237659907	0.17312782461575696
84	4	classification	1.0	2.203972804325936	0.25255662276505
85	5	clustering	1.0	2.6094379124341005	0.37808950386861345
86	5	is	1.0	1.2231435513142097	0.17722504002599546
87	5	a	1.0	1.3566749439387324	0.19657281517240482
88	5	technique	1.0	2.203972804325936	0.31934041433091737
89	5	used	1.0	1.916290731874155	0.27765727194730444
90	5	to	1.0	1.916290731874155	0.27765727194730444
91	5	group	1.0	2.6094379124341005	0.37808950386861345
92	5	similar	1.0	2.6094379124341005	0.37808950386861345
93	5	documents	1.0	2.203972804325936	0.31934041433091737
94	5	together	1.0	2.6094379124341005	0.37808950386861345

Rows: 1-94 | Columns: 5

Notice how we can get the idf weight/score of each word in each row. We can also get the results in a more convient form by switching the pivot parameter to True. But for large datasets this is not ideal.

Advanced Analysis

In the above result, we can observe some less informative words such as “is” and “a”, which may not provide meaningful insights.

To address this issue, we can make use of the max_df parameter to exclude words that occur too frequently and might be irrelevant. Similarly, we can leverage the min_df parameter to eliminate words with low frequency that may not contribute significantly.

Let’s apply these parameters to remove common words like “is” and “a.”

model = TfidfVectorizer(max_df = 4, min_df = 1,)
model.fit(
    input_relation = data,
    index = "id",
    x = "docs",
)
model.transform(
    vdf = data,
    index = "id",
    x = "docs",
)

	123 row_id Integer	Abc word Varchar(4049)	123 tf Double	123 idf Double	123 tfidf Double
1	7	tfidf	1.0	2.203972804325936	0.3599704622538046
2	7	technique	1.0	2.203972804325936	0.3599704622538046
3	7	widely	1.0	2.6094379124341005	0.4261942659718019
4	7	used	1.0	1.916290731874155	0.31298392575964307
5	7	in	1.0	1.916290731874155	0.31298392575964307
6	7	information	1.0	2.6094379124341005	0.4261942659718019
7	7	retrieval	1.0	2.6094379124341005	0.4261942659718019
8	6	python	1.0	2.6094379124341005	0.3475186445132981
9	6	popular	1.0	2.6094379124341005	0.3475186445132981
10	6	programming	1.0	2.6094379124341005	0.3475186445132981
11	6	language	2.0	2.203972804325936	0.5870395596337948
12	6	natural	1.0	2.203972804325936	0.2935197798168974
13	6	processing	1.0	2.203972804325936	0.2935197798168974
14	6	tasks	1.0	2.6094379124341005	0.3475186445132981
15	4	newsgroups	1.0	2.6094379124341005	0.3107020913110948
16	4	dataset	1.0	2.6094379124341005	0.3107020913110948
17	4	collection	1.0	2.6094379124341005	0.3107020913110948
18	4	of	1.0	1.916290731874155	0.22817003428833435
19	4	text	2.0	2.203972804325936	0.5248478656907996
20	4	documents	1.0	2.203972804325936	0.2624239328453998
21	4	used	1.0	1.916290731874155	0.22817003428833435
22	4	classification	1.0	2.203972804325936	0.2624239328453998
23	4	the	1.0	2.6094379124341005	0.3107020913110948
24	4	20	1.0	2.6094379124341005	0.3107020913110948
25	5	clustering	1.0	2.6094379124341005	0.39207100431193614
26	5	technique	1.0	2.203972804325936	0.33114941219743876
27	5	used	1.0	1.916290731874155	0.28792485470509516
28	5	to	1.0	1.916290731874155	0.28792485470509516
29	5	group	1.0	2.6094379124341005	0.39207100431193614
30	5	similar	1.0	2.6094379124341005	0.39207100431193614
31	5	documents	1.0	2.203972804325936	0.33114941219743876
32	5	together	1.0	2.6094379124341005	0.39207100431193614
33	3	machine	1.0	2.203972804325936	0.29665466794755513
34	3	learning	1.0	2.203972804325936	0.29665466794755513
35	3	algorithms	1.0	2.6094379124341005	0.35123025834234434
36	3	can	1.0	2.6094379124341005	0.35123025834234434
37	3	be	1.0	2.6094379124341005	0.35123025834234434
38	3	applied	1.0	2.6094379124341005	0.35123025834234434
39	3	to	1.0	1.916290731874155	0.2579326703302803
40	3	text	1.0	2.203972804325936	0.29665466794755513
41	3	data	1.0	2.203972804325936	0.29665466794755513
42	3	classification	1.0	2.203972804325936	0.29665466794755513
43	1	natural	1.0	2.203972804325936	0.31436127094398053
44	1	language	1.0	2.203972804325936	0.31436127094398053
45	1	processing	1.0	2.203972804325936	0.31436127094398053
46	1	field	1.0	2.6094379124341005	0.3721943469502448
47	1	of	1.0	1.916290731874155	0.27332805050394926
48	1	study	1.0	2.6094379124341005	0.3721943469502448
49	1	in	1.0	1.916290731874155	0.27332805050394926
50	1	artificial	1.0	2.6094379124341005	0.3721943469502448
51	1	intelligence	1.0	2.6094379124341005	0.3721943469502448
52	9	data	2.0	2.203972804325936	0.5471117563820143
53	9	preprocessing	1.0	2.6094379124341005	0.3238819818101776
54	9	an	1.0	2.203972804325936	0.27355587819100713
55	9	important	1.0	2.6094379124341005	0.3238819818101776
56	9	step	1.0	2.6094379124341005	0.3238819818101776
57	9	in	1.0	1.916290731874155	0.23784893942348248
58	9	preparing	1.0	2.6094379124341005	0.3238819818101776
59	9	machine	1.0	2.203972804325936	0.27355587819100713
60	9	learning	1.0	2.203972804325936	0.27355587819100713
61	2	tfidf	1.0	2.203972804325936	0.35335631585664906
62	2	stands	1.0	2.6094379124341005	0.4183633143678389
63	2	term	1.0	2.6094379124341005	0.4183633143678389
64	2	frequencyinverse	1.0	2.6094379124341005	0.4183633143678389
65	2	document	1.0	2.6094379124341005	0.4183633143678389
66	2	frequency	1.0	2.6094379124341005	0.4183633143678389
67	8	an	1.0	2.203972804325936	0.2848499257850261
68	8	algorithm	1.0	2.6094379124341005	0.337253796525324
69	8	set	1.0	2.6094379124341005	0.337253796525324
70	8	of	1.0	1.916290731874155	0.24766878778426263
71	8	instructions	1.0	2.6094379124341005	0.337253796525324
72	8	designed	1.0	2.6094379124341005	0.337253796525324
73	8	to	1.0	1.916290731874155	0.24766878778426263
74	8	perform	1.0	2.6094379124341005	0.337253796525324
75	8	specific	1.0	2.6094379124341005	0.337253796525324
76	8	task	1.0	2.6094379124341005	0.337253796525324

Rows: 1-76 | Columns: 5

Notice how we have removed the unnecessary words.

We can also see which words were omitted using the stop_words_ attribute:

model.stop_words_

See also

VastColumn.pivot() : pivot VastFrame.

__init__(name: str = None, overwrite_model: bool = False, lowercase: bool = True, vocabulary: Annotated[list | ndarray, 'Array Like Structure'] | None = None, max_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, min_df: Annotated[int | float | Decimal, 'Python Numbers'] | None = None, norm: Literal['l1', 'l2', None] = 'l2', smooth_idf: bool = True, compute_vocabulary: bool = True) → None

Methods

__init__([name, overwrite_model, lowercase, ...])
contour([nbins, chart])	Draws the model's contour plot.
deploySQL([X])	Returns the SQL code needed to deploy the model.
drop()	Drops the model from the VAST DataBase.
export_models(name, path[, kind])	Exports machine learning models.
fit(input_relation, index, x[, return_report])	Applies basic pre-processing.
get_attributes([attr_name])	Returns the model attributes.
get_match_index(x, col_list[, str_check])	Returns the matching index.
get_params()	Returns the parameters of the model.
get_plotting_lib([class_name, chart, ...])	Returns the first available library (Plotly, Matplotlib) to draw a specific graphic.
import_models(path[, schema, kind])	Imports machine learning models.
set_params([parameters])	Sets the parameters of the model.
summarize()	Summarizes the model.
to_binary(path)	Exports the model to the VAST Binary format.
to_python([return_proba, ...])	Returns the Python function needed for in-memory scoring without using built-in VAST functions.
to_sql([X, return_proba, ...])	Returns the SQL code needed to deploy the model without using built-in VAST functions.
transform(vdf, index, x[, pivot])	Transforms input data to tf-idf representation.

Attributes

object_type