LibreTimes

Search results for “ARTM”

Categories

Political Philosophy

39 results

Machine Learning 2. Lecture 5. Topic Modelling

2026LectureКонстантин Вячеславович Воронцов

Probabilistic topic modelling of text document collections: the bag-of-words and conditional independence hypotheses, the maximum likelihood criterion, three interpretations of the problem (soft clustering, stochastic matrix factorisation, autoencoder); the lemma on maximising a function on unit simplices; the ill-posedness of the problem and the additive regularisation of topic models ARTM, the EM algorithm and its interpretation; PLSA and LDA, the Dirichlet distribution, the maximum a posteriori probability and giving up Bayesian inference; smoothing, sparsing, background topics, decorrelation; multimodal and multilingual models, unigrams and bigrams; regularisers for classification and regression on texts, for word co-occurrence in short texts, for links between documents and for topic hierarchies.
0
130

Non-Bayesian Additive Regularization for Multimodal Topic Modeling of Large Collections

2015Conference paperKonstantin Vorontsov, Oleksandr Frei, Murat Apishev, Peter Romov +2

Probabilistic topic modeling of text collections is a powerful tool for statistical text analysis based on the preferential use of graphical models and Bayesian learning. Additive regularization for topic modeling (ARTM) is a recent semiprobabilistic approach, which provides a simpler inference for many models previously studied only in the Bayesian settings. ARTM reduces barriers to entry into topic modeling research field and facilitates combination of topic models. In this paper we develop the multimodal extension of ARTM approach and implement it in BigARTM open source project for online parallelized topic modeling. We demonstrate the ability of non-Bayesian regularization to combine modalities, languages and multiple criteria to find sparse, diverse, and interpretable topics.
0
4

Mining Ethnic Content Online with Additively Regularized Topic Models

2016Journal articleMurat Apishev, Sergei Koltcov, Olessia Koltsova, Sergey Nikolenko +1

Computación y Sistemas

Social studies of the Internet have adopted large-scale text mining for unsupervised discovery of topics related to specific subjects. A recently developed approach to topic modeling, additive regularization of topic models (ARTM), provides fast inference and more control over the topics with a wide variety of possible regularizers than developing LDA extensions. We apply ARTM to mining ethnic-related content from Russian-language blogosphere, introduce a new combined regularizer, and compare models derived from ARTM with LDA. We show with human evaluations that ARTM is better for mining topics on specific subjects, finding more relevant topics of higher or comparable quality.
0
3

Regularized Multimodal Hierarchical Topic Model for Document-by-Document Exploratory Search

2019Conference paperAnastasia Ianina, Konstantin Vorontsov

In the exploratory search paradigm of information retrieval, the user has a complicated search demand that can not be formulated in a short query. The user collects thematically relevant information iteratively in a “query-browse-refine” process being motivated by learning, understanding, and knowledge acquisition purposes. We consider an elementary step of this scenario in which the search intent can be expressed by a long text query. For this case, we develop an exploratory search engine based on probabilistic topic modeling. Topic model gives a low-dimensional sparse interpretable vector representation (topical embedding) of a text. The search engine uses these embeddings for ranking documents by their similarity to the query. We show that performing only one query, the topic-based search engine achieves better precision and recall that human assessors do spending up to one hour in a conventional browse-refine loop. We use additive regularization for topic modeling (ARTM) to make the model simultaneously sparse, decorrelated, n-gram, multimodal and hierarchical. We show experimentally that each of these features of the model is important to achieve precision and recall higher than 90
0
4

Learning Topic Models with Arbitrary Loss

2020Conference paperMurat Apishev, Konstantin Vorontsov

Topic modeling is an area of text analysis actively developing over the past 20 years. A probabilistic topic model (PTM) finds a set of hidden topics from a collection of text documents. It defines each topic as a probability distribution over words and describes each document as a probability mixture of topic distributions. Learning algorithms for topic models are usually based on Bayesian inference or log-likelihood maximization. In both cases, EM-like algorithms are used. In this paper, we propose to replace the logarithm in the log-likelihood by an arbitrary smooth loss function. We prove that such a modification preserves both the structure of the algorithm and compatibility with any regularizers in terms of additive regularization of topic models (ARTM). Moreover, in the case of a linear loss, the Estep becomes much faster due to the omission of a normalization. We study combinations of the fast and usual E-steps and compare them to regularization using different number of topics in both offline and online versions of EM-algorithm. For an empirical comparison of the algorithms, we estimate perplexity, coherence, and learning time. We use an efficient parallel implementation of the EM-algorithm from the BigARTM open-source library. We show that in most cases the two-stage strategy wins, which uses fast E-steps at the beginning of iterations, then proceeds with usual E-steps.
0
6

Incremental Learning of Topic Models for Finding Trend Topics in Scientific Publications

2022Journal articleN. A. Gerasimenko, A. S. Chernyavsky, M. A. Nikiforova, M. D. Nikitin +1

Doklady Mathematics

With a soaring number of scientific publications and rapid emergence of new directions and approaches, the scientific community faces the task of timely identification of trends. By a trend, we mean a semantically homogeneous topic characterized by a steady lexical kernel and a sharp, often exponential increase in the number of publications [1]. Examples of trends in machine learning are “LSTM,” “deep learning,” “word2vec,” “BERT,” and “fake news detection.” For real-time detection of trend topics from a stream of scientific publications, we use incremental methods of probabilistic topic modeling. An ARTM-based approach to early trend detection has been shown to outperform popular classical and neural network approaches to this task. A dataset of 91 trends for performance evaluation has been manually collected and made available for public use.

0
7

Incremental Topic Modeling for Scientific Trend Topics Extraction

2023Conference paperNikolai Gerasimenko, Alexander Chernyavskiy, Maria Nikiforova, Anastasia Ianina +1

Computational Linguistics and Intellectual Technologies

Rapid growth of scientific publications and intensive emergence of new directions and approaches poses a challenge to the scientific community to identify trends in a timely and automatic manner. We denote trend as a semantically homogeneous theme that is characterized by a lexical kernel steadily evolving in time and a sharp, often exponential, increase in the number of publications. In this paper, we investigate recent topic modeling approaches to accurately extract trending topics at an early stage. In particular, we customize the standard ARTM-based approach and propose a novel incremental training technique which helps the model to operate on data in real-time. We further create the Artificial Intelligence Trends Dataset (AITD) that contains a collection of early-stage articles and a set of key collocations for each trend. The conducted experiments demonstrate that the suggested ARTM-based approach outperforms the classic PLSA, LDA models and a neural approach based on BERT representations. Our models and dataset are open for research purposes.
0
4

Convergence of the Algorithm of Additive Regularization of Topic Models

2021Journal articleI. A. Irkhin, K. V. Vorontsov

Proceedings of the Steklov Institute of Mathematics

The problem of probabilistic topic modeling is as follows. Given a collection of text documents, find the conditional distribution over topics for each document and the conditional distribution over words (or terms) for each topic. Log-likelihood maximization is used to solve this problem. The problem generally has an infinite set of solutions and is ill-posed according to Hadamard. In the framework of Additive Regularization of Topic Models (ARTM), a weighted sum of regularization criteria is added to the main log-likelihood criterion. The numerical method for solving this optimization problem is a kind of an iterative EM-algorithm written in a general form for an arbitrary smooth regularizer as well as for a linear combination of smooth regularizers. This paper studies the problem of convergence of the EM iterative process. Sufficient conditions are obtained for the convergence to a stationary point of the regularized log-likelihood. The constraints imposed on the regularizer are not too restrictive. We give their interpretations from the point of view of the practical implementation of the algorithm. A modification of the algorithm is proposed that improves the convergence without additional time and memory costs. Experiments on a news text collection have shown that our modification both accelerates the convergence and improves the value of the criterion to be optimized.
0
3

Interpretable probabilistic embeddings: bridging the gap between topic\n models and neural networks

2017PreprintPotapenko, Anna, Popov, Artem, Vorontsov, Konstantin

arXiv (Cornell University)

We consider probabilistic topic models and more recent word embedding\ntechniques from a perspective of learning hidden semantic representations.\nInspired by a striking similarity of the two approaches, we merge them and\nlearn probabilistic embeddings with online EM-algorithm on word co-occurrence\ndata. The resulting embeddings perform on par with Skip-Gram Negative Sampling\n(SGNS) on word similarity tasks and benefit in the interpretability of the\ncomponents. Next, we learn probabilistic document embeddings that outperform\nparagraph2vec on a document similarity task and require less memory and time\nfor training. Finally, we employ multimodal Additive Regularization of Topic\nModels (ARTM) to obtain a high sparsity and learn embeddings for other\nmodalities, such as timestamps and categories. We observe further improvement\nof word similarity performance and meaningful inter-modality similarities.\n

0
5

Fast and modular regularized topic modelling

2017Conference paperDenis Kochedykov, Murat Apishev, Lev Golitsyn, Konstantin Vorontsov

Topic modelling is an area of text mining that has been actively developed in the last 15 years. A probabilistic topic model extracts a set of hidden topics from a collection of text documents. It defines each topic by a probability distribution over words and describes each document with a probability distribution over topics. In applications, there are often many requirements, such as, for example, problem-specific knowledge and additional data, to be taken into account. Therefore, it is natural for topic modelling to be considered a multiobjective optimization problem. However, historically, Bayesian learning became the most popular approach for topic modelling. In the Bayesian paradigm, all requirements are formalized in terms of a probabilistic generative process. This approach is not always convenient due to some limitations and technical difficulties. In this work, we develop a non-Bayesian multiobjective approach called the Additive Regularization of Topic Models (ARTM). It is based on regularized Maximum Likelihood Estimation (MLE), and we show that many of the well-known Bayesian topic models can be re-formulated in a much simpler way using the regularization point of view. We review some of the most important types of topic models: multimodal, multilingual, temporal, hierarchical, graph-based, and short-text. The ARTM framework enables easy combination of different types of models to create new models with the desired properties for applications. This modular “lego-style” technology for topic modelling is implemented in the open-source library BigARTM.
0
6

Iterative Improvement of an Additively Regularized Topic Model

2024PreprintGorbulev, Alex, Alekseev, Vasiliy, Vorontsov, Konstantin

arXiv (Cornell University)

Topic modelling is fundamentally a soft clustering problem (of known objects – documents, over unknown clusters – topics). That is, the task is incorrectly posed. In particular, the topic models are unstable and incomplete. All this leads to the fact that the process of finding a good topic model (repeated hyperparameter selection, model training, and topic quality assessment) can be particularly long and labor-intensive. We aim to simplify the process, to make it more deterministic and provable. To this end, we present a method for iterative training of a topic model. The essence of the method is that a series of related topic models are trained so that each subsequent model is at least as good as the previous one, i.e., that it retains all the good topics found earlier. The connection between the models is achieved by additive regularization. The result of this iterative training is the last topic model in the series, which we call the iteratively updated additively regularized topic model (ITAR). Experiments conducted on several collections of natural language texts show that the proposed ITAR model performs better than other popular topic models (LDA, ARTM, BERTopic), its topics are diverse, and its perplexity (ability to "explain" the underlying data) is moderate.
0
4
Константин Вячеславович Воронцов
@konstantin_vorontsov

Константин Вячеславович Воронцов — профессор РАН, доктор физико-математических наук. Заведующий кафедрой «Математические методы прогнозирования» ВМК МГУ, где с 2021 года — профессор; заведующий лабораторией «Машинное обучение и семантический анализ» Института искусственного интеллекта МГУ (с 2022 года); заведующий кафедрой «Машинное обучение и цифровая гуманитаристика» МФТИ (с 2021 года); заведующий кафедрой «Интеллектуальные системы» МФТИ (с 2022 года), на которой работает с 2004 года; главный научный сотрудник отдела «Интеллектуальные системы» ВЦ ФИЦ ИУ РАН, где работает с 2001 года. Окончил МФТИ (1994) и аспирантуру ВЦ РАН (1997). Кандидатская диссертация «Локальные базисы в алгебраическом подходе к проблеме распознавания» (1999, специальность 01.01.09); докторская диссертация «Комбинаторная теория надёжности обучения по прецедентам» (2010, специальность 05.13.17). Автор комбинаторной теории надёжности обучения по прецедентам, оценивающей вероятность переобучения с учётом эффектов расслоения и сходства в семействах алгоритмов; основное направление его исследований — теория и методы аддитивной регуляризации тематических моделей (ARTM). С 2014 года — научный руководитель разработки библиотеки тематического моделирования с открытым кодом BigARTM, с 2019 года — поисково-рекомендательной системы SciSearch. Один из идеологов и администраторов ресурса MachineLearning.ru (с 2008 года). Медаль РАН с премией для молодых учёных (1999, совместно с Е. В. Нарыжным); почётное звание профессора РАН (2015); юбилейная медаль РАН «300 лет Российской академии наук» (2024); премия Yandex ML Prize в номинации «Зал славы» (2025). Автор курсов «Вероятностные тематические модели», «Математические методы анализа текстов», «Прикладной статистический анализ данных» и спецкурса по теории надёжности обучения по прецедентам; в 2008–2020 годах преподавал в Школе анализа данных Яндекса. С 2004 года читает годовой курс «Машинное обучение» в МФТИ и ВМК МГУ.

Does Annotating Multi-Spans Improve Classification in Considerable Text Collections?

2024Conference paperArchil Maysuradze, Olga Rink, Artem Fedorov, Andrey Tabachenkov +1

The Universal data markup structures empower the annotation of multiple text fragments (multi-spans or multi-fragments furthermore) to analyze large collections of content. Multi-spans have demonstrated helpful in tackling issues related to the programmed discovery of semantic blunders in school essays or human values in social media writings. Labeling multi-fragment information has made it conceivable to form an interdisciplinary classification of human values. This classifier consists of 105 labels grouped into 7 categories, and a corresponding dataset has been created. Subsequent ML experiments have been designed to demonstrate the effectiveness of the multi-spans structure in recovering annotations of human values. The accuracy of the multi-fragment detector is 0.943 for material values and 0.957 for legal awareness (a subject of civic engagement and citizenship).
0
4

Topic Balancing with Additive Regularization of Topic Models

2020Conference paperEugeniia Veselova, Konstantin Vorontsov

This article proposes a new approach for building topic models on unbalanced collections in topic modelling, based on the existing methods and our experiments with such methods. Real-world data collections contain topics in various proportions, and often documents of the relatively small theme become distributed all over the larger topics instead of being grouped into one topic. To address this issue, we design a new regularizer for and matrices in probabilistic Latent Semantic Analysis (pLSA) model. We make sure this regularizer increases the quality of topic models, trained on unbalanced collections. Besides, we conceptually support this regularizer by our experiments.
0
6

Communicative Type “Municipal Employee” in the Media Space: Development of an Automatic Information and Analytical Assessment System

2024Journal articleIrina Karabulatova, Konstantin Vorontsov, Daniil Okolyshev, Ludan Zhang

Vestnik Volgogradskogo gosudarstvennogo universiteta Serija 2 Jazykoznanije

The article examines the issue of representing municipal government in the media space, followed by the proposed solution for automatically identifying signs of destructive and constructive positioning of communicative types of municipal employees in the public information space. The definition of the concept of the communicative type “municipal employee” with verification features is introduced. The results of the analysis of the organization of local self-government on the example of the Moscow region allowed us to conclude that the communicative type “municipal employee” reflects a diversified system of territorial communicative position within the regional government. The information obtained during the analysis of public information space attitudes regarding the activities of municipal employees can be automated with the method of identifying linguistic markers of emotivity to determine the communicative position of territorial authorities. The suggested methodology for effective automation of the studied subject area in the humanities has been verified as possessing a high scientific potential for further research. It is concluded that the development of technology for monitoring and forecasting public threats based on “soft power” methods through automatic and expert work to identify markers of evaluative presentation of communicative types of municipal employees is designed to help regional authorities achieve the desired results in ensuring territorial identity.
0
5

Sharpness Estimation of Combinatorial Generalization Ability Bounds for Threshold Decision Rules

2021Journal articleSh. Kh. Ishkina, K. V. Vorontsov

Automation and Remote Control

This article is devoted to the problem of calculating an exact upper bound for the functionals of the generalization ability of a family of one-dimensional threshold decision rules. An algorithm is investigated that solves the stated problem and is polynomial in the total number of samples used for training and validation and in the number of training samples. A theorem is proved for calculating an estimate for the functional of expected overfitting and an estimate for the error rate of the method for minimizing empirical risk on a validation set. The exact bounds calculated using the theorem are compared with the previously known quick-to-compute upper bounds so as to estimate the orders of overestimation of the bounds and to identify the bounds that could be used in real problems.
0
6