Search results for “EM-алгоритм”

Estimating a probability density from a sample: the maximum likelihood principle, the multivariate Gaussian density and its geometric meaning, multicollinearity and regularisation of the covariance matrix; separating a mixture of distributions, the EM algorithm as a method of simple iterations, the Gaussian mixture (GMM); the nonparametric Parzen–Rosenblatt estimate, the choice of kernel and window width; Bayesian classification theory: the generative and discriminative approaches, the optimal Bayes classifier, the Parzen window method, the quadratic discriminant, Fisher's linear discriminant, the radial basis function network and its training by the EM algorithm.

Probabilistic topic modelling of text document collections: the bag-of-words and conditional independence hypotheses, the maximum likelihood criterion, three interpretations of the problem (soft clustering, stochastic matrix factorisation, autoencoder); the lemma on maximising a function on unit simplices; the ill-posedness of the problem and the additive regularisation of topic models ARTM, the EM algorithm and its interpretation; PLSA and LDA, the Dirichlet distribution, the maximum a posteriori probability and giving up Bayesian inference; smoothing, sparsing, background topics, decorrelation; multimodal and multilingual models, unigrams and bigrams; regularisers for classification and regression on texts, for word co-occurrence in short texts, for links between documents and for topic hierarchies.

The clustering problem and its ill-posedness, the aims of clustering, types of cluster structure, sensitivity to the choice of metric; the problem of semi-supervised learning and transductive learning, why it reduces neither to classification nor to clustering; criteria of clustering quality, the silhouette coefficient, the BCubed measures of precision and recall; the k-means method (Lloyd's algorithm) as a simplification of the EM algorithm for a mixture of Gaussians; the DBSCAN algorithm; agglomerative hierarchical clustering, the Lance–Williams formula and its special cases, the dendrogram, monotonicity, contracting and dilating distances, Ward's distance; semi-supervised learning based on classification: self-training, co-training, co-learning, the transductive SVM, expectation regularisation (XR).

Anomaly and novelty detection; heuristic anomaly scores: local density, isolation forest, a mixture with a background component, robust principal component analysis; outlier removal in nonparametric regression, the LOWESS algorithm; anomaly as the value of the loss function; the theory of robust learning: slowly growing functions, iterative reweighting of objects with IRS and IRLS, robust loss functions, the median, the quantile and the censored mean, robust aggregation functions and their smoothing, the IR-ERM algorithm; one-class SVM, learning from positive and unlabeled objects (PU-learning), problems with new and unknown classes.

Topic modeling is an area of text analysis actively developing over the past 20 years. A probabilistic topic model (PTM) finds a set of hidden topics from a collection of text documents. It defines each topic as a probability distribution over words and describes each document as a probability mixture of topic distributions. Learning algorithms for topic models are usually based on Bayesian inference or log-likelihood maximization. In both cases, EM-like algorithms are used. In this paper, we propose to replace the logarithm in the log-likelihood by an arbitrary smooth loss function. We prove that such a modification preserves both the structure of the algorithm and compatibility with any regularizers in terms of additive regularization of topic models (ARTM). Moreover, in the case of a linear loss, the Estep becomes much faster due to the omission of a normalization. We study combinations of the fast and usual E-steps and compare them to regularization using different number of topics in both offline and online versions of EM-algorithm. For an empirical comparison of the algorithms, we estimate perplexity, coherence, and learning time. We use an efficient parallel implementation of the EM-algorithm from the BigARTM open-source library. We show that in most cases the two-stage strategy wins, which uses fast E-steps at the beginning of iterations, then proceeds with usual E-steps.

Real-time monitoring of scientific papers and technological news requires fast processing of complicated search demands motivated by thematically relevant information acquisition. For this case, the authors develop an exploratory search engine based on probabilistic hierarchical topic modeling. Topic model gives a low dimensional sparse interpretable vector representation (topical embedding) of a text, which is used for ranking documents by their similarity to the query. They explore several ways of comparing topical vectors including searching with thematically homogeneous text segments. Topical hierarchies are built using the regularized EM-algorithm from BigARTM project. The topic-based search achieves better precision and recall than other approaches (TF-IDF, fastText, LSTM, BERT) and even human assessors who spend up to an hour to complete the same search task. They also discover that blending hierarchical topic vectors with neural pretrained embeddings is a promising way of enriching both models that helps to get precision and recall higher than 90

The problem of probabilistic topic modeling is as follows. Given a collection of text documents, find the conditional distribution over topics for each document and the conditional distribution over words (or terms) for each topic. Log-likelihood maximization is used to solve this problem. The problem generally has an infinite set of solutions and is ill-posed according to Hadamard. In the framework of Additive Regularization of Topic Models (ARTM), a weighted sum of regularization criteria is added to the main log-likelihood criterion. The numerical method for solving this optimization problem is a kind of an iterative EM-algorithm written in a general form for an arbitrary smooth regularizer as well as for a linear combination of smooth regularizers. This paper studies the problem of convergence of the EM iterative process. Sufficient conditions are obtained for the convergence to a stationary point of the regularized log-likelihood. The constraints imposed on the regularizer are not too restrictive. We give their interpretations from the point of view of the practical implementation of the algorithm. A modification of the algorithm is proposed that improves the convergence without additional time and memory costs. Experiments on a news text collection have shown that our modification both accelerates the convergence and improves the value of the criterion to be optimized.

In the exploratory search paradigm of information retrieval, the user has a complicated search demand that can not be formulated in a short query. The user collects thematically relevant information iteratively in a “query-browse-refine” process being motivated by learning, understanding, and knowledge acquisition purposes. We consider an elementary step of this scenario in which the search intent can be expressed by a long text query. For this case, we develop an exploratory search engine based on probabilistic topic modeling. Topic model gives a low-dimensional sparse interpretable vector representation (topical embedding) of a text. The search engine uses these embeddings for ranking documents by their similarity to the query. We show that performing only one query, the topic-based search engine achieves better precision and recall that human assessors do spending up to one hour in a conventional browse-refine loop. We use additive regularization for topic modeling (ARTM) to make the model simultaneously sparse, decorrelated, n-gram, multimodal and hierarchical. We show experimentally that each of these features of the model is important to achieve precision and recall higher than 90

A survey lecture: machine learning as formulations of optimisation problems, a model, a loss function, a regulariser. Regression and classification, continuous approximations of the threshold loss function, SVM and logistic regression, quantile, robust and SVM regression; the L0, L1, L2, Elastic Net, SFM, RFM regularisers and feature selection; learning to rank: ordinal regression, the pairwise approach and AUC, LambdaRank; estimating a density and a mixture of distributions, clustering as a special case of a mixture, the one-class SVM; semi-supervised learning, the transductive SVM, PU-learning; matrix factorisations, autoencoders, multidimensional scaling, graph factorisations, GraphEDM; transfer learning, multi-task and self-supervised learning, distillation, learning using privileged information, generative adversarial networks; a typology of machine learning problems.