Search results for “обучение с подкреплением”

Categories

35 results

Machine Learning 2. Lecture 10. Reinforcement Learning

2026LectureКонстантин Вячеславович Воронцов

Reinforcement learning as learning from the environment's response: the multi-armed bandit problem, estimating the value of an action, the greedy and ε-greedy strategies, the exploration–exploitation trade-off, the UCB method, the exponential moving average, the pursuit method and reinforcement comparison; an environment with states, the Markov decision process, how it differs from ordinary machine learning problems, the discounted return, the state and action value functions, the Bellman equation, the SARSA and Q-learning methods, on-policy and off-policy; deep Q-learning DQN on the example of Atari games, experience replay; gradient optimisation of the policy (policy gradient) and its connection with log-likelihood maximisation; model-based and model-free approaches, a regression model of the reward with an upper confidence bound, LinUCB, recommending news articles, evaluating a policy on historical data.

550

Machine Learning 2. Lecture 11. Active Learning

2026LectureКонстантин Вячеславович Воронцов

Active learning: the model itself chooses which objects to give the teacher for labelling. The problem setting, why active learning is faster than passive learning, learning curves; pool-based sampling, query synthesis and stream-based selective sampling, cost-sensitive methods; uncertainty sampling (least confidence, minimum margin, maximum entropy), query by committee, version space reduction, expected model change, expected error reduction; query synthesis by the Nelder–Mead method and by the variance reduction criterion; density weighting; exploratory actions, the ε-active and EG-active algorithms with the exponentiated gradient; active learning in crowdsourcing: reconciling annotators' labels, models of their reliability and of object difficulty, assigning tasks to annotators.

450

Machine Learning 2. Adaptive Forecasting Methods

2026LectureКонстантин Вячеславович Воронцов

Time series forecasting problems: the time series model, the forecasting horizon, the least squares method; econometrics as the main source of problems — trends, seasonalities, change points; examples of forecasting sales volumes and electricity prices; the linear autoregression model; a brief survey of forecasting methods; adaptive short-term forecasting methods: the exponential moving average and the smoothing parameter, the models of Holt, Theil–Wage, Winters with linear and exponential trend, adaptive autoregression; the tracking signal and a statistical test of model adequacy, the Trigg–Leach model; adaptive selection and adaptive composition of models, the LAVR method — local adaptation of weights with regularisation, the monotonicity constraint as a regulariser; experiments on the sales forecasting problem.

140

Константин Вячеславович Воронцов

@konstantin_vorontsov

Константин Вячеславович Воронцов — профессор РАН, доктор физико-математических наук. Заведующий кафедрой «Математические методы прогнозирования» ВМК МГУ, где с 2021 года — профессор; заведующий лабораторией «Машинное обучение и семантический анализ» Института искусственного интеллекта МГУ (с 2022 года); заведующий кафедрой «Машинное обучение и цифровая гуманитаристика» МФТИ (с 2021 года); заведующий кафедрой «Интеллектуальные системы» МФТИ (с 2022 года), на которой работает с 2004 года; главный научный сотрудник отдела «Интеллектуальные системы» ВЦ ФИЦ ИУ РАН, где работает с 2001 года. Окончил МФТИ (1994) и аспирантуру ВЦ РАН (1997). Кандидатская диссертация «Локальные базисы в алгебраическом подходе к проблеме распознавания» (1999, специальность 01.01.09); докторская диссертация «Комбинаторная теория надёжности обучения по прецедентам» (2010, специальность 05.13.17). Автор комбинаторной теории надёжности обучения по прецедентам, оценивающей вероятность переобучения с учётом эффектов расслоения и сходства в семействах алгоритмов; основное направление его исследований — теория и методы аддитивной регуляризации тематических моделей (ARTM). С 2014 года — научный руководитель разработки библиотеки тематического моделирования с открытым кодом BigARTM, с 2019 года — поисково-рекомендательной системы SciSearch. Один из идеологов и администраторов ресурса MachineLearning.ru (с 2008 года). Медаль РАН с премией для молодых учёных (1999, совместно с Е. В. Нарыжным); почётное звание профессора РАН (2015); юбилейная медаль РАН «300 лет Российской академии наук» (2024); премия Yandex ML Prize в номинации «Зал славы» (2025). Автор курсов «Вероятностные тематические модели», «Математические методы анализа текстов», «Прикладной статистический анализ данных» и спецкурса по теории надёжности обучения по прецедентам; в 2008–2020 годах преподавал в Школе анализа данных Яндекса. С 2004 года читает годовой курс «Машинное обучение» в МФТИ и ВМК МГУ.

Machine Learning 2. Lecture 2. Neural Networks for Unsupervised Learning

2026LectureКонстантин Вячеславович Воронцов

Kohonen networks: clustering as a two-layer neural network, the hard (WTA) and soft (WTM) competition rules, self-organising Kohonen maps and their interpretation; autoencoders: the formulation of the problem, the linear autoencoder and the principal component method, sparse, denoising, contractive, relational and variational autoencoders, reparametrisation, autoencoders for supervised learning, multilayer autoencoders; joint training of several models: pre-training and transfer learning, multi-task learning and few-shot learning, self-supervised learning, distillation and surrogate modelling, learning using privileged information (LUPI), generative adversarial networks (GAN).

210

Machine Learning 1. Lecture 1. Basic Concepts and Examples of Applied Problems

2026LectureКонстантин Вячеславович Воронцов

Objects, features and answers; the types of supervised learning problems; parametric models and the learning method; loss functions and empirical risk minimization; the Runge example and overfitting; hold-out, leave-one-out and cross-validation; applied problems from medical diagnosis to sales forecasting; the CRISP-DM standard.

210

Machine Learning 1. Lecture 15. A Short Review of the Semester

2026LectureКонстантин Вячеславович Воронцов

A closing review of the first semester: empirical risk minimisation and gradient methods; underfitting, overfitting, multicollinearity and regularisation; optimisation formulations of supervised learning (regression, classification, ranking) and unsupervised learning (density, mixtures, clustering, one-class classification, semi-supervised learning, PU-learning, principal component analysis, autoencoders); the six scientific schools of machine learning — symbolism, evolutionism, analogism, Bayesianism, connectionism, compositionism — and their Soviet roots; estimate-calculating algorithms; practice: CRISP-DM, data preprocessing, model evaluation and selection, loss analysis, A/B testing, meta-learning and AutoML.

170

Machine Learning. Additional Lecture 2. A Survey of Optimisation Problems in Machine Learning

2026LectureКонстантин Вячеславович Воронцов

A survey lecture: machine learning as formulations of optimisation problems, a model, a loss function, a regulariser. Regression and classification, continuous approximations of the threshold loss function, SVM and logistic regression, quantile, robust and SVM regression; the L0, L1, L2, Elastic Net, SFM, RFM regularisers and feature selection; learning to rank: ordinal regression, the pairwise approach and AUC, LambdaRank; estimating a density and a mixture of distributions, clustering as a special case of a mixture, the one-class SVM; semi-supervised learning, the transductive SVM, PU-learning; matrix factorisations, autoencoders, multidimensional scaling, graph factorisations, GraphEDM; transfer learning, multi-task and self-supervised learning, distillation, learning using privileged information, generative adversarial networks; a typology of machine learning problems.

210

Machine Learning 2. Lecture 6. Learning to Rank

2026LectureКонстантин Вячеславович Воронцов

The learning-to-rank problem: recovering a partial order relation, the pointwise, pairwise and listwise approaches, the linear ranking model, applications; ordinal regression and the ordinal SVM (SVOR); the pairwise criterion, Ranking SVM, RankNet, the transition to the listwise method LambdaRank; ranking search results: query–document objects, assessor ratings, text, link and click features, TF-IDF and its probabilistic meaning, Okapi BM25, PageRank; quality criteria: precision and recall, MAP, the share of defective pairs and its link with AUC, DCG and NDCG, the pFound user behaviour model; the DSSM neural model: a siamese network, word hashing by letter trigrams, training on clicks with negative sampling.

130

Machine Learning 1. Lecture 2. Stochastic Gradient and the Linear Classifier

2026LectureКонстантин Вячеславович Воронцов

The general scheme of supervised learning for regression, binary and multiclass classification; the margin and continuous approximations of the threshold loss; the linear classifier as a model of a neuron; the stochastic gradient method, momentum, Nesterov's accelerated gradient and heuristics; multicollinearity and L2 regularization; the maximum likelihood principle, its link to empirical risk minimization and the probabilistic meaning of regularization.

130

Machine Learning 1. Lecture 10. Linear Ensembles

2026LectureКонстантин Вячеславович Воронцов

Ensembles of models: base algorithms, the space of estimates, the decision rule and correcting functions; the problem of diversity of base algorithms; bagging, the random subspace method and a common pseudocode of stochastic ensembling; the unbiased out-of-bag estimate and estimating feature importance; converting simple voting into weighted voting; the random forest; boosting for binary classification with abstentions, the exponential approximation of the threshold loss function, the main theorem of AdaBoost with proof, convergence, classical AdaBoost and heuristics; why boosting does not overfit: experiments, bounding the probability of error through margins, the distribution of margins, comparison with bagging; the ComBoost algorithm.

160

Machine Learning 2. Lecture 9. Incremental and Online Learning

2026LectureКонстантин Вячеславович Воронцов

Learning on a data stream: the formulation of the online learning problem, the learning curve, the issues: efficient model updates, making the model more complex as the sample grows, the equivalence of online and offline, storing the sample and catastrophic forgetting; how online learning differs from incremental learning, continual learning, anytime algorithms; lazy learning and online prototype selection; the online naive Bayes classifier with exponential distributions of features; the perceptron, Passive-Aggressive and online gradient descent; the recursive least squares method and the Sherman–Morrison formula; incremental training of decision trees ID5R; online training of an ensemble by the Hedge(β) algorithm and its financial interpretation; online deep learning and Hedge BackProp; incremental learning of new classes, iCaRL.

200

Machine Learning. Additional Lecture 1. The Theory of Overfitting

2026LectureКонстантин Вячеславович Воронцов

The combinatorial theory of overfitting: the probability of overfitting and why the Vapnik–Chervonenkis bounds are loose; experiments with overfitting — a monotone chain of algorithms, the effect of splitting and connectivity of the family on the probability of overfitting and on the complete cross-validation estimate, overfitting of chains, the exact formula for the probability of overfitting when choosing between two algorithms; the splitting–connectivity graph of a family of algorithms, examples for linear classifiers, a monotone chain and a two-dimensional net; generating and forbidden sets, upper connectivity and inferiority of an algorithm; the splitting–connectivity upper bound, the idea of its proof and its properties.

220

Machine Learning 1. Lecture 7. Generalizations of Linear Models

2026LectureКонстантин Вячеславович Воронцов

Nonlinear regression: the Newton–Raphson and Gauss–Newton methods as a series of linear regressions; logistic regression and iteratively reweighted least squares (IRLS); the generalized additive model and backfitting; the probabilistic meaning of least squares; the generalized linear model (GLM), the exponential family of distributions and the link function; logistic regression as a special case of GLM; the median, quantile regression and robust loss functions.

130

Machine Learning 1. Lecture 14. Anomaly Detection and Robust Methods

2026LectureКонстантин Вячеславович Воронцов

Anomaly and novelty detection; heuristic anomaly scores: local density, isolation forest, a mixture with a background component, robust principal component analysis; outlier removal in nonparametric regression, the LOWESS algorithm; anomaly as the value of the loss function; the theory of robust learning: slowly growing functions, iterative reweighting of objects with IRS and IRLS, robust loss functions, the median, the quantile and the censored mean, robust aggregation functions and their smoothing, the IR-ERM algorithm; one-class SVM, learning from positive and unlabeled objects (PU-learning), problems with new and unknown classes.

480

Machine Learning 2. Lecture 3. Vector Representations of Texts and Graphs

2026LectureКонстантин Вячеславович Воронцов

Vector representations (embeddings) of texts and graphs: the distributional semantics hypothesis, syntagmatic and paradigmatic similarity of words, the CBOW and Skip-gram models of the word2vec program, hierarchical SoftMax on a Huffman tree, negative sampling (SGNS) and the connection of word2vec with the matrix factorisation of the PMI matrix, semantic analogies of words, the FastText model on character n-grams; texts as graphs; multidimensional scaling and the stress criterion, stochastic neighbour embedding SNE, perplexity, t-SNE; matrix factorisations of graphs, the random-walk models DeepWalk and node2vec; graph embedding methods as autoencoders, the generalised GraphEDM model, graph neural networks and message passing.

250

Machine Learning 2. Lecture 4. Attention Models and Transformers

2026LectureКонстантин Вячеславович Воронцов

Sequence-to-sequence problems: the recurrent network, the encoder–decoder architecture and its limitations; the recurrent network with an attention model, interpreting the attention matrix in machine translation and image captioning; similarity functions, the query–key–value attention model, the attention formula, self-attention; multi-head attention and its regularisation, hierarchical attention for document classification, the graph attention model GAT; the transformer for machine translation: the encoder, positional encoding, layer normalisation and skip connections, the decoder with masking, the training criterion, BLEU and WER; the transformer encoder BERT, the MLM and NSP criteria, fine-tuning, BPE and WordPiece tokenisation, model sizes.

450

Machine Learning 1. Lecture 8. Feature Selection, Model Evaluation and Model Selection

2026LectureКонстантин Вячеславович Воронцов

Measures of classification quality: type I and type II errors, the ROC curve and AUC, gradient maximization of AUC, log-loss, precision and recall, micro- and macro-averaging, the F-measure; internal and external criteria of generalization ability; hold-out, leave-one-out, q-fold and t×q-fold cross-validation, stratification; model consistency; regularization, AIC, BIC and the Vapnik–Chervonenkis bound; multi-criteria model selection; feature selection: full search, Add, Add-Del, depth-first search, beam search (GMDH), the genetic algorithm and random search with adaptation.

150

Machine Learning 1. Generalization Theory

2026LectureКонстантин Вячеславович Воронцов

The problem of estimating generalization ability; the error matrix and the probability of overfitting; the case of a single algorithm and the hypergeometric distribution; the principle of uniform convergence of frequencies and Boole's inequality; the Vapnik–Chervonenkis bound, the growth function and the capacity of a family of algorithms; inverting the bound and the complexity penalty; structural risk minimization; the splitting and similarity effects; the Occam's razor bound and its link to regularization.

140

Machine Learning 1. Bayesian Classification Theory

2026LectureКонстантин Вячеславович Воронцов

The probabilistic formulation of classification: prior and posterior class probabilities, class likelihood functions, the maximum a posteriori principle; the optimal Bayes classifier and losses from errors; empirical estimation of prior probabilities and class densities; the naive Bayes classifier, features with exponential-family densities and the linear naive Bayes classifier; a survey of Bayesian classifiers: the Parzen window method, the quadratic discriminant, Fisher's linear discriminant, Gaussian mixtures with diagonal covariance matrices, and the radial basis function network.

180