Search results for “Q-обучение”

Reinforcement learning as learning from the environment's response: the multi-armed bandit problem, estimating the value of an action, the greedy and ε-greedy strategies, the exploration–exploitation trade-off, the UCB method, the exponential moving average, the pursuit method and reinforcement comparison; an environment with states, the Markov decision process, how it differs from ordinary machine learning problems, the discounted return, the state and action value functions, the Bellman equation, the SARSA and Q-learning methods, on-policy and off-policy; deep Q-learning DQN on the example of Atari games, experience replay; gradient optimisation of the policy (policy gradient) and its connection with log-likelihood maximisation; model-based and model-free approaches, a regression model of the reward with an upper confidence bound, LinUCB, recommending news articles, evaluating a policy on historical data.

Measures of classification quality: type I and type II errors, the ROC curve and AUC, gradient maximization of AUC, log-loss, precision and recall, micro- and macro-averaging, the F-measure; internal and external criteria of generalization ability; hold-out, leave-one-out, q-fold and t×q-fold cross-validation, stratification; model consistency; regularization, AIC, BIC and the Vapnik–Chervonenkis bound; multi-criteria model selection; feature selection: full search, Add, Add-Del, depth-first search, beam search (GMDH), the genetic algorithm and random search with adaptation.

The general scheme of supervised learning for regression, binary and multiclass classification; the margin and continuous approximations of the threshold loss; the linear classifier as a model of a neuron; the stochastic gradient method, momentum, Nesterov's accelerated gradient and heuristics; multicollinearity and L2 regularization; the maximum likelihood principle, its link to empirical risk minimization and the probabilistic meaning of regularization.

The neuron as a linear model; implementing Boolean functions and the exclusive-OR problem; two-layer networks as universal approximators, Cybenko's theorem; the multilayer network and the backpropagation method; the adaptive gradient methods RMSProp, AdaDelta, Adam, Nadam; gradient explosion and gradient clipping; Dropout, ReLU activation functions, batch normalization, initialization and network pruning (Optimal Brain Damage).

The optimal separating hyperplane and the width of the separating strip; the hinge loss and L2 regularization; the Karush–Kuhn–Tucker conditions, the dual problem and the types of objects; kernels, rectifying spaces and the constructive building of kernels; SVM as a two-layer neural network; choosing the constant C; SVM regression; L1 regularization, Elastic Net SVM, SFM and RVM.

Multiple linear regression and least squares, the pseudoinverse and the geometric meaning of least squares; the solution through the singular value decomposition; multicollinearity and the condition number; ridge regression, choosing the regularization parameter and the effective dimension; LASSO and non-smooth regularizers for feature selection; principal component analysis as a low-rank matrix decomposition; the spectral method of least squares.

Nonlinear regression: the Newton–Raphson and Gauss–Newton methods as a series of linear regressions; logistic regression and iteratively reweighted least squares (IRLS); the generalized additive model and backfitting; the probabilistic meaning of least squares; the generalized linear model (GLM), the exponential family of distributions and the link function; logistic regression as a special case of GLM; the median, quantile regression and robust loss functions.

The problem of estimating generalization ability; the error matrix and the probability of overfitting; the case of a single algorithm and the hypergeometric distribution; the principle of uniform convergence of frequencies and Boole's inequality; the Vapnik–Chervonenkis bound, the growth function and the capacity of a family of algorithms; inverting the bound and the complexity penalty; structural risk minimization; the splitting and similarity effects; the Occam's razor bound and its link to regularization.

Bias–variance analysis: the decomposition of the error into noise, bias and variance, the bias and variance of simple voting, why complex ensembles do not overfit; gradient boosting for an arbitrary loss function as a parametric approximation of a gradient step; stochastic gradient boosting; special cases: regression, AdaBoost, AnyBoost, variants of loss functions; XGBoost: the tree as a linear model, L0 and L2 regularisation, analytical leaf values and the splitting criterion; CatBoost: leakage in the gradients, ordered boosting on subsamples of doubling length, unbiased target statistics for categorical features.

A closing review of the first semester: empirical risk minimisation and gradient methods; underfitting, overfitting, multicollinearity and regularisation; optimisation formulations of supervised learning (regression, classification, ranking) and unsupervised learning (density, mixtures, clustering, one-class classification, semi-supervised learning, PU-learning, principal component analysis, autoencoders); the six scientific schools of machine learning — symbolism, evolutionism, analogism, Bayesianism, connectionism, compositionism — and their Soviet roots; estimate-calculating algorithms; practice: CRISP-DM, data preprocessing, model evaluation and selection, loss analysis, A/B testing, meta-learning and AutoML.

Reminder: the neuron model, the multilayer network, stochastic gradient, momentum, vanishing and exploding gradients, ReLU, dropout, batch normalisation; depth matters more than width; feature generation and end-to-end learning; convolutional neural networks: convolution and pooling layers, the standard CNN scheme, AlexNet and ImageNet, residual networks ResNet and Highway Networks; convolutional networks for speech, texts and games; recurrent neural networks, training by BPTT, long short-term memory networks LSTM, the peephole variant, GRU.

Kohonen networks: clustering as a two-layer neural network, the hard (WTA) and soft (WTM) competition rules, self-organising Kohonen maps and their interpretation; autoencoders: the formulation of the problem, the linear autoencoder and the principal component method, sparse, denoising, contractive, relational and variational autoencoders, reparametrisation, autoencoders for supervised learning, multilayer autoencoders; joint training of several models: pre-training and transfer learning, multi-task learning and few-shot learning, self-supervised learning, distillation and surrogate modelling, learning using privileged information (LUPI), generative adversarial networks (GAN).

Time series forecasting problems: the time series model, the forecasting horizon, the least squares method; econometrics as the main source of problems — trends, seasonalities, change points; examples of forecasting sales volumes and electricity prices; the linear autoregression model; a brief survey of forecasting methods; adaptive short-term forecasting methods: the exponential moving average and the smoothing parameter, the models of Holt, Theil–Wage, Winters with linear and exponential trend, adaptive autoregression; the tracking signal and a statistical test of model adequacy, the Trigg–Leach model; adaptive selection and adaptive composition of models, the LAVR method — local adaptation of weights with regularisation, the monotonicity constraint as a regulariser; experiments on the sales forecasting problem.

Learning on a data stream: the formulation of the online learning problem, the learning curve, the issues: efficient model updates, making the model more complex as the sample grows, the equivalence of online and offline, storing the sample and catastrophic forgetting; how online learning differs from incremental learning, continual learning, anytime algorithms; lazy learning and online prototype selection; the online naive Bayes classifier with exponential distributions of features; the perceptron, Passive-Aggressive and online gradient descent; the recursive least squares method and the Sherman–Morrison formula; incremental training of decision trees ID5R; online training of an ensemble by the Hedge(β) algorithm and its financial interpretation; online deep learning and Hedge BackProp; incremental learning of new classes, iCaRL.

Ensembles of models: base algorithms, the space of estimates, the decision rule and correcting functions; the problem of diversity of base algorithms; bagging, the random subspace method and a common pseudocode of stochastic ensembling; the unbiased out-of-bag estimate and estimating feature importance; converting simple voting into weighted voting; the random forest; boosting for binary classification with abstentions, the exponential approximation of the threshold loss function, the main theorem of AdaBoost with proof, convergence, classical AdaBoost and heuristics; why boosting does not overfit: experiments, bounding the probability of error through margins, the distribution of margins, comparison with bagging; the ComBoost algorithm.

Anomaly and novelty detection; heuristic anomaly scores: local density, isolation forest, a mixture with a background component, robust principal component analysis; outlier removal in nonparametric regression, the LOWESS algorithm; anomaly as the value of the loss function; the theory of robust learning: slowly growing functions, iterative reweighting of objects with IRS and IRLS, robust loss functions, the median, the quantile and the censored mean, robust aggregation functions and their smoothing, the IR-ERM algorithm; one-class SVM, learning from positive and unlabeled objects (PU-learning), problems with new and unknown classes.

Interpretability and explainability of machine learning models: goals, tasks and basic notions, who needs explanations and why; interpretable models; feature importance in linear regression: the t-statistic, net effects, contributions and situational importance; partial dependence plots (PDP) and individual conditional expectation plots (ICE); permutation feature importance; the Shapley value from cooperative game theory, its axioms and Monte Carlo estimation, its use in linear regression; surrogate modelling in the neighbourhood of an object, the LIME, Anchors, SHAP, Shapley Kernel and SAGE methods; the Shapley value for training objects, Data Shapley and Gradient Shapley; counterfactual explanations.

A survey lecture: machine learning as formulations of optimisation problems, a model, a loss function, a regulariser. Regression and classification, continuous approximations of the threshold loss function, SVM and logistic regression, quantile, robust and SVM regression; the L0, L1, L2, Elastic Net, SFM, RFM regularisers and feature selection; learning to rank: ordinal regression, the pairwise approach and AUC, LambdaRank; estimating a density and a mixture of distributions, clustering as a special case of a mixture, the one-class SVM; semi-supervised learning, the transductive SVM, PU-learning; matrix factorisations, autoencoders, multidimensional scaling, graph factorisations, GraphEDM; transfer learning, multi-task and self-supervised learning, distillation, learning using privileged information, generative adversarial networks; a typology of machine learning problems.