Search results for “градиентный бустинг”

Bias–variance analysis: the decomposition of the error into noise, bias and variance, the bias and variance of simple voting, why complex ensembles do not overfit; gradient boosting for an arbitrary loss function as a parametric approximation of a gradient step; stochastic gradient boosting; special cases: regression, AdaBoost, AnyBoost, variants of loss functions; XGBoost: the tree as a linear model, L0 and L2 regularisation, analytical leaf values and the splitting criterion; CatBoost: leakage in the gradients, ordered boosting on subsamples of doubling length, unbiased target statistics for categorical features.

Learning on a data stream: the formulation of the online learning problem, the learning curve, the issues: efficient model updates, making the model more complex as the sample grows, the equivalence of online and offline, storing the sample and catastrophic forgetting; how online learning differs from incremental learning, continual learning, anytime algorithms; lazy learning and online prototype selection; the online naive Bayes classifier with exponential distributions of features; the perceptron, Passive-Aggressive and online gradient descent; the recursive least squares method and the Sherman–Morrison formula; incremental training of decision trees ID5R; online training of an ensemble by the Hedge(β) algorithm and its financial interpretation; online deep learning and Hedge BackProp; incremental learning of new classes, iCaRL.

Nonlinear regression: the Newton–Raphson and Gauss–Newton methods as a series of linear regressions; logistic regression and iteratively reweighted least squares (IRLS); the generalized additive model and backfitting; the probabilistic meaning of least squares; the generalized linear model (GLM), the exponential family of distributions and the link function; logistic regression as a special case of GLM; the median, quantile regression and robust loss functions.

A closing review of the first semester: empirical risk minimisation and gradient methods; underfitting, overfitting, multicollinearity and regularisation; optimisation formulations of supervised learning (regression, classification, ranking) and unsupervised learning (density, mixtures, clustering, one-class classification, semi-supervised learning, PU-learning, principal component analysis, autoencoders); the six scientific schools of machine learning — symbolism, evolutionism, analogism, Bayesianism, connectionism, compositionism — and their Soviet roots; estimate-calculating algorithms; practice: CRISP-DM, data preprocessing, model evaluation and selection, loss analysis, A/B testing, meta-learning and AutoML.

Reminder: the neuron model, the multilayer network, stochastic gradient, momentum, vanishing and exploding gradients, ReLU, dropout, batch normalisation; depth matters more than width; feature generation and end-to-end learning; convolutional neural networks: convolution and pooling layers, the standard CNN scheme, AlexNet and ImageNet, residual networks ResNet and Highway Networks; convolutional networks for speech, texts and games; recurrent neural networks, training by BPTT, long short-term memory networks LSTM, the peephole variant, GRU.

Kohonen networks: clustering as a two-layer neural network, the hard (WTA) and soft (WTM) competition rules, self-organising Kohonen maps and their interpretation; autoencoders: the formulation of the problem, the linear autoencoder and the principal component method, sparse, denoising, contractive, relational and variational autoencoders, reparametrisation, autoencoders for supervised learning, multilayer autoencoders; joint training of several models: pre-training and transfer learning, multi-task learning and few-shot learning, self-supervised learning, distillation and surrogate modelling, learning using privileged information (LUPI), generative adversarial networks (GAN).

The neuron as a linear model; implementing Boolean functions and the exclusive-OR problem; two-layer networks as universal approximators, Cybenko's theorem; the multilayer network and the backpropagation method; the adaptive gradient methods RMSProp, AdaDelta, Adam, Nadam; gradient explosion and gradient clipping; Dropout, ReLU activation functions, batch normalization, initialization and network pruning (Optimal Brain Damage).