Search results for “BERT”

With a soaring number of scientific publications and rapid emergence of new directions and approaches, the scientific community faces the task of timely identification of trends. By a trend, we mean a semantically homogeneous topic characterized by a steady lexical kernel and a sharp, often exponential increase in the number of publications [1]. Examples of trends in machine learning are “LSTM,” “deep learning,” “word2vec,” “BERT,” and “fake news detection.” For real-time detection of trend topics from a stream of scientific publications, we use incremental methods of probabilistic topic modeling. An ARTM-based approach to early trend detection has been shown to outperform popular classical and neural network approaches to this task. A dataset of 91 trends for performance evaluation has been manually collected and made available for public use.

We consider probabilistic topic models and more recent word embedding\ntechniques from a perspective of learning hidden semantic representations.\nInspired by a striking similarity of the two approaches, we merge them and\nlearn probabilistic embeddings with online EM-algorithm on word co-occurrence\ndata. The resulting embeddings perform on par with Skip-Gram Negative Sampling\n(SGNS) on word similarity tasks and benefit in the interpretability of the\ncomponents. Next, we learn probabilistic document embeddings that outperform\nparagraph2vec on a document similarity task and require less memory and time\nfor training. Finally, we employ multimodal Additive Regularization of Topic\nModels (ARTM) to obtain a high sparsity and learn embeddings for other\nmodalities, such as timestamps and categories. We observe further improvement\nof word similarity performance and meaningful inter-modality similarities.\n

Two pathological phenomena for algebraic groups over general bases — a group degenerating between the multiplicative and additive group across a DVR, and a non-affine identity component — built as centralizers in SL_n.