ruSciFact: Open Benchmark for Verifying Scientific Facts in Russian

Against the backdrop of active LLM development, their tendency to hallucinate, as well as the growing volume of texts they generate, the validation of facts has become increasingly important and relevant.We propose ruSciFact 1 , a new benchmark for fact-checking scientific claims in Russian.ruSciFact is structured as a NLI task, the goal is to verify whether a fact is confirmed by a given abstract.To generate facts, we used an 3-step pipeline based on LLaMA-405B, validating the resulting sentences with the help of assessors-terminologists.The ruSciFact dataset consists of 1128 pairs in the format , which we are releasing as open source together with the benchmark code.Additionally, we are opensourcing the fact-generation pipeline 2 , which facilitates the expansion of the dataset to specific scientific domains.We evaluated several popular language models on ruSciFact, including text embedders and generative models.The results show that this benchmark allows to effectively assess the fact-checking capabilities of LLMs in Russian.