The Use of Statistical Characteristics to Reduce the Volume of Textual Information while Preserving Its Informativeness

Abstract

The paper examines the possibility of researchers using specialized algorithms for an information system that provides a reduction in the volume of the analyzed text information in the process of information retrieval. The relevance of the work is justified by the complexity of information retrieval associated with the user’s solution of a particular task and by the need to process large amounts of text data. The goal is to reduce the volume of the analyzed text information of Russian-language texts, while preserving their semantic component. The description of the previously developed information system for reducing the volume of textual information in the process of information retrieval is given. A description of two different approaches to text analysis is presented, which allows for a comparative analysis of the results obtained. These approaches were implemented based on the previously developed information system. Corresponding changes were made in the structural scheme and algorithm of the information system functioning. The results of the experimental study are presented. It follows from the results of the application of this approach that the main part of the proposals corresponding to the user's request for the selected text is shown in the final part of the text, which allows the researcher to pay attention to this part of the analyzed document. Results that can be used to compose abstracts and annotations of analyzed documents are obtained. In the future it is supposed to form author's semantic groups of words that can be used by the researcher to synthesize new knowledge.