Automatic multi-document summarization using original paragraph based extraction technique

Summarization is generally defined to determine the important parts of a document so that shorten the length of the document. Although summarization is exhausting for big documents it is a job could be done by man power. However a document pool has been formed starting by the Internet come into human's life last quarter of the century and it is almost not controllable last decade. An inquiry results in references to the hundreds/thousands resources in this pool. But it is not possible to read all these documents then it would be a convenience work developing a summary from them. Multi-document summarization is a research area that is also commercial beside of being scientific. Summarization can be done by two techniques in the literature. The first one is to present units (sentence, paragraph or text segment) as a summary whose are selected from current document. This method is called extraction technique. The second one is reproducing new sentences from the current sentences in the document. This method is called abstraction technique and it is difficult to apply because it requires natural language processing. Sentence unit is generally preferred for extraction technique. The main reason is that sentence is the smallest meaningful unit in the document. There is a few works use paragraph based extraction in the literature accordingly and not enough research is done. Multi-document summarization requires works about increasing readability (correct sentence ranking) and decreasing the repetitive information. Automatic summaries which are close to human reader summaries can not be produced currently. This doctorate work is about developing a new multi-document summarization technique based on paragraph extraction using English language. Work is fundamentally composed of three phases. The first phase is aimed to construct a vector structure which represents document set better. This part is based on usage of particular percentage of the different words obtained from documents which are more distributed through documents (included in different documents) to use for document set representation vector. The second phase is to determine the outlier documents in the document set which are far away from two sigma (2σ) distance and don't use them in the summarization. The outlier detection done by reader eye is used first time in multi-document summarization scientifically. The third phase is calculating similarities of paragraph term vectors and document set representation vector and presenting paragraphs in order until reach to the length of summary. Similarity measure called matching percent is used in this phase, by this means the effect of paragraph lengths is put away. Thesis is applied to the experimental document sets (DUC 2006) which is constructed by NASA for summaries use sentence based extraction and the ROUGE metrics of obtained summaries use paragraph based extraction are compared with the participant summaries metrics. Success of this paragraph based extraction technique is almost double of DUC 2006 participants. Moreover it is observed that this technique produces successful results for sentence based extraction as much as paragraph based extraction.