CoDet: Yazılı dokümanlarda kapsama ve benzerlik tespiti için yeni bir algoritma

In this thesis, we investigate containment detection, which is a generalized versionof the well known near-duplicate detection problem concerning whether adocument is a subset of another document. In text-based applications, there arethree ways of observing document containment: exact-duplicates, near-duplicates,or containments, where first two are the special cases of containment. To detectcontainments, we introduce CoDet, which is a novel algorithm that focusesparticularly on containment problem. We also construct a test collection using anovel pooling technique, which enables us to make reliable judgments for the relativeeffectiveness of algorithms using limited human assessments. We compare itsperformance with four well-known near duplicate detection methods (DSC, fullfingerprinting, I-Match, and SimHash) that are adapted to containment detection.Our algorithm is especially suitable for streaming news. It is also expandable todifferent domains. Experimental results show that CoDet mostly outperforms theother algorithms and produces remarkable results in detection of containments intext corpora.