Theses supervised by Doç. Dr. Tevfik Metin Sezgin
10 theses · Koç University
Perceptually validated precise local editing for facial action units with styleGAN
The ability to edit facial expressions has a wide range of applications in computer graphics. The ideal facial expression editing algorithm needs to satisfy two important criteria. First, it should allow precise and targeted editing of individual facial actions. Second, it should generate high fidelity outputs without artifacts. We build a solution based on StyleGAN, which has been used extensively for semantic manipulation of faces. As we do so, we add to our understanding of how various semantic attributes are encoded in StyleGAN. In particular, we show that a naive strategy to perform editing in the latent space results in undesired coupling between certain action units, even if they are conceptually distinct. For example, although brow lowerer and lip tightener are distinct action units, they appear correlated in the training data. Hence, StyleGAN has difficulty in disentangling them. We allow disentangled editing of such action units by computing detached regions of influence for each action unit and restrict editing to these regions. We validate the effectiveness of our local editing method through perception experiments conducted with a total of 43 subjects. The results show that our method provides higher control over local editing and produces images with superior fidelity compared to the state-of-the-art methods.
Domain-adaptive self-supervised pre-training for face & body detection in drawings
Drawings are powerful means of pictorial abstraction and communication. Understanding diverse forms of drawings, including digital arts, cartoons, and comics, has been a major problem of interest for the computer vision and computer graphics communities. Although there are large amounts of digitized drawings from comic books and cartoons, they contain vast stylistic variations, which necessitate expensive manual labeling for training domain-specific recognizers. In this work, I show how self-supervised learning, based on a teacher-student network with a modified student network update design, can be used to build face and body detectors. My setup allows exploiting large amounts of unlabeled data from the target domain when labels are provided for only a small subset of it. I further demonstrate that style transfer can be incorporated into my learning pipeline to bootstrap detectors using a vast amount of out-of-domain labeled images from natural images (i.e., images from the real world). My combined architecture yields detectors with state-of-the-art (SOTA) and near-SOTA performance using minimal annotation effort. Through the utilization of this detector architecture, I accomplish a set of additional tasks. First, I extract a large set of facial drawing images (~1.2 million instances) from unlabeled data and train SOTA generative adversarial network (GAN) models to generate and a SOTA GAN inversion model to reconstruct faces. When the detector-aided data is leveraged, these generative models successfully learn diverse stylistic features. Secondly, I implement an annotation tool to enlarge the existing set of annotated data. This tool offers users to annotate bounding boxes of panels, speech bubbles, narrations, faces, and bodies; to associate text boxes with faces and bodies; to transcript the text; to match the same characters in the image.
Intelligible emotional voice conversion with starGAN assisted by DTW and speaker classifier
Human speech carries not only linguistic content and speaker identity but also emotional content. The ability to alter emotional colouring of speech has the potential to enable a variety of tasks such as producing affective speech for intelligent dialogue systems and guiding people in their emotional expression abilities. While performing emotional voice conversion, special focus needs to be given to both the quality of speech (e.g., naturalness, intelligibility) and the perceived emotion of the generated speech. In this study, we propose a method for converting the emotion of speech across multiple emotion categories with a single trained model. Our StarGAN-based model, enhanced by the DTW algorithm and auxiliary speaker classifier, can change the emotion of a given speech signal into 3 emotion classes: angry, happy and sad. When building our model, we determine the loss functions targeting distinct attributes of speech including the authenticity, linguistic information, speaker identity, and emotional expression. We evaluate the performance of our model through objective and subjective evaluation criteria for both audio quality and emotional content of the converted speech. The results show that our method compares favourably with the state-of-the-art method in terms of both speech quality and emotional articulateness.
Çizim vuruşlarının bölütlenmesi ve çizim tanıma için bütünleşik bir yaklaşım
Hardware supporting pen-based interaction have been around for a long time, however progress in efficient and intelligent processing of input has been lagging far behind. This is partly due to the complicated nature of the sketch recognition problem. Optimal sketch recognition is intractable even for moderate-sized sketches. Recent methods deal with the problem either by making simplifying assumptions or by adopting sub-optimal methods. In this thesis, as an alternative to the sub-optimal methods, we describe an optimal and polynomial-time trainable framework for multi-domain sketch recognition. Our solution handles offline and interspersed sketches as well as online sketches, and it does not make assumptions about user input. Our unified framework is based on supervised machine learning techniques, graph theory, and dynamic programming. We apply the framework to two fundamental problems of bottom-up sketch recognition: stroke fragmentation and sketch segmentation. Dynamic programming approach is directly applicable to the fragmentation of strokes and segmentation of ordered primitives. For other cases, such as offline and interspersed sketches, we introduce the \textit{spatial serialization} concept to impose an order on the primitives. We propose different graph theoretic methods and coherence models to convert 2D points into an ordered set of primitives. We evaluate the accuracy and runtime of different serialization schemes on multiple datasets. For both fragmentation and segmentation problems, experiments show that the accuracy of the unified framework either matches with the state-of-the-art, or it surpasses them by a large margin.
Algısal benzerlik yoluyla serbest formlu çizim sahnelerinin gruplandırılması
Similarity is a fundamental concept in psychology that underlies object recognition, classification and clustering. Psychologists have theorized many explanations for human mind's ability to perceive similarity. Yet, there is still no agreed upon theory, partly because similarity is quite subjective and varies with the features attended by the subject. In this thesis, we explored the concept of similarity for hand-drawn sketches, and address the problem of building a gold standard for assessing similarity and clustering free-form sketch scenes through perceptual similarity. Toward this end, we collected a large dataset consisting of 2400 hand-drawn scenes. We further designed a table-grouping protocol for obtaining a measure of similarity through similarity ratings of human assessors. We verified the validity of the constructed gold standard through inter-rater agreement. We evaluated the performance of the clustering system by measuring the degree of agreement with the constructed gold standard. We obtained high agreement scores, showing that the clustering system operates very similar to human way of grouping sketch scenes.
Az örnek ile çizim tanıma
Sketch recognition is the task of converting hand-drawn digital ink into symbolic computer representations. Since the early days of sketch recognition, the bulk of the work in the domain has focused on building accurate recognition algorithms for specific domains, and well defined databases. These lines of work adopt traditional machine learning approaches. They assume the presence of a fixed set of symbol classes, and availability plenty of annotated examples per class. However, in practice, these assumptions do not hold. In reality, the designer of a sketch recognition system starts with no labeled data at all, and faces the burden of data annotation. In this work, we propose to alleviate the burden of annotation by building systems that can learn from very few labeled examples, and large amounts of unlabeled data. Our systems perform self-learning by automatically extending a very small set of labeled examples with new examples extracted from unlabeled sketches. The end result is a sufficiently large set of labeled training data, which can subsequently be used to train classifiers. We present four self-learning methods with varying levels of implementation difficulty and runtime complexities. One of these methods leverages contextual co-occurrence patterns to build verifiably more diverse set of training instances. Rigorous experiments with large sets of data demonstrate that this novel approach based on exploiting contextual information leads to significant leaps in recognition performance.
Göz bakış hareketlerinin kalem-temelli sistemlere entegrasyonu
In typical human-computer interaction, users convey their intentions through traditional input devices (e.g. keyboards, mice, joysticks) coupled with standard graphical user interfaces elements. Recently, pen-based interaction has emerged as a more intuitive alternative to these traditional means. However, existing pen-based systems are limited by the fact that they rely heavily on auxiliary mode switching mechanisms during interaction (e.g. hard or soft modifier keys, buttons, menus). In this thesis, we describe how eye gaze movements that naturally occur during pen-based interaction can be used to reduce dependency on explicit mode selection mechanisms in pen-based systems. In particular, we show that a range of virtual manipulation commands, that would otherwise require auxiliary mode switching elements, can be issued with an 88% success rate with the aid of users' natural eye gaze behavior during pen-only interaction. We believe the non-intrusive and transparent use of gaze modality as a complementary information channel will bring us closer to the goal of truly intuitive pen-based interaction. To this end, we (1) investigate the nature of gaze behavior during various pen-based interaction scenarios, (2) mine for, extract, and create statistical models for useful and usable gaze behavior patterns while users keep their normal habits and ways to interact, (3) use these models to create fully integrated gaze-based intelligent information visualization systems that are able to dynamically adapt to user's spontaneous task-related intentions and goals, and (4) evaluate these systems via a thorough usability study involving 19 participants and 5 different user interface scenarios. Evaluation results demonstrate that we can successfully establish a shared understanding between the user and the adaptive interface based on users' natural eye gaze behavior, and without interrupting the interaction flow.
Çiz, konuş ve arat: Çok kipli bir video arama sistemi
With the increasing amount of multimedia content available on the web, the focus on video retrieval engines has been shifting from text-based systems to content-based methods that allow indexing and retrieval based on video contents. This trend has sparked a quest for efficient and effective video retrieval systems on large video collections. Most video retrieval systems rely only on hand-crafted features and manual annotations. Motion of the individual objects, the most decisive information conveyed in videos, is usually overlooked in video retrieval. From a user interaction perspective, motion can be given as a query using speech and sketch simultaneously. Speech allows easy specification of content, events and relationships, while sketching brings in spatial expressiveness. Unfortunately, we have insufficient knowledge of how sketching and speech can be used for video retrieval, because there are no existing retrieval systems that support such interaction. In this paper, we describe a Wizard-of-Oz protocol and a set of tools that we have developed to engage users in a sketch- and speech- based video retrieval task. We report how the protocol and the tools fit together to establish an ecologically valid testbed using retrieval of soccer videos as a use case scenario. Using the data collected in the studies, we developed a model capable of interpreting simultaneous speech and sketching to infer the sequence of motions described by a user. The performance results of the model suggest that the protocol and the tools together have the potential to serve as effective means for studying a wide range of multi-modal use cases. Moreover, a video retrieval system was built by integrating the multimodal interpretation model to a database back-end designed for big multimedia collections. The retrieval system was assessed through user evaluation studies. The evaluation results demonstrate that the given query interpretation mechanism and the database system make a good couple for motion-based video retrieval on big video collections.
Diyalog oluşturmada duyguyu dahil etme
Due to its expressivity, natural language is paramount for explicit and implicit affective state communication among humans. The same linguistic inquiry (e.g. How are you?) might induce responses with different affects depending on the affective state of the conversational partner(s) and the context of the conversation. Yet, despite few studies which perceive affect as constitutive aspect of response generation, most of the existing dialog systems produce identical responses to a given inquiry, irrespective of affective information. In this thesis, we introduce AffectON, an approach for integrating affect into dialog generation. For generating language in a targeted affect, our approach leverages a probabilistic language model, an affective space and word embeddings. AffectON is language model agnostic since the probabilistic distribution can originate from any probabilistic language model (i.e. sequence-to-sequence models, neural language models, n-grams). Hence, it can be employed for both affective dialog and affective language generation. We experiment with affective dialog generation and conduct subjective and objective evaluations. For the subjective part of the evaluation, we design a designated user interface for rating and provide recommendations for the design of such interfaces. The results, both subjective and objective show that our approach is successful in pulling the generated language toward the targeted affect, with little sacrifice in syntactic coherence.
Basit çizimleri kullanarak 3 boyutlu model ve animasyon üretimi
Generating 3D models from 2D images or sketches is a widely studied important problem in computer graphics. We describe the first method to generate a 3D human model from a single sketched stick figure. In contrast to the existing human modeling techniques, our method requires neither a statistical body shape model nor a rigged 3D character model. We exploit Variational Autoencoders to develop a novel framework capable of transitioning from a simple 2D stick figure sketch, to a corresponding 3D human model. Our network learns the mapping between the input sketch and the output 3D model. Furthermore, our model learns the embedding space around these models. We demonstrate that our network can generate not only 3D models, but also 3D animations through interpolation and extrapolation in the learned embedding space. In addition to 3D human models, we produce 3D horse models in order to show the generalization ability of our framework. Extensive experiments show that our model learns to generate reasonable 3D models and animations.