Theses supervised by Prof. Dr. Uğur Güdükbay
16 theses · İhsan Doğramacı Bilkent University
İnsan animasyonunda kişilik aktarımı: El yapımı ve veri odaklı yaklaşımlarının karşılaştırılması
The ability to perceive and alter personality traits in animation has significant implications for fields such as character animation and interactive media. Research and developments that use systematic tools or machine learning approaches show that personality can be perceived from different modalities such as audio, images, videos, and motions. Traditionally, handcrafted frameworks have been used to modulate motion and alter perceived personality traits. However, deep learning approaches also offer the potential for more nuanced and automated personality augmentation than handcrafted approaches. To address this evolving landscape, we compare the efficacy of handcrafted models with deep-learning models in altering perceived personality traits in animations. We examined various approaches for personality recognition, motion alteration, and motion generation. We developed two methods for modulating motions to alter OCEAN personality traits based on our findings. The first method is a handcrafted tool that modifies bone positions and rotations using Laban Movement Analysis (LMA) parameters. The second method involves a deep-learning model that separates motion content from personality traits. We could change the overall animation by altering the personality traits through this model. These models are evaluated through a three-part user study, revealing distinct strengths and limitations in both approaches.
Kişilik ifadeli hedefe duyarli kişilerarasi animasyon üretimi
Expressing personalities is essential in human animation for realism and improved communication. We focus on interpersonal animations that portray target-sensitive actions, such as handshaking, to build a data-driven model for personality-based dyadic motion synthesis. However, since datasets with personality annotations are rare, we first introduce personality labels to a subset of animations from well-known datasets with high style variance. To this end, we perform a user study for participants to rate the animation samples regarding perceived personality. We form a set of motion parameters based on various distances, angles, and volumes calculated over different combinations of body joints. Then, we identify the motion parameters that influence personality expression through an in-depth correlation analysis. These motion parameters influence the personality expression in our generative model. We build a transformer-based generative autoencoder that inputs the previous and current poses of a secondary agent, together with the previous pose and the desired personality of the main agent, to output the current pose of the main agent expressing the desired traits. We train this system using dyadic motion samples to learn the semantic actions, also utilizing a randomly selected personality sample for learning to replicate different personality styles based on their motion parameters. We evaluate the generative model by synthesizing motions where the main agent expresses different personality traits. The results suggest that our approach can generate a significant effect on perceived extraversion and agreeableness, while the influence on the remaining factors is limited, likely due to the secondary agent's motion being a limiting factor. Consequently, the resulting model can effectively generate personality-specific poses in time but has less variety based on motion speed.
Üç boyutlu insan-nesne etkileşim animasyonlarında kişilik ifadesi
Personality expression is an essential component of communication between virtual characters and the viewers. Human motion, encompassing temporal facial expressions, body movements, and posture, conveys personalities while controlled by high-level parameters, such as the OCEAN personality model. Research in personality expression, however, focuses solely on isolated behavior, where the subjects don't interact with their environment in a physical sense. Even though object interaction is not a social context, we investigate whether the aforementioned features regarding human motion can be used to communicate the personality of the subjects. Arbitrary object types, actions, and multiple iterations of such interactions are subject to differences in perceived expression, which we quantitatively and qualitatively analyze and synthesize in this work. We applied crowdsourcing to annotate our custom subset of a three-dimensional object interaction dataset and discuss the perceived personalities in multiple object categories. Then, we developed a personality-aware motion augmentation framework to increase the scale and diversity of our dataset. The dataset is used to train a neural motion field-based network architecture to alter a given motion's expressed personality, controlled via OCEAN factors. We validated our approach with a separate user study to assess the resulting motions' personality, accuracy, and realism. We compared the performance of synthetic motions from our neural network and augmentation-only motions. Results suggest that while augmentation-only motions are better at differentiating positive and negative traits of personality, their realism and semantic accuracy are lower than neural-based motions. Neural-based motions' control over the personality via a single factor is observed to be subtle, but multiple factors display a more noticeable difference.
Serebral palsi hastalığına yönelik ergoterapi egzersizlerinin tanınması
Depth camera-based virtual rehabilitation systems are gaining traction in occupational therapy for approaching patients with cerebral palsy. When developing such a system, a domain specific exercise recognition method is vital. In order to design a successful gesture recognition solution for this specific purpose, some obstacles needs to be overcome, namely; detection of gestures that are not related to the defined exercise set and recognition of incorrect exercises that are performed by the patients to compensate for their lack of ability. A combination of solutions, that are based on hidden Markov models, targeting aforementioned obstacles are proposed and elaborated on. The proposed solution works for upper extremity functional exercises and critical compensation mistakes together with restrictions for classifying these mistakes are determined with the help of occupational therapists. Afterwards, we first aim to eliminate the undefined gestures by designing two models that produce adaptive threshold values. Then, we utilize specific negative models based on an approach named feature thresholding and train them specifically for each exercise to distinguish the compensation mistakes. We conducted various tests using our method in a laboratory environment under the supervision of occupational therapists and presented the results of our proposed approach.
Gerçek kalabalık videolarında gezilebilir alanların belirlenmesi, yayaların tespiti ve sanal bireyler eklenmesi
Crowd simulations imitate the behavior of crowds and individual agents in the crowd with personality and appearance, which determines the overall model of a multi-agent system. In such studies, the models are often compared with real-life scenarios for assessment. Yet apart from side-by-side comparison and trajectory analysis, there are no practical, out-of-the-box tools to test how a given arbitrary model simulate the scenario that takes place in the real world. We propose a framework for augmenting virtual agents in real-life crowd videos. The framework locates the navigable areas on the ground plane using the automatically-extracted detection data of the pedestrians in the crowd video. Then it places the three-dimensional (3D) models of real pedestrians in the 3D model of the scene. An interactive user interface is provided for users to add and control virtual agents, which are simulated together with detected real pedestrians using collision avoidance algorithms.
Çok kipli uzun kısa-süreli bellek ve evrişimli sinir ağları ile videoda kişilik tanıma
Personality computing and affective computing, where recognition of personality traits is essential, have gained increasing interest and attention in many research areas recently. The personality traits are described by the Five-Factor Model along five dimensions: openness, conscientiousness, extraversion, agreeableness, and neuroticism. We propose a novel approach to recognize these five personality traits of people from videos. Personality and emotion affect the speaking style, facial expressions, body movements, and linguistic factors in social contexts, and they are affected by environmental elements. For this reason, we develop a multimodal system to recognize apparent personality traits based on various modalities such as the face, environment, audio, and transcription features. In our method, we use modality-specific neural networks that learn to recognize the traits independently and we obtain a final prediction of apparent personality with a feature-level fusion of these networks. We employ pre-trained deep convolutional neural networks such as ResNet and VGGish networks to extract high-level features and Long Short-Term Memory networks to integrate temporal information. We train the large model consisting of modality-specific subnetworks using a two-stage training process. We first train the subnetworks separately and then fine-tune the overall model using these trained networks. We evaluate the proposed method using ChaLearn First Impressions V2 challenge dataset. Our approach obtains the best overall "mean accuracy" score, averaged over five personality traits, compared to the state-of-the-art.
Işın izleme için sınırlayıcı hacim hiyerarşisi-dörtyüzleme hibrit hızlandırıcı yapısı
The computational cost of the ray-tracing method is directly proportional to the number of ray-surface intersection tests. The naive ray-tracing algorithm requires $\mathcal{O}(N)$ computational cost for the ray-surface intersection calculations where $N$ is the number of primitives in the scene. Ray tracing acceleration data structures like the regular grid, bounding volume hierarchy (BVH), kd-tree, constrained tetrahedralization, has been developed to reduce the number of ray-object intersection tests to speed-up ray tracing. We propose a hybrid acceleration structure, the Bounding Volume Hierarchy-Tetrahedral mesh hybrid (BTH) acceleration structure, that can be used to speed-up ray tracing. BTH structure is composed of a BVH hierarchy where some of the leaves of the BVH hierarchy contain tetrahedralizations. We propose an algorithm for the construction of the BTH structure. We describe methods for approximating the average nearest-hit cost of a tetrahedralization, which we use for the construction of BTH. Besides, we can adapt the proposed BTH structure for dynamic scenes with hierarchical motion. We describe a two-level BVH-BTH acceleration structure for rendering animated scenes. We test the proposed BTH structure using various scenes. For some of the experiments, the BTH structure performs better against other acceleration structures in terms of rendering times. We perform experiments for animated scenes. We show that the two-level BTH structure outperforms the two-level BVH structure for the tested dynamic scenes.
Üç boyutlu sahnelerin dörtyüzlü örgüler ile görselleştirilmesi
We propose compact and efficient tetrahedral mesh representations to improve the ray-tracing performance. We reorder tetrahedral mesh data using a space-filling curve to improve cache locality. Most importantly, we propose efficient ray traversal algorithms. We provide details of the regular ray tracing operations on tetrahedral meshes and the Graphics Processing Unit (GPU) implementation of our traversal method. We demonstrate our findings through a set of comprehensive experiments. Our method outperforms existing tetrahedral mesh-based traversal methods and yields comparable results to the traversal methods based on the state-of-the-art acceleration structures such as k-dimensional (k-d) tree and Bounding Volume Hierarchy (BVH) in terms of speed. Storage-wise, our method uses less memory than its tetrahedral mesh-based counterparts, thus allowing larger scenes to be rendered on the GPU. We also describe additional applications of our technique specifically for volume rendering, two-level hybrid acceleration structures for animation purposes, and point queries in two-dimensional (2-D) and three-dimensional (3-D) triangulations. Finally, we present a practical method to tetrahedralize very large scenes.
Bilimsel görüntüleme için hacim boyama yöntemleri
Direct volume rendering is widely used in many applications. We explored several aspects of the problem. First, we proposed a view-dependent selective refinement scheme in order to reduce the high computational requirements without affecting the image quality significantly. Then, we explored the parallel implementations of direct volume rendering. Finally, we used direct volume rendering approaches to create a material visualization tool. Achieving interactive rates in volume-rendering applications is a real challenge. We present a selective refinement scheme that dynamically defines the mesh according to the camera parameters. The importance of different parts of the mesh is automatically determined and the mesh is refined accordingly. Thanks to low overhead dynamic view-dependent refinement, we achieve interactive frame rates for rendering common datasets at decent image resolutions. Using CUDA, we propose a GPU-based volume rendering algorithm that is based on a cell projection-based ray-casting algorithm. We favor image quality over rendering speed. Our algorithm has a low memory footprint, allowing us to render large datasets with significant speed-ups. Visualization of materials is an indispensable part of their structural analysis. We developed a visualization tool called MaterialVis. Unlike existing tools, MaterialVis represents materials as a volume and a surface manifold, in addition to plain atomic coordinates. MaterialVis provides a wide range of functionality to visualize topological structures and crystal defects interactively. Direct volume rendering techniques are used to visualize crystal defects. In addition, the tool provides surface visualization to extract hidden topological features within the material.
Artırılmış gerçeklik için model tabanlı kamera takibi
Augmented reality (AR) is the enhancement of real scenes with virtual entities. It is used to enhance user experience and interaction in various ways. Educational applications, architectural visualizations, military training scenarios and pure entertainment-based applications are often enhanced by augmented reality to provide more immersive and interactive experience for the users. With hand-held devices getting more powerful and cheap, such applications are becoming very popular. To provide natural AR experiences, extrinsic camera parameters (position and rotation) must be calculated in an accurate, robust and efficient way so that virtual entities can be overlaid onto the real environments correctly. Estimating extrinsic camera parameters in real-time is a challenging task. In most camera tracking frameworks, visual tracking serve as the main method for estimating the camera pose. In visual tracking systems, keypoint and edge features are often used for pose estimation. For rich-textured environments, keypoint-based methods work quite well and heavily used. Edge-based tracking, on the other hand, is more preferable when the environment is rich in geometry but has little or no visible texture. Pose estimation for edge based tracking systems generally depends on the control points that are assigned on the model edges. For accurate tracking, visibility of these control points must be determined in a correct manner. Control point visibility determination is computationally expensive process. We propose a method to reduce computational cost of the edge-based tracking by preprocessing the visibility information of the control points. For that purpose, we use persistent control points which are generated in the world space during preprocessing step. Additionally, we use more accurate adaptive projection algorithm for persistent control points to provide more uniform control point distribution in the screen space. We test our camera tracker in different environments to show the effectiveness and performance of the proposed algorithm. The preprocessed visibility information enables constant time calculations of control point visibility while preserving the accuracy of the tracker. We demonstrate a sample AR application with user interaction to present our AR framework, which is developed for a commercially available and widely used game engine.
Artırılmış gerçeklik uygulamaları için hızlandırılmış çekim videolarda güneş pozisyonu tahmini
Realistic illumination of virtual objects on Augmented Reality (AR) environments is important in terms of achieving visual coherence. This thesis proposes a novel approach that facilitates the illumination estimation on time-lapse videos and gives the opportunity to combine AR technology with time-lapse videos in a visually consistent way. The proposed approach works for both outdoor and indoor environments where the main light source is the Sun. We first modify an existing illumination estimation method that aims to obtain sparse radiance map of the environment in order to estimate the initial Sun position. We then track the hard ground shadows on the time-lapse video by using an energy-based pixel-wise method. The proposed method aims to track the shadows by utilizing the energy values of the pixels that forms them. We tested the method on various time-lapse videos recorded in outdoor and indoor environments and obtained successful results.
İlgi noktalarını tahmin ederek sanal kalabalıklar için bakış davranışı simülasyonu
Creating realistic crowd behavior is one of the major goals in crowd simulations. Simulating gaze behavior and predicting interest points of virtual characters play a significant role in creating believable scenes, however this aspect has not received much attention in the field. This study proposes a saliency model, which enables virtual agents to produce gaze behavior. The model measures the effects of distinct pre-defined saliency features that are implemented by examining the state-of-the-art perception studies. When predicting an agent's interest point, we compute the saliency scores by using a weighted sum function for other agents and environment objects in the field of view of the agent for each frame. Then we determine the most salient entity in the virtual scene according to the viewer agent by comparing the scores. We execute this process for each agent in the scene, thus agents gain a visual understanding about their environment. Besides, our model introduces new aspects to crowd perception, such as perceiving characters as groups of people, gaze copy phenomena and effects of agent velocity on attention. For evaluation, we compare the resulting saliency gaze model with real world crowd behavior in captured videos. In the experiments, we simulate the gaze behavior in real crowds. The results show that the proposed approach generates plausible gaze behaviors and is easily adaptable to varying scenarios for virtual crowds.
Kalabalık simülasyonları için iletişim modelleri
Modeling and animation of behaviorally plausible virtual crowds are important problems of crowd simulation research. We propose a communication model in order to equip virtual agents with the ability to autonomously communicate with each other. We investigate whether such a communication model would improve the plausibility of the simulated crowds. Initially, our efforts were towards a model that is as human-like as possible and towards combining this model with an agent architecture that contains psychological attributes. Early experimental results showed that when we look at a crowd, the influences such as different agent personalities causing different communicative behavior are hardly visible. Besides, achieving these effects introduces complexity. Thus, a generic and easy-to-use communication model instead of a human-like one became the target and psychological agent attributes were dropped. The proposed communication model and its application in several scenarios are presented in this dissertation. As a second contribution, one of the application scenarios led us to develop a planning algorithm for an agent in an unknown environment. Simulation results are analyzed both visually and by using various measurements and metrics. Our conclusion is that in addition to improving observed behavioral variety, the effects of employing the communication model are clear in the quantitative results and these effects are in line with our expectations in each scenario.
Gerçek olaylardan öğrenme: Gerçekçi sanal kalabalıklar için veriye dayalı duygu bulaşıcılığı
We propose a data-driven approach for tuning, validating and optimizing crowd simulations by learning parameters from real-life videos. We discuss the common traits of incidents and their video footages suitable for the learning step. We then demonstrate the learning process in three real-life incidents: a bombing attack, a panic in subway and a Black Friday rush. We reanimate the incidents using an existing emotion contagion and crowd simulation framework and optimize the parameters that characterize agent behavior with respect to the data extracted from the video footages of the incidents.
Büyük üç boyutlu üçgensel modellerin bellek verimli kısıtlı Delaunay dörtyüzlemesi
We propose a divide-and-conquer algorithm that can solve the Constrained Delaunay Tetrahedralization (CDT) problem. It consists of three stages: Input Partitioning, Surface Closure, and Merge. We first partition the input into several pieces to reduce the problem size. We apply 2D Triangulation to close the open boundaries to make new pieces watertight. Each piece is then sent to TetGen [Hang Si, TetGen, a Delaunay-Based Quality Tetrahedral Mesh Generator, ACM Transactions on Mathematical Software, Vol. 41, No. 2, Article No. 11, 36 pages, January 2015] for processing. We finally merge each tetrahedral mesh to calculate the final solution. In addition, we apply post-processing to remove vertices we introduced during the input partitioning stage to preserve the input triangles. An alternative approach that does not insert new vertices and eliminates the need for post-processing is also possible but not robust. The benefit of our method is that it can reduce memory usage or increase the speed of the process. It can even tetrahedralize meshes that TetGen cannot do due to the memory's insufficiency. We also observe that this method can increase the overall tetrahedral mesh quality.
Düzensiz hacimsel ağların donanımsal hızlandırıcı yöntemleri ile doğrudan görüntülenmesi
Computational fluid dynamic simulations often produce large clusters of finite elements with non-trivial, non-convex boundaries and uneven distributions among compute nodes, posing challenges to compositing during interactive volume rendering. Correct, in-place visualization of such clusters becomes difficult because viewing rays straddle domain boundaries across multiple compute nodes. We propose a GPU-based, scalable, memory-efficient direct volume visualization framework suitable for in situ and post hoc usage. Our approach reduces memory usage of the unstructured volume elements by leveraging an exclusive or-based index reduction scheme and provides fast ray-marching-based traversal without requiring large external data structures built over the elements. Moreover, we present a GPU-optimized deep compositing scheme that allows correct order compositing of intermediate color values accumulated across different ranks that works even for non-convex clusters. Furthermore, we illustrate that we can achieve secondary effects such as shadows and gradient shading using our method for single GPU setups. Our approach scales well on large data-parallel systems and achieves interactive frame rates during visualization. We can interactively render Fun3D Small Mars Lander (14 GB / 798.4 million finite elements) and Huge Mars Lander (111.57 GB / 6.4 billion finite elements) data sets at 14 and 10 frames per second using 72 and 80 GPUs, respectively, on the Frontera supercomputer at The Texas Advanced Computing Center (TACC).