181
Archived Theses
0
DOIs Assigned
0%
DOI Rate
Discipline
Domain-adaptive self-supervised pre-training for face & body detection in drawings
Drawings are powerful means of pictorial abstraction and communication. Understanding diverse forms of drawings, including digital arts, cartoons, and comics, has been a major problem of interest for the computer vision and computer graphics communities. Although there are large amounts of digitized drawings from comic books and cartoons, they contain vast stylistic variations, which necessitate expensive manual labeling for training domain-specific recognizers. In this work, I show how self-supervised learning, based on a teacher-student network with a modified student network update design, can be used to build face and body detectors. My setup allows exploiting large amounts of unlabeled data from the target domain when labels are provided for only a small subset of it. I further demonstrate that style transfer can be incorporated into my learning pipeline to bootstrap detectors using a vast amount of out-of-domain labeled images from natural images (i.e., images from the real world). My combined architecture yields detectors with state-of-the-art (SOTA) and near-SOTA performance using minimal annotation effort. Through the utilization of this detector architecture, I accomplish a set of additional tasks. First, I extract a large set of facial drawing images (~1.2 million instances) from unlabeled data and train SOTA generative adversarial network (GAN) models to generate and a SOTA GAN inversion model to reconstruct faces. When the detector-aided data is leveraged, these generative models successfully learn diverse stylistic features. Secondly, I implement an annotation tool to enlarge the existing set of annotated data. This tool offers users to annotate bounding boxes of panels, speech bubbles, narrations, faces, and bodies; to associate text boxes with faces and bodies; to transcript the text; to match the same characters in the image.
Learning markerless robot-depth camera calibration and end-effector pose estimation
Robot arms are being used more and more in unstructured environments. As such, they are relying more on vision sensors compared to traditional factory robots which are placed in highly structured, controlled and caged work-cells. Some applications that rely on vision data include bin-picking, box picking and placing, assembly and part feeding in mixed human-robot work cells, inspection, quality control etc. Vision data is mainly required to localize the objects to be manipulated, perform measurements and detect near-by humans. Vision based robot systems require extrinsic calibration between the robot and camera in order to work properly. This is a time consuming and tedious procedure which can be expensive as well. Fast, flexible and precise robot-camera calibration is essential for not only industrial environments but also academic lab environments where the location of the camera and/or robot needs to be frequently or is accidentally changed. Extrinsic calibration between a robot arm and camera is a decades old challenge still prevalent to this day. Traditional techniques work by estimating pose of the camera relative to a fiducial marker from multiple points and matching these estimations with the robot's pose. Recent learning based approaches predict extrinsic calibration from images relying heavily on simulation data. In this thesis, we present a learning based markerless extrinsic calibration system that uses a depth camera. We learn models for end-effector (EE) segmentation, single-frame rotation prediction and keypoint detection, from automatically generated real-world data. Our models are based on MinkUNet and PointNet++ architectures. We use a transformation trick to get EE pose estimates from rotation predictions and a matching algorithm to get EE pose estimates from keypoint predictions. We further utilize the iterative closest point (ICP) algorithm, multiple-frames and outlier detection to increase calibration robustness. Our results on the test set with previously unseen camera locations give sub-centimeter (0.74 cm) and less than 0.05 radians (1.69 degrees) average calibration errors and 1.00 cm and 2.74 degrees average pose estimation errors. In addition, we released an open source easy to use tool for robot users to handle robot-camera calibration with a few mouse clicks by seamlessly integrating all the models and algorithms discussed in this thesis.
Addressing the static scene assumption and the scale ambiguity in self-supervised monocular depth estimation
Self-supervised monocular depth estimation is the task of estimating per-pixel depth from a single image without any supervision. Typically, there are two networks to estimate depth and camera pose between consecutive frames, which are then used to reconstruct one view from another for self-supervision. There are two problems with this approach. Firstly, the scene is assumed to be static and the only motion is due to the camera, namely the static scene assumption, however, this is frequently violated in real-world driving scenarios. Due to this assumption, monocular depth methods struggle to produce accurate predictions in the moving regions of the scene. Current methods either ignore moving regions or require an additional instance segmentation input to identify and separately process moving regions. In this thesis, we first propose MonoDepthSeg to jointly estimate the depth and decompose the scene into moving regions to model the motion of dynamic objects. We show that going beyond the static scene assumption improves the accuracy of depth prediction, especially in moving regions. The second problem of self-supervised monocular depth estimation methods is the scale ambiguity. The estimated depth values are in an unknown scale which is typically handled with normalization with respect to the ground truth scale value during inference. We revisit the traditional paradigm of plane and parallax to address this issue and propose DepthP+P to estimate depth in metric scale. Our method shows promising results that are metric scale without any additional normalization.
Two-level temporal relation model for video instance segmentation
In Video Instance Segmentation (VIS), current approaches either focus on the quality of the results, by taking the whole video as input and processing it offline; or on speed, by handling it frame by frame at the cost of competitive performance. In this work, we propose an online method that is on par with the performance of the offline counterparts. We introduce a message-passing graph neural network that encodes objects and relates them through time. We additionally propose a novel module to fuse features from the feature pyramid network with residual connections. Our model, trained end-to-end, achieves state-of-the-art performance on the YouTube-VIS dataset within the online methods. Further experiments on DAVIS demonstrate the generalization capability of our model to the video object segmentation task. We also evaluate our work on autonomous driving setting and show comparable results in KITTI MOTS dataset.
Answering spatial density queries under local differential privacy
Spatial density queries are fundamental in many geospatial data analysis tasks and have numerous applications in the real world, such as determining crowded areas, estimating traffic density, navigation, passenger demand analysis, and so forth. However, answering spatial density queries based on users' data may violate users' privacy by exposing their true locations to an untrusted third party (e.g., a server acting as the service provider or data collector). In this thesis, we propose a solution for answering spatial density queries while preserving Local Differential Privacy (LDP), a state-of-the-art privacy protection standard. Our solution consists of four main steps: partitioning, finding sensitivity, user-side noisy response computation, and server-side estimation. For the first step, we initially propose three basic partitioning strategies: Singleton Partitioning, Holistic Partitioning and Random Partitioning. Based on our qualitative and empirical analysis of the three basic strategies, we design and implement an improved strategy called Advanced Partitioning. For the second step, we adapt graph-based modeling of query sets from the centralized DP literature. Advanced Partitioning also leverages and extends this technique by formulating the partitioning problem as a vertex coloring problem on the graph representation of a query set. For the third and fourth steps, in addition to adapting two popular LDP protocols (GRR and RAPPOR) to our solution, we propose an extension for the Optimized Unary Encoding (OUE) protocol so that it can be employed in our solution. We call the extended protocol Optimized Bitvector Encoding (OBE). OBE is applicable to not only the problem of answering spatial density queries, but also in arbitrary LDP problems with bitvector encodings. We formally prove that the user-side perturbation step of our OBE protocol satisfies LDP and its server-side estimation step produces unbiased estimates. Combining the different partitioning strategies and LDP protocols, we obtain a total of 8 different approaches for answering spatial density queries under LDP, all of which can be parsed as instances of our four-step solution with different choices in the individual steps. We perform an extensive experimental evaluation of these approaches using 4 real-world datasets, varying number of queries, varying query sizes, varying degrees of privacy, and multiple error metrics. Results show that Advanced Partitioning and OBE protocol typically yield the lowest error, demonstrating the superiority of our proposed methods.
Keyframe demonstration seeded and Bayesian optimized policy search
Reinforcement learning (RL) is a promising approach to endow robots with skills. However, RL requires many trials to get satisfactory results. Learning from Demonstration (LfD) seeded RL alleviates this problem by learning an initial skill from human demonstrations. Nevertheless, this approach still requires robots to perform a non-trivial amount of trials. In this thesis, we develop an approach to further reduce these for manipulation skills with perceptual goals. Our main contributions are (1) an algorithm to focus the exploration by using the learned relationship between action and perception and a (2) Black-Box RL Policy Search (PS) method that improves upon the popular Policy Improvement with Path Integral (PI²) algorithm, called the Bayesian Optimized PI² (BO-PI²), that uses reward predictive UCB-type exploration. Our underlying LfD framework utilizes a Dynamic Bayesian Network (DBN) learned from keyframe demonstrations to jointly model the action (end-effector pose) and the goal (object-specific perceptual data) of the skill. The action part is used to generate robot trajectories, and the goal part is used to monitor the success of trajectory executions and to create a Partially Observable Markov Reward Model in order to learn rewards. BO-PI$^{2}$ is used to improve the action part of the DBN with trial-and-error using the learned returns. The novelty of BO-PI$^{2}$ comes from its exploration strategy. The coupling between the action and the goal is used to pick the part of the model to focus on to reduce the effort, in a sense to solve the credit attribution problem. After picking the part to focus on, BO-PI$^{2}$ samples trajectories from the action model to get rollouts, which is typical of PS approaches. In addition, BO-PI$^{2}$ uses a Gaussian Process (GP) to learn local returns from these rollouts, which is improved with each executed trajectory. The next samples are selected by utilizing an Upper Confidence Bound (UCB) approach, using the predicted return and uncertainty of the possible candidate points. This is in contrast to random sampling, used in most PS approaches. BO-PI$^{2}$ also utilizes a skill success based termination criteria, using the goal model to monitor success autonomously. We evaluate BO-PI$^{2}$ with expert and non-expert keyframe demonstrations for three skills. In the expert case, the models are perturbed so that the initial skill execution starts from a failure condition. In the non-expert case, we pick skill models that fail to begin with. We test our approach against the current state-of-the-art PI$^{2}$-ES-Cov algorithm using three metrics: (1) skill success rate, (2) total accumulated reward, and (3) number of trials. In both the expert case and the non-expert case, on average, our approach performed better than the baseline on all three metrics. Our results show that utilization of keyframes allows us to focus on failed sub-goals rather than the entire trajectory, and combined with reward predictive exploration strategies, are beneficial to improve RL performance and reduce the number of trials to endow robot arms with real-life manipulation skills.
Hierarchical clustering attention for unsupervised object-centric representation learning
Extracting object-centric representations from a complex multi-object scene is indeed a crucial milestone for modern neural network architectures to achieve near human level cognition capabilities. Nevertheless, most of the contemporary neural networks that address object-centric representation learning problem require apriori initialization of a fixed set of object describing vectors or cannot manage to handle images of higher resolution. Contrary to long-standing paradigms in the literature, this work proposes Query Breaking Visual Attention (QBVA) module, an efficient and effective building block that introduces a divide and conquer strategy to object-centric representation learning while solving the unsupervised scene segmentation task. QBVA is essentially a stand-alone attention based clustering module that is capable of extracting object-centric representations from a multi-object scene when cascaded into a hierarchical network architecture. QBVA leverages a novel, fully differentiable and non-parametric clustering scheme named Query-Breaking Clustering (QBC) which eliminates the need for initializing a fixed set of clusters and holds the promise to provide dynamic representation for a variable number of objects. We demonstrate that QBVA-Net is indeed a competitive approach to address object-centric representation learning paradigm and prove to be advantageous compared to the state-of-the-art in the sense that it can provide better segmentation performance at the end of the encoder network and theoretically scale up to images of higher resolution.
Incentive compatible and provably secure blockchain applications
With the introduction and advent of blockchain technologies, money transfers have become easier with lower costs, no matter how far the recipient is from the sender. Moreover, the blockchain technology has not only affected the contemporary economical model, but also inspired many novel applications that have been thought impossible without the help of a trusted third party (such as governments or well-recognized companies). However, being a newly developed technology, blockchain still lacks enough research to ensure its security (in terms of mining, transactions, or its applications) and incentive compatibility (i.e., the participants are motivated for following the expected protocol steps). Cryptography and game theory are two well-known tools with traditional proving methods to help us for this task. In this thesis, we utilize game theory and cryptography based security models for building secure applications on blockchain. First, we develop a provably secure blockchain application that we name as e-donation, for secure, fair and decentralized online donations. Second, we develop a practical attribute based digital signature scheme. Third, we analyze and improve the security of blockchain by providing an incentive compatible defense solution against the selfish mining attack of Eyal and Sirer (CACM '18) on proof-of-work blockchain. Forth, we develop a network-based simulation tool for both attacks and defenses. Fifth, we propose a framework for better game-theoretical analysis of multi-player mechanisms when the sizes of the coalitions can be bounded by a threshold, via successfully combining threshold security ideas of cryptography and transferable utility of game theory. Sixth, we apply our threshold coalition notions to show the incentive compatibility of our solution to the outsourced incentivized computation problem in blockchain with multiple outsourced parties.
Elastic pipeline load balancing for dynamic DNNS
Training of dynamic models is gaining traction in DNNs as it reduces computational and memory requirements of large-scale training. Gradual pruning, one of the prominent approaches for dynamic training, prunes (or sparsifies) the parameters of a model during training. However, one of the side effects of gradual pruning is that sparsification introduces an imbalanced workload across accelerators, which in turn affects the pipeline parallelism efficiency. This work introduces DynPipe which dynamically load balances the stages of the pipeline to offset the negative performance effects of pruning. On top of load balancing dynamic models, DynPipe can dynamically pack work into fewer GPUs, while sustaining performance. DynPipe works on single nodes with multi-GPUs and also on systems with multinodes. Experimental results show that DynPipe can speed up the training up to 5.64% in a single node, and 8.43% in a multi-node setting, over state-of-the art solutions used in training production large language models. DynPipe is available at https://anonymous.4open.science/r/DynPipe-1EC5
Building, analyzing and interpreting classroom engagement: Apps and machine-learning models for an affordable programming education
Ensuring students remain engaged in the classroom is crucial for their success in any given topic. In programming education, promoting hands-on interactions and demonstrating real-world use cases are effective methods to foster engagement. However, schools located in socio-economically disadvantaged areas often lack adequate digital infrastructure, such as computer laboratories, to support such engagement-building tasks. Nevertheless, utilizing mobile devices can support programming education due to their availability and affordability. Additionally, mobile devices can help augment the tangible materials to use in collaborative programming experiences. In this respect, the first goal of my research is to develop an affordable tangible programming education environment using mobile devices that supports the collaborative work of students. On top of building tools to foster engagement, analyzing and interpreting student engagement is also a critical component of the teaching process. Analyzing classroom engagement requires a multi-component evaluation of affective, behavioral, and cognitive states. Yet, limited research has been conducted to create a multimodal classroom engagement dataset and analysis model. To fill this gap, my second goal is to build a multimodal engagement evaluation tool using a single camera to ease teachers' workload in group activities. Overall my thesis contributes to Human-Computer Interaction (HCI), Artificial Intelligence (AI), and educational technology research areas with six research outputs: 1. An open-source, user-centered, AI-powered, affordable tangible programming environment that was informed by iterative user studies. 2. A set of design considerations for developing paper-based intelligent tangible programming environments which are informed by iterative and classroom-wide user experience studies. 3. A set of curricular activities to help teachers adapt our programming environment into Turkish national curricula. 4. An open-source audio-visual dataset comprising eight-hour-long video recordings of thirty-three students to predict classroom engagement levels using their self-evaluation scores. 5. Multimodal machine-learning models that can address the multi-component definition of engagement. The image models achieved up to 84% test accuracy on person-based engagement level prediction. The real-time video model that can run on streaming videos achieved 71% test accuracy. 6. A student-centric interactive dashboard to help students view their engagement over time and interpret the results of engagement model prediction.
HyperGAN-CLIP: A versatile framework for CLIP-guided imagesynthesis and editing using hypernetworks
Generative Adversarial Networks, particularly StyleGAN and its variants, have shown exceptional capability in generating highly realistic images. However, training these models remains challenging in domains where data is scarce, as it typically requires large datasets. In this thesis work, we introduce a versatile framework that enhances the capabilities of a pre-trained StyleGAN for various tasks, including domain adaptation, reference-guided image synthesis, and text-guided image manipulation even when only a small number of training sample are available. We achieve this by integrating the CLIP space into the generator of StyleGAN using hypernetworks. These hypernetworks introduce dynamic adaptability, enabling the pre-trained StyleGAN to be effectively applied to specific domains described by either a reference image or a textual description. To further improve the alignment between the synthesized images and the target domain, we introduce a CLIP-guided discriminator, ensuring the generation of high-quality images. Notably, our approach shows remarkable flexibility and scalability, enabling text-guided image manipulation with text-free training and seamless style transfer between two images. Through extensive qualitative and quantitative experiments, we validate the robustness and effectiveness of our approach, surpassing existing methods in terms of performance.
Comicverse: Expanding the frontiers of ai in comic books with holistic understanding
Comics are a unique and multimodal medium that conveys stories and ideas through sequential imagery often accompanied by text for dialogue and narration. Comics' elaborate visual language exhibits variations from different authors, cultures, periods, technologies, and artistic styles. Consequently, the computational analysis of comic books requires addressing fundamental challenges in computer vision and natural language processing. In this thesis, I aim to enhance neural comic book understanding by making use of comics' unique multimodal nature and processing comics in a character-centric approach. The primary data source for this thesis is the Golden Age of American Comics due to its public accessibility and abundance of comic series. However, the availability of annotated data is limited. Thus, to achieve my goal, I have adopted a holistic approach composed of four main steps ranging from curating datasets to proposing novel tasks and architectures for comics. The first three steps aim to create a machine-readable comics database by locating comic book panels, identifying their components, and transforming character identities into a dialogue-like structure and the final step uses this database to train a transformer- based model. The first step involves extracting high-quality text data from speech bubbles and narrative box images using OCR models. I decompose comic pages into their constituent components in the second step through detection, segmentation, and association tasks with a refined Multi-Task Learning (MTL) model. Detection involves identifying panels, speech bubbles, narrative boxes, character faces, and bodies. Segmentation focuses on isolating speech bubbles and panels, while the association task involves linking speech bubbles with character faces and bodies. In the third step, I utilize the paired character faces and bodies obtained from the previous stage to create character instances and, subsequently, reidentify and track these instances across sequential panels. In the final step of my thesis, I propose a multimodal framework by introducing the ComicBERT model, which exploits the abovementioned structure. Cloze-style tasks were used to evaluate ComicBERT's contextual understanding capabilities. Furthermore, I propose a new task called Scene-Cloze, which predicts the next panel given n previous panels as context. As a result, my approach achieves a new state-of-the-art performance in Text-Cloze and Visual-Cloze tasks with accuracies of 69.5% and 77.1%, respectively, thus getting closer to the human baseline. Overall, the highlights of my contributions are as follows: 1. I curated and shared COMICS Text+ Dataset with over two million transcrip- tions of textboxes from the golden age of comics. In addition, I open-sourced the text detection and recognition models that are fine-tuned for the task and datasets used in their training. 2. I refined a MTL framework for detection, segmentation, and association tasks and achieved SOTA results in comic character face and body-to-speech bubble association tasks. 3. I proposed a novel Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification framework to generate unified and identity-aligned comic character embeddings and identity representations. Furthermore, I generated two new datasets: the Comic Character Instances Dataset, encompassing over a million character instances used in the self-supervision phase, and the Comic Sequence Identity Dataset, containing annotations of identities within sets of four consecutive comic panels used in semi-supervision phase. 4. I introduced the multimodal Comicsformer, a transformer-encoder architecture capable of processing sequential panels and their constituents. It serves as the backbone for the Masked Comic Modeling (MCM) task, a novel self- supervised pre-training strategy for comics, resulting in ComicBERT, a potential foundation model for golden age comics. ComicBERT achieves SOTA performance in cloze-style tasks, particularly in text-cloze and visual-cloze tasks, approaching human-level comprehension.
Investigating the missing pieces of sensorimotor reinforcement learning agents for autonomous driving
Reinforcement Learning (RL) has the potential to surpass human capabilities in self-driving without needing any expert supervision. Despite its promise, the state-of-the-art in sensorimotor self-driving is dominated by imitation learning methods due to the inherent challenges of RL algorithms. Nonetheless, RL agents are able to discover highly successful policies when provided with privileged ground truth representations of the environment. In this work, we investigate what separates privileged RL agents from sensorimotor agents for urban driving in order to bridge the gap between the two. We propose vision-based deep learning models to approximate the privileged representations from sensor data. In particular, we identify aspects of state representation that are crucial for the success of the RL agent such as desired route generation and traffic light prediction, and propose solutions to gradually remove privileged information for each with the existing computer vision approaches. Through rigorous evaluation on the CARLA simulation, we shed light on the significance of the state representation in RL for autonomous driving and outline unresolved challenges for future research.
Advancing toward temporal and commonsense reasoning in vision-language learning
Humans learn to ground language to the world through experience, primarily visual observations. Devising natural language processing (NLP) approaches that can reason in a similar sense to humans is a long-standing objective of the artificial intelligence community. Recently, transformer models exhibited remarkable performance on numerous NLP tasks. This is followed by breakthroughs in vision-language (V&L) tasks, like image captioning and visual question answering, which require connecting language to the visual world. These successes of transformer models encouraged the V&L community to pursue more challenging directions, most notably temporal and commonsense reasoning. This thesis focuses on V&L problems that require either temporal reasoning, commonsense reasoning, or both simultaneously. Temporal reasoning is the ability to reason over time. In the context of V&L, this means going beyond static images, i.e., processing videos. Commonsense reasoning requires capturing the implicit general knowledge about the world surrounding us and making an accurate judgment using this knowledge within a particular context. This thesis comprises four distinct studies that connect language and vision by exploring various aspects of temporal and commonsense reasoning. Before advancing to these challenging directions, (i) we first focus on the localization stage: We experiment with a model that enables systematic evaluation of how language-conditioning should affect the bottom-up and the top-down visual processing branches. We show that conditioning the bottom-up branch on language is crucial to ground visual concepts like colors and object categories. (ii) Next, we investigate whether the existing video-language models thrive in answering questions about complex dynamic scenes. We choose the CRAFT benchmark as our test bed and show that the state-of-the-art video language models fall behind human performance by a large margin, failing to process dynamic scenes proficiently. (iii) In the third study, we develop a zero-shot video-language evaluation benchmark to evaluate the language understanding abilities of pretrained video-language models. Our experiments reveal that the current video-language models are no better than the vision-language models, processing static images as input in processing daily dynamic actions. (iv) In the last study, we work on a figurative language understanding problem called euphemism detection. Euphemisms tone down expressions about sensitive or unpleasant issues. The ambiguous nature of euphemistic terms makes it challenging to detect their actual meaning within a context where commonsense knowledge and reasoning are necessities. We show that incorporating additional textual and visual knowledge in low-resource settings is beneficial to detect euphemistic terms. Nonetheless, our findings on these four studies still demonstrate a substantial gap between current V&L models' abilities and human cognition.
RbA: Segmenting unknown regions rejected by all using mask classifiers
Fine-grained visual semantic understanding is essential for autonomous driving and many other computer vision tasks. To this end, segmentation tasks have been witnessing rapid advancements as a result of the growing number of benchmarks proposed. However, existing segmentation benchmarks generally assume a fixed set of semantic categories. Consequently, the development of segmentation methods has been centered around this assumption, while little attention has been poured into handling novel or out-of-distribution (OoD) samples that can potentially be encountered in real-life scenarios. This poses an issue for an autonomous vehicle, as it is crucial to identify unknown objects so that a safety warning can be issued to avoid disastrous consequences in case of failure. As a result, the task of OoD segmentation has been addressed separately, leading to the emergence of methods that adapt to the existing segmentation methods and disregard the performance of the main segmentation tasks. Additionally, these methods also generally suffer from a lack of smoothness and objectness in their predicted anomaly maps due to the reliance on models that follow the per-pixel classification paradigm. In this thesis, we explore the potential of region-level classification models for unknown segmentation as a unified architecture with an inherent ability to express uncertainty. We show that the object queries in mask classification models tend to behave like one \vs all classifiers. Based on this finding, we propose a novel outlier scoring function called Rejected by All (RbA) by defining the event of being an outlier as being rejected by all known classes. We also propose an objective that optimizes this proposed score for boosting the unknown segmentation performance using pseudo-outlier data without hurting the closed-set performance. RbA performs well under high domain shifts and is capable of separating sources of uncertainty, such as at the boundaries, due to known class ambiguity. We evaluate RbA on several unknown segmentation benchmarks and show that it achieves state-of-the-art performance with significant margins compared to previous pixel-level unknown segmentation methods. We report extensive ablation experiments that validate the effectiveness of RbA.
A multilayer perceptron framework for sparse multiple kernel learning
Advancing cancer research and improving patient care heavily rely on understanding how biological mechanisms change as cancer progresses. Identifying the active biological processes within tumors is crucial for developing targeted treatments and potential biomarkers for early detection and prognosis. In this thesis, we focused on distinguishing between early-stage and late-stage cancers using gene expression profiles, aiming to develop a powerful and interpretable model. Machine learning methods have shown promise in cancer research by enabling the analysis of vast genomic and clinical data to uncover hidden patterns and predictive features. Our work centered around harnessing the capabilities of a multilayer perceptron (MLP) model to create a sparse solution within the context of multiple kernel learning (MKL). This enabled our model to proficiently differentiate between early-stage and late-stage cancers based on gene expression profiles. Our model not only offered high predictive performance but also provided valuable insights into the crucial genes and pathways driving cancer progression. To evaluate our MLP model, we benchmarked it against three well-established machine learning algorithms: random forest, support vector machine, and MKL. Remarkably, our model consistently achieved better or comparable predictive performance, as measured by the area under the receiver operating characteristic curve, across 15 cancer cohorts. The findings demonstrate that our proposed MLP model effectively identifies critical genes and pathways driving cancer progression, offering valuable insights into early-stage and late-stage cancer classification.
HAİSTA-NET: Dikkat yoluyla insan destekli obje segmentasyonu
Instance segmentation is a fundamental computer vision task with a wide range of applications. Some instance segmentation tasks such as medical image analysis, and image/video editing require high levels of precision. However, this precision is often beyond the reach of what even state-of-the-art, fully automated instance segmentation algorithms can deliver. The performance gap becomes particularly prohibitive for small and complex objects. Practitioners typically resort to fully manual annotation, which can be a laborious process. In order to overcome this problem, we propose a novel approach to enable more precise predictions and generate higher-quality segmentation masks for high-curvature, complex and small-scale objects. Our human-assisted segmentation model, HAISTA-NET, augments the existing Strong Mask R-CNN network to incorporate human-specified partial boundaries. We also present a dataset of hand-drawn partial object boundaries, which we refer to as "human attention maps." In addition, the Partial Sketch Object Boundaries (PSOB) dataset contains hand-drawn partial object boundaries which represent curvatures of an object's ground truth mask with several pixels. Through extensive evaluations, we show that HAISTA-NET outperforms state-of-the art methods such as Mask R-CNN, Strong Mask R-CNN, and Mask2Former, achieving respective increases of +36.7, +29.6, and +26.5 points in AP-Mask metrics for these three models. Our novel approach sets a baseline for future human-aided deep learning models by combining fully automated and interactive instance segmentation architectures.
FLAGS framework and decentralized federated learning under device volatility
Federated Learning (FL) has become a key choice for distributed machine learning. Initially focused on centralized aggregation, recent works in FL have emphasized greater decentralization supported by standardization of serverless interaction in the next-generation communication networks. However, the diversity of devices, data distributions, and communication settings, compounded by dynamic operating conditions, result in multiple challenges for Decentralized FL (DFL). There have been various approaches to DFL, from utilizing intermediate edge servers to fully device-to-device approaches. In decentralized settings, communication cost and learning performance are usually assessed together and certain trade-offs are made based on scenarios. However, there is a lack of existing work on comparing DFL approaches in an apples-to-apples manner in a multitude of scenarios and operating conditions. To bridge this gap between methods and their comparative analysis, we design and develop the Federated Learning Algorithms Simulation (FLAGS) Framework. One important challenge we noticed that most DFL methods struggle with is the extreme fluctuations in device availability, especially for purely decentralized approaches. This \textbf{device volatility} leads to poor learning performance. To address this issue, we investigate the effects of neighborhood selection, memory and multi-hop information passing on DFL performance. We introduce a fully decentralized FL approach that can operate under realistic and highly volatile device participation settings. The key contributions of this thesis are as follows: (i) development of a lightweight FL framework for benchmarking a large plethora of methods, (ii) analysis and comparison of multiple FL methods with this framework under multiple operating conditions, (iii) empirical analysis of various node selection strategies under heavy device volatility, and (iv) utilizing memory and relayed communication to enhance device-to-device FL by developing a novel algorithm that can operate under realistic operating conditions and heavy device volatility. Federated Learning supports a wide variety of node interactions and autonomous operations across the network edge. With the aim to encompass this multi-faceted heterogeneity, the FLAGS framework was proposed and developed as a lightweight FL implementation and testing platform. FLAGS framework allows for a wide range of device behaviors and cooperation mechanisms, enabling rapid testing of multiple FL algorithms. FLAGS's built-in features allow it to subject existing and novel FL algorithms to a wide range of data distributions, simulating the nodes with multiple neural networks as well as participation conditions ranging from homogeneous to highly volatile. Different network tiers and communication mechanisms enable various FL algorithms to be configured by employing various combinations of the aforementioned factors. In order to consolidate this very extensive FL landscape and offer an objective analysis of the major FL algorithms, comprehensive cross-evaluations for a wide range of operating conditions have also been conducted. Starting with the three foundational FL algorithms, including Hierarchical FL (HFL), Decentralized FL (DFL), and Gossip FL (GFL), this work evaluates six derived algorithms ranging from fully centralized to fully decentralized. The experiments indicate that fully decentralized FL algorithms achieve comparable accuracy under multiple operating conditions, including asynchronous aggregation and the presence of stragglers. Furthermore, DFL can also operate in noisy environments and with a comparably higher local update rate. However, the impact of extremely skewed data distributions on DFL is much more adverse than on centralized variants. The analysis of the cross-evaluation indicates that DFL performance is considerably impacted by node participation. This part of the thesis focuses on improving DFL performance under realistic and volatile device behavior. Node selection in various forms has been experimented with to improve both communication efficiency and convergence rate. We experimented with multiple node selection mechanisms and also proposed and evaluated a time-varying parameterized node selection method for DFL employing validation accuracy and its per-round change. The mentioned criteria are evaluated using both hard and stochastic/soft selection on sparse networks. The results indicate that the bias associated with node selection adversely impacts performance as training progresses, and a uniform random selection is preferable under extremely limited participation conditions. Continuing with volatile conditions, we investigate and propose mechanisms to improve DFL operating on sparse graphs in the presence of stragglers and non-participating nodes. We first propose two algorithms: Memory-Assisted DFL (MA_DFL) and Augmented-Graph Assisted DFL (AG_DFL). These algorithms employ memory and selective relaying to improve DFL performance. Both algorithms outperform the baseline DFL and gossip interaction for volatile node participation. Then, we propose a hybrid of these two algorithms, Memory and Augmented-Graph Assisted DFL (MAG_DFL), that employs memory and graph augmentation to improve the performance of DFL under highly volatile devices and extreme data conditions. The research conducted in this thesis evaluates the multi-faceted challenges to the DFL operation in volatile conditions and proposes mechanisms to improve its performance. Our work indicates that DFL holds the potential to assist learning operations distributed across the edge network. It may be used to augment the FL in the presence of costly upstream communication or limited connectivity. However, node density has a major impact on DFL, and sparse networks, along with volatile device behavior and non-IID distributions, tend to reduce its convergence rate. The enhanced neighborhood interaction and intelligent use of local information has the potential to improve DFL performance under such adverse conditions based on the presented results. The analysis, algorithms and results presented in this thesis pave the way for additional developments and more practical applications of DFL in the next-generation communication networks.
Autonomous execution for multi-GPU systems: CPU-free blueprint and compiler support
As multi-GPU systems become more prolific in the field of supercomputing, scientific applications are adapted and scaled up to take advantage of the highly parallel accelerators for increased performance. However, the traditional model of GPU programming leaves much to be desired in multi-GPU settings, wherein communication among devices - one of the largest points of contention and bottlenecks in scientific applications - is controlled by the CPU. This kind of one-sided control leads to undue latencies incurred by the constant back-and-forth of synchronization and API calls between the host and devices, and harms application scaling as the number of GPUs grows. This work first proposes the fully autonomous CPU-Free execution model for multi-GPU applications that completely excludes the involvement of the CPU beyond the initial kernel launch. We systematically combine several techniques such as persistent kernels, thread block specialization, and GPU-initiated communication and synchronization to significantly reduce host-incurred latencies and facilitate further optimizations. We benchmark our proposed model on a broadly used iterative solver, 2D/3D Jacobi Stencil and improve 3D stencil communication latency by 58.8% compared to CPU-controlled baselines on 8 NVIDIA A100 GPUs. The second part of this work adds compiler support to easily write performant CPU Free code in high-level Python by extending the DaCe framework with GPU-centric communication intrinsics. We compare automatically generated CPU-Free code to existing distributed facilities in DaCe and observe over 96\% performance improvement in Stencil benchmarks.
Localizing knowledge in large language model representations
Large language models (LLMs) are very proficient in NLP tasks. In the first part of this work, we evaluate the performance of LLMs on the task of finding the locations of characters inside a long narrative. The objective of the task is to generate the correct answer when the input is a piece of a narrative followed by a question asking the location of a character. For the evaluation of the task, we generate two new datasets by annotating the characters and their locations in the narratives: Andersen and Persuasion. We show that the LLM performance is not satisfactory on these datasets when compared to the simple baseline we designed that does not use machine learning. We also experiment with in-context learning to improve the performance and report results. Moreover, we address the problem that the LLMs are limited by the bounded context length. We hypothesize that if we localize the character-location relation information among the activations inside an LLM, we can store those activations and inject them into other models that are run with a different prompt so that the LLM can answer the questions about the information that was carried from another prompt, even though the character and location relation is not mentioned explicitly in the current prompt. We develop five different techniques to localize the character-location relation information occurring in the LLMs: Moving and adding LLM activations to other prompts, adding noise to LLM activations, checking cosine similarity between LLM activations, editing LLM activations, and visualizing attention scores during answer generation. We report the observations we made using these techniques.
Investigating the potential of incorporating protein language models (pLMs) into ML/DL approaches for enhanced prediction of allosteric sites in proteins
Allosteri, proteinin bir bölgesindeki değişikliğin, mesela başka bir moleküle bağlanmanın, proteinin uzak bir bölgesini etkilediği süreç olarak tanımlanabilir. Allosteri protein fonksiyonu üzerindeki önemli etkisi sebebiyle ilaç geliştirme alanında önemli bir odak noktasıdır. Allosterik ilaçlar proteinleri aktive veya inhibe edebilir, allosterik olmayan ilaçlara göre avantajlar sunar. Bununla birlikte, allosterik bölgelerin tanımlanması zorlu bir iştir. Geçmişte allosterik bölgeleri tahmin etmek için Normal Mod Analizi (NMA), Moleküler Dinamik (MD) ve Makine Öğrenimi (MÖ) gibi hem statik cep özelliklerini hem de proteinlerin dinamiklerini kullanan çeşitli hesaplama teknikleri geliştirilmi olmakla birlikte bu yöntemlerin performansının daha da geliştirilmesi gerekmektedir. Bu araştırmada, pDM'lerin (örneğin, ProtTrans pDM ailesinden BERT mimarisine dayalı ProtBERT'in) allosterik kalıntıların tahminini iyilrştirmek için Protein Dil Modellerini (pDM'ler), MÖ ve/veya DÖ yaklaşımlarıyla birlikte kullanılma potansiyelini araştırılıyor. Tezde, amino asitler arasındaki mekansal ilişkiyi etkili bir şekilde öğrenerek, sonuçta allosterik alanların/ceplerin tanımlanmasını hedeflenmektedir. ProtBERT-BFD (ProtTrans), test veri kümesinde %61,54'lük bir F1 puanıyla allosterik kalıntıları tahmin eden protein dizilerinin Allosterik Veri Kümesine (AVK) göre ince ayar yapılmıştır. XGBoost, SVM, AutoML ve GNN'ler dahil olmak üzere çeşitli MÖ ve DÖ yaklaşımlarından yararlanılmıştır, İnce ayarlı pDM özelliklerinin dahil edilmesiyle, yukarıda belirtilen yaklaşımların tümü, allosterik bölgelerin tahmin performansını önceki çalışmalara göre önemli bir farkla artırdığı bulunmuştur. Bu çalışmada en yüksek performansa sahip model olan XGBoost, ince ayarlı ProtBERT'ten çıkarılan özellikleri FPocket tarafından çıkarılan cep özellikleriyle birleştirerek sonuçları iyileştiriyor ve allosterik cepler/bölgeler için %75,76'lık bir F1 puanı erişmektedir. Bilinen allosterik bölgelere sahip proteinler üzerinde örnek çalışmaların yanı sıra, farklı proteinler üzerindeki yeni allosterik bölgeleri tahmin etmek için de çalışmalar yapılmıştır.
Unsupervised multi-object discovery and tracking using memory-augmented slot attention
Learning object-centric representations from static images is a promising research direction in the field of deep learning. However, adapting this approach to videos poses certain challenges due to the necessity of capturing the temporal dynamics of video content. Recent works have made significant progress in object discovery within synthetic video datasets. Nevertheless, these works do not fully exploit the motion of objects in videos and temporal cues. In this thesis, we aim to enhance the performance of object-centric representation learning on video frames by using the temporal information more carefully. To achieve this goal, we propose a new unsupervised learning method that utilizes a memory-augmented slot attention model for multi-object discovery and tracking. The key component of our approach is the integration of memory slots, which store information from past video frames, alongside object slots into the learning architecture. Object slots simultaneously attend to both memory slots for information from past frames and the current image input. Training memory slots requires longer video sequences. However, the most suitable learning structures for this, recurrent neural networks (RNNs), are not very effective in learning long-term temporal data due to the issues of exploding and/or vanishing gradients. To train our memory-augmented model more effectively on long video sequences, we employ truncated back-propagation through time. Experiments conducted on synthetic yet realistic video datasets have yielded promising results, indicating that memory slots significantly improve multi-object tracking and object segmentation performance. Our fully unsupervised learning method contributes to the problem of object-centric representation learning in videos and opens up new possibilities in this field.
Self-collision aware reaching and pose control in large workspaces using deep reinforcement learning
Reaching, pose control, and inverse kinematics are fundamental robotic manipulator tasks that underpin other tasks and as such, there is a vast body of related literature from various fields. Control, planning, and more recently learning fields are among the main ones. Traditional control algorithms are prone to failure around singularities and joint limits and do not naturally handle self-collisions. Planning methods are not fast enough for reactive behaviours and require additional infrastructure, including controllers, to work. Learning-based methods have emerged to tackle these issues. However, most of them do not handle arbitrary initial and target poses, ignore self-collisions, do not include orientation information in their targets, work in small workspaces and evaluate themselves with coarse success metrics. In this thesis, we introduce a novel hybrid approach that combines Pseudo-inverse control (PinvC) and model-free reinforcement learning (RL), including state space and reward function design, to fill these gaps in the context of reaching, pose control and inverse kinematics. PinvC already calculates joint velocities given desired task-space (e.g. the end-effector pose) velocities and only requires the kinematic structure of the robot. PinvC is mostly reliable away from joint limits, singularities and when individual links are not prone to collisions. The main idea behind our approach is to use RL to handle these situations. Towards this end, we design a novel state space and reward functions. Our reward function aims to minimize position (reaching task) or pose (inverse kinematics and pose control tasks) errors, reduce self-collisions and reduce joint velocities near the target. Furthermore, we develop a curriculum learning methodology to aid learning. Lastly, we introduce a simple modification, which we call "switching" to further improve task performance. We evaluate our approach with four simulated robots for various problem settings and compare it against traditional and learning-based approaches. Our results show that our approach decidedly outperforms the baselines in terms of mean error, success rates at various thresholds and terminal speed for reaching tasks. In addition, we reduced the number of self-collisions across all the scenarios. Our approach achieved better results when orientation was included, but none of the methods performed very well, especially the learning baselines. We note that the learning-based methods in the literature almost always ignore orientation. As a result, we comprehensively discuss the reasons for orientation failure and potential remedies.
Audio-driven image generation and editing with pretrained diffusion models
We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, in this thesis we propose SonicDiffusion to enable audio-conditioning in large scale image diffusion models. Our method first maps features obtained from audio clips to tokens that can be injected into the diffusion model in a fashion similar to text tokens. We introduce additional audio-image cross attention layers which we finetune while freezing the weights of the original layers of the diffusion model. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods to enable audio conditioned image editing. We demonstrate our method on a wide range of audio and image datasets. We perform extensive comparisons with recent methods and show favorable performance.
Interpretable cancer stage classification using sparse bayesian neural networks
Cancer requires an in-depth exploration of its molecular characteristics. For targeted treatment strategies, distinguishing cancer stages is essential. This thesis has focused on the differentiation of early- and late-stage cancers using gene expression profiles. With the integration of computational techniques into medical research, machine learning models excel in this task, offering insights into biological mechanisms. To harness these insights, we proposed a novel approach, which is Bayesian Neural Networks (BNNs) with sparsity-inducing priors. The proposed sparse BNNs are designed to deliver high predictive performance in identifying cancer stages while maintaining a high level of interpretability. To evaluate our sparse BNN models, we benchmarked them against three machine learning algorithms across 15 different cancer cohorts. The results of our study revealed that our sparse BNN models achieve predictive performances comparable to traditional benchmark models. Additionally, we addressed the black-box issue of neural networks in medicine, which obscures which input features are crucial for predictions, a serious issue in decision-making with significant implications. To address this issue, our primary contribution has been the development of a novel BNN architecture that considerably enhances data interpretability. In our approach, we have integrated three types of sparsity inducing priors, namely, Laplace, Student's t, and Spike-and-Slab. Each prior has a mean of zero and low variance, promoting a reduction in connections and thus enabling a focused feature selection process. This methodology allows us to identify and concentrate on the most influential gene expressions. Our analysis revealed that sparse BNNs show a distinct preference for specific gene sets. In conclusion, the development of sparse BNNs offers a biologically informative and interpretative tool, enhancing the field of cancer research by shedding light on key gene pathways and significantly improving the process of cancer staging.
Cha and core discovery on intel chips and generating optimized thread binding
In modern multi-core architectures with distributed directory-based cache coherence, each memory address is overseen by a distributed directory unit, known as a Caching/Home Agent (CHA), that monitors cache line state and location. Neither the CHA nor core locations in a processor are directly exposed to the programmer. In this work, we firstly analyze and compare the methodologies for uncovering both the CHA and core topology of Intel Xeon Scalable processors, as well as the methods to reveal the mapping of memory addresses to CHAs. Leveraging the topology and the address mapping information, we investigate the impact of spatial proximity between communicating cores and CHAs on application performance, and propose a thread mapping heuristic that assigns threads to cores by considering cache coherence traffic. We expect our heuristic to achieve significant performance gains on applications with high amount of on-chip cache coherence traffic due to high percentage of shared written data. We evaluated our heuristic on applications that exhibit high amount of on-chip communication traffic. The heuristic achieves up to 5.6% speedup over compact placement on merge-based SpMV application, up to 8% with an average of around 4.4% on Barnes application, around 25% for Fluidanimate application to simulate 60 frame per second, and lastly approximately 6% for LU across different matrices. We also prove the improved performance is in fact related to reduced on-chip traffic on the mesh.
An automated framework for concurrent graph processing on GPU
GPUs have become the bleeding edge in high-performance systems used for artificial intelligence and scientific computations. Running parallel jobs and algorithms on GPUs yields results that are orders of magnitude faster than running the same jobs on CPUs that employ fewer cores. A significant body of computational tasks incorporates graph traversals and computations associated with traversal of large graphs. These tasks are either graph traversal tasks in nature or are simplified or transformed into graph traversal tasks. Kernel fusion has been proposed in several bodies of research as a means of fusing computation cores (kernels) in parallel jobs so that the performance can be optimized. Specifically, kernel fusion is used to fuse accesses to same graph nodes and corresponding regions of memory by multiple parallel jobs so that memory caches are more efficiently utilized, resulting in performance improvements. As kernel fusion's fundamental idea is sharing resource access to increase utilization by multiple jobs, it is even better suited for GPUs that run hundreds of jobs in parallel than CPUs that run a few jobs in parallel. However, kernel fusion is a computationally expensive operation that incorporates control flow changes and manipulation of the order of execution for computational jobs. As such, kernel fusion is harder to utilize on GPUs compared to CPUs. This work introduces a streamlined kernel fusion framework for concurrent graph processing on GPUs. The framework enables definition and implementation of graph jobs that can be executed in parallel in GPUs, and are automatically fused by the kernel fusion algorithm implemented in the framework, to achieve better performance. The framework also introduces novel data handling structures, as well as a meta-compiler that enables static polymorphism for running multiple different jobs as a job queue on the GPU, without performance hits that are associated with typical polymorphism. The framework is then extensively evaluated by defining four common graph jobs (BFS, SSSP, PageRank, Label Propagation) and running up to 200 parallel instances of these jobs in homogeneous and heterogeneous manners, with and without kernel fusion. The evaluations show that kernel fusion provides about 5% to 10% performance improvement when job parallelism is reasonably high (i.e., more than 10 parallel jobs).
Distributed multi-party fair exchange
Over the years, researchers focused on either two-party cases or multi-party computation securely, fairly and optimistically. Most of the studies in the literature require a trusted third party (TTP) to be present for fairness either at every step of the protocol or only optimistically intervene in case of malicious actions. However, having a single TTP creates a single point of failure (in terms of security) in the protocol. To distribute this responsibility, literature has different solutions. Blockchain is one of the most used solutions for decentralizing exchange protocols and multi-party computations. Secret sharing is another procedure used for distributing responsibility, however, it is mostly used for 2-party settings. As far as we know, there is no optimistic multi-party fair exchange solution using multiple TTPs in the literature. In this work, we present a Distributed Multi-Party Fair Exchange Protocol, building on top of a previous research [Alper and Küpçü, 2021], by distributing the responsibility of a single TTP to multiple (m) TTPs. We are using secret sharing to distribute the decryption share used for fairness to TTPs. Even with a malicious subset of TTPs, the protocol provides a fair result, as long as there are threshold-many honest TTPs. Even though the performance of the protocol is worse (slower, as expected) than the previous study, it carries the importance of being the first optimistic multi-party fair exchange protocol with multiple trusted third parties in the literature.
O1O: Grouping of known classes to identify odd-one-out
Object detection methods trained on a fixed set of known classes struggle to detect objects belonging to unknown classes in real-world scenarios. Open-world methodologies have emerged in recent years as a solution for the limitations of closed-set approaches. The main goal of open-world object detection is to detect and identify novelties while maintaining closed-set abilities. One common approach involves incorporating approximate supervision with pseudo-labels corresponding to candidate locations of objects, typically obtained in a class-agnostic manner. While previous attempts mainly rely on the appearance of objects, we propose that geometric cues provide a better solution as the source of pseudo-labels. By considering not just how objects look but also their shapes and relative locations, we aim to improve the system's ability to detect unfamiliar objects. Although additional supervision from pseudo-labels improves unknown object detection, it also introduces confusion for known classes. We observed a notable decline in the model's performance for detecting known objects in the presence of noisy pseudo-labels. To address this problem, we drew inspiration from human cognitive science. Studies about how humans mentally represent objects found that humans group objects based on their common attributes, which then helps to compare and identify the different ones given a group of objects. We applied a similar concept by organizing known object classes into a smaller set of superclasses by learning discriminative superclass representations. By doing so, our model can identify similarities between classes within a superclass, thereby facilitating the detection of unknown classes through an odd-one-out scoring mechanism. Our experiments on open-world detection benchmarks demonstrate significant improvements in unknown recall consistently across all tasks. Crucially, we achieve this without compromising known performance, thanks to better partitioning of the feature space with superclasses.
Multiview contrastive autoencoder-transformer approach for protein-protein interface representation: Unveiling biological and functional insights
Protein-protein interactions (PPIs) play pivotal roles in various biological processes, orchestrating cellular functions essential for life. The interfaces where these interactions occur serve as focal points for understanding the mechanisms underlying disease pathways. Accurate representation of these interfaces is crucial for deciphering their biological significance and designing therapeutic interventions. This thesis introduces a novel approach for representing protein-protein interfaces using a graph-based multiview contrastive autoencoder combined with a transformer, which learns representations from a large dataset. Comprehensive evaluations demonstrate the method's effectiveness in capturing the structural and functional characteristics of protein-protein interfaces. The learned representations are applied to tasks such as biological relevance prediction, biological vs. crystal classification, and Gene Ontology term prediction, showcasing their versatility and utility in understanding PPIs. By integrating explainable AI techniques, key features contributing to model predictions are identified, enhancing the interpretability of the results. A detailed case study illustrates the practical application of these methods, highlighting their potential to provide actionable insights for biological research and drug discovery. Overall, this thesis advances the understanding of protein-protein interactions by providing interpretable representations that capture the complex structural and functional characteristics of interfaces, thereby facilitating biomedical studies and therapeutic developments.
Grounding language in motor space: Exploring robot action learning and control from proprioception
Language development, particularly in its early stages, is deeply correlated with sensory-motor experiences. For instance, babies develop progressively via unsupervised exploration and incremental learning, such as labeling the action of "walking" by first discovering to move their legs via trial and error. Drawing inspiration from this developmental process, our study explores robot action learning by trying to map linguistic meaning onto non-linguistic experiences in autonomous agents, specifically for a 7-DoF robot arm. While current grounded language learning (GLL) in robotics emphasizes visual grounding, our focus is on grounding language in a robot's internal motor space. We investigate this through two key aspects: Robot Action Classification and Language-Guided Robot Control, both within a "Blind Robot" scenario by relying solely on proprioceptive information without any visual input in pixel space. In Robot Action Classification, we enable robots to understand and categorize their actions using internal sensory data by leveraging Self-Supervised Learning (SSL) through pretraining an Action Decoder for better state representation. Our SSL-based approach significantly surpasses other baselines, particularly in scenarios with limited data. Conversely, Language-Guided Robot Control poses a greater challenge by requiring robots to follow natural language instructions, interpret linguistic commands, generate a sequence of actions, and continuously interact with the environment. To achieve that, we utilize another Action Decoder pre-trained on sensory state data and then fine-tune it alongside a Large Language Model (LLM) for better linguistic reasoning abilities. This integration enables the robot arm to execute language-guided manipulation tasks in real time. We validated our approach using the popular CALVIN Benchmark, where our methodology based on SSL significantly outperformed traditional architectures, particularly in low-data scenarios on action classification. Moreover, in the instruction following tasks, our Action Decoder-based framework achieved on-par results with large Vision-Language Models (VLMs) in the CALVIN table-top environment. Our results underscore the importance of robust state representations and the potential of the robot's internal motor space for learning embodied tasks.
OPTKIT: Unleash the Green Potential of Your CPU Code
The analysis of energy consumption in software has traditionally taken a back seat to performance analysis. However, increasing environmental concerns, power and cooling limitations in expanding HPC systems and data centers, and the need for energy efficiency in power-constrained devices such as mobile phones, tablets, laptops, and IoT devices have brought significant focus to this area. While performance analysis has a wide range of tools, those specifically designed for software energy efficiency are scarce. This study introduces the Optimizer Toolkit (optkit), a dedicated library and toolset designed for the energy analysis and optimization of software runtime. optkit utilizes perf_event_open kernel call for PMU and RAPL, and msr-safe for cpu frequency operations. Inside the tool, there are Query* classes where the user can query anything regarding cpu, energy, and performance before taking any action. It includes useful utility tools that help users streamline the optimization process. In this work, we leverage optkit to evaluate the potential for reducing a system's energy consumption by strong-scaling core numbers in a given application. We then use these data to further reduce energy consumption through co-scheduling applications. In another use-case, we investigate the impact of CPU frequency on energy consumption and develop a methodology to dynamically detect the optimal frequency at runtime and adjusts frequencies based on execution phases to maximize energy savings. We find that not all applications scale linearly, and allocating appropriate resources yields better energy savings. Co-scheduling applications by using unassigned resources often saves more energy than serial execution counterparts. Although manually finding the best frequency offers the highest energy savings, our automated approach achieves up to 31.83% energy savings compared to normal execution of applications. The optkit tool is publicly available on GitHub
Feature generation for SME behavioral credit scoring model using graph embeddings
Banks lend credit regarding customers' credit scores from in-house credit risk models. Corporate customers hold greater risk and are evaluated with already high-performing models. In this thesis, corporate customers' financial interactions (money transfers) with each other are used to develop an additional feature set to improve the behavioral credit risk scoring model of small and medium-sized enterprises. Graph representation learning does a good job in terms of extracting the essence of such relationships and projecting them into a multidimensional space. For this purpose, a graph of enterprises is constructed where they are nodes and their sum of money transfer amount over time are edges. An inductive graph representation learning algorithm, GraphSAGE, is employed. Therefore, the graph is created with directed and weighted edges and reduced to a strongly connected graph. Then, the algorithm is run in various settings to achieve the best performance. A credit scoring pipeline using different machine learning algorithms is added to improve the current model with these new embedding features. QNB Finansbank's real banking data for 2022 is used to perform this study. After looking at the results of the computational experiments, the most important contribution came from the concatenation of multiple embeddings generated with different aggregator functions. Starting from here, a potential economic savings is calculated. Additionally, it is seen that the new architecture improves the credit scoring of nodes with either low transfer amount or number of edges the most.
Enhancing scene sketch understanding through a dual-network: Visio-temporal segmentation and context-aware sketch recognition
Understanding scene sketches involves segmenting and categorizing individual objects within the sketch. Semantic segmentation in scene sketches is crucial for distinguishing distinct sketches, but current methods often treat sketches as bitmap images, which can result in a loss of stroke order information. However, people tend to draw objects sequentially, so leveraging this temporal order could enhance segmentation performance. Moreover, traditional methods typically focus on class-level segmentation, failing to differentiate between instances within the same category. Another important aspect of scene sketch understanding is classifying individual sketch objects within a scene. These objects often lack the detail needed for standalone recognition, making their identification challenging without contextual information. For instance, a sketch that is fluffy and circular could be interpreted as a "bush" or a "cloud," depending on its position and size within the scene. Bushes are typically drawn on the ground, while clouds are usually sketched in the sky. Despite their similar appearances, their interpretation can vary based on their context. To enhance recognition accuracy, information about relative position and size is often used in computer vision tasks like object recognition and image classification. However, many current sketch recognition methods treat sketches in isolation, overlooking the contextual information present in the scene. To address these issues, I propose a dual-network approach comprising two novel networks for separate tasks: scene sketch segmentation and scene sketch recognition. The first network, the Class-Agnostic Visio-Temporal Network (CAVT), detects individual objects in a scene sketch using a class-agnostic object detector and groups strokes with its post-processing module. This network can distinguish object instances at the stroke level, independent of their categories. The second network, Context-Aware Graph Attention Transformer Network (CGAT-Net), processes individual sketch objects within the scene and leverages inter-object relationships to find their appropriate categories. This work is the first to apply a context-based sketch recognition approach by leveraging a novel Transformer-based Graph Attention Network within scene sketches. Additionally, the literature lacks free-hand scene sketch datasets with both instance and stroke-level class annotations. To fill this gap, I collected the largest Free-hand Instance- and Stroke-level Scene Sketch dataset (FrISS) that contains 1,000 scene sketches and covers 403 different object classes with dense annotations. Extensive experiments on FrISS and other scene sketch datasets demonstrate that the dual-network approach, combining CAVT and CGAT-Net, as well as each network individually, outperforms existing methods in their respective domains.
Hierarchical spatial decompositions under local differential privacy
The popularity of smartphones, GPS-equipped devices, social networks, and connected vehicles continues to increase the volume of spatial data available for collection and analysis. Spatial decompositions assist in handling big spatial data, and they have been commonly used in the centralized differential privacy (DP) literature for range query answering, spatial indexing, count-of-counts histograms, data summarization, and visualization. However, their applications under the emerging local differential privacy (LDP) notion are relatively scarce. In this thesis, we study the problem of building hierarchical spatial decompositions under LDP, focusing on two methods: quadtrees and kd-trees. We develop two solutions for quadtrees: a baseline solution that is inspired by the centralized DP literature, and a proposed solution that utilizes a single data collection step from users, propagates density estimates to remaining nodes, and performs structural corrections to the quadtree. Since kd-trees rely on node medians which are data-dependent, we observe that it is not feasible to build kd-trees using a single data collection step. We therefore propose an iterative solution that constructs kd-trees in top-down fashion by utilizing a novel algorithm for estimating node medians at each tree depth. We experimentally evaluate our quadtree and kd-tree algorithms using four real-world spatial datasets, multiple utility metrics, varying privacy budgets, and tree parameters. Results demonstrate that our algorithms enable the building of accurate spatial decompositions that provide high utility in practice. Notably, our quadtrees and kd-trees achieve substantially lower errors in answering spatial density queries (up to 10-fold improvement) when compared with a state-of-the-art method.
Aerogel-based 3D printed optofluidic waveguided multi-channel photocatalytic microreactors
This thesis investigates the use of additive 3D printing technology for creating micro photoreactors within aerogels. The study focuses on developing a low-cost 3D printing system tailored for aerogels and optimizing an ink formulation suitable for 3D printing processes. The research begins with customizing an off-the-shelf 3D printer to fabricate a low-cost system specifically for aerogel-based structures. This customized printer integrates a 3D-printed syringe pump system and custom software for accurate calibration and printing of sub-millimeter lines. The ink formulation designed for 3D printing comprises a silica aerogel starter solution with additives and titanium dioxide nanoparticles. Post-printing steps including gelation and solvent exchange refine the aerogel's structures enhancing stability for micro photoreactors designed to interact with water. Rheological characterization studies inform ink optimization ensuring durability and usability over time. The optimized 3D printing system and ink formulations demonstrate advancements in printing aerogels with additives where applications promise impactful solutions and future advancements in materials engineering and environmental sciences.
Self-supervised prediction contrast time frequency for industrial fault detection
Industrial informatics produce vast data, but fault detection is hindered by dynamic processes and non-linear interactions. Deep learning and machine learning help, but sensor errors, transmission failures, and scarce labels remain issues. Self-supervised learning (SSL) uses unlabeled data to train models and generate labels. Prediction contrast learning, a key SSL method, preserves temporal patterns, making it ideal for time-series data while avoiding constant parameter tuning required by traditional methods. To tackle fault detection task, this thesis proposes a novel self-supervised learning architecture using prediction contrast learning to enhance fault detection (FD) in industrial settings. Our approach integrates time and frequency domain information to capture crucial temporal dependencies and improve model robustness. The key contributions of this work include: the novel combination of time-frequency information in SSL, the first application of prediction contrast learning for FD, a complex network analysis for real-time performance and scalability, and an extensive evaluation including ablation studies. To address the challenges of data scarcity and enhance fault detection accuracy in complex environments, our proposed model incorporates data transformations that improve robustness while preserving essential temporal relationships for time series analysis. We evaluated our approach using the Tennessee Eastman Process (TEP) benchmark datasets, including both Ricker and Rieth versions, which are widely recognized standards for testing FD systems in industrial processes. We also evaluate our model in Secure Water Treatment (SWaT) and Water Distribution (WADI) dataset, which contain anomalies such as intentional attack scenarios injected into the water management system. Experimental results on the TEP datasets demonstrate that our approach significantly outperforms existing benchmark models, achieving a True Positive Rate (TPR) of 92.04% and reducing the Average Detection Delay(ADD) to 22.18 steps, Correct Diagnosis Rate (CDR) of 97.62% effectively addressing data scarcity challenges and enhancing fault detection accuracy in complex industrial environments. Similar outperforming results have also been observed in other benchmark datasets. To ensure practical applicability, we conduct complex network topology analysis to evaluate our model's real-time performance and scalability in both cloud and edge computing environments, considering industrial networks as complex systems with non-trivial topological features. This analysis provides a comprehensive comparison of latency differences between edge and cloud computing architectures, addressing a critical gap in the fault detection literature. The proposed SSL architecture, combined with our comprehensive network analysis, offers a robust solution for FD in industrial chemical facilities. By integrating advanced deep learning techniques with practical considerations for implementation in various computing environments, our work paves the way for more efficient and reliable fault detection systems in industry.
High-performance multi-GPU concurrent queues: Case study with parallel Bellman-Ford SSSP
This thesis presents a novel implementation of a concurrent FIFO queue algorithm for both single-GPU and multi-GPU environments, applied to the Single-Source Shortest Path (SSSP) Bellman-Ford algorithm. The primary contribution of this work is the design and implementation of a multi-GPU concurrent queue system using NVIDIA's NVSHMEM, which has not been previously explored. The Bellman-Ford algorithm is used as a case study to evaluate the performance of the proposed queue system, with this multi-GPU implementation being the first known instance of its kind. Experimental results demonstrate that the multi-GPU queue implementation achieves a maximum speedup of 3.92x and an average speedup of 3.04x over the single-GPU baseline on four NVIDIA A100 GPUs. When applied to the Bellman-Ford algorithm, the multi-GPU system achieves a maximum speedup of 3.794× and an average speedup of 3.573× compared to the single-GPU implementation, tested on a generated benchmark and 10 graphs of different kinds taken from the SuiteSparse Matrix Collection. These findings highlight the efficiency of the multi-GPU queue system for graph processing tasks and contribute to advancements in high-performance computing by addressing practical challenges in parallel computing.
Leveraging weak supervision for cell localization in digital pathology using multitask learning and consistency loss
Cell detection and segmentation are integral parts of automated systems in digital pathology. Encoder-decoder networks have emerged as a promising solution for these tasks. However, training of these networks has typically required full boundary annotations of cells, which are labor-intensive and difficult to obtain on a large scale. However, in many applications, such as cell counting, weaker forms of annotations--such as point annotations or approximate cell counts--can provide sufficient supervision for training. This thesis proposes a new mixed-supervision approach for training multitask networks in digital pathology by incorporating cell counts derived from the eyeballing process--a quick visual estimation method commonly used by pathologists. This thesis has two main contributions: (1) It proposes a mixed-supervision strategy for digital pathology that utilizes cell counts obtained by eyeballing as an auxiliary supervisory signal to train a multitask network for the first time. (2) This multitask network is designed to concurrently learn the tasks of cell counting and cell localization, and this thesis introduces a consistency loss that regularizes training by penalizing inconsistencies between the predictions of these two tasks. Our experiments on two datasets of hematoxylin-eosin stained tissue images demonstrate that the proposed approach effectively utilizes the weakest form of annotation, improving performance when stronger annotations are limited. These results highlight the potential of integrating eyeballing-derived ground truths into the network training, reducing the need for resource-intensive annotations.
Development of algorithms and signal enhancement techniques for seismocardiogram-based hemodynamic parameter estimation
Continuous monitoring plays a critical role in bridging the information gap that exists between periodic hospital visits. Recent advancements in sensor technology and biomedical signal processing have paved the way to the development of sensor systems capable of continuous monitoring of physiological signals and diagnosing various pathologies. Among various types of physiological signals, the seismocardiogram (SCG) is particularly popular in wearable system design. The SCG corresponds to the chest micro-vibrations caused by blood ejection and heart contraction during each cardiac cycle. This thesis focuses on developing algorithms to derive various hemodynamic parameters and enhancement methods for SCG signal analysis. These methods aim to improve signal quality, enable accurate feature extraction, and enhance the reliability of SCG-based physiological assessment. Chapter 2 explores the temporal and spectral relationships between SCG signals and hemodynamic parameters like pre-ejection period (PEP) and left ventricular ejection time (LVET). It presents regression models to estimate these parameters and investigates the relationship between thorax impedance and SCG-derived features. Chapter 3 introduces a two-step hierarchical framework for apnea detection and respiration pace assessment using SCG signals. This model includes binary classification for detecting breath-holding episodes and multi-class classification for categorizing breathing patterns (normal, slow, fast). Chapter 4 presents a comparative evaluation of various denoising algorithms aimed at improving SCG signal quality during exercise. It assesses the performance of different heart rate estimation methods and compares the impact of seven different denoising algorithms on SCG-based heart rate estimation. Overall, these findings collectively support the versatility and potential of SCG in advancing personalized healthcare, particularly in wearable health monitoring and continuous physiological assessment.
Learning Bayesian networks under local differential privacy
Bayesian networks are widely used for causal discovery and probabilistic modeling across diverse domains including healthcare, multi-dimensional data analysis, environmental modeling, and industrial processes. Although previous work has studied the learning of Bayesian networks under centralized differential privacy, to the best of our knowledge, the problem of learning Bayesian networks under local differential privacy (LDP) remains open. In this thesis, we address this problem by proposing two solution methods for learning Bayesian networks under LDP: LDP-BN and LDP-BN+. Our first solution called LDP-BN utilizes a novel algorithm for computing mutual information values necessary for building a Bayesian network under LDP, but it suffers from high utility loss since the privacy budget needs to be divided into many pairs of attributes and candidate parent sets. To reduce the amount of noise, we propose LDP-BN+ which utilizes a novel density-aware covering design algorithm that ensures all necessary mutual information values will be computed while the privacy budget is used more effectively. We experimentally evaluate LDP-BN and LDP-BN+ using multiple utility metrics and datasets. Results show that LDP-BN+ outperforms LDP-BN and enables the generation of high-utility Bayesian networks that can be used in practice.
Locally differentially private mechanisms for sequential and high-dimensional data analysis
With the increasing need for data privacy and protection, local differential privacy (LDP) has emerged as a widely accepted standard for privacy-preserving data col- lection. While LDP has been studied extensively for singular data, its application to sequential and multidimensional data remains underexplored. In this thesis, we propose novel LDP mechanisms for advancing the state-of-the-art in the application of LDP to these two data types. First, we propose Prima for learning discrete-time Markov chain models from sequential data under LDP. Markov chains are frequently used in the analysis and modeling of sequential data such as location traces, time series, natural language, and speech. However, considering that many such data sources are privacy-sensitive, it is imperative to design privacy-preserving methods for learning Markov chains. Prima addresses this need. In Prima, each user locally encodes and perturbs their sequential record on their own device using LDP protocols. For this purpose, we adapt two bitvector-based LDP protocols (RAPPOR and OUE); and furthermore, we develop a novel extension of the GRR protocol called AdaGRR. We also propose to utilize custom privacy budget allocation strategies for perturbation, which enable uneven splitting of the privacy budget to better preserve utility in cases with uneven sequence lengths. On the server side, Prima uses novel algorithms for estimating Markov probabilities from perturbed data. We experimentally evaluate Prima using three real-world datasets, four utility metrics, and various combinations of privacy budget and budget allocation strategies. Results show that Prima enables learning Markov chains with high utility and low error compared to Markov chains learned without privacy constraints. Second, we propose MCM (Matrix-Based Data Collection Mechanism), a novel LDP mechanism for the collection of multidimensional data. In MCM, each user encodes their multidimensional record into a bitmatrix. The rows of the bitmatrix are perturbed in a way that satisfies LDP. Then, the key contribution of MCM lies in its novel server-side estimation process, which enables the server to recover co- occurrence counts of all pairs of values in attribute domains. We utilize MCM to perform feature selection from multidimensional data using two popular feature se- lection metrics: information gain and chi-square. We experimentally evaluate MCM and the accompanying feature selection algorithms using three datasets, two utility metrics, and varying privacy budgets. Furthermore, we compare our solution with LDP-FS, a state-of-the-art solution for feature selection under LDP. We experimen- tally show that our solution yields more accurate information gain and chi-square values compared to LDP-FS, thereby improving the state-of-the-art. Furthermore, we demonstrate that correlations between attributes are accurately preserved in feature selection while LDP is satisfied.
Fairnessdplab: Diferansiyel mahremiyetin gözetimli yz algoritmalari üzerindeki adalet etkisinin analizi
This study investigates the impact of differential privacy (DP) on fairness in supervised artificial intelligence (AI) algorithms through the development and evaluation of a benchmark platform, FairnessDPLab. The platform systematically examines the interplay between privacy preservation and fairness across three widely studied datasets: Adult, COMPAS, and German Credit. Utilizing both machine learning and deep learning models, the study explores the effects of varying privacy budgets and their implications on various fairness metrics. The findings indicate that the balance between privacy and fairness can vary under different conditions, such as dataset characteristics or model choices. For instance, lower privacy budget values enhance privacy but may lead to undesirable changes in model accuracy and fairness measures. Deep learning models show higher sensitivity to privacy settings than traditional machine learning methods, such as logistic regression and random forests. The study also highlights the importance of carefully selecting privacy metrics, suggesting that they may need to be adapted based on the domain of application. This work contributes to the growing body of literature on equitable AI systems by providing a robust framework for analyzing fairness under differential privacy constraints. The insights derived from this study offer practical guidelines for designing privacy-preserving AI systems that balance utility and fairness in diverse applications.
Backdoor Attacks Against Text Classification Models: A Comprehensive Benchmark and Ensemble-Based Defenses
Textual backdoor attacks present a critical threat to the security and trustworthiness of NLP systems by embedding stealthy triggers in training data. Such attacks enable adversaries to manipulate model predictions without harming performance on clean inputs. This thesis provides a comprehensive benchmark study of backdoor attacks targeting text classification models, encompassing both traditional machine learning models (Logistic Regression, Naive Bayes, Decision Tree, Random Forest) and neural architectures (LSTM, DistilBERT, BERT, RoBERTa). We evaluate various attack methods—including AddSent, WordInj, SynBkd, StyleBkd, and BITE—across multiple benchmark datasets (IMDb, SST-2, HateSpeech, Tweet) under varying poison rates (0.5% to 10%). Empirical results reveal that transformer-based models, while achieving high clean accuracy, are especially vulnerable to backdoor triggers. AddSent emerges as the most potent attack, consistently achieving the highest attack success rates (ASRs). Style-transfer attacks (e.g., using Bible or Shakespeare styles) also remain highly effective, often reaching ASRs above 98% at just 3% poison rate. Increasing poison rates amplify both ASR and detectability, posing trade-offs for adversaries and defenders. Traditional models suffer greater drops in clean accuracy at higher poison rates, while transformer models often preserve clean accuracy despite hidden triggers—making them harder to detect in real-world settings. To counter these attacks, we propose ensemble-based defenses that combine multiple model architectures at inference time. Through experimental evaluations, we observe that ensembles of traditional models successfully reduce ASR but also compromise accuracy. Ensembles of transformer models achieve high accuracy, but do not succeed in reducing ASR to acceptable levels. We therefore propose joint ensembles that combine traditional models with modern transformer models. Our approach significantly reduces ASR while maintaining competitive clean accuracy, highlighting the benefits of architectural diversity and serving as a potential defense mechanism for building backdoor-resilient text classification systems.
Online long-term point tracking in the foundation model era
Point tracking aims to identify the same physical point across video frames and serves as a geometry-aware representation of motion. This representation supports a wide range of applications, from robotics to augmented reality, by enabling accurate modeling of dynamic environments. Most existing long-term tracking approaches operate in an offline setting, where future frames are available to refine predictions and recover from occlusions. However, real-world scenarios often demand online predictions: the model must operate causally, using only current and past frames. This constraint is critical in streaming video and embodied AI, where decisions must be made immediately based on past observations. Under such constraints, viewpoint invariance becomes essential. Visual foundation models, trained on diverse large-scale datasets, offer the potential for robust geometric representations. While they lack temporal reasoning on their own, they can be integrated into tracking pipelines to enrich spatial features and improve robustness. In this thesis, we address the problem of long-term point tracking in an online setting, where frames are processed sequentially without access to future information or sliding windows. We begin by evaluating the suitability of visual foundation models for this task and find that they can serve as useful initializations and be integrated into tracking pipelines. However, to enable long-term tracking in an online setting, a dedicated design is still required. In particular, maintaining coherence over time in this causal regime requires memory to propagate appearance and context across frames. To address this, we introduce Track-On, a transformer-based model that treats each tracked point as a query and processes video frames one at a time. It predicts correspondences through patch classification followed by local refinement. To ensure temporal consistency, memory modules are incorporated to carry relevant information across frames. Track-On sets a new state of the art across seven public benchmarks, demonstrating the feasibility of long-term tracking without future access.
Kuş bakışı görünüm segmentasyon modellerinin sağlamlığını ve genelleme yeteneğini geliştirme
Kuş Bakışı Görünüm, çevrenin üç boyutlu mekânsal düzenini yapılandırılmış bir şekilde sunması nedeniyle, otonom sürüşteki pek çok görev için kritik öneme sahiptir. Araç kameralarından kuş bakışı görünüm çıkarmak, LiDAR tabanlı yöntemlere kıyasla daha uygun maliyetli ve ölçeklenebilir bir alternatif sunar. Ancak yalnızca görsel verilere dayanan kuş bakışı görünüm üreten modeller bazı zorluklarla karşılaşmaktadır: Bu modeller, bozulmuş girdilerle (örneğin bulanık kamera görüntüleri veya kamera arızaları) karşılaştıklarında yeterince dayanıklı değildir ve mevcut veri kümeleri yeterli çeşitliliğe sahip olmadığından genelleme konusunda da yetersiz kalmaktadır. Dayanıklılık ve genelleme, otonom sürüş sistemlerinin gerçek dünya koşullarında güvenli ve güvenilir bir şekilde çalışabilmesi için kritik faktörlerdir. Bu tezde, her iki soruna da çözüm sunmayı hedefliyoruz. İlk olarak, kuş bakışı görünüm üreten modellerin dayanıklılığını artırmak amacıyla, büyük görsel temel modellerin (vision foundation models) güçlü ve genellenebilir özelliklerinden faydalanarak bu modelleri parametre açısından verimli bir adaptasyon tekniği ile BEV algılama sistemlerine entegre ediyoruz. Yaptığımız deneyler, çeşitli girdi bozulmaları altında modelin dayanıklılığında artış sağlandığını ve model boyutu ile girdi çözünürlüğü arttıkça bu kazanımların daha da belirginleştiğini göstermektedir. Ayrıca yöntemimizin daha hızlı yakınsama sağladığını ve daha az öğrenilebilir parametreye ihtiyaç duyduğunu da ortaya koymaktayız. İkinci olarak, sınırlı veri kümesi kapsamı ve çoklu görüşlü sürüş sahnesi üretiminde esnek girdi kontrolünün olmaması sorununu ele almak amacıyla, orta düzeyde zorluk içeren ve veri kümesinde az temsil edilen sahne yapılarını hedefleyen yönlendirmeli bir yerleşim üretim hattı öneriyoruz. Gerçek sürüş sahnelerinin dağılımını analiz ederek, bir üretici modeli bu dağılımın belirli bölgelerinde sahneler üretmeye yönlendiriyoruz. Bu yönlendirmeli örneklerle kuş bakışı çıkaran modellerin eğitim kümesini genişletmek şu an için doğrudan bir performans artışı sağlamasa da, üretilen yerleşimlerin büyük ölçüde gerçek dünya yapılarına uygun olduğu ve hedeflenen dağılım bölgesine denk düştüğü gözlemlenmiştir. Bu durum, önerdiğimiz yöntemin, kuş bakışı algı modellerinin genelleme yeteneğini artırmaya yönelik hedef odaklı sentetik veri üretimi açısından gelecek vadettiğini göstermektedir.
Inference attacks and defenses in continuous location sharing with local differential privacy
Local differential privacy (LDP) has recently emerged as a commonly used privacy standard. With the growing popularity of LDP, recent works started applying LDP to location data collection and privacy-preserving usage of location-based services (LBS). However, in many practical applications of location data sharing and LBS usage, the user must continuously share his/her location with the data collector (e.g., LBS provider). Although the privacy of each individual location is protected with LDP, correlations between the user's consecutive locations can be exploited to infer the user's true locations. Following this idea, in this thesis we propose novel inference attacks against continuous location sharing under LDP. We develop attacks in two categories: statistical attacks based on Bayesian adversary formulation for stationary users and Hidden Markov Model (HMM) based attacks for mobile users. We also propose two extensions for our HMM-based attacks (informed and chain attacks), which enable the attacker to incorporate aggregate population statistics or use multiple iterations of HMM construction to improve attack effectiveness. We adapt and apply our attacks to four popular LDP protocols (GRR, RAPPOR, OUE, OLH), three datasets, and varying privacy levels. Experiment results show that our attacks are highly effective, which highlights the privacy risks of using LDP in continuous location sharing. Furthermore, results show that the attacks are better suited for the types of users they were designed for, i.e., statistical attacks achieve higher effectiveness than HMM-based attacks in case of stationary users, whereas HMM-based attacks achieve higher effectiveness in case of mobile users. Although our HMM-based attacks are effective on bitvector-based LDP protocols (such as RAPPOR and OUE), we observe that they suffer from efficiency problems, i.e., high memory usage and execution time. These hinder the scalability of our attacks, especially when grid sizes are large or when the attacker aims to achieve precise location inference. We therefore propose heuristic methods to improve the time and memory-efficiency of our attacks by removing states with low occurrence probability from the attack HMMs. We experimentally show that our improvements can reduce the memory and time costs of the attacks by more than an order of magnitude, while maintaining similar effectiveness compared to their original versions. Finally, we propose three defense strategies against our attacks: Memoization, Replay, and Replication. In memoization, each user stores a perturbed version of their original value, which is perturbed for a second time before being reported to the data collector. In replay, each user maintains a cache of previous locations and their perturbed counterparts. When a certain location is repeated, the user replays the old perturbed version rather than performing a fresh perturbation. In replication, each user replicates their previously reported value if their current location is similar to their previous location. We evaluate the three defenses using the same experiment setup containing four LDP protocols, three datasets, varying privacy levels, and mobile and stationary users. Results show that our defenses are successful in reducing attack effectiveness. We critically analyze the success, efficiency, and utility aspects of the three defenses under varying conditions, and provide recommendations regarding when to use which defense.
Efficient autonomous driving with foundation models
Driving safely in urban environments is a task that requires a deep understanding of complex, dynamic environments and the ability to react swiftly and effectively to unexpected events. Recent advances in large foundation models have shown impressive capabilities to reason and generalize. In this thesis, we explore the potential and feasibility of foundation models for driving. First, we start by building on top of previous work showing promising performance in robotics tasks by formulating reinforcement learning as a language modeling problem using auto-regressive transformers. These approaches, however, assume the state is relatively simple, often representable as a single vector. In driving, the state is more complex, involving multiple agents that interact with each other, raising the need to find a compact, effective state representation. In Carformer, we propose a language model using learned, self-supervised, object-centric representation based on slot attention. We analyze different possible state representations in a privileged setting and find that our proposed representation outperforms both scene-level and hand-crafted object-centric representations, achieving the state-of-the-art on the Longest6 benchmark. After showing strong results with a language modeling approach in a privileged setting, we then focus on a more realistic, end-to-end setting. While foundation models further improve with scale, this results in additional inference latency. Latency is one of the main hurdles facing the deployment of large foundation models for real-time applications like driving. To address this, we propose a dual approach pipeline, ETA, that utilizes a large foundation model in tandem with a smaller, lightweight model. To minimize latency, we shift the computation of the large foundation model to an earlier time-step following it with a forecasting module to adapt the features to the present. Unlike previous work on dual approaches, we batch the large model inference to enable the decisions to benefit from the large model features at every time-step. Outperforming all previous approaches at a fraction of the latency, we highlight the feasibility of large foundation models in self-driving.
Learning object-centric representations based on slots in real world scenarios
A central goal in artificial intelligence is to enable machines to perceive the visual world as a composition of distinct objects. This ability for object-centric understanding is essential for generative models that support fine-grained, controllable content creation and editing. However, state-of-the-art diffusion models process images holistically and are conditioned on text, creating a semantic misalignment when tasked with object-level manipulation. As a result, researchers face a fundamental challenge: either adapt powerful but text-biased models or build specialized models from scratch, often with reduced capacity. This dissertation addresses this problem by introducing a framework that adapts pretrained generative models for object-centric image and video synthesis. Our analysis highlights a core challenge in current approaches: achieving high-quality generation requires balancing global scene coherence with disentangled, object-level control. To address this, we propose an adaptation strategy that integrates object-specific conditioning into pretrained models while preserving their valuable priors. Extending this framework to video further amplifies the difficulty, as maintaining temporal coherence and consistent object identity across frames is critical. For static images, we introduce SlotAdapt, a method that augments diffusion models with lightweight slot-based modules. A register token captures background and style, while slot-conditioned components encode object-specific information. This dual-pathway design mitigates text-conditioning bias and provides precise, object-centric control, leading to state-of-the-art results in object discovery, segmentation, compositional editing, and controllable image generation. We then extend the framework to video. Using Invariant Slot Attention (ISA) to disentangle object identity from pose, combined with a Transformer-based temporal aggregator, our approach ensures consistent object representation and dynamics across time. This framework sets new benchmarks in unsupervised video object segmentation and reconstruction, while enabling advanced video editing capabilities, including object removal, replacement, and insertion, all without explicit supervision. Overall, this work establishes a general and scalable approach to object-centric generative modeling for both images and videos. Beyond setting new technical baselines, it expands the design space for interactive and controllable generative tools, bridging the gap between human object-based perception and machine learning models. These contributions open new directions for structured, intuitive, and user-driven AI applications in creative, scientific, and practical domains.
Dynamical phases of short-term memory: The emergence of slow-point and limit cycle mechanisms in recurrent neural networks
This thesis investigates the computational principles of short-term memory, specifically examining how information is maintained through sequential neural activity in recurrent neural networks (RNNs). While a longstanding theory posits that persistent activity holds information in memory—an idea that has dominated neuroscience for decades and is conceptually aligned with fixed-point attractors in dynamical systems—this work explores the alternative hypothesis that memory can be encoded in the transient, sequential firing patterns of large neural populations. We identify and rigorously characterize two primary mechanisms capable of supporting this dynamic process: slow- point manifolds, which generate direct, non-oscillatory sequences through a saddle-node bifurcation, and limit cycles, which provide a periodic, oscillatory basis for temporal coding. Through the use of simplified, yet analytically tractable, dynamical system models, we derive theoretical scaling laws that precisely relate the critical learning rate— the threshold beyond which training becomes unstable—to the duration of the memory delay. These laws predict that the difficulty of learning increases as a power-law function of the delay, but with different exponents for each mechanism. We provide rigorous empirical validation for these theoretical predictions by training and evaluating a large-scale dataset of over 80,000 RNNs on memory-dependent tasks. The findings reveal a fundamental and robust trade-off among memory duration, learning speed, and network stability, offering a principled computational explanation for the historical difficulty of learning long-term dependencies in RNNs. This framework also offers a new perspective on the biophysical constraints on memory in biological systems, suggesting why certain computational strategies might be favored over others. Furthermore, this work highlights how subtle alterations in task design can fundamentally change the underlying memory mechanisms learned by a network, providing concrete, experimentally testable predictions for systems neuroscience aimed at differentiating these computational strategies in vivo.