206
Arşivlenen Tez
0
DOI Atanmış
0%
DOI Oranı
Anabilim Dalı
Derin sinir ağ tabanlı dosya ve veri parçası sınıflandırılması
Bu çalışmada sunulan araştırma, adli bilişim ve bilgi güvenliği uygulamalarında hayati önem taşıyan dosya ve veri türü sınıflandırmasına yönelik bir çözüm önermektedir. Son onbeş yılda dosya ve veri türü sınıflandırması araştırmalarında kullanılan yöntemler, dosya uzantısı tabanlı yöntemler, sihirli bayt tabanlı yöntemler ve içerik tabanlı yöntemlerdir. Bu yöntemlerden uzantı tabanlı ve sihirli bayt tabanlı yöntemler, dosya başlığında yer alan sihirli baytlar ve dosya uzantıları kolayca değiştirilebildiğinden dolayı yetersiz yöntemlerdir. İçerik tabanlı yöntemler sihirli bayt ve dosya uzantıları gibi değişikliklere karşı dirençli olduğundan son yıllarda bu alanda yapılan çalışmalar hızlı bir şekilde artmıştır. İçerik tabanlı yöntemlerin kullanıldığı çalışmaların çoğunda çok az sayıda dosya ve veri türü kullanılmaktadır. Bu alanda yapılan çok az sayıda çalışmada ise çok sayıda dosya ve veri türü kullanılmaktadır. Ancak bu çalışmalardaki dosyaların bazıları işletim sistemlerinde çok az kullanılan dosya türleridir. Bu çalışmada en çok kullanılan 15 dosya ve veri türünü içeren içerik tabanlı dosya ve veri parçası sınıflandırma yöntemi sunulmuştur. Sınıflandırma alanında son yıllarda derin sinir ağları yaygın bir şekilde kullanılmaya başlanmıştır. Kullanılan sınıflar eğitim setinde yeterince iyi genellediğinde çok iyi sınıflandırma performansı elde edilmektedir. Bu çalışmada da dosya ve veri sınıflandırması problemine derin sinir ağ mimarileri kullanılarak çözüm aranmaktadır. Önerilen yöntemde iki seviyeli hiyerarşik model kullanılmakta olup bu hiyerarşik sınıflandırma sisteminde ilk seviyede birkaç alternatif sınıflandırma modeline dayanan deneyler yapılmıştır. Alternatif sınıflandırma modelleri entropi bazlı dört farklı durum ve sınıflandırma bazlı üç farklı algoritma kullanılmaktadır. ıkinci seviyede ise kazanan model üzerinden derin sinir ağları kullanılmıştır. İşletim sistemlerinde kullanılan en küçük küme birim büyüklüğü olan 4 kilobayt ve 8 kilobaytlık dosya ve veri parçaları kullanılarak 2-gram analizi ile öznitelikler çıkartılmaktadır. Çıkarılan bu öznitelikler üç farklı makine öğrenmesi algoritması kullanılarak entropiye dayalı olarak gruplara ayrılmaktadır. Daha sonra bu ayrılan gruplar üzerinden dosya ve veriler derin sinir ağlar kullanılarak tür tabanlı sınıflandırma yapılmakadır. 4 kilobayt ve 8 kilobayt için sınıflandırma doğruluk oranları sırasıyla %92.80 ve %94.67 dir. Yapılan bu çalışmada doğruluk oranını önemli ölçüde azaltan şifrelenmiş veri türü olan aes256 kullanılmasına rağmen benzer dosya türü kullanılarak yapılan en iyi çözüm ile karşılştırıldığında bizim önerdiğimiz yöntem doğruluk oranını %6.87 oranında artırdığı görülmektedir.
Yapay sinir ağları ve bulanık mantık ile sanayi üretim endeksi tahmini
Sanayi, hammaddelerin insan kullanımına elverişli hale getirilmesi için temini, işlenmesi ve üretilmesidir. Günümüzde sanayi sektörünün durumunu gösteren, üretim faaliyetindeki artış ya da azalışını karşılaştırmalı olarak izleyen göstergelerden biri de sanayi üretim endeksidir. Sanayi sektörü gayrisafi yurt içi hasılanın belirlenmesinde önemli bir öncüldür. Bunun en temel nedeni sanayi üretiminin ekonomiyi pek çok açıdan etkilemesidir. Yurt içi tüketim olsun ihracat olsun sanayi üretimindeki artış ekonomik büyüme ile alakalıdır. Sanayi üretiminin hem tüketim ihtiyacına hem de faiz oranlarına duyarlı olmasından dolayı sanayi üretim endeksi büyüme ile doğrudan ilişkilidir. Sanayi üretim endeksi büyümenin en önemli göstergelerinden biri olduğu için, sanayi üretim endeksinin gelecekte ne yönde olacağını tahmin etmek de önemlidir. Yapılmış olan bu yüksek lisans tez çalışmasında bir yıla ait elektrik üretim endeksinin tahmin edilmesi için önceki yıllardaki veriler kullanılarak yapay sinir ağı ve bulanık mantık modelleri üretilmiştir. YSA'da transfer fonksiyonları ve nöron sayılarının değiştirilmesi ile farklı modeller elde edilmiş ve bu modeller tahminde kullanılmıştır. Benzer şekilde, ANFIS modelleri için de çeşitlendirme üyelik fonksiyonu ve üyelik fonksiyon sayısını değiştirerek elde edilmiştir. ANFIS'de yapılan tahminler bu modeller üzerinde gerçekleştirilmiştir. YSA modelleri incelendiğinde ANFIS'de olduğu gibi zaman, maden ve imalat üretim endeksleri girdilerinden oluşan sistemin sadece zaman veya sadece imalat-maden üretim endekslerine dayalı tahmin yapan sistemden daha iyi sonuç verdiği de görülmüştür.
Detecting novel behavior and process improvement with multi-modal process mining
The importance of data has increased, especially with the spread of Internet of Things (IoT)-like technologies as a result of the 4th Industrial Revolution. In order to make sense of the data, valuable sciences have also gained importance in this direction. Data science is one of these sciences. Although it differs from data science in order to make sense of the data, process mining (PM) plays a dominant role in modeling process-based systems by intersecting with data science. When PM intersects with data science and includes other perspective data in the analysis, problems may occur in performing error-free operations due to data science-related errors. One of these processes is; to detect new behavior with the combination of other perspective data (e.g., data perspective). For existing studies, multi-perspective analysis can be performed with the support of data science by including other perspective data in the analysis. Within the scope of this thesis, by adding other perspective data to the control-flow perspective, multi-modal analysis can be carried out with a PM-intensive approach without loss of context. Multi-modal analysis is the analysis of an activity in combination with the attributes that affect that activity. In addition, instead of using ready-made event logs, event logs can be produced through virtual factories that shine with the 4th Industrial Revolution, and improvement can be made on modeled processes without data from real processes. Essentially, with this strategy, the gap for approaches that use other perspective data with loss of context is filled, but unlike existing process improvements, process improvement can be worked on without waiting for real system event logs. With these advantages, a PM framework that can produce event logs between independent units (i.e., from producer to processor) is proposed. The transmission of these logs between units in a distributed structure is carried out with a broker-based architecture. Root-cause analysis (RCA) can be performed with the help of metrics through other perspectives added to the control-flow perspective for the taken records, and the detection of new behavior in desired level can be done with multi-modal analysis, without loss of context, unlike especially multi-perspective analysis with data exploration mode. In order to verify the framework within the scope of the thesis, an event log is produced via the Business Process Modeling Notation (BPMN) model found with the repairExample.xes event log. Verification is carried out for the generated event logs with multi-perspective BPMN and multi-modal process discovery. Event logs that are verified are transferred to the processor part. The records taken in the processor part are first written to disk. Afterwards, one of the new event logs is selected and root cause analysis is performed. Through other perspective data added to the control perspective, root cause analysis is done multi-modal. Token and alignment based replay is used for predictor and response. If necessary, process improvement is provided through model repair. Fitness is the most important metric for improvement. Precision and generalization can provide insight into the system. In time-related mode, analysis can become a hybrid with variant and query support. Keywords: Process mining, software architecture, machine learning, multi-modal analysis, quality in process mining, root cause analysis, process enhancement.
A skill assessment model for human resources skill development enabling digital transformation: Dx-hrsam
In today's era of rapid technological advancement, digital transformation stands at the forefront of Industry 4.0, revolutionizing both personal and professional spheres. This paradigm shift is fueled by the integration of cutting-edge technologies such as artificial intelligence, the Internet of Things, smart machines, and automation. These disruptive innovations, intertwined with ongoing societal shifts, are not only reshaping our daily lives but also redefining the workplace, giving rise to novel professions and necessitating a transformation of essential skills and competencies. Human Resources (HR) management must proactively embrace these changes, developing strategies that foster learning and adaptability to optimize the current workforce and prepare for the workforce of the future. To this end, a MLR was conducted, examining emerging professions and skills/competencies arising from digital transformation. This analysis delved into the current state of the literature, revealing critical gaps to inform future research endeavors. An extensive review of 6055 studies culminated in a refined pool of 23 studies, providing valuable insights to address key research questions. Recognizing a need for a comprehensive assessment model in this domain, a model was developed by defining new professions and skills, culminating in the creation of distinct role sets and skill sets, named as Digital Transformation Human Resource Skills Assessment Model (DX-HRSAM). A five-level proficiency scale was established, and a meticulous role-skill mapping was performed for each role, resulting in a robust and insightful model. This model was subsequently subjected to expert scrutiny, leading to further refinements based on valuable feedback. A pilot study was then conducted, followed by multiple studies to validate the model's consistency and comprehensiveness. The outcome is a validated model that offers a valuable framework for understanding and navigating the evolving landscape of skills and professions in the digital age.
Deprem tweetleri üzerinde retweet tahmini
On February 6, 2023, an earthquake centered in Kahramanmaraş killed or damaged many people. In the aftermath of these devastating earthquakes, the efficiency of communication channels in the crisis zone is of vital importance. While a few decades ago there was no indication of the existence of social media, today social media platforms have become people's main communication channels. Twitter, one of these platforms, is widely used in Turkey. Social media provides the opportunity to reach millions of people with a shared post. The amount of interaction a post receives increases the possibility of being noticed by other users on social media. In this thesis, using the tweets posted during and after the earthquake centered in Kahramanmaraş on February 6, 2023, the retweet interaction amounts were divided into two classes. These classes are 'non-low' and 'moderate-high' classes. The data was captured with Python's Snscrape Library as 38 days of data covering February 6, 2023 - March 15, 2023. The following operations were then performed respectively: Tweet text was cleaned. Spelling mistakes were corrected with the python Zemberek Module. Words were parse to their roots with Zeyrek Module. Stop words were deleted. Stop words were deleted. The dataset was simplified and idf values of unique words in the first week tweets were calculated. Unique words were grouped according to their idf value ranges. By adding 400 unique words from different idf ranges to the dataset, 7 dataset versions consisting of different unique word groups were obtained. Among these sets, the word set that best represents the tweet text was investigated. The XGBoost model was used in the analysis. We also investigated the interaction type and class threshold limit that would be the best class label. The best class label was 'Retweet' and the best class distinction limit was observed as 2. The words that best represents the dataset were found to be the 400 words with the lowest idf value. These words were added to the dataset as Binary Bag of Words. Then, classification was performed with various Deep Learning and Machine Learning models. These models are Random Forest, XGBoost, LSTM and DistilBERTurk. The XGBoost model gave the best performance. The results of the XGBoost model are as follows: Non-low class precision 0.75, recall 0.70, f1 score 0.73, Moderate-high class precision 0.72, recall 0.77, f1 score 0.74. Average accuracy 0.7340 and ROC-AUC score 0.81.
Explaining artificial neural networks with decision tree ensembles
With the development of efficient algorithms, artificial intelligence (AI) applications have become ubiquitous in almost every aspect of our lives. They have even started to be used in critical areas such as defense industry, economy, and healthcare. However, the use of AI models in these important areas raises concerns about their reliability. Therefore, explaining how these black box models work has become an important goal. This thesis, we propose a simple and fast model to explain the decisions of any black box model. To achieve this, we attempt to explain the basic behavior of the model through a set of semi-random decision trees. Our approach only requires the data used to train the black box model and the model itself to work. Current state-of-the-art explainable AI (XAI) models typically produce local explanations for a black box model's decision regarding a single observation. On the other hand, models that produce global explanations use complex computations to understand the effect of each feature on the model's decisions. However, our proposed approach defines separate regions in the model's general decision space to explain the decision-making process of the model, and requires significantly less computational power than other advanced XAI techniques while producing both local and global explanations.
A novel communication protocol design for internet of flying things
In this thesis a new communication protocol, Message Queuing Telemetry Transport Extended (MQTT-XT) is proposed for the Internet of Flying Things (IoFT) network. The main objective of the proposed protocol is to enable UAVs to be considered as part of IoT nodes. The MQTT-XT protocol bridges the gap between UAVs and IoT devices by converting MQTT and MAVLINK message types to make them compatible with each other. Three different client types have been proposed to support long-distance and distributed communication with UAVs and IoT devices. A testbed was set up in simulation to evaluate the performance of MQTT-XT and MAVLINK protocols in terms of UAV management messages and 4 different IoFT scenarios. The experimental results show that MQTT-XT consumes 5 times less message size to transmit command messages and 20 times less for status messages. Additionally, the MQTT-XT protocol-based network completes missions 30% faster, and the packet loss ratio is reduced from 20% to 2%. In conclusion, MQTT-XT outperforms MAVLINK in IoFT autonomous applications.
Suicidal ideation detection from social media
Suicidal ideation is a global cause of life-threatening injury and, most of the time, death. Mental health issues have been rapidly increasing, and most are being avoided without adequate treatment. Due to the developments in social media platforms and the online anonymity that these platforms provide, people would like to interact more with others on social platforms. Social platforms are surveillance tools for mining social content and suicidal tendencies. The current thesis attempts to present a solution to detect depression/suicidal ideation by using state-of-the-art natural language processing (NLP) and deep learning (DL) approaches (BiLSTM, BERT Transformer). Three different novel approaches are proposed for three different datasets of textual content. The SuicideDetection dataset is a publicly available dataset which is a collection from the social platform of Reddit's subreddits ("SuicideWatch", "depression", "bipolar", "offmychest", "anxiety") in Kaggle and the SWMH dataset is a collection from only "SuicideWatch" subreddit. The CEASEv2.0 dataset is another used dataset which is a collection of 4932 suicide notes. The proposed models outperformed the latest models by 2% and 1% F1 scores on SuicideDetection and CEASEv2.0 datasets, respectively. The best models for each dataset have been analyzed and discussed in terms of performance, along with the characteristics of the datasets and limitations in the suicidal ideation classification. This performance can be measured by common metrics such as Accuracy, Precision, Recall, F1-Score, and ROC curve. As its application in the real world, this project can assist psychologists in the early identification of suicidal ideation before the suicidal person harms him/herself. The thesis also demonstrates the potential of employing DL algorithms such as transformers along with the latest word embedding techniques and NLP techniques that could improve the issue of suicidal ideation.
Graphical user interface customization for the elderly in ambient assisted living environments
In today?s computerized world, we usually face various types of computerized devices such as notebooks, personal digital assistants (PDAs) and even several machines at home. These devices have started to be operated by graphical user interfaces because of their ease of use compared to the text-based interfaces. Adaptation of young people or everyday-computer-users to these interfaces is quite natural; however, acceptability of graphical user interfaces (GUI) by elderly, as a rapidly growing group in today?s world, has become a challenging issue. Providing simplicity for user interfaces in order to be acceptable by elderly people is the main concern of the studies that try to address this challenge. Thus, designing simple and hence acceptable user interfaces for elderly requires considering their preferences and impairments such as visual, audial, psychomotor and cognitive impairments as the main design parameters.In this thesis, we study graphical user interfaces for elderly people with different impairments in the context of in-home healthcare system that involves daily living monitoring, focusing on the relation between two main design parameters of a graphical user interface: page complexity which is the number of interface elements on each page and the page hierarchy which is the number of the pages to be traced in order to finish a task. In order to study the dichotomy between these two phenomena, we designed two versions of the interface: One version is the flat version which has less page hierarchy with more complex pages and the other version is the deep version which has deeper page hierarchy with less complex pages.We conducted experiments with 18 elderly people having different impairments in order to understand whether the flat version or the deep version would be more acceptable / preferable. We asked them to complete some tasks such as making a video-call to a friend and setting a reminder to the agenda. Using the errors, time, inter-touch time between two consequent interactions, key strokes and clicks as the experiment metrics, the results revealed that the flat version with a reasonable complexity level would be more acceptable by elderly. We also report different findings about elderly perception of interacting by a user interface such as touching phenomena and perception of interface elements like buttons and interactive sentences.
Hyperparameter optimization of autoencoders for deeplearning of collective variables
Understanding the conformational space of proteins is integral to deciphering the mechanism of action of the proteins in drug-discovery studies. Molecular dynamics (MD) simulations have enabled the realistic simulation of protein dynamics. However, the computational demands of processing extensive MD data necessitate efficient dimensionality reduction techniques. An automated method for extraction of collective variables (reduced representations of protein movements) for the accurate representation of inherent dynamics of proteins is important. For this purpose, this thesis involves implementation of a semi-supervised autoencoder (SAE) for the extraction of collective variables from MD simulations of MurD and adenosine kinase (AdK) proteins, selected for their distinct dynamic states. This model, which incorporates extra output nodes coming from the latent layer and encoder hidden layers carrying the extra information such as opening angle, significantly outperforms the traditional vanilla autoencoder by achieving better results in reconstructing protein dynamics. The research also explores the optimization of the semi-supervised autoencoder architecture using genetic algorithm and enhances the quality of the extracted collective variables. By focusing on features such as Cα coordinates and dihedral angles, and adjusting the model's hyperparameters, the study advances our ability to capture the essential movements within proteins. Moreover, the semi-supervised approach shows an important improvement in handling the highly flexible regions of proteins, particularly the C-terminal domain of MurD, which is crucial for its functional transitions. The findings also shows the potential of machine learning in bioinformatics to refine simulation data analysis.
Dissimilarity based multiple instance learning using dictionary ensembles
Multiple Instance Learning (MIL) is one of important topics in the pattern recognition research field. It differs from many traditional machine learning problems in terms of real-world object representations. In MIL problems, samples are represented by multi-sets which are commonly named as bags where each bag include a set of feature vectors called instances. Many real world problems such as image, text or document classification, drug activity prediction and etc., can be formalized as MIL problems. Mostly, MIL problem, deals with supervised learning paradigm, which aims to learn models from the on hand samples and use it to solve regression or classification problems. Most of the cases, MIL problem is referred to binary classification where each bag has to be classified into one of the two categories; "positive" or "negative". Many MIL algorithms are developed considering the assumption that a bag is positive if at least it has one positive instance. Although this assumption has good results in some domains such as drug activity predictions, it may be restrictive for other domains of MIL problems such as computer vision MIL problems can be solved using instance based, bag based or embedded space algorithms. In this thesis, embedded-based strategy has been applied which converts the MIL problem to a standard supervised learning. In this approach, instead of using the mentioned assumption which relates the instance labels to the bags' labels, the dissimilarity of bags to the selected training instances which are called prototypes are taken into consideration. This mapping makes the bags be represented by standard fixed sized feature vectors. Most of the time, this feature mapping may include lots of redundant and irrelevant features. In this work we use Dictionary Learning as a classifier, which generates the sparse representation of each signal and classify them simultaneously. Using dictionaries would result in more efficient, fewer noises, simple and sparse manner, which make signals have more global look. As a third Approach, Ensemble Learning technique is implemented. This algorithm, apply two mentioned methods in its strategy and combine their strength where leads to higher and more reliable classification performances. Random Subspace and Bagging are two strategy implemented as Ensemble Learning approach. The proposed algorithm is evaluated on 11 different MIL datasets and compared with a recently proposed dissimilarity based ensemble MIL algorithm that uses Support Vector Machines (SVM). Experimental results show that the proposed algorithm outperforms the counterpart algorithm that uses SVM.
Çizge ve içerik verilerinde kolektif sınıflandırma algoritmalarının karşılaştırılması
Nesnelerin kategorilerine, bağlantılarına, içeriklerine göre sınıflandırılmasına finans, sosyal bilimler, iletişim, bilgisayarda görü ve biyoloji gibi birçok araştırma alanında ihtiyaç duyulur. Özellikle son yıllarda bu çalışma alanlarında kullanılan ağ verilerinin içerikleri ve birbirleriyle olan ilişkilerini çözmeye çalışmak günden güne önem kazanmaktadır. Sınıflandırma problemlerinde, nesnelerin sadece öznitelikleri kullanılarak yapılan sınıflandırma işlemleri, metin sınıflandırma gibi uygulama alanlarında İçerik-Tabanlı sınıflandırma olarak adlandırılmaktadır. Geleneksel sınıflandırma algoritmaları yalnızca bağımsız içerik özelliklerine bakarak sınıflandırma işlemini gerçekleştirirken Kolektif Sınıflandırma algoritmaları hem bağımsız nesneleri (içerik verileri) hem de bağımlı nesneleri (çizge verilerini) kullanarak sınıflandırma işlemini gerçekleştirir. Bu algoritmalar doküman, görüntü, ilişki tipi sınıflandırma ya da farklı özelliklerdeki düğümlerin tespiti gibi işlemlerde sınıflandırıcının performasını arttırmak için nesneler arasındaki çizge verilerini kullanırlar. Kolektif sınıflandırma üzerine yapılan çalışmalar, ağ verilerinin bilgilerini de sınıflandırıcılara dahil ettiği için yalnızca içerik verileri kullanan sınıflandırma algoritmalarına göre daha yüksek doğruluk sonuçları elde eder. Ağ verisinin yapısı çizge şeklindedir ve içeriğindeki her bir satır iki düğüm arasındaki ilişkileri temsil eder. Bu ilişkiler dökümanlar arası atıf, kişiler arası akrabalık, görüntüler arası benzerlik olabilir. Bu ilişki bilgilerinin sınıflandırıcılara dahil edilmesi nesneler arası benzerliklerin daha ön plana çıkmasını sağlar. Sınıflandırma işlemini çizge verileri arasında gerçekleştiren algoritmalardan biri Iterative Classification Algorithm (ICA)' dir. ICA algoritması sınıflandırmayı gerçekleştirirken etiketi belli olmayan düğümün komşularının etiket bilgilerini kullanır. Bu yüzden bu algoritma yerel (lokal) sınıflandırma algoritması olarak adlandırılır. ICA algoritması etiketsiz bir düğümün etiketini belirlerken incelediği komşu düğümlerin de etiketi belli değilse özyinelemeli bir şekilde etiketsiz komşulara geçici değerler atar. İçerik verisi ile ağ verisini birlikte kullanabilmek için yerel sınıflandırıcı olan Naïve Bayes ve K-En Yakın Komşuluk (KNN) algoritmaları ICA algoritması ile birleştirilerek Naïve Bayes-ICA, Naïve Bayes-KNN şeklinde kullanılır. Her iki algoritmada ele alınan etiketsiz düğümler için Naïve Bayes ve KNN içerik verisini sınıflandırırken, ICA algoritması ağ verisini sınıflandırır. Yerel sınıflandırıcılara alternatif olarak Loopy Belief Propagation (LBP) ve Mean Field Relaxation Labeling (MF) gibi global sınıflandırıcılar da bulunmaktadır. Bu algoritmaların global olarak adlandırılma sebebi, ağ verilerinde sınıflandırma yaparken global bir amaç fonksiyonu (global objective function) kullanması ve Markov rasgele alanlar ile ağın belli parçalarından sınıflandırılacak düğümleri elde etmesidir. Özetle bu çalışmada, yerel ve global sınıflandırıcılar içerik ve bağlantı tabanlı veri kümelerine uygulanmış, sınıflandırıcıların doğruluk değerleri karşılaştırılmış ve bu sınıflandırıcıların birleştirilmesiyle daha yüksek başarım elde edilebilmesi amaçlanmıştır.
Transfer learning for continuous control
Agents trained with deep reinforcement learning algorithms are capable of performing highly complex tasks including locomotion in continuous environments. In order to attain a human-level performance, the next step of research should be to investigate the ability to transfer the learning acquired in one task to unknown tasks. Concerns on generalization and overfitting in deep reinforcement learning are not usually addressed in current transfer learning research. This issue results in simplistic benchmarks and inaccurate algorithm comparisons due to rudimentary assessments. In this thesis, we propose novel regularization techniques exclusive to policy gradient algorithms for continuous control through the application of sample elimination and early stopping. By discarding samples that lead to overfitting via strict clipping we will generate robust policies for a humanoid with high generalization capacity. We also suggest the inclusion of training iteration to the hyperparameters in deep transfer learning problems. We recommend resorting to earlier snapshots of parameters depending on the target task due to the occurrence of overfitting to the source task. We demonstrate that a humanoid is capable of performing forward locomotion in unseen environments with different gravities and tangential frictions using strict clipping and early stopping. Furthermore, we evaluate our propositions on a delivery task where a humanoid is required to carry a heavy box while walking and inter-robot transfer tasks where the humanoid transfers its learning to taller and shorter robots. Because source task performance is not indicative of the generalization capacity of the algorithm we propose three different transfer learning evaluation methods. We increase the generalization capacity of a state-of-art adversarial algorithm by introducing entropy bonus, proposing different critic architectures and using simpler adversaries. Finally, we evaluate the robustness of these adversarial algorithms on morphologically modified hopper environments and environments with unknown gravities according to the criteria we proposed.
A gan-based hybrid data augmentation framework on chest x-ray images and reports
Classical data augmentation techniques are widely used by many image classification applications in the absence of adequate training data. These data augmentation techniques consists of but not limited with reflection, random cropping, re-scaling existing images and transformations. These techniques are widely used in practice during training classifiers with extended versions of real-world datasets. Increasing dataset size with realistic synthetic data allows us to improve the classification accuracy by making use of additional realistic variety. With the great representational power of GANs, learning the distribution of real data with a consistent level of variety allows us to generate samples with nearly-unobserved discriminative features. In our approach we used the aforementioned generative capability of GANs by utilizing state of the art GAN augmentation framework titled as StyleGAN2-ADA. After the training SytleGAN2-ADA in class conditional setting, we extended the dataset with different numbers of additional generated samples in order to observe the correlation of accuracy and augmentation strength. We extended our approach by using StyleCLIP to experiment disentangled feature augmentations which is a novel approach in the field of GAN augmentation. To make use of StyleCLIP more efficiently, we fine-tuned CLIP with x-ray images and modified entities which are extracted from corresponding medical reports. We used the DeepAUC framework which is proven to be efficient for multi-disease labelled x-ray classification tasks to test the performance of the GAN augmentation. In our approach, we observed that the classification accuracies were improved compared to without text-manipulated GAN augmented setting.
Efficient realtime image scaling and warping in hardware
Downscaling and warping are found in many image/video processing applications. This thesis offers an area-efficient downscaler hardware architecture and an implementation of a warping algorithm on hardware. The proposed downscaler is called "ODD (Output Domain Downscaler)". Both warping and output domain downscaler are demonstrated based on the implementation of bilinear interpolation method. Same interpolation method used in a different setting caused the difference between downscaler and warping implementations. Memory read and write methods of both warping and downscaler are also implemented with a single general FIFO. FIFO size calculation tool and a scheduler tool was used when implementing warping unit. Output domain downscaler is also combined with edge detection and sharpening spatial filter. This thesis compares output domain downscaler to a straight-forward implementation of the same combination of methods, which is called "Input Domain Downscaler". Input domain downscaler tries to output a new pixel of the downscaled video frame every time a new pixel of the original video frame is received. However, every once in a while, there is no downscaled pixel to produce. Input domain downscaler sometimes also skips a complete row of input pixels. Output domain downscaler, on the other hand, spreads out the job of producing downscaled pixels almost uniformly over a frame. As a result of that, output domain downscaler is able to do more resource sharing, i.e., can do the same job with fewer arithmetic units, thus offers a more area-efficient solution than input domain downscaler. In this thesis, output domain downscaler architecture is implemented with a downscale ratio between 1 and 2 with no loss of generality. That is because it is best to achieve larger downscale ratios of bilinear interpolation by applying a downscale ratio between 1 and 2 multiple times.
Akıllı ev ağlarına uzaktan erişim için güvenli SIP iletişimi sağlama
In recent years, smart home network applications are more and more popular with the help of networked appliances and wireless technology. The users can control appliances in their homes by remote access devices. However, there are certain security issues that can arise. Regarding the ultimate importance of people?s private home security, it is essential to provide a secure remote access mechanism.SIP (Session Initiation Protocol) is a common protocol which is convenient for remote connection to smart home network systems. In this work, two different SIP communication models for smart home networks are suggested to meet the security requirements needed for verifying user?s identity (authentication) and data privacy (confidentiality) of the connection. While the first method uses HTTP Digest Authentication for user registration security and IPSec tunnel for data confidentiality, the second method uses TLS for a secure authentication and SRTP in order to provide data confidentiality during the connection. SRTP key sharing is also provided by TLS before connection is established between a user and a home network.These methods are compared from security and performance aspects. They are evaluated against seven different well-known security threats. Furthermore, a pilot system is developed by using third party SIP stack implementation to make various data transfer tests to observe the performance of the two alternatives. Results of these evaluations are presented.
A new unit testing framework for enhanced software development methodologies
Unit testing is a method for validating that the source code of a program is working properly. In an ideal situation, each test case is independent from the others. Thus when applied to all parts of the code under development, it makes the whole code more reliable.Although unit testing is used widely, there is no specialized unit testing tool for web- based applications. We developed a new unit testing framework called YUnit which facilitates mobile & web developer's work, which can measure the performance of source code and allows us to compare with previous test results.The main goal of YUnit is demonstrating how, much of the work which people do manually during conventional testing, will be done automatically with the advanced unit testing tools, especially in web applications.A brief survey on unit testing and unit testing frameworks, design, implementation and evaluation of entire YUnit system are presented in this thesis. In addition, suggestions to enhance the system are presented.
Analysis and evaluation of big data clusters in different cloud computing environments
Cloud technology has become the supreme way to deliver enterprise applications and the preferred solution for companies launching innovations or extending their infrastructure. NoSQL databases are often used to provide availability and flexibility for big data processing. However, in distributed and parallel processing environments, there has not yet been a comprehensive evaluation of NoSQL databases. NoSQL was developed to better meet the data storage needs of a large number of records, but there are different kinds of NoSQL technologies, and most have not been thoroughly compared. Understanding the performance of NoSQL databases is essential for the selection of the correct database for a specific application. Several databases have emerged that can handle the cloud domain to take advantage of the flexibility of the cloud environment. The aim of this dissertation is to develop a model to evaluate and analyze NoSQL databases using a cloud environment where these systems are deployed (i.e., in a cloud or a local data center cluster). This model is then evaluated with the most popular NoSQL databases in cloud computing (MongoDB and Riak KV) and internal cluster environments to analyze the performance of those two systems. A number of aspects of performance, including monitoring throughput and latency, are compared. The model architecture has the ability to use other databases as well as other clouds. An evaluation of the model system in three cloud computing infrastructures is also presented. The model architecture can use other databases as well as other clouds. Furthermore, a decision tree is constructed based on many experiments to determine the performance criteria for the databases on the selected cloud computing platforms. The main findings of this work are as follows: (i) Performance gain is achieved by integrating MongoDB with Google Cloud, especially in terms of reading and writing a massive amount of data in a cluster environment; (ii) The infrastructure of Google Cloud is more manageable and efficient in dealing with the distributed environment and big data, and it achieved better results than the other clouds; (iii) The results also showed the limitations of Riak KV in both the Google and OpenStack clouds, especially in terms of update latency. Keywords: NoSQL; cloud computing; MongoDB; Riak KV; big data; cluster; YCSB; Basho-bench
Ses teli damar ve şekil özelliklerini değerlendirerek ses telipatolojilerinin sınıflandırılması
Ses telleri üzerinde oluşan nodül, polip, larenjit gibi hastalıklar nedeniyle, ses tellerinin yapısı bozulmakta bu da ses kısıklığı, nefes alma ve yutma güçlüğü gibi problemlere sebebiyet vermektedir. Ses teli patolojilerinin teşhisi video laringostroboskop, endoskopik yüksek hızlı kamera gibi medikal görüntüleme cihazları ile alınan görüntülerde ses tellerinin yapısal bozulmalarının ve ses teli titreşimlerinin incelenmesi ile mümkün olmaktadır. Ancak bazı patolojilerin ses tellerinde meydana getirdikleri şekilsel bozulmalar birbirleri ile benzerlik gösterebildikleri için ses teli hastalıklarının teşhisi hataya açıktır. Bu gibi durumlarda net tanı ancak ameliyat esnasında patoloji açıldığında konulabilmektedir. Son yıllarda yapılan tıbbi çalışmalar ses telleri üzerindeki patolojilerin kan damar yapılarını da değiştirdiğini ortaya koymuştur. Araştırmalarda, kan damarlarının patoloji cinsine göre belli bir örüntü sergilediği gözlenmiş ve kan damarlarının yardımcı bir teşhis metodu olarak kullanılması gündeme gelmiştir. Ancak literatürde görüntü işleme ve şekil tanıma yöntemleri ile ses telleri üzerindeki kan damar yapılarını değerlendiren bir çalışma mevcut değildir. Bu tez çalışmasında, direkt mikro laringoskop ve video laringostroboskop cihazları kullanılarak alınan renkli ses teli görüntülerinde, ses teli üzerindeki kan damarı bozulmalarını patolojiler ile ilişkilendirerek ses teli şekil bozulmaları ile birlikte değerlendiren ve böylece teşhis koymanın zor olduğu vakalarda net tanı için hekime yardımcı olabilecek özgün bir sistem tasarlanmış ve gerçeklenmiştir. Çalışmanın ilk adımında, Yönlü Gradyan Histogramı (YGH) kullanılarak ses teli görüntüleri üzerinde ses tellerinin lokasyonu tespit edilmekte ve tohumlu bölge büyütme algoritması ile ses telleri arasında kalan glottal açıklık bölgesi bölütlenmektedir. Ses teli kenar özelliklerini çıkarmak için ses teli kenar hattı analiz edilerek patolojilerin lokasyonu, şekli ve büyüklüğü ölçülmektedir. Ses teli damar özelliklerini çıkarabilmek için ise ilk önce tohumlu bölge büyütme algoritması ile damar yapıları bulunmakta ardından yanlış damarların temizlenmesi ve damar kopukluklarının giderilmesi işlemleri ile damarların orta eksenleri çıkarılmaktadır. Son adımda ses teli kenar ve damar özellikleri kullanılarak ses teli hastalıkları sınıflandırılmaktadır. Çalışma kapsamında kist, nodül sınıflandırması ve sağlıklı, nodül, polip, sulcus vocalis, larenjit sınıflandırması olmak üzere iki farklı sınıflandırma problemi ele alınmıştır. Kist,nodül sınıflandırması için ses teli damar özelliklerinden faydalanılmıştır. Destek Vektör Makineleri (DVM), K-En Yakın Komşuluk (KEK) ve Rastgele Orman (RO) algoritmaları ile sınıflandırma yapılarak başarıları karşılaştırılmıştır. DVM algoritması ile %75 değerinde bir ağırlıklı ortalama kabul oranı elde edilmiştir. Sağlıklı, nodül, polip, sulcus vocalis ve larenjit sınıflandırması için ise ses teli kenar ve damar özelliklerini beraber değerlendiren bir ikili karar ağacı yapısı tasarlanmıştır. Tasarlanan ikili karar ağacı yapısının ağırlıklı ortalama kabul oranı %79 olarak hesaplanmıştır.
A new fusion reranking pipeline for Turkish datasets using fine-tuned RAG components
This study addresses the gap in the multilingual capabilities of Retrieval Augmented Generation (RAG) systems for the Turkish language, particularly in the medical domain. With the rise of Large Language Models (LLMs) and their widespread applications, the reliance on external knowledge through retrieval components has become crucial to mitigate hallucinations and improve response accuracy. However, most existing retrieval components, including embeddings and rerankers, are predominantly trained on English datasets, highlighting a significant limitation in multilingual and domain-specific capabilities. To address this, the study introduced Pubmed-RAG-TR, a Turkish-language medical dataset, and fine-tuned retrieval components on both Pubmed-RAG-TR and WikiRAG-TR, a Turkish RAG dataset. A novel RRF-based reranker pipeline was also developed to improve the context construction for LLMs. Experimental results demonstrated that fine-tuning retrieval components on domain-specific datasets significantly enhanced the retrieval and post-retrieval quality, improving the accuracy of LLM responses. The study concludes that incorporating domain-specific semantics into retrieval and reranking models can substantially boost the performance of RAG systems in multilingual contexts.
Konuşma dili işleme
İster Türkçe, ister Tagalog dilinde olsun, herhangi bir konuşma dilini kullanarak iletişim kurma kabiliyeti sadece insan ırkına mahsus bir üstünlüktür. Bilgisayarlar, bu dil kullanma yeteneğini insanlarla paylaşana kadar insanların günlük hayatlarında yapabildikleri pek çok işi yerine getiremeyeceklerdir. Sadece 3 yaşında olan bir çocuk, geçerli bir satranç oyunu oynayamamasına, en azından bir ustayı yenememesine rağmen kendi ana dilini rahatça konuşur ve anlar. Buna karşılık dünyada hala insanların bu dil yeteneğinin üstesinden gelebilmiş bir bilgisayar programı geliştirilebilmiş değildir. Konuşma dilinin yapısını anlayabilmek çok zor bir işlemdir. Bunun için kullanılan dilin gramer bilgisine ve tartışılmakta olan konuya bağlı olarak genel kültüre ihtiyaç vardır. Ancak, her şeyden önce bilgisayarın, işleyeceği konuşma dili hakkında yeterli bir alt yapıya sahip olması gerekir. Bunun için de o dilin kelimeleri, kelimelerin fonksiyonları, cümle kurarken kullanılan yapılar ve kurulmuş cümlelerin çeşitleri hakkında ayrıntılı bilgisi olmalıdır. Bu bilgilerin veri tabanına yerleştirilmesinden sonra dışarıdan girilen bilginin analiz edilmesi, cümlenin manalandırılması ve metin içinde ne anlama geldiğinin değerlendirilmesi aşamaları gelir. Bu aşamaların ortak adı "konuşma dili işleme"dir. Bu tezde, zekânın ve yapay zekâ'nın tanımları ayrıntılı olarak incelendikten sonra, bu kavramları kullanarak anlama için gerekli olan dilbilimsel kültürün düzgün bir konuşma dili için nasıl kullanıldığı incelenmiştir. Daha sonra PROLOG dili kullanılarak, bir Türkçe cümle parser'ı yazılmış ve Yunus Emre'nin 180 beyiti üzerinde kullanılarak, şairin kullandığı kelimelerin köken ve görev dağılımları çıkarılmıştır.
World-Wide-Web üzerinde çokluortam veritabanı uygulaması
ÖZET Son yıllarda entegre devre ve iletişim teknolojilerindeki ilerileme ile bilgi işlemede daha önce görülmemiş işlem ve iletişim hızlarına erişilebilmiştir. Bunun doğal bir sonucu olarak da önceden işlemesi çok pahalı olan çokluortam veri de bilgi işlemde yaygın olarak kullanılmaya başlandı. Ses, video ve resim gibi çokluortam verisi daha çok gerçek hayatın aynen bilgi işleme hazır hale getirilmesi ile oluşturulur ve sekizli başına en düşük anlamı ifade eden veri tipidir. Sonraki kuşak bilgisayar sistemleri ise daha da büyük bilgiyi yine daha hassas ve yaygın işleme yeteneğine sahip olacaklardır. Bu gelişmeler doğrultusunda, çokluortam verisini daha etkin işlemek için yeni bilgisayar algoritmaları ve işlem teknikleri geliştirilmeye başlanmıştır. Yeni teknikler çokluortam verisinin hatalara karşı neredeyse duyarsız olmasından faydalanıp işlem ve/veya iletişim hızını arttırmaya yönelik olarak geliştirilmişlerdir. Öte yandan Internet dünya çapında ve her geçen gün yayılarak, özellikle uzak mesafeler arasında haberleşmenin en önemli araçlarından biri haline gelmiştir. Internet' in böylesi genişlemesindeki en önemli etmen ise basitleştirilmiş bazı iletişim kurallarına uyabilen her bilgisayara kullanım imkanı vermesidir. Bu tezde yukarıda sayılar gelişmelerin genel yapısı üzerinde durulup, çokluortam verisinin yönetimi, düzenlenmesi ve WWW üzerinde yayınlanmasını sağlayacak bir "Çokluortam Haberler Veritabanı" geliştirilmiştir. XII
Hiyerarşik grupsal kurumlarda kullanılacak bir şifre sistemi
ÖZET Kurumlar, etkinlik alanlarına bağlı olarak askeri, devlet güvenliği ya da ticari kaygılardan dolayı bünyelerinde mevcut bilgilerin güvenliğini sağlamak isterler. Alınabilecek fiziksel güvenlik yöntemleri yetersiz kaldığında ise bilgi güvenliğini sağlamanın tek yolu, şifreleme yöntemlerinin kullanılmasıdır. Kurumlar, işlevsel farklılıklar içeren gruplardan oluşurlar. Bilgiler konularına göre farklı grupları ilgilendirebilir. Gruplar içinde de personelin görev farklılığım temel alan bir hiyerarşik seviyelendirme mevcuttur. Bir bilgi bir grubun yalnızca en üst seviyedeki yöneticisini ilgilendirirken, başka bir bilgi daha alt seviye personelini de ilgilendirebilir. Bu çalışma, hiyerarşik grupsal kurumlarda veri aktarma iletişim güvenliğini sağlamak için kullanılabilecek bir modern (açık anahtarlı) şifre sistemi tanımım içermektedir. Anahtarların üretim, korunma ve dağıtımı ile mesajların dağıtımından Mesaj Dağıtıcısı (MD) sorumludur. MD, sistemde mevcut her kişiye ait olduğu gruba ilişkin bir, ve hiyerarşik seviyeye ilişkin olarak bir olmak üzere iki anahtar dağıtacaktır. Mesajlar, sistemde dağıtılmadan önce hedef grup ile hiyerarşik anahtarların kullanımı ile iki kez şifrelenecektir. İlgili grup ile seviyede bulunan kişiler, ellerindeki anahtarları kullanarak şifrelenmiş mesajları çözebilirler; böylece mesaj larm ilgili seviye ve gruptaki personel tarafından okuyabilmesi garantilenmiş olur. Anahtar mevcut olmadığı durumda ise şifrelenmiş mesajlardan asıl mesajları üretmek mümkün değildir. Çalışma: kullandığı algoritma ile, tanımlanan grupsal hiyerarşik modelde şifrelemeyi sağlaması, alt seviye anahtarlarının basit bir işlem ile üretilebilmesi ve yalnızca iki anahtarın muhafazasına ihtiyaç duyurması nedenleri ile özgün bir algoritmayı içermekte olup, bu algoritmada RSA ile El Gamal şifre sistemleri tarafından kullanılmış ve güvenli oldukları ispatlanmış olan tek-yönlü NP-Complete fonksiyonlar temel olarak alınmıştır.
Real-time intelligent strawberry harvesting and quality determination system using computer vision and deep learning
Strawberries have a comparatively extended harvesting period, which poses the need for intelligent harvesting systems that identify various stages of strawberry ripeness to alleviate fatigue and reduce the costs associated with this task. These systems offer potential solutions to enhance productivity while minimizing labor requirements. Research in real-time strawberry detection still has several gaps to address. These gaps include managing imbalance label distribution, exploring data augmentation techniques, optimizing preprocessing and training parameters, and investigating advanced topics such as fine-tuning. This research focuses on developing an accurate and efficient real-time strawberry detection model. The main objective is to locate and identify strawberries in agricultural orchards and assess their quality by detecting overripe and decaying fruits. For this purpose, a diverse High-Quality Annotated Strawberry Dataset(HQASD) was collected with 5000 RGB images belonging to ten strawberry maturity levels. Different types of data augmentation techniques were applied, and the most advanced and novel approach used in this context was the Cycle-Consistent Generative Adversarial Network (CycleGAN) for generating images of an underrepresented class from an overrepresented one. The latest real-time object detection model, You Look Only Once (YOLOv7), is trained on real and synthetically generated data. Deep experiments and various scenarios were conducted to investigate all the aspects that impact the model's overall performance. Hyper-tuning is performed on the model's parameters to optimize its performance. The best-obtained models for identifying ten classes achieved Mean Average Precision (mAP) 98.7% at Intersection over Union (IoU) threshold 0.5, and the best model for identifying three classes scored 96.51% mAP for all types at the same threshold, where the model trained on synthetic images performed 98.4% mAP for all classes which shade the effectiveness of employing CycleGAN along with fine-tuning techniques. In addition, the research succeeded in creating HQASD, which provides a comprehensive representation of the strawberry maturity spectrum, which makes it well-suited for computer vision tasks.
The detection and recognition of faces in the internet of things for security applications
In recent years, the security constitutes are the most important section of human life. Security of the house and the family is important for everybody. Automation of a home is an exciting field for security applications. This area has developed with new technologies such as the Internet of things (IoT). In IoT, each device behaves as a small part of an internet node and each node communicates and interacts. Currently, security cameras are used in order to construct safety in areas, cities and homes. The camera records the events, and when a problem occurs, it will detect by monitoring the old recording. At this time, the cost is the greatest factor. This system is very helpful to reduce the cost of monitoring the movement from outside. The purpose of this thesis is to describe a security alarm application by utilizing low preparing power chips and Internet. Also, a new online method is proposed to detect and recognize faces on Raspberry Pi in the IoT. Raspberry Pi operates and controls movement detectors. It will monitor and record the motions for future playback. This thesis proposes an analysis of images via computer vision to detect and recognize faces in the analyzed images. If these frames contain a face, the system will detect and recognize the face. This system is appropriate for small personal range surveillance, as, in personal office security, home, parking entrance and bank locker room. The face detection and recognition in the IoT is a very important problem for a security and surveillance system. Also, face detection and recognition is presently a very active research area. The proposed system is very helpful to reduce the cost of monitoring the movement from outside. On the other hand, in this research, an IoT-based system is combined with computer vision in order to detect human's body. A Raspberry Pi 3 cards with the size of a credit card is used for this purpose. A motion is detected by the PIR sensor mounted on the Raspberry Pi. PIR sensor helps to monitor and get alerts when a movement is detected. Afterward, the human's body detects in the captured image and is sent to a smartphone by using telegram application.
ComScribe: A communication monitoring tool for multi-GPU platforms
GPU communication plays a critical role in performance and scalability of multi-GPU accelerated applications. With the ever increasing methods and types of communication, it is often hard for the programmer to know the exact amount and type of communication taking place in an application. Though there are prior works that detect communication in distributed systems for MPI and multi-threaded applications on shared memory systems, to our knowledge, none of these works identify intra-node GPU communication. In this work we present ComScribe, a tool that identifi es and categorizes types of communication among all GPU-GPU and CPU-GPU pairs in a node. Our tool is built on top of NVIDIA's pro lfier nvprof for capturing intra-node point-to-point communication resulting from explicit communication primitives, Uni ed Memory operations, and Zero-copy Memory transfers. For monitoring collective GPU-GPU communication in a node, ComScribe intercepts NCCL's collective primitives at runtime and records data transfers among GPUs. It visualizes data movement as a communication matrices for both number of bytes transferred and the number of transfers. To validate our tool on 16 GPUs, we present communication patterns of 13 micro and 3 macro-benchmarks from NVIDIA, CommjScope, and MGBench benchmark suites. To demonstrate tool's capabilities in real-life applications, we also present insightful communication matrices of three deep neural network models. All in all, ComScribe can guide the programmer in identifying groups of communicating GPUs, the volume of communication, and types of primitives used. This o ers avenues to detect performance bottlenecks and more importantly communication bugs in an application.
Increasing efficiency of combinatorial optimization problems on quantum annealers using classical computers
Google's claim of quantum supremacy is a big milestone in the history of quantum computing. Despite the claim, the practical applicability of quantum computers remains questionable due to the low number of quantum bits and high noise rates. An alternative model of quantum computing is quantum annealing, which is capable of solving only an optimization problem in a specific format. Quantum annealing is being actively researched due to the fact that the scale of such devices has increased up to thousands of qubits. This relatively high number of qubits enables quantum annealers to solve problems of larger sizes, hence makes them usable in real-life scenarios. The specific format of the solved optimization problem is called an Ising formulation, which can also be represented as quadratic unconstrained binary optimization (QUBO). The QUBO consists of a set of qubits with corresponding bias weights and the quadratic weights between the qubits. This thesis presents two schemes for weight optimization in the QUBO formulation of two different combinatorial optimization problems. Both schemes involve a classical computer redefining the QUBO weights that is interfaced with an annealing device, which solves the QUBOs. The first combinatorial problem is the task assignment problem, in which the biases represent the computational costs of tasks and quadratic terms model communication between tasks. The second problem is the circuit mapping problem, where the biases represent the fidelity of quantum gates, while the quadratic terms model qubit movement. The first approach named weight optimization algorithm (WOA) searches for a desirable ratio between the qubit biases responsible for fidelity of mapping quantum gates to physical qubit topology and the quadratic terms responsible for qubit movement. The desirability of the ratio is defined by the total fidelity resulting from both qubit movement and mapping. The second presented model uses ant colony optimization (ACO) to update the quadratic terms of the QUBO that solves the task assignment problem. However, this model can be generalized to any combinatorial optimization problem solvable by the ACO. Efficient updates of the weights based on the answers from previous QUBOs are expected to guide the reformulated QUBOs towards the optimum of the objective function. At the same time, this algorithm would allow utilizing the stochasticity of quantum annealers for better exploration of the solution space and their speed for faster generation of candidate solutions. The introduction of the WOA into the quantum annealing workflow for quantum circuit mapping resulted in reduced qubit movement in 72.9% of all problem samples. Moreover, it allowed to increase the total fidelity of the mapped circuit by 39% on the IBM Vigo device and 107% on IBM QX2. The experiments have been performed on the tabu search QUBO solver from the D-Wave quantum annealing software stack. The results for the ant colony weight optimizer are limited due to the unavailability of a quantum annealing device.
Platform and data-aware execution of sparse triangular solve on CPU-GPU heterogeneous systems
Sparse triangular solve (SpTRSV) is an important computational kernel used in many scientific and numerical linear algebra applications such as direct methods, iterative solvers and least square problems. In comparison with other sparse kernels such as sparse matrix-vector multiplication (SpMV), SpTRSV is an inherently sequential operation owing to the presence of dependencies among computations of different unknowns. Often, it has also been observed to be one of the most time consuming operations in an application. The performance of a given algorithm highly depends upon the sparsity characteristics of the input matrix and the underlying hardware. Several SpTRSV algorithms and their implementations are available for the CPUs and the GPUs. Unfortunately, there is no single algorithm or hardware platform that has been shown to achieve the best performance for all input matrices. In this dissertation, we propose tools and techniques aimed at extracting higher SpTRSV performance for a given input matrix on modern CPU-GPU heterogeneous systems. Towards this end, we adopt a two-pronged approach: Depending upon the sparsity characteristics of the input matrix, we propose to (i) automatically select the best CPU or GPU SpTRSV algorithm for the matrix, (ii) split a single SpTRSV execution into parallel and sequential parts such that the parallel part is executed with a parallel-friendly algorithm and the sequential part is executed with a sequential-friendly algorithm. The aim is to achieve a higher SpTRSV performance than using a single algorithm. The two algorithms can also potentially execute on two different platforms (CPU and GPU). For the SpTRSV algorithm selection, we propose a supervised machine learning-based prediction framework to predict the SpTRSV implementation giving the fastest execution time for a given sparse matrix based on its structural features. The framework works by extracting matrix features, collecting algorithm performance data, and training a prediction model with around 1000 real square matrices from the SuiteSparse Matrix Collection. Once trained on a given machine, the model can predict the fastest SpTRSV implementation for a given matrix by paying a one-time matrix feature extraction cost. The framework is also capable of taking into account CPU-GPU communication overheads that might be incurred in scientific applications such as iterative solvers. We test the framework on a modern CPU-GPU machine. Experimental results on a modern CPU-GPU platform (Intel Xeon Gold + NVIDIA Tesla V100 GPU) show that the fastest algorithm is selected with a reasonable accuracy (87%) as well as the predicted SpTRSV implementation achieves significant speedups compared (1.4-2.7× harmonic mean) with a lazy choice of a single algorithm. Secondly, for dividing the SpTRSV execution between the CPU and GPU, we propose an SpTRSV split execution model. The model has been designed based on the empirical evidence from the existing research that; (i) a highly parallel algorithms perform well for SpTRSV requiring few sequential steps, with high number of unknowns computed per step, (ii) a sequential algorithms perform better for SpTRSV requiring more steps with few unknowns computed per step. For matrices having a mix of highly parallel and sequential steps, a parallel algorithm is expected to provide benefits for steps with large number of unknowns, its performance is expected to deteriorate for steps with few unknowns. The converse can be said about the performance of sequential algorithms for such matrices. With the aim of performing split-execution for such matrices using a highly parallel and a sequential algorithm, our split execution model can automatically determine the suitability of an SpTRSV for split-execution, find the appropriate split point, and execute SpTRSV in a split fashion using two SpTRSV algorithms while automatically managing any required inter-platform communication. The model is implemented as a C++/CUDA library supporting multiple CPU-GPU algorithms. Experimental evaluation of the model on two CPU-GPU with a matrix dataset of 327 matrices from the SuiteSparse Matrix Collection shows that our approach correctly selects the fastest SpTRSV method (split or unsplit) for 88% of the matrices on Intel Xeon Gold + NVIDIA Tesla V100 and 83% of the matrices on Intel Core I7 + NVIDIA G1080 Ti platform achieving speedups up to 10× and 6.36×, respectively. We expect that the tools and methods proposed in this dissertation will benefit both academia and industry while at the same time will pave the way for further research in the area of efficient exploitation of CPU-GPU systems for sparse linear algebra computations.
Cognitively-inspired deep learning approaches for grounded language learning
Designing machines that can perceive the surrounding world and interacting with us using human language is one of the long-standing goals of artificial intelligence. Although tremendous progress has been made to model the linguistic meanings computationally, how to best integrate linguistic and perceptual processing in multi-modal tasks is a significant open problem. This thesis explores several cognitively-inspired neural architectures that consider the different aspects of the language's role in cognition, visual perception, and task execution. Proposed models incorporate design choices motivated by cognitive science studies and are based on the common patterns in vision-language tasks. We begin by presenting an encoder-decoder network with a novel channel-based perceptual attention mechanism and its application to the navigational instruction following task. The perceptual processing component of this architecture is designed to focus on individual objects and properties within the environment using the language priors while preserving the spatial relations. To benefit from the designed component, we also propose an improved agent-centric world representation to allow the model to reason over the perception spatially. Next, we explore the usage of the Neural Module Networks approach in a real robotic system for the first time. Since collecting large-scale real world data is a labor-intensive and expensive work, the system learns the language grounding on simulated data and the perceptual representation separately to overcome the scarce data problem. However, because of the separate learning processes, inconsistencies arise between the user's and robot's world models. To overcome this, we propose a Bayesian learning approach that uses the implicit information in the instruction to update the perceptual belief to align what the user sees and what the robot perceives. In both parts, we demonstrate systems that use the high-level effect of language on visual processing, which operates on high-level representations. In addition to this, in the last part, we investigate the effect of language on low-level visual processing. To this end, we condition one or both low-level and high-level visual processing branches of a backbone architecture on language using language filters and apply these models to the image segmentation from referring expression task. Experiments show that modulating both low-level and high-level visual processing with language significantly improves the language grounding performance.
Self-supervised representation learning from demonstration
Given the growing demand for the application of robotics in an increasingly wide range of tasks and environments, robot learning as opposed to programming is getting more relevant by the day, since programming robots is tedious and usually only feasible in controlled domains. However, gathering data with robots is an expensive and time-consuming task, so the learning methods must cope with the limited amount of data available. When working with high dimensional perceptual data, learning low-dimensional, useful representations is a key aspect in dealing with the low-data setting. In this thesis, we develop a neural network architecture to learn perceptual representations from few human skill demonstrations in a self-supervised manner. The developed models take the sequentiality of the data and the low-data nature of the problem into account. These representations are used to learn perceptual goal models of the demonstrated skills. These models can monitor the learned skill executions and be used in reinforcement learning to generate reward signals, without explicit reward engineering. Our simulated and real robot evaluations with object manipulation skills show that the learned representations result in better goal models in terms of monitoring and reinforcement learning performance compared to generic dimensionality reduction methods. We further introduce transfer learning approaches in the context of learning from demonstration and show positive transfer between different objects for the same skill, between the same object for different skills under certain conditions, and also between different perceptual domains. Transferring knowledge as enabled by our modular neural architecture allows us to leverage existing data for continual learning into the future. Overall, we show that our proposed self-supervised representation learning architecture has the potential to improve learning from demonstration approaches with a perceptual component.
Stacked frequency-timeGRUs for continuous arousal recognition from musical audio
We address the problem of continuous arousal detection for emotion recognition in musical audio pieces where emotions are represented in the two-dimensional arousal-valence space. We propose a novel method which is a combination of two recurrent neural networks using mel-spectrogram features: A bidirectional GRU network along the frequency dimension as a feature extractor, stacked with a GRU network along the temporal dimension, which is unidirectional for real-time adaptability. The method is evaluated on the MediaEval2015- Emotion in Music Dataset, achieving an RMSE of 0.215 which is better than the results reported by real-time adaptable state-of-the-art models.
Precise event sampling: In-depth analysis and sampling-based profiling tools for data locality
Precise event sampling is a profiling feature in current commodity CPUs that allows sampling of hardware events and identifies the instructions that trigger the sampled events. It offers the ability to detect performance bottlenecks with low overhead as well as the locations of the bottlenecks in source code. There have been a number of profiling tools developed using this feature that detect various sources of performance bottlenecks. However, none of these tools detects inter-thread data movement nor measures data locality in multithreaded applications, which have become widely used due to the ubiquity of multicore architectures. Furthermore, though this hardware facility has been used in multiple profiling tools, there have been only few works that analyze it in terms of accuracy and overhead. All of these works target only the facility in Intel architecture, and none of these works evaluates other aspects of precise event sampling such as memory overhead, stability, and functionality of the facility. In this dissertation, we present threefold major contributions. First, we perform the most comprehensive and in-depth qualitative and quantitative analyses to date on PEBS and IBS, which are the precise event sampling facilities of two major vendors, Intel and AMD, respectively. Next, we show the potential for imaginative use of precise event sampling in developing low overhead yet accurate profiling tools for multicore and design two diagnostic tools with a particular focus on data movement as it constitutes the main source of inefficiencies. First of such tools is ComDetective that detects inter-thread communications, classifies them into true sharing or false sharing, and records them in the form of communication matrices. Second is ReuseTracker that measures data locality in private and shared caches of multithreaded applications. ComDetective and ReuseTracker leverage precise event sampling to profile multithreaded applications accurately and with low overheads compared to their state-of-the-art alternatives. To analyze key differences between Intel PEBS and AMD IBS, we firstly developed a series of carefully designed microbenchmarks. Through our qualitative analysis and quantitative study using the microbenchmarks, we found that Intel PEBS samples hardware events more accurately and with higher stability in terms of the number of samples that it captures, while AMD IBS records richer set of information at each sample. We also discovered that both PEBS and IBS are afflicted with bias when sampling the same event across multiple different instructions in a code. Moreover, we also show how our findings from the quantitative experiments using the microbenchmarks are relevant for a full-fledged profiling tool that runs on Intel and AMD machines. We develop ComDetective, a profiling tool that captures inter-thread communications accurately and with low runtime and memory overheads. ComDetective employs precise event sampling to sample memory accesses and utilizes hardware debug registers to detect inter-thread communications. In addition to detecting communications, ComDetective can also classify them into true or false sharing. Its time and memory overheads are only 1.30× and 1.27×, respectively, for the 18 applications studied under 500K sampling interval. Using ComDetective, we generate insightful communication matrices from several microbenchmarks, PARSEC benchmark suite, and some CORAL applications and compare the produced matrices against the matrices of their MPI counterparts. Using ComDetective, we identify communication bottlenecks in a few codes and achieve up to 13% speedup from code refactoring those codes. We also design ReuseTracker, which is a profiling technique that measures reuse distance - a widely used metric that measures data locality. Reuse distance is a measurement of data locality as it is the number of unique memory locations that are accessed between two consecutive accesses to a particular memory location (use and reuse). ReuseTracker leverages precise event sampling to capture uses and debug registers to detect reuse in measuring reuse distance. ReuseTracker can measure reuse distance in multithreaded applications by also considering cache-coherence effects with much lower overheads than existing tools. It introduces only 2.9x time and 2.8x memory overheads. It achieves 92% accuracy when verified against a carefully crafted configurable microbenchmark that can generate user-specified reuse distance patterns. We demonstrate in two use cases how ReuseTracker can be used to guide code refactoring by detecting spatial reuses in shared caches that are also false sharing and how it can also be used to predict whether certain applications can benefit from adjacent cache line prefetch optimization. We expect that the analysis, algorithms, and the tools presented in this dissertation will benefit hardware architects in designing new precise event sampling features and performance engineers in performance tuning of their software while also paving the way for a new generation of low-overhead profiling tools. Moreover, the outcomes of the dissertation can be used by the end-users (e.g., data analysts, engineers, compiler developer) to identify the performance issues and increase the data locality aspects of their software.
Utilizing coarse-grained data in low-data settings for event extraction
Annotating text data for event information extraction systems is hard, expensive, and error-prone. We investigate the feasibility of integrating coarse-grained data (document or sentence labels), which is far more feasible to obtain, instead of annotating more documents. We utilize a multi-task model with two auxiliary tasks, document and sentence binary classification, in addition to the main task of token classification. We perform a series of experiments with varying data regimes for the aforementioned integration. Results show that while introducing extra coarse-grained data offers greater improvement and robustness, a gain is still possible with only the addition of negative documents that have no information on any event.
Exploring mixed and multi-precision SpMV for GPUs
Sparse Matrix-Vector Multiplication (SpMV) is one of the key memory-bound kernels commonly used in industrial and scientific applications. To improve its data movement and benefit from higher compute rates, there are several efforts to utilize mixed precision for SpMV. Most of the prior-art focus on performing the SpMV in different precisions throughout the entire application, such as an iterative solver (e.g., CG, GMRES) where certain steps can be done with lower precisions. More recently, methods of using mixed-precision within a single SpMV has been of consideration. For instance, one can decide precision for each non-zero value in the matrix, and then split the input into multiple matrices with their respective precisions; or, decide the precision for each block in a given block-diagonal matrix format. In this work, we are interested in this more fine-grained approach of mixedprecision SpMV. To this extent, we extend an existing entry-wise precision based approach by deciding precision for each row, motivated by the granularity of parallelism on a GPU where groups of threads process rows in row-compressed sparse matrices. We propose mixed-precision CSR storage methods with row permutations and describe its greater load-balance compared to the existing method. We also consider a multi-precision case where single and double-precision copies of the matrix are stored priorly, and further extend our mixed-precision SpMV approach to comply with it. To evaluate our methods, we apply them in two real-life applications: a multiprecision Jacobi method and a multi-precision Cardiac modeling application. We further extend our mixed-precision methodology to be used with the ELLPACK-R format. We demonstrate the effectiveness of the proposed SpMV methods on an extensive dataset of real-valued large sparse matrices from the SuiteSparse Matrix Collection using an NVIDIA V100 GPU.
GAN inversion based image manipulation with text-guided encoders
Style-based Generative adversarial networks (StyleGAN) enable very high quality image synthesis while learning disentangled latent spaces. Hence, there is a lot of recent work focusing on semantic image editing by latent space manipulation. A particularly emerging field is editing images based on target textual descriptions. Existing approaches tackle this problem either by performing instance-level latent code optimization which is not very efficient or by mapping predefined text prompts to editing directions in the latent space. In contrast, in this thesis work, we present two novel approaches that enable image editing guided by textual descriptions. Our idea is to use either a text-conditioned encoder network or a text-conditioned adapter network that predicts a residual latent code in a feed forward manner. Both quantitative and qualitative results demonstrate that our methods outperform competing approaches in terms of manipulation accuracy, i.e., how well the synthesized images match the textual descriptions while ensuring highly realistic results and preserving features of the original image. We also demonstrate that our method can generalize to various domains including human faces, cats, and birds.
Domain-adaptive self-supervised pre-training for face & body detection in drawings
Drawings are powerful means of pictorial abstraction and communication. Understanding diverse forms of drawings, including digital arts, cartoons, and comics, has been a major problem of interest for the computer vision and computer graphics communities. Although there are large amounts of digitized drawings from comic books and cartoons, they contain vast stylistic variations, which necessitate expensive manual labeling for training domain-specific recognizers. In this work, I show how self-supervised learning, based on a teacher-student network with a modified student network update design, can be used to build face and body detectors. My setup allows exploiting large amounts of unlabeled data from the target domain when labels are provided for only a small subset of it. I further demonstrate that style transfer can be incorporated into my learning pipeline to bootstrap detectors using a vast amount of out-of-domain labeled images from natural images (i.e., images from the real world). My combined architecture yields detectors with state-of-the-art (SOTA) and near-SOTA performance using minimal annotation effort. Through the utilization of this detector architecture, I accomplish a set of additional tasks. First, I extract a large set of facial drawing images (~1.2 million instances) from unlabeled data and train SOTA generative adversarial network (GAN) models to generate and a SOTA GAN inversion model to reconstruct faces. When the detector-aided data is leveraged, these generative models successfully learn diverse stylistic features. Secondly, I implement an annotation tool to enlarge the existing set of annotated data. This tool offers users to annotate bounding boxes of panels, speech bubbles, narrations, faces, and bodies; to associate text boxes with faces and bodies; to transcript the text; to match the same characters in the image.
Learning markerless robot-depth camera calibration and end-effector pose estimation
Robot arms are being used more and more in unstructured environments. As such, they are relying more on vision sensors compared to traditional factory robots which are placed in highly structured, controlled and caged work-cells. Some applications that rely on vision data include bin-picking, box picking and placing, assembly and part feeding in mixed human-robot work cells, inspection, quality control etc. Vision data is mainly required to localize the objects to be manipulated, perform measurements and detect near-by humans. Vision based robot systems require extrinsic calibration between the robot and camera in order to work properly. This is a time consuming and tedious procedure which can be expensive as well. Fast, flexible and precise robot-camera calibration is essential for not only industrial environments but also academic lab environments where the location of the camera and/or robot needs to be frequently or is accidentally changed. Extrinsic calibration between a robot arm and camera is a decades old challenge still prevalent to this day. Traditional techniques work by estimating pose of the camera relative to a fiducial marker from multiple points and matching these estimations with the robot's pose. Recent learning based approaches predict extrinsic calibration from images relying heavily on simulation data. In this thesis, we present a learning based markerless extrinsic calibration system that uses a depth camera. We learn models for end-effector (EE) segmentation, single-frame rotation prediction and keypoint detection, from automatically generated real-world data. Our models are based on MinkUNet and PointNet++ architectures. We use a transformation trick to get EE pose estimates from rotation predictions and a matching algorithm to get EE pose estimates from keypoint predictions. We further utilize the iterative closest point (ICP) algorithm, multiple-frames and outlier detection to increase calibration robustness. Our results on the test set with previously unseen camera locations give sub-centimeter (0.74 cm) and less than 0.05 radians (1.69 degrees) average calibration errors and 1.00 cm and 2.74 degrees average pose estimation errors. In addition, we released an open source easy to use tool for robot users to handle robot-camera calibration with a few mouse clicks by seamlessly integrating all the models and algorithms discussed in this thesis.
Addressing the static scene assumption and the scale ambiguity in self-supervised monocular depth estimation
Self-supervised monocular depth estimation is the task of estimating per-pixel depth from a single image without any supervision. Typically, there are two networks to estimate depth and camera pose between consecutive frames, which are then used to reconstruct one view from another for self-supervision. There are two problems with this approach. Firstly, the scene is assumed to be static and the only motion is due to the camera, namely the static scene assumption, however, this is frequently violated in real-world driving scenarios. Due to this assumption, monocular depth methods struggle to produce accurate predictions in the moving regions of the scene. Current methods either ignore moving regions or require an additional instance segmentation input to identify and separately process moving regions. In this thesis, we first propose MonoDepthSeg to jointly estimate the depth and decompose the scene into moving regions to model the motion of dynamic objects. We show that going beyond the static scene assumption improves the accuracy of depth prediction, especially in moving regions. The second problem of self-supervised monocular depth estimation methods is the scale ambiguity. The estimated depth values are in an unknown scale which is typically handled with normalization with respect to the ground truth scale value during inference. We revisit the traditional paradigm of plane and parallax to address this issue and propose DepthP+P to estimate depth in metric scale. Our method shows promising results that are metric scale without any additional normalization.
Two-level temporal relation model for video instance segmentation
In Video Instance Segmentation (VIS), current approaches either focus on the quality of the results, by taking the whole video as input and processing it offline; or on speed, by handling it frame by frame at the cost of competitive performance. In this work, we propose an online method that is on par with the performance of the offline counterparts. We introduce a message-passing graph neural network that encodes objects and relates them through time. We additionally propose a novel module to fuse features from the feature pyramid network with residual connections. Our model, trained end-to-end, achieves state-of-the-art performance on the YouTube-VIS dataset within the online methods. Further experiments on DAVIS demonstrate the generalization capability of our model to the video object segmentation task. We also evaluate our work on autonomous driving setting and show comparable results in KITTI MOTS dataset.
Answering spatial density queries under local differential privacy
Spatial density queries are fundamental in many geospatial data analysis tasks and have numerous applications in the real world, such as determining crowded areas, estimating traffic density, navigation, passenger demand analysis, and so forth. However, answering spatial density queries based on users' data may violate users' privacy by exposing their true locations to an untrusted third party (e.g., a server acting as the service provider or data collector). In this thesis, we propose a solution for answering spatial density queries while preserving Local Differential Privacy (LDP), a state-of-the-art privacy protection standard. Our solution consists of four main steps: partitioning, finding sensitivity, user-side noisy response computation, and server-side estimation. For the first step, we initially propose three basic partitioning strategies: Singleton Partitioning, Holistic Partitioning and Random Partitioning. Based on our qualitative and empirical analysis of the three basic strategies, we design and implement an improved strategy called Advanced Partitioning. For the second step, we adapt graph-based modeling of query sets from the centralized DP literature. Advanced Partitioning also leverages and extends this technique by formulating the partitioning problem as a vertex coloring problem on the graph representation of a query set. For the third and fourth steps, in addition to adapting two popular LDP protocols (GRR and RAPPOR) to our solution, we propose an extension for the Optimized Unary Encoding (OUE) protocol so that it can be employed in our solution. We call the extended protocol Optimized Bitvector Encoding (OBE). OBE is applicable to not only the problem of answering spatial density queries, but also in arbitrary LDP problems with bitvector encodings. We formally prove that the user-side perturbation step of our OBE protocol satisfies LDP and its server-side estimation step produces unbiased estimates. Combining the different partitioning strategies and LDP protocols, we obtain a total of 8 different approaches for answering spatial density queries under LDP, all of which can be parsed as instances of our four-step solution with different choices in the individual steps. We perform an extensive experimental evaluation of these approaches using 4 real-world datasets, varying number of queries, varying query sizes, varying degrees of privacy, and multiple error metrics. Results show that Advanced Partitioning and OBE protocol typically yield the lowest error, demonstrating the superiority of our proposed methods.
Keyframe demonstration seeded and Bayesian optimized policy search
Reinforcement learning (RL) is a promising approach to endow robots with skills. However, RL requires many trials to get satisfactory results. Learning from Demonstration (LfD) seeded RL alleviates this problem by learning an initial skill from human demonstrations. Nevertheless, this approach still requires robots to perform a non-trivial amount of trials. In this thesis, we develop an approach to further reduce these for manipulation skills with perceptual goals. Our main contributions are (1) an algorithm to focus the exploration by using the learned relationship between action and perception and a (2) Black-Box RL Policy Search (PS) method that improves upon the popular Policy Improvement with Path Integral (PI²) algorithm, called the Bayesian Optimized PI² (BO-PI²), that uses reward predictive UCB-type exploration. Our underlying LfD framework utilizes a Dynamic Bayesian Network (DBN) learned from keyframe demonstrations to jointly model the action (end-effector pose) and the goal (object-specific perceptual data) of the skill. The action part is used to generate robot trajectories, and the goal part is used to monitor the success of trajectory executions and to create a Partially Observable Markov Reward Model in order to learn rewards. BO-PI$^{2}$ is used to improve the action part of the DBN with trial-and-error using the learned returns. The novelty of BO-PI$^{2}$ comes from its exploration strategy. The coupling between the action and the goal is used to pick the part of the model to focus on to reduce the effort, in a sense to solve the credit attribution problem. After picking the part to focus on, BO-PI$^{2}$ samples trajectories from the action model to get rollouts, which is typical of PS approaches. In addition, BO-PI$^{2}$ uses a Gaussian Process (GP) to learn local returns from these rollouts, which is improved with each executed trajectory. The next samples are selected by utilizing an Upper Confidence Bound (UCB) approach, using the predicted return and uncertainty of the possible candidate points. This is in contrast to random sampling, used in most PS approaches. BO-PI$^{2}$ also utilizes a skill success based termination criteria, using the goal model to monitor success autonomously. We evaluate BO-PI$^{2}$ with expert and non-expert keyframe demonstrations for three skills. In the expert case, the models are perturbed so that the initial skill execution starts from a failure condition. In the non-expert case, we pick skill models that fail to begin with. We test our approach against the current state-of-the-art PI$^{2}$-ES-Cov algorithm using three metrics: (1) skill success rate, (2) total accumulated reward, and (3) number of trials. In both the expert case and the non-expert case, on average, our approach performed better than the baseline on all three metrics. Our results show that utilization of keyframes allows us to focus on failed sub-goals rather than the entire trajectory, and combined with reward predictive exploration strategies, are beneficial to improve RL performance and reduce the number of trials to endow robot arms with real-life manipulation skills.
Hierarchical clustering attention for unsupervised object-centric representation learning
Extracting object-centric representations from a complex multi-object scene is indeed a crucial milestone for modern neural network architectures to achieve near human level cognition capabilities. Nevertheless, most of the contemporary neural networks that address object-centric representation learning problem require apriori initialization of a fixed set of object describing vectors or cannot manage to handle images of higher resolution. Contrary to long-standing paradigms in the literature, this work proposes Query Breaking Visual Attention (QBVA) module, an efficient and effective building block that introduces a divide and conquer strategy to object-centric representation learning while solving the unsupervised scene segmentation task. QBVA is essentially a stand-alone attention based clustering module that is capable of extracting object-centric representations from a multi-object scene when cascaded into a hierarchical network architecture. QBVA leverages a novel, fully differentiable and non-parametric clustering scheme named Query-Breaking Clustering (QBC) which eliminates the need for initializing a fixed set of clusters and holds the promise to provide dynamic representation for a variable number of objects. We demonstrate that QBVA-Net is indeed a competitive approach to address object-centric representation learning paradigm and prove to be advantageous compared to the state-of-the-art in the sense that it can provide better segmentation performance at the end of the encoder network and theoretically scale up to images of higher resolution.
Incentive compatible and provably secure blockchain applications
With the introduction and advent of blockchain technologies, money transfers have become easier with lower costs, no matter how far the recipient is from the sender. Moreover, the blockchain technology has not only affected the contemporary economical model, but also inspired many novel applications that have been thought impossible without the help of a trusted third party (such as governments or well-recognized companies). However, being a newly developed technology, blockchain still lacks enough research to ensure its security (in terms of mining, transactions, or its applications) and incentive compatibility (i.e., the participants are motivated for following the expected protocol steps). Cryptography and game theory are two well-known tools with traditional proving methods to help us for this task. In this thesis, we utilize game theory and cryptography based security models for building secure applications on blockchain. First, we develop a provably secure blockchain application that we name as e-donation, for secure, fair and decentralized online donations. Second, we develop a practical attribute based digital signature scheme. Third, we analyze and improve the security of blockchain by providing an incentive compatible defense solution against the selfish mining attack of Eyal and Sirer (CACM '18) on proof-of-work blockchain. Forth, we develop a network-based simulation tool for both attacks and defenses. Fifth, we propose a framework for better game-theoretical analysis of multi-player mechanisms when the sizes of the coalitions can be bounded by a threshold, via successfully combining threshold security ideas of cryptography and transferable utility of game theory. Sixth, we apply our threshold coalition notions to show the incentive compatibility of our solution to the outsourced incentivized computation problem in blockchain with multiple outsourced parties.
Elastic pipeline load balancing for dynamic DNNS
Training of dynamic models is gaining traction in DNNs as it reduces computational and memory requirements of large-scale training. Gradual pruning, one of the prominent approaches for dynamic training, prunes (or sparsifies) the parameters of a model during training. However, one of the side effects of gradual pruning is that sparsification introduces an imbalanced workload across accelerators, which in turn affects the pipeline parallelism efficiency. This work introduces DynPipe which dynamically load balances the stages of the pipeline to offset the negative performance effects of pruning. On top of load balancing dynamic models, DynPipe can dynamically pack work into fewer GPUs, while sustaining performance. DynPipe works on single nodes with multi-GPUs and also on systems with multinodes. Experimental results show that DynPipe can speed up the training up to 5.64% in a single node, and 8.43% in a multi-node setting, over state-of-the art solutions used in training production large language models. DynPipe is available at https://anonymous.4open.science/r/DynPipe-1EC5
Building, analyzing and interpreting classroom engagement: Apps and machine-learning models for an affordable programming education
Ensuring students remain engaged in the classroom is crucial for their success in any given topic. In programming education, promoting hands-on interactions and demonstrating real-world use cases are effective methods to foster engagement. However, schools located in socio-economically disadvantaged areas often lack adequate digital infrastructure, such as computer laboratories, to support such engagement-building tasks. Nevertheless, utilizing mobile devices can support programming education due to their availability and affordability. Additionally, mobile devices can help augment the tangible materials to use in collaborative programming experiences. In this respect, the first goal of my research is to develop an affordable tangible programming education environment using mobile devices that supports the collaborative work of students. On top of building tools to foster engagement, analyzing and interpreting student engagement is also a critical component of the teaching process. Analyzing classroom engagement requires a multi-component evaluation of affective, behavioral, and cognitive states. Yet, limited research has been conducted to create a multimodal classroom engagement dataset and analysis model. To fill this gap, my second goal is to build a multimodal engagement evaluation tool using a single camera to ease teachers' workload in group activities. Overall my thesis contributes to Human-Computer Interaction (HCI), Artificial Intelligence (AI), and educational technology research areas with six research outputs: 1. An open-source, user-centered, AI-powered, affordable tangible programming environment that was informed by iterative user studies. 2. A set of design considerations for developing paper-based intelligent tangible programming environments which are informed by iterative and classroom-wide user experience studies. 3. A set of curricular activities to help teachers adapt our programming environment into Turkish national curricula. 4. An open-source audio-visual dataset comprising eight-hour-long video recordings of thirty-three students to predict classroom engagement levels using their self-evaluation scores. 5. Multimodal machine-learning models that can address the multi-component definition of engagement. The image models achieved up to 84% test accuracy on person-based engagement level prediction. The real-time video model that can run on streaming videos achieved 71% test accuracy. 6. A student-centric interactive dashboard to help students view their engagement over time and interpret the results of engagement model prediction.
HyperGAN-CLIP: A versatile framework for CLIP-guided imagesynthesis and editing using hypernetworks
Generative Adversarial Networks, particularly StyleGAN and its variants, have shown exceptional capability in generating highly realistic images. However, training these models remains challenging in domains where data is scarce, as it typically requires large datasets. In this thesis work, we introduce a versatile framework that enhances the capabilities of a pre-trained StyleGAN for various tasks, including domain adaptation, reference-guided image synthesis, and text-guided image manipulation even when only a small number of training sample are available. We achieve this by integrating the CLIP space into the generator of StyleGAN using hypernetworks. These hypernetworks introduce dynamic adaptability, enabling the pre-trained StyleGAN to be effectively applied to specific domains described by either a reference image or a textual description. To further improve the alignment between the synthesized images and the target domain, we introduce a CLIP-guided discriminator, ensuring the generation of high-quality images. Notably, our approach shows remarkable flexibility and scalability, enabling text-guided image manipulation with text-free training and seamless style transfer between two images. Through extensive qualitative and quantitative experiments, we validate the robustness and effectiveness of our approach, surpassing existing methods in terms of performance.
Comicverse: Expanding the frontiers of ai in comic books with holistic understanding
Comics are a unique and multimodal medium that conveys stories and ideas through sequential imagery often accompanied by text for dialogue and narration. Comics' elaborate visual language exhibits variations from different authors, cultures, periods, technologies, and artistic styles. Consequently, the computational analysis of comic books requires addressing fundamental challenges in computer vision and natural language processing. In this thesis, I aim to enhance neural comic book understanding by making use of comics' unique multimodal nature and processing comics in a character-centric approach. The primary data source for this thesis is the Golden Age of American Comics due to its public accessibility and abundance of comic series. However, the availability of annotated data is limited. Thus, to achieve my goal, I have adopted a holistic approach composed of four main steps ranging from curating datasets to proposing novel tasks and architectures for comics. The first three steps aim to create a machine-readable comics database by locating comic book panels, identifying their components, and transforming character identities into a dialogue-like structure and the final step uses this database to train a transformer- based model. The first step involves extracting high-quality text data from speech bubbles and narrative box images using OCR models. I decompose comic pages into their constituent components in the second step through detection, segmentation, and association tasks with a refined Multi-Task Learning (MTL) model. Detection involves identifying panels, speech bubbles, narrative boxes, character faces, and bodies. Segmentation focuses on isolating speech bubbles and panels, while the association task involves linking speech bubbles with character faces and bodies. In the third step, I utilize the paired character faces and bodies obtained from the previous stage to create character instances and, subsequently, reidentify and track these instances across sequential panels. In the final step of my thesis, I propose a multimodal framework by introducing the ComicBERT model, which exploits the abovementioned structure. Cloze-style tasks were used to evaluate ComicBERT's contextual understanding capabilities. Furthermore, I propose a new task called Scene-Cloze, which predicts the next panel given n previous panels as context. As a result, my approach achieves a new state-of-the-art performance in Text-Cloze and Visual-Cloze tasks with accuracies of 69.5% and 77.1%, respectively, thus getting closer to the human baseline. Overall, the highlights of my contributions are as follows: 1. I curated and shared COMICS Text+ Dataset with over two million transcrip- tions of textboxes from the golden age of comics. In addition, I open-sourced the text detection and recognition models that are fine-tuned for the task and datasets used in their training. 2. I refined a MTL framework for detection, segmentation, and association tasks and achieved SOTA results in comic character face and body-to-speech bubble association tasks. 3. I proposed a novel Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification framework to generate unified and identity-aligned comic character embeddings and identity representations. Furthermore, I generated two new datasets: the Comic Character Instances Dataset, encompassing over a million character instances used in the self-supervision phase, and the Comic Sequence Identity Dataset, containing annotations of identities within sets of four consecutive comic panels used in semi-supervision phase. 4. I introduced the multimodal Comicsformer, a transformer-encoder architecture capable of processing sequential panels and their constituents. It serves as the backbone for the Masked Comic Modeling (MCM) task, a novel self- supervised pre-training strategy for comics, resulting in ComicBERT, a potential foundation model for golden age comics. ComicBERT achieves SOTA performance in cloze-style tasks, particularly in text-cloze and visual-cloze tasks, approaching human-level comprehension.
Investigating the missing pieces of sensorimotor reinforcement learning agents for autonomous driving
Reinforcement Learning (RL) has the potential to surpass human capabilities in self-driving without needing any expert supervision. Despite its promise, the state-of-the-art in sensorimotor self-driving is dominated by imitation learning methods due to the inherent challenges of RL algorithms. Nonetheless, RL agents are able to discover highly successful policies when provided with privileged ground truth representations of the environment. In this work, we investigate what separates privileged RL agents from sensorimotor agents for urban driving in order to bridge the gap between the two. We propose vision-based deep learning models to approximate the privileged representations from sensor data. In particular, we identify aspects of state representation that are crucial for the success of the RL agent such as desired route generation and traffic light prediction, and propose solutions to gradually remove privileged information for each with the existing computer vision approaches. Through rigorous evaluation on the CARLA simulation, we shed light on the significance of the state representation in RL for autonomous driving and outline unresolved challenges for future research.
Advancing toward temporal and commonsense reasoning in vision-language learning
Humans learn to ground language to the world through experience, primarily visual observations. Devising natural language processing (NLP) approaches that can reason in a similar sense to humans is a long-standing objective of the artificial intelligence community. Recently, transformer models exhibited remarkable performance on numerous NLP tasks. This is followed by breakthroughs in vision-language (V&L) tasks, like image captioning and visual question answering, which require connecting language to the visual world. These successes of transformer models encouraged the V&L community to pursue more challenging directions, most notably temporal and commonsense reasoning. This thesis focuses on V&L problems that require either temporal reasoning, commonsense reasoning, or both simultaneously. Temporal reasoning is the ability to reason over time. In the context of V&L, this means going beyond static images, i.e., processing videos. Commonsense reasoning requires capturing the implicit general knowledge about the world surrounding us and making an accurate judgment using this knowledge within a particular context. This thesis comprises four distinct studies that connect language and vision by exploring various aspects of temporal and commonsense reasoning. Before advancing to these challenging directions, (i) we first focus on the localization stage: We experiment with a model that enables systematic evaluation of how language-conditioning should affect the bottom-up and the top-down visual processing branches. We show that conditioning the bottom-up branch on language is crucial to ground visual concepts like colors and object categories. (ii) Next, we investigate whether the existing video-language models thrive in answering questions about complex dynamic scenes. We choose the CRAFT benchmark as our test bed and show that the state-of-the-art video language models fall behind human performance by a large margin, failing to process dynamic scenes proficiently. (iii) In the third study, we develop a zero-shot video-language evaluation benchmark to evaluate the language understanding abilities of pretrained video-language models. Our experiments reveal that the current video-language models are no better than the vision-language models, processing static images as input in processing daily dynamic actions. (iv) In the last study, we work on a figurative language understanding problem called euphemism detection. Euphemisms tone down expressions about sensitive or unpleasant issues. The ambiguous nature of euphemistic terms makes it challenging to detect their actual meaning within a context where commonsense knowledge and reasoning are necessities. We show that incorporating additional textual and visual knowledge in low-resource settings is beneficial to detect euphemistic terms. Nonetheless, our findings on these four studies still demonstrate a substantial gap between current V&L models' abilities and human cognition.
RbA: Segmenting unknown regions rejected by all using mask classifiers
Fine-grained visual semantic understanding is essential for autonomous driving and many other computer vision tasks. To this end, segmentation tasks have been witnessing rapid advancements as a result of the growing number of benchmarks proposed. However, existing segmentation benchmarks generally assume a fixed set of semantic categories. Consequently, the development of segmentation methods has been centered around this assumption, while little attention has been poured into handling novel or out-of-distribution (OoD) samples that can potentially be encountered in real-life scenarios. This poses an issue for an autonomous vehicle, as it is crucial to identify unknown objects so that a safety warning can be issued to avoid disastrous consequences in case of failure. As a result, the task of OoD segmentation has been addressed separately, leading to the emergence of methods that adapt to the existing segmentation methods and disregard the performance of the main segmentation tasks. Additionally, these methods also generally suffer from a lack of smoothness and objectness in their predicted anomaly maps due to the reliance on models that follow the per-pixel classification paradigm. In this thesis, we explore the potential of region-level classification models for unknown segmentation as a unified architecture with an inherent ability to express uncertainty. We show that the object queries in mask classification models tend to behave like one \vs all classifiers. Based on this finding, we propose a novel outlier scoring function called Rejected by All (RbA) by defining the event of being an outlier as being rejected by all known classes. We also propose an objective that optimizes this proposed score for boosting the unknown segmentation performance using pseudo-outlier data without hurting the closed-set performance. RbA performs well under high domain shifts and is capable of separating sources of uncertainty, such as at the boundaries, due to known class ambiguity. We evaluate RbA on several unknown segmentation benchmarks and show that it achieves state-of-the-art performance with significant margins compared to previous pixel-level unknown segmentation methods. We report extensive ablation experiments that validate the effectiveness of RbA.