Theses supervised by Yrd. Doç. Dr. Gökhan İnce
12 theses · İstanbul Technical University
Audio visual attention for robots from a developmental perspective
Developmental robotics is associated with embedding a human-like behavior in a developmental manner within the robot. The field takes inspiration from the human cognitive model for the syntheses of a cognitive-like decision-making model. The key concepts associated with the topic are hierarchy of the system, incremental learning and multi-sensory cooperation. To achieve a developmental like structure, the system has to be hierarchical in nature. Specifically, the perception level has to be separated from (and put beneath) the decision level. The cognitive-like structure is achieved by ascending from the lower-level to the upper level. The ascension should entail a reduction in the engagement of processing resources and a suppression of response to stimuli of lower priority. The process is similar to filtering where only important information propagated further in the model. Not only that, but it is also modified so that the meaning is conveyed in a symbolic manner rather than transporting the raw information itself. However, for the establishment of such hierarchy to be truly developmental in essence, incremental learning is a crucial factor. After all, the human cognitive model achieves its highly complex dynamics through a lifetime of evolution. As a matter of fact, incremental learning seems to offer, in this case, more than a way out of the scarcity of data at time zero of the experiment dilemma. But it also offers an opportunity of allowing a unified gradual growth of multiple prospective cognitive -functions embedding on a larger scheme and over the long run of the field's research. The multi-sensory cooperation comes then naturally in the picture because the majority of cognitive functions feed on the fused information provided by the different lower level sensory. Among the highly hierarchical cognitive functions, attention is one that has been heavily addressed by the literature. More specifically, audiovisual attention relies heavily on the concept of fusing multi-sensory information. In order to design an audiovisual attention system within a developmental framework, the nature of the attention phenomenon found in the human model has to be understood and translated into the machine learning and artificial intelligence view. Concepts related to attention like "audiovisual objects" and "binding" found in the neuroscience literature seem to provide key insight to how audiovisual attention occurs. "Audiovisual objects" are the cognitive-association representative of an auditory quantity emitted by a visual object perceived by both eyes and ears. The "binding" concept is the one responsible for creating this symbolic association. Examining these concepts for design inspiration, and taking into account that visual and auditory quantities are perceived through "features" from the scope of Artificial intelligence, a system-design that attempts to replicate such concepts can be established. The audiovisual attention system introduced here is of a two level hierarchy. The low level modules are auditory and visual perception modules; the second-level module fuses the information provided in the low level ones to achieve a higher-order cognitive like behavior. The auditory module relies heavily on the concept of incremental unsupervised learning, where the classifier labels the incoming new data instances, assumes the label to be true, append the instance to its past knowledge and retrains again. Three classification model structures that make use of a support classifier (which can be thought of as a short term auditory memory) are introduced to attempt to decrease the drop down of accuracy overtime. These models are Shared base-knowledge life-long model, Separate base-knowledge life-long model, Separate base-knowledge epoch-based model. The models investigate the accuracy behavior of unsupervised learners. Six different classifier algorithms are monitored for their behavior and the results of the models are compared to the offline learning case for the same algorithms. The performance in terms of accuracy and training time of each classifier is also compared within different feature set frames and different amount of data per training set frame. Audiovisual attention is then introduced through a two level architecture and three phases of learning. The first level of the architecture is comprised of the auditory module and the visual module. The auditory module operates with the Separate base-Knowledge life-long model suggested for incremental learning. The visual model uses a pre-trained (in an offline manner) object recognition model of deep neural network with a Single Shot Multi-Box Detector (SSD) architecture with the tensorflow object detection API. The second level module makes use of a meta classifier that operates on the classes labels probability of the low level modules in order to fuse the audiovisual information. The meta classifier is evolved developmentally in three phases. The first phase (Exposure) establishes the base knowledge for the meta classifier and realizes the audiovisual "binding" concept. In this phase the symbolic representation of an audiovisual object is realized through concatenating the label probability vectors of the auditory and visual modules to make up a meta feature vector. This way the association is introduced via correlation between label probability (low level modules decisions) values of the two sensory models. The meta classifier is trained with the meta feature vectors in an offline manner at this stage. The second phase (Expansion) attempts to expand and change the nature of a pre-established class in the first phase through enumerations of mismatched audio visual information. Mismatch events are not forced on the system but are rather taken when the meta classifier makes a false classification of a meta audiovisual instance. Therefore, learning at this stage is supervised by the caregiver, where the system is told of its false labeling and informed of the true label in order to enumerate the instances correctly. The last phase (Reappraisal) attempts to equip the system with the ability to correct its false guess of an instance through previous knowledge of related instances and via storing the wrong guessed instances in memory to go back to later after more knowledge has been obtained. For this phase two methods (subroutines) for correction are used. The first subroutine looks at the past instances in the stream of which the instance at hand was labeled falsely. It takes the maximum vote over the labels of these instances and applies it to the instance at hand. If the first subroutine fails (or if the instance in question is the first in the stream) then the second subroutine tries to infer the label by looking only at the audio meta information or the visual meta information. The subroutine tries first to decide which of these two information vectors is more likely to be accurate (or less noisy). Then, it picks the label whose audio/visual association (that was established in the exposure phase) is voted by the more likely to be accurate among the two types of meta information. The communication mechanism by which the low level modules and the second (meta) module communicate and broadcast information is maintained by the topic/message architecture of ROS. The timestamp feature is utilized for audio and visual capturing to avoid synchronization issues between all modules and insure that all audiovisual information being fed to the system correspond to each other. The system was tested in two parts. The first part addresses the audio classification by incremental learning problem. The second part addresses the audiovisual integration and audiovisual object classification problem. First, for the incremental learning models test, the auditory data were tested with different feature sets that consist of Mel Frequency Cepstrum Coefficients (MFCC), delta MFCC, energy filter banks and a range of other features that include spectrograms and Chroma-grams. Of those, the performance accuracy and training time were compared. Single MFCC features were found to be efficient since they train the fastest and their accuracy is comparable to a more complex feature set. Then, the support based incremental learning models were tested for their behavior. Results showed that most of the classifier such as support vector machines and random forests experienced a lift in performance accuracy with models introduced as opposed to regular incremental learning or even offline learning in the case of multilayer perceptron. Both the Shared Base-Knowledge lifelong model and Separate Base-Knowledge lifelong model performed well and obtained good results. The Separate Base-Knowledge epoch-based model however failed to achieve a good performance except for the case of k nearest neighbor algorithm. As for the audiovisual integration problem, the learning phases were tested with a custom made test set that includes the fused auditory and visual features obtained (as the labels probability ) from the lower level auditory and visual modules. The exposure phase achieved an accuracy of 89% on the test set. The expansion phase failed however, to achieve a good performance and the accuracy fell down to 7% because the meta classifier classified all incoming instances as a "don't-care" class after expanding said class. The reappraisal phase was thus mounted on the exposure phase directly. Its defined subroutines and behaviors managed to correct the false label estimations on the test set and the end accuracy was lifted to 100%.
Automatic posture evaluation for professional voice users
Nowadays voice trainings are executed by therapists in the worldwide to their knowledge, experiences and skills. In these therapies, tension spots of a body are evaluated. A body is relieved with guidance of a therapist and applied physical exercises. Breathing exercises and implementations are executed. Breathing and body posture relation hybridization with voice resonance is carried out. Workouts are executed by combining breath, body posture and voice resonance. Trainings are conducted to help subjects improve their voice qualities and to feel correlation between body posture and voice. Enhancements are executed practically by talking or singing a song. In order to master the usage of muscle groups as confident as possible, breathing exercises are required. Afterwards, body locations with the lack of resonances are sought using subjective sense of a therapist and continue to treatment with a focus to chosen locations. After each therapy session, important points are highlighted and if it is needed a schedule of exercises are given to a patient. Voice recordings are saved for the assessment of voice both before and after the therapy. To sum up, weekly improvement of a patient is observed. So far, voice quality evaluation of professional voice users are carried out subjectively. However, classical voice evaluations cannot be done by an objective approach, yet done with therapists' intuition. In this thesis, a measurement system was proposed to evaluate voice quality objectively by the posture of a patient. In the first step, voice quality metrics were determined and it was defined as a human voice which propagates to significant distances without any deterioration. With respect of this, three level of voice quality hypothesis was created. In the first step, a voice which faded out in a patients' body and cannot be propagated to even close proximity points were labeled as low quality voice. In the second step, a voice which has a natural amplitude and reached to some extend distance without any deterioration was labeled as normal quality sound. In the last step, a voice which reached to great extent of distant points in big halls namely opera hall without any deterioration was labeled as high quality voice. In the first stage, in the presence of a professional sound therapist, five people were asked to voice ``aaaaa'' in all three levels. Sounds extracted by the subjective approach of the voice therapist were labeled according to the three classes identified. Thus, a sound quality data set was created. 26 different acoustic features have been extracted from the extracted audio data set. Some of the extracted features were created as features suggested by the voice therapist and voice therapist. Other attributes have been identified in the literature as attributes used to classify different sound properties. 57-dimensional feature array was created from each audio window received from the audio data using 26 attributes. The difference with the reconstructed system is that it can extract attributes in real time. It has been observed that by selecting 5 parameters which are most successful in classifying the sound quality classification of the extracted sound parameters, the feature size is reduced and the classification success is increased. In particular, it has been observed that the energy value of the mel-frequencies generated by the principle of transmitting the emitted energy values to the brain's hearing center by clustering the incoming sound of the human inner ear within certain frequency ranges and thus being audible is a very successful sound quality classifier parameter. The features extracted from voice data labeled by voice therapist were used for the training of an artificial neural network and the system was created to predict voice quality. In the second step, correlation between qualitative parameters of given voice and body postures were examined and results were obtained. In order to extract body posture data, Motion Capture Device(MOCAP) was used. It is a device that extracts angular position information of joints in body to calculate distances between neighboring joints and finds angles between them in real time. In the presence of a voice therapist, five users wearing the MOCAP device were asked to make an ``aaaa'' sound and voice and posture data were collected with eight static postures and two dynamic motions accompanying. The three-dimensional center of mass of the extracted postural data was calculated, and the correlation between each voice parameter and the center of mass of the body in terms of the distance from zero position in each dimension was inspected at and it was observed that some voice parameters are significantly related to the center of mass change of body. An interface was written on the Unity platform to simultaneously record posture data and audio data. The real time posture change at the interface can be observed using an avatar, and the sound data and posture data can be extracted and recorded to the computer at the desired time. The recorded postural data was again analyzed by a built-in function created on the Matlab platform to extract the three-dimensional center of mass deviation data of a body. In the third stage, an objectively sound focus was determined from the instantaneous changes in the body's postures. Thus, a system has been developed in which a wearer of a MOCAP device can sense, which predicts optimal posture. In order to be able to create this system, the audio features extracted during the experiments were fed into the artificial neural network model which was formed in the first stage, which determined the sound quality, and as a consequence the instantaneous sound data was labeled. In respect, the labels of the changing postural parameters were accepted as labels of the instantaneous voice data, and the changed postural parameters were trained in another artificial neural network model. Thus, by testing the changing postural parameters in the model, it is possible to predict what quality natural voice will be likely to emerge at the time of that posture. 10-fold cross-validation score of the artificial neural networks that predict the voice quality was found to be 85.19\% and the 10-fold cross-validation score of the model analyzing the possible voice quality of the postural parameters was found to be 81.33\%. In future work, the objective sound quality will be determined to have more classes than three, and a system will be developed that evaluates a closer objective sound quality to reality. By doing this, the possible sound quality of the posture that the person is in can be examined in more detail and the feedback given to the individual will be elaborated more. At this point, a subject will be able to better predict the possible optimal posture when trying different postures.
Sound morphing using auditory p300 based attentional brain-computer musical interface
Thesis describes the development and testing of a Brain Computer Musical Interfacing (BCMI) system that allows a user to transform one element of a musical texture by paying attention to that particular element. Building on previous works, the system makes use of Auditory Evoked Potentials (AEP) to elicit P300 waves. An oddball paradigm with multiple target stimuli was developed, and sound stimuli were diffused through a multi-channel speaker array for improved focus on the sound stimuli. A sound synthesis model was created for transforming sound texture based on neural oscillations. Artificial Neural Network was used to predict possible P300 waves showing the attentional focus. Using this technique, P300 waves were successfully predicted for the most of the test subjects.
Comparative assessment of mobile navigation applications using 2D maps and augmented reality interfaces
Navigation has always been a problem for people, when they visit a location which they have not been before. Especially in universities with big campuses, new comers and guests need a guide for finding their way and getting to know the environment. Most of the universities provide guidance with signposts and web based maps. However, these tools are not the most user friendly ways of providing guidance service to people. In order to provide a better service for guidance purposes, mobile solutions should be developed. Since, nowadays, almost everyone carries a smart phone, mobile applications are useful tools to reach to the people and provide service. Mobile navigation applications are effective tools for helping people to find their way and explore their environment. The most popular navigation applications in the market are capable of navigating people to their destination as well as providing extra functionalities to help users to get familiar with their surroundings and to access information about locations that are in users' preferences. The main disadvantage of these applications are the weak GPS accuracy, which may confuse the users. Another disadvantage of these applications is that they do not have the detailed map of every area on the earth, which disables the users to use the applications if the roads or the pavements around them are not presented on the map. Therefore, there is a need for guidance service specialized for such areas. The main challenge for the mobile applications is to provide user-friendly interfaces, which also reveals a good user experience. In the design process of applications, developers should not only consider usability issues, but also consider the user satisfaction among the interaction between the user and the application. The type and quality of interaction that an application provides may take it to the top listings in the application stores, or may make people not even notice it. This thesis focuses on the user experience of mobile navigation applications by providing two types user interaction styles. The first type of user interaction style examined in the thesis provides guidance service with a 2D map interface. 2D map interfaces present a satellite map of earth and shows locations with 2D content, such as pinpoints, images, text to the users. This type of interface provides a bird's eye view of an area. The route information is shown to the user using lines on the map. Users' interaction with this interface may include finger gestures or voice commands. Users can manage the map by tilting, rotating, viewing, zooming in, and zooming out using fingers. The second type of user interaction examined in the thesis is the Augmented Reality (AR) browser interface. An AR interface enhances the real vision of the user with the digital information in order to provide new functionalities or more detailed information about what user sees. AR applications has been categorized as marker based and location based applications. In the scope of this thesis only the location based AR applications were examined. The location-based AR applications have been known as the AR browser applications with which users can browse or discover their environment. AR browsers use device's camera to access the real world image and show any digital media, such as, 2D images, text, videos, 3D models, onto the live camera view of the device. Users can explore their environment by browsing points of interests in such applications, without using any finger gestures. Another important contribution of this thesis is the guidance method called waypoint approach implemented in the AR interface, where users can get to their target location by following a series of waypoints displayed one after another onto the camera view. These interaction styles were implemented into two mobile navigation applications, which were intended to serve to visitors and newcomers of Istanbul Technical University for guidance purpose. The comparison of these two interfaces in terms of user experience was carried out by developing one sample application for each interface and conducting a field test and applying couple of questionnaires. Two groups of 10 participants were asked to follow two consequent routes. The first group took the first route guided by 2D map interface followed by the second route by AR interface and the second group took the first route guided by the AR interface followed by the second route by the 2D maps interface. Participants were asked to think aloud during the tests. After each test, participants were asked to answer NASA TLX and Post Study System Usability Questionnaires. The NASA TLX questionnaire was useful to compare these interfaces in terms of mental and physical workload, frustration, and performance of the participants. The PSSUQ were used to analyze these two interfaces in terms of user satisfaction, usefulness, information quality, and interface quality. In addition, participants' route completion times were also recorded. The results of the user experience tests revealed that the application with 2D map interface outperformed the AR based application in terms of all criteria. Participants travelled the routes in shorter times with the 2D map interface. The NASA TLX results showed that the 2D map interface was less demanding both mentally and physically, caused less frustration and required less effort from the participants. According to the PSSUQ results 2D map interface was more satisfactory for the users than the AR interface. In terms of degree of exploration of the environment, AR interface was found to be more useful. Participants stated that the combination of these two interfaces would be an improvement on top of the existing solutions, which is one of the future research ideas of these thesis. The user preferences may be also included in such systems. Moreover, instead of using smartphone, using optical head-mounted device could also create a whole new type of interaction. Both of the interfaces were also evaluated according to their usage of resources, such as GPU load, CPU load, and battery power. It was observed that the AR interface was more CPU and GPU power demanding since it uses camera and sensors of the device more intensely and frequently. Because of high ratio of usage of CPU and GPU, the AR interface consumed the battery power faster than the 2D map interface.
A distributed human identification system for indoor environments
Nowadays "internet of things" and "robotics" topics are getting more and more popular because "state of the art" technologies are easier to be applied in our daily lives. Utilization of "internet of things" started home automation systems to use devices at home remotely. While these automation systems aim at giving access to designated areas, they also offer solutions for unauthorized access problems by keeping logs of video records and control mechanisms as well as features for more security measures. Besides, developers are able to implement and monitor video surveillance systems to support security in the IoT platforms using low cost embedded computers. This study presents an entire system, which is similar to home automation system and allows the login of a person entering a private property using face recognition system, video calls, sending notifications and having more components such as threat detection. In this study statically positioned camera systems were implemented to identify people thanks to face recognition algorithms. Captured images from fixed positioned camera systems(FPFRS) had problems to process on identification. The problems were differences in distance of people to the camera and differences caused by posing. Therefore a mobile robot that has camera, was incorporated to recognize faces of people in the environment. The mobile robot can track people and keep the distance to the people constant. Thus, the mobile robot can solve the distance problem. However, the mobile robot can not track people perfectly without miss. Sometimes it can have trouble while tracking people. Thus, integrating the mobile robot to the Fixed Positioned Face Recognition System has a big potential to deal with the face recognition problem in indoor environments. Therefore an distributed approach was proposed to recognize humans in this study. The system utilizes 1) Cameras statically mounted on wall devices (FPFRS), 2) Cameras mounted on mobile robots. FPFRS were built on a tiny embedded computer. Motion sensor and infrared camera components were used to gather data about environment. However, sometimes they can fail to recognize faces because of their fixed positions. This decreases face recognition performance of the system when FPFRS tries to recognize person from captured images. Because distance between camera and person is very important in order to obtain features of face. To improve the performance of this system proposed method given in this thesis aims at solving the problem of fixed position with robots. Therefore a mobile robot is added to improve face recognition accuracy gathering face images from certain distance and moving around face image that is aligned to the camera of robot before captured. With the help of the mobile robot, the system becomes more interactive and information gathering from environment process is done more efficiently. As a result proposed hybrid system in this study not only uses fixed positioned captured images by FPFRS but also uses images that are taken from mobile robot. To provide network communication between FPFRS and the mobile robot which are linked to in this distributed system, modified techniques are used which follow the rules of client-server approach. Each of the FPFRS and the mobile robot runs main program during scenarios. Inputs can be taken from the environment and enable the trigger mechanism. An FPFRS schedules its jobs and behaves according to the active jobs. After the job is done, FPFRS deactivates the jobs in the database records. These records can be manipulated from other devices. Thus communication and trigger mechanism between devices is established. These systems are more complex and it is hard to follow this kind of systems, actions. Architecture is designed to expand by adding new FPFRSs or mobile robots to the system and each FPFRS can communicate via server. One FPFRS and a mobile robot have scenario to capture more images of people from different positions. Then each device runs face recognizer and obtains a result which consists of name of person and confidence of face recognition score. Using the results of face recognizer, various techniques, including machine learning algorithms, are applied to improve accuracy of face recognition. In addition to choosing prediction based on highest confidence in face recognition, random forest tree, decision tree, and linear SVM methods are applied to address the issue of face recognizer improvement. Described systems are implemented and tested systematically based on various scenarios. To show the impact of the effect of changes on accuracy of face recognition system, experiments were done with and without the mobile robot. The proposed system was evaluated using 1)Face recognition scores on a face database, 2)Performance of video calls, and 3)Case scenarios performed on human test subjects. We have verified the effectivity of our approach by using six scenarios. In the first scenario button triggered human identification has been evaluated. In the second scenario motion triggered human identification has been analyzed. In the third scenario human identification using rotational tracking has been investigated. In the fourth scenario human identification using displacement tracking has been examined. In the fifth scenario human identification using displacement and voice combined tracking has been illustrated. In the sixth scenario human identification using robot initiated dialog has been analyzed. One or more FPFRS and the mobile robot have scenario to route the mobile robot to capture images of people on designated area of motion sensor triggered FPFRS based on the scenarios given above.
Real-time emotion recognition from EEG signals using one electrode device
Since the day the computer was first invented, there has been an interaction between people and computers. This interaction can be performed through software or hardware. The interfaces that provide interaction between people and computers are defined as Brain-Computer Interfaces (BCI). Electroencephalography (EEG) signals are preferred because of their practical use and portable structure in BCI applications. EEG technology measures the electrical activity of the brain. Many applications developed in recent years benefit from this activity. In recent years, researchers have concentrated on the development of ElectroEncephaloGraphy (EEG) based Brain-Computer Interfaces (BCI) to increase the quality of life using medical applications. In this context, BCI applications were introduced that could meet the basic needs of people with physical disabilities. In addition, the impact of the content produced in the marketing field on the users was tried to be determined with BCI applications. In recent times it has become an indispensable product for meditation practices and neurotherapy. Many games and entertainment applications were further enriched through the BCI. BCIs are used for marketing, gaming, and entertainment to provide users with a more personalized experience. Both medical and non-medical applications require the ability to interpret the user's multimedia-induced perception and emotional experience. The changes that both internal and environmental factors create in a person-specific manner are called emotions. Estimating the emotional state of a person is an important step to build effective BCI applications. Therefore, studies are needed to identify the correct emotional state. In the literature, there are few studies on real-time emotion recognition with EEG signals. With this study, it is aimed to close these deficiencies. It is also possible to foresee that neurotechnology will be the technology of the future. The day when people can upload their thoughts to computers is an unquestionable fact that these studies will prevail with applications that can increase the cognitive capacity of people. With the developments in EEG technology in recent years, EEG devices can be obtained with more reasonable costs. In addition, more complex advances are being made every day by using less channel EEG devices. The number of studies using a single-electrode device for emotion recognition is low, because single-channel EEG devices have recently been marketed. Single electrode devices are economical and easy to use. It is predicted that it will be included in the literature more widely in the coming years. In this thesis, a system is designed which detects instantaneous emotional state by using EEG data obtained in real-time with a single-channel commercial BCI device. While designing this system, it comes from the top of the following challenging problems: The collection of EEG signals, the selection of the most appropriate features, and the creation of the most appropriate detection model using these features. A software was written to collect the EEG signals. This software can report data coming in real time. After this software has been written, it has been tested whether the software works correctly by looking at the reported data. For the selection of the right features, firstly visual and auditory content was created to warn emotions correctly. For this reason, visual and auditory items that are both short in duration and as effective as possible have been chosen. The sufficiency of the content found was determined as a result of several pre-evaluation tests. After the content selection was completed, decision trees and random forest algorithms, which are widely used in the literature, were used to correlate EEG data with the emotional state of the content. The order of importance of the features is determined using the random forest algorithm. It has been decided which features should be in the final model by taking the order of importance of the features. In order to create the most appropriate model, the most suitable parameters were determined by using the deep learning algorithm and the grid search method for different parameters. For the suitability of the specified model, the data were divided into groups as training, validation and test data and the success rate was measured. It is also shown that the response time is fast enough to work in real time. The generated model is tested in real time and the response time is measured. It has been tested to produce accurate results under the required time constraints. The success of the system created on this issue has been revealed. Both the real-time and non-real-time classification success of the generated model is measured. In both cases, a success rate was achieved that would meet expectations. In addition, the system has been tested for both supervised and unsupervised learning using an incremental learning approach. All the parameters affecting the success of the system are discussed. The created model and method will be a pioneer for such studies. With the experimentation and the increase of collected data, even the ordinary events in human life will reveal the relation with the brain signals. To sum up, first, while the users watched a video consisting of film fragments and binaural beats that would trigger different emotional states, the EEG data from the users were collected. Then, pre-processing and feature selection were performed on this EEG data and an emotion recognition model with an accuracy of more than 87% was generated using deep learning neural networks. Using this model, a system which detects the instant emotional state on the queuing port structure of a certain length buffer using real time EEG data is designed. As a result, we present a real-time emotion recognition model using EEG signals. This thesis tackled to figure out the specific link between brain waves and the multimedia-induced emotions. Using an ANN-based classification scheme, the proposed system classifies the multimedia genres such as funny, horror, weepy with an average accuracy of 87% which cause emotional or psychological experiences that are induced in viewers. The system can be used for both medical and non-medical applications. In the future, we plan to develop a real-time game based on the fears of the players which will impact all events in the game. The game will come with a horror story which sets players up to solve a mystery while overcoming their fears. In the future, with the increase in work in this area, a society is hoped that people will not be sick and will be happy. But it is also necessary to work to ensure that these studies are not used outside ethical and moral boundaries. There is also the possibility that the ideological apparatus of the state may turn into a dystopic cluster by using such studies. For this reason, it is aimed to develop protocols in order to keep such applications in ethical limits in the future.
Virtual and augmented reality based interfaces for choreography generation
Choreography is a vital creative process of defining movement. Digital applications go a long way in helping choreographers achieve their tasks in choreography generation. Thus, since the late 1960's researchers have tried to use the computer in the choreographing process. In the early days choreographing applications made use of 2D stick figures to represent characters. However, as computer processing power and graphics improved 3D realistic models became the norm. These advances in technology have also seen different techniques to interact with computing systems emerging, this has been punctuated by the advent of Natural User Interfaces (NUIs), whose interaction models the way humans naturally interact. NUIs remove the reliance on traditional input methods like keyboards and mouse, but rather focuses on gestures, touch and voice. Furthermore, technologies like Augmented Reality (AR) and Virtual Reality (VR) have transformed interface development. AR presents an interesting way to interface design. It allows users to interact with both the real world and the virtual world simultaneously. Whereas, VR immerses the user into a synthetic world. Thus, it is important to embrace these technologies in the choreographing process. In this study, we investigate the effects of different interaction techniques on user experience for a choreography generator. We evaluated different interactive techniques, which are the direct input methods like touch screen and gestures, and indirect methods like the mouse and keyboard. We also evaluated VR and AR as interfaces for choreographing tools. Our research contributes to the study of how different interaction methods of the same application affect user experience. It also extends the evaluation of VR and AR in training applications. Previous research on VR and AR has shown varying results which differ from domain to domain. Therefore, it is imperative to evaluate these two technologies on choreography tools so that stakeholders in choreography know which interface to invest on. For this reason, we develop an AR based choreography generator and a VR-based choreography generator and compare with a Personal Computer (PC) based choreography generator and a touch-based mobile application for choreography generation. The PC based generator uses the keyboard and mouse as the interaction technique, whilst the mobile application utilizes the touch screen. The AR interface utilizes the touch screen and gestures, whereas the VR interface utilizes the touchpad and gestures. These interfaces allow choreographers to complete the same tasks on them. Using the choreography generator, users can add actors and other props into the scene. The applications allow the choreographer to be in charge of the actors just like in the real world, he/she can define movements of the 3D actors. The choreographer can add from a diverse list of characters. The props are essential to decorate the scene or to help emphasize concepts. The interfaces also have interactive items that allow the choreographer to control the speed of the characters. The PC based interface, the touch-based mobile application interface and the virtual reality based interfaces present the same graphics to the user. These three mainly differ in the interaction method. However, the VR interface also gives the user the immersion effect, whereby a user is immersed into the choreographing world and feels like he/she is inside the virtual environment. Since AR allows users to interact with the real world and virtual objects simultaneously, the AR interface graphics differs significantly from the other interfaces, however it offers the same functionality. AR gives the users the power to use real world objects with computer generated imagery. It presents a flexible way which is important especially with interfaces for training and teaching applications. In this research, we evaluate user performance and user experience on the interfaces in terms of task completion times, as well as subjective criteria, such as mental stress, physical stress, pleasure and frustration experienced. The physical stress scores give us an indication of the physical requirements needed to perform a task, whilst mental stress gives us the cognitive load needed to perform the tasks. These criteria help us investigate the four factors of usability: satisfaction, effectiveness, ease of learning and efficiency. We used usability testing methods to evaluate the four interfaces using a total of 20 test subjects. To obtain objective results we recorded the task completion times, whereas to obtain subjective results a post-test questionnaire was used. Furthermore, the think-aloud technique assisted us to understand the user's psychological experience whilst using the interface. The combination of time it takes to complete tasks, mental load and physical load gives a fair understanding of the efficiency and effectiveness of the interfaces. The experiments were conducted in a controlled environment that was the same for all users. Furthermore, the users used the interfaces randomly to avoid task adaptation. Participants were given four tasks to complete on the interfaces. Firstly, they had to add two dancers to the left and right of the scene. After that they had to add props to the scene. After adding the dancers and props users had to define the path for the dancers to follow. Finally, they had to play the defined choreography. Whilst playing the choreography users were encouraged to explore the system further. This included changing zooming levels and viewing angles. The results show that touch-based approaches produce faster task completion rates. This was attributed to the naturality of using the approach, especially when defining the path using the finger. The results also demonstrated that NUIs are an effective way to enhance user experience. They give users an easier and faster way to complete tasks using intuitive methods. Furthermore, this approach does not require pre-training as users can use their intuition to understand how to interact. The AR interface's ability to zoom in or out and change viewing angles by only moving the mobile device around the marker, gave users a natural approach of performing the zooming task, which is analogous to moving near or further from an object in real life. This gives users a compelling effect which enhances user experience. We also noted that pointer based approaches can be very effective depending on the tasks. Especially with tasks that require a small area of contact to interact the pointer proves to be a good option as it provides precise pointing. The results also verify the effectiveness of AR in developing training and design applications. In the choreographing process AR has an additional advantage as both computer generated imagery and physical objects can be used to emphasize concepts. Therefore, AR gives the choreographer flexibility allowing him/her to define his/her own stage in a way that is necessary for a particular task. Furthermore, the results showed that AR gives users excitement and pleasure which comes from the curiosity it creates. The major downside of mobile AR is the physical stress it exerts on users, which is due to hand movements as the user interacts with the interface. Another downside of AR is due to the errors that arise as the mobile device tilts whilst the user is performing the tasks. This is due to the natural hand tremor. Although VR gives pleasure to users, its high tasks completion rates, and high physical stress scores negatively affect its efficiency and effectiveness in choreography. This adversely impacts its overall usability scores. Therefore, most of the users preferred the AR interface as compared to the three other interfaces. We conclude by proposing an AR interface that utilizes virtual buttons and voice commands to interact. In this way, we can achieve a system that completely embraces the natural user interface approach. We strongly suggest that choreographers should invest in AR, as it is flexible and allows them to use both computer generated imagery and physical objects to illustrate concepts.
Ses tabanlı otomatik medya takibi
Televizyon, reklamlar için alternatif mecralar içinde en etkili ve kritik olanıdır. Dijital çağda internet ve mobil mecralar önem kazanmaya başlasa dahi reklamverenler için televizyon reklamları hala yeri doldurulamayacak niteliktedir. Her ne kadar özellikle genç yaş kitlesi için televizyon izleme oranları azalsa ve yayıncılık anlayışı internet tabanlı yeni nesil ürünlere dönüşse de, tüketicileri ve markaların satış rakamlarını etkileme konusunda en büyük paya sahip araç hala televizyon reklamları olmaktadır. Özellikle televizyon izleme süreleri ve alışkanlıkları baz alındığında ülkemiz, diğer ülkelere göre televizyon reklamcılığının daha da önemli olduğu ve pastadan daha büyük pay aldığı bir ülkedir. Televizyon reklamlarına harcanan paraların büyüklüğü, reklamverenlere, reklam ajanslarına ve medya takibi uzmanlarına, çıkılan reklam kampanyasının denetlenmesi gerekliliğini de ortaya çıkarmaktadır. Yayıncı kuruluşların, reklamların yayınlanma toplam süre ve zaman için yapılan sözleşmelere sadık kalıp kalmadığının kontrol edilmesi için çoğu durumda insan çabasından faydalanılmaktadır. Bütün televizyon kanallarının gözle takip edilip, reklamların yayınlandığı zamanların not edilip raporlanması; insanlar için sıkıcı olduğu gibi zor, hataya açık ve zahmetli bir işlemdir. Bu hataya açık işlem, ses ve görüntü işleme teknikleri kullanılarak otomatikleştirilebilir. Böylece maliyet azalmakla beraber kalite de artmış olacaktır. Günümüzde özellikle ses işleme alanındaki çalışmalar hayret verici derecede başarılı olan ürünler ortaya çıkarmaktadır. Hemen hepimizin kullandığı akıllı telefonlarda, şarkı ya da ses tanıyan uygulamalar bulunmaktadır. Üstelik yoğun gürültü altında dahi, örneğin kalabalık bir kafede bile çalınan bir eseri cep telefonlarımızdaki uygulamalar kolayca arayarak, saniyeler içinde hangi şarkı olduğunu bulabilmektedir. Ses arama alanındaki bu gelişmeleri, reklam takibi ve yayın izleme sahasına uygulamak mümkündür. Zaten halihazırda uygulamada olan, telifli içerik için iletim kanallarını gözlemlemek konusu da, ses işleme alanındaki yeni gelişmelerle daha kesin ve hızlı sonuçlara erişebilir. İşitsel veri ile örnek bazlı arama oldukça tatmin edici doğrulukta sonuçlar üretmektedir. Bununla birlikte görüntü işleme alanındaki tekniklerin, ses işleme alanına göre daha karmaşık olduğu iddia edilir. Bu yüzden geliştirilen sistemde, televizyon daha çok görsel bir medya olmasına rağmen ses ile aramak yeterli olmuştur. Yalnızca ses verisini kullanmanın getirdiği bir yarar da, aynı uygulamanın değiştirilmeden radyo reklamları için de kolayca çalıştırılmasını sağlayacaktır. Çalışmamızda, televizyon yayınlarındaki reklamları ses işaret işleme teknikleri kullanarak gerçek zamanlı olarak algılayan ve tanıyan bir sistem geliştirilmesi amaçlanmıştır. Bu sayede reklamverenlerin televizyon mecrasında yayınladıkları reklamları takip edip raporlandırabilecekleri bir araç ortaya çıkarılacaktır. Geliştirilen bu sistem yayınların uygun şekilde kaydedilip, depolanması ve dizinlenmesini sağlayan yüksek erişilebilirliğe sahip stabil bir altyapıya sahiptir. Ses parmak izi çıkarma ve eşleştirme teknikleri geliştirilmiş, yeterli hız ve doğrulukta çalışan algoritmalar gerçek zamanlı akışlarda kullanılabilmek üzere uygunlaştırılmıştır. Önerilen sistemin etkin bir şekilde çalıştığı gerçek veriler üzerinde var olan yöntemlerle kıyaslamalı bir şekilde gösterilmiştir.
The effect of visual and text interfaces in teaching robot programming
Programming is a popular subject in education and experts emphasize the importance of teaching programming to the children in the high school or even earlier. In European countries, ministries of education give efforts to integrate programming education to school curricula. Some of these countries already have programming classes in high schools and consider to give programming class to younger students. In this respect, methods of the programming education is an important subject and deciding on more successful methods is the problem of this area. In this work, we address this problem and compare different programming interfaces to each other. While teaching programming, we use an educational robot to increase the understanding of the programming environment. Thymio, which is the educational robot we choose, does not require any knowledge of electronics beforehand and it is in no need of hardware related constructions before the programming. Therefore, when using Thymio our students can focus only the programming part and use different prebuilt functions of the robot. Thymio is simple structured, has a special programming language and development environment for different operating systems and more importantly it is more accessible and suitable for children because it was made of simple and limited number of sensors and actuators. Also Thymio supports two different programming interfaces. First one is text based programming language which is called Aseba Event Scripting Language (AESL) and the other is icon based programming interface which is called Visual Programming Language (VPL). This study focuses on the programming interfaces and their effects on the performance of programming education. Therefore, we use text programming and visual programming interfaces to teach programming to students. VPL is a purely icon based programming interface and it does not have keywords. Robots can be programmed using only drag and dropping the required icons on the interface. Students do not need to use keyboard and they are not required to know any technical details. On the other hand, AESL is keyword based text programming language which has syntax rules with different keywords. Basic programming features such as variables, decision mechanism, loops etc. are included in the language. It has been designed as event based language to be compatible with robot events. To be able to use AESL interface, students must know basic English and must understand English keywords. To test the effect of this requirement, in addition to original AESL interface we developed an equivalent Turkish keyword based AESL, which we call TAESL. In TAESL, all the keywords and native functions are replaced with Turkish keywords. To be able to do this, we modify the development environment and robot's firmware. To test our interfaces with the students, we also prepared a curriculum for teaching programming with a robot. We prepare three lessons according to text programming and visual programming interface structures and Thymio's features. First lesson generally covered introductory subjects to programming and development environment. In the second lesson, we focused on the decision mechanism with minor sensors and actuators of Thymio. In the third lesson, we covered complementary elements to robot components and programming features. In addition to new topics, we have reinforced previous topics to reach a better understanding of algorithms as well as introducing new topics. Our teaching strategy as well as the consequent assessment experiments focused on basic programming subjects. We kept the lectures rather simple, so that we can ignore effects of personal skills and cognitive levels of the students. Also the languages, VPL and AESL, do not need long code portions to control the robot; so we avoid using complex algorithms and long code parts. Almost all of the examples consist of a few lines of code. This minimizes the complexity of teaching, thus we ignore the algorithm creation, debugging and testing effects on our evaluation. In order to reduce the effects of lecturer, we prepare in the form of recorded lectures based on video for each lesson. In addition to that, we organize the lectures as one to one training, thus we minimized the effects of the classroom environments in teaching. Each lecture is planned to be a traditional 45-minutes long session. First 20 minutes of the lesson is devoted to the lecture part. In this section, students watch the lecture videos and ask their questions. Next 10 minutes are reserved for practicing with the robot freely. The final 15 minutes section is dedicated to the exam part. In the evaluation phase, collected codes are tested by our team in predesigned evaluation environment which has obstacles and cliffs at necessary locations. Each objective is assessed in the given order and the student is considered successful only if the code produces the expected outcome. In our experiments, 32 students, within ages of 7 to 24 with an average 14.3 and standard deviation 5.25 participated. The majority of the students had no experience with programming before and rest of them had barely some experience on programming. We divided students into 3 different groups, students who studied visual programming (Group 1), students who studied text programming (Group 2) and students who studied text programming after visual programming (Group 3). According to the results of experiments, students who studied with visual programming have been more successful than students who studied only text programming. This is an expected result because visual programming is faster and simpler to program. So, it can be said that the visual programming languages are better for learning programming. When we compare the performance of the students who studied only text programming, with the performance of the students who studied visual programming first and then text programming, we observed that students who studied text programming after visual programming have been more successful than students who studied text programming only. This comparison shows that learning programming with visual languages helps to learn text based programming languages later. Another performance criterion in our tests is the time spent on each interface. While time spent on the visual programming interface is always less than the others because of the simplicity of the visual programming, the effects of the visual programming languages can be seen on the time spend for programming on between text programming after visual programming and just text programming interfaces. According to these result we can say that learning programming with visual languages helps to adapt text based programming faster. We also compared the results of the students who study with original text programming to the results of the students who study with TAESL. In our test results there is no significant difference between the success rates of original AESL and TAESL. Average success rates is nearly identical in original AESL and TAESL. In conclusion, we used different programming interfaces to teach programming using robots and compared the results of the experiments. According to result of our experiments, visual programming languages are found to be easier and quicker to learn. In the experiments, using visual programming students had higher success rates overall. In addition to that, when learning text programming, if students have visual programming background before, they had more successful results on learning text programming concepts. So, we verified that if visual programming languages are used for learning programming first time, students can adapt to other programming languages easier. Also, according to our experimental results, using a mother-tongue based programming language does not have any effect on the learning of programming.
Sahne analizi için ses kaynağı tespiti
Ses işaretlerini yorumlayarak nesneleri ve olayları tanıma kabiliyeti insanın önemli yeteneklerinden biridir. Bu sayede insanlar bulundukları ortam hakkındaki farkındalıklarını artırarak, karar verme aşamasında bu bilgiler ışığında daha doğru hareketlere yönelirler. Örnek olarak, kapalı bir kutu içerisinden gelen cam sesi, kişiyi o kutuyu taşıma esnasında daha dikkatli olmaya sevk eder. Robotlarda ise nesneleri ve olayları çıkardıkları seslere göre sınıflandırma son derece önemli ancak oldukça zor bir görevdir. Bu tez çalışmasında, robotların ortamda tespit ettikleri sesin özelliklerini kullanarak nesneleri ve olayları otomatik olarak sınıflayabilecekleri bir sistem kapsamında ses verilerinin ayırt edici özelliklerini kullanan farklı gözeticili öğrenme yöntemleri kıyaslanmıştır. Çalışmalar kapsamında geliştirilen sistem temel olarak üç aşamadan meydana gelmektedir. Bunlar; önişleme, öznitelik çıkarma ve sınıflandırmadır. Önişleme, sisteme mikrofon aracılığıyla aktarılan ses sinyalinin öznitelik çıkarma öncesinde gerekli eşikler ve dönüşümlerden geçtiği bölümdür. Eşikleme ses sinyalinin sistem açısından faydalı kısımlarını almak için yapılır. Eşiklemeden sonra ise ses ayrık Fourier dönüşümüne (DFT) tabi tutularak spektral özniteliklerin çıkarımı için hazırlanır. Öznitelik çıkarma işlemlerinde ise önişlemeden gelen sinyal ve verilerden yararlanarak sesin spektral ve tınısal öznitelikleri ile sınıflandırma algoritmalarına uygun öznitelik vektörleri hesaplanmaktadır. Bu vektörler hem eğitim hem de test aşamalarında aynı şekilde oluşturulmaktadır. Çalışmaların sınıflandırma bölümü eğitim ve test aşamalarından meydana gelmektedir. Sınıflandırma, öncelikle kaynağı bilinen sesleri ile sınıflandırma algoritmasının eğitilerek bir model oluşturması daha sonra ise kaynağı bilinmeyen sesin bu model yardımıyla doğru şekilde sınıflandırılmasından oluşmaktadır. Çalışmalarda destek vektör makinesi (SVM) ve çok katmanlı algılayıcı (MLP) olmak üzere iki adet makine öğrenme algoritması kullanılmıştır. Sistem geliştirme ve seslerin özniteliklerini analiz etme amacıyla tez çalışmaları sırasında bir seri deneyler icra edilmiştir. Yapılan deneyler iki bölümde değerlendirilmektedir. Bunlar; 1) Laboratuvar deneyleri, 2) Robot deneyleri. İlk bölümdeki deneyler daha çok sistemin geliştirilmesi ve sesin öznitelik analizi amacıyla yapılmış, ikinci bölümdeki deneyler ise sistemin robotik sistemler üzerinde çalışma performanslarını ölçmek amacıyla yapılmıştır. Deneylerde kullanılmak üzere bir nesne seti oluşturulmuştur. Nesneler cam, metal, porselen, karton ve plastik gibi farklı malzemelerden imal edilmiş tipik ev eşyalarıdır. Bu nesnelerden ses çıkarmak amacıyla vurma ve düşme olmak üzere iki etki yöntemi kullanılmıştır. Çıkarılan sesler ayrıştırıcı özelliğe sahip öznitelikler açısından incelenmiştir. İncelemeler için seslerin spektral ve tınısal öznitelikleri çıkarılmıştır. Bu özniteliklerden bazıları sınıflandırma algoritmaları için faydalı olurken, bazılarının ya hiç etkisi olmamış ya da olumsuz etkileri olmuştur. Gözeticili öğrenme yöntemlerinin önerilen öznitelikler ile malzemeleri sınıflandırma başarımı analiz edildikten sonra sistemin düşme ve çarpma olaylarını ayırt edebilme becerisi de incelenmiştir. Deneyler için iki farklı robot platformu kullanılmıştır: tekerlekli gezgin bir robot ve yedi serbestlik dereceli bir robot kolu. Önerilen sistem, malzemeleri ve olayları sınıflandırmada yüksek sınıflandırma başarımı ile çalışmaktadır. Ayrıca sistem açık kaynak kodlu platformlar üzerinde geliştirilmeye ve farklı uygulamalarda kullanıma uygun şekilde geliştirilmiştir.
Robotlar için aktif işitsel-görsel insan takibi
In this thesis, a multimodal system is designed in the form of an active audio-visual perception in order to improve the perceptual capability of a robot in a noisy environment. The system running in real-time uses 1) audition modality, 2) a complementary vision modality and 3) motion modality incorporating intelligent behaviors based on the data obtained from both sensory modalities. In this system, the detection, localization and tracking of the speaker are the main tasks of audition and vision modalities independently.
Ses parmakizi kullanılarak reklam tanıma
Akıllı telefonların kullanımı her geçen gün daha da fazla artmaktadır. Artık insanlar günlük hayatlarında birçok faaliyetle eş zamanlı olarak akıllı telefon kullanmaktadırlar. Akıllı telefonun eş zamanlı olarak en çok kullanıldığı faaliyetlerden biri de televizyon izlemektir. Akıllı telefon kullanıcılarının \%84'ü televizyon izlerken eşzamanlı olarak akıllı telefonlarını kullanmaktadırlar. Televizyon izlerken akıllı telefon kullanımına yönlendiren sebeplerden biri de televizyonda izlenen konu ile ilgili daha detaylı bilgiye ulaşmak için telefondan arama yapmaktır. Metinsel olarak arama yapmak zahmetli ve uzun bir işlemdir. Özellikle reklam sektörünün, tanıttıkları ürünle ilgili kullanıcıların daha kolay ve hızlı şekilde bilgiye ulaşması için başka çözümlere ihtiyacı vardır. Çalışmamızda, televizyonda oynayan reklamın, akıllı telefondaki bir uygulamaya dinletilip tanınmasını sağlayacak bir algoritmanın geliştirilmesi amaçlanmıştır. Reklam tanıması için ses parmakizi (audio fingerprinting) yöntemlerinin kullanılmasına karar verilmiştir. Geliştirilecek yöntemin, ses tanıma sistemlerinin sahip olması gereken gürbüzlük, güvenilirlik, veri boyutu küçüklüğü, parçalılık, arama süresi kısalığı ve hesaplama maliyeti küçüklüğü özelliklerine sahip olması hedeflenmiştir. Farklı reklamlarda, şarkılardan farklı olarak, aynı müzik veya konuşma bölümleri geçebilmektedir, bu nedenle güvenilirlik kontrolü aşamasında, müzik tanıma sistemlerinden farklı bir yaklaşım geliştirilmesi amaçlanmıştır. Ses parmakizi olarak, spektrogramdaki zirve noktalarının aralarındaki zaman ve frekans farklarından yararlanılarak üretilen karmalar kullanılmıştır. Gürültü ve sinyal bozulmaları durumlarında bile spektrogramdaki zirve noktaların en azından bir kısmının korunması beklenmektedir. İlgili yöntemin prototip geliştirmesi yapıldıktan sonra deneylerde bazı zayıf yönleri tespit edilmiştir. Genelde, spektrogramdaki zirve noktaların zaman veya frekans yönünde küçük kaymaları nedeniyle ortaya çıkan bu problemlere çözümler geliştirilmiştir. Yöntemin aynı başarım oranlarına daha az veri kullanarak ulaşabilmesi için de katkılar sunulmuştur. Güvenilirlik kontrolü aşamasında da iki eşik değeri parametreli bir çözüm üretilmiştir. Yapısal katkılar tamamlandıktan sonra, yapılan birçok deney ile yöntemin en iyi başarım oranlarını verdiği sistem parametreleri belirlenmiştir. Deneylerde şarkılar ve reklamlardan oluşan 2 deney kümesinin, çeşitli seviyelerde beyaz gürültülü, pembe gürültülü, kahverengi gürültülü, kırpılma uygulanmış, bar ortamı etkisi uygulanmış, canlı kayıt etkisi uygulanmış, akıllı telefon kayıt etkisi uygulanmış, akıllı telefon çalma etkisi uygulanmış alt-deney kümeleriyle, ayrıca bir de İstanbul'daki bir alış veriş merkezinde akıllı telefon ile kaydedilmiş versiyonları kullanılmıştır. Deney sonuçları, anma, kesinlik, kullanılan veri boyutu ve tanıma süresi açılarından değerlendirilmiş, baz alınan yöntemle karşılaştırılmıştır. Bahsedilen bütün kriterlerde baz alınan yönteme göre daha iyi sonuçlar elde edildiği tespit edilmiştir.