Virtual and augmented reality based interfaces for choreography generation

Choreography is a vital creative process of defining movement. Digital applications go a long way in helping choreographers achieve their tasks in choreography generation. Thus, since the late 1960's researchers have tried to use the computer in the choreographing process. In the early days choreographing applications made use of 2D stick figures to represent characters. However, as computer processing power and graphics improved 3D realistic models became the norm. These advances in technology have also seen different techniques to interact with computing systems emerging, this has been punctuated by the advent of Natural User Interfaces (NUIs), whose interaction models the way humans naturally interact. NUIs remove the reliance on traditional input methods like keyboards and mouse, but rather focuses on gestures, touch and voice. Furthermore, technologies like Augmented Reality (AR) and Virtual Reality (VR) have transformed interface development. AR presents an interesting way to interface design. It allows users to interact with both the real world and the virtual world simultaneously. Whereas, VR immerses the user into a synthetic world. Thus, it is important to embrace these technologies in the choreographing process. In this study, we investigate the effects of different interaction techniques on user experience for a choreography generator. We evaluated different interactive techniques, which are the direct input methods like touch screen and gestures, and indirect methods like the mouse and keyboard. We also evaluated VR and AR as interfaces for choreographing tools. Our research contributes to the study of how different interaction methods of the same application affect user experience. It also extends the evaluation of VR and AR in training applications. Previous research on VR and AR has shown varying results which differ from domain to domain. Therefore, it is imperative to evaluate these two technologies on choreography tools so that stakeholders in choreography know which interface to invest on. For this reason, we develop an AR based choreography generator and a VR-based choreography generator and compare with a Personal Computer (PC) based choreography generator and a touch-based mobile application for choreography generation. The PC based generator uses the keyboard and mouse as the interaction technique, whilst the mobile application utilizes the touch screen. The AR interface utilizes the touch screen and gestures, whereas the VR interface utilizes the touchpad and gestures. These interfaces allow choreographers to complete the same tasks on them. Using the choreography generator, users can add actors and other props into the scene. The applications allow the choreographer to be in charge of the actors just like in the real world, he/she can define movements of the 3D actors. The choreographer can add from a diverse list of characters. The props are essential to decorate the scene or to help emphasize concepts. The interfaces also have interactive items that allow the choreographer to control the speed of the characters. The PC based interface, the touch-based mobile application interface and the virtual reality based interfaces present the same graphics to the user. These three mainly differ in the interaction method. However, the VR interface also gives the user the immersion effect, whereby a user is immersed into the choreographing world and feels like he/she is inside the virtual environment. Since AR allows users to interact with the real world and virtual objects simultaneously, the AR interface graphics differs significantly from the other interfaces, however it offers the same functionality. AR gives the users the power to use real world objects with computer generated imagery. It presents a flexible way which is important especially with interfaces for training and teaching applications. In this research, we evaluate user performance and user experience on the interfaces in terms of task completion times, as well as subjective criteria, such as mental stress, physical stress, pleasure and frustration experienced. The physical stress scores give us an indication of the physical requirements needed to perform a task, whilst mental stress gives us the cognitive load needed to perform the tasks. These criteria help us investigate the four factors of usability: satisfaction, effectiveness, ease of learning and efficiency. We used usability testing methods to evaluate the four interfaces using a total of 20 test subjects. To obtain objective results we recorded the task completion times, whereas to obtain subjective results a post-test questionnaire was used. Furthermore, the think-aloud technique assisted us to understand the user's psychological experience whilst using the interface. The combination of time it takes to complete tasks, mental load and physical load gives a fair understanding of the efficiency and effectiveness of the interfaces. The experiments were conducted in a controlled environment that was the same for all users. Furthermore, the users used the interfaces randomly to avoid task adaptation. Participants were given four tasks to complete on the interfaces. Firstly, they had to add two dancers to the left and right of the scene. After that they had to add props to the scene. After adding the dancers and props users had to define the path for the dancers to follow. Finally, they had to play the defined choreography. Whilst playing the choreography users were encouraged to explore the system further. This included changing zooming levels and viewing angles. The results show that touch-based approaches produce faster task completion rates. This was attributed to the naturality of using the approach, especially when defining the path using the finger. The results also demonstrated that NUIs are an effective way to enhance user experience. They give users an easier and faster way to complete tasks using intuitive methods. Furthermore, this approach does not require pre-training as users can use their intuition to understand how to interact. The AR interface's ability to zoom in or out and change viewing angles by only moving the mobile device around the marker, gave users a natural approach of performing the zooming task, which is analogous to moving near or further from an object in real life. This gives users a compelling effect which enhances user experience. We also noted that pointer based approaches can be very effective depending on the tasks. Especially with tasks that require a small area of contact to interact the pointer proves to be a good option as it provides precise pointing. The results also verify the effectiveness of AR in developing training and design applications. In the choreographing process AR has an additional advantage as both computer generated imagery and physical objects can be used to emphasize concepts. Therefore, AR gives the choreographer flexibility allowing him/her to define his/her own stage in a way that is necessary for a particular task. Furthermore, the results showed that AR gives users excitement and pleasure which comes from the curiosity it creates. The major downside of mobile AR is the physical stress it exerts on users, which is due to hand movements as the user interacts with the interface. Another downside of AR is due to the errors that arise as the mobile device tilts whilst the user is performing the tasks. This is due to the natural hand tremor. Although VR gives pleasure to users, its high tasks completion rates, and high physical stress scores negatively affect its efficiency and effectiveness in choreography. This adversely impacts its overall usability scores. Therefore, most of the users preferred the AR interface as compared to the three other interfaces. We conclude by proposing an AR interface that utilizes virtual buttons and voice commands to interact. In this way, we can achieve a system that completely embraces the natural user interface approach. We strongly suggest that choreographers should invest in AR, as it is flexible and allows them to use both computer generated imagery and physical objects to illustrate concepts.