Research Projects

Metrics for Assessing Drone Piloting in VR

To understand motor-cognitive learning in virtual reality, we are conducting a study where participants learn how to pilot a drone using a real-world remote drone controller. The objective of the study is to capture performance and biomarkers for learning across distinct modules and training outcomes. The study was designed after reviewing NIST’s Standard Test Methods for Small Unmanned Aircraft Systems, and it consists of four training and one evaluation module. The participants complete these modules on three consecutive days. From the preliminary data, we are able to predict the users' performance in the evaluation module from their training modules' data with an accuracy ranging from 93.33% to 100%, based on the number of training modules used in the ML analysis. (in-progress)

Effectiveness of Augmented Reality (AR) Adaptation on AR triaging tasks

We are currently working on an AR learning study, where the participants learn how to triage patients in a Mass Casualty Incident (MCI) using a physical and a virtual AR triage tag. The objective of this study is to extract markers for human sensorimotor learning in AR. Participants (firefighters and others) in this study will learn how to use an AR headset and complete a virtual and triaging exercise through multiple training levels. We will monitor the participant’s physiological and cognitive states using wearable ECG, EDA, Eye tracker, and fNIRS device, and extract physiological, neural and behavior (PNB) markers for learning how to interact in the AR environment and how to perform the triage task. (in-progress)

Track the Learning Curve of Adapting to Exoskeletons

We are conducting a study to track user sensorimotor behaviors when using a shoulder exoskeleton across tasks with varying workload demands. We hypothesize (1) that task performance under exoskeleton use will exhibit unique behaviors, e.g., how users adapt to the kinematics of the exoskeleton or changes in their proprioception. Markers of learning captured in this activity will inform the development of adaptive models and associated training strategies for the patient handling exercises. (in-progress)

Brain Activity-Based Metrics for Assessing Learning States in VR under Stress among Firefighters: An Explorative Machine Learning Approach in Neuroergonomics

The nature of firefighters’ duties requires them to work for long periods under unfavorable conditions. To perform their jobs effectively, they are required to endure long hours of extensive, stressful training. Creating such training environments is very expensive and it is difficult to guarantee trainees’ safety. In this study, firefighters are trained in a virtual environment that includes virtual perturbations such as fires, alarms, and smoke. The objective of this paper is to use machine learning methods to discern encoding and retrieval states in firefighters during a visuospatial episodic memory task and explore which regions of the brain provide suitable signals to solve this classification problem.

Task Engagement as Personalization Feedback for Socially-Assistive Robots and Cognitive Training

Socially-Assistive Robotics (SAR) has been extensively used for a variety of applications, including educational assistants, exercise coaches, and training task instructors. The main goal of such systems is to provide a personalized and tailored session that matches user abilities and needs. While objective measures (e.g., task performance) can be used to adjust task parameters (e.g., task difficulty), towards personalization, it is essential that such systems also monitor task engagement to personalize their training strategies and maximize the effects of the training session. We designed an Interactive Reinforcement Learning (IRL) framework that combines explicit feedback (task performance) with implicit human-generated feedback (task engagement) to achieve an efficient personalization.

9PM: A Novel Interactive 9-Peg Board for Cognitive and Physical Assessment

Cognitive assessments are a crucial part of rehabilitation in persons with a neurological disorder and vocational rehabilitation, where people need to be trained to improve their cognitive abilities. While human action involves using several cognitive skills and physical skills, most assessment systems focus on detecting or assessing either the cognitive ability or just physical ability. There is a need for a system that bridges the gap between real-world activity, which involves physical activity and cognition, and clinical tests that are tailored for a specific use. To address this need, we designed a novel interactive 9-Hole Pegboard called the 9-Peg Move (9PM) capable of performing both cognitive and physical assessments in the same system. The system incorporates wearable sensors to collect data for objective evaluation.

Adaptive Robotic Rehabilitation using Muscle Fatigue as a Trigger

Fatigue is a pervasive symptom following brain injury or disease. It has been known to impact recovery and hence is an important factor in rehabilitation. Robotic rehabilitation may be one way to reduce fatigue because of the robot's capability to adapt to the user's performance. This paper explores an adaptive rehabilitation system to provide personalized upper limb rehabilitation. The system collects EMG data from the major muscles responsible for movement and adapts the forces used for rehabilitation (assistive and resistive) in real-time based on muscle fatigue. Experimental results and the user survey outcomes show that the system was able to detect the onset of fatigue within ±10 seconds error margin. Overall, it was found that the subjects experienced lower fatigue and had a higher probability of compliance and engagement with the proposed robotic rehabilitation system.

User Skill Assessment using Informative Interfaces for Personalized Robot-Assisted Training

Robot-Assisted Training (RAT) systems have been successfully deployed to assist with a training task, promoting an efficient interaction with the user. Personalization can improve the efficiency of the interaction and thus enhance the effects of the training session. Personalization can be achieved through user skill assessment to choose an appropriate robot behavior that matches user abilities and needs. Graphical User Interfaces have been used to enable human supervisors to control robots and guide the interaction in RAT-based systems. This work focuses on how such interfaces can be used to enable human supervisor users (e.g., therapists) to assess user skills during a robot-based cognitive task. In this study, we investigate how different visualization features affect decision making and efficiency, towards the design of an intelligent and informative interface.

APSEN: Pre-screening Tool for Sleep Apnea in a Home Environment

APSEN system is a pre-screening tool to detect sleep apnea in a home environment and track user’s sleeping posture. The system was designed and evaluated in two parts: apnea detection using SpO2 and posture detection using IR images. The two parts can work together or independently. During the preliminary study, the apnea detection algorithm was evaluated using an online database, and the right algorithms for detecting the sleep posture were determined. In the overnight study, both of the subsystems were tested on 10 subjects. The average accuracy for the apnea detection algorithm was 71.51% for apnea conditions, and 98.68% for normal conditions. For the posture detection algorithms, during the overnight study, the average accuracies are 74.91% and 89.71% for SVM and CNN, respectively. The results represented in the paper indicate that the APSEN system could be used to detect apnea and postural apnea in a home environment.

VoTrE: A Vocational Training and Evaluation System to Compare Training Approaches for the Workplace

Extensive research has been carried out in using computer-based techniques to train and prepare workers for various industry positions. Most of this research focuses on how to best enable the workers to perform a type of task safely and efficiently. Many of the accidents in manufacturing and construction environments are due to the lack of proper training needed for employees. In this study, we compare the impact of three types of training approaches on the planning and problem-solving abilities of a trainee while he/she performs the Towers of Hanoi (TOH) task. The three approaches are (a) traditional (with a human trainer), (b) gamification (game-based training simulation), and (c) computer-aided training. This study aims to evaluate a worker’s level of functioning and problem-solving skills based on a specific training approach. The exact assessment of functional capacities is an important prerequisite to ensure effective and personalized training. The study uses workplace simulation to collect different types of performance data and assess the impact of these training approaches.

Multimodal Analysis of Serious Games for Cognitive and Physiological Assessment

Serious games refer to virtual games used for training, simulation, or education and can engage users in cognitive and physical tasks. The design of serious games may offer insights into users' cognitive and physical behaviors while they try to accomplish structured tasks. In this study, we concentrate on the design principles of serious games that can be used for assessment, which we employed for our design. To test our prototype, we experimented with control participants. Results from surveys, our collected game features, and sensor outputs were compared and analyzed with hypotheses based on previous research studies. Finally, we interpret the results of our experiment, and we describe issues and real-life uncertainties that associate with sensor errors.

Brain-EE: Brain Enjoyment Evaluation using Commercial EEG Headband

Previous EEG studies have mainly been experimentally driven projects to help identify and elucidate our understanding of many neuroscientific, cognitive, and clinical issues (e.g., sleep, seizures, memory). However, advances in technology have made EEG more accessible to the population. This opens up lines for EEG to provide more information about brain activity in everyday life, rather than in a laboratory setting. To take advantage of the technological advances that have allowed for this, we introduce the Brain-EE system, a method for evaluating user engaged enjoyment that uses a commercially available EEG tool (Muse). During testing, fifteen participants engaged in two tasks (playing two different video games via tablet), and their EEG data were recorded. The Brain-EE system supported much of the previous literature on enjoyment; increases in frontal theta activity strongly and reliably predicted which game each participant preferred. We hope to develop the Brain-EE system further to contribute to a wide variety of applications (e.g., usability testing, clinical or experimental applications, evaluation methods, etc.).