PhD Dissertation
My PhD research and dissertation focused on mobile remote presence (MRP) systems and developing technologies to help reduce the control burden for people operating these systems. MRP systems enable remote communication between distally located people in a familiar face-to-face format. These systems combine video conferencing capabilities with mobile robots and allow the operators of these systems (referred to as pilots) autonomy to move around and interact with various people in the distant environment.
Numerous MRP systems currently exist and are being used to enable remote face-to-face like communication in various application domains including medical, eldercare, education and home use. The following video and plot show examples of MRP systems in use today and how the development of these systems have evolved for use in various application over the past 20 years.
My dissertation focused on improving the maneuverability of mobile robot platforms used by MRP systems and developing gaze aware user interfaces and control systems for the pilot operating these systems. An overview of an MRP system and the areas of focus for my dissertation is shown in the figure below.
Examples of various MRP Systems and use cases
Evolution of MRP systems development in various application domains
Overview of an MRP System with dissertation topics highlighted
Holonomic Multi-Ball Locomotion
For the MRP Robot I evaluated the use of ball driven mobility platforms to enhance the maneuverability of mobile ground robots and eliminate the non-holonomic mobility constraints of existing methods of locomotion used by these systems. Spherical wheels have an isotropic surface geometry which is ideal for holonomic ground locomotion as they can be used to enable omnidirectional motion over a wide range of ground terrains.
As part of my work I explored the rich design space for ball driven ground robots and developed a generalized kinematic model that can be used for calculating motion commands for various new ball drive configurations. Prototypes for a number of these configuration were developed and tested. I also worked on developing a novel magnetically coupled ball drive with improved actuation performance that can be used as a drive system for holonomic multi-ball mobility platforms.
Generalized kinematic model diagram for ball driven mobility platforms
Dual ball drive concept
Single ball drive concept
Example ball drive configurations
Magnetically coupled ball drive (MCBD) concept
Prototype of spherical wheel for MCBD concept
Adaptive Gaze Assisted Control
For the user interface I examined the use of eye gaze tracking to determine the pilot's focus of attention and showed how this information can be used to help automate certain tasks that can reduce the control burden for the pilot.
Continuously transmitting real-time video from a mobile system connected to a wireless network can be challenging as the network conditions can changes from location to location. Maintaining a consistent high frame rate video is important for remotely operating an MRP system reliably. Good video resolution and compression quality, while desirable, are less important then maintaining a consistent high frame rate, low latency video for remote operation performance.
I developed an adaptive, multi-resolution, gaze directed video compression algorithm to enable real-time (buffer-less) high frame rate video transmission over variable bandwidth wireless networks. Using eye tracking hardware the algorithm uses the gaze location of the pilot to generate a composite video frame composed of multiple resolution levels of a single frame. By varying the resolution level used, the number of pixels that need to be encoded and transmitted over the wireless network can be dramatically reduced. At the gaze location high resolution is always maintained, while the resolution in the periphery is gradually degraded at high resolution.
During video transmission, a controller continuously monitors the network and makes adjustments to the resolution level and compression quality of each frame that is transmitted with the goal of maintaining a constant video frame rate. After the composite video frame has been transmitted it is reconstructed into a multi-resolution video frame where the original video resolution is maintained at the gaze location and the video resolution is gradually degraded towards the periphery of the image based on the available network bandwidth.
Pixel reduction achievable at various resolution levels
Example of a level 3 resolution composite image used for generating a multi-resolution image
Example of level 3 multi-resolution image reconstructed from the composite image shown above.
Examples of multi-resolution images at various resolution levels and gaze locations
Functional prototype of MRP UI for adaptive gaze assisted control
Flow diagram of adaptive multi-resolution gaze assisted video compression algorithm developed for MRP systems
MRP Robot used for testing UI
In my dissertation I showed how eye tracking can be used to improve the performance of the pilot by using gaze assisted multi-resolution video compression to minimize the frame rate variability associated with operating mobile systems on wireless networks. Eye tracking also has the potential to help enable gaze cueing for MRP System. Gaze awareness is one of the areas that MRP systems and other computer mediated visual communication systems fall short when compared to face-to-face communication. Gaze is an important non verbal cue that is lost when using current systems, therefore being able to accurately convey gaze in an MRP system would help to further enrich MRP systems and bring them closer to the level of fidelity of face-to-face communication. Gaze cueing is not a topic I covered in my dissertation, but it is one of the topic that I plan to work on in the future.
Gaze cueing problem for MRP systems and other computer mediated visual communication systems
High level concept models of a robot and UI for an MRP systems that would utilize the technologies developed during my PhD research is shown below.
High level concept model of an MRP system
High Level concept model of UI for MRP system
Links to my dissertation, published papers and patents can be found in the papers and patents page