VR and Robotics Team Up to Heal •
From NIH Grant to Primary Care and Neurorehabilitation •
Virtual reality tricks the brain into believing we are somewhere else. That other place can be anything we can imagine, like a fantasy world where we are warriors that fight monsters. We move along a rocky path in the sky, block with a large shield, and swing a heavy sword. We duck under rock bridges and thorny vines. We reach for treasure and save animals with magic. We forget about our pain and limitations because we are someone else, somewhere else, while in the real world we are being held safely by the robotic arm of a unique treadmill called the KineAssist MX.
The KineAssist MX (KAMX), the MX stands for Mobility eXtreme, is sold by Woodway and was engineered by HDT Global. The treadmill has an intelligent robot arm that is designed to provide safety and assistance in walking. The treadmill, unlike traditional exercise equipment, is driven by the user’s movement, not by the machine. The machine, however, can work with the robotic arm to offer resistance and perturbation to safely introduce physical challenges that are not possible on other devices. The arm also provides enough safety to enable a user to be entirely in a virtual world.
The KAMX is controlled by an internal computer system, but the designers had the foresight to include a means by which an external computer could also integrate with the system. A computer or internet router can be connected to the KAMX through it’s ethernet port. When the KAMX is put into “game mode” it looks for what is called an MQTT broker at a specific IP address. MQTT is a standard messaging protocol for the Internet of Things (IoT) that provides an efficient way for multiple devices to have bidirectional communications. Through MQTT an external computer system, such as a VR headset, can read the walk speed, get steering values for the arm, and even tell the KAMX to set resistance so it feels like the user is walking uphill or downhill. How many treadmills can do that?
The History
In 2017, the NIH awarded an SBIR grant for a study proposal with the hefty title, Mixed Reality Gaming Combined With a Novel Treadmill for Low Back Pain. One of the results of the study was a VR game called Lucid. Lucid is a sword and shield fantasy game that was designed to work with the KAMX and an HTC Vive VR system. The study was designed and conducted though the University of Alabama at Birmingham (UAB) by Dr. Rebecca Hennessy PhD, Dr. Deanna Rumble PhD, Mike Christian BS, David A Brown PT, PhD, FAPTA, and Zina Trost PhD. The group came from a variety of backgrounds, such as rehab science, physical therapy, engineering, and psychology. The study was published in a paper, with an even more impressive title, called A Graded Exposure, Locomotion-Enabled, Virtual Reality Application during Walking and Reaching for Individuals with Chronic Low Back Pain: Cohort Gaming Design.
When the study was conducted in 2018, the best VR solution was the HTC Vive VR system because it included external hardware, called Vive Base Stations, that could track the position of the VR headset and controllers. Plus, it had the horsepower to drive quality, high-frame rate visuals through a PC with an advanced graphics card. This PC could also run the MQTT broker software and communicate directly with the KAMX via an ethernet cable.
The VR system was great for a study but it had some major challenges for commercial use, primarily centered around the cables and the technical friction of getting started. The headset had a cable leading to the PC. The PC, in addition to normal cables and power cords, had an ethernet cable running to the KAMX. The base stations, two of them, were on tripods around the KAMX and also had power cords. The PC had to be started. The SteamVR software had to run. Sometimes it did not due to updates. The MQTT broker had to be running. The headset, controllers, and base towers had to connect to the PC via SteamVR. If someone bumped one of the tripods then the whole system had to be put through a calibration process with the headset centered on the treadmill. What average physical therapist, or really any human with a full-time job, has time to remember and deal with all of that if there are issues? Every piece of software and hardware needed to run a system can be a failure point and it is what we call technical friction, the resistance to getting a system working well and consistently. High technical friction can be a huge barrier against use.
Mobile VR and Raspberry Pi
Fast forward to 2021 and mobile VR systems. A mobile VR system, such as the Oculus Quest, Vive Focus, or the Neo Pico are self-contained VR hardware. The systems consist of just the headset and the controllers, no external PC or advanced graphic card is needed. Tracking is inside out, meaning it does not require external hardware to track position in a room and there are no cables to get in the way. The headsets available in 2021 are also very capable when it comes to rendering graphics. Though not nearly as powerful as PC based systems with high-end graphics cards, they were powerful enough to get the job done in the hands of our skilled software developers.
Porting Lucid to a mobile VR headset would have taken only a few days if the software had been developed using From the Future’s DRIVE platform. DRIVE stands for Distributed Real Time Intelligent Virtual Environment. One of the features of DRIVE is a VR abstraction layer that supports all popular VR headsets. Lucid was created before DRIVE, so sadly this was not the case. Instead, we ported Lucid to DRIVE, replacing all Vive-specific SDK calls with DRIVE’s abstraction layer. This effort took about four weeks for one programmer. Another two weeks of art time was needed to optimize the art and to change the UI to be more suitable for an in-game, mobile user. Previously, the UI was controlled externally on a PC monitor.
The FTF team chose the Oculus Quest 2 for the first implementation of Lucid, though it could have been the Pico or the Focus. This was mainly because a great deal of testing had already been done with the Quest 2. This was the quickest way to deliver. Plus, the app looked and ran great on the Quest 2.
The other half of the hardware equation was communication from Lucid, in the Quest 2, to the KAMX treadmill. The KAMX requires an ethernet connection. The Quest 2 has WiFI capabilities, so a router was needed. What was also required was an MQTT broker to facilitate the communications between Lucid and the KAMX. The broker used to run on the PC that controlled the Vive VR hardware. Either the broker could be ported to the Quest 2 headset or needed to run somewhere else. Since the headset needed to keep operating at a high frame rate for Lucid, and there were some unknown as to how well an MQTT broker would run in the background on the headset, we chose a different solution.
The primary goal of porting to mobile VR was to reduce tech friction. The last thing the team wanted to do was make the system on location any more complicated than absolutely necessary. With this in mind, we decided to use a Raspberry Pi 4 to run the MQTT broker. The Pi is small and designed to work with IoT devices, hence a version of the MQTT broker we were already using could run on the system. We found a small router that had an ethernet connection and WiFi support and put it in a small container with the Pi. The router connected directly into the KAMX via an ethernet cable. The Pi and the Quest 2 connected to the router through WiFi. The Pi was given a static IP address that the KAMX and Lucid, on the Quest 2, would both be looking for. It sounds complicated, and it is to some degree, but only on the development side. On the user side, the box can be connected to the KAMX and be left on and forgotten. Or a single button on the side can power it all on and off. Once turned on, the system will automatically boot and be ready for operation.
Physical and Neuro Rehabilitation
The KAMX and the Lucid VR game are installed in two rehabilitation facilities with a third on the way. These facilities are the UTMB Primary Care Pavilion (PCP) in Galveston, Texas and the Moody Neurorehabilitation Institute with a location also in Galveston and one in Lubbock, Texas. PCP’s use of the KAMX will focus on treatment for physical issues related to injuries or musculoskeletal disorders involving back, hip, and knee pain. The Moody center’s use of the robotic treadmill will be for people with brain injury that have difficulties walking. In all uses, the KAMX’s robotic arm safely and intelligently holds a patient upright as they move to improve their mobility.
When you have an injury or physical disability that affects your walking it can be frightening to engage in rehabilitation. Physical therapy’s purpose is to regain and restore pain-free and comfortable movement. In order to accomplish that, the therapy will take a patient right up to the edge and even cross their comfort zone. Sometimes patients resist doing what they need to out of fear of pain. The KAMX can be intimidating at first. Despite its safety features, it is new and must earn the trust of patients before they feel completely comfortable using it. Even then, they may be reluctant to move at the level they need to for recovery. This is where the VR system comes in.
The VR game, titled Lucid, turns a patient into a monster fighting warrior. Not only that, the battles take place on earthen walkways in the sky with no visible terrain below. When they move on the treadmill in the real world they move an equal distance along a sky-path in Lucid. Before long, they have a frogman, flying one-eyed beast, or giant rock golem to do battle with. With a shield in one hand to block projectiles and a sword in the other to strike down the monsters, the patient is quickly engaged in their new role in this new world. The patient is now doing something fun and possibly minimizing or even forgetting about their pain while they do the physical movements necessary for their journey to recovery.
The game, in combination with the capabilities of the KAMX treadmill, provides a broad range of movement challenges. At the simplest level, the patient walks at their own pace and engages with the monsters as little or as much as they can. There is no game death from injury and no possibility of falling off the sky path. Simply holding the shield and sword will get them past adversaries. On the other extreme, the patient can run and swing their sword as hard as they can to send monsters flying. Some levels have features that the patient can duck under while others encourage a patient to reach for food and treasure. One level encourages speed to rescue animals with a magical shield. The combination of robotic intelligence and VR game provides safety and encouragement while also enabling a broad range of physical skill from simply moving to athletic running while ducking, reaching, and swinging.
The future combined powers of VR and robotics has the potential to profoundly change the face of physical and neuro rehabilitation.
Body and mind need to work together for success. VR and robotics can assist and enhance both. The KAMX treadmill alone has a lot of potential that can be explored in VR. Resistance to simulate going up hill and assistance to simulate going downhill are possible. Perturbance, the robotic arm basically shaking you (safely of course) could be used to simulate walking through a stream or in an extreme case, being attacked by a bear! Imagine being in an earthquake and trying to walk to safety. It requires some physical skill, a skill that is valuable in later stages of rehabilitation. The machine could even simulate flying. How would flying be useful in physical therapy? We don’t know yet, but wouldn’t it be interesting to find out?