The Interface that Touches the Mind:
Advancing Beyond Autonomous Vehicles

by Dennis Hong

From the Editor: On Thursday afternoon, July 8, 2010, Dr. Dennis Hong, director of the Robotics and Mechanisms Laboratory at the Virginia Polytechnic Institute, addressed the convention. He is leading the group of Virginia Tech engineering students who are working with the NFB to develop the blind-drivable car. Here is what Dr. Hong said:

Driving—sighted people like me, we do it every day, and we take it for granted. When you need to go to the grocery store, you just get in your car and start driving. You drive to school, pick up your dearly loved children, and then take them to soccer practice. You hit the open road and enjoy a road trip with your friends and family with freedom and joy. Well, in modern society driving is really a necessity. It takes you from point A to point B. It’s a means of getting you to your destination whenever and wherever that may be. At the same time driving is fun and exciting. Some people even consider it an expression of power. However, most important, driving is really about freedom. Driving is really about independence. Unfortunately, however, not everyone has the privilege of driving, mostly because of physical challenges, blindness being one of the reasons affecting people the most.

We want to change this. We want to give the blind the ability to drive. So in 2007 Virginia Tech accepted a challenge proposed by the National Federation of the Blind Jernigan Institute to develop a car that can be driven by the blind safely and independently. It was called the Blind Driver Challenge. As far as I know, we are the only group in the nation that has accepted this call. A lot of people thought we were crazy. Some people still do, and, as a matter of fact, to be honest, half the time I actually think we are crazy myself.

But the real story behind the challenge goes like this. At the time when NFB announced the Blind Driver Challenge, we already had a very active research program in autonomous vehicles at Virginia Tech. For example, we won third place at the DARPA Urban Challenge and won a half-million-dollar award. This competition was about developing a fully autonomous vehicle that can maneuver a sixty-mile course in the urban environment. The vehicle had to obey all the California traffic laws, merge into moving traffic, navigate traffic circles, negotiate intersections, avoid a variety of free-standing obstacles, and even park itself--all with no human intervention. So we thought we could tackle the challenge proposed by the NFB. We had already successfully developed an autonomous vehicle, so we thought, "How hard could it be to develop a car for the blind?" Well, we couldn't have been more wrong. We quickly realized that what the NFB wanted was not a vehicle that could drive a blind person around, but rather a vehicle that a blind person could actually operate by making active decisions.

Realizing this, we had to start from scratch; we had to go back to the drawing board and rethink how we could pull it off. Sometimes we doubted whether it was even possible, but, when we realized the importance of this mission for the blind community and the huge positive impact it could have on society, we understood that the potential for the technologies we'll be developing along the way would have more far-reaching impact than just driving. So in 2008, with thirteen very talented and hard-working, smart undergraduate students and only $3,000 in funding, we started developing our first vehicle for the blind.

Let me tell you a story. I still remember when the folks from the NFB first visited my lab, the Robotics and Mechanisms Laboratory (RoMeLa) at Virginia Tech, and the stupid mistakes and awkward situations I created due to lack of experience and understanding of blindness. At the time I did not know anyone personally who was blind, and I do not have any blind members of my family, so I had all those stereotypes in my head about blind people and erroneous assumptions about blindness.

Though we are good friends now, I have to confess that I felt very uncomfortable when I first met Mark Riccobono, the executive director of the NFB Jernigan Institute. He was the very first blind person that I had a true conversation and interaction with. When he first visited our lab a few years ago, actually I was not at all prepared. I didn't know what to do. We have so many visitors through our lab from high-profile sponsors to K-12 students, so we are always ready and prepared to greet visitors and give tours. However, that day, when Mark came to our lab in the basement of Randolph Hall, he was sharply dressed in a suit and tie as usual. First I tried to grab his arm to guide him. I was giving a tour of our laboratory, and I constantly said, "As you can see," then I quickly remembered and thought to myself, "Ahho, I just said, `As you can see' to this blind man," which then created this awkward pause. I know better now. Then in the afternoon I remembered the movie, Daredevil, with Ben Affleck, this heroic blind guy. I wanted to be polite and actually brushed my teeth twice after lunch, remembering from the movie that blind people have a heightened, almost super-hero-like ability to smell. You might laugh, but that was how I thought of the blind at the time. You would be surprised how many people in our society still think the way I did.

Then, as I and my students constantly met with the folks at the NFB Jernigan Institute, visiting the NFB headquarters in Baltimore overnight, interacting with students from schools for the blind, and working together with blind engineers, we started to learn more about all the misunderstandings about blindness. One of the biggest among them is the ability of the blind to perform at jobs. I have learned and personally witnessed that, contrary to general belief, there are really very few jobs that blind people cannot do well. Throughout my work on this project I've been talking and emailing back and forth with blind people from all over the world and was surprised at the jobs these people have. They range from office managers, farmers, IT specialists to auto mechanics. Some ride horses as a hobby, and I even talked to a hobbyist who is a drag racer. Gradually I understood that with just a little technology the blind can really do almost anything that a sighted person can do.

We need for the rest of society to understand this, and what better way to deliver this message to society than for a blind person to drive a car? At the same time, if this vehicle becomes a reality, which it will, and is available to the general public, the impact on the blind would be huge, opening even more doors to new jobs by providing a safe, independent means of transportation.

What is this car for the blind? How does it work? We don't have a lot of time, and the vehicle is very complicated, so I will give you a very quick overview of how it works. There are three parts to the system: part 1, perception; part 2, computation; and part 3, nonvisual user interfaces. So in the first stage, perception, this vehicle has different kinds of sensors all around it, from laser range-finder sensors to cameras. The laser range-finder sensor shoots out a laser, and, if there is an object in front of it, it bounces back, and a computer measures the time of flight. So, if you know the speed of light, which you do, then you can measure the distance of objects. The laser shoots around, scans the environment, and makes a map around the vehicle. The camera system looks all around the vehicle by use of some very sophisticated computer vision algorithms, to identify and classify objects so that the vehicle knows, oh, a tree is over there; a rock is over here. These are the lanes. A vehicle is to the left, and it's going at such-and-such speed and direction.

The second step is computation. This is a vast amount of data from the sensors, and that sensory information is fed into the computer, and the computer tries to generate a world model. This is essentially a map around the vehicle that the computer can understand. Now the challenge is the third stage--nonvisual user interfaces. How do we move or channel these vast amounts of real-time information to a person driving the vehicle without using vision? This is a challenge, so during the past three or four years we've been working on many, many different types of nonvisual interfaces. I'm sure you've probably heard some about the vibrating vest, the AirPix device, and the glove. By the way, some of my students were here. They left yesterday, but they brought some of the interfaces and did a demonstration. Did anybody have a chance to play with those? [applause] We also brought the vehicle that will eventually become the next generation Blind Driver Challenge vehicle. It is a Ford hybrid Escape, very exciting.

One thing I want to point out is a more philosophical approach: what really is this Blind Driver Challenge vehicle? You know, we already have a fully autonomous vehicle. Is the challenge just to put a blind person in it? Is that the Blind Driver Challenge vehicle? No. We want people actually to drive the vehicle, so we have two types of interfaces. One is called the instructional cue interface. The other is the informational cue interface. Let me explain a little more, because this is a rather important concept. The instructional user interface includes the drive grip, which is like a glove, that has five ring motors on the knuckles and tells you how to turn the steering wheel. With this device the computer makes the decision: turn right, turn left, stop, go, push on the brake. So this information or instruction is transmitted to the driver through these interfaces. We call this the "backseat driver problem." This is not really driving. You're driving, but you are really following orders from a computer. You can call that a "Blind Driver Challenge" vehicle, but it is not our vision.

We started from there. Now we are moving towards the information cue devices. The AirPix is a good example. AirPix is a small tablet-like device that has holes on it and compressed air comes out and forms an image of the map around the vehicle. You put your hand over it and feel, "Oh these are the roads. That's a tree over there; there is a moving vehicle to my right." The computer provides information about the vehicle so that it is you, the driver, who make active decisions. That is the concept.

We started with instruction cues and are moving toward information cues, and the future is great, and it’s looking good. Let me give you a brief timeline. In 2008 we started a feasibility study. We started with this low-cost dune buggy that we bought on eBay for $2,000. We only had $3,000, so we used two-thirds of it. Then we got a bunch of donations of equipment from companies, and then we generated these first-generation interfaces, which included vibrating chairs and vibrating vests, a click-wheel interface, and other things.

In 2009, last year, we had our first successful test run in early summer, and, as Mark Riccobono mentioned, we brought the vehicle to the Youth Slam event at the University of Maryland and had two hundred students from all over the nation who came. Some of them had the chance to drive it, and the experience was tremendous. When I think of it, I get tears in my eyes.

This year we are developing the next-generation vehicle. The red buggy demonstration that we had last year was really a feasibility experiment; it was run in a parking lot. The lanes were defined by red traffic cones. It was a very controlled environment, but now the next-generation vehicle is going to be running on real roads. This is a real car, and this is going to be the real Blind Driver Challenge vehicle. I am very excited about this.

As you have probably heard, this has been all over the news, even internationally. It has been on the cover of several magazines, on TV news, everywhere. I am literally getting hundreds and hundreds of emails, letters, and phone calls from people all over the world. Most of them are positive: "Dr. Hong, this is great. Thanks for doing this." Some of them give us advice and feedback. But from time to time this is a controversial project, and I do get letters, most of them from sighted people, saying, "Dr. Hong, are you out of your mind? We already have teenagers texting while driving, which is dangerous. What do you think you are doing putting blind people on the road?" To be honest with you, it is rather a valid concern, so this is good news, bad news, good news, bad news, but I get a lot of questions from the community saying, "When can I buy this vehicle. When can I drive the vehicle?" Well you will be able to drive the vehicle soon in a test track, in a controlled situation. When will you be able to buy it? That's the bad news. This vehicle will not be a real product for the general consumer until it's proven 100 percent safe, at least as safe as a regular vehicle today. The good news is that I truly believe it can be done. [applause]

Now again, the bad news is that, aside from these technical difficulties, a hurdle which we really can tackle, there are many, many other issues. How is a driver's license going to be issued? How is insurance going to cover this? The social acceptance. A lot of issues need to be addressed; nonetheless, this is a very exciting project. You will actually have a chance to drive this vehicle. January 29 at the Daytona International Raceway, we will have the first sneak peek, public demonstration, and next year at the national convention we will have the full demonstration and it’s very, very exciting.

When I talk to my students who work on this project, I always ask them, how many chances in your lifetime do you have an opportunity to change the world? This is actually that moment, so we are the Virginia Tech Blind Driver Challenge Team, and we expect to see spectacular things coming in the next few years. Thank you. [Applause]