American Action Fund for Blind Children and Adults
Future Reflections Special Issue on Tactile Fluency INNOVATIONS
Semiconductors, Streetscapes, and Learning Through Touch
A Conversation with Josh Miele
From the Editor: Josh Miele is a longtime advocate of tactile access for the blind, and he has channeled his energy and creativity into a variety of innovative projects. In this interview he discussed his work and his ideas about the importance of tactile exploration.
Deborah Kent Stein: When I started to plan a special issue of Future Reflections around the theme of tactile fluency, you are one of the first people I decided to contact. I know that this is an important area for you and one where you seem to be very comfortable.
Josh Miele: A lot of my work has been around using tactile perception to present information, especially spatial information that is very hard to convey auditorially. As a designer and appreciator of art, I also think a lot about the aesthetics of tactile design—quite literally, the feel of things. My work has had tactile aspects from the very beginning.
One of my very first projects involved developing a program called Outspoken for Windows for a company called Berkeley Systems. I was in charge of documentation, and I developed a set of tactile graphics to represent Windows concepts. At that time hardly anybody in the blind community knew what Windows was. Menus, dialog boxes, buttons, combo boxes—they were completely new ideas for the blind world. I developed a set of tactile graphics that went along with our documentation. They illustrated what it means when a window is on top of another window, or when there are elements in a dialog box. They showed what it looks like when a menu pops down from the menu bar. The response was amazing! We had people calling up who didn't want to buy the software, they just wanted the tactile graphics for Windows.
DKS: Did your parents and teachers encourage you to explore through touch? Did they help you develop an interest in the tactile world?
JM: Absolutely! From very early on! My mother was an artist, and her main medium was collage. It's a very tactile medium. We had tons of materials around—different textures of paper, and scissors and razor blades everywhere. It was completely routine to cut things out and glue them on top of each other to create tactile representations.
My father is an architect, and I would go and play in his office. When you're a little kid and your dad is an architect, you play at drafting floor plans. In his office there were rolls of tape called layout tape. It came in different widths and different colors. I would make floor plans with this tape. I could feel the lines I was making, and because the tape had different colors, it was perceivable to the sighted people around me, too. I remember playing that way when I was six or seven.
I could see until I was almost five. I wasn't very interested in art when I had sight, and I wasn't especially skilled. But later, after I became blind, I got interested in drawing things for other people so that they could see what I was talking about.
When I was eight or nine I had an amazing art teacher who really got it. While the other kids were busy making drawings, she gave me tools for cutting linoleum. They were sharp tools—maybe the school administration didn't know about it! She gave me scratch paper, which is a heavy black oaktag with a white veneer. When you scratch the paper, you create black lines that can be felt as well as seen. She gave me lots of different types of sandpaper to use for making collages. It was great! It's amazing what you can do with high expectations! Her name was Pat Gazer, and she really was a remarkable teacher. It wasn't because she knew anything about blindness; she just accommodated the student that I was. I am forever grateful to her!
Another person who was a very important influence was my itinerant TVI (teacher of the visually impaired), Joan Smith. She was my TVI from the time I was in third grade until I graduated from high school. She did all of my Braille transcription, including tactile graphics that went along with my books. She did a great job! She hadn't studied higher math, but she transcribed all of my books for geometry and trigonometry and calculus. Can you imagine having to learn geometry without tactile graphics? She was very conscientious about making sure I had all the graphics I needed! Through her I learned how important tactile graphics are.
From the time I was in third grade, there was a bookshelf in my classroom filled with Braille books. In middle school I had the Webster Dictionary; it took up an entire wall! That 72-volume dictionary was part of the furniture of the classroom, and I could browse through it whenever I had free time.
Another book that was on my shelves in middle school was a basic primer on tactile maps. It was full of maps of imaginary places that illustrated map concepts. For instance, there was a map about a family vacation. It showed where the cabin was, where the pine trees were, where the stream was. It showed the road that went across the bridge. I spent a lot of time with that little book of maps! I loved it! But I wished I had access to real maps. Imaginary maps were cool, but I sure wished I had a map of the area around my school or a map of the area around my house. That longing led me to develop the TMAP Project later on.
DKS: Explain a bit about the TMAP Project. What is it? How does it work?
JM: TMAP is a website that takes geographical information from a database of streets and roads and walking paths and buildings. It turns that information into formatted tactile street maps with Braille labels. It's a very simple concept. It's an automated way of making a tactile street map of any place you want, any place in the world. It uses free street map data from openstreetmap.org. It's a way for blind people to get street maps of any place they want.
You can't really describe a street network and hold the whole thing in your mind. A street map is a unique and irreplaceable experience. To follow routes on a tactile street map is like wandering around in the real world. You can build in your mind a cognitive map of the physical world that you can't build from a description.
I had always envied sighted people their ability to look at a street map and learn the layout of an area where they had never been. I wanted that for myself and for my blind friends. I developed TMAP at Smith-Kettlewell Eye Research Institute as part of my early postdoc work in about 2005.
Most of the maps that were available were visual maps. Visual maps are very different from tactile maps. On a tactile map you want very different kinds of information. To make a tactile map, you can't go directly from a visual map to a tactile one. You want to go from the underlying data. You need to avoid a lot of extra clutter, a lot of extraneous features. The map should be very clean. The more dense the features are on the map, the harder it is to interpret. Tactile graphics need to be designed so you can extract the important information. In designing a good tactile street map, all you want are streets. You can add a few other things such as particular buildings that you want to represent, but you can't put all of the buildings on there.
DKS: Can you show more if you make the map bigger?
JM: Yes, but if you make it bigger, you lose the sense of an overview. When you zoom in you can include a lot more detail. What you include depends on the scale and the size of the paper.
I have designed maps for the BART system, Bay Area Rapid Transit in California. Those maps work with a Smart Pen, and they include much more information. There's a set of maps for each of the forty-three BART stations. There are three views of each station. One is the streetscape around the station, showing the entrances and exits. One is the concourse, the area where the faregates and the ticket machines are. Then there's a map of the platform level. On the map of the street level we include much more than the streets. But you need to keep the feature density below a certain threshold or it becomes too confusing, even for a very good map reader. It's not so much about skill, it's about the fundamental aspects of tactile discrimination.
DKS: How does the Smart Pen work?
JM: The Smart Pen is a mainstream product that was developed as a tool for college students and reporters. The pen has a little computer inside it and a camera in the tip. It also has the ability to do audio recording. You take notes or draw on special paper in a special notebook. You can record at the same time you're taking written notes with the pen. Embedded in the background of each page in the notebook, sort of like a watermark, is a very high-density pattern of dots. Every square millimeter of every page is unique. There is a different dot pattern at every spot on every page. The camera sees those dots as you're writing. So when the pen touches the top righthand corner on page 52, it knows that that is different from any other spot. It knows you're at page 52 on the top righthand corner.
The pen was designed to let students take notes while they recorded a lecture or an interview. When you go back and review your notes, you can tap that part of the notebook with the pen and it jumps to the spot in the recording where you were when you took that note. The pen is a way of instantly connecting your visual notes to an audio recording. As I said, it's a mainstream product that you can buy at Target or on Amazon.
We designed a bunch of tactile graphics and programmed the pen. When you touch the pen to a page with a tactile graphic on it, the pen will give you information about the part of the tactile graphic you're touching. It's a portable, inexpensive way of creating audio/tactile graphics.
My research looked at ways to use the Smart Pen as a platform for audio/tactile graphics. Other organizations such as Touch Graphics and the American Printing House for the Blind (APH) and the San Francisco LightHouse collaborated with me to use this idea and created audio/tactile materials of various types.
DKS: Did you have to work out any copyright issues with the developers of the Smart Pen?
JM: There were some, but they were fairly straightforward. We were buying the pen and using it, and they were happy to have a little secondary market.
Incidentally, a set of the BART maps of the forty-three stations is at the Cooper-Hewitt Museum of Design. It's an example of universal design accessibility. So I have a piece in the Cooper-Hewitt! It makes me happy.
DKS: Where can someone get the BART maps? Can you buy them? Do you have to look for them at a BART station?
JM: You can buy them from the San Francisco LightHouse, along with the pen.
DKS: To go back a little bit, what did you study as an undergrad student?
JM: I studied physics as an undergrad. In grad school I studied something called psychoacoustics, which is about how hearing works.
DKS: Those were areas that must have used a lot of charts and diagrams.
JM: Yes, and I no longer had a dedicated TVI. I taught my readers to create graphics for me using Sewell paper [light plastic sheets used with the Sewell Raised Line Drawing Kit available from APH). It's a nice quick way of doing drawings. As an undergraduate I found that there wasn't a lot of variety to the diagrams I needed. Once you figure out the general pattern, it's just a matter of labeling. For tests I would draw diagrams on the Sewell paper, and my reader would copy my drawing into the bluebook. When I worked on semiconductor circuits, I had my reader draw circuit diagrams on Sewell paper.
Sewell paper is awful stuff to keep long-term. I would have to tape each sheet to a sheet of regular paper. Nowadays I would probably use swell paper instead. Sewell paper is nice, though, because you can draw on it with a regular pen.
Once I started working with experiments and data in grad school, I needed ways to do my data analysis and data representation. I had to create software to handle my graphics. My colleagues were using a program called Matlab from a company called Mathworks. Basically Matlab is like Excel on steroids for scientists and engineers. It allows you to visualize lots of numbers at once by putting them into graphs and images. There was no way for me to see the graphs that I was making, so I created a program that would take the data from Matlab and turn it into tactile graphics that I could print out on my Braille embosser. It became really easy for me to create scatter plots or bar graphs or other sorts of data representations. I used that same software when I created TMAP. Instead of plotting perception characteristics from my research data, I was plotting street coordinates.
DKS: After you got your PhD in psychoacoustics, what did you do for your postdoc work?
JM: Once I started working with circuits and semiconductors I started looking for information about how blind people did such things. All roads led to Smith-Kettlewell. I met the folks there and acquired enough knowhow from them to complete my undergraduate semiconductors course. I reconnected with them in grad school because it seemed like a good thing to do. They were very excited about the work I was doing. I got a fellowship there while I was still in grad school, and when I finished my PhD, I immediately started a postdoc there. Eventually I transitioned to being a scientist there. It was a very lucky situation.
DKS: How long were you there?
JM: I finished my degree in 2003, and I stayed at Smith-Kettlewell until the beginning of this year, so I was there for about sixteen years altogether.
DKS: Did you work on any projects that had to do with tactile learning and creating tactile access?
JM: I did the Smart Pen stuff while I was at Smith-Kettlewell, and also the TMAPs. I also worked on some basic research about tactile exploration, one hand vs. two hands, one finger vs. multiple fingers. I did that research with a brilliant young scientist named Val Morash. I've always been most interested in applied science, but she was more interested in the fundamentals of tactile perception.
There's an awful lot of bad science in the history of blindness research. One of the myths that has been around for a long time is that blind people really can't understand tactile graphics.
DKS: I've always thought a lot of that has to do with lack of exposure.
JM: It does. It also has to do with the poor design of a lot of tactile graphics. It has to do with the fact that blind people are actually discouraged from touching things a lot of the time, so they come to feel uncomfortable about tactile exploration, which is an absolute crime!
One of the pieces of bad science in the blindness literature says that you don't get any increased benefit from using more than one finger or from using both hands instead of one hand for tactile exploration. This bad research would have you believe that using one finger is just as good as all the fingers of both hands.
DKS: Really! That's ridiculous!
JM: So Val and I set out to destroy that myth!
DKS: Is Val a blind person?
JM: No, she was a sighted doctoral candidate at Berkeley, and this was part of her dissertation. I was a collaborator and adviser. She did extraordinary work.
One of the problems with a lot of the research that has been done in this area is a lack of rigor. Val was an incredibly brilliant young person, but she very tragically passed away. She would have done some incredibly cool stuff if she had lived.
Val and I set out to debunk the notion that one finger is just as good as multiple fingers for exploration, and she brought her mathematical modeling skill to the project. I brought my knowledge of tactile design and my ideas about the practicalities of tactile exploration. We published two articles in Perception, which is a very well-respected journal. Our research showed that two fingers are better than one finger, and two hands are better than one hand. The first two fingers of both hands add significantly as you bring them into the exploration process. We used a very rigorous mathematical finger tracking model, and we had lots of subjects. Interestingly, when we did the experiments with both blind subjects and sighted subjects under blindfolds, both groups showed the same trends. More fingers and both hands were better for both groups. Our findings not only showed that the blind subjects were better at tactile exploration, but that they were qualitatively different from the sighted subjects in the way they used their hands. They tended to use both hands in parallel, while sighted people tended to use one hand. One of the important lessons from this research is that you can't generalize about blind people based on studies of blindfolded sighted people. That's a mistake that researchers make over and over again.
DKS: One of the things I hear a lot from parents and teachers is, "I'd love to show him things, but he doesn't like to touch things." Do you have any ideas for helping kids get past a reluctance to explore?
JM: I've worked with a lot of kids through the Blind Arduino Project, using computers to design and build things. I notice that sometimes you hand something to a kid and say, "Here, check this out!" and they just hold onto it limply. They don't touch it all over. I think it's a general passivity that kids acquire. They get a lot of messages like, "Stop feeling around, it makes you look blind!" and "Don't touch that! It's not yours!" or "Don't reach over there, you're going to get hurt!" All these messages extinguish the urge to explore and check things out. It's terribly detrimental to the child's capacity to learn about the world.
When I show something to someone, often the first thing I have to teach is not about the object itself, but about how to look at it tactually. I have to say, "Nobody is watching you. Nobody is going to tell you to stop touching. We want you to use your hands to find out as much as you can about this thing." You have to do exercises like that with individuals or groups before you can do anything else. You have to scaffold up. You might start by asking, "What are some of the things you notice about this object?" You'll always have a few kids who are all over it, who find out everything about it in the first three seconds. But half of the kids will just sit there, holding onto it, not exploring at all.
DKS: Do they explore more effectively once you start to build that scaffold?
JM: A lot of them do. But you can't fix everything in a couple of hours or even in a few days. I think it's extremely important to give young blind kids the clear message that they have a right to feel around. There's nothing wrong with touching things, and it's the only way they're going to find stuff out.
Tactile graphics are a great way to teach exploration. You can start building expectations. When you hand the kid a drawing on a piece of paper, you can say, "Nobody's going to yell at you for touching everywhere on this page. You need to feel every square inch of it to know what's on it."
DKS: Do you think a lot of kids' reluctance to explore has to do with our society's negative attitudes about touch? People get very strong lifelong messages that you shouldn't touch things. Things might be dirty, they might hurt you, they aren't yours. I think sighted adults may unconsciously inculcate blind kids with those attitudes. To get blind kids more opportunities to touch things, maybe we have to change the way sighted people feel about touch.
JM: The idea of destigmatizing touch is really complicated! I'm not saying we shouldn't try—but wow! it's a big problem!
I was always given the message that I was entitled to touch pretty much anything I wanted to feel. There were some moments when I was embarrassed to touch things because nobody else was doing it—at museums, for example, when I was the only one allowed to touch things. But the message got through. Having access to all those tactile art materials and tactile graphics, and being encouraged by my family and friends to use touch as a means of expressing myself and learning about the world, that was very powerful. I feel so lucky to have had that kind of upbringing! Spatial information and tactile representations are important to being able to do so many of the things that I love—studying maps, representing scientific data, drawing, creating and enjoying sculpture. I derive so much pleasure from those things, and they give me independence. All of that is missing from the lives of many blind kids. That's one of the places where I hope I can make a difference.
Media Share
