Two Milestones Crossed at AIM Conference on Automated Nemeth Braille Translation

by Al Maneki

From the Editor: Al Maneki is a tireless advocate for blind people who want to work in Science, Technology, Engineering and Mathematics. He knows that it takes a good deal of initiative and desire to excel in these fields, and he also knows it takes good textual materials, ones that come as close to giving the blind reader what his or her colleague using print gets. Dr. Maneki isn’t content simply to observe that the problem of converting complicated mathematics into Braille is difficult. He is determined to do something about it, and he knows the work cannot be done alone. So part of his work as a pioneer in this field of using digital technology to bring meaningful mathematical reading materials to the blind has been to gather smart, committed colleagues to work with him and the organized blind movement of the National Federation of the Blind. Here is what Al has to say about the latest workshop and progress made on creating a system to automatically convert mathematics represented in print into Braille:

Author’s note: I want to begin this article by acknowledging the valuable assistance of fellow team member Karen Herstein, who verified most of the facts and citations given here. I want to thank all of the participants of the AIM workshop on Automated Nemeth Braille Translation for reviewing the first draft and making suggestions and corrections. In writing this article, I relied heavily on the workshop summary that was prepared by Alexei Kolesnikov. As the author of this article, I assume full responsibility for any errors, misrepresentations, and oversights. Readers who wish to contact me may do so by email at [email protected]. – Al Maneki

Introduction

In 2019 the Automated Nemeth Translation Team (ANTT) applied to hold a workshop at the American Institute of Mathematics (AIM). AIM hosts workshops in all areas of mathematics, including topics in mathematics education and accessibility in mathematics. AIM, one of six institutes sponsored by the National Science Foundation, fully funds all costs associated with the workshop. The goal of our team’s workshop was to bring together software developers, mathematicians, and blind consumers to advance our expertise in automated Nemeth translation of mathematics text. Workshop proposals were evaluated by an external advisory committee and are funded on a competitive basis. Fortunately, our proposal was selected based on merit and the qualifications of our team members. This workshop was held from August 3-August 7, 2020.

The purpose of the workshop was to extend the availability of mathematics textbooks in Braille to students and professionals who are visually impaired. The goal was to produce mathematical texts accurately, inexpensively, and in a timely manner by developing a user-friendly software workflow using open-source software to automatically, or nearly automatically, translate mathematical texts ready for embossing. Volker Sorge further suggests that, “An important goal of our work is also to provide the means of translating content into Nemeth to blind users directly, which will give them greater independence from publishers or transcribers. It will also empower blind students and researchers to get real time access to material given to them by their professors and peers.”

This was the first AIM workshop to run virtually after the closure necessitated by COVID-19. The organizers are grateful to AIM staff for thinking through the logistics of such meetings. Due to difference in time zones (the participants on the West Coast of the United States were separated by nine time zones from the participants in Europe), the daily meeting period of the workshop was somewhat reduced compared to the usual AIM workshop.

The workshop participants were selected from a list of respondents to our open invitation that was included in the public announcement of the selection of our workshop proposal. They came from various parts of the US and Europe and brought a wide variety of skills, backgrounds, and insights to the solution of this problem. What was most important was that none of us held particular biases about the difficulties and challenges of this problem. We came to the workshop with open minds and positive attitudes. Although this workshop did not completely solve the problem of automated Nemeth translation, we can cite two major accomplishments:

• The development of a framework and methodology for producing automated properly scaled tactile graphics with properly placed Braille labels.
• The identification of shortcomings in the Nemeth Braille code. Some symbols and conventions currently used in mathematics do not have Nemeth equivalents.

A gentle warning: As you read this article please do not be discouraged or turned off by unfamiliar terms such as: LaTeX, TikZ, SVG and MathML. A familiarity with these terms is not critical to grasp the significance of this work. I have included these terms here to convey the level of technical details that were needed to solve the problems of automated Nemeth translation. Frankly, many of the technical details involved here are also beyond my comprehension.

The support and commitment we have received from the general mathematics community is unprecedented. No other assistive technology efforts to aid blind people have had this degree of external support. At the same time, let us not forget that we, the blind community, were able to harness and use this support to our advantage because we were ready when the opportunity presented itself. For a long time, we have been articulating the need for automated Nemeth translation. We laid the groundwork for external cooperative efforts when this help was forthcoming. In short, our preparation could not have taken place without the leadership and direction of the National Federation of the Blind and the Jernigan Institute.

Before reviewing these two milestones, we need to introduce everyone to the participants and to review the subject matter of the workshop itself.

The Participants

A total of seventeen persons took part in this workshop. Six participants were the members of the original ANTT. They were Rob Beezer, Karen Herstein, Alexei Kolesnikov, Al Maneki, Martha Siegel and Volker Sorge. These team members were introduced to you in my article which appeared in the May 2020 issue of this magazine.

https://www.nfb.org/images/nfb/publications/bm/bm20/bm2005/bm200507.htm

Brief educational and professional backgrounds of the remaining participants are listed below:

Michael Cantino, accessibility specialist, Portland Community College, Portland, Oregon (BS in Liberal Studies from Portland State University): "I've been transcribing Braille for about 8 years. In 2014, I received certification in literary Braille transcription through the Library of Congress. I began transcribing Nemeth regularly in 2014 under the guidance of a large staff of Teachers of the Visually Impaired. I eventually enrolled in and completed NFB's Nemeth certification course. I was stopped short of completing the final Nemeth certification manuscript when I was presented with an exciting opportunity; I was asked to join an NSF research project studying the use of interactive 3D printed models for visually impaired learners. This project merged 3D printed graphics with an augmented reality app to provide audio and Braille annotations. 

I am passionate about the creation and use of tactile graphics. I began creating tactile graphics around 2013 and soon began leveraging existing technologies to make complex graphics that are clear and engaging. I regularly produce traditional embossed graphics, but I've also done quite a bit of work with 3D printers, vinyl cutters, laser cutters, and other advanced production methods. 

I've been transcribing in the UEB literary code since 2015. I started my Braille transcription journey with music Braille, and I spent several years teaching Braille music basics and supporting students in interpreting their scores.”

Davide Cervone, professor of mathematics, Union College, Schenectady, NY (PhD in Mathematics from Brown University): “My interest is through my work on MathJax, for which I am the primary author. MathJax has always had math accessibility as an important priority, and when Volker Sorge joined the MathJax project, that allowed us to realize that priority in a significant way. So my interest in the subject is through that avenue.”

John Gardner, formerly professor of physics, Oregon State University, Corvallis, Oregon (PhD, Physics, University of Illinois at Urbana-Champaign): He became totally blind after an eye operation in 1988. He continues to teach; however, he could no longer evaluate the data from experiments. This difficulty prompted him to establish a university team to research methods for better accessibility to graphical information. In 1996 his team developed a new high resolution embossing technology and patented it. He was unable to persuade any Braille embosser manufacturer to license this technology. They told him that there was no need for blind people to access graphics. He did not believe them, so he and his wife founded ViewPlus to produce embossers. In the year 2000 the first product reached the market, the ViewPlus Tiger Advantage embosser, built on a dot matrix printer.

Chris Hughes, staff tutor, Faculty of Science, Technology, Engineering & Mathematics, Open University, Milton Keynes, England (PhD in Applied Math from University of Reading, UK):
“I was interested in the workshop in my role as accessibility lead for maths & stats at The Open University. My reasons for participating were to gain and develop understanding on converting mathematical content into Braille.”

Alex Jordan, professor of mathematics, Portland Community College, Portland, Oregon (Ph.D. in Mathematics from University of Oregon): He is a mathematics instructor at Portland Community College, 2009 to present. He is a contributor to PreTeXt development. He is an author/coauthor/editor for several PreTeXt books. He brings an eye for accessibility issues to several projects, including PreTeXt, WeBWorK (an open online homework platform), and miscellaneous issues as they arise in his department.

Mitch Keller, assistant professor of mathematics, Morningside College in Sioux City, Iowa (PhD from the Georgia Institute of Technology): He began writing the open-source text Applied Combinatorics with his PhD advisor, William T. Trotter. Applied Combinatorics has been a PreTeXt book since August 2016. Keller is also an editor for three other mathematics texts written in PreTeXt and the co-author of a chapter of another PreTeXt mathematics text. “My interest in the AIM workshop stems from a desire to further enhance the quality of open-source math textbooks, particularly those authored in PreTeXt. The PreTeXt HTML is already designed to be accessible to screenreader users, but we know that Braille and tactile graphics can provide a better reading and learning experience for blind readers.”

Dr. Peter Krautzberger is a mathematician by training working as an independent consultant and developer in Bonn, Germany, working primarily to help academic publishers improve their content conversion workflows for STEM content (Ph.D. [Dr. rer. nat.]) in Mathematics, Freie Universität Berlin).

After leaving mathematical research, Peter managed the MathJax Consortium from 2012 to 2017, and he currently leads the work on the American Mathematical Society's HTML platform, including "MathViewer" for journal articles and EPUB production for textbooks and monographs.

Peter is also an invited expert with the W3C Accessible Rich Internet Applications Working Group (ARIA WG).

Oscar Levin, associate professor of mathematics, University of Northern Colorado, Greeley, Colorado (PhD in Mathematics from the University of Connecticut): “My interests in the workshop came from being an author of an open source textbook in PreTeXt (Discrete Mathematics: An Open Introduction) and as an occasional contributor to PreTeXt development.”

Michael Reynolds, associate professor of mathematics, Indian River State College, Stuart, FL. (PhD in Mathematics Education from University of Central Florida): “I was one of the new folks to this project, so I did not have very much to contribute to the sessions. But I was enthralled the whole time and am eager to get more involved in these efforts. I do not know much about PreTeXt, MathML, TikZ Liblious, and many of the other technical computer processes discussed in the workshop, but I am very interested in the process of making all mathematics materials accessible to all learners and all readers.

My research areas are the history of mathematics, mathematics education (particularly math anxiety), and graph theory. I've had no particular experience with Braille or blind learners that led me to participate in the AIM workshop. I just found it fascinating, both because of the complexity of the processes involved and because of the important issues of equity and accessibility.”

Richard Scalzo, US government, retired (PhD in Applied Mathematics, Northwestern University, Evanston, Illinois): He was very active in reading mathematics textbooks and research papers to blind students in the mid 60’s at Illinois Institute of Technology. He has spent his entire career working for the US government and nonprofit defense institutions, working in the area of software development and systems acquisitions.

Richard’s interest in automated Nemeth translation and the production of automated tactile graphics was further stimulated in 2010 when he and Al Maneki began to read and discuss Foundations of Geometry by Gerard Venema. To get around the problem of presenting Venema’s diagrams to Al, Karen Herstein was asked to reproduce the diagrams in Venema’s book in tactile form. She first traced the diagrams with a needlepoint tracing wheel. These diagrams had to be constructed in “reverse order” since the raised lines appear on the reverse side of the paper. Braille labels were then affixed in their proper positions. This “exercise” brought back all the painful memories of comprehending very complex mathematical diagrams, to which Richard and Al groaned in agony—there must be a better way!

As we were planning this workshop, Martha Siegel encouraged us to invite persons who did not necessarily have relevant backgrounds or expertise in Nemeth translation but who could view these problems with fresh insights and contribute with truly original solutions. “We don’t want to be talking to ourselves in this workshop,” Siegel cautioned. Considering the backgrounds of these ten individuals, we have successfully met Siegel’s advice.

It is worth noting that none of the workshop participants were intimidated by Braille. They did not view Braille as a mysterious system of writing that was difficult to learn and inconvenient to use, criticisms that we have often heard from many professionals in work with the blind.
Ignoring the need for sensitizing the fingers to recognize Braille characters, they took Braille for what it really is, just another “programming language” with its well-defined rules for constructing meaningful sequences of six dot characters.

Workshop Activities

AIM workshops generally consist of lectures to the entire group to provide relevant background information and smaller breakout sessions in which participants may focus on specific problem areas. Martha Siegel advised the lecturers to start with the most basic information since most of the participants were not familiar with PreTeXt or automated Nemeth translation. We will summarize the general lectures and the work of the breakout sessions:

Background Lectures

Rob Beezer introduced us to his PreTeXt authoring tool and pointed out its advantages over the many versions of LaTeX currently in use. PreTeXt is ideally suited for Braille translation when used in combination with Liblouis and Volker Sorge’s Speech Rule Engine (SRE).

John Gardner stressed the importance of audio tactile graphics since very few blind people are competent Braille readers. His company, ViewPlus, has developed IVEO, audio-tactile graphics software in which enriched SVG files can be embossed, then placed on a touchscreen. After simple calibrations, the associated text is read by a screen reader as the user touches the various graphic elements.

Volker Sorge presented an overview of his SRE that was initially built to produce spoken math output through a screen reader. The input to SRE is the math content of MathJax, an open-source JavaScript display engine for use with LaTeX, MathML, and AsciiMath notations. Instead of writing a separate rule engine for Nemeth output, Sorge decided to incorporate Nemeth output into his SRE.

Alexei Kolesnikov spoke to us about his work on automatically scaling print diagrams to fit on an individual Braille page. Proportional scaling of an entire diagram may be sufficient to provide the blind reader with comprehensible tactile graphics in the simpler cases. This turned out to be true for the embossed diagrams I examined before and during the workshop. However, for more complex diagrams, more sophisticated scaling and positioning of the labels will be needed.

Susan Osterhaus, a statewide mathematics consultant, and a long-time math teacher at the Texas School for the Blind and Visually Impaired, was a guest lecturer. She serves on the Braille Authority of North America’s (BANA) Nemeth Code Technical Committee and Tactile Graphics Technical Committee. She spoke to us about forthcoming updates to the Nemeth Braille code that BANA plans to make. She invited us to submit our recommendations to BANA.

Jonathan Godfrey, an invited lecturer, received his PhD in Statistics from Massey University, New Zealand. He created and maintains the Braille R statistical package. The R package is a complete and powerful set of general statistical routines that can be used in many applications. His discussion served as the basis for developing navigation-enriched diagrams for files produced by other scientific software.

Breakout Sessions

Our breakout sessions did not focus only on tactile graphics and Nemeth translation. There was considerable interest in producing audio-described graphics as proposed by John Gardner and providing advice to authors on creating easily enlargeable print graphic images into tactile form.

Tactile Graphics: Producing raised versions of print graphs and diagrams is a difficult problem because tactile legibility is different from the visual one. A way to automatically convert diagrams written in the LaTeX package TikZ (pronounced tik-zee) to a PDF file with Braille labels was known before the workshop began. However, due to the shortcomings found in this conversion and the inherent limitations of the PDF file format, we knew that we needed to turn to a different file structure.

The participants outlined a way to convert to a more versatile SVG file format with Braille labels from a TikZ file. A sample SVG file was tested on an embosser, and its output was satisfactory. However, shortcomings were also identified for the conversion of other SVG files. This group continues its work on SVG file conversion.

Audio Description of Graphics: The capability to produce an SVG file with Braille labels from a TikZ file was further enhanced during this workshop when we developed the ability to “navigate” through an SVG file with a screen reader. The navigation can allow a user interacting with an image on a computer screen to move between different components of the image. This technology can further be integrated with audio-tactile graphics software, IVEO.

Chemistry Diagram Demonstration: Accessible chemistry diagrams developed by Progressive Accessibility Solutions were demonstrated in another breakout session. Enriched SVG files allow the user to navigate around the skeletal formulas of molecules, from group level to individual atoms and bonds between them. Similar technology is available for navigating complex mathematics formulas.

Advice for Authors: A group of participants met to discuss the advice that could be given to authors to design graphics that would be more comprehensible to blind readers upon initial translation. Conceivably properly designed graphics should have a simpler appearance and be more appealing and comprehensible to the sighted reader as well. This group has also compiled a list of previously developed documents (including those produced by Benetech and BANA) containing graphics guidelines. The advice for mathematical graphics includes both general design principles, as well as specifics, such as spacing guidelines for graphics elements. An understanding of these guidelines should help us to develop more effective programming rules to produce more comprehensible tactile graphics.

Nemeth Braille Code: On our work with the SRE we found that there are no Nemeth equivalents for some symbols in common use today. For example, there are no Nemeth code representations for blackboard-bold and calligraphic fonts, or the equals sign with vertical bar on left, the hyphen with vertical bar on left, and the letter x with vertical bar on right. There are commonly used mathematical symbols today that simply did not exist in 1972, the last time revisions were made to the Nemeth Braille code. Nemeth rules describe how to transcribe complicated fractions and large matrices. Similar guidelines are needed for other 2-dimensional mathematical expressions that frequently occur in undergraduate texts: commutative diagrams and logical inference rules.

Workshop Perspectives

There is much cause for joy when we reflect on the many accomplishments of this workshop. Primarily, we got away from the mindset that automated Nemeth Braille translation and producing comprehensible tactile diagrams were the most difficult problems. This idea has been so ingrained in our thinking that it simply stymied us from making any progress at all. However, among the new participants, there was never the thought that these problems were unsolvable.

From my years of academic and government problem solving, I have learned that solutions are difficult to come by if we first decide that a problem is “hard.” When examining a problem, it is always best to take a first step toward a solution to see if a fresh perspective or further insights can be gained.

During our first session, as we introduced ourselves, both Martha Siegel and Alexei Kolesnikov admitted that Braille translation turned out to be more difficult than they originally thought. Siegel and Kolesnikov are experienced researchers in their own right, and they never let negativity creep into their modes of thinking. All of the workshop participants followed their lead of thinking positively.

We did not solve all of the problems related to automated translation. However, the two milestones mentioned in the introduction deserve further elaboration:

• We developed a framework and methodology for producing comprehensible tactile graphics. The ability to produce enlarged graphics with Braille labels will soon be solved. But this ability alone is not sufficient for producing comprehensible tactile graphics for more complicated print diagrams. What is needed is an enlargement in which more space is allotted to the most critical portions of a diagram, i.e., “spreading out” the more intricate parts to increase comprehensibility. In some cases, this may mean that some portions of a diagram may have to be “discarded” to simplify it. Then these elements may be reintroduced in a later diagram. How this is to be automated is not yet completely understood. Perhaps with experience, and by examining the guidelines of the Benetech and BANA documents, we may be able to develop program coding rules to accomplish these tasks. Karen Herstein suggested that one possibility is to consider the strategy employed in designing street maps in which the more detailed sections are enlarged to show the complex street patterns more clearly.

In his review of the first draft of this article, Richard Scalzo makes a further suggestion: “Even with fully automated translation from text formats there will be maintenance issues. These issues include incorporating new text and graphics formats and extending Nemeth Braille to include new symbols. The question of automating the labeling of graphics with Braille is one of the long-term issues. Enabling the automated labeling of graphics files may require modification of standards for graphics file formats. So, at some point in the future the project should investigate gaining support for enabling the automated labeling of graphics files.”

• We were presented with a list of all of the symbols and notations used in the abstract algebra textbook by Thomas Judson and in the calculus textbook, which is being used at the University of Nebraska, Lincoln. From these two lists, we identified the symbols which have no Nemeth equivalents. These have already been submitted to BANA for inclusion in the forthcoming revisions to the Nemeth code.

Our work of automatically translating the Judson textbook remains largely unfinished. I am hopeful that we will soon be able to make considerable progress on it because of the progress that the workshop has made on tactile graphics and because of the enhancements to the Nemeth Braille code. Rob Beezer will also be writing an article for this publication on PreTeXt in which he will describe the success he has had in producing a Braille Calculus textbook for a college student in Nebraska.

Although this workshop did not necessarily have audio tactile graphics under its purview, it seemed rather natural for us to take it up. We must accept the fact that we live in a multi-media world, and we should seize every advantage that multi-media offers us. At this time, however, I should point out that in so many mathematical instances, there is no substitute for Braille. Often in a mathematical diagram, points are labeled with letters or numbers. The best an audio description can give you is the label for that point; however, what is important about a diagram may be the paths and connections between points. The reader examines these possible paths and, for whatever reason, decides which paths are most significant.

There is a classic example in high school geometry. Starting from an arbitrary triangle, one constructs a set of line segments relating to this triangle. In this way, we define a set of nine points. By examining these nine points, i.e., running one’s fingers from one point to the next, one may conclude that these nine points lie on a circle. In my case, lacking sufficient geometric intuition, the circularity of these nine points was explained to me. The circle formed by these points is known as the Feuerbach nine-point circle. For a description, see the article https://en.m.wikipedia.org/wiki/nine-point_circle.

We did not investigate the use of 3D printers to construct tactile diagrams. There simply was not sufficient time for this. I believe that the advantage of using 3D printers is their ability to produce solid raised lines as well as dotted ones. Also, they offer a greater variety of textured surfaces.

With regard to automated Nemeth translation, where do we stand today? I think that in the very near future, given a textbook written in PreTeXt, it will be possible to produce separate BRF files for the text itself and SVG files for the diagrams. These diagrams may be of limited use because they will not be “smartly” scaled. Having these diagrams in tactile form may be better than having no diagrams at all.

Conclusion

I am intrigued by Volker Sorge’s suggestion that our automated Nemeth translation software should provide blind persons with the means of translating their own content into Nemeth. This means that blind users will have to acquire the additional skill of writing PreTeXt code. This is not an unreasonable demand. It also suggests the possible need for a reverse Nemeth to PreTeXt translator.

The automated translation to Nemeth Braille, including tactile diagrams, is a huge problem. While we have made a start at this workshop, this work is by no means complete. I think that once we have arrived at a solution, we will find certain aspects of that solution to be unsatisfactory. This will cause us to make improvements and refinements, leading to better solutions. The persons involved in this work will not be limited to those who took part in this workshop. There is still enormous opportunity and need for others to help in this effort. Anyone who is interested may contact me at [email protected]. We can make all of the workshop documents available to you.

Additional help needed! If you are:

• A blind college student planning to take a math course,
• A college-level math instructor anticipating a blind student in one of your courses, or
• A staff member of a disabled students services office assisting a blind student with a math course.

Please contact me at [email protected]. While we might not currently be able to render all of the assistance you need, we would like to discuss your situation in order to understand future needs for our Nemeth translation software.

For learning and studying mathematics and the other STEM subjects, the future has never been brighter. My modest accomplishments to the mathematical sciences will pale in comparison to what future blind scholars and scientists may achieve. So be it!