American Action Fund for Blind Children and Adults
Future Reflections       Winter 2021     STEM

(back) (contents) (next)

Two Milestones Reached at AIM Conference On Automated Nemeth Braille Translation

by Al Maneki

From the Editor: Al Maneki is a mathematician who is deeply committed to enhancing access to mathematics education for blind students. His article "Math that Feels Good: Enabling Sighted and Blind People to Share the Mathematical Experience" appeared in the Summer 2020 issue of Future Reflections. In that article he explained how a team of mathematicians launched an effort to develop automatic translation of mathematics materials into Nemeth Code. In this article he updates our readers on some exciting new developments.

Author's Note: I wish to acknowledge the valuable assistance of my fellow team member Karen Herstein, who verified most of the facts and citations given here. I want to thank all of the participants at the AIM Workshop on Automated Nemeth Braille Translation for reviewing the first draft of this article and making suggestions and corrections. In writing this article, I relied heavily on the workshop summary that was prepared by Alexei Kolesnikov. I assume full responsibility for any errors, misrepresentations, and oversights.

Introduction

In 2019 the Automated Nemeth Translation Team (ANTT) applied to hold a workshop at the American Institute of Mathematics (AIM). AIM hosts workshops in all areas of mathematics, including mathematics education and accessibility in mathematics. One of six institutes sponsored by the National Science Foundation, AIM fully funds all costs associated with the workshop. The goal of our team's workshop was to bring together software developers, mathematicians, and blind consumers to advance our expertise in automated Nemeth translation of mathematics text. Workshop proposals were evaluated by an external advisory committee and were funded on a competitive basis. Fortunately, our proposal was selected based on merit and the qualifications of our team members. This workshop was held from August 3 to August 7, 2020. 

The purpose of the workshop was to expand the availability of mathematics textbooks in Braille to students and professionals who are blind or have low vision. The goal was to develop a user-friendly software workflow using open-source software to translate mathematical texts automatically, or nearly automatically. Volker Sorge further suggests, "An important goal of our work is also to provide the means of translating content into Nemeth to blind users directly, which will give them greater independence from publishers or transcribers. It will also empower blind students and researchers to get real time access to material given to them by their professors and peers."

This was the first AIM workshop to run virtually after the closure necessitated by COVID-19. The organizers are grateful to AIM staff for thinking through the logistics of such meetings. Due to differences in time zones (the participants on the West Coast of the United States were separated by nine time zones from the participants in Europe), the daily meeting period of the workshop was somewhat reduced compared to the usual AIM workshop.

The workshop participants were selected from a list of respondents to our open invitation that was included in the public announcement when our workshop proposal was selected. They came from various parts of the US and Europe and brought a wide variety of skills, backgrounds, and insights to the solution of this problem. What was most important was that none of us held particular biases about the difficulties and challenges of this problem. We came to the workshop with open minds and positive attitudes. Although this workshop did not solve the problem of automated Nemeth translation completely, we can cite two major accomplishments.

A gentle warning: As you read this article please do not be discouraged by unfamiliar terms such as LaTeX, TikZ, SVG, and MathML. A familiarity with these terms is not critical for you to grasp the significance of this work. I have included them to convey the level of technical detail that was needed to solve the problems of automated Nemeth translation. Frankly, many of the technical details involved here are also beyond my comprehension.

The support and commitment we have received from the general mathematics community is unprecedented. At the same time, let us not forget that we, the blind community, were able to harness and use this support to our advantage because we were ready when the opportunity presented itself. For a long time we have been articulating the need for automated Nemeth translation. We laid the groundwork for external cooperative efforts when this help was forthcoming. The work could not have taken place without the leadership and direction of the National Federation of the Blind and the NFB Jernigan Institute.

Gallery view of a Zoom meeting, showing members of the AIM team

Workshop Participants

Before reviewing the two milestones listed above, we need to introduce the participants and to review the subject matter of the workshop itself.

Sixteen persons took part in this workshop. Rob Beezer, Karen Herstein, Alexei Kolesnikov, Martha Siegel, Volker Sorge, and I were members of the original team. I'd like to share a bit of background about the other team members.

Michael Cantino, accessibility specialist, Portland Community College, Portland, Oregon. Michael is a certified transcriber of literary Braille, and he has completed the NFB’s course for Nemeth Braille certification. He is passionate about tactile graphics, and he has worked with 3D printers, vinyl cutters, laser cutters, and other advanced production methods.

David Cervone, professor of mathematics, Union College, Schenectady, New York. David is the primary author of MathJax, a program that has always made accessibility a priority.

John Gardner, formerly professor of physics, Oregon State University, Corvallis, Oregon. John became totally blind after an eye operation in 1988. He continued to teach, but he no longer could evaluate the data from experiments. To overcome this problem he established a university team to research methods for better accessibility to graphical information. In 1996 his team developed and patented a high-resolution embossing technology. The ViewPlus Tiger Advantage embosser, built on a dot matrix printer, was released in 2000.

Chris Hughes, staff tutor, Faculty of Science, Technology, Engineering and Mathematics, Open University, Milton Keynes, England. Chris serves as accessibility lead for maths and stats at The Open University. He is eager to learn more about converting mathematical content into Braille.

Alex Jordan, professor of mathematics, Portland Community College, Portland, Oregon. Alex is a contributor to PreTeXt development. He has been involved with several PreTeXt books as author, coauthor, or editor. He brings concern for accessibility issues to several projects, including PreTeXt, WeBWorK (an open online homework platform), and miscellaneous issues as they arise in his department.

Mitch Keller, assistant professor of mathematics, Morningside College, Sioux City, Iowa. Mitch wrote the open-source text Applied Combinatorics with his PhD advisor, William T. Trotter. His interest in the AIM workshop stems from a desire to further enhance the quality of open-source math textbooks, particularly those authored in PreTeXt. The PreTeXt HTML is designed to be accessible to screenreader users, but he knows that Braille and tactile graphics can provide a better reading and learning experience for blind readers.

Peter Krautzberger is a mathematician by training who works as an independent consultant and developer in Bonn, Germany. He works to help academic publishers improve their content conversion workflows for STEM content. Peter managed the MathJax Consortium from 2012 to 2017, and he currently leads the work on the American Mathematical Society's HTML platform, including "MathViewer" for journal articles and EPUB production for textbooks and monographs. He is an invited expert with the W3C Accessible Rich Internet Applications Working Group (ARIA WG).

Oscar Levin, associate professor of mathematics, University of Northern Colorado, Greeley, Colorado. Oscar is one of the authors of an open source textbook in PreTeXt (Discrete Mathematics: an Open Introduction), and he is an occasional contributor to PreTeXt development.

Michael Reynolds, associate professor of mathematics, Indian River State College, Stuart, Florida. Michael is very interested in making all mathematics materials accessible to all learners and all readers. His research areas are the history of mathematics, mathematics education (particularly math anxiety), and graph theory. He had had no experience with Braille or blind learners, but he found the AIM workshop fascinating, both because of the complexity of the processes involved and because of the important issues of equity and accessibility.

Richard Scalzo, US government, retired. In the mid 1960s Richard was active in reading mathematics textbooks and research papers to blind students at Illinois Institute of Technology. Richard's interest in automated Nemeth translation and the production of automated tactile graphics was stimulated in 2010 when he and I began to read and discuss Foundations of Geometry by Gerard Venema. Karen Herstein was asked to reproduce the diagrams in Venema's book in tactile form. She first traced the diagrams with a needlepoint tracing wheel. These diagrams had to be constructed in "reverse order," since the raised lines appear on the reverse side of the paper. Braille labels were then affixed in their proper positions. This exercise brought back our painful memories of creating and comprehending very complex mathematical diagrams. We groaned in agony, "There must be a better way!"

As we planned this workshop, Martha Siegel encouraged us to invite persons who did not necessarily have relevant backgrounds or expertise in Nemeth translation, but who could view these problems with fresh insights and contribute with truly original solutions. "We don't want to be talking to ourselves in this workshop," she cautioned. Considering the backgrounds of these ten participants, we have followed Siegel's advice.

It is worth noting that none of the workshop participants were intimidated by Braille. They did not view Braille as a mysterious system of writing that was difficult to learn and inconvenient to use, criticisms that we have heard from many professionals in work with the blind. Ignoring the need to sensitize the fingers to recognize Braille characters, they took Braille for what it really is, just another programming language with its well-defined rules for constructing meaningful sequences of 6 dot characters.

Workshop Activities

AIM workshops generally consist of lectures to the entire group to provide relevant background information, followed by smaller breakout sessions in which participants may focus on specific problem areas. Martha Siegel advised the lecturers to start with the most basic information, since most of the participants were not familiar with PreTeXt or automated Nemeth translation. We will summarize the general lectures and the work of the breakout sessions.

Background Lectures

Rob Beezer introduced us to his PreTeXt authoring tool and pointed out its advantages over the many versions of LaTeX currently in use. PreTeXt is ideally suited for Braille translation when used in combination with Liblouis and Volker Sorge's Speech Rule Engine (SRE). 

John Gardner stressed the importance of audio tactile graphics, since very few blind people are competent Braille readers. His company, ViewPlus, has developed IVEO, audio-tactile graphics software in which enriched SVG files can be embossed and then placed on a touchscreen. After simple calibrations, the associated text is read by a screen reader as the user touches the various graphic elements.

Volker Sorge presented an overview of his SRE that was initially built to produce spoken math output through a screen reader. The input to SRE is the math content of MathJax, an open-source JavaScript display engine for use with LaTeX, MathML, and AsciiMath notations. Instead of writing a separate rule engine for Nemeth output, Sorge decided to incorporate Nemeth output into his SRE.

Alexei Kolesnikov spoke to us about his work on automatically scaling print diagrams to fit on an individual Braille page. Proportional scaling of an entire diagram may be sufficient to provide the blind reader with comprehensible tactile graphics in the simpler cases. This turned out to be true for the embossed diagrams I examined before and during the workshop. However, for more complex diagrams, more sophisticated scaling and positioning of the labels will be needed.

Susan Osterhaus, a statewide mathematics consultant and a long-time math teacher at the Texas School for the Blind and Visually Impaired, was a guest lecturer. She serves on the Nemeth Code Technical Committee and Tactile Graphics Technical Committee of the Braille Authority of North America (BANA). She spoke to us about forthcoming updates that BANA plans to make to the Nemeth Braille code. She invited us to submit our recommendations to BANA. 

Jonathan Godfrey, an invited lecturer, received his PhD in Statistics from Massey University, New Zealand. He created and maintains the Braille R statistical package. The R package is a complete and powerful set of general statistical routines that can be used in many applications. His discussion served as the basis for developing navigation-enriched diagrams for files produced by other scientific software.

Breakout Sessions

Our breakout sessions focused on much more than tactile graphics and Nemeth translation. There was considerable interest in producing audio-described graphics as proposed by John Gardner. There was also discussion about providing advice to authors on creating easily enlargeable print graphic images that can be converted into tactile form. 

Hands examine a raised-line diagram.Tactile Graphics: Producing raised versions of print graphs and diagrams is a difficult problem because tactile legibility and print legibility are quite different. A way to automatically convert diagrams written in the LaTeX package TikZ (pronounced tik-zee) to a PDF file with Braille labels was known before the workshop began. However, due to the shortcomings found in this conversion and the inherent limitations of the PDF file format, we knew that we needed to turn to a different file structure.

The participants outlined a way to convert from a TikZ to a more versatile SVG file format with Braille labels. A sample SVG file was tested on an embosser and its output was satisfactory. However, shortcomings were also identified for the conversion of other SVG files. This group continues its work on SVG file conversion. 

Audio Description of Graphics: The capability to produce an SVG file with Braille labels from a TikZ file was further enhanced during this workshop when we developed the ability to navigate through an SVG file with a screen reader. The navigation can allow a user interacting with an image on a computer screen to move between different components of the image. This technology can further be integrated with audio-tactile graphics software, IVEO.

Chemistry Diagram Demonstration: Accessible chemistry diagrams developed by Progressive Accessibility Solutions were demonstrated in another breakout session. Enriched SVG files allow the user to navigate around the skeletal formulas of molecules, from group level to individual atoms and the bonds between them. Similar technology is available for navigating complex mathematics formulas.

Advice for Authors: A group of participants met to discuss the advice that could be given to authors to design graphics that would be more comprehensible to blind readers upon initial translation.  Conceivably, properly designed graphics should have a simpler appearance and be more appealing and comprehensible to the sighted reader as well. This group has also compiled a list of previously developed documents (including those produced by Benetech and BANA) containing graphics guidelines. The advice for mathematical graphics includes general design principles as well as specifics such as spacing guidelines for graphics elements. An understanding of these guidelines should help us develop more effective programming rules to produce more comprehensible tactile graphics.

Nemeth Braille Code: On our work with the SRE we found that there are no Nemeth equivalents for some symbols in common usage today. For example, there are no Nemeth code representations for blackboard-bold and calligraphic fonts, for the equals sign with a vertical bar on the left, for the hyphen with a vertical bar on the left, or for the letter x with a vertical bar on the right. These are commonly used mathematical symbols that did not exist in 1972, the last time revisions were made to the Nemeth Braille Code. Nemeth rules describe how to transcribe complicated fractions and large matrices. Similar guidelines are needed for other 2-dimensional mathematical expressions that frequently occur in undergraduate texts, such as commutative diagrams and logical inference rules.  

Workshop Perspectives

There is much cause for joy when we reflect on the many accomplishments of this workshop. Primarily, we got away from the mindset that automated Nemeth Braille translation and the production of comprehensible tactile diagrams were the most difficult problems. This idea has been so ingrained in our thinking that it simply stymied us from making any progress at all. However, among the new participants, there was never the thought that these problems were unsolvable.

From my years of academic and government problem solving, I have learned that solutions are difficult to come by if we first decide that a problem is hard. When we examine a problem, it is always best to take a first step toward a solution to see if a fresh perspective or further insights can be gained.

During our first session, as we introduced ourselves, Martha Siegel and Alexei Kolesnikov admitted that Braille translation turned out to be more difficult than they originally thought. Siegel and Kolesnikov are experienced researchers in their own right, and they never let negativity creep into their modes of thinking. All of the workshop participants followed their lead of thinking positively.

We did not solve all of the problems related to automated translation. However, the two milestones mentioned in the introduction deserve further elaboration.

We developed a framework and methodology for producing comprehensible tactile graphics. The ability to produce enlarged graphics with Braille labels will soon be solved. However, this ability alone is not sufficient for producing comprehensible tactile graphics for more complicated print diagrams. What is needed is an enlargement in which more space is allotted to the most critical portions of a diagram, i.e., "spreading out" the more intricate parts to increase comprehensibility. In some cases, this may mean that portions of a diagram have to be discarded to simplify the graphic. These elements may be reintroduced in a later diagram. How this is to be automated is not yet completely understood. Perhaps with experience, and by examining the guidelines of the Benetech and BANA documents, we may be able to develop program coding rules to accomplish these tasks. Karen Herstein suggested that one possibility is to consider the strategy employed in designing street maps in which the more detailed sections are enlarged to show complex street patterns more clearly. 

In his review of the first draft of this article, Richard Scalzo makes a further suggestion. "Even with fully automated translation from text formats, there will be maintenance issues. These issues include incorporating new text and graphics formats and extending Nemeth Braille to include new symbols.  The question of automating the labeling of graphics with Braille is one of the long-term issues. Enabling the automated labeling of graphics files may require modification of standards for graphics file formats. So, at some point in the future, the project should investigate gaining support for enabling the automated labeling of graphics files."

We were presented with a list of all of the symbols and notations used in the abstract algebra textbook by Thomas Judson and in the calculus textbook that is being used at the University of Nebraska, Lincoln. From these two lists, we identified the symbols that have no Nemeth equivalents. These symbols have already been submitted to BANA for inclusion in the forthcoming revisions to the Nemeth Code.

Our work of automatically translating the Judson textbook remains largely unfinished. I am hopeful that we will soon be able to make considerable progress on it because of the progress that the workshop has made on tactile graphics and because of the enhancements to the Nemeth Braille code. Rob Beezer will also be writing an article for this publication on PreTeXt, in which he will describe the success he has had in producing a Braille calculus textbook for a college student in Nebraska.

Although this workshop did not necessarily have audio tactile graphics under its purview, it seemed rather natural for us to take up this topic. We live in a multimedia world, and we should seize every advantage that multimedia offers us. At this time, however, I should point out that in so many mathematical instances, there is no substitute for Braille. Often in a mathematical diagram, points are labeled with letters or numbers. The best an audio description can give you is the label for that point. However, what is important about a diagram may be the paths and connections between points. The reader examines these possible paths, and for whatever reason, decides which paths are most significant.

Here is a classic example from high school geometry. Starting from an arbitrary triangle one constructs a set of line segments relating to this triangle. In this way, we define a set of nine points. By examining these nine points, i.e., running one's fingers from one point to the next, one may conclude that these nine points lie on a circle. In my case, as I lacked sufficient geometric intuition, the circularity of these nine points was explained to me. The circle formed by these points is known as the Feuerbach Nine-Point circle. For a description, see the article https://en.m.wikipedia.org/wiki/Nine-point_circle.

We did not have time to investigate the use of 3-D printers to construct tactile diagrams. I believe that the advantage of using 3-D printers is their ability to produce solid raised lines as well as dotted lines. Also, they offer a greater variety of textured surfaces.

With regard to automated Nemeth translation, where do we stand today? I think that in the very near future, given a textbook written in PreTeXt, it will be possible to produce separate BRF files for the text itself and SVG files for the diagrams. These diagrams may be of limited use because they will not be "smartly" scaled. However, having these diagrams in tactile form may be better than having no diagrams at all.  

Conclusion

I am intrigued by Volker Sorge's suggestion that our automated Nemeth translation software should provide blind persons with the means of translating their own content into Nemeth. This means that blind users will have to acquire the additional skill of writing PreTeXt code. This is not an unreasonable demand. It also suggests the possible need for a reverse Nemeth to PreTeXt translator.

The automated translation to Nemeth Braille, including tactile diagrams, is a huge challenge. While we have made a start at this workshop, this work is by no means complete. Even when we have arrived at a solution, I think we will find certain aspects of that solution to be unsatisfactory. We will be forced to make improvements and refinements, leading to better solutions. The persons involved in this work will not be limited to those who took part in this workshop. There is still enormous opportunity and need for others to help in this effort. Anyone who is interested may feel free to contact me. We can make all of the workshop documents available to you.

Additional help is needed! Please contact me if you are:

While we might not currently be able to render all of the assistance you need, we would like to discuss your situation in order to understand the future needs for our Nemeth translation software.

For blind students studying mathematics and the other STEM subjects, the future has never been brighter. My modest accomplishments in the mathematical sciences will pale in comparison to what future blind scholars and scientists may achieve. So be it!

With questions or ideas, you may contact Al Maneki at [email protected].

(back) (contents) (next)