Math that Feels Good: Enabling Sighted and Blind People to Share the Mathematical Experience

by Al Maneki

From the Editor: Al Maneki is a living legend. He has gone where few blind people have gone before, into the field of mathematics, and beyond this, he thinks it is a field in which more blind people should be involved. He knows technology and the software sighted people use to perform all of the mathematical functions required to be a skilled mathematician. He continuously presses the National Federation of the Blind to be a leading force in the incremental steps required to give us full access to the tools of science, technology, engineering, art, and mathematics. He is quick to credit the efforts of others in the work that he does, and this you will clearly see in his article. Here it is:

Author’s note: I want to begin this article by acknowledging the valuable assistance of fellow team member Karen Herstein, who verified most of the facts and citations given here. I also want to thank the other members of the Nemeth Braille translation team: Martha Siegel, Alexei S. Kolesnikov, Rob Beezer, and Volker Sorge, for reviewing the first draft and making suggestions. I also want to thank David Farmer, director of programs, and the American Institute of Mathematics, for its support and assistance with our work on Nemeth Braille translation. As the author of this article I assume full responsibility for any errors, misrepresentations, and oversights. Readers who wish to contact me may do so by email, [email protected]. – Al Maneki

The Beginning

The press release issued by the American Institute of Mathematics (AIM) on January 16, 2020, said in part:

Mathematics and science Braille textbooks are expensive and require an enormous effort to produce—until now. A team of researchers has developed a method for easily creating textbooks in Braille, with an initial focus on mathematics textbooks. The new process is made possible by a new authoring system which serves as a “universal translator” for textbook formats, combined with enhancements to the standard method for putting mathematics in a web page. Basing the new work on established systems will ensure that the production of Braille textbooks will become easy, inexpensive, and widespread.

This press release was issued in conjunction with the annual Joint Mathematics Meetings (JMM) of the American Mathematical Society (AMS), Mathematical Association of America (MAA), and several other professional mathematics organizations. There was a special session at this year’s JMM, held in Denver, on open educational resources. A one-hour portion of this session was devoted to talks by me and two of our team members on the progress we have made so far. We will have more to say about AIM later in this article.

The work described in this press release has been taking place quietly but with considerable NFB involvement since 2017. Given the considerable progress that has been made to date, the overwhelmingly positive response to this press release and to our JMM talks from both the mathematics and the blind communities, this is the right moment to inform readers of this magazine about this effort. Let me emphasize here that it is still too early for anyone to expect that mathematics textbooks can be produced on demand and delivered promptly.

For a very long time the development of an easy, inexpensive, and widely used piece of software to convert printed mathematics texts into Nemeth Braille has been considered the Holy Grail of providing mathematics textbooks to blind students. Marc Maurer, the Immediate Past President, told me that years ago, he asked Abraham Nemeth and Tim Cranmer to work on automated Nemeth Braille translation software. This was to no avail. Both leading technology giants of that day did not take up Dr. Maurer’s request.

Anyone who has experienced the pleasure of learning mathematics from a hard copy Braille text can tell you how much simpler the learning process and self-discovery can be. By keeping the fingers of one hand on an initial calculation and by keeping the fingers of the other hand on a concluding calculation, one can mull over the sequence of calculations that leads from the first one to the final one. In the case of logical arguments, one can mull over the sequence of logical steps that would lead from an initial proposition to a conclusion. To save textbook space and to challenge reader’s mathematical abilities, textbook authors will frequently leave out this intermediate material. Hard copy format of a mathematical text gives the reader the necessary time to think about the material and fill in the details left by the author to the readers.

In the case of tactile diagrams, a reader may use both hands to explore the patterns and the relationships displayed by the entire diagram. This type of exploration is simply not possible when a diagram is described verbally. A verbal rendering of a diagram may introduce the describer’s own interpretations of that diagram, depriving the listener of the opportunity for self-discovery of that diagram’s content. This is not to say that the use of refreshable Braille displays has no place in mathematics. We will have more to say about refreshable Braille displays later in this article.

The Back Story

Our story begins with Natalie Shaheen, the project director for the National Federation of the Blind’s Spatial Ability and Blind Engineering Research Project and the program director for the associated NFB EQ summer programs, undertaking her doctoral studies at Towson University under Dr. Jonathan Lazar. As Martha Siegel, professor emerita, Department of Mathematics, Towson University, tells it, Shaheen and Lazar discussed the difficulties and delays in obtaining a Braille textbook for a required statistics course. This tale of woe is well known to many of us. Lazar suggested that Shaheen take this matter up with Siegel.

The outcome of this conversation was an enraged Dr. Martha Siegel, determined to do something about this. At this point, in early 2017, Siegel and I had an extensive conversation and began to map out a strategy. Our first collaboration was “Guidelines for Collegiate Faculty to Teach Mathematics to Blind and Visually Impaired Students” which appeared in the summer 2017 issue of Future Reflections, the NFB’s quarterly publication for Parents of Blind Children. See: https://nfb.org/images/nfb/publications/fr/fr36/3/fr360307.htm, or https://www.maa.org/
sites/default/files/cupm/FacultyGuidelinesForTeachingVisuallyImpairedStudents.pdf

Siegel is the ideal spokesperson to promote our quest for the Holy Grail of automated Braille translation software. She has boundless energy and is totally committed to our Braille project. Siegel served as the editor of Mathematics Magazine from 1991 to 1996 and as secretary of the Mathematical Association of America (MAA) from 1996-2010. She was a faculty member in the mathematics department of Towson University from 1971 until 2015, when she became professor emerita. From her extensive service, she has many contacts in the mathematics community.

The Team and the Method

From the beginning, Siegel has held a firm mental image of how Braille translation software might work. She realized that Braille translation from all renditions of LaTeX and other presentation software languages would be virtually impossible because we cannot anticipate the form and structure of all presentation languages. Despite the overabundance of just LaTeX versions alone, Siegel’s plan was to develop a standard version of LaTeX to provide the framework for Braille translation software. She insisted that authors could be persuaded to write their textbooks in this standardized LaTeX version, enabling straightforward Braille translations.

In the Summer of 2018, Siegel recruited Alexei Kolesnikov, a colleague at Towson University. Although he had no prior experience with Braille translation, he was very familiar with LaTeX. He realized, along with Martha and me, that converting the words is easy: Braille is just another alphabet. The hard part is converting the structure of the book in a nonvisual way, converting the mathematics formulas, and converting the graphs and diagrams. “Given the amazing technology available today, we thought it would be easy to piece together existing tools into an automated process,” Kolesnikov said later.

Our approach to developing Braille translation software depended on breaking up the overall problem into three distinct problems, then solving each piece separately:

• First: The overall structure. A typical mathematics textbook uses visual clues to indicate chapters, sections, captions, and other landmarks. In Braille all the letters are the same size and shape, so these structural elements are described with special symbols.
• Second: Accurately conveying complicated mathematical formulas. In Braille, mathematics formulas are represented using the Nemeth Braille Code. The existence of the Nemeth Braille Code simplifies the problem of automated Braille translation for us, i.e., when the translation software encounters a mathematical expression or formula, simply convert it into its Nemeth equivalent.
• Third: Providing a nonvisual tactile way to represent graphs and diagrams. The nonvisual representation of graphs and diagrams remains a unique challenge. Many of the usual tools of presenting information using color, thickness of a line, shading, etc., are not available in tactile graphics. The tips of our fingers have a much lower resolution than our eyes, so the size of the image must be bigger (yet still fit on the Braille page). The labels that include a picture must be translated to Braille and placed so that they do not interfere with the drawn lines. Diagrams that show three-dimensional shapes are particularly hard to read in a tactile format.

Our plan to develop a standardized version of LaTeX began to change when Siegel introduced us to the American Institute of Mathematics (AIM) in January 2019, at the JMM which was held in Baltimore. AIM is an organization devoted to advancing mathematical knowledge through collaboration, to broadening participation in the mathematical endeavors, and to increasing the awareness of the contributions of the mathematical sciences to society. Since 2002 AIM has been part of the National Science Foundation (NSF) Mathematical Sciences Institutes program.
One of the participants from AIM was Rob Beezer, a mathematics professor at the University of Puget Sound in Washington state. His participation was most fortunate for us. He had already developed an authoring tool which he called PreTeXt, open-source software that contains all the properties of the standardized version of LaTeX that we were hoping to develop.

In August 2018 Siegel and Kolesnikov initiated a call with Beezer and David Farmer from AIM about PreTeXt. But it was the 2019 JMM where Beezer spoke with Volker Sorge about Braille (more on Volker Sorge later in this article). Later that day Siegel, Kolesnikov, Volker, Beezer, Farmer, Herstein, and I gathered at a table where we were introduced to Brian Conrey, AIM’s director. Looking back, it would be fair to say that this is when our Nemeth Braille team was formed and our collaboration with AIM was solidified.

Beezer sees automated Braille translation software as a natural extension of a dream he has been pursuing for several years: “We have been developing a system for writing textbooks which automatically produces print versions as well as online, EPUB, Jupyter, and other formats. Our mantra is ‘Write once, read anywhere.’” Beezer added Braille as an output format in his system when we decided to use PreTeXt as our standardized file format and authoring tool. According to Beezer’s scheme, we may consider all the Braille codes, Nemeth, music, etc., as just another presentation language on par with EPUB, Jupyter, etc. What a novel view of Braille!
All the elements of the first problem will be handled through PreTeXt. The non-mathematical elements in a PreTeXt file may be transferred as inputs to Liblouis, an open-source Braille translator and back-translator. The output BRF file for the translated mathematics textbook is created by Liblouis.

The codes in the PreTeXt file identify the mathematical elements for us. These elements are then inserted into a Nemeth Braille translator, and the translated Nemeth Code is transferred into its proper positions in the BRF file. For our Nemeth Braille translator, we are using MathJax, a standard, open-source package for displaying mathematics formulas on web pages.

When we met Volker Sorge, a reader in scientific document analysis in the School of Computer Science at the University of Birmingham, England, he had already extended MathJax to produce accessible mathematics content on the web through verbal screen readers. This MathJax accessibility was initially introduced as part of a grant from the Alfred P. Sloan Foundation with support from AMS and the Society for Industrial and Applied Mathematics (SIAM). Sorge was added to our team when he agreed to extend MathJax accessibility to include electronic Braille devices and hard copy Braille printers. Sorge noted, “We have made great progress in having MathJax produce accessible mathematics content on the web, so the conversion to Braille was a natural extension of that work.” Sorge is a member of the MathJax consortium and the sole developer of Speech Rule Engine, the system that is at the core of the Nemeth translation and provides accessibility features in MathJax and other online tools.

When I inquired about the connection between AIM and MathJax, I got three responses from Sorge, Farmer, and Siegel. Here are their direct quotations:

According to Sorge: “We (i.e., MathJax and Alex Jordan (mathematics department, Portland Community College)) organized an AIM workshop on mathematics accessibility on the web a few years ago. That’s where Rob and I started experimenting with Braille translation. But I assume most people were aware of MathJax before that as it is the main solution to transform and render math on the web.”

According to Farmer: “I was going to say it as: AIM is not involved with MathJax, but AIM is involved with Volker because of the workshop he co-organized in May 2018.”

According to Siegel: “And we are involved with AIM (and therefore MathJax) because I saw that AIM had sponsored the workshop on web accessibility for the blind, and I contacted Brian Conrey. MAA has been recommending MathJax for a while.”

Currently, we have no solutions to the third problem, i.e., the automated generation of tactile graphs and diagrams. At best, graphs and diagrams may be generated from PDF files and scaled to size to fit a standard 11” x 11.5” Braille sheet. Individual graphs and diagrams, once they have been embossed, are then manually inserted into the proper positions in a Braille volume that has already been generated from the BRF file. Alexei Kolesnikov has done much of the work and gained valuable experience in producing tactile graphs and diagrams. He has used a Tiger Max Braille embosser to produce them. Converting the print labels into Braille labels also requires an extra level of attention to details. All these difficulties will have to be taken into consideration for us to automate textbook production completely.

The Grant

The initial work for automated Nemeth Braille translation was funded by a grant issued in 2019 to Towson University from the American Action Fund for Blind Children and Adults. Under this grant, we began to develop some of the methods for automated translation as stated in the three problem areas. Kolesnikov downloaded a set of calculus II lecture notes with the belief that if we could translate these notes, we would be able to translate anything at an equivalent or lower-class level. Our translation of the chapter on double integrals of functions over regions in these lecture notes revealed several problems with the structure elements of PreTeXt that were intended for Braille translation and the Nemeth translation from MathJax. After several iterations of proofreading by Herstein and myself, improvements and corrections were made to PreTeXt, and some improvements were made to the MathJax Nemeth translator. I think we have the PreTeXt translation software working quite well. But we don’t have all the features that we need in Speech Rule Engine and MathJax.

To avoid the difficulties inherent in embossing three dimensional diagrams, we decided not to attempt a complete translation of these calculus II lecture notes. However, we gained valuable experience in the art and craft of automated Braille translation.

To complete work on the current grant, we began a Braille translation of the open-source textbook, Abstract Algebra: Theory and Applications (AATA) by Thomas W. Judson, written initially in PreTeXt. In the standard mathematic curriculum, abstract algebra is taught at a comparable level to or a slightly higher level than calculus II. As of this writing we have Brailled the first five chapters, and Herstein and I are now proofreading them. Diagrams have been produced for these five chapters. The diagrams are accurate; but work is still needed to automate the process of producing Braille labels and placing these labels in their proper positions, and if necessary, providing additional text explaining what is in the diagram.
For the purposes of demonstration, we brought a Braille copy of chapter two (with no diagrams) and a set of embossed diagrams from chapter one (both items containing imperfections) of Judson’s book to the JMM. The other two speakers from our team were Sorge and Kolesnikov. The overall response to our three presentations was most enthusiastic. Members of the audience went away with a better understanding of PreTeXt, Nemeth Braille, and the importance of hard copy Braille textbooks for blind and visually impaired students. They sympathized with and supported the need for PreTeXt. However, in addition to the target audience at JMM, we must develop a greater appreciation for PreTeXt and Nemeth in the overall mathematics community. This was the very reason for AIM’s press release at this time. The challenges of getting more textbooks authored in PreTeXt are immense. There is the matter of inertia. Authors are accustomed to doing what they have always done. Too often they don’t want to learn anything new. Textbook publishers must also buy into PreTeXt so that they will be willing to mass produce books from PreTeXt files.

Now What?

Despite the overwhelming difficulties inherent in the PreTeXt/Liblious/MathJax (PLM) approach to automatic translation, this is our best strategy for getting Braille books produced rapidly. Eventually, this approach can be made to work for books in all scholarly fields, especially those involving heavy use of footnotes. While direct Braille translations from presentation languages, such as LaTeX, may be desirable, there seems to be the universal belief that Braille translation from any presentation language is impractical. Since all textbooks written in any presentation language contains so much code that is irrelevant to Braille, the process of extracting the relevant material for Braille is daunting. For this reason, there have been virtually no successful efforts at translation from presentation language files.

In today’s learning and research environment, downloading and uploading of mathematical content to the web is absolutely essential. Although many limitations are imposed using one-line refreshable Braille displays, these are the only devices we have for web communications. We will use them until better refreshable Braille displays become available. We must continue to insist that all mathematical materials on the web are nonvisually accessible.

In order to simplify the PLM translation software for the average blind or sighted user, we must improve all the software components so that file transfers and specification of parameters are easy to perform and foolproof. We must also write good documentation to guarantee ease of use. Of course, the procedures and use of PLM translation software will be nonvisually accessible. I don’t know when all of this will be finished. At present, if anyone wishes to write a mathematics book in PreTeXt and to translate it into Nemeth Braille, it would be best to get in touch with Rob Beezer, email [email protected].

For the remainder of this year, we hope to perfect our techniques in order to produce an error-free Nemeth Braille copy of AATA with diagrams. Also, on the calendar this year, there will be a one-week workshop on Braille translation sponsored by AIM from August 3-7. At this workshop we will reconsider all aspects of PLM translation software. We will pay special attention to the problem of automating tactile graphics. Ultimately, PLM translation software should be good enough so that any book that is translated into Nemeth Braille from it will pass the proofreading of a certified Nemeth proofreader. You can be sure that the NFB’s technical and mathematical expertise will be well represented at this workshop.

My Take

This work on PLM translation software has been a most pleasant experience for me. It has brought back many of the memories of my student days—the struggles and the triumphs of mastering mathematical concepts without benefit of hard copy Braille textbooks. The work we are doing now will make life easier for the next generation of blind students. Despite the difficulties I faced, compared to the advantages that the upcoming students will have, there is no doubt in my mind; I would do it all over again just the way I did it. My accomplishments in mathematics are all the proof I need.

I have been an NFB member for almost fifty years. Over this half century, we have accomplished enormous successes because of both our collective and individual actions. Just remember that PLM translation software would never have been started if Natalie Shaheen hadn’t taken the individual action of paying a visit to Martha Siegel. My heart and soul are inextricably tied to NFB. We are the only game in town!