Supporting Students with Blindness and Visual Impairments to Learn Computational Thinking Through Astronomy

By Eric D. Hochberg, James K. L. Hammerman, and Santiago Gasca

Eric D. Hochberg is a senior researcher at TERC, a non-profit STEM education research and development organization in Cambridge, Massachusetts. His research and evaluation work focuses on development and refinement of materials and resources to support rigorous, meaningful, relevant, and accessible science, mathematics, and integrated computer science learning experiences.

James K. L. Hammerman is Co-Director of the STEM Education Evaluation Center (SEEC) at TERC. He works with K-12 and higher ed educators in formal and informal environments, addressing a wide range of STEM topics, including data and statistics, computing and artificial intelligence, and climate change and environmental justice, with a focus on equitable access and broadening participation.

Santiago Gasca is a researcher and evaluator at TERC. His work focuses on computational thinking, climate change, and the integration of science practices.

Abstract

Astronomy is often thought of as a highly visual field, but underlying its beautiful images are data that astronomers analyze using computational tools. The NSF-funded IDATA project developed astronomy and computing education materials for blind and visually impaired (BVI) and sighted middle and high school students, and then assessed the impact of these on students’ understanding of computational thinking (CT), confidence and interest in computing, and beliefs about who can engage in computing. We developed and tested an accessible CT assessment of the data and computational problem-solving practices (Weintrop et al., 2016) at the core of the project. BVI and sighted students made significant gains in computational thinking, with slightly higher gains for BVI students. All students had similar declines in their computing interest and confidence. Finally, both BVI and sighted students had slight, but not significant increases in their equity beliefs about visually impaired people engaging in computing.

Keywords

STEM-education, computational thinking, astronomy, assessments, attitudes

Introduction

Computational thinking (CT), the human thought process that guides the organization of problems and their solutions in a way that enables solutions to be generated by a computer (Wing, 2006), has received growing attention as an important set of competencies that students should develop as part of their K-12 schooling. Most states have adopted policies mandating computer science standards (Code.org Advocacy Coalition et al., 2021; Santo & DeLyser, 2020), which often identify computational thinking as a component (e.g., Arkansas Department of Education, 2018; Nevada Department of Education, 2019; Virginia Department of Education, 2017). CT practices are also described in the framework of the widely implemented AP Computer Science Principles course (College Board, 2020). However, broadening access to CT development opportunities for individuals with a range of sight ability is a challenge, as many K-12 learning applications providing an opportunity to engage in CT require sight (Riazy et al., 2020); for example, learning environments with graphical output (e.g., geometric explorations in Logo; Clements & Battista, 1989) and programming environments that use drag-and-drop blocks (e.g., Scratch; Resnick et al., 2009). The field has made progress addressing this challenge in recent years (Hadwen-Bennett et al., 2018; Milne & Ladner, 2018), such as through development of ways of creating and understanding block-based code using just a keyboard and screen reader (Mountapmbeme et al., 2022), and through the use of tactile materials in place of screen-based blocks to develop code that generates auditory output (American Printing House, 2023; Morrison et al., 2021).

Innovators Developing Accessible Tools for Astronomy (IDATA), a National Science Foundation funded project (see Notes 1 and 2), engaged both blind and visually impaired (BVI) and sighted middle and high school students in computational thinking processes by making analysis of astronomical data accessible to BVI individuals. Astronomy, though typically understood to be reliant on images from telescopes, is actually an especially data-intensive field, where primary activities of professional scientists involve using computational tools to take measurements and conduct analyses to understand phenomena, such as stellar and galactic evolution, exoplanet transits, variable stars, asteroid motion, and much more. In this regard, astronomy offers a fitting context for learning to engage in computational thinking. This occurs through processes, such as instructing remote telescopes to point towards an object and capture the light for a specified period of time, using a charge-coupled device (CCD) camera to convert the light into a data array which can be analyzed, conducting these analyses through data manipulation, and representing data in a form that is accessible to both sighted and visually impaired users—all of which require use of an information processing agent or computer.

The IDATA project aimed to promote equity in astronomy and computational thinking for individuals from groups historically underrepresented in the astronomy and computing fields (specifically, BVI and female-identifying or gender non-binary individuals; see Note 3) by providing supports for learning and engagement that increase accessibility. By positioning empathy and identity development at the nexus of astronomy and computing as central to the project’s equity objectives (Jackson et al., 2021)—incorporating opportunities for all students to learn with, from, and about, other students and practicing scientists or science educators representing these underrepresented groups—IDATA intended to increase access to astronomy and computational thinking for students regardless of visual ability or gender. Products developed by the project have also been translated into Spanish to further expand access.

IDATA was designed to provide both BVI and sighted students with opportunities for meaningful engagement in astronomy and computational thinking, with intended learning objectives in each of these areas. Learning opportunities occurred through a series of online modules (Associated Universities, Inc., 2019), developed by a team of astronomy and physics educators and undergraduates, many of whom had considerable experience living with blindness or a visual impairment, and some of whom were experienced BVI educators. Specific opportunities to engage in computational processes included learning to program in Quorum (a programming language, designed with BVI accessibility in mind that works well with screen readers, commonly used by BVI individuals; Stefik et al., 2019) and analyzing astronomical data collected through a remote robotic telescope network using sonification. The focus of the culminating astronomical investigation was tracking an asteroid and graphing its light curve.

This paper highlights IDATA’s BVI-accessible computational thinking learning opportunities and addresses the following research questions about IDATA participants’ computational thinking:

1. How does students’ computational thinking change through participation in IDATA, a learning experience designed to integrate computational thinking and astronomy?
2. How do students’ interest, confidence, and beliefs about who can engage in computing change through participation in IDATA?
3. To what extent do BVI students differ from sighted peers in their understanding of computing/CT and beliefs about who can engage in it?

Theoretical Framework

This study’s central hypothesis is that engagement with BVI-accessible resources through a substantial, long-term, computer-based astronomy-computing learning experience will build the capacity of both BVI and sighted middle and high school students to engage in CT. Specifically, the IDATA project envisioned that the learning experiences it created would (a) support development of CT understanding, and (b) increase levels of interest and confidence regarding computing in STEM, as well as the perception of other-abled individuals (and the self-perception of BVI individuals) to engage in computing in STEM. Ultimately, the project expected that enhanced accessibility of ways to use computing to do astronomy, coupled with facilitated opportunities to engage in meaningful learning, would yield equitable outcomes for BVI and sighted individuals, i.e., would be as effective for BVI students as for non-BVI students. In this section, we describe the theoretical foundations that led to this hypothesis.

Computational Thinking

As its importance in education has increased in the past few years, there have been extensive debates about how to define computational thinking (International Society for Technology in Education & Computer Science Teachers Association, 2011; Grover & Pea, 2013; Shute et al., 2017; Weintrop et al., 2016). Many go back to the seminal description by Wing (2006, p. 33): “Computational thinking involves solving problems, designing systems, and understanding human behavior, by drawing on the concepts fundamental to computer science. [It includes] thinking recursively… choosing an appropriate representation for a problem…[and] using abstraction and decomposition when attacking a large complex task,” among other things. While grounded in the concepts and ways of thinking of computer science, this definition, like most others, acknowledges that computational thinking is more than just programming a computer.  

The Computer Science Teachers Association (CSTA) identified “abstraction as CT’s keystone” and described several “widely accepted elements” of computational thinking: “Abstractions and pattern generalizations (including models and simulations); Systematic processing of information; Symbol systems and representations; Algorithmic notions of flow of control; Structured problem decomposition (modularizing); Iterative, recursive, and parallel thinking; Conditional logic; Efficiency and performance constraints; and Debugging and systematic error detection” (Grover & Pea, 2013, p. 39-40). In addition to elaborating on the meaning of “abstraction,” this description includes attention to important programming concepts and processes, as well as attention to computer hardware and systems.  

Weintrop et al. (2016) describe a four-fold taxonomy of computational thinking in math and science classrooms: “data practices, modeling and simulation practices, computational problem-solving practices, and systems thinking practices” (p. 127). This taxonomy includes many of the computational thinking elements described by the CSTA and others, but also adds an emphasis on data including data manipulation and representation practices, which are important in astronomical investigations. The curriculum and assessment development in IDATA focused on two of these four categories: data practices and computational problem solving (CPS) practices.  

Learning Experiences That Support Development of CT

School-age students have opportunities to develop CT skills and understanding in different ways (Grover, 2022). One way is through the process of programming or coding, where students use either a programming language (e.g., Python; Python Software Foundation, 2023) or a drag-and-drop block-based programming environment (e.g., Scratch; Resnick et al., 2009) to make something happen using a computer. Another way is through non-programming activities, such as puzzles and games that may require students to engage in algorithmic thinking processes, or to define a problem and solve it in steps after first unpacking the problem’s component parts.

Although many puzzles and games used to develop CT still involve a device with a screen (e.g., Zoombinis; Asbell-Clarke et al., 2021), many fully unplugged activities and approaches have been created that support CT development without the use of a computing device (Bell et al., 2009). A common activity, especially at the elementary school level, involves students writing out a sequence of instructions for performing a common task, such as making a sandwich or tying shoes, and exchanging instructions with a partner who follows them exactly as written. Decomposing a process into steps and engaging in debugging that occurs as someone else follows the process (e.g., by putting a jar of jam on top of a slice of bread, rather than opening the jar and using a knife to spread the jam) are among the computational thinking practices students draw on in this activity.

Especially as students get older, computational thinking learning experiences often marry unplugged and programming approaches, which is necessary to support a depth of computational thinking understanding (Caeli & Yadav, 2020). Opportunities to combine programming and non-programming activities can be accomplished through integration of computational thinking with content from other disciplines, including science (Arık & Topçu, 2022). IDATA took this integrated approach in the design of learning experiences, combining BVI-accessible non-programming activities (usually with tactile components) with the use of a programming language suitable for novices that was incorporated into IDATA activity modules.

Barriers to Accessibility of CT Development Opportunities for BVI Students

Despite varied access points for students to engage in CT, barriers often prevent BVI students from engaging fully. As with other content, especially in STEM disciplines, BVI students may struggle with having visual concepts or representations sufficiently explained to them, and with alternative tactile representations lacking sufficient detail to support their understanding in the same way as sighted students (Bell & Silverman, 2019). Access limitations with CT involve learning experiences relying on visual references as well as incompatibility with assistive technology for BVI individuals. The absence of BVI representation in CT learning experiences may also indirectly limit BVI students’ engagement in CT.

Reliance on Visual References

Although there are many unplugged ways to teach computational thinking using hands-on activities that do not require computer programming, when programming is used in educational contexts, it often requires navigating visually to assemble code or interpret its output. Block-based languages have proven quite powerful to help novice programmers create code without getting hung up in syntactical errors, by providing structures to cue for necessary inputs of different types and appropriate connections between code sequences (Weintrop, 2019). Turtle graphics (via any of the several flavors of Logo) and animated story-telling output (Scratch, Alice, etc.) have proven quite motivating contexts for students to engage in programming and, thus, computational thinking (e.g., Resnick et al., 2009). However, with some notable recent exceptions expanding some functionality of block-based programming to BVI individuals (Mountapmbeme et al., 2022; Morrison et al., 2021), these structures and output modalities that are supportive for those who can see fully can be barriers for those who cannot see them, or who can only see them with great difficulty. Unplugged activities, seemingly more accessible because they generally do not rely on a screen, fall along a range of BVI accessibility, often using visual content like diagrams, although accessibility can be improved through the use of tactile materials (Gupta et al., 2017). IDATA adopted principles of universal design in the development of learning experiences, aiming to enable BVI and sighted students to have equivalent experiences, in part by using tactile materials to explore ideas at the nexus of astronomy and computational thinking.  

Incompatibility with Assistive Technology

Professional BVI programmers as well as BVI college students have described challenges related to the incompatibility of programming tools with their own assistive technologies. Baker (2017) found that blind college students learning programming had to do additional work compared to sighted peers because they could not carry out instructions within the programming environments being used with the tools to which they were accustomed. Similarly, Mealin & Murphy-Hill (2012) describe professional BVI programmers’ use of text editors outside of the specific programming environments in which they work because of difficulty navigating within those programming environments when debugging programs. Depending on BVI condition, the ability to access learning experiences may depend on compatibility with assistive technologies like screen readers (e.g., JAWS, NVDA, VoiceOver; Baker et al., 2019) and/or magnification tools like ZoomText (Access Computing, 2022). The development of IDATA materials recognized this, ensuring compatibility of module materials with these technologies, and using the Quorum language—with an online environment that conforms to Web Content Accessibility Guidelines (WCAG 2.1; Kirkpatrick et al., 2018) and principles of Accessible Rich Internet Applications (ARIA; Diggs et al., 2017). Survey tools used as part of this study were intentionally compliant with WCAG 2.1 guidelines as well, and BVI students and educators who were not participating directly in the project tested surveys to ensure compatibility with commonly used assistive technologies.

Lack of Representation

The ability to recognize oneself among individuals engaged in STEM fields can drive interest and persistence (Lawner et al., 2019; MacDonald, 2014). There are no computational thinking learning initiatives of which we are aware that focus explicitly on both the learning needs of BVI students and the representation of BVI individuals and role models as part of the learning experiences. IDATA centered the astronomy-CT learning materials and activities around the experiences of BVI scientists and role models, as well as BVI students. For example, BVI scientists (e.g., astronomer, Dr. Nic Bonne), as well as a non-BVI actor representing a BVI student persona named “Penny,” are featured in embedded media throughout IDATA modules. These individuals are also the subject of some IDATA activities, highlighting both how technologies could create or expand access, and how BVI individuals do astronomy and computing.

Computational Thinking in IDATA

IDATA’s focus on the integration of astronomy and computational thinking meant that two categories of CT practices from Weintrop et al.’s (2016) taxonomy were appropriate to emphasize: Data Practices and Computational Problem-Solving Practices. The project focused specifically on data practices of collecting, analyzing, manipulating, and representing data, and on these specific computational problem-solving practices: preparing problems for computational solutions, computer programming, assessing different approaches to a problem, developing modular computational solutions, creating computational abstractions, and troubleshooting and debugging. The project’s focus on development of CT within the context of astronomy, as well as the prioritizing of BVI students’ access and engagement in activities, informed the design of activities.

Data Practices

As noted above, the CCD cameras attached to telescopes naturally create data arrays by recording the number of photons that hit each bin/pixel in the CCD during the time when the telescope is open and pointed at the object of interest. This insight, explored through an unplugged hands-on activity in IDATA (using a 3 x 3 tactile matrix on foam board to represent the camera, students in motion to represent objects in space, and foam shapes and push pins to represent photons of different colors; Associated Universities, Inc., 2019), that astronomical “images” are, essentially, arrays of numbers representing the amount of electromagnetic radiation received at different points was key to helping students understand that data are at the core of astronomical investigations, and that we can make choices about representing those data in ways that are accessible. As a result, several IDATA activities focused on methods for manipulating data using matrix mathematics to achieve important astronomical goals.

When comparing data from two images, resizing, cropping and rotating the data arrays allows them to be aligned. Rescaling the values to adjust for differences in exposure time, light sensitivity of the equipment, and viewing conditions allows stars with constant brightness to be represented the same way in two images. Once aligned and calibrated, subtracting the data of one image from the other removes objects that do not move across the sky, leaving only data for objects that seem to move, e.g., asteroids. Students in IDATA were walked through each of these data manipulation steps, first using pre-collected image data in AfterGlow Access software, which the project developed, to illustrate the concept, and then using Quorum to trace and apply code. (See Note 4.)   

Computational Problem-Solving Practices

Many IDATA activities engaged students in using computational problem-solving practices. For example, students were told that between 18,000 and 84,000 meteorites larger than 10 g strike the earth each year, and were asked to calculate the likelihood of one hitting the roof of their house. Exploring this problem involved creating variables and approximating their values (e.g., the size of my roof), computing several components and then building a relationship between them (the surface area of the earth in square kilometers; the number of roofs in a square kilometer), translating these into Quorum language code, checking the answer for reasonableness and debugging as needed. (See Note 5.) Enacting the thought processes to find an asteroid trail via the data manipulations described above using Quorum or AfterGlow Access also involved computational problem solving.  

Interest, Confidence, and Beliefs About Equitable Participation

Providing experiences to learn computing and STEM is often insufficient to catalyze long-term interest and engagement in these disciplines (Maltese et al., 2014). To become “the kind of people who do STEM and computing” (MacDonald, 2014), students need to have meaningful learning experiences along with opportunities to build confidence in themselves as knowers and doers of science (Happe et al., 2021). The mutual development of interest and confidence goes hand-in-hand with recognizing oneself represented in the field and with having role models or mentors who do the work (Aish et al., 2017; Lawner et al., 2019). The complementary objectives of building interest, confidence, and a sense of universal capability of success in computing and astronomy, were central to the IDATA project’s approach.

These priorities were evident across a range of activities in IDATA. Curriculum modules include several non-programming and programming activities to provide BVI and sighted students opportunities to learn computational thinking and astronomy using tactile and sonified representations. Accessible unplugged activities were designed with universal access in mind, and many called for sighted students to complete them with BVI condition simulation goggles or blindfolds to provide some insight into the experience of individuals with disabilities. Although there is some evidence that participating in disability simulation may actually diminish perceptions of people with disabilities (Nario-Redmond et al., 2017; Silverman et al., 2015), the IDATA project engaged in these simulations only after both providing education about the range of BVI conditions and providing students with in-person and/or Zoom engagement opportunities with BVI college students and scientists. Involving BVI individuals (Silverman, 2015) in combination with the simulation was intended to build empathy and understanding of how individuals can use their particular abilities to participate in science, especially when participation opportunities are designed with a range of abilities in mind. 

BVI scientists and near-peer (undergraduate) mentors were featured throughout IDATA module activities, and activities were intentionally designed with BVI access as a central objective to allow BVI students to experience success with challenging concepts, and thus to build confidence, alongside sighted peers. BVI representation served to normalize BVI participation in STEM, to provide role models to BVI and sighted students, and to point toward how computing can help to increase access to science. We expected that the presence of these features would help to increase levels of interest and confidence with computing in STEM, as well as the perception of BVI individuals as able to succeed in computing.

Methods

Sample

There were 111 students in grades 6 through 12 (average grades 9-10), from 12 groups spread across the US who participated in the test year of IDATA, engaging with the final set of materials during after school time. Groups were facilitated by middle- and high-school teachers (some of whom had specialization as teachers of the visually impaired), who received training and support for implementing IDATA resources from the project team. Participating students included 19 (17%) who identified as blind or low-vision. Groups had between 2 and 21 students, nine of which had at least one BVI student, with one group having five such students. Not all students who began the project in the fall “completed” the year, defined as engaging with curriculum materials on the SJS website at least until March 1 of the academic year, and/or completing one or more post-assessments. There were 86 students (78% overall) who completed the project (18 BVI students, 95%; 65 sighted students, 74%). The 25 students who dropped out of the program included one teacher and his entire group of five seventh graders as well as individual students in seven of the classes. Four of the remaining classes had 100% completion rates; the others ranged between 57% and 88% completion. Though most students completed the program, not all completed both pre- and post-assessments, allowing us to calculate difference scores for use in regression models. In fact, only 43 students from nine groups completed computational thinking (CT) pre- and post-assessments, 10 of whom (23%) identified as BVI. Our analyses, below, are limited to the 43 students who completed activities and for whom we have pre- and post-test data. Because our analyses could only include students with complete data, we checked whether the pre- and post-scores of those included in our models differed overall from those who were excluded. We found no statistically significant differences in CT or attitude scores between these groups.

Measure of Computational Thinking

To address our research questions, we needed a way of measuring those aspects of students’ computational thinking that are relevant to IDATA—computational problem solving (CPS) and data practices—that would be accessible to BVI middle and high school students. Just as many programming activities to develop computational thinking are inaccessible to those with vision impairments, we found that many existing computational thinking assessments, or at least several items from them, are inaccessible to BVI individuals because they require vision-based programming (Brennan et al., 2016; Brennan & Resnick, 2012; Rowe et al., 2021; Werner et al., 2012) or rely on analysis of images that cannot be described by text without revealing key information for their solution (SRI International, 2016; Weintrop et al., 2014). In addition, many CT assessments presume students’ pre-existing familiarity with specific programming languages or environments (Brennan et al., 2016; Brennan & Resnick, 2012; SRI International, 2016) which prevented their use as both pre- and post-assessments as measures of CT independent of programming ability. Finally, we wanted to be sure that the survey administration platform that we selected would support easy navigation for those using screen readers, and that we could select colors and fonts that would provide sufficient contrast for those with limited vision or color blindness. Ultimately, we opted to administer the survey using SurveyMonkey (https://www.surveymonkey.com).

Through selection and modification of existing computational thinking assessment items (see Note 6), as well as the creation of new ones designed to address the specific, focal computational thinking practices and sub-practices of the IDATA project, we developed a computational thinking assessment accessible to BVI and sighted students. The 26-item assessment—nine addressing data practices, 17 addressing computational problem solving—is scored using a rubric, which we also developed, with which we were able to achieve inter-rater reliability sufficient for research purposes (i.e., greater than 0.7). Items sometimes had several parts or could be answered with different levels of completeness or sophistication (e.g., just an answer, or also an explanation of reasoning), so the rubric allowed 1, 2, or occasionally 3 points for each item. Thus, the 26 items yielded a rubric with a maximum of 41 points (19 points for data practices, 22 for CPS practices).

We piloted the assessment and rubric with middle and high school students who were participating in IDATA. The instrument has strong internal consistency as a single scale (standardized Cronbach’s α = 0.93), and it can be split along the two Practice dimensions into two separate, internally consistent scales (Data α = 0.84, CPS α = 0.91), which are highly correlated with one another (r = 0.8).

Measure of Interest, Confidence, and Beliefs About Who Can Engage in Computational Thinking

We used a modified version of the Computer Science Attitudes and Beliefs Assessment (CSABA), originally developed with NSF funding for undergraduates (Hoegh & Moskal, 2009), but also tested and refined for use with high school students (Heersink & Moskal, 2010). The original 37-question, Likert-type scale includes five dimensions: students’ Confidence in their own ability to learn computing skills, Interest in computing, Gender (the perception of computing as a male field), beliefs about the Usefulness of learning computing, and beliefs about Professionals in computing.

To align with the IDATA project, we used items from the Confidence, Interest, and Gender constructs. Because of the project’s focus on broadening participation and building confidence and interest of BVI students in computing and astronomy, we added a parallel set of items to those used to measure attitudes about the ability of women compared to men, to engage in computing (i.e., the Gender construct) that instead compared the ability of BVI individuals and sighted individuals to engage in computing. For example, parallel to the statement, “Men and women are equally capable of solving computing problems,” we included the statement, “Sighted people and people with blindness or impaired vision are equally capable of solving computing problems.” Response options were on a 5-point scale, from Strongly Disagree to Strongly Agree, with a neutral position.

Operationalizing CT Engagement

To assess the effectiveness of the IDATA curriculum materials, we wanted to connect changes in students’ pre- to post-CT scores, with information about the extent of their engagement with CT content. Although there were some hands-on activities, IDATA content was primarily presented through pages on the Skynet Junior Scholars (SJS) website (https://skynetjuniorscholars.org), and addressed several core ideas—astronomy, computational thinking, equity, and accessibility. To get a rough measure of engagement with CT content, we identified SJS web pages that substantially addressed computation—27 of the 60 pages—and then, using web analytics installed on the SJS site, counted the number of times students accessed these pages. In instances where students worked in a group using a single group member’s device, we counted engagement related to one group member’s access with all other members of the declared group. We limited the count to pages that were accessed for at least 30 seconds at a time, to focus on pages with which students had substantive engagement, rather than pages that they just passed through while navigating to other pages.    

Results

In this section, we describe results of the test of the IDATA curriculum materials. We first describe the variability in students’ engagement with pages presenting CT content. Then we look at their learning of CT ideas, using the assessment we created as the measure, examining differences in learning based on students’ visual abilities, gender, and engagement with the CT content. Finally, we look at changes in students’ computing interest and confidence, as well as their beliefs about whether BVI individuals can be successful in computing, and whether those changes differ by students’ visual abilities. In this way, we address the three research questions.

Engagement in CT

The 43 students for whom we had both pre- and post- CT scores accessed a wide range of CT related content—a mean of 42 substantive page views (SD = 31), median of 39 page views, with a total range of zero to 123 page views and interquartile range from 16 (Q1) to 58 (Q3) page views. Blind and low-vision students accessed somewhat more CT pages than sighted students, about 10 pages more on average, but these differences were not statistically significant. These data were skewed right, with a lower limit of pages to the left and a long tail to the right, so we normalized the distributional shape for use in regression analyses by applying a square root transformation, but we reversed the transformation for interpretation. 

Change in CT Over Time

Students who took both pre- and post-IDATA CT assessments made modest statistically significant gains on the measure (average gain of 2.9 points, t = 3.2, p = .003, d = .24). Although BVI students had lower scores on the pre-assessments (average 4.9 points lower) and post-assessments (average 4.1 points lower) than sighted students, their gains were slightly higher (average 3.5 points compared with 2.7 points for sighted students), though none of these were significantly different from the scores or gains of sighted students (Model 1, Table 1). 

Among students with similar pre-scores, BVI students had post-scores that were slightly higher (0.53 points) than their sighted peers, though these differences were not statistically significant (Model 3, Table 1). A similar analysis showed female students had post-scores that were 3.6 points higher than their male peers, and these differences were marginally significant (t = 1.81, df = 39, p = .08, Model 4). Table 1 shows these and other regression models—centering the pre-scores means the intercept can be interpreted as the predicted value for sighted (or male) students with average pre-scores.

To assess the effectiveness of IDATA curricular materials, we added the square root transformed measure of engagement with CT-related pages to our regression models, again centered at its mean value so that the intercept represents average gains or post-scores for students with average engagement. For students who had similar pre-scores, post-scores were higher when they engaged with more CT pages (0.9 points for a shift from median to Q3) and these differences were marginally significant (Model 5, Table 1). CT score gains were also slightly higher when students engaged with more CT pages, and blind and low-vision students had slightly higher gains (0.5 points) than their sighted peers, but these differences were not statistically significant (Model 2, Table 1). 

Among students with similar pre-scores who engaged with similar numbers of CT pages, girls had post-scores that were 4.0 points higher than boys (t = 1.98, p = .055, marginally significant) and engaging with CT pages was a significant predictor of scores (1.2 points higher for a shift from median to Q3 engagement; t = 2.15, p = .038). In this model (Model 6, Table 1), BVI students had slightly (0.12) but not significantly higher scores than their sighted peers.

These models suggest that students learned about computational thinking by engaging with IDATA materials, that blind and low-vision students learned as much (or more) as their sighted peers, and girls learned as much or more than boys.

Change in Interest, Confidence, and Beliefs About Who Can Engage in CT

Several questions on pre- and post-surveys were designed to assess students’ interest and confidence in engaging in computing. Unfortunately, both student interest in computing, and their confidence in learning about and solving problems with computing declined slightly over the course of the project for both sighted and BVI students. (Note: The group who completed pre- and post-IDATA attitude surveys is slightly larger, N = 54, than those who completed both CT assessments, N = 43.) For sighted students, these changes were not statistically significant; for BVI students changes in confidence were comparable to those of sighted students, but changes in interest in computing were significantly less than that of sighted students. For sighted students, confidence in computing declined (–0.23 points) from pre- to post-; for BVI students this difference was –0.41 (Model 1, Table 2), but these differences are not statistically significant (d = –0.36). For sighted students, interest in computing declined from pre- to post- (–0.17 points), a marginally significant difference. For BVI students this difference was –0.63 points (Model 2, Table 2), which was a significant drop (t = –2.22, p = .031, d = –0.39). Correlation between pre- and post- values is r = 0.43 for confidence and r = 0.72 for interest.

We asked students about their beliefs about whether BVI people can succeed in computing careers and problem solving as well as sighted people. Scores at pre- were correlated with scores at post- (r = 0.69), and overall scores for sighted students increased from pre- to post-survey (average difference = 0.27), a statistically significant difference (t = 2.35, p = 0.023, d = 0.22). BVI students had slight declines in these views (average difference = –0.05) but this was not significantly different from sighted students (Model 3, Table 2).

Thus, the project was not able to increase interest or confidence in computing for sighted students. BVI students’ confidence in computing was comparable to that of sighted students’, but their interest in computing was significantly lower than their sighted peers. The project was able to increase sighted students’ equity beliefs about whether BVI students can engage with computing as well as sighted students, though these views did not change for BVI students.

Limitations

The measure of engagement with computational thinking content was necessarily rough, relying on counts of CT related SJS web pages accessed for at least 30 seconds. Yet, we do not know what students did after navigating to these pages—did they engage with the content either alone or with other students? Engage in unrelated discussions while the page sat open? Or even walk away from the computer? In addition, not all CT related pages were the same—some had extensive, multi-faceted readings, videos or opportunities to program; others had just a few. Our method cannot distinguish between these.

Though we had a substantial fraction (17%) of BVI students in our test group, the absolute numbers in the analytical sample (i.e., the subset of participants who engaged with the materials in the final year and had both pre- and post-data on computational thinking assessments) were relatively small (total N = 43, BVI N = 10), which limits the power of our analyses. Some of the non-significant findings above are likely due to this limited power, and could be addressed by a larger study, with the effect sizes found here, even if not statistically significant, useful in estimating the necessary sample sizes.

The CT assessment was developed to measure computational thinking in the domains of data practices and computational problem solving practices in general, rather than being aligned with the specific learning objectives of the IDATA project activities. This makes it more useful as a general tool for measuring CT in projects beyond IDATA, but may have limited the gains that students could have demonstrated with an assessment more closely aligned to the specific astronomy plus computation skills and knowledge of the IDATA activities (e.g., there were no questions about CCD cameras or asteroid motion in our CT assessment).  

Discussion

The results suggest that we were able to design a measure of two key components of computational thinking—data practices and computational problem-solving practices—that has good psychometric properties and is accessible to BVI middle- and high-school students as well as their sighted peers. Using this new measure, our study found that BVI students’ CT gains were at least as great as those of fully sighted students, and that engagement with IDATA project resources was positively associated with these gains. This leads to two important conclusions:

1. We developed a high-quality new measure of computational thinking that is a) accessible to both BVI and sighted students, as intended, and b) can detect differences in CT for both groups.
2. The IDATA materials are also accessible and effective, providing opportunities to learn about CT that are comparable for both visually impaired and sighted students, even using a measure which is only generally aligned with the content.

The project was also able to address its equity goals in other ways. Girls did as well as or better than boys in learning about CT. Although students’ interest in and confidence about engaging with computing declined slightly—not what we intended to happen—these changes were small, and confidence differences were not statistically significant. However, there was a marginally significant decline in interest among sighted students, and a statistically significant decline in interest among BVI students. A possible explanation for this decline in interest and confidence is that, consistent with other findings in the literature (e.g., Gorson & O’Rourke, 2020), students initially discover that engagement in computational problem-solving and data practices is a challenge, and although they can learn to do these things, their beliefs about their ability to succeed in computational thinking tasks diminishes, at least temporarily, as they come to understand the difficulties.

Finally, we hoped that both the social interactions between sighted and BVI students, and the content of the IDATA modules that provided role models of blind astronomers and directly addressed the capacity of visually impaired people to engage in computing and astronomy, could help all students realize that visual impairment is not an impediment to engaging with computing. In this we were modestly successful, though for sighted students only, whose beliefs about whether BVI people can engage successfully in computing increased slightly (albeit not statistically significantly) through engagement with the project. However, perhaps in connection with the challenges they experienced engaging with IDATA computing activities, BVI students’ beliefs about their own ability to engage in computing declined slightly, though not statistically significantly.

Implications for Practitioners and Families

This study shows that it is possible to design BVI-accessible materials and assessment tools for student development of computational thinking, even in a seemingly visual field like astronomy. Through participation in project activities, BVI students were able to learn as much computational thinking as their sighted peers. The nature of project activities and materials—especially including tactile learning experiences and models wherever possible, along with the deliberate use of programming tools that work optimally with commonly used BVI assistive devices like screen-readers—suggests that careful attention to the design of learning experiences for universal access can help to remove barriers to meaningful engagement with computational thinking.

Findings also indicate that, despite relative parity in growth in computational thinking between BVI and sighted students, confidence to engage in work that uses computational thinking may actually diminish at first. Thinking computationally is hard, and developing skills and confidence may not always increase even as students gain computational thinking knowledge. Acknowledging this difficulty so that students do not get discouraged by it, and providing appropriate technical resources (e.g., screen reader accessible curriculum and programming tools) and other scaffolding to support learning is crucial.  

Finally, greater representation of BVI students and scientists—especially those for whom computational thinking is an especially critical component of their professional activities—in the daily lives of BVI students (and sighted students) may be important to improve attitudes about equitable access to STEM activities for all. When the experiences of BVI people are demystified, when their engagement in scientific activities is celebrated, and when the use of computational tools to support their engagement is highlighted, students can develop more positive beliefs about what BVI people can accomplish, and about how computing can be a tool for equity.

References

Access Computing. (2022). Quorum Programming Language. https://www.washington.edu/accesscomputing/quorum-programming-language-0

Aish, N., Asare, P., & Miskioğlu, E. E. (2017). People like me increasing likelihood of success for underrepresented minorities in STEM by providing realistic and relatable role models. In 2017 IEEE frontiers in education conference (FIE) (pp. 1-4). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/FIE.2017.8190454

American Printing House. (2023). Code Jumper. https://codejumper.com/

Arık, M., & Topçu, M. S. (2022). Computational thinking integration into science classrooms: Example of digestive system. Journal of Science Education and Technology, 31(1), 99-115. https://doi.org/10.1007/s10956-021-09934-z

Arkansas Department of Education. (2018). Arkansas computer science standards for grades 9-12. https://dese.ade.arkansas.gov/Files/20210127201133_HS_Computer_Science_042018_pdf.pdf

Asbell-Clarke, J., Rowe, E., Almeda, V., Edwards, T., Bardar, E., Gasca, S., Baker, R. S., & Scruggs, R. (2021). The development of students’ computational thinking practices in elementary- and middle-school classes using the learning game, Zoombinis. Computers in Human Behavior, 115, Article 106587. https://doi.org/10.1016/j.chb.2020.106587

Associated Universities, Inc. (2019). IDATA module 1. IDATA. https://idataproject.org/idata/

Baker, C. M. (2017). Increasing access to computer science for blind students. ACM SIGACCESS Accessibility and Computing, (117), 19-22. https://doi.org/10.1145/3051519.3051523

Baker, C. M., Bennett, C. L., & Ladner, R. E. (2019). Educational experiences of blind programmers. In Proceedings of the 50th ACM technical symposium on computer science education (pp. 759-765). Association for Computing Machinery. https://doi.org/10.1145/3287324.3287410

Bell, E. C., & Silverman, A. M. (2019). Access to math and science content for youth who are blind or visually impaired. Journal of Blindness Innovation and Research, 9(1). https://doi.org/10.5241/9-152

Bell, T. C., Alexander, J., Freeman, I., & Grimley, M. (2009). Computer science unplugged: School students doing real computing without computers. The New Zealand Journal of Applied Computing and Information Technology, 13(1), 20-29.

Brennan, K., Chung, M., Martin, W., Cervantes, F., Tally, B., & Resnick, M. (2016). Assessing: How do I assess the development of CT? Computational Thinking with Scratch. http://scratched.gse.harvard.edu/ct/assessing.html  

Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. In Proceedings of the 2012 annual meeting of the American educational research association (Vol. 1, pp. 1-25). American Educational Research Association. https://web.media.mit.edu/~kbrennan/files/Brennan_Resnick_AERA2012_CT.pdf

Caeli, E. N., & Yadav, A. (2020). Unplugged approaches to computational thinking: A historical perspective. TechTrends, 64(1), 29-36. https://doi.org/10.1007/s11528-019-00410-5

Clements, D. H., & Battista, M. T. (1989). Learning of geometric concepts in a Logo environment. Journal for Research in Mathematics Education, 20(5), 450-467. https://doi.org/10.5951/jresematheduc.20.5.0450

Code.org Advocacy Coalition, Computer Science Teachers Association, & Education Computing Education Pathways Alliance. (2021). 2021 State of computer science education: Accelerating action through advocacy. https://advocacy.code.org/2021_state_of_cs.pdf

College Board. (2020). AP computer science principles: Course and exam description. https://apcentral.collegeboard.org/media/pdf/ap-computer-science-principles-course-and-exam-description.pdf

Diggs, J., Nurthen, J., Cooper, M., MacLeod, C., McCarron, S., Schwerdtfeger, R., & Craig, J. (Eds.). (2017). Accessible rich internet applications (WAI-ARIA) 1.2. World Wide Web Consortium. https://www.w3.org/TR/wai-aria/

Gorson, J., & O'Rourke, E. (2020). Why do CS1 students think they're bad at programming?: Investigating self-efficacy and self-assessments at three universities. In ICER ’20: Proceedings of the 2020 ACM conference on international computing education research (pp. 170-181). Association for Computing Machinery. https://doi.org/10.1145/3372782.3406273

Grover, S. (2022). Computational thinking today. In A. Yadav, & U. Berthelsen (Eds.), Computational thinking in education: A pedagogical perspective (pp. 18-40). Routledge.

Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38-43. https://doi.org/10.3102/0013189X12463051

Gupta, R., Balakrishnan, M., & Rao, P. V. M. (2017). Tactile diagrams for the visually impaired. IEEE Potentials, 36(1), 14-18. https://doi.org/10.1109/MPOT.2016.2614754

Hadwen-Bennett, A., Sentance, S., & Morrison, C. (2018). Making programming accessible to learners with visual impairments: A literature review. International Journal of Computer Science Education in Schools, 2(2), 3–13. https://doi.org/10.21585/ijcses.v2i2.25

Happe, L., Buhnova, B., Koziolek, A., & Wagner, I. (2021). Effective measures to foster girls’ interest in secondary computer science education. Education and Information Technologies, 26(3), 2811-2829. https://doi.org/10.1007/s10639-020-10379-x

Heersink, D., & Moskal, B. M. (2010). Measuring high school students’ attitudes toward computing. In SIGCSE ’10: Proceedings of the 41st ACM technical symposium on computer science education (pp. 446-450). Association for Computing Machinery. https://doi.org/10.1145/1734263.1734413

Hoegh, A., & Moskal, B. M. (2009). Examining science and engineering students' attitudes toward computer science. In 2009 39th IEEE frontiers in education conference (pp. 1-6). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/FIE.2009.5350836

International Society for Technology in Education, & Computer Science Teachers Association. (2011). Operational definition of computational thinking for K-12 education. Computer Science Teachers Association. https://csta.acm.org/Curriculum/sub/CurrFiles/CompThinkingFlyer.pdf

Jackson, C., Mohr-Schroeder, M. J., Bush, S. B., Maiorca, C., Roberts, T., Yost, C., & Fowler, A. (2021). Equity-oriented conceptual framework for K-12 STEM literacy. International Journal of STEM Education, 8, 1-16. https://doi.org/10.1186/s40594-021-00294-z

Kirkpatrick, A., O’Connor, J., Campbell, A., & Cooper, M. (Eds.). (2018). Web content accessibility guidelines (WCAG) 2.1. World Wide Web Consortium. https://www.w3.org/TR/WCAG21/

Lawner, E. K., Quinn, D. M., Camacho, G., Johnson, B. T., & Pan-Weisz, B. (2019). Ingroup role models and underrepresented students’ performance and interest in STEM: A meta-analysis of lab and field studies. Social Psychology of Education, 22(5), 1169-1195. https://doi.org/10.1007/s11218-019-09518-1

MacDonald, A. (2014). “Not for people like me?” Under-represented groups in science, technology, and engineering. South East Physics Network. https://www.sciencecentres.org.uk/documents/50/not_for_people_like_me.pdf

Maltese, A. V., Melki, C. S., & Wiebke, H. L. (2014). The nature of experiences responsible for the generation and maintenance of interest in STEM. Science Education, 98(6), 937-962. https://doi.org/10.1002/sce.21132

Mealin, S., & Murphy-Hill, E. (2012). An exploratory study of blind software developers. In M. Erwig, G. Stapleton, & G. Costagliola (Eds.), 2012 IEEE symposium on visual languages and human-centric computing (VL/HCC) (pp. 71-74). Institute for Electrical and Electronics Engineers. https://doi.org/10.1109/VLHCC.2012.6344485

Milne, L. R., & Ladner, R. E. (2018). Blocks4All: Overcoming accessibility barriers to blocks programming for children with visual impairments. In CHI ’18: Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1-10). Association for Computing Machinery. https://doi.org/10.1145/3173574.3173643

Morrison, C., Villar, N., Hadwen-Bennett, A., Regan, T., Cletheroe, D., Thieme, A., & Sentance, S. (2021). Physical programming for blind and low vision children at scale. Human–Computer Interaction, 36(5-6), 535-569. https://doi.org/10.1080/07370024.2019.1621175

Mountapmbeme, A., Okafor, O., & Ludi, S. (2022). Accessible Blockly: An accessible block-based programming library for people with visual impairments. In J. Froehlich, K. Shinohara, & S. Ludi (Eds.), ASSETS ’22: Proceedings of the 24th international ACM SIGACCESS conference on computers and accessibility (pp. 1-15). Association for Computing Machinery. https://doi.org/10.1145/3517428.3544806

Nario-Redmond, M. R., Gospodinov, D., & Cobb, A. (2017). Crip for a day: The unintended negative consequences of disability simulations. Rehabilitation Psychology, 62(3), 324–333. https://doi.org/10.1037/rep0000127

Nevada Department of Education. (2019). Nevada academic content standards for computer science and integrated technology. https://webapp-strapi-paas-prod-nde-001.azurewebsites.net/uploads/Nevada_Academic_Content_Standards_forrev_adf847c3ac.pdf

Python Software Foundation. (2023). Python. https://www.python.org

Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., & Kafai, Y. (2009). Scratch: programming for all. Communications of the ACM, 52(11), 60-67. https://doi.org/10.1145/1592761.1592779

Riazy, S., Weller, S. I., & Simbeck, K. (2020). Evaluation of low-threshold programming learning environments for the blind and partially sighted. In H. C. Lane, S. Zvacek, & J. Uhomoibhi (Eds.), Proceedings of the 12th international conference on computer supported education (Vol. 2, pp. 366-373). SCITEPRESS. https://doi.org/10.5220/0009448603660373

Rowe, E., Almeda, M. V., Asbell-Clarke, J., Scruggs, R., Baker, R., Bardar, E., & Gasca, S. (2021). Assessing implicit computational thinking in Zoombinis puzzle gameplay. Computers in Human Behavior, 120, Article 106707. https://doi.org/10.1016/j.chb.2021.106707 

Santo, R., & DeLyser, L. (2020). CS policy to practice: Understanding emerging approaches to state-level computer science education policy design in the United States. CSforALL. https://www.csforall.org/projects_and_programs/cspolicytopractice/

Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142-158. https://doi.org/10.1016/j.edurev.2017.09.003

Silverman, A. M. (2015). The perils of playing blind: Problems with blindness simulation, and a better way to teach about blindness section. Journal of Blindness Innovation and Research, 5(2). https://doi.org/10.5241/5-81

Silverman, A. M., Gwinn, J. D., & Van Boven, L. (2015). Stumbling in their shoes: Disability simulations reduce judged capabilities of disabled people. Social Psychological and Personality Science, 6(4), 464-471. https://doi.org/10.1177/1948550614559650

SRI International. (2016). Download the ECS assessments developed by SRI Education. Principled Assessment of Computational Thinking. https://pact.sri.com/ecs-assessments.html

Stefik, A., Ladner, R. E., Allee, W., & Mealin, S. (2019). Computer science principles for teachers of blind and visually impaired students. In SIGCSE ’19: Proceedings of the 50th ACM technical symposium on computer science education (pp. 766-772). Association for Computing Machinery. https://doi.org/10.1145/3287324.3287453

Virginia Department of Education. (2017). 2017 Computer science standards of learning for Virginia public schools. https://www.doe.virginia.gov/home/showdocument?id=9930

Weintrop, D. (2019). Block-based programming in computer science education. Communications of the ACM, 62(8), 22-25. https://doi.org/10.1145/3341221

Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127-147. https://doi.org/10.1007/s10956-015-9581-5

Weintrop, D., Beheshti, E., Horn, M. S., Orton, K., Trouille, L., Jona, K., & Wilensky, U. (2014). Interactive assessment tools for computational thinking in high school STEM classrooms. In D. Reidsma, I. Choi, & R. Bargar (Eds.), Intelligent technologies for interactive entertainment: 6th international conference, INTETAIN 2014 (pp. 22-25). Springer. https://doi.org/10.1007/978-3-319-08189-2_3

Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012). The fairy performance assessment: Measuring computational thinking in middle school. In SIGCSE ’12: Proceedings of the 43rd ACM technical symposium on computer science education (pp. 215-220). Association for Computing Machinery. https://doi.org/10.1145/2157136.2157200

Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33-35. https://doi.org/10.1145/1118178.1118215

Table 1
CT Scores and CT Page Engagement

	Dependent Variable:
	CT Difference Scores		CT Post Scores
	(1)	(2)	(3)	(4)	(5)	(6)
Constant	2.67* (1.04)	2.85* (1.09)	25.05*** (1.05)	23.81*** (1.14)	25.11*** (0.94)	23.34*** (1.33)
CT Pre Score (centered)			0.94*** (0.08)	0.89*** (0.08)	0.89*** (0.08)	0.81*** (0.09)
Blind Low Vision	0.83 (2.16)	0.51 (2.20)	0.53 (2.20)			0.12 (2.14)
Female				3.59~ (1.99)		3.99~ (2.04)
Sqrt # CT Pages (centered)		0.44 (0.37)			0.67~ (0.40)	0.84* (0.40)
N	43	41	43	42	41	40
R²	0.00	0.04	0.79	0.81	0.81	0.82
Adj R²	-0.02	-0.01	0.78	0.80	0.80	0.80
F-statistic (df)	0.15 (1; 42)	0.75 (2; 39)	77.13 (2; 41)	81.09 (2; 40)	79.14 (2; 39)	41.09 (3; 37)
Note. ***p < 0.001. **p < 0.01. *p < 0.05. ~p < 0.1.

Table 2
CT Confidence, Interest, and Equity Beliefs

	Dependent Variable:
	Confidence Difference	Interest Difference	BVI Equity Difference
Constant	-0.23 (0.14)	-0.17~ (0.10)	0.27* (0.11)
Blind Low Vision	-0.18 (0.29)	-0.46* (0.21)	-0.32 (0.23)
N	54	54	54
R²	0.01	0.09	0.04
Adj R²	-0.01	0.07	0.02
F-statistic (df)	0.39 (1; 53)	4.94 (1; 53)	1.89 (1; 53)
Note. ***p < 0.001. **p < 0.01. *p < 0.05. ~p < 0.1.

Notes

Note 1. We gratefully acknowledge support for this work from the US National Science Foundation under an award to Associated Universities Inc. (AUI), "Research Supporting Multisensory Engagement by Blind, Visually Impaired, and Sighted Students to Advance Integrated Learning of Astronomy and Computer Science (IDATA)" (DRL-1640131). The opinions, findings, and conclusions or recommendations expressed are those of the author(s) and do not necessarily reflect the views of AUI or the National Science Foundation.

Note 2. We also gratefully acknowledge contributions to the development of IDATA project materials from project collaborators: Timothy Spuck, Kate Meredith, Kathy Gustavson, Dan Reichart, Josh Haislip, Tyler Linder, Vladimir Kouprianov, Yasmin Catricheo, Andreas Stefik, Patrick Daleiden, Nic Bonne, Al Harper, Erika Labbé Waghorn, Demian Schkolnik, Alexandra Grossi, and Bret Feranchak; undergraduate mentors Tia Bertz, Katya Gozman, Chris Mathews, Kendall Mehling, Andrea Salazar, Benjamin Schafer, Alex Traub, and Sophia Vlahakis; as well as the many students and teachers who participated in the project and administrative staff and interns at all collaborating organizations, who provided valued support.

Note 3. Throughout the paper, we use the terms "women," "female," and "girls" to refer to students who identify as female, and the terms "men," "male," and "boys" to refer to students who identify as male.

Note 4. See, for example, https://skynetjuniorscholars.org/explorations/idata-module4/activities/section2/steps/k.

Note 5. See, for example, https://skynetjuniorscholars.org/explorations/idata-module4/activities/section1/steps/c.

Note 6. Some items from the Exploring Computer Science (ECS) Assessments were used. The ECS Assessments are a product of SRI Education and their development has been supported by the National Science Foundation.

The Journal of Blindness Innovation and Research is copyright (c) 2024 to the National Federation of the Blind.