Evaluation Report: Sensational Activities in Summer Science 2017

Donna Posont; Jason Meddaugh; Fred Wurtzel; Lydia Schuck

Evaluation Report: Sensational Activities in Summer Science 2017

By Donna Posont, Jason Meddaugh, Fred Wurtzel, and Lydia Schuck

Donna Posont is a Naturalist at the University of Michigan—Dearborn, and has served in several positions in the Michigan Affiliate and Western Wayne County Chapter of the National Federation of the Blind. Donna conducts year-round science and nature programs for blind youth.

Jason Meddaugh is the Owner of the AT Guys technology company, and has served on the state board of the Michigan Affiliate of the National Federation of the Blind. Jason has also been involved in the Michigan Association of Blind Students and many activities for blind youth.

Fred Wurtzel is Past President of the National Federation of the Blind of Michigan and has served in numerous positions on the state board and the local Lansing chapter He is retired from supervising the Business Enterprise Program at the Michigan Commission for the Blind. Fred is a passionate amateur scientist and helps to conduct youth programs throughout the year.

Lydia Schuck is the parent of a young adult who is blind with other disabilities. She has published stories and articles about blindness in Future Reflections and in the Journal of Visual Impairment and Blindness. Lydia works as an instructional designer at Hadley Institute for the Blind and Visually Impaired.

Abstract

This article reports the results of an evaluation of a summer science program for blind and visually impaired youth that took place in 2017. The program was conducted by members of a state affiliate of the National Federation of the Blind.

Keywords

Science, blind, visually impaired, weather

Background

The Michigan affiliate of the National Federation of the Blind has provided STEM-based learning opportunities for middle and high school-aged blind youth since 2010. The 2017 program ran for eight days, beginning at the end of July, based at a state university. In the summer of 2017, the National Federation of the Blind conducted a week-long residential summer science program for visually impaired high school-aged students. The theme of the camp was weather, its cycles, climate, and the effects of weather on human systems. This was the tenth annual program. Institutional Review Board approval was not necessary for this evaluation report. Evaluation data were collected with the limited intent of evaluating and improving existing programs.

The Sensational Adventures in Summer Science program takes over the University of Michigan—Dearborn’s Environmental Interpretive Center facility for the week. The program has full use of a college-level laboratory equipped with lab tables and a classroom with tables and seating ordinarily used in college classrooms. These features provide an authentic experience of a college environment.

The National Federation of the Blind places a strong emphasis on successful blind role models and qualified staff who can serve as mentors to the participants. Staff ranged in age from college student to a retired individual. The staff included a professional naturalist, chemist, child development professional, and small business owner, all of whom were blind. The sighted staff included a school teacher and driver. Two staff members were former SASS participants.

The program produces an annual evaluation report. The evaluation of the program has three purposes: to measure student progress toward the academic and other goals of the week, to provide a report to funders and other stakeholders, and to identify improvements for the next year's program. The evaluation process for the SASS program has expanded over several years' time from a focus on participant satisfaction to a retrospective post- then pre- self-report methodology measuring change in learning and attitudes. In 2017, the SASS evaluation included a staff-created pre-assessment and post-assessment of weather knowledge, science attitudes and perceptions, and future plans. The 2017 report includes participant demographics, description of program components, participant-reported science attitudes and perceptions, and assessment results of pre- and post-program knowledge of weather concepts.

Upon arrival, a staff member administered an assessment to each participant to determine basic understanding of the topic of the week-long program, weather. Participants also answered questions about their attitudes and perceptions toward science classes and labs, and the extent to which they participate in science activities at school. A matching post-assessment was used at the end of the program. This part of the program evaluation was designed to identify academic and social factors that might be affected by the SASS program.

Youth Scientist-Participant Demographics

The 2017 class of youth scientist-participants ranged in age from 13 to 19 years. Four were females, and nine were males. Eleven of the participants reside in Michigan, where the program was conducted. One came from Maryland, and another from Belize, a country in Central America. Four participants were totally blind and nine were students with low vision of varying visual acuities. Six were braille users with varying skill levels.

Table 1(a). Demographics by Age (N = 13)

	n	%
12	1	7.7
14	2	15.4
15	5	38.5
16	2	15.4
17	2	15.4
19	1	7.7

Note 1: Mean = 15.4 years
Note 2: Median = 15 years

Table 1(b). Demographics by Gender

	n	%
Male	9	69.2
Female	4	30.8

Table 1(c). Demographics by Race

	n	%
African American	5	38.5
White	4	30.8
Asian	2	15.4
Hispanic	2	15.4

Program Components

Focus on the Expanded Core Curriculum

The staff plans the week's activities for each SASS program with a purposeful emphasis on the nine areas of the Expanded Core Curriculum for Students who are Blind or Visually Impaired (Texas School for the Blind and Visually Impaired, 2014). Staff members and participants use and talk about their assistive technology during the week. Blind staff and guest presenters act as mentors for self-determination, career education, and orientation and mobility skills.

To further develop orientation and mobility skills, participants took turns leading the several-block walk across campus to the university cafeteria. Staff strongly encouraged participants to use white canes. Meals at the hotel, university cafeteria, and other venues provided opportunities to learn and practice independent living and social interaction skills. At the university cafeteria, participants selected and paid for their own meals, and then found and cleared their own tables. Participants were expected to be on time to meals, to serve themselves at the hotel breakfast buffet each morning, and to clear their own tables.

Participants used varied approaches for recording data and for exploring hands-on chemistry and other activities. Each participant used their preferred literacy medium and other tools, applying compensatory and sensory efficiency skills strategies.

Finally, the program promoted physical activity by scheduling swimming as an almost daily activity. In 2017, participants also walked several blocks between locations on the university campus and while at the farm and the living history museum.

Expanded Core Curriculum skills were practiced individually, alongside peers, and with the help and encouragement of blind mentors. Participants shared their individual experiences of daily "highs and lows" at evening times of reflection for the staff and participants.

Learning About Weather

The 2017 SASS program focused on weather. Activities included the following:

Participants collected daily quantitative weather data from a nearby weather station and an internet-enabled remote weather station 150 miles away. Participants learned to use the Beaufort Wind Scale to estimate wind speed and direction. Participants also estimated the temperature and humidity without instruments. They recorded their own estimations, the nearby weather station data, and the remote data each day. Participants later compiled the data and created a story using this data, with the goal of demonstrating an interesting and accessible approach to data collection and use.
A certified science teacher on the SASS staff used tactile three-dimensional models to introduce participants to cloud types.
An internationally recognized naturalist met with the participants to discuss the relationship between weather and trees. The naturalist, staff, and participants observed and counted tree rings while discussing the reaction of trees to annual weather cycles. Participants listened to a selection from Aldo Leopold's Sand County Almanac in which the author reflected on sawing through a very old tree from outer edge to center, describing the events of the day for each ring.
A SASS staff member and past participant presented the “Storm Shield” program. The National Weather Service trains a network of citizen-scientists to report weather, providing valuable information to weather forecasters during times of dangerous weather.
The same staff member presented valuable information about emergency preparedness for weather events and other emergency situations. Participants took home a bag containing emergency food and tools which could be used in survival situations, with instructions on what to add to the bag to make it more useful for them and their families.
Scientist and entertainer, Paul Gross, from WDIV Channel 4 in Detroit, discussed the science of weather forecasting and his views on the climate change debate. He has authored a book on extreme weather in Michigan. One scientist later asked Mr. Gross how she could visit him at the television station to see what he does on the job, as this is a career interest of hers.
A lead naturalist from the Environmental Interpretive Center presented on plant and animal responses to weather cycles. Participants examined animal pelts and other artifacts. Participants predicted which pelts would be suited to which climate.
Participant groups researched and later presented information about a science-related word, reinforcing what they learned during the week. The words assigned to the four groups were “sunny,” “snowy,” “anemometer” (representing the concept “windy”), and “stormy.” Each team devised a project to demonstrate some aspect of the word, which they presented to parents and guests on the last day of the program.
Participants used commonly available glue traps to collect samples of debris carried in the ordinary breeze. The traps were suspended on strings outdoors in the woods, on lawn areas and near the building. The traps were retrieved the next day and analyzed for content. The exercise demonstrated what is all about us in the air all the time, reinforcing Paul Gross’s comments that the greatest danger to people in a high wind event is from the debris carried by the storm.
The staff informally assessed weather knowledge at the end of the week during the annual SASS Jeopardy game. Staff members created questions of varying difficulty for the high-spirited game, based on topics presented during the week of SASS. Participants earned "SASS Bucks" for correct answers, which they spent at the SASS Auction that followed. Auction items included headphones, Bluetooth speakers, adapted games such as Braille Uno, and other playing cards.

The SASS program also offered other hands-on activities that were not directly related to weather, including the following:

Participants traveled to a working dairy farm where they fed calves, ducks, turkeys, chickens, lambs, and goats in and around the barns and other buildings on site. Participants took advantage of the opportunity to milk a cow, with several commenting on the large size of a cow. Many of the participants complained about the smell of the barn. Participants, staff, and dairy workers discussed dairy cattle care and management while enjoying ice cream.
A blind chemist applied principles of chemistry to make “Dippin’ Dots”, the ice cream novelty often found in sports venues. She sprayed streams of melted ice cream into a bowl of liquid nitrogen, forming pea-sized balls of ice cream. The blind chemist carried out chemistry experiments independently, demonstrating the skills of blindness as applied to her career.
Participants toured a living history museum, a village with buildings that had been moved to the site from all over the United States. Participants toured the village with a guide who had been trained in accessibility strategies. The groups visited Henry Ford’s birth home, the Wright Brothers' Bicycle shop, and the Susquehanna Plantation, followed by a play about slavery. Participants also visited a large barn, observing the functions of different areas of the barn: shelter for animals during storms, warm quarters for newborn livestock, and ventilated places for drying and storing grain and hay. The village brings together in one location examples of architecture from several periods in American history, authentic arrangements of buildings and orchards, and examples of homemade clothing, trees, and plants to suit various seasons and climates.
A professional electrical engineer and blind role model presented the basics of Newton's Laws of Motion, giving participants topics of conversation in the van as the group traveled around in the following days.
Participants and staff traveled to a local farmers' market to purchase the ingredients for the lunch they prepared for parents and guests on the final day of the program. Participants developed the menu in collaboration with staff, preparing a list that was then divided between groups who found and purchased the needed items.

On the final day of the 2017 SASS program, parents and other guests joined participants and staff for team presentations and lunch. Participants reflected on the week's activities, using the high and low reflection strategy that had been used each evening as a debriefing activity. Each scientist and staff person described a high and a low for the week. In this way parents and staff learned what was meaningful to each scientist.

Assessment of Science Attitudes and Perceptions

The daily high-low reflection informally revealed science attitudes and perceptions as the program progressed. Science attitudes and perceptions were measured more formally at the beginning of the week through an assessment administered individually to participants by program staff. A staff member read the items to the participant and recorded the responses. Results of the assessment of attitudes and perceptions are in Table 2, below, and in the text that follows.

Table 2. Participant-Reported Science Attitudes and Perceptions Before (Pre) and After (Post) Program

Statement	Never 1	Some of the time 2	Most of the time 3	Always 4	Mean
I look forward to science class	Pre 0.0% - Post 7.7%	Pre 66.7% - Post 30.8%	Pre 41.7% - Post 46.2%	Pre 25.0% - Post 15.4%	Pre 2.9 - Post 2.7
I feel confident around science lab equipment	Pre 8.3% - Post 7.7%	Pre 66.7% - Post 23.1%	Pre 16.7% - Post 30.8%	Pre 41.7% - Post 38.5%	Pre 2.9 - Post 3.0
I am as good in science as other kids my age	Pre 0.0% - Post 7.7%	Pre 25.0% - Post 15.4%	Pre 25.0% - Post 46.2%	Pre 50.0% - Post 30.8%	Pre 2.8 - Post 3.0
I participate in the same science classes as my peers	Pre 8.3% - Post 0.0%	Pre 25.0% - Post 15.4%	Pre 25.0% - Post 30.8%	Pre 41.7% - Post 53.8%	Pre 3.0 - Post 3.0
I am able to explain to my teachers how to adapt science activities so I can participate	Pre 0.0% - Post 0.0%	Pre 25.0% - Post 15.4%	Pre 16.7% - Post 23.1%	Pre 58.3% - Post 61.5%	Pre 3.1 - Post 3.3
Outside of class, I plan on reading a STEM-related book in the next year	Pre 41.7% - Post 46.2%	Pre 25.0% - Post 23.1%	Pre 25.0% - Post 23.1%	Pre 8.3% - Post 7.7%	Pre 2.0 - Post 1.9
I am interested in pursuing a career in a STEM field	Pre 58.3% - Post 53.8%	Pre 8.3% - Post 7.7%	Pre 25.0% - Post 38.5%	Pre 8.3% - Post 0.0%	Pre 1.8 - Post 1.8

Please note: Pre-assessment N = 12; Post-assessment N = 13

In several of the assessment items, participants responded to and provided comments about four statements.

Statement 1: "I do the same activities in science class as the sighted kids." The number of participants who gave each response is shown in parentheses.
Responses: “Never” (2), “Some of the time” (2), “Most of the time” (2), and “Always” (7).
Comments: Participants who did not always do the same activity reported that they did the following:

Caught up on school work
Teachers had low expectations, didn't want me to burn myself or get chemicals on me.
Was expelled from the science program.
Play video games and chemistry kits
Didn't do the activity.
Given an alternative assignment.

Statement 2: “My teacher has made adaptations to science labs or activities so that I can participate.”

Reponses: “Yes” n = 11, 84.6%; “No” n = 2, 15.4%

Comments: “Accessible models of DNA/RNA”; “tactile punnett squares”; “taking down offensive chemistry posters, e.g. if you don't wear goggles you will go blind”; “uses computer”; “use popsicle sticks for diagrams and braille for the corresponding numbers”; “uses a microscope to help me see during labs”; “large print”; “she reads the board”; “gets more powerful microscope”; “makes the print bigger font”; “teacher let me feel the bag of gas and let me get materials”; “would help measure stuff”; “let me slide on doing maps”; “excluded from the activity”; “labeled beakers in black marker”

Statement 3: “There are activities in science class that frustrate me.”

Reponses: “Yes” n = 7; 53.8%; “No” n = 6; 46.2%

Comments: “Teacher has tried to adapt APH materials but in a way that is not useful”; “a lot of calculating”; “other classmates wouldn't let me do much when working in groups”; “looking through devices to see small things like DNA hurts my eyes”; “taking notes when the teacher won't let me touch something is frustrating”; “using the microscope, measuring, plotting points, and diagrams”

Participants also responded to and provided comments about four questions.

Question 1: “Which areas of science most interest you?” Some participants mentioned more than one area. The number of participants who gave each response is shown in parentheses after the name of the science area. Responses before the SASS program included the following:

Anatomy (1), anything medical (1), astronomy (2), biology (4), chemistry (6), engineering (1), evolution (1), math (1), natural science (2), neurosciences (1), physics (4), weather (1), and zoology (1)

Responses after the SASS program included the following:

Astronomy (3), biology (7), chemistry (5), math (1), medical science (1), meteorology (1), natural science (1), physics (2), technology (1), and zoology (1)

Question 2: “Which science class have you taken most recently?”

Responses: biology (5), general science (3), physics (1), physical science (1), doesn't know (1), N/A (1)

Question 3: “What science class are you planning on taking this fall?”

Responses: chemistry (5), biology (2), unsure (3), multiple unit science class (1), physics or environmental science (1), regular 9th grade science (1)

Question 4: “What assistive technology do you use regularly?”

Responses: Windows computer +/- magnification, NVDA screenreader, BrailleNote Apex, monocle, fancy sunglasses, voiceover on iPod, CCTV, magnifying glass, phone +/- large print (iPhone, Android), Large Print, iPad +/- magnification, Zoomtext

Weather Knowledge Pre- and Post-Assessment

The pre- and post-assessments of weather knowledge were administered by staff at the same times as the assessments of science attitudes and perceptions. A staff member asked each participant individually to respond true or false to ten statements. The assessment identified a statistically significant increase in weather knowledge. Twelve participants participated in both the pre- and post-assessments. The questions and number of correct responses are shown in Table 3.

Table 3. Results of Pre- and Post-Assessment of Weather Knowledge

Statements	Pre-test correct (n = 12)	Post-test Correct (n = 13)	Percent change in this item
An anemometer is used to measure humidity. (False)	4 (33.3%)	10 (77%)	+43.7%
A tornado watch means that a tornado is imminent and you should seek shelter immediately. (False)	5 (41.7%)	6 (46.2%)	+4.5%
The National Weather Service is the government organization that creates weather forecasts and issues warnings. (True)	12 (100%)	13 (100%)	0%
Sound travels further in winter than in summer. (True)	3 (25%)	10 (77%)	+52.0%
Climate refers to how the atmosphere "behaves" over relatively long periods of time. (True)	11 (91.7%)	12 (92.3%)	+0.6%
The deadliest flood in the United States took place in Johnstown, Pennsylvania in 1889. (True)	8 (66.7%)	6 (46.2%)	-20.5%
A cumulonimbus cloud would typically be seen during a thunderstorm. (True)	6 (50%)	11 (84.6%)	+34.6%
The deadliest tornado in Michigan happened in 1953 in Benton Harbor. (False)	7 (58.3%)	5 (38.5%)	-19.8%
In the Northern hemisphere, you are closer to the sun in the winter than in the summer. (True)	5 (41.7%)	11 (84.6%)	+42.9%
During the spring, it is generally warmer near Lake Michigan than it is further inland. (False)	6 (50%)	3 (23.1%)	-26.9%

SASS participants increased their average score on the SASS activity-related weather knowledge assessment [t(11) = 2.56, p = 0.026], from the pre-assessment average score of 56.0% to the post-assessment average of 67.0%. This increase is statistically significant and reflects a large effect size, d = 1.102 (Field, 2009).

Qualification for Workforce Investment Opportunities Act

The SASS program included features that qualified the program for financial support from the state rehabilitation agency. The Workforce Investment Opportunities Act mandates services to transition age youth, including the following which took place at SASS:

Conclusion

The 2017 SASS program evaluation had three purposes: to document increases in science knowledge and positive attitudes toward science, to inform stakeholders including program funders, and to improve implementation and evaluation of future programs. The first purpose has been achieved: pre- and post-assessment results were tabulated, documenting an increase in weather knowledge and changes in science attitudes and perceptions. The second purpose will be achieved with the dissemination of this report. The third purpose will be achieved as the evaluation is used by staff to develop the 2018 program and its evaluation. The team has used the evaluation report to expand understanding of program planning and learning design after each year's program.

References

Field, A. (2009). Discovering Statistics Using SPSS (3rd ed). London, England: Sage Publications, Ltd.

Texas ECC Committee, Texas School for the Blind and Visually Impaired. (2014). What is the Expanded Core Curriculum? Retrieved from http://www.tsbvi.edu/math