Braille Test

Braille Test

Content Validity of the National

Literary Braille Competency Test

by Carol B. Allman and Sandra Lewis

From the Editor: Because Braille users and those who

wish they had been taught to use it have such strong

convictions about the importance of effective teaching of

the code to children, members of the National Federation of

the Blind have worked to ensure that teachers of the

visually impaired know the code well themselves so that they

can teach it. Unfortunately, a number of teachers have

opposed our efforts. They offer a variety of arguments in

support of their position, but we have been made skeptical

through the years by transparently poor teaching of Braille

and, too often, a rigid determination to teach print if at

all possible. In short we have become convinced that

insecurity and fear underlie a large part of the teacher

resistance to the movement toward demonstrated competency in

Braille reading and writing for teachers of blind students.

A few months ago word began to circulate about an

astonishing article that had appeared in the Fall, 1996,

issue of RE:view, the journal of the Association for

Education and Rehabilitation of the Blind and Visually

Impaired (AER). Here is the article by Carol Allman and

Sandra Lewis as it appeared:

Criticism of teacher competence in using and teaching

Braille contributed to the start of a Braille literacy

movement in the 1980's. Because of the Braille movement and

the general agreement that Braille is a literary code of

importance for some people with a visual impairment, twenty-

five states [now twenty-eight], including Florida, have

passed "Braille Bills" (Turco, 1993; personal communication,

B. Pierce, April 13, 1994). Such legislation reiterates the

importance of Braille for some students with severe visual

impairments and, in most cases, requires testing the Braille

competence of teachers of students with visual impairments.

As a result of the Braille literacy movement, the

Braille section of the National Library Service for the

Blind and Physically Impaired (NLS) of the Library of

Congress has developed the National Literary Braille

Competency Test (NLBCT), a criterion-referenced test that

assesses skill in reading and writing (transcription)

Braille. No other test of this kind exists, and states that

have passed Braille legislation have considered using the

test for one aspect of certifying teachers of students with

visual impairments. If the test, which has not been used as

yet, is to be considered for partial use in teacher

certification, its content must be determined to be valid.

In 1989, national organizations for the blind (American

Council of the Blind, American Foundation for the Blind,

Association for Education and Rehabilitation of the Blind

and Visually Impaired, Blinded Veterans Association,

Canadian Council of the Blind, Canadian National Institute

for the Blind, National Federation of the Blind, and the

NLS) formed the Joint Organizational Effort (JOE) in part to

promote Braille literacy. The group chose the NLS to devise

a competency test because it had expertise in Braille codes

and no affiliation with teachers, universities, or other

education or rehabilitation organizations (National Library

Service, 1993).

An editorial committee of eleven professionals involved

in education or rehabilitation for blind and visually

impaired persons was formed in 1991 to advise on test

development. The committee recommended limiting the test to

literary Braille (excluding math or music codes), including

slate and stylus writing, and not testing Braille teaching

methodology. It recommended that university training

programs assure proficiency in teaching methodology testing

through certification standards.

In the spring of 1992, the editorial committee reviewed

a trial test, which NLS revised on the basis of that

evaluation. Subsequently, thirty-two peer reviewers (64

percent return rate) in fifteen states evaluated the test.

The editorial committee or NLS selected those reviewers from

a list of individuals who had expressed an unsolicited

interest in reviewing the test. Their responses were

positive; most agreed that slate writing was important,

although a few thought it unnecessary. Most recommended more

multiple-choice questions. NLS revised the test based on

these recommendations (Stark, 1993b).

The NLBCT is described in news releases (National

Library Service, 1992, 1993, 1994) as a three-part

evaluation of general knowledge of the Braille literary

code. The test assesses (a) the ability to write by using a

slate and stylus to Braille one medium-length paragraph, and

a Braillewriter to transcribe one full print page, and (b)

the ability to identify Braille errors in four medium

Braille paragraphs. It also requires the candidate to answer

twenty-five questions on the use of Braille rules.

Candidates can use a dictionary, but not Braille reference

materials, to complete the test within four to six hours.

NLS will grade the tests and set passing scores. NLS has set

prerequisites for taking the test the first time and

guidelines for subsequently retaking it.

The Braille literacy concerns of JOE indicate that

professionals and consumers in the field generally support

the concept of a Braille competency test. However, the NLBCT

has not been rigorously validated, and testing and

measurement specialists agree that assessments used to

obtain teaching certification should have psychometric

characteristics that include assurance that the instrument

used has job relevance (Gorth and Chernoff, 1986). The

measurement literature on validation of teacher

certification tests, although limited (Schmitt and Borman,

1993), supports the need for content validation,

particularly for tests like the NLBCT that are

criterion-referenced tests of skills used in teacher

certification (Shimberg, 1981).

Unfortunately, the current development of the NLBCT

consists of expert judging and peer review based on personal

expertise (Stark, 1993a) and not of job analysis data. The

recommendation by peer reviewers to assess only knowledge of

the Braille code and not methodology raises particularly the

question of the need for teachers to demonstrate ability to

transcribe materials using a slate and stylus. There is

agreement in the literature that teachers should teach slate

and stylus to students. However, if the NLBCT is designed to

assess demonstration of the Braille code and not teaching of

Braille and related communication skills, the requirement of

slate and stylus writing is questionable.

Issues Surrounding Content Validity

Technical adequacy of any test through the use of

psychometric techniques is considered standard procedure as

outlined in the Standards for Educational and Psychological

Testing (American Educational Research Association, American

Psychological Association, and National Council on

Measurement in Education, 1985). Primary standards include

evidence of validity. Validity of criterion-referenced tests

is widely discussed in the literature, but content validity

is generally recommended as the primary validation of

interest. For teacher certification purposes, test content

validation is generally determined through an investigation

of practitioners who either report or demonstrate the skills

tested while on the job.

As prospective teachers are tested for competency, it

is critical that the competence be based on the knowledge,

skill, and ability that is demonstrated by practicing

teachers. The job relevance of testing for certification

purposes is upheld by Standards for Educational and

Psychological Testing (American Educational Research

Association et al., 1985) and the Equal Employment

Opportunity Commission's overview of the Adoption by Four

Agencies of Uniform Guidelines on Employee Selection

Procedures (Equal Employment Opportunity Commission, 1978).

The assessment of skills through performance rating of

demonstrated skills has traditionally been focused on

occupational areas that lend themselves to completion of a

product through simulated performance of specified skills,

such as secretarial (typing exams) and mechanical

functioning (car repair, building construction, or

assembly-line skill)(Fleishman, 1982; Hambleton and Rogers,

1991). Typically, teacher skills are not defined in ways

that allow their simulation for skill performance

assessment. However, Braille transcription competence is a

skill that lends itself to a performance rating on a

criterion-referenced test.

Legal opinions support the critical nature of content

validity for teacher certification testing. In a 1971 case,

Griggs v Duke Power Company, employees of Duke Power Company

challenged the legality of an employer's using general

ability tests to hire and advance employees. That landmark

case in personnel testing established the concept of job

relatedness in finding that the general ability tests were

not validated by correlation with job relevant tasks

(Bershoff, 1981). In a 1975 case, Albemarle Paper Company

v Moody, an employer's method of determining validity of an

employee test was found defective. A psychologist hired to

validate an employee test compared test scores of current

employees with supervisors' judgements of competence. The

court stated that the validation did not analyze attributes

or particular skills needed in the job and thus was

defective methodology. (Bershoff, 1981).

Two cases in 1981 further addressed the validity of

employment testing. The United States v City of St. Louis

case challenged multiple-choice questions and simulation

exercises on an employee test developed by a panel of

experts. The court concluded that the items were based on

opinion rather than actual observation of correlation

between mastery of knowledge and abilities measured by the

test. Although the test assessed reading and writing skills

that were dissimilar to those needed in the work situation,

it threatened the use of simulated situations in the testing

situation. In Guardians Association of New York City v Civil

Service Commission, perhaps the most sophisticated opinion

on employment test validity, the court found the functional

approach of job relatedness for content validation to be

appropriate. The court upheld the city's use of content

validation strategies, supporting further use of

job-relevant content validation for tests of teacher

certification. (Bershoff, 1981).

Bishop (1993), Harrell and Curry (1987), Heinze (1986),

Olmstead (1991), and Torres and Corn (1990) provided

descriptions of duties of a teacher of visually impaired and

included the transcription of materials into Braille and the

interlining of Braille materials with print. Teachers may

demonstrate these transcription skills by using a

Braillewriter, a slate and stylus, a Braille computer

program, or a Braille transcriber aide. Any assessment of

these Braille transcription skills must necessarily reflect

on-the-job activities as carried out by teachers.

Unfortunately there are no quantitative studies or job

analyses to suggest how teachers proceed with Braille

transcription duties.

The Present Study

In the present study, we address the content validity

of Braille transcription on the NLBCT. The need for the test

is not at issue. If the test is an instrument for

demonstrating "basic competency in literary Braille"

(National Library Service, 1994, p.1), then proper

validation procedures should show that the test is related

to the job performed by the teacher of students with visual

impairments.

Method

The Questionnaire

We designed a Braille Skills Analysis Questionnaire

(BSAQ), choosing the questions by reviewing the literature

on Braille- related communication skills and by evaluating

the purpose, content, and process of Braille transcription

skills on the NLBCT practice test. We designed the

questionnaire so that the teacher respondents could indicate

how and how often they used Braille transcription skills,

particularly their use of slate and stylus, Braillewriter,

Braille computer programs, and transcriber aides in

transcribing materials and transcribing with the use of

Braille reference materials.

We asked fifteen visual impairment professionals to

review a draft of the BSAQ. Using responses from nine of

those reviewers (60 percent), we revised the draft. We then

asked ten potential participants to complete the revised

test to determine its test-retest reliability. We had

established a priori that a test-retest reliability of .85

would be acceptable for ascertaining that the questionnaire

would produce reliable information. Eight participants (80

percent) returned the completed questionnaire. Two weeks

later we sent a second questionnaire to those participants

who returned the first one. Six of the eight participants

(75 percent) returned the questionnaires for test-retest

reliability computation. We obtained an average test- retest

reliability of .87 from those six responses. The field test

participants were not part of the initial review, and both

groups of participants were deleted from the participant

pool.

Participants

The participants were 233 teachers of students with

visual impairments in Florida, whose names we obtained from

the Florida Instructional Materials Center for the Visually

Handicapped, which maintains an annually updated list of all

teachers of students with visual impairments in Florida.

These individuals would be a source of current on-the-job

information about skill in Braille transcription. The sample

of teacher participants consisted of thirty-two men (14

percent) and 201 women (86 percent).

Procedure

The Florida Department of Education mailed the test and

a return-address, stamped envelope to the 233 teachers. The

tests were coded to the addresses of the participants. The

coding was accessed only by a research assistant who

maintained records on the return of the tests and sent

follow-up letters four weeks after the original mailing to

those who had not completed the questionnaires.

Results

Eighty-one percent (189 of 233) returned the test. Of

those 189, 181 questionnaires (96 percent) were completed in

a usable manner. The eight unusable questionnaires were not

completed because the recipients were no longer teaching.

Twenty-six men (14 percent) and 155 women (86 percent)

completed the questionnaire. Seven participants reported

that they were tactual Braille readers. Seventy-five percent

of the respondents were teachers of visually impaired

children; 7 percent were orientation and mobility

specialists; 14 percent had dual assignments; and 4 percent

gave no identification. Of these teachers, 42 percent served

prekindergarten through secondary school children; the

remaining 58 percent, in about equal proportion, taught

secondary school children only, elementary and secondary

school children, elementary school children only, or some

other combination of ages. Sixty-seven percent of the

teachers reported that they had taught for more than ten

years; 59 percent stated that they had taught students with

visual impairments for more than ten years. The data in

Tables One and Two indicate the locations of teaching

assignments and the number of visually impaired children

each teacher taught.

Teachers who did not use Braille in their teaching

assignments were asked not to answer the remaining

questions. Eighty-nine teachers (49 percent) reported

transcribing Braille; 96 percent of those used a

Braillewriter, 64 percent used computer software, and only

12 percent used a slate and stylus.

We had decided before mailing the tests that to be

reported as content-valid a skill had to be used by 85

percent of transcribing teachers. Based on the report of

eighty-nine teachers in Florida who transcribe materials for

students as part of their current job, the NLBCT skill of

producing print into Braille by a Braillewriter can be

considered content-valid. The NLBCT skill of producing print

into Braille using a slate and stylus is not content-valid

based on the Florida responses. Although the skill of using

a slate and stylus is described in the literature as

desirable for teachers who teach students with visual

impairments, the reproduction of materials into Braille by

that method is not a skill that many teachers in Florida

use. Most teachers who transcribe materials do so with a

Braillewriter or computer software.

Of the 104 teachers responding to the question about

using Braille transcriber aides to transcribe materials,

sixty-five (63 percent) reported not using an aide. However,

sixteen (15 percent) reported using an aide for 3-5 hours a

week; 16 (15 percent) reported using an aide less than 1 to

2 hours; and 7 (7 percent) used an aide for transcription

from 6 hours to more than 10 hours weekly.

One hundred teachers answered the question about using

reference materials for transcription, and eighty-five

teachers answered the question on using reference materials

for interlining Braille with print. The data in Table three

show the frequency of reported use of reference materials.

Table four contains data on the amount of time teachers

spend weekly in interlining and transcribing Braille.

Interpretations of the findings in this study need to

consider the following limitations:

1. Participants in this study were volunteers and may not be

representative of the population of teachers.

2. Participants were limited to the state of Florida.

3. Participants in this study may have previously

participated in some aspect of the NLBCT development.

4. Data from this study are self-reported information and

may reflect the participants' biases.

TABLE One. Teaching Assignments of Respondents

Assignment Number Percent

Residential School 25 14%

Resource Room 24 13%

Itinerant Teaching 99 55%

Special Class 21 12%

Other (supervisor, media spec.) 12 6%

Total 181 100%

TABLE Two. Number of Students With Visual Impairment That

Respondents Serve

Program Mode Range M Mode

Itinerant teaching 2-32 14.1 10

Resource room 1-65 11.5 9

Residential School 3-100 27.4 27

TABLE Three. Frequency of Teacher Use of Braille Reference

Materials

Use Always Sometimes Never

Transcribing Braille Materials 21 70 9

Interlining Print Materials 14 47 24

TABLE Four. Time Teachers Spend per Week (in Hours) in

Transcription

and Interlining

Skill <1 1-2 3-5 6-10 >10

Slate and stylus 10 1 0 0 0

Braillewriter 18 30 21 11 5

Computer Software 7 21 13 6 3

Interlining Braille 9 27 15 7 2

Conclusions and Discussion

We designed the collection of data in this study to

determine if teachers transcribe Braille using a

Braillewriter and a slate and stylus without the use of

reference materials as assessed on the NLBCT. The data from

this survey support the assertion that transcribing Braille

with a Braillewriter is a valid skill to assess as a

certification requirement for prospective teachers of

students with visual impairments. Using a slate and stylus

and transcribing Braille without using reference materials

are not valid components for certification requirements.

Wittenstein (1993a, 1993b) found that over half of the

subjects he surveyed felt that it was not desirable for

teachers to be certified transcribers of Braille. Currently,

Braille transcribers are certified through a test similar to

the NLBCT that requires transcription on a Braillewriter

with use of reference materials and with particular

attention given to format, structure, and lack of errors on

the transcribed document. If teachers are to be assessed on

their ability to transcribe Braille for student use, those

skills should be assessed through ways typically used by

teachers and with attention to lack of errors on the

transcribed document. The data from this study indicate that

transcribing using a Braillewriter and reference materials

is a skill used by over 90 percent of the teachers who use

Braille in their classrooms. Unlike the NLBCT, the

applicants for the Braille transcriber test may use

reference materials and may complete the test in a setting

of their choice.

Over half of the teachers using Braille spent one to

five hours a week interlining print with Braille.

Interlining appears to be a critical skill for a number of

teachers and is necessary in the management of students with

visual impairments in regular classrooms. If regular

classroom teachers are readily to accept students with

visual impairments in their classrooms, they need assurance

that the materials are accessible. This suggestion is

supported by Bishop (1986), who identified factors in the

successful mainstreaming of students with visual

impairments. This finding supports the suggestion that

teacher preparation programs include the skill of

interlining in Braille coursework.

The data from this study indicate that 64 percent of

teachers using Braille in their classrooms transcribe with

computer software. This skill may reflect a future trend and

probably reflects teachers' desires to complete needed

transcription in a timely, simplified fashion. It does not

suggest that teachers are illiterate in the Braille code.

The finding of some use of aides for transcription may

indicate a trend in the use of trained transcribers, which

is supported by Currey and Hatlen (1989), who reported that

teacher "aides are often assigned the job of Braille

transcription and that teachers of the visually impaired are

assigned the job of training those aides in the fundamentals

of Braille transcription" (p.61). This information suggests

that a Braille skill that may need to be included in teacher

competency in the future is the ability to train teacher

aides in the transcription of Braille.

Based on the results of this study, we do not recommend

using the NLBCT in its present format in certifying teachers

of students with visual impairments. Competence in

transcription of Braille on a Braillewriter using reference

materials is a job-relevant, content-valid skill expected of

any teacher of students with visual impairments. In

addition, teachers should be competent in interlining

materials.

Teachers must have the opportunity to continuously

renew and upgrade teaching skills. In this study, we report

that 51 percent of the respondents indicated that they do

not currently use Braille, and often teachers go for several

years with no Braille-reading students; these teachers

require Braille and Braille device updates (Olmstead, 1991).

New and improved technology, methodologies, and materials

become available and require learning or renewing (Maron,

1983). Commonwealth of Virginia (1991) and Wittenstein

(1993a, 1993b) report that although teachers generally feel

confident in their Braille skills, they desire some level of

inservice training on various Braille-related communication

devices.

Based on the findings of this study concerning the

content validity of the NLBCT, the Braille Competency

Committee of the Florida Department of Education recommended

that this test not be used for teacher certification. The

Braille Competency Committee established Braille competence

standards for teachers and recommended that prospective

teachers' competence in Braille be assured through the

content of university courses, including passing an

examination that allows use of reference materials while (a)

transcribing a lengthy passage from print to Braille on the

Braillewriter and (b) interlining Braille to print. This

committee, recognizing that caseloads of teachers may only

sporadically include students who are Braille readers, also

recommended that regular inservice training in Braille be

initiated for teachers who believe that their skills are

rusty. More than 100 individuals participated in four

regional two-day Braille refresher workshops in the spring

of 1996. Future plans are to provide advanced Braille

updating, which would include teacher competence in the use

of software for transcription and the training of teacher

aides to assist in the transcription of Braille.

NLS has announced that it will proceed with a

nationwide effort to validate the content of the NLBCT. Our

research, conducted in only one state, can be used as a

pilot study for the larger investigation. It will be

interesting to see if the transcribing practices of Florida

teachers are similar to those of teachers in other states.

Should the nationwide validation confirm that teachers

primarily use Braillewriters and reference materials when

transcribing materials from print to Braille, it seems

reasonable that the NLBCT can be made more content-valid by

making changes to the testing procedures that reflect these

job-related practices.

In the meantime, states that have adopted the current

version of the NLBCT for teacher certification may want to

reevaluate their decision. If the test lacks content

validity, as determined in this study, continued use of the

NLBCT as a determinant of employability may not be upheld in

the courts.