Peggy Elliott

Peggy Elliott

What's Round and Mean?
Standardized Testing and Blind Students

by Peggy Elliott

                                                                             

            From the Editor: A little over a year ago, at the 1999 Mid-Winter Conference of the National Association of Blind Students, Peggy Elliott gave a speech on the subject of standardized testing and the problems faced by blind test-takers. It was the clearest explication of the situation I, at least, had ever heard. Conference planners returned to the subject on this year's agenda. Because of the importance of the topic, here is Peggy's original speech. Next month we will carry the follow-up agenda item from this year's seminar. Peggy Elliott is Second Vice President of the National Federation of the Blind and President of the NFB of Iowa. This is what she said:

                                                                    **********

            The best description I know of standardized testing as applied to blind people is the old joke: What is round and mean? A vicious circle. I have discussed this topic with hundreds of blind people over the years, and it seems to me that those facing standardized testing need to know what is going to happen, why it is going to happen, the inevitability of experiencing one or more of the problems others have already experienced, and some techniques for positioning yourself for maximum benefit and minimum hassle. With this base of knowledge you can then consider what you personally and we as the Federation can do about standardized testing. My observation is that most blind people experience one or more problems in the standardized testing sphere, soldier through as best they can, achieve the goal in some way, and then move on, leaving the next blind person to experience the exact same set of problems anew. I think it is time to begin to pool our knowledge, our experience, and our wit to find solutions that will be effective for all blind people.

            Any blind person needs to grasp three concepts to understand the context of standardized testing as we experience it. These are validation theory, the tip letter, and the gateway. As we discuss each of these three concepts in turn, please keep in mind what is round and mean. It will help in understanding the context.

            Validation theory for standardized testing is arcane, complicated, understood by only a few people, and will hit you in the face every time you walk into a standardized testing setting. Therefore I recommend very strongly that at a minimum you study the basic concept of validation theory. It is one that you will be dealing with from now until you no longer need the services of any standardized testing agency. And on that bright day I hope that you are still interested in the effect of this validation theory on fellow blind people and will be willing to continue working to effect change in the current, unacceptable circumstances.

            Validation theory holds that the administrator of a test can determine and vouch for the validity of a score to the user of the score. In this context validity is equivalent to predictive effect. The user of the test score wants a prediction of how you, the test taker, will perform in the thing for which you are being tested, and standardized test administrators claim to be able to provide this prediction. So, for most practical purposes, validity equals prediction.

            The test administrator provides this validated score on a standardized test by a three-step process which will remind us of what is round and mean. First, the tester awards you a score on the test. Second, the test administrator tracks your actual performance in the context for which the test was created. And third, the administrator compares actual performance to score, adjusting the interpretation of the score so that it accurately predicts how someone else with the exact same score will perform in the context.

            Let me untangle that a little bit by giving a concrete example. Colleges and universities almost all use either the SAT or the ACT test for admission into undergraduate school. The people who own and administer those tests have given them to hundreds and hundreds of thousands of high school students and then gathered data about the performance of those same students as they complete their freshman year of college. The SAT and ACT people then compare the two (score and performance) and adjust their interpretation of the test score so that the next group of people that comes along is awarded a score that is now validated to predict performance in that first year of college.

            Another way of saying the same thing is that a certain score received by thousands of students is statistically correlated to a certain level of performance in a college freshman year. This correlation is then provided to score users as a valid predictor of the performance for persons who have taken the test and not yet entered college. College admissions offices can then make decisions based on this predictive score, confident that the prediction will largely be valid since it has been validated through the correlation process using hundreds of thousands of comparisons between score and performance.

            This is the bare bones of validation theory. There is a great deal more to the detail and application of the theory. For example, test administrators include unvalidated questions or sections in the test everyone takes and are always performing this validation process on those unvalidated sections. Some unvalidated questions or sections are tossed out as unusable or non-predictive. These unvalidated questions or sections do not get reported in your score. They're for the benefit of future test users. You are just a cog in that round and mean wheel of validation theory. But the basic progression is score, performance, correlation, prediction, use as predictor.

            The statistical underpinning of validation theory means that it is not valid on a small set of people. For a score's predictive value to be valid, it must be validated on sets of hundreds of thousands of people. Validation theory for a standardized test holds that, if you test enough people and then check their performance in relation to their original scores, you can make the score predictive. For example, the Library of Congress' test concerning Braille literacy is not validated, as I understand it, because they haven't had enough people take their test to look at what the people actually do and relate it back to the score on the test. Validation theory is much more complicated to apply and use than what I have just said, but that's the basic theorem.

            Validation theory holds that all the conditions under which the test is administered must be standard or, in other words, as much the same as you can get them, for the score to be susceptible of correlation with performance in these huge, statistically significant sets of test takers. Everyone is familiar with many of the so-called standardized conditions. You sit in the same room as everyone else, use a No. 2 pencil, use your eyes to read the test, sit at a desk in front of and behind other people, are supervised and proctored by exactly one person who sits at a desk in front of the room. You can add a bunch of other subparts. Basically you have to be one of the herd. You have to be part of that big group that is tested and whose scores are then compared to your actual performance in the first year of college. One person, five people, ten people do not a validation yield. It has to be hundreds of thousands of people, and the conditions under which the test is taken are part of what is considered necessary for the scoring, for the prediction, and then for the validation.

            If you happen to be an Orthodox Jew who can't take the test on a Saturday, your score will not be validated because you didn't take the test in the room, under the same lights, with people sitting around you in the desks. You didn't take it in the same conditions. You can be as sighted as you want, but the test is not validated because you did not take it under all the standardized conditions. Or consider people who use wheelchairs who can read their tests but can't get to the testing room because there happen to be steps in the way. When wheelchair users take the test in a different room, same day, same time, exact same everything except that they are not in the room with all the other people, those wheelchair users will not have their tests validated because the conditions of test administration are not the standardized conditions. "Standardized conditions" doesn't just mean using your eyes. "Standardized conditions" means all of the conditions--the herd conditions.

            Validation theory is precise, specific, and based on the scores of masses of people that are used to validate, compare, and predict. You and I as blind people will never produce a valid score. It can't be done under current validation theory.

            Since the Americans With Disabilities Act was passed, I have heard numerous blind people say in genuine puzzlement and frustration: "But they could just validate the score!" Test Administrators are not going to "just validate" the scores of blind people because validation theory mandates the concepts of large-number correlation and of standardization I have described. There are not hundreds of thousands of us, and we don't sit in the same room and take the test under the same conditions as other people.

            I can understand the frustration that causes blind people to ask why can't our scores just be validated? But I can tell you this: The test owners are not going to. This is round and mean. It's validation theory, and blind people do not fit anymore than people who are Orthodox Jews and can't take tests on the same day. Anymore than people who read the test in print and happen to be in a wheelchair in a different room. It doesn't matter who you are as long as you don't take the test under exactly the same conditions as the herd, as the hundreds of thousands of people. Your score will not be valid.

            Why does it matter that your test score is not valid? The reason for the existence of most standardized tests is to predict for the institution we are seeking to enter how we are going to perform. As a blind person you can simply never produce a valid score. And, unfortunately, the people who developed validation theory aren't governed by the Americans with Disabilities Act. Validation theory is a law unto itself--the law of averages, and as nearly as I can determine, it supersedes the Americans with Disabilities Act. We could litigate this from now until every single person in this room is dead, and the law of averages is still going to supersede the Americans with Disabilities Act. That is a fact of life, no matter what kind of frustration it creates inside you.

            This leads to the second fact that all blind people taking standardized tests will deal with: the tip letter or flagging, as it is currently called. When you as a blind person take a standardized test of any kind, a flag letter is going to be sent to the institution that you specify for receiving your scores. The letter is going to say in so many words that the accompanying score is not valid or not validated (the terms are essentially interchangeable). Many blind people do not even know their scores will be flagged. It is vital that you know it so that you know the context in which you are functioning. It does not matter what you do or how much you dislike it. All that matters here is that we have something that is round and mean. Your test score is going to be flagged.

            I want to give you one more concept before we talk about what we can do in response. The final concept I want to discuss is the gateway. Standardized tests are used essentially in two different ways. One is the gateway of admission to a post-secondary academic program. You're trying to get into undergraduate school or graduate school or law school or social work school. You're trying to get into some program for which the number of applicants exceeds the number of places. The principal method devised for sorting out applicants and accepting only some for admission is standardized testing with the predictive value of a valid score behind it. So the gateway of admission to a program is one type of standardized test.

            The second type is a gateway to licensure. We have all heard of the bar exam and the social work exam. Many professions have such exams, and many of these are standardized and function as a gateway. We therefore have gateways to admission and gateways to licensure.

            A gateway to licensure is pretty much a single-option gateway. There's only one way into the profession in question, and that is to pass the test. These same single-option gateways are now beginning to show up in elementary and secondary schools, used to determine admission to programs such as gifted and talented programs or to determine eligibility for high school graduation.

            However, with regard to a gateway to admission, please consider the concept that a gateway test is not the only way to gain admission to a program or course of study. Let's take for example the few colleges and universities around the country that do not award letter or number grades. You can actually get into some schools in which you earn merely a pass or fail in college. In other words, when graduates of such schools apply for graduate school, they do not have grades to present. Yet graduates of these colleges do achieve admission into graduate schools. How do they do it? They deal directly with the admissions department for the school and say, "I don't have grades. But there's some way you can fairly assess whether I'm a good candidate for your program or not despite the fact that I do not have any grades."

            I want to commend to each of you the like thought for us as blind people appearing at a gateway without a validated test score. The objective here, please keep in mind, is not to take the test. I cannot tell you how many people I have talked to who have said to me, "But I want to take the test. I just want to be like other people."

            I don't want to be mean about this, but I don't care how much you want to be like other people. Because of validation theory, you're not when it comes to standardized tests. You're blind, and your score is never going to be validated. So talk to me about getting into the program. Don't talk to me about wanting to take the test. Don't talk to me about just wanting to be like other people. Who cares if you take the test or not? The question is: Did you get into the program you wanted? Please, please keep in mind that there is more than one gateway into an academic program. Please also keep in mind that you do not have to go through the same gateway that other people go through to get into that program. The smarter you are, the more persistent you are, the better you network, the more you think about it, the likelier it is that you'll figure out that there is more than one gateway. And I want to commend to each of you that, in the case of gateways to admission to an academic program, you don't have to take the test to get in. You cannot get a valid score anyway.

            These then are the three concepts that you need to have in your hands before you, as an individual blind person, can make decisions about what you are going to do regarding standardized testing. Your scores are not going to be validated. Your scores are going to be flagged. There are two different types of testing, for admission and for licensure. With those three concepts here are some things to think about in the case of standardized testing.

            Many standardized tests are not fair to blind people. I've gotten some large national test administrators to admit in private that blind people as a class test lower than sighted people do. I can believe that because of lack of training and skills on our part. If you tell the admissions department that your score is not going to be valid, they will already know it because of validation theory, assuming they know you are blind. Then, if your score comes in a little lower than you would have liked, guess what? You've already undermined the score, and they are probably going to ignore or discount it.

            The opposite, however, is not true; if you score very high, they'll believe it and possibly even rate you yet higher since, in their unspoken view, you had to overcome such obstacles to do well. It always helps to scope out how others will think and use it to your advantage. Boldly pointing out that your score will be invalid can't hurt, may help, and classifies you as proactive rather than reactive--someone knowledgeable about and in charge of his or her world instead of crushed by it. It's to your advantage to recognize that you are different in this context and to make a different approach to getting in. The objective is not to take the test. The objective is to get into the program and get the chance to show what you can do.

            Now I want to back up one step and reiterate what I just said. I think one of the biggest problems that blind people have in taking standardized tests is that as a class our skills are still not as strong as those of any randomly selected class of sighted people. How many people in this room didn't start learning Braille until you were in high school or college? How many of you really prefer to take the standardized test using a tape recorder or reader because aural administration is currently your best test-taking skill? How many of you are really comfortable with your Braille skills and are sure that you could be as competitive in Braille as you could be using print? How many of you are proud of your Braille and are working very hard on it but, when it comes to a gateway test, are not going to trust your future to your Braille skills? Nobody is answering me out loud, but you should think about what I am saying.

            When you get to that gateway, you're better off if your skills are strong and you can choose any of the options offered. The biggest problem we have at test time is the method of presentation of the test. How many of us go to a test and say we need extra time because we are using tape recorders, and that takes more time? Or we're using Braille, and we're not quite as good and can't be sure that we can rely on our skills to get us done in the same amount of time? I submit to you, and I want you to take this home and think about it, that we don't need extra time. Let me say that again. We don't need extra time. We need the skills before we step into the room. Take that one home and think about it. Argue about it among yourselves. Do we need extra time to perform all the regular tasks that we are asked to perform when we get a job? We better not get used to extra time and better not ask for it on the job.

            The skills that you're going to take into the room where you're going to take that standardized test are probably weaker as a class than the skills that sighted people take into that room. Think about that. Don't blame it on the test. It's very convenient, I know, to blame weak skills on the testing authority, to blame them for not offering you the right type or level of accommodation. That's convenient. It feels good. Is it honest? Is it fair? Think about it, and work on your skills. But don't blame the messenger. If your skills are not fully up to your being competitive, don't say that the fault is in the administration of a standardized test. Find ways to work around the problem, but don't fib to yourself about it.

            When you get to the test itself, if you need to take it, if you want to take it, or if you're dealing with a gateway to licensure where you really don't have a choice, this is where another set of problems comes out. How are you going to take the test? We know it's not going to be valid. We know that the person who receives your score is going to be told that it's not valid. But for whatever reason you still have to take it. What do you do?

            Please don't ever tell anybody that the Americans with Disabilities Act is there to take care of this testing situation for you--that the Americans with Disabilities Act gives you this, provides for that. If you start down that road, then oh, I wish you good luck for the rest of your life because you will always be looking for somebody to take care of the hard stuff. Do it yourself. Don't count on the testing authority or the testing site to arrange that the test and the correct accommodations will be there.  Whatever your choice of medium, make sure yourself that whatever is needed is there. Make sure that you push all of the buttons and jump through whatever hoops need jumping through.

            I've heard many testing horror stories in which blind test takers themselves could have taken the initiative and solved the problem before it occurred. It's your job as the blind test taker to be sure that the testing situation is as completely handled as you can get it. It's not the job of the testing authorities, and it's most certainly not the job of the Americans with Disabilities Act. It's your job. So call that testing authority. Bug them. Become absolutely insufferable until you're sure that you're going to get exactly what you want. And make your choice carefully. Assess your own skills. As I say, you should have that full quiver of skills. But you probably don't. So pick your best one and be sure that you get it.

            The last topic I want to talk about is the one that creates the most fierce difficulties for blind test takers. What if you choose a reader as your method of taking the test? Your skills in Braille just aren't good enough. Tape recorders are so cumbersome. Your sight isn't good enough; a CCTV isn't good enough. Your personal assessment of your skills yields the conclusion that your best choice is a reader. That's where most of the worst testing problems begin.

            When you show up at the testing site, they have a reader who's inadequate if they have one at all. In this no-reader situation, they'll recruit one on the spot, and you can be sure that he or she will be inadequate. What do you do? Remember in these difficult situations that the Americans with Disabilities Act gives you a right to reasonable accommodation, the technical term that means I get a reader, or I get Braille, or I get tape. You've done everything you can ahead of time. But keep this simple fact in mind. The United States Department of Justice and everybody who wrote the regulations that implement the Americans with Disabilities Act forgot one simple thing. What they forgot or didn't take the trouble to find out is that your own reader is the most qualified reader. I've been around, around, and around that vicious circle with the Department of Justice on this very topic.

            This is what we may be taking into federal court, the issue that when you bring your own reader to a test site, you have a right to use that reader. Nowhere in federal law is a qualified reader defined as your own choice of reader. Many times, by persistence and self-advocacy, you can overcome this omission in federal law, but not always. Some of you are going to end up in testing situations in which the testing authority says, "I'm going to pick your reader." This is probably the single most important thing we as blind people can change right now--to find a way to get the proposition that "qualified reader" equals my choice of reader into federal law. Until that's true, I can make the absolute, unequivocal statement without fear of contradiction that the Americans with Disabilities Act has hurt blind people because there are a lot of us who have suffered the injustice of having a test site administrator say, you are going to use this reader we have provided, a reader who proves to be inadequate, to put it mildly.

            I'm going to say one last thing about standardized testing, and I'm guessing that it will stimulate discussion. I just had another round of debate with a Department of Justice lawyer about a week ago during which he said that the testing authority with which my husband Doug is dealing has never had any complaints about its handling of blind test takers. According to the attorney from Justice, Doug is the first and only blind person who has objected to his treatment as a blind test taker. The testing authority, by the way, says very sanctimoniously that it accommodated the blind long before the passage of ADA, and it knows very well what the blind need because of its long record. So where is the evidence that we blind people are not getting qualified readers in the standardized tests that we are taking?

            We're going to have to start generating that evidence. As I explained to the Department of Justice lawyer, once we pass the gateway, it is common for blind people simply to move on and put the experience behind them. The experience is often a miserable one; sometimes we pass through the gateway even though we have a bad reader; sometimes we simply back up and go a different way, entering through a different door or choosing to put our efforts into some other project because we can't get past that gateway. But I ask you: how many of you in this room and how many people back home have had a miserable experience with an imposed reader and simply walked away from the experience once it was over? Whether you pass that test or whether you fail it, if you have a reader not of your choice, we need to start documenting the poor quality of those readers.

            The only qualified reader, in my opinion, is the one that you choose. But we are not going to have that option unless we as blind people stand up and say that, in the case of readers, the only qualified reader is the one I choose. At the absolute worst, I should have the choice to bring my own reader or have one assigned. The choice should always be mine. Of course, when bringing a personal reader, we all expect to be proctored by someone from the testing authority sitting there, watching to make sure that you don't cheat, as they say. But, if we are ever going to get the choice under the Americans with Disabilities Act to bring our own readers, people in this room are going to have to do it. Right now the law says that a qualified reader is a reader that someone else chooses for you.

            I hope I've said provocative things about standardized testing. Maybe I've also said helpful things. There is no magic answer to this question. I think it is time for all of us to start discussing the framework and context of standardized testing so that we can make our own individual choices as informed citizens. And I also think it is time for us to start discussing this topic as an organization so that we can forge solutions to this difficult problem that we have for so long confronted as individuals and then moved on with our lives, never looking back on what is usually an unpleasant experience for everyone but doubly so for a blind person. Please remember that there are two basic things to keep in mind about standardized testing: we're going to do it better if we work together as Federationists to solve the problem, and the whole thing is round and mean.

                                                                    **********