This is being provided in a rough-draft format. Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.
MEGAN SCHULLER: As some of you may notice Maria is not yet on the stage, but she is on her way and I'm going to slide over so I can manipulate the slides. Are those up for folks? Thank you so much.
Good morning, everyone. When we were asked to do this and I said 8:40? I said, are you sure?
[ Laughter ]
But I am very happy to see all of you. I'm very impressed that you're all here very early and on time. We're excited to talk about AI, the Americans with Disability Act and Public Policy. I have to reach into my bag, my entire presentation is handwritten and on sticky notes which feels appropriate, somehow. I am a white woman in my middle wonderful aged years and have short blonde hair and wearing a black mock turtle neck sweater today. I'm accompanied by Brian on stage and I'll let Brian reintroduce themselves when we get to the discussion portion of the morning.
First I want to start by doing a little table setting and making sure since this an opening plenary, I imagine people with varying levels of AI and its impact on the disability community want to talk about what do we even mean when we're talking about AI and how do these laws apply. So as we've said the speakers this morning are myself, Megan Schuller I use she/her pronouns. I'll be accompanied by Maria Town American Association of People with Disabilities and Brian Dimmick from the ACLU disability rights program. Our objectives this morning are really to create a forum to bring forth the topics that concern the disability community. And how both laws and public policy can address those concerns.
Now this is not the first time for almost anyone in this room that we have confronted a major development in technology that has been life altering and world changing. I can remember when I got our first home computer. I remember we had to tear the little edges off the printer paper and how when I would make big signs for happy birthday I had little X Os to create the big letters. I remember when I got my first laptop, my first cell phone, and very well remember when my now husband told me we should really invest in stock in Apple because they're going to release this iPad. I said that's ridiculous. My mistake. We didn't buy that stock.
We also all know, most of us firsthand, how that technology can help increase access and independence for people with disabilities. As well as how it can isolate and increase discrimination when people with disabilities are not included in the development, the regulation the roll out and the conversation about those technologies. And my hope here is we have an opportunity to perhaps address some of those issues a little earlier on with AI than we did with some other technologies, where we are still hoping to soon see an exciting new regulation about web access.
AI is one of evolving public policy, but what does not get enough attention, and for those who know me, you've heard me say this many times before, is the impact on people with disabilities. There has been a growing discussion about equity and discrimination as a result of AI, but too often disability is not part of that conversation, and the disability community is not part of that conversation. And I think there is a huge opportunity here for us to change that.
How AI is defined in policies in place directly affect the laws that are already established to provide accessibility to the disability including the Americans with Disabilities Act and the Rehabilitation Act. Our objectives are to examine the numerous areas as well as the intersectionality of race and disability discrimination in the AI space. To explain the application and importance of existing civil rights law like the ADA and Rehab Act to address the impacts of AI on the disability community, to discuss whether there's a need for additional regulations and policies that address the impact of AI on the disability community, and to look ahead toward the future, including both the promise and the perils of AI and how we balance those when they're competing.
So what are we talking about? On this slide there's a picture of a red puzzle piece connecting a bunch of white puzzle pieces. So I want to talk about the pieces that make up this broad and frankly vague term that very often gets thrown around of AI for artificial intelligence. So what is AI? The reality is there are many definitions and people often use it without fully necessarily understanding what they mean or what the other person who's talking about it means, and they may have different definitions. I've provided here one definition from President Biden's executive order. It says quote, a machine based system that can, for a given set of human defined objectives, make predictions is, recommendations, or decisions, influencing real or virtual environments. That pretty much clears it up and we all now fully understand what AI is.
[ Laughter ]
So I put up another definition here from the National Institute of Standards and Technology, NIST which defines AI as the capability of a device to perform functions that are normally associated with human intelligence, such as reasoning, learning, and self improvement. That is a little clearer, but note how broad a term that is. A term encompassing all matters from self-driving cars to ChatGPT, to police surveillance to facial recognition technology. That are used to determine who gets a job, who gets a loan, who gets an apartment, who gets held in jail, who gets to keep custody of their child. And who doesn't.
AI is now being used for very high stakes decisions. Often with the outward stated purpose of reducing human and unconscious bias. So I also want to talk about what is AI bias? Because that's going to be a topic of several different sessions today, as well as a lot of our conversation. On this slide there's a picture of an iceberg and as is through with the iceberg, about a third is visible above the surface. Is fairly well known and understood and about two thirds of it is below the surface. Now I'm stealing this analogy also from NIST. Above the surface we have data and training analysis, which there is a lot more conversation about and is better understood. Bias in AI systems is often seen as a tactical problem. But we know, and I know everyone in this room knows very well, that a great deal of AI bias stems from human bias. And systemic and institutional bias.
So I want to talk about each of these briefly. What do we mean when we say data and training biases? We're talking about machine learning processes and data that's used to train AI software. So I want you to imagine that you have software algorithm and you're trying to teach it to identify a dog. And you show it a bunch of pictures of dogs until it can identify what a dog is. But when you're doing that, you're only showing it pictures of big dogs. You've got some golden retrievers, maybe some black labs. It's a very limited data set that has been identified by a human. And then when you show that same AI that is now able to identify many dogs, a very little dog, a Chihuahua, it can't identify that dog. It doesn't recognize that that's a dog at all. This is a very simplified explanation of what we're talking about, but this is what we're talking about with data and training biases. How are you training that AI tool? What's the data set that you're choosing and who's being excluded from that data set?
It often is being trained on data sets that under represent a particular group, perhaps a gender, an ethnic group. I can tell you that almost all data sets, if not all, used in training AI, underrepresent people with disabilities. And I have a colleague who has many, many times said to me that every disability is a disability of one. And that presents unique challenges, and is something we can talk about more during our conversation with Brian and Maria, about how do we address that when we're talking about AI and equity and trying to address AI bias. When everybody disability is also so unique.
The result is that people with disabilities become outliers or are flagged. Now, this Schwartz has said context is everything. AI systems do not operate in isolation. They help people make decisions that directly affect other people's lives. Bias in AI can hurt people. There are very real impacts and we're going to talk about some of those today. So when we talk about this context, you also have to talk about human biases. Choosing the data set as I just talked about. Also choosing the variables that go into an algorithm. How each variable is weighted. What's the question that the algorithm is even being asked to decide?
We'll talk a little about the family regulatory system and the tools that are being used to screen calls of purported neglect to do with children. And there is a tool in Allegheny County, Pennsylvania, where there's been a lot of reporting about it and there's an unusual amount of information about it. The question in that tool has been asked is not what's the likelihood of harm to this child. The question is what's the likelihood that this parent is going to have their child removed in the next two years, based on the people we we've removed children from in the past. Not rightly removed children from in the past, not where we found was a real risk to the child. We all know very well that the answer to that question, the correct answer for the algorithm to provide you, is going to be Black and Brown people, low income people, and people with disabilities. And now you’re asking the algorithm to zone in only on that subset of the population, and flag them as high risk.
So these are the kinds of human biases that we need to think about and talk about. And that also raises systemic biases and institutional biases throughout our see site that operate in ways that disadvantage certain groups including discrimination based on race, income, disability, as well as other protected classes. Now, I have been also, as we just talked about terminology, alternating back and forth between AI and algorithms. And there is a difference. Some algorithms can be AI, some are not. AI is made up of algorithms, but let's talk about algorithms versus AI and what's the difference, at a very, very high simple level.
I put up here a picture of a car and wrote cars versus self driving cars. The algorithm is the car. The AI is the self driving car that operates more like a human and can think and can do human operations, as opposed to being a tool humans use to be more refigures, to automate a human function. Perhaps a better analogy is a chef. Recipes versus robot chefs. An algorithm is effectively a lot like a recipe where you plug in information or variables and it will spit out, it may be the same answer for algorithms that have been used for decades. Other algorithms can use machine learning and can provide different answers each time but it's effectively a human recipe that's automating a human function. Versus the robot chef that's taking lots of information, is learning, maybe it will make adjustments to the recipe as it goes and improve upon it over time.
So when we talk about algorithms versus AI, I think it's an important distinction when we talk about how current civil rights laws apply, and about public policy and regulation in this space. Because there is a very significant difference between some of these classic statistical algorithms that are effectively doing the same thing that a lot of people have been doing filling out a form manually in the past and now they're doing it through a computer based algorithm, versus when we're talking about a self driving car or ChatGPT which is a very different type of AI with different considerations.
So how does AI impact the disability community? I've sort of already gotten to the punchline ahead of myself. In everything we do. And here again we have a picture of that red puzzle piece connected to a bunch of white puzzle pieces, but now I want you to imagine that that red puzzle piece is the AI, and the white puzzle pieces are all of our daily activities. And every societal and governmental system. AI is connected to all of them, and whether we like it or not, it's here to stay. And it's not going anywhere and it's growing and it's expanding and developing rapidly.
The next slide we have a image that many of you are going to be very familiar with of the Disability Rights Movement and of a large group of people with disabilities who are demonstrating and holding a sign that says: Injustice anywhere is a threat to justice everywhere, quoting Martin Luther kin king junior. Well, when we're talking about AI, we're more or less talking about injustice everywhere. These tools are everywhere now, and most of them, when it comes to the disability community, are not yet taking into account how that is impacting people with disabilities and the bias that's baked in.
So as a quick example, before we get to our broader discussion, I want to talk about AI in the criminal legal system. And some of you may or may not be familiar with the sequential intercept model, but the imagine up here to depict that and it's a circle with a bunch of bubbles that are connected by arrows. At step zero we have community services, step 1 is law enforcement with the image of a police officer. Step 2 is the initial detention and initial court hearings with an image there of handcuffs.
And step 3 we have the jails and courts and a picture of a gavel. Step 4 we have reentry and the image of a calendar and step 5 we have community corrections with an image of a house.
And I put this up here, because AI and algorithms are being used at every single step of the criminal legal system. And I've put up just a few examples that came quickly to mind as I thought about each step. When we're talking about community services, AI is being used for benefits determinations. It's being used for Medicaid and Medicare benefits determinations. It's being used to determine what services you get in the community and what the level.
At step 1 we have predictive policing and surveillance tools which are being used in cities and jurisdictions around the country, as well as more often in schools. And you can think about a student that perhaps has diabetes and needs to regularly go to perhaps a bathroom or nurse's office they may carry a device to check their insulin and it may flag them for unusual behavior or for having an object that is flagged as being potentially threatening.
We've got step 2, with pretrial sentencing tools. And this is what I want to quickly pause on. There's a great Ted Talk X that I like to share with people who are new to this topic, by Honey freed who talks about pretrial sentencing tools how they are using algorithms to make decisions in courtrooms. One popular widespread algorithm used to make bail decisions at the point of arrest, effectively gets information from the arrestee, they feed it into a computer algorithm, and it outputs a risk factor or a risk score for that individual that is meant to quantify the likelihood that that person is going to commit a crime or fail to appear for their next court hearing. It's effectively a very simple minority report, without the people in pools of water.
If you're flagged as high risk, it means you're going to stay in jail. More or less, I'm simplifying this a little. If you're low risk, you get bail. There was a well known Pro Publica report about a very popular version of this tool being used around the country, that found that for individuals with sorry. For individuals who are Black, there was almost two times the likelihood that they would get a false positive, would be flagged as likely to commit a crime when they did not.
For white individuals, who were screened using this tool, they were almost two times as likely to be identified as not likely to commit a crime when they then, in fact, did.
The algorithm did not know the race of the individuals, so it raises the question of how is this happening. And it raises the issues of proxies which is another piece of the puzzle that we need to be aware of and talk about. Unfortunately, for people with disabilities, a lot of the team, in fact, these tools are very overtly using disability as a risk factor or a variable, but there's also a lot of proxies for race, for disability, for other protected classes. And those are a part of this decision that we're having about how AI can perpetuate systemic and institutional biases, the ways in which our society is already set up in such a way that if you are a Black person, or a person with a disability, you are more likely to have an encounter with law enforcement.
You are more likely to experience being unhoused or homeless. We all know that people with disabilities are not fairly represented in employment and don't have equal access to employment. Those are the factors that these tools are taking into account. Employment, criminal history. Whether or not you have stable housing.
And other ways that this can come up for people with disabilities, is also that this tool takes into account is if you have previously failed to appear. But I can assure you it's not checking whether you have failed to appear because are a person who's blind who didn't receive notice of your court hearing in an accessible manner, so you didn't know you had a court hearing.
So these are the things that are coming in underlying these tools that we need to be thinking about. We're going to talk more about this, so I do want to getting to discuss another example of this, that I've already touched on, the family regulatory system, that are using predictive risk modeling and we talked a little about the Allegheny Family Screening Tool. This slide has a picture of two parents with their child who are looking very happy. That I took off the website of the Allegheny Family Screening Tool and I think it drives home the fact these stakes could not be higher. These tools are being used for decisions that impact people's lives and whether or not their family remains together.
And for the Allegheny Family Screening Tool they have shared what the risk factors they're taking into account in determining who's high risk getting screened and investigated and it expressly includes whether or not you have a mental health diagnosis, whether or not you've received mental health treatment, those are two separate factors that increase your score. You're higher risk if you've received treatment. It also includes substance use disorder, whether you have a diagnosis of a substance use disorder and whether you've received treatment for a substance use disorder, as well as certain iterations that it's pulled from Medicaid data and Medicare data. And is very clearly, as reported by the ACLU in a report that they released about the tool, based on overt disability factors alone, your risk score on a scale of 0 12 can raise by 3 full points based on the fact that you have a disability. Never mind the numerous other proxies that are included in that tool.
So these are some examples of how AI is impacting the disability community, and we'll talk about a few more with Brian and Maria in a bit. We've also touched on how race and disability intersect in this space. It is critical that we keep in mind the multi marginalization that is happening, and that's being perpetuated by some of these tools. So, for example, when we're talking about the criminal legal system, we know that one study found that Black people with mental health disabilities were more likely to be incarcerated than any other racial group. That pretrial sentencing tool is going to flag a Black person with a mental health disability as high risk, most likely, if we look at the statistics and the facts of our society.
The CCD joint task force comments on a rule noted that Black and indigenous youth placed in institutional settings in the child welfare system stay in care longer, are segregated longer, and have race and disability are deeply intertwined in this space, and there's been a lot of discussion of racial equity and racial bias in these tools, and not enough discussion of how that intersects with disability and that impacts people with disabilities as well. And so we want to have that conversation and raise those intersections. I want to briefly touch on the last puzzle piece here, which is how is current laws apply.
There's a session later on just this, on AI, the law and governance so we're going to do a very high level touch on topic here and folks can go to that session later if you want to talk more about it, but I want to at least touch on how the Americans with Disabilities Act, as well as the rehabilitation act, do apply in this space. ADA prohibits discrimination based on disability. And that includes as recognized by the Supreme Court, discrimination based on assumptions, myths and stereotypes. Discrimination includes using methods of administration that result in discrimination. Think about how that would apply to the use of a tool that would make these decisions. Also you can't use eligibility criteria that screen out or tend to screen out a person with a disability or a class of people with disabilities.
And an entity providing an aid, benefit, or service, including through contractual or licensing arrangements, another topic that we'll touch on in a little bit is on vendors, vendor liability and developers, and whether or not these laws might apply to them, and if not what are their incentives for compliance. But when you're providing a benefit or service, you may not provide that, a benefit or service, in a way that is not as effective in affording an equal opportunity to obtain the same result, or to gain the same benefit, or to reach the same level of achievement as provided to others. And in many instances, that's exactly what these tools are doing. They're providing for an equal opportunity to gain the same benefit, to reach the same level of achievement, for certain individuals, including people with disabilities.
The bottom line here is there is no exception in the ADA for discrimination resulting from AI, or any other type of technology. Automated discrimination is still discrimination.
Now, that's without talking about some of the challenges that come with applying these laws that we're going to talk about those in just a little bit. I also want to briefly flag and touch on some of the emerging laws and regulations that are coming out in this space. The EU artificial intelligence Act was just passed by the European parliament on March 13, 2024. So that is how quickly the law is evolving here. Some folks may be familiar with the fact that president Biden issued an executive order on October 30th of 2023, just last year in the fall. And prior to that, there was the blueprint for an AI Bill of Rights that was issued by the White House Office of Science and Technology which was more of a nonbinding road map.
And I want to very briefly just explain, since it is the latest development, the EU AI act, which notably was approved by an overwhelming majority, with 523 votes in favor. There were only 46 "no" votes and 49 abstentions.
The way that this law is structured, which gives us I think a little food for thought in our conversation, is based on potential risks and level of impacts with certain AI applications actually being fully banned, including emotion recognition in work and schools. Social scoring, predictive policing that is based solely on profiling a person, among others. And most of what we have just talked about and what we're going to talk about later, are also deemed high risk uses of AI, where the law requires them to assess and reduce risks, maintain use locks, be transparent and accurate and ensure human oversight.
And so we can talk a little about some of those ideas for solutions and possibilities for the future. And I see Maria is just walking in at the perfect moment as I finish up my remarks. So what we're going to transition to now is a discussion with Brian and Maria to get a little bit further into some of the challenges, some additional examples, as well as looking towards the future and solutions and opportunities.
So I'm going to give Maria a moment to get settled on stage and then we'll start the conversation. Brian, could you share some other ways that you're seeing AI impact the disability community?
BRIAN DIMMICK: Sure, good. I'm a senior staff attorney with the ACLU disability rights program. I'm a short white male wearing glasses and hair we will call brown ish and great to be here this morning. So I'm going to start with talking about AI and employment and this is an area that's getting a lot of attention when you talk about AI and disability, because it's an area where people see the impacts here. But especially hiring is becoming more and more automated with each passing year. Employers will tell you, hundreds of thousands of applications we've got to come down and screen out and the question is, what are you screening out based on and how are you doing it and are you reinforcing biases by doing that.
So and we the hiring process is being automated and they are bringing in products that use machine learning and algorithms to automate selection and testing. So some examples, resume screeners that look at resumes to bring in candidates, personality assessments. These are things that have been around for years, and in some cases the AI allows you to do it more to do it more efficiently and more selection criterias, but there are basically tools that have been out.
There are also newer tools that are based on AI, for example, things like chat bots have been around getting information, or video interviews or even a test where you play a game.
So these are sort of new technologies, and in addition to AI and governance workshop, there's also a workshop on AI employment this morning, I'm going to try not to steal this and go through this at a pretty high level.
But the first issue with these systems is the way they interact with people with disabilities. They often don't provide it's hard to request a reasonable accommodation. The chat bot may not understand what you're talking about. It may not know how to communicate. That's one issue. The second issue is the actual results that come up based on either directly measuring things in a way that impacts disability like trying to evaluate facial expressions of someone who has different facial expressions through a disability. There is also, as Megan has talked about extensively, based on data. So that's really high level issues these tools and they are I will say that EEOC of both issue guidance on AI, they coordinated on the same day. The EEOC guidance in particular in detail, and I would recommend looking at it, if you're interested. But it talks a lot about how these violate the ADA, how they what the pitfalls are and also best practices for employers.
I'll also air area where we're seeing this.
MARIA TOWN: Good morning, everyone. My name is Maria Town. I serve as the President and CEO of the American Association of People with Disabilities. My pronouns are she/her/hers. I'm a white woman wearing a red mask, a red sweater and black dress. My hair is all over the place. I just had to push a mobility scooter that died on me on my way here. So shout out to the disability advocates who picked me up and got me here. Thank you all so much. Interdependence for the win today and always.
In addition to employment, we're also seeing AI and automated decision tools having big impacts on access to benefits for disabled people. You know, and I want to make it clear, we've actually been seeing this for years, right? Because as you laid out in your opening, many entities, including statement governments, have been using algorithms to help them make decisions around benefits for more than a decade at this point. And AI is just taking everything kind of up to another level. So you have states like Idaho that have gathered data from potential benefit recipients. They put this data into an algorithm, and it establishes a score, which then determines how much funding they receive for services like home and community based services and other health services.
What resulted was potential beneficiaries and existing beneficiaries getting access to fewer resources. And only if a reviewer determined that an individual needed more resources for their health and safety, a very ambiguous category, could they get additional services. So there was a lawsuit, there continues to be a lawsuit, and we just saw people getting access to fewer and fewer services.
Similarly in the Medicare space, we are seeing AI being used to deny people with disabilities and older adults access to more involved care. People are being sent home from rehab facilities after short amounts of time even though they are reporting that their pain is increasing, they aren't to walk and stand by themselves and unable to dress and care for themselves. So there's the merging efforts there to try to address AI's role in Medicare in making sure that particularly older adults can get access to the services they need in order to thrive.
I want to be clear too, we have also seen AI and algorithms used to great effect to increase access to benefits and services. States were able to use Medicaid data to figure out who else was eligible for SNAP. So we actually saw an increase in SNAP enrollments. But this is when states were specifically trying to use AI as a pro equity tool. And I think we have to examine the values frameworks that undergird all of these tools. Not only the data themselves that are being put into it, but building on Brian's example, if an employer is using AI to be more efficient in sorting their through applications, if a state is using AI to address budget pitfalls and save money, ultimately those things will likely harm disabled people.
If a state is using AI and algorithms to better identify potential beneficiaries for things like SNAP or TANIF, then there will be positives.
Specifically what I'll say, in the Medicare and Medicaid space, many of the AI tools are removing the opportunity for people to receive individualized assessments that's actually core to the policies that underlie these programs.
There's a tension there between the use of these AI tools and the value of individualization and customization that exists within not only for Medicaid and Medicare, but also disability rights legislation. In addition to benefits, we're also seeing AI being used in the education space. And current data show that students with disabilities are more likely to use AI in school. We see this with students who have learning disabilities, who might use generative AI, like ChatGPT, to help start a paper. Grammarly is used as an accommodation in schools. This is an AI grammar tool, right?
What we also know is that students with disabilities are more likely to face disciplinary action for their use of these tools, even if they're using them as a kind of reasonable accommodation. I want to name, especially because of the moment that we're in as a nation right now, students with disabilities are in other likely to be LGBT, autistic students are more likely to be trans. And so with AI, and we saw this really increase during the COVID 19 pandemic, which continues today, an increase in surveillance. And some schools, like schools in Texas, banned students going to websites like the Trevor Project, an organization that works to prevent LGBT youth suicide.
We have seen the use of these AI surveillance tools result in outings of students that they did not want to happen. Which places them in danger. And I want to be clear, these are very likely disabled, queer students. Similarly these AI surveillance tools aren't necessarily monitoring student engagement. They're monitoring whether or not the particular tools are used in the right way.
So we've also seen students with disabled parents who are experiencing accessibility failures, get reported as truant. Because their parent was not able to use the school assigned tool, because it was inaccessible. Many schools are reporting the surveillance data directly to law firm. So this is contributing to the overwhelming criminalization of disability students, he special disabled students of color, who are more likely to rely on school provided technology to do their work.
MEGAN SCHULLER: Thank you, Maria. You've just touched on a lot of topics that I want to follow up on. One that you've raised about individualization, and this question that we've talked a little bit about among ourselves, when we talk about how the ADA applies we know that there is a requirement to do an individualized assessment for a person with a disability. Can AI comply with that requirement? Is there a way for there to be an assessment that is individualized enough in terms of how somebody's disability manifests, how it impacts them, how it impacts their daily living, their treatment, that would meet that requirement?
And I'm wondering, Brian, if you can touch on some of what we've seen happening in the employment space or in other contexts, in terms of looking at how does the ADA apply here, but what also are the challenges of applying the existing civil rights laws to these emerging technologies?
BRIAN DIMMICK: Sure. To start with, I mean, I think basically you know, the law is the law. As Megan said, there's no there's no robot exception to the ADA. You can't say that the computer made me do it. But, so, the legal pitfalls are the same, there are challenges when applying. But if you know the background of the ADA, the question and the devil is in the details. And if you're talking about employment, for example, I mean these tools can be standards that screen out people with disabilities.
They can be criteria or administration that have the effect of discriminating sort of under the existing statutory framework, if they are actually using their rules or their AI, their machine learning in a way that is negatively impacting the success of one person with a disability or you can look at this, screening out one person because their disability is too many, but you can also have claims that on the whole, this tool, because the way it works, screens out people with disabilities at a higher rate. That's sort of a impact claim.
That's one area. The other area that I think we think a lot about is reasonable accommodations. Now, that is one way where you might mitigate the harm of one of these tools, but it is also an area that the tools often don't get right. They don't know how to reasonable accommodations, often they're not making accommodations available, and I think we have to have a decision, and we'll get into this more a little later, what accommodations will work. Like what kind of accommodation for let's say a skills test that relies on AI and gives you adaptive choices based on your answers, how do you accommodate that? Is it does it change the way that test is administered, is it a new test? I mean those are somewhat of a lot of somewhat thorny questions, and I think that is something that we're going to be playing out a lot over the coming years.
In employment also you cannot make disability related inquiries for the author or do medical combinations. That can come into play, especially with things like personality assessments, but also another tools that ask for or obtain information about your disability. So that's how it applies in employment and I agree that the individualized nature question is really important. Like we have to figure out how can these these tools are claiming to make individualized decisions, but in fact the way they're doing that is based on aggregate data often that is biased and or is measuring things that aren't related to what they're trying to decide.
So that's employment. I think the principles under Title II of the ADA are also similar and deal with a lot of screening out and reasonable accommodation. I don't know if Megan and Maria if you want to say more about that.
MEGAN SCHULLER: Well, I think one of the pieces that you raised there, one way to challenge these tools that sort of seems to lend itself when you look at how they're operating is disparate impact. But we also know that there have been challenges whether or not you can have a disparate impact theory under the Rehab Act and the ADA. This came up in the CVS case before the Supreme Court.
So we're talking about an already challenging new application of the law, and then using a theory that's under attack in the courts, makes it challenging. I do think the ADA applies.
I think it's very important that we make that clear, and I think particularly when we're talking about algorithms that are effectively automating what has already been happening. If a social worker is not permitted to screen calls based on certain factors when they were doing it manually, it should not now be permitted because they're using a computer to do it and they did not know what was actually happening. And I think that's the piece we talk about here, when we talk about applying these laws and the challenges, is what many have called the black box problem.
A lot of times we can't get the data, we can't get the tool. When we talk about the pretrial sentencing tools, well if the tool doesn't know what their race is, how did it have the bias, the first answer would be let's look at the tools, let's look at the factors taken into account, how they were weighted and. But no that's proprietary and the developers and vendors have fought hard and have had some success if defending these are proprietary and we cannot get access to that information. And in fact the courts don't have access to that information. They do not know how this score is being calculated in appearing on their screen. The social workers do not know how that score is being calculated, and then they are seeing bright red and a number appear that of course is going to make you view that individual differently.
And we have this issue also of what is typical, to when you think about in a courtroom, an expert bias. That if you call somebody an expert, people are more likely to defer to them to think that what they're saying must be true and correct. There's also an AI algorithm bias that we are seeing, and that studies are showing, that even when there's a human check, even when that's supposed to be just information for a human to then consider, that what one study in Allegheny County found was that the social workers started deferring to the tool and changing their own assessments instead of being a check on the tool. They assumed without having any idea how the score is being calculated or what's being used to do it, that it must be correct, it must be objective. It's math, it's science.
And I think that's a real challenge for us when we go into court, in terms of the framing and how you convince the judge that this in fact could be very arbitrary. Could be very discriminatory. But also how do we get the data and the evidence to bring those cases? And I will put one plug for the fact I think one thing we all can do as advocates is filing complaints with the federal government, with DoJ Civil Rights Division, with the EEOC, with other federal agencies Office of Civil Rights, because they do have tools to get information that may be harder to get.
We can also partner with the P&As and there's been some work to explore P&A access and how that might apply in this space and whether or not they can get access to that information more readily than the public. And we have our Freedom of Information Act and Sunshine Law requests that we can try and see how it goes. But I think that's a real challenge that we need to explore. And I think when we talk in a minute about solutions, transparency is a big concern. I don't know if there's anything that you two want to add before we move on
BRIAN DIMMICK: I think the transparency issue is a big one and I will highlight. Often the entities using these tools don't even know what they do. They are buying them off the shelf based on the sales claims that the vendors make, which may or may not have any relation to reality. They don't even know what's going on. They're just and as Megan said, there's an AI bias, and I think a lot of what we need to do is create a healthy skepticism among ourselves and among courts and users of these tools as to what they can actually do. These tools are not they're not thinking. They're looking for patterns. They are mining large chunks of data. But they are not thinking. So I think that is one thing here that I do think the transparency.
And the other transparency issue is people who are subjected to these tools don't know that they're being used a lot of the time. And so they don't even know to challenge something or ask for accommodations.
And I think again, when we get the solutions and we're going to talk what is the solutions. I think we have to recognize that these problems are out there, that they're hard to find. Because we don't know I couldn't give you a list of companies that are using these tools either in employment or in law enforcement. We don't know who's using these things, because they don't tell us. They don't have to tell us. And we're going to have to do come up with creative ways to investigate and figure out whether that's testing, whether that's, you know, strategies to find out who's using so we can figure out how to fight this. So that to me is a big looking into a online personality assessment that's used in hiring and a lot of the reasons we know this particular one is the company itself published data in an attempt to make themselves look cutting edge. But not companies public their data not all vendors public their data. And if they don't, how are we going to know.
MARIA TOWN: I wanted to add one piece on the proprietary kind of issue and the lack of transparency. One of the arguments that companies use to continue to keep their data and formulas private is that they don't want people learning how to game the system. And so this age old stereotype of disabled people who are just trying to pull one over and scam benefits systems, that again has existed for far longer than these AI systems have, is actually informing how companies address arguments for transparency and address a need to actually highlight how they're making these decisions.
MEGAN SCHULLER: Before we move on to some of sort of the emerging laws and public policy questions, I did want to also touch on this question of the vendors and the developers, which ties into this issue of it being proprietary, the transparency. Does the ADA reach the vendors? Does it reach the developers? And if not, then what's their incentive to comply?
BRIAN DIMMICK: I can start with that. The ADA wasn't really designed to reach vendors. It was designed to reach the entities that are making decisions and interacting with people with disabilities and the vendors are often behind the scenes. There are some theories that we've been thinking about, like employment agencies under the are subject to Title I of the ADA. So casting a vendor, especially if they're contracted with a company to buy a test or do some screening, and there's a good example on the EEOC guidance about this, where they say that a company that is basically acting as an agent of the employer in the solution process could be held liable.
So that's one area. And there are some other things that we can think about. In general, the vendors don't really think they're liable and aren't really thinking about this in that way. So I think that's going to be an area we're going to have to push on. I think it's really going to be finding the way to make the vendors directly liable or putting pressure or having companies who use these tools or governments push the vendors that they're going to do the right thing. Vendors will listen to their customers and I think that's an area where we push them.
The other thing to touch on briefly on is the vendors incentives. They want to sell products and they want this technology to advance. A lot of times I think A, they make sales claims that these products can do things that they can't necessarily do. They will make claims that these products are unbiased or they have validated them for race and disability when it's very questionable. A, if they could even do that, and B if they have done that, and that is a concern.
And also these vendors, especially when we get into public policy we're going to get into, they will often use the potential purported benefits to people with disabilities as a way to say well you have to let us innovate and do what we need to do. You can't hand us during this technology with regulation because we'll look at all the benefits people with disabilities with get and maybe there's something to that in some circumstances. But when we're thinking about what vendors are saying, we need to think about what their incentives are. How we did them good incentives and don't buy into what they're saying just to sell the products and advance the technology.
MARIA TOWN: I would love to build on that, actually. So one tool that we've seen begin to be used are these AI civil rights audits. And there are now a few different civil rights frameworks for these audit tools. And I'll use New York as an example. The city of New York established a local law that contractors who use AI and automated decision tools in their work have to put these tools through a civil rights audit framework. And the audit framework addressed things like race and gender, but it did not include disability.
And as the city was considering finalizing this local law, many disability advocates testified that disability needed to be integrated into the framework. And the response the city consistently gave is that there was not enough data around disability for it to be incorporated into an audit framework. So this issue of a lack of data continues to contribute to major issues around disability bias in AI, but it also contributes to a real limitation on the types of tools that are being developed to address and mitigate that potential bias.
MEGAN SCHULLER: Thank you. I think as we think about this, some of this is obviously to be seen. The law is still very much evolving and we're going to see how it applies. I mention there's the provision in the regulations about the fact that this applies through contractual or other licensing arrangements, so at least entities can't say well, it's not my fault. I didn't design the tool. Or we didn't know what it was doing or how it worked. Should not be valid defenses. But I think it's also for advocates, there are opportunities that both Brian and Maria have raised here, where there is a discussion happening. There are folks who are interested in creating fairness in AI and equity. There are companies that are coming out, like Checker, that advertise themselves as creating an unbiased and more fair background check.
There are opportunities for us to engage for folks who say that they're interested in doing this, to make sure that people with disabilities are part of those conversations. And even also with researchers. With the experts on these issues. I have often found that when I talk to experts on predictive algorithms, on risk assessment tools, and you ask them if they've thought about disability, the answer is no. But they're interested.
And then I ask them who is thinking about this, who is working on that, and I often get, I'll get back to you. And they don't. Because there's not a lot of people who are thinking about this and researching this, outside of the disability rights community. So there is a moment here, I think, and an opportunity for us to raise awareness, to engage with folks, to get folks who are interested in these topics, who are researching these topics, to think about disability. To help them understand how it's impacting the disability community. And to start a conversation. So that hopefully some of this can also happen as the tools are being developed, before they are literally everywhere.
And I think about, for example, we've been looking at the use of police to respond to 911 calls related to mental health emergencies. Dispatch is now using AI. A lot of different police departments are using AI for their call tree and for screening calls to determine if you send the police or fire or where it is available a mobile crisis unit or mental health response. If we can talk to the developers, whether they're required to make the changes or not, maybe there's an opportunity here for us to get in and weigh in on what those questions are, how it should be done.
So I think we need to both emphasize and talk about the importance of existing civil rights laws applying, but also think about our other advocacy tools and how we can make an impact and get in early and make sure that people with disabilities are in the room, are part of the conversation, are part of the development and the roll out of these tools.
So with that I want to open it up. We had a bunch of other topics that we wanted to talk about on promise and perils and solutions, so I also welcome there's a lot of expertise in the room, if folks want to make comments on some of those things. But also we welcome questions. I see there's a couple of hands in the audience. I think we have somebody with a microphone.
Attendee: My name is Julia, and I have brown frizzy hair, curlyish, I'm fat, I have a beige jacket, I'm white, brown eyes. Yeah. And I use they/them pronouns.
So I worry a lot about the conversation around AI moving away from holding ourselves accountable for our own actions. For example, the actions that I do create the data that outputs racist, ableist, and discriminatory answers.
For example, the way we are taught to view disabled Black mothers will inform the bias CPS workers have for removal. Our statutes that is an algorithm that can be put into a computer and it does get the output we want. For example, it gives us the output of racism and ableism because we created the law to do that. And so I think about like how if we remove the AI, you would still have the same input and the same output because our human behavior isn't being addressed. And I worry about how are we going to ensure that while we're addressing the AI conversation, we're addressing what created the data in the first place. There's no such thing as an unbiased AI, that's the same thing as saying someone is colorblind. Because someone is going to always put in that data and we are always going to create that data. And anybody born in the U.S. is inherently racist, ableist, all of the above. So yeah. I was just thinking a lot about that.
MEGAN SCHULLER: I saw you nodding, Maria.
MARIA TOWN: You're right. And I think your point speaks to a number of things that were mentioned up here, and also speaks to the limitations of potential accommodations or human alternatives, right? So Brian in his discussion of employment, talked about notice and requests for accommodation. But when the accommodation for an AI screening tool is an interview with a human, does that produce a benefit when it is likely that that person would have discriminated anyway. And I wanted to take this opportunity to mention two things that are happening right now that I think we have a real chance at influencing.
One is the White House Executive Order on Artificial Intelligence and the other is what is happening in Congress. It articulates requirements for federal agencies to examine how these tools are being used across their enterprise. And I see a couple of federal partners here who I know are working on this. It also kind of asks these questions of on ask agencies to pursue questions of what do we need to do to make our systems more equitable.
In conjunction with the executive order, the office of management and budget released a memo focused on AI governance. And all of these things are in motion right now. Now, executive orders have limits. When the administration changes, what's going to happen to all of that is a huge question, which is one of the reasons it's so important to pay attention to what's happening on Capitol Hill. Because Senator Schumer and senator Hawley are working on creating national AI legislation. And I think there's a big question in front of us which I think is how new should this piece of legislation be? How much should it incorporate existing civil rights laws and mandates, and what can we actually do to attempt to undo all of the systemic biases that we've built into our existing policies.
So I really hope that you all will kind of look at what this bipartisan coalition, I think they're being called the Gang of Five are putting out with. Disability is not really there yet. So I think we need a big push, especially in the Senate, to make sure our elected officials are aware how much is at stake for people with disabilities with AI
MEGAN SCHULLER: You raised what I was talking about the levels of bias in the system, it raises the question is the right solution what the EU AI Act is doing? Do we need to ban the use of certain types of AI, when we're talking about high risk uses and these decisions that impact people's lives in such major ways, what are the solutions? Because also the systems are already very biased and discriminatory. So I've certainly had conversations with parental rights advocates about which is worse? ? The AI tool or what's already been happening. Is there an opportunity here. And so that's, you know, something that the Bazelon Center has been talking to the CPT and our partners at AAPD and other organizations what do solutions look like. And whether we're going to challenge through the law or we're doing it through public policy, are there ways that these tools could be used to reduce human bias? Because there is a lot of human bias in the system.
If we changed what the question was, if we used it at a different phase in the process, I don't have those answers yet, but I'm really interested in coming to some of the sessions and hearing people's thoughts on that and would welcome additional conversations. I think this is also where we need to get more, like, research and academics and folks thinking about this and talking about it and we need to talk with developers and we need to engage, I think, with the industry, who are, you know, working on these tools. How do they operate? Is there a way that we could do this in a more equitable way? Are there places where it shouldn't be used? Are there places where it could be used better
Attendee: I use she/her pronouns and we heard a lot of doom and gloom this morning about AI, rightly so in the spaces that you've been talking about. But I'm concerned about things sort of being overlooked, in terms of the expansion to information that AI has created for the disability community, and our need to protect it. For example, you mentioned initially in your opening remarks around seeing AI and the Be My Eyes application to that is giving people access to information that we have not been able to receive previously. In addition there was a few years back where Facebook Meta rolled back its use of facial recognition that changed and diminished our access to that platform.
Because in terms of photos and video, all of a sudden we were able to know who was in it. Or if we were in it. Whereas before that, we didn't. We couldn't. And so there is some discussions in the world around privacy in AI and of course those are important conversations to have. Though I'll tell you I don't quite understand why somebody can walk around and take a picture with their phone and that's okay but my AI that's going to tell me what it is, can't. I don't know about that. Side note. But I think it's really important that we actually also talk about the really good things that AI is doing to advance inclusion and participation for the disability community and that this group and others really work hard to protect those aspects of AI. Can you talk a little about that?
MEGAN SCHULLER: Absolutely. I'm so glad you raised that. It was something we wanted to touch on. So I'll open it up to Brian and Maria first.
BRIAN DIMMICK: I'm happy to start. Thank you for raising that. I think it's really important. There are good things that can come from AI, especially when it's AI the person with disabilities is using and has control over. We've been talking a lot about systems where the government or the employer has the control and they decide. But where the person with disabilities has the control, I think it could be really, really beneficial. I do think the privacy concerns are really important. That's a conversation too long to have right now except to say I work at the ACLU, we have some strong opinions on privacy.
But I think the question has become who has the data and who controls the data that these systems use. It's one thing to take a picture where the data only goes to your phone and it's not in the world. But these vendors what they're really trying to do is collect data and use that data for their own purposes. So I think we need to push for yes, the benefits, these tools coming to people with disabilities, but also our community retaining control over our data and deciding how to protect our own privacy and not just accepting that the trade off for getting the tools is giving away all of our data.
MARIA TOWN: I think this is an excellent point, and again it's one of the reasons we're seeing more disabled students use AI tools in school. Students with disabilities, dyslexia using tools like Grammarly or ChatGPT. There's an advocate AAPD works with in Ohio to help her write a script to tell her parents she wanted to live on her own and move out of their house. That conversation did not go well, even though the script was very good.
Using a personal example, I wanted to go to the movies with friends, they wanted to meet up beforehand, because you can't really chat in the movies. And suggested a restaurant. I have a mobility disability. I use a mobility scooter. I needed to find out if the restaurant was accessible. I looked online. No information, right? I tried to look at pictures, no information. So I called the restaurant, and my phone call is answered by AI. And it says, please wait for this chat bot to chat you. And I'm like I get so excited, right? Because this is actually a very, in my opinion, straightforward application of AI, where it can give me factual information, and I don't have to listen to a person say oh, well, it's just one step. We can pick you up. Right?
So sure enough, the chat bot texts my phone and says thanks for thinking about Open Roast, how can I help you. And I ask is your restaurant accessible. And it doesn't know what I mean. So then I have to ask a few different ways for it to respond, and I finally say is your restaurant wheelchair accessible. And it says, well, naturally, by federal law, we are ADA compliant.
[ Laughter ]
Compliance with federal law is not natural. It doesn't happen naturally.
[ Laughter ]
So to your point, I really firmly believe that there are many, many beneficial applications of AI, especially for people with disabilities. But that example is meant to highlight that all of these tools need so much more training, even the ones that we know to be beneficial. Because that script, for example, for that bot, could have been written so much better. And with input from disabled people. And it's very clear that that did not happen. And just so you know, I wasn't able to get to the restaurant, because the elevator on the Metro was broken. So anyway, I completely agree, and I'm glad you raised the point.
MEGAN SCHULLER: I will very briefly say I think this underscores the fact that people with disabilities need to be part of the development, part of the conversation. I think there's a lot of ways that technology can increase independence, can increase access, when it's done in partnership with people with disabilities. When disability is considered. And I do think there are going to be competing interests and that needs to be part of the conversation and it's going to be challenging as we navigate it, but it's not something new to this community.
So I think that's a really important piece. The other thing I will say is even when we're talking about the doom and gloom, AI is not going anywhere. Algorithms aren't going any way. So we need to think about solutions. We need to think about opportunities and how we make this technology, even when it's not currently working for us, work for us. And how we can increase the equity and access through AI. Because I don't think the ultimate answer is don't use it, make it go away. It's not going to happen.
All right, I believe we are at time. I wonder first from our panelists are there any final closing thoughts? Though I don't know how you're going to beat what they just said. Thank you so much.
[ Applause ]