This is being provided in a rough-draft format. Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.
MAITREYA SHAH: Hi, everyone. Thank you for coming to this session. So we will quickly introduce ourselves and then move to part of the discussion. So the left of me is Ariana Aboulafia, who is Policy Counsel at the Center for Democracy and Technology. And works on disability and technology and does a lot of good research and heads CDT's efforts in that direction.
ARIANA ABOULAFIA: And all the way to the right of me at the end is Lydia X. Z. Brown. They are the Director of Public Policy at NDI. They also are a professor at Georgetown and engage in all sorts of disability-related advocacy.
Lydia, do you want to do our last intro?
LYDIA X.Z. BROWN: And sitting in the middle up here is Maitreya Shah, who comes to us from Harvard Law School's Berkman Klein Center for Internet and Society. He is a fellow there. He is an attorney working on AI and disability rights issues. We are very privileged have Maitreya here. And he actually had the idea to present this session at tenBroek this year. So thank you for organizing and making that happen.
MAITREYA SHAH: Thank you so much. Thank you both. So I think we'll -- we will jump right in as the title says, total is the session, discussion -- again, the frame it as discussion is on the conversation of AI and disability and the impact of policy of policies in the United States. We had some impact of AI on people with disabilities, we will try not to repeat what was already said but also go deeper with some of the impact and how things can be addressed legally, what are the legal recourses, and what are the public policy implications of these that is part of our work.
Probably, to start with, I think Ariana, what do you think, when we say AI's impact for people with disabilities, we've heard a lot today from sources how there is this miscommunication, biases, and exclusion from AI algorithmic systems, employment, many sectors. But what do you think makes disability a unique case here? What issues do you think are very, very crucial for people with disabilities to think about when you say a AI and algorithms impact disproportionately. Let's start with that and probably go into a deeper conversation on the actual impact.
ARIANA ABOULAFIA: Yeah. I think that's a good place to start. So I think sort of when you talk about AI and disability and what we are going to talk about a lot here today is sort of discriminatory outcomes or disproportionate outputs that we are seeing as a result of AI or algorithmic systems. And we are seeing those, basically, anywhere that AI or technology or algorithmic systems are being integrated. So that could be in education, which we are going to talk a bit about, in employment systems, which we are going to talk about, healthcare, but also in all sorts of things, in tenet screenings and benefits determinations, which we may not touch on today.
As these algorithmic systems or AI are continuing to be incorporated into more and more areas of life, we are going to see these discriminatory or disproportionately outcomes for folks with disabilities. One of the things I think is important to talk about as a threshold matter is one of the reasons why it is that we see dips outputs as a result of these algorithmic systems.
Sometimes when folks talk about algorithms, they use the term, quote-unquote, black box. Meaning we don't necessarily know what happens with an algorithm. But when you have a discriminatory outcome or output, there are lines that can be drawn from those outputs to under-inclusive or non-inclusive inputs. So when you have data that's being used to create or deploy or train or even on the back end, to audit algorithmic systems, and that data is either under-inclusive or inaccurate or non-inclusive of people with disabilities, the outputs that wind up being discriminatory are at least partially as a result of that. And disability, it's very difficult to get accurate and inclusive data related to disability. And I think my colleague is going to talk a little bit more about that. But it's really important, I think, to -- I mean, one of the really important things is to ensure that people with disabilities are being included at every stage of, again --
SPEAKER: Whoa. Sorry to interrupt.
ARIANA ABOULAFIA: We are glad to have you. Welcome. To ensure that people with disabilities are included at every stage whether it is creation of algorithms, deployment of algorithms, and, again, auditing. Because when people with lived experience with disability, whether or not they are coders or computer scientists, folks with lived experience of disability, when they are included in every step of creating these algorithms or technologies, I think that makes it less likely we are going to have these discriminatory outputs.
So I want to start there and do some table setting. And then I want to turn it back over to my colleague.
MAITREYA SHAH: Thank you so much for introducing that so well and talking about the crucial relation of disability data and the discriminatory outcomes of AI and algorithmic technologies.
What we also realized through our work is the lack of epistemic knowledge. As Lawrence, attorneys, even as people with disabilities, we have very little understanding of how these technologies work and why do these technologies have discriminatory outcomes. When we say that disability poses a unique gaze, you know, when you think about -- so there is a lot of conversation around AI, fairness, ethics. Companies come and tell you that we are trying to make our AI systems fairer. But it's usually for different identity groups like race and gender. But disability usually doesn't come in because the technical AI fairness metrics are usually lobbed on a group fairness basis where you group different identities into your data sets and in your algorithm training, and disability, because of its identity, doesn't feature in.
It's even more complex for people who live with marginalizations, intersection Al identities, the term that is used in technical trainings is outliers. People with identities and people with different intersectionalities are labeled as, quote-unquote, outliers. While you are building a system and you are cleaning your data for the purposes of accurate decisions, you think to remove your outliers so that you can have better and more accurate decisions and people with disabilities are usually those outlier cases. When you can now imagine that if an entire category of people are removed from your training systems, then the overall outcome of that AI system will be discriminatory.
So with that ground setting, I think we can probably jump into a couple of the issues that we think we want to discuss including employment, education, and healthcare, and discuss what is the exact intersection here between law, policy, and the impact of AI on disability. So Lydia, you would like to maybe start talking about the exact issues in what we have been seeing in employment context in how people with disabilities are disproportionately impacted with the rise of decoders using AI systems?
LYDIA X.Z. BROWN: Sorry. We have to keep moving this microphone. This is Lydia speaking. I think it's important to preface my remarks by explicitly naming that one of the issues and conversations about AI, civil rights, and social justice is a misapprehension and an incorrect and nonproductive framing of the issue being that of unfairness or of inequality. The word fairness in particular is widely used, for example, in the phrase algorithmic fairness, to indicate that an algorithm acts unfairly if the algorithms affect whether intended or not is to increase or to perpetuate existing patterns of discrimination.
But this framing fundamentally misunderstands the foundational problem of algorithmic harm, which is actually that of injustice. Fairness within our systems as they currently are, within our educational system, within access to the workforce, within any other consideration where people with disabilities are present does not actually mean and to be ableist practices and ableist practices or end of ableist results.
So in the context of employment and automatic decision making tools that recruiters and hiring managers are increasingly using, a hiring manager may adopt an algorithmic tool that is incredibly fair. It assesses every single applicant according to identical criteria. Perhaps they have taken great care to include data like was just discussed now that they made effort to include data about people with disabilities in training the algorithm. Which is not that common, but maybe they did. So their algorithm was trained on more representative data. It is applying the same criteria equally across all people that it's considering for a particular job. And yet, it continues to produce ableist results where disabled people are disproportionately screened out for employment. Why is that? It's not a question of fairness. It's a question of justice.
If the algorithm, for instance, involves video recording somebody's answers to interview questions and analyzing the recorded videos of the applicant for their tone of voice, for the semantic content that they use, for what their eye movements are, we can understand pretty intuitively in a room of many disabled people and lawyers who work on disability rights issues, that a blind person whose eyeballs move atypically, autistic person who does not make typical eye contact, a person with cerebral palsy who impacts their speech, a Deaf person whose oral speech is impacted by being deaf, and many other people with a wide range of disabilities will automatically be judged differently and likely adversely for an algorithm that is making assumptions about a person's reliability, their honesty, their congeniality, and, therefore, their potential future ability to be successful in the workforce based on factors related to their eye movements or their voice and the types of words that they use. It is fair. It is not just.
If an employer or a recruiter is also using an algorithm that automatically screens resumes for content to determine which candidate should be passed on, many HR departments may have adopted that algorithm because they claim that it is removing human bias. It is reducing, if not eliminating human bias, that, for instance, the algorithm will not make a human judgment about a name that is coded as Black or as Muslim, that an algorithm will not necessarily differentiate between names that are gendered as more feminine versus those that are gendered as more masculine.
But in reality, as we know, if the data that it is trained on is what the algorithm deploys in making its assessments and making its recommendations, that reflects systematic inequalities. Depressed rates of employment or promotion for Black people versus white people and other non-Black people of color, or that women on average even before disaggregating for race already earn less and are less likely to be promoted than men, those disparities grow when you factor in racial difference along with already being gendered as woman, then those algorithmic results are going to replicate the same patterns of systematic discrimination. They will do so intersectionally and will amplify them.
One great example that of this that I like to cite often, one company that uses its own program, promote the future success of employees and tell them who to prioritize for internal professional development and the promotion of raises and promotions, two factors which strongly correlated with future successes. Those were having the first name Jarred and having played Lacrosse.
(Laughter).
Both of which are traits that code very white and very male. Now, you don't have to be a white man to be named Jarred, and you, certainly, don't have to be a white man to play Lacrosse, but that sport is more often available to people of upper income levels, generally. And it also is pretty irrelevant to your ability to do almost any job except if you were being considered for the job of Lacrosse sports reporter or Lacrosse coach. In those two jobs, I think it matters a great deal if you have played Lacrosse before. For literally any other job on the planet, it does not matter or necessarily relate to your future performance if you have played Lacrosse. But aside from being a really funny story, what this antidote illustrates for me is that algorithmic tools don't simply reflect societal values, but they perpetuate and replicate the same systems and patterns of injustice. And they do so at scale. They are not creating new injustices. They are new tools perpetuating and amplifying the same injustices.
So the question is one of is it fairness, we are missing the point. If the question is one merely of equal treatment, we are missing the point. And we know that as Disability Rights Advocates. Equal treatment in education does not mean an actually meaningfully -- a meaningful education. There was another adjective that was supposed to be there and my brain forgot what it was. So that's where we are off to this week. And yet that is where a lot of policy is still stuck when we talk about AI and algorithms. It's stuck on we need our algorithms to be fair.
If we include more data from disabled people, it will be better. But that's not necessarily true because if as we know disabled people are largely underemployed, higher rates of unemployment twice that of the overall population, are more likely to have precarious or contingent employment and over a lifetime are likely to have lower lifetime earnings and lower net wealth, those disparities grow for every other access of marginalized identity that is introduced. NDI and the financial something network, now I feel really bad, Saturday funders network. As set funders network and NDI published about a year ago that illustrated in detail disaggregating by race and gender what these statistics look like for the disability community.
Well, if overall over our lifetime we have had less access to credit especially accounting for racial oppression and gender-based oppression on top of disability discrimination, overall, we have had a less stable employment record, lower-paying jobs, and lower ability to attain a job commensurate to experience and education, then those realities, those patterns that are the result of systematic discrimination will be replicated and exacerbated by algorithmic tools that function favorable. Because if they were fair, they would rely on the existing data.
MAITREYA SHAH: Thank you so much, Lydia, for laying that out. That so many issues I think we have seen a lot of evidence of people being screened out in these recruitment tools and how discriminatory that is. I think we will move to a more specific tangent here, another disability law, the way crucial rights for people with disabilities in employment has been reasonable accommodations. I will ask you, Ariana, what do you think has been the impact of AI tools on reasonable accommodations specifically l it is at a time of recruitment or during jobs, at the workplace, so this entire life cycle of someone who is applying and then getting into a job and how algorithmic tools interact with your right of feasible accommodations. As lawyers sitting in the room, how can we contextualize the violation of conditional accommodation rights with algorithmic tools?
ARIANA ABOULAFIA: Sure, I will start on this and then, hopefully, you can fill in some of the gaps as well. So I think in the -- I think we can also kind of copy/paste when we talk about female education as well. I think in so far and employment, if you are talking about some of the tools that Lydia had mentioned, tools that may be as an example resume screeners, tools that may track your eye contact or measure your vocal cadence during an interview, we sort of refer to those as automated employment decision tools or automated employment decision systems. That's an umbrella term for all of those things. One of the issues with those sorts of tools is that a person with a disability may not know that they are being used during an interview. So if a people with disabilities doesn't know that a tool is being used and they don't know to ask for an accommodation.
So that can be sort of one accommodation issue related with these tools or systems. Secondarily, these tools may or may not actually measure things that are related to essential job functions. They may or may not measure things that somebody may be able to do with the proper accommodation actually on a job. But, again, if they don't know it to ask for an accommodation because they haven't been made aware that these tools exist or are being used, or secondarily, it could just be a different sort of measurement that is somewhat attenuated from what the essential job function actually is.
So there is sort of those set or suite of issues in the hiring process, and there is more there. And I know some of our colleagues early this morning mentioned some of these tools.
Then there is sort of issues with technology that are being integrated to sort of on the job uses. There is sort of, again, kind of an umbrella term of boss ware, which refers to all sorts of surveillance technologies that are used throughout the employment process. So these can be tools that measure. It can be sort of key stroke monitors, things that measure, quote-unquote, productivity. And it can be tempting to sort of write these off as tools that are only used in, quote-unquote, white collar or office jobs, but they are absolutely used in other contexts. Truck drivers, warehouse workers.
For folks with disabilities as an example who may need extra breaks or something like that, these tools are going to potentially measure or flag you negatively. And then those tools can then be sometimes combined with also automated tools that formally jobs done by individual people, that will say whether or not this person should get a promotion, whether or not this person should be disciplined. And when you have all of those things sort of combined, it can really impact a person with a disability at work and impact their ability to ask for the proper accommodations, and also to go through the personalize the accommodation process in the way in which that is supposed to be done. And that is also going to be compounded any time a person doesn't know that these tools are being used. Right? So I would say those are some of the main things that employment -- and, again, we will kind of circle back to some of the ADA questions when we talk about education.
MAITREYA SHAH: Thank you so much, Ariana. Maybe the examples you gave us in helping us make this point that all these reasonable accommodations, I don't know if tangible is the right word, but one of these issues that we could comprehend, because at times with AI systems, it's often difficult to understand if there has been a violation of your right. But with reasonable accommodations, probably, it's possible to comprehend and to realize that your right has been violated. One of those very crucial points under ADA.
And I think I will also like to touch upon discrimination, this broader umbrella term of discrimination, and also disparate impact on ADA and how AI tools result in discrimination. One of the things that I can think of is the medical information that gets leaked. So we know there is specific provisions under ADA for employers to keep medical information of people private. But if say, for example, they are using a software or a tool that is built by an outside vendor, also, sometimes maintained by an outside vendor, and collects medical information of the employees and is leaked outside the system, it could be a case where you could hold the employer liable for discrimination here for leaking medical information. And there are many other cases and instances of discrimination that I think I like both of you to probably touch on and you could also bring in -- because the equal opportunity commission I think has been taking employment discrimination and AI tools very seriously. So maybe touch upon the EEOC guidance and what do you think has been the impact, what is lacking, what are the gaps, and what do you think could people in this room better address in their work in laws and policies? It's off to both of you whoever wants to touch on this aspect of employment discrimination.
LYDIA X.Z. BROWN: This thing keeps making rumbling noises. Sorry, this is Lydia. Mic set up is assuming that we all wish to be very intimate with each other or have to move the microphone around every two minutes. Those of you who do not know, the EEOC jointly with the DOJ was the first federal agency, or I guess the two of them jointly, was the first federal agencies to explicit -- I can't speak, to explicitly issue guidance related to AI discrimination and to disability rights issues. And that guidance was specifically focused on the use of these automated decision making tools in the hiring and recruitment context.
And it was fortunately for our community, there are many folks who had a background in disability rights law who were involved in drafting that guidance and they were in community with advocates as well. And so that guidance reiterates and clarifies what employers' obligations are under the Americans with Disabilities Act. In order to affirmatively avoid discrimination against perspective candidates with disabilities. That means in addressing the specific ways in which automated decision making tools and hiring recruitment can actually discriminate. One of which is to be outright inaccessible without providing an alternative or the ability to provide reasonable accommodations.
The other major way that they tend to discriminate is by disparately selecting out candidates with disabilities who might be otherwise qualified and are functionally screened out. That's the phrase I was looking for, functionally screened out. There are other ways these tools can also be discriminatory, fact tow medical tests that is relate to perform the essential functions of the job, but primarily they tend to discriminate by not being accessible and by not making reasonable accommodations available include to provide vial notice in advance and meaningful alternative and the ability to equally be considered in one's candidate see having used an alternative assessment or an automated assessment with an accommodation in place. And, of course, that other ways to have the effect of screening out people with disabilities.
So, for instance, an algorithmic hiring tool that depends upon personality testing that tends to screen out applicants with psycho, social, or mental health disabilities could be in violation with the Americans with Disabilities Act. Again, even if it is technically fair and that it is being administered to all applicants and all applicants are answering the same question.
Or conversely, if an assessment that is used in the process of onboarding somebody and determining if they are going to be hired is one in which all people who have any form of blindness are screened out because of inability to use a visual-based interface that does not have any capabilities for screen reader or alternative access and requires visual acute and the job does not require visual acute, say being a bus driver, which right now without automated vehicles, does require visual acute, but if it is for literally anything else, that could constitute discrimination under the Americans with Disabilities Act.
To avoid discrimination requires careful attention to the ways in which an employer chooses to design its assessments and hiring procedures, selection procedures, for candidates, as well as how an employee continues to reassess and regularly assess the tools they use and the impact that they have. But it also requires an approach to hiring candidates that is in alignment with the existing requirements of ADA for an interactive process to determine reasonable accommodations to which somebody is entitled in the process for applying for a job for which they are otherwise qualified.
That individualized and interactive process requires the ability of the employer to make sure that the assessments they provide actually relate to essential job functions, this they are narrowly tailored to do that, and that there are alternatives available this do not necessarily cause and inaccessible -- barrier of inaccessible or another discriminatory barrier, or also that the candidate has a meaningful opportunity to understand what the nature is of the assessment process and any tools being used in that process so that the candidate can actually request accommodations in advance of having interface with that particular tool.
That guidance now being written at the highest level in our federal government means that most employers it do have even more of an affirmative obligation to know what it means to actually apply with the Americans with Disabilities Act in their hiring processes. It also provides advocates with better tools for being able to bring action against employers that violate the provisions of nondiscrimination within the ADA.
Importantly, while advocacy within the EEOC can often be limited in its reach and effect because most cases reach settlement, those settlements can still have persuasive impact on future issues that come up before the EEOC in their investigations. Last summer, the EEOC for the first time, specifically ruled in a public settlement against i tutor group, that a territory algorithm that had outright excluded all applicants above a certain age through an automated resume screening process was, in fact, engaging in unlawful employment discrimination. While that was focused on be age rather than disability, that is informative and useful for disability advocates not only because many aging people also develop aging related disabilities, but also because it points to the future of a fertile ground for further effective advocacy at the EEOC and beyond where algorithms discriminate against disabled candidates.
MAITREYA SHAH: Thank you so much. That gives us this idea that there is progress here on the side of the government. And we also realized that compared to a lot of other contexts, we have recognized some harms are able to also it cans conceptualize them in the respect of ADA and how to challenge some of these tools. We could take more questions on employment later from the audience. But I think we will now like to switch to the second big issue that we wanted to focus on, which is education.
Just like employment, I think education is also a sector where we've seen a lot of discriminatory outcomes of AI and algorithmic technologies.
I think specifically, we would like to talk about ed tech, I think you will be very familiar with education technologies and tools that are used in education settings and how they disproportionately impact people with disabilities.
Ariana, you would like to address the issues of ed tech and what is the impact specifically in its relation with disability law?
ARIANA ABOULAFIA: Sure, yes. I will talk briefly about some of these things. I think the first thing that I want to kind of layout, and that's why I have this paper in front of me, so I can get the numbers right, is just sort of how common a lot of these education
technologies are.
So there is all sorts of different technologies that are being used in education systems. There are things like content filtering and blocking. Those are things that filter out what may or may not be able to be shown either on a school Wi-Fi network or on a school issued device and lots of schools issue devices at this point. Then also student monitoring softwares, which are more akin to being surreal technologies that are often put into, again, school issued devices whether laptops, chrome books are or iPads. Those monitors can monitor what programs or a student accesses on those devices. It can also access things like what they are searching on search engines and that sort of thing.
Then there is emerging technologies, genitive AI. Includes things like it chat bots, ChatGPT 4, 5, and other language models. These technologies are, I think, probably a little bit more common than many folks would know. And so CDT actually recently -- in 2023, published some research. So 98% of teachers say that their school uses filtering or blocking technology. And this is of the folks that we polled. 88% of teachers say their school uses some form of student activity monitoring. And 51% of teachers say that their school uses some form of generative AI. Their students have used either for personal or school use.
Particularly those first two, filtering blocking and student activity monitoring, that's like the significant vast majority of schools. Right? And so, then we need to think about students with disabilities in those schools, and that students with disabilities as a threshold matter, the ADA matter, of course, also Section 504 of the Rehabilitation Act, IDEA. There is a bunch of statutes that protect students with disabilities and youth with disabilities in schools. And so, there are all sorts of potential ways that we can dream up where the use of these softwares can lead to harms for students with disabilities. And so in this paper that CDT published, one of the things that we posited was a potential scenario where a student, let's say with major depressive disorder or mental health condition is using a school issued device to look up things related to that disability, those words are then flagged by a student activity monitoring, and that person is then repeatedly pulled out of class.
If a student is going to be repeatedly pulled out of class as a result of their disability facilitated by technology, we can see a world where that can lead to a denial of free and appropriate education under Section 504. We can see a world where something like that winds up being discrimination under the ADA. And so one of the ways that I like to think about sort of these sorts of scenarios is that this is just disability discrimination that we've been dealing with. Disability Rights Advocates have been dealing with. But it's been facilitated by technology. I tend to use the words technology facilitated technology to drive home the point that these outcomes are not necessarily novel. It's just that the tech is facilitated now.
So I hope that sort of helps to answer some of that.
MAITREYA SHAH: Absolutely. Thank you so much. And Lydia, you would like to tell us a bit about the larger policy questions here when we think about education discrimination, and as Ariana said, technology facilitated technology. What are the larger policy questions here? In employment, we said EEOC guidance, other questions about reasonable accommodations, so on. What is probably the way forward for us when we think about working on these things?
LYDIA X.Z. BROWN: This is Lydia. The bigger issues, again, going back to my earlier comments, at the beginning of our session are questions about justice. And it is what are these algorithms doing? What are the outputs? What are the objectives of them? And what social policies do those algorithms serve to further or do they undermine? And that's also another way of reframing and reiterating what I had said before, that the conversation should not be one merely about fairness, but it is also one that is about justice. Because at the social policy aim is to say, uphold status quo and keep people with disabilities out of the workforce, to keep people with disabilities out of the economic mainstream, to keep people with disabilities out of the same opportunities that could be availed of by nondisabled people, then that's not a social policy that's worth furthering, and we should not be advocating for simply regulating algorithms that further those social policy aims.
But, on the other hand, if we think about the social policy aims that we all have been deeply committed to furthering as individual advocates or as advocates working on systems change, then that's how we should think about what we do to regulate algorithms. For instance, if we cared about people with disabilities being able to have equal access to the workforce, to be able to have the free and appropriate public education and the least restrictive environment possible, if we cared about people with disabilities being able to be treated fairly in a healthcare setting without worrying about disability bias and prejudice leading to discriminatory treatment or medical, that's how we should aim to do what algorithms can do. If the algorithm used by a credit scoring agency is one that consistently depressing the credit rating of people with disabilities, or that includes false or unreliable information relating to people with disabilities that disabled people's ability to access credit, that affects disabled people's ability to rent an apartment, to pass a background check and secure a conditional offer of employment.
The problem isn't is this algorithm fair but what social policy is that pursuing? Is that a social policy worth pursuing? Is this an algorithm that is actually helping society become more just for people with disabilities and other marginalized communities, or is this algorithm reinforcing and replicating existing forms and patterns of injustice?
So the broader policy questions to me are less about how we respond to algorithms in particular. I think there is room for doing that in how we shape guidance and detailed regulatory guidance. But more, how we enforce existing civil rights laws and improve our existing civil rights monitoring and enforcement mechanisms to account for the experiences of disabled people?
So many of our civil rights laws do not meaningfully address or understand disability and disability rights law, has in effect been segregated from other civil rights legal advocacy, how people with disabilities are defined under law. Sometimes that has been to our benefit. One scholar, of course, I forgot their name, who argued, for instance, that the way the ADA creates an asymmetrical standard for who is entitled to access, rights, or benefits under the ADA as compared to the Civil Rights Act of 1964, Title VI or Title IX, which create a symmetrical standard, doesn't matter what race, anyone can claim discrimination on the basis of race. A white person can claim discrimination on the basis of race. It doesn't matter what sex or gender a person is perceived of being. You could be a cisgender man who is claiming discrimination on the basis of sex. And in the law, you may actually have a claim that prevails.
What does that mean? (Phone dinging). I don't think that was me.
You may have a claim that prevails. That is why Abigale Fisher was able to go to the Supreme Court twice in a row as a white woman, who was claiming that she was discriminated against on the basis of her race and sex in the admission's policy of UT Austin. If you don't know what I am talking about, you can research this and be enraged. If you do know what I'm talking about, you are already enraged. Oh, no, you did mention her.
So there is an argument that is a good thing in some contexts, but other contexts, this has been detrimental. For instance, the Fair Credit Reporting Act does not address disability at all. The Equal Credit Opportunity Act does not mention or include disability. Those are really important consumer protections laws that relate to the ability to meaningfully enforce and have fulfilled the sitter civil right to housing, employment, fair dignified employment. What we in the disability community also describe as having access to competitive integrated employment doesn't just mean you can get a job, it means you are fairly considered for a job that you might apply for.
So if an algorithm is perpetuating discrimination that prevents full consideration for a job or prevents full inclusion in educational context or prevents fair access to credit or to housing, then the problem isn't just that there is a lack of technical understanding. And frankly, we are all not going to become technical experts, that's unrealistic and beyond our field. It's also that we need strengthened civil rights protections and enforcement and monitoring protections of rights and actions and robust -- at the state and federal level. Even if we have an unfavorable federal administration, state advocacy becomes even more important to make sure that disabled people and advocates have the tools they need to challenge discriminatory outcomes and impact.
ARIANA ABOULAFIA: So in our last sort of couple of minutes -- and thank you for laying all that out so well, Lydia, I think we sort of also just wanted to talk very briefly about some of Maitreya's research and what he has been working on in so far as algorithmic systems and people with disabilities in healthcare:
MAITREYA SHAH: Thank you. Thank you so much. We've talked about employment in education and I also wanted to briefly touch about healthcare which is not usually discussed as we ought to.
So my recent work has been on algorithmic triage and disability discrimination. For folks if triage is a new word, it's basically rationing of scarce medical resources in times of emergency. We've seen that during COVID-19 pandemic when ventilators, ICU beds, oxygen, vaccines were scarce. The supplies were less and the demand was high. At that time, clinicians and ethicists were calling for triage protocols where you decided who gets these scarce resources and you prioritize some patients over others. And it's usually the people who are deprioritized are people with disabilities.
Although this has been done for a long time now, this is not new, we have triage in place since the wars, what is alarming is these health institutions and clinicians using algorithmic technologies to decide who gets to live and who gets to die in emergency settings. So to give you two very quick examples of how people with disabilities are specifically discriminated, here is, one, your algorithmic systems are trained using your electronic health records. We know that your electronic health records are always full of clinician biases against people with disabilities. Always been medical gatekeeping. People with disabilities don't get to self-report their disability status. And there is clinician misjudgments and misconceptions about disability. So all of that makes way into your electronic health records, that data is used in training triage algorithm.
You can understand there are biases that come in. The second big issue is, these algorithms are built using existing triage protocols such as you have these clinical scales, subsequently organ assessment, and standards of care, which also deprioritize people with disabilities. So the sequential organ failure assessment uses many different matrix or many different prediction scales, one of them is a scale which measures acute brain injury and how much chances of survival someone has. It does not take into account the baseline status of, say, a person with cerebral palsy. It would treat somebody with an acute brain injury and cerebral palsy equally and, basically, giving them high ranks. These are ways of sanctioning people with disabilities.
These are some very alarming situations of, I think disability, discrimination in healthcare. There are many others because technology is being rapidly integrated into modern medicine from diagnosis to prognosis. And there are many instances of how this could be identified and challenged. I think it was said, we need better civil rights protections. I think that was my work on healthcare and discrimination.
And do you both have any final comments? Then we could go to the audience for their questions as we are cognizant of time?
ARIANA ABOULAFIA: I think I will close out by reiterating something that Lydia said stating it differently. I think we can think that AI algorithms systems technology as a force multiplier. So when we are thinking about these systems, like healthcare is one that can be very ableist, when you integrate and incorporate algorithmic systems, you may be multiplying those forces of ableism. And that stands as well for racism or gender-based bias or anything like that. And so I think it's really important to ask those questions that Lydia spoke about in the beginning and sort of throughout her session. Lydia, do you have anything to close out as well?
LYDIA X.Z. BROWN: This is Lydia. I recognize that in these conversations, I often appear to inhabit the role of perpetual pessimist. Sorry. But on a potentially more hopeful note, one thing that I very buoyed -- buoyed? Oh, my God. I can't even speak any more. Pleased, encouraged. Very encouraged by -- I had to wake up very early today. One thing I'm very encouraged by, even seeing the agenda for this year, there is a growing interest in and expertise on the intersections specifically of tech and disability rights advocacy. And I see this happening both in the technology advocacy space around tech and civil rights and in the disability civil rights and disability advocacy spaces too. That to me suggests that unlike say four years ago when I was one of, like, four people working in this space, there is now enough of us where we have multiple sessions on this two day conference agenda that are addressing tech and disability rights. And where I am seeing disability being written about and considered very seriously as a necessary topic in other conversations about technology and civil rights and social justice issues.
And that to me suggests that we are building and seating the field with more expertise and more passion for this growing area of advocacy that is all the more important, again, not because tech is necessarily creating new forms of discrimination, but because there are new and constantly being developed tools to continue the same types of discrimination and try to evade detection. But even as companies and other orient tease creating those physiologies are creating new ways to discriminate against disabled people with those tools, there are more and more advocates coming up now and honing and refining their skills and their knowledge and community connections and relationships to be able to fight against them. So that gives me a little bit of hope.
ARIANA ABOULAFIA: I think that's a great way to close out.
MAITREYA SHAH: Yeah.
(Applause).
ARIANA ABOULAFIA: I think we are just about at time. We are happy to answer questions.
MAITREYA SHAH: Yeah, if folks want to break now.
ATTENDEE: I just have one question. Thank you. Because I was wondering, if I really want to know how those intersectionality of technology and disability rights and discrimination, is there a book, an article, I can look for and I can systematically understand what is going on in the field?
MAITREYA SHAH: There is a lot.
ATTENDEE: There is a lot?
MAITREYA SHAH: Basically, even when you start googling that Ariana and Lydia is saying there is a lot of articles. But I think you would both like to suggest specific favorites?
ARIANA ABOULAFIA: I would recommend against techno ableism.
LYDIA X.Z. BROWN: I second that recommendation. Also, all three of us have written a lot of articles. If you two aren't going to shamelessly self-promote, I will shamelessly self-promote me and shamelessly co-presenter promote you. If you look up any of our names and tech and disability, you will find an enormous amount of resources availability. Articles, popular audience articles, podcasts, other recordings, and so on. Self-promotion accomplished.
ATTENDEE: Thank you.
(Laughter).
ATTENDEE: This is Julia. Brown curly hair, beige jacket. This is a very important topic for me. For one, I'm also thinking of this topic since the 2000s, that's why I want to actually push you three to consider de-professionalizing the model and looking toward cross-collaboration. As artists about this, there are tons of artists that are disabled working on the tech side and addressing this. But more specifically, I always think about to actually even create a just algorithmic model in AI or an AI that passes the mirroring test.
This AI would need to, one, not be something -- the mirror test essentially requires that the AI provides me an output that uses normal atypical speech, usually white or affluent elite sounding. So it has to pass it for human but the human we are using it for is not disabled, not -- like, not diverse in anyway. And on top of that, everything that we input has to come with human action. So arguably, the only way to actually create a just model is to create either fictionally or real, a universal design system, where people's behavior creates the output -- the data we want to be able to input it into an algorithm that can have an output they want.
LYDIA X.Z. BROWN: Yeah, thank you for sharing that intervention. My background is in community organizing, and I am also an artist. So I'm already there with you.
ATTENDEE: I know.
ARIANA ABOULAFIA: I think your points about the creation of just models is well taken as well. I think that this sort of -- sort of what you are alluding to is that algorithms are trained on pattern recognition. So anyone who has any sort of difference including disabilities but also including all sorts of other things, that algorithms sort of may be -- that there may be an inherent compatibility here. In order to resort that incompatibility, you have to completely change or upend or recreate the structure of how these things are created. So I think that point is very well-taken. And I'm assuming we are going to get kicked out of here soon.
(Laughter).
Thank you all. Thanks all for being here. And we are here for more.
MAITREYA SHAH: Yeah, we are here for more. Thank you so much.