People should not be slaves to machines, a coalition of evangelical church congregations from more than 30 countries preached to leaders of the European Union earlier this summer.
The European Evangelical Alliance believes all forms of AI with the potential to harm people should be evaluated, and AI with the power to harm the environment should be labeled high risk, as should AI for transhumanism, the alteration of people with tech like computers or machinery. It urged members of the European Commission for more discussion of what’s “considered safe and morally acceptable” when it comes to augmented humans and computer-brain interfaces.
The evangelical group is one of more than 300 organizations to weigh in on the EU’s Artificial Intelligence Act, which lawmakers and regulators introduced in April. The comment period on the proposal ended August 8, and it will now be considered by the European Parliament and European Council, made up of heads of state from EU member nations. The AI Act is one of the first major policy initiatives worldwide focused on protecting people from harmful AI. If enacted, it will classify AI systems according to risk, more strictly regulate AI that’s deemed high risk to humans, and ban some forms of AI entirely, including real-time facial recognition in some instances. In the meantime, corporations and interest groups are publicly lobbying lawmakers to amend the proposal according to their interests.
At the heart of much of that commentary is a debate over which kinds of AI should be considered high risk. The bill defines high risk as AI that can harm a person’s health or safety or infringe on fundamental rights guaranteed to EU citizens, like the right to life, the right to live free from discrimination, and the right to a fair trial. News headlines in the past few years demonstrate how these technologies, which have been largely unregulated, can cause harm. AI systems can lead to false arrests, negative health care outcomes, and mass surveillance, particularly for marginalized groups like Black people, women, religious minority groups, the LGBTQ community, people with disabilities, and those from lower economic classes. Without a legal mandate for businesses or governments to disclose when AI is used, individuals may not even realize the impact the technology is having on their lives.
The EU has often been at the forefront of regulating technology companies, such as on issues of competition and digital privacy. Like the EU's General Data Protection Regulation, the AI Act has the potential to shape policy beyond Europe’s borders. Democratic governments are beginning to create legal frameworks to govern how AI is used based on risk and rights. The question of what regulators define as high risk is sure to spark lobbying efforts from Brussels to London to Washington for years to come.
Within the EU, some legal structures do exist to address how algorithms are used in society. In 2020, a Dutch court declared an algorithm used to identify fraud among recipients of public benefits a violation of human rights. In January, an Italian court declared a Deliveroo algorithm discriminatory for assigning gig workers reliability scores that penalize people for things like personal emergencies or being sick.
But the AI Act would create a common regulatory and legal framework for 27 countries in the European Union. The draft proposes the creation of a searchable public database of high-risk AI that’s maintained by the European Commission. It also creates a European AI Board tasked with some yet-to-be-decided forms of oversight. Significantly, the AI Act hinges upon defining what forms of AI deserve the label “high risk.”
Work on the AI Act started in 2018 and was preceded by a number of reports about the making of trustworthy AI systems, including work with a 52-member expert group and processes involving thousands of business, government, and society stakeholders. Their feedback helped inform which forms of AI are listed as high risk in the draft proposal.
EU leaders insist that addressing ethical questions that surround AI will lead to a more competitive market for AI goods and services, increase adoption of AI, and help the region compete alongside China and the United States. Regulators hope high-risk labels encourage more professional and responsible business practices.
Business respondents say the draft legislation goes too far, with costs and rules that will stifle innovation. Meanwhile, many human rights groups, AI ethics, and antidiscrimination groups argue the AI Act doesn’t go far enough, leaving people vulnerable to powerful businesses and governments with the resources to deploy advanced AI systems. (The bill notably does not cover uses of AI by the military.)
While some public comments on the AI Act came from individual EU citizens, responses primarily came from professional groups for radiologists and oncologists, trade unions for Irish and German educators, and major European businesses like Nokia, Philips, Siemens, and the BMW Group.
American companies are also well represented, with commentary from Facebook, Google, IBM, Intel, Microsoft, OpenAI, Twilio, and Workday. In fact, according to data collected by European Commission staff, the United States ranked fourth as the source for most of the comments, after Belgium, France, and Germany.
Many companies expressed concern about the costs of new regulation and questioned how their own AI systems would be labeled. Facebook wanted the European Commission to be more explicit about whether the AI Act’s mandate to ban subliminal techniques that manipulate people extends to targeted advertising. Equifax and MasterCard each argued against a blanket high-risk designation for any AI that judges a person’s creditworthiness, claiming it would increase costs and decrease the accuracy of credit assessments. However, numerous studies have found instances of discrimination involving algorithms, financial services, and loans.
NEC, the Japanese facial recognition company, argued that the AI Act places an undue amount of responsibility on the provider of AI systems instead of the users and that the draft’s proposal to label all remote biometric identification systems as high risk would carry high compliance costs.
One major dispute companies have with the draft legislation is how it treats general-purpose or pretrained models that are capable of accomplishing a range of tasks, like OpenAI’s GPT-3 or Google’s experimental multimodal model MUM. Some of these models are open source, and others are proprietary creations sold to customers by cloud services companies that possess the AI talent, data, and computing resources necessary to train such systems. In a 13-page response to the AI Act, Google argued that it would be difficult or impossible for the creators of general-purpose AI systems to comply with the rules.
Other companies working on the development of general-purpose systems or artificial general intelligence like Google’s DeepMind, IBM, and Microsoft also suggested changes to account for AI that can carry out multiple tasks. OpenAI urged the European Commission to avoid the ban of general-purpose systems in the future, even if some use cases may fall into a high-risk category.
Businesses also want to see the creators of the AI Act change definitions of critical terminology. Companies like Facebook argued that the bill uses overbroad terminology to define high-risk systems, resulting in overregulation. Others suggested more technical changes. Google, for example, wants a new definition added to the draft bill that distinguishes between “deployers” of an AI system and the “providers,” “distributors,” or “importers” of AI systems. Doing so, the company argues, can place liability for modifications made to an AI system on the business or entity that makes the change rather than the company that created the original. Microsoft made a similar recommendation.
Then there’s the matter of how much a high-risk label will cost businesses.
A study by European Commission staff puts compliance costs for a single AI project under the AI Act at around 10,000 euros and finds that companies can expect initial overall costs of about 30,000 euros. As companies develop professional approaches and become considered business as usual, it expects costs to fall closer to 20,000 euros. The study used a model created by the Federal Statistical Office in Germany and acknowledges that costs can vary depending on a project’s size and complexity. Since developers acquire and customize AI models, then embed them in their own products, the study concludes that a “complex ecosystem would potentially involve a complex sharing of liabilities.”
Overall, the study predicts that by 2025 the global AI industry would pay 1.6 to 3.3 billion euros in compliance costs each year. A separate impact assessment of the AI Act estimates that 5 to 15 percent of AI can be categorized as high risk and expect additional costs. That assessment also warns that if the European Commission fails to adopt legislation like the AI Act, a fragmented approach could lead to higher compliance costs for European businesses working with high-risk forms of AI.
The Center for Data Innovation—part of the Information Technology and Innovation Foundation, which receives support from major tech firms as well as companies like AT&T, Procter & Gamble, Merck, and Pfizer—says EU cost estimates are much too low. It estimates that businesses will pay more than 10 billion euros in annual costs by 2025 and that costs for some small to medium-size enterprises could reach up to 400,000 euros. The center’s analysis also predicted that the AI Act would drain Europe of AI talent and reduce AI investment and adoption across the EU. These findings and figures were later reported by news outlets and cited by the US Chamber of Commerce in its letter to the European Commission.
Meeri Haataja, CEO of an AI startup in Finland, and Joanna Bryson, a professor of ethics and technology at the Hertie School Data Science Lab in Germany, have pushed back on the center’s accounting. In a recently published analysis, the two review AI Act compliance costs and feedback on the draft legislation from business interests, think tanks, and civil society organizations. They estimate that the AI Act could cost companies deploying high-risk AI roughly 13,000 euros per system to stay in compliance with human oversight mandates, much closer to the EU estimate.
Haataja and Bryson say the Center for Data Innovation picked the biggest, scariest number from an EU report, but they conclude that additional review of compliance costs associated with the AI Act is called for.
Senior policy analyst Ben Mueller was author of the Center for Data Innovation study. He says his figure takes into account the cost of a quality management system, a regulatory compliance standard that costs hundreds of thousands of euros to set up. European Commission staff said that the cost of a quality management system is not taken into account in the EU’s impact assessment, because most companies making AI systems already have such regulatory systems in place and to do so would be “misleading.”
“The debate rests on what fraction of AI systems will be deemed high risk,” Mueller said, pointing to a segment of the EU’s impact assessment report that said the definition of high risk will determine how much of AI development spending goes toward compliance costs.
Bryson rejects the idea that compliance costs for high-risk forms of AI will be too much of a burden, particularly since corporations deal with compliance requirements all the time in sensitive areas like privacy or cybersecurity. And she expects that companies developing AI that is not considered high risk may voluntarily adopt similar standards.
“If a company isn't willing to spend a certain amount of money reviewing these kinds of problems, then maybe they shouldn't be going into hazardous kinds of software,” Bryson said.
Whereas business and tech interests say the AI Act goes too far, others say it doesn’t go far enough. These critics are largely civil society organizations and human rights advocates, as well as some AI researchers, some of whom have up until now shaped the legislation behind the scenes.
International human rights group Access Now says the AI Act needs a number of major changes or it will fall short of protecting fundamental rights. In January, Access Now joined more than 100 members of the European Parliament and dozens of human rights organizations in urging the European Commission to ban AI in a number of use cases. Some of those requested bans or high-risk designations made it into the draft document, including restrictions on AI for border or migration control, AI that can assign people social scores, and AI with the power to manipulate people. The draft also bans real-time facial recognition by law enforcement, except in certain scenarios.
In its commentary, Access Now officials characterized the draft ban language as vague and containing too many loopholes. The group also said the act needs a clearer definition of what level of risk is considered unacceptable, so there’s clear criteria in place for how to ban additional forms of AI in the future.
The Future of Life Institute, a nonprofit research group, argues in favor of a lower threshold for what’s considered subliminal manipulation, a banned category of use under the AI Act. Adtech that maximizes ad clicks to be addicting, the group claims, can lead people to poor mental health, the spread of misinformation, or extremism. It also agrees with Google’s assertion that the AI Act should be amended to account for AI that can be used for multiple purposes but for different reasons, stressing that judging systems by a single use “could allow increasingly transformative technologies to evade regulatory scrutiny.”
The Civil Liberties Union for Europe wants the act to require a third-party audit for all high-risk AI systems; the current draft requires it only for some forms of biometric identification like facial recognition.
“These systems are a threat to our individual freedoms, including the right to education, the right to a fair trial, the right to privacy, and the right to freedom of speech. They often present a situation of severe power imbalance and have huge implications on people’s fundamental rights. It is unacceptable to delegate their risk assessment to profit-oriented businesses who focus on obeying the rules when they have to and not on protecting fundamental rights,” the group wrote in a statement to the European Commission.
Other groups are requesting more forms of AI to receive a high-risk label. The Standing Committee of European Doctors asks that AI for determining an insurance premium or assessing medical treatments should be considered high risk. Climate Change AI, a group of machine learning researchers, wants the carbon footprint of an algorithm considered as part of the risk calculation.
Bryson, the professor in Germany, said she and her coauthor Haataja are skeptical that every company producing risk today will actually be classified as high risk under the AI Act but that overall she’s optimistic. By her estimation, things have moved from ethics principles to policy rather quickly.
She said some businesses’ rhetoric about the AI Act reminds her of when she heard colleagues at Big Tech companies ahead of the release of GDPR claim they would be forced to leave Europe. But six months after GDPR passed, Bryson says, the same people called the privacy legislation amazing, like a single API for dozens of European countries. In time, she expects to hear similar things about the AI Act should it be enacted.
“We know that there isn't like a perfect end game,” she said, but passage of the AI Act will help Europe be “better for having the structures there that can help us achieve the next stage.”
To become law, the AI Act will next go through amendment changes by the European Parliament and European Council, giving groups additional bites at the lobbying apple. The question of who is held responsible when AI or other forms of emerging technology harm people is planned for the next phase of EU tech policy scheduled to begin later this year. Should the AI Act become law, the European Commission will be able to update the list of high-risk systems. The law would take two years to go into effect.
Lobbying to influence AI governance happening in Brussels now is just the beginning. A similar gathering of powerful business interests, trade unions, and civil society and activist groups is likely coming to Washington soon.
A person familiar with the matter told WIRED that lawmakers in Congress are working on a version of the Algorithmic Accountability Act. Introduced in 2019, the act requires businesses to assess high-risk machine learning and AI for bias, inaccuracy, or discrimination.
The National Institute of Standards and Technology is also developing an AI risk-assessment tool and seeking public comment and ways to measure how much people trust AI based on risk level. A report released last month by the UC Berkeley Center for Long-Term Cybersecurity includes interviews with NIST staff and urges the federal agency to adopt stricter rules for its AI risk-assessment framework than it did for previous versions made around privacy and cybersecurity. Study author Au Yeung also urged that the tool incorporate environmental harm and impact on marginalized communities in its definition of what’s considered a high-risk AI system.
- 📩 The latest on tech, science, and more: Get our newsletters!
- Looks that quill: The dark side of Hedgehog Instagram
- Is the robot-filled future of farming a nightmare or utopia?
- How to send messages that automatically disappear
- Deepfakes are now making business pitches
- It's time to bring back cargo pants
- 👁️ Explore AI like never before with our new database
- 🎮 WIRED Games: Get the latest tips, reviews, and more
- 🏃🏽♀️ Want the best tools to get healthy? Check out our Gear team’s picks for the best fitness trackers, running gear (including shoes and socks), and best headphones
