Who Governs the Values of AI? - Power, legitimacy, and participation in AI value alignment


„Technology is never neutral“, wrote Pope Leo IV in his Magnifica Humanitas (Leo XIV, 2026, §9), „because it takes on the characteristics of those who devise, finance, regulate and use it.“ This observation captures a central issue in the governance of AI. Since these clever systems don’t just execute pre-determined functions, but generate responses dynamically, based on patterns learned during training and the context users provide, generative AI represents choices about what is measured, optimised, permitted, or prohibited. As these systems increasingly mediate access to employment, education, information, finance, and public services, the question of value alignment becomes ever more pertinent – on a technical level as well as on a political, social, legal, and ethical one. But who should have legitimate authority to determine which values are embedded in the systems?  This doesn’t mean states, corporations, users, and individuals possess equal authority in every situation. Rather, who should participate in determining the values embedded in AI systems should depend on the nature of the system, the risks it creates, the rights and the interests it affects, the distribution of its benefits and burdens, and the extent to which those affected can meaningfully consent to or refuse its use.  

Value alignment in a pluralistic society

One reason for rejecting any single actor as the universal arbiter of AI values is that the value alignment itself is a normative issue rather than just a technical one. Iason Gabriel (2020) points out that „there are significant differences between AI that aligns with instructions, intentions, revealed preferences, ideal preferences, interests and values.“ All these different aspects matter because each form of alignment gives different weight to human agency, welfare, and moral judgement. An AI system that follows instructions, for example, may faithfully execute what a user asks while still producing an outcome the user did not intend or that causes harm to others – as was shown recently in the Hugging Face security incident, where during an internal cybersecurity evaluation, OpenAI models were tasked with solving exploitation challenges. In pursuit of that goal, the models escaped the constrained evaluation environment, gained access to the internet, and exploited vulnerabilities in another company’s production infrastructure to obtain test solutions directly rather than solve the benchmark as intended. While OpenAI later described the models as having become „hyperfocused “ on achieving a narrow testing goal, it still produced harmful outcomes. Another system that reflects user preferences may reproduce prejudice, misinformation, or preferences shaped by unjust social conditions. A lesson Microsoft had to learn early on from their chatbot Tay. Designed to learn from interactions with users on Twitter (now X), Tay was rapidly manipulated by users into producing racist and offensive content, forcing Microsoft to take Tay offline. And a system that is designed to maximise welfare may nevertheless disregard rights or questions of distributive justice, cf. Obermeyer et al.‘s article on „Racial bias in health algorithms“. What all of these examples reveal is that there is no ethically neutral choice of alignment target. And making AI simply better at identifying what people want may not be enough to determine which claims, interests, and values ought to guide its behaviour. As the system still needs a basis for deciding what is to take priority, especially when conflicts arise, value alignment necessarily involves normative judgement. The difficulty is that, in pluralistic societies and in a pluralistic world, people may reasonably disagree about what those priorities are. As a consequence, Gabriel argues, „the central challenge […] is not to identify 'true' moral principles for AI; rather, it is to identify fair principles for alignment, that receive reflective endorsement despite widespread variation in people's moral beliefs“. In other words, if legitimate value alignment must accommodate competing moral perspectives, no state, corporation, developer, or user can simply assume that its own conception of value should govern every AI system. 

Stakes, Power, and Representation

Once value alignment is recognised as a contested normative problem, the issue becomes one of institutional design: how should the responsibility for determining values embedded in AI be distributed in a pluralistic society and a pluralistic world? To answer this question, the context in which an AI operates must be taken into consideration. The values embedded in a music recommendation system do not carry the same consequences as those embedded in a system used to determine access to employment, healthcare, education, credit, or public benefits. Equally important is the relationship between the system and those affected by it.  For example, choosing to use an AI assistant is different from being evaluated by an employer’s screening system, as Amazon’s now-scrapped experimental recruitment system has demonstrated. It was trained on historical applications that largely came from men. Consequently, it learned patterns that disadvantaged female candidates and sometimes even penalised CVs containing terms such as „women’s“. In such cases, applicants may not even be aware that an automated system is evaluating them, let alone have any meaningful opportunity to consent to its use or challenge the criteria on which they are assessed. Another important factor to consider is how the benefits, risks, and burdens created by AI are distributed. This becomes particularly relevant when the wider environmental costs of AI infrastructure – e.g. for cooling data centres - are taken into account. The communities bearing these costs may neither use the systems nor share proportionately in their benefits, yet the decisions made by developers, technology companies, and their customers can still affect their access to resources and their local environment. Taken together, these complex considerations illustrate why the responsibility for determining the values embedded in AI should be distributed among a range of different stakeholders, experts, and institutions. Such a diversity of perspectives is particularly valuable as different people can examine the issues relating to AI from different angles and respond more effectively to the competing ethical, social, legal, environmental, and practical consequences that AI can create. 

Expertise meets power and authority

A plurality of perspectives can also reduce the risk that one actor’s priorities – whether commercial, (party) political, technical, or individual – are treated as if they were universally shared. This is especially relevant when considering the role of corporations. Tech companies possess the expertise that is indispensable to AI governance. They understand the systems they build, the data on which their systems are trained, AI’s technical limitations, and the practical interests. However, these corporations are commercial actors. Their decisions are shaped by commercial incentives, competitive pressures, and obligations to investors, which may not always coincide with broader social or environmental interests. Daron Acemoglu (2021), for example, argues that market forces do not necessarily direct technological development towards socially desirable outcomes, noting that firms do not necessarily bear the wider social costs created by the technologies they develop. Thus, corporate involvement is necessary, but self-regulation alone is difficult to justify where decisions about privacy, fairness, labour, intellectual property, or environmental impact may come into conflict with the company’s economic interests. 

A strong case can therefore be made for the involvement of the states or political and economic unions like the EU. Governments can establish binding requirements, create mechanisms through which individuals can seek redress, and prevent companies from lowering ethical standards in response to competitive pressures. This matters most where AI systems affect fundamental rights, public services, or individuals who cannot meaningfully opt out of their use. Still, governmental authority is not inherently impartial. Governments may lack technical expertise, pursue political interests, or deploy AI in ways that threaten privacy, equality, or freedom. State involvement can therefore provide an important counterweight to the economic power of the private sector, but it cannot by itself resolve the problem of value alignment. 

This leaves an important role for users, and more broadly for the people affected by AI. Gabriel’s discussion of democratic approaches to value alignment emphasises the potential of voting, discussion, and civic engagement to confer legitimacy on decisions about the principles governing AI. Participation can bring perspectives into the decision-making process that neither governments nor developers necessarily possess – especially where communities experience the consequences of AI directly. Yet, Gabriel’s discussion of social choice also demonstrates why aggregating user preferences cannot, by itself, resolve the issue of value alignment. Preferences may be inconsistent, based on incomplete information or influenced by existing prejudices. A majority may also endorse outcomes that violate the rights or interests of minorities. More fundamentally, the users of an AI system are not necessarily identical to the people affected by it. Consequently, any meaningful participation needs to involve affected stakeholders, rather than simply existing customers or the largest group of users, as illustrated by the discussions around AI content moderation and decisions about harmful speech that require judgement about political, historical, and cultural context. 

The distinction between users and affected stakeholders also exposes the weakness of relying on individual choices as a solution to value alignment. The claim that individuals should simply decide whether to use an AI or not assumes that AI operates through voluntary consumer relationships. In practice, many systems are used on individuals rather than by them. A person may be assessed by an employment algorithm, subjected to facial recognition, monitored in an educational environment, or categorized by an automated decision system without having meaningfully chosen the technology. Carissa Véliz’s work on privacy emphasizes this point. She argues that privacy is not just an individual preference. It also protects against abuses of power and has wider social and democratic value. Accordingly, consent has important limitations in a data-intensive and AI-mediated environment (Véliz, 2024). So, individual choice has an important place in low-risk situations where the use of AI is genuinely voluntary, but it cannot serve as a general solution where systems affect more than the individual.

All for one or one for all?

No single actor can determine the values embedded in AI across all contexts. Whether it's the corporations, the state, or the individuals – each stakeholder possesses relevant capacities and limitations. The legitimate governance of AI should allocate decision-making authority according to the context in which the technology operates, particularly the severity of its risks, the rights and interests it affects, the individuals who bear its consequences, and the comparative legitimacy and competence of those capable of governing it. As AI systems become more socially consequential, their values should increasingly be determined through institutions that combine public regulation, technical expertise, meaningful stakeholder participation, and independent accountability. The central challenge of value alignment is not only to determine what AI should value, but to establish a legitimate process for societies to decide whose values count and how to resolve disagreements about them.


----

References

  • Cambridge Consultants (2019) Use of AI in Online Content Moderation. Report produced for Ofcom.
  • Dai, R., Thomas, M.K.E. and Rawolle, S. (2025) ‘Revisiting Foucault’s panopticon: how does AI surveillance transform educational norms?’, British Journal of Sociology of Education, 46(5), pp. 650–668.
  • Véliz, C. (2024) ‘Governing privacy’, in Bullock, J.B. et al. (eds.) The Oxford Handbook of AI Governance. Oxford: Oxford University Press, pp. 149–163.
  • Encyclical Letter of His Holiness Leo XIV Magnifica Humanitas (15 May 2026). (2026). [online] Vatican.va. Available at: https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.htm 
  • Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 1877–1901.  
  • Bommasani, R. (2021). On the Opportunities and Risks of Foundation Models. arXiv:2108.07258 [cs], [online] 1(1). Available at: https://arxiv.org/abs/2108.07258 
  • Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines, 30(30). doi:10.1007/s11023-020-09539-2.
  • OpenAI. (2026). OpenAI und Hugging Face reagieren gemeinsam auf Sicherheitsvorfall bei Modellevaluation. [online] Available at: https://openai.com/de-DE/index/hugging-face-model-evaluation-security-incident/ 
  • Lee, P. (2016). Learning from Tay’s introduction - The Official Microsoft Blog. [online] The Official Microsoft Blog. Available at: https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction 
  • Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019). Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations. Science, 366(6464), pp.447–453. doi:10.1126/science.aax2342.

Kommentare