There exist several Open Problems in the existing field of AI governance. These problems often have no clear answers, and discussion of these questions can promote better governance. For effective AI governance, we mainly discuss international and open-source governance, hoping to promote the safe development of AI through our discussion.
International Governance
We examine the significance and viability of international AI governance from three aspects within this section: Manage Global Catastrophic AI Risks, Managing Opportunities in AI, and Historical and Rresent Efforts, with both generational and intergenerational perspectives. We aim to contribute innovative thoughts for the prospective structure of international AI governance.
Recommended Papers List
Artificial intelligence and the political legitimacy of global governance
Click to have a preview.
Although the concept of “AI governance” is frequently used in the debate, it is still rather undertheorized. Often it seems to refer to the mechanisms and structures needed to avoid “bad” outcomes and achieve “good” outcomes with regard to the ethical problems artificial intelligence is thought to actualize. In this article we argue that, although this outcome-focused view captures one important aspect of “good governance,” its emphasis on effects runs the risk of overlooking important procedural aspects of good AI governance. One of the most important properties of good AI governance is political legitimacy. Starting out from the assumptions that AI governance should be seen as global in scope and that political legitimacy requires at least a democratic minimum, this article has a twofold aim: to develop a theoretical framework for theorizing the political legitimacy of global AI governance, and to demonstrate how it can be used as a compass for critially assessing the legitimacy of actual instances of global AI governance. Elaborating on a distinction between “governance by AI” and “governance of AI” in relation to different kinds of authority and different kinds of decision-making leads us to the conclusions that much of the existing global AI governance lacks important properties necessary for political legitimacy, and that political legitimacy would be negatively impacted if we handed over certain forms of decision-making to artificial intelligence systems.
The global governance of artificial intelligence: Some normative concerns
Click to have a preview.
The creation of increasingly complex artificial intelligence (AI) systems raises urgent questions about their ethical and social impact on society. Since this impact ultimately depends on political decisions about normative issues, political philosophers can make valuable contributions by addressing such questions. Currently, AI development and application are to a large extent regulated through non-binding ethics guidelines penned by transnational entities. Assuming that the global governance of AI should be at least minimally democratic and fair, this paper sets out three desiderata that an account should satisfy when theorizing about what this means. We argue, first, that an analysis of democratic values, political entities and decision-making should be done in a holistic way; second, that fairness is not only about how AI systems treat individuals, but also about how the benefits and burdens of transformative AI are distributed; and finally, that justice requires that governance mechanisms are not limited to AI technology, but are incorporated into a range of basic institutions. Thus, rather than offering a substantive theory of democratic and fair AI governance, our contribution is metatheoretical: we propose a theoretical framework that sets up certain normative boundary conditions for a satisfactory account.
Manage Global Catastrophic AI Risks
The continual advancements in AI technology promise immense potential for global development and prosperity. However, they inevitably harbor underlying risks.
International Institutions for Advanced AI (Ho et al., 2023)
International Institutions for Advanced AI
Click to have a preview.
International institutions may have an important role to play in ensuring advanced AI systems benefit humanity. International collaborations can unlock AI’s ability to further sustainable development, and coordination of regulatory efforts can reduce obstacles to innovation and the spread of benefits. Conversely, the potential dangerous capabilities of powerful and general-purpose AI systems create global externalities in their development and deployment, and international efforts to further responsible AI practices could help manage the risks they pose. This paper identifies a set of governance functions that could be performed at an international level to address these challenges, ranging from supporting access to frontier AI systems to setting international safety standards. It groups these functions into four institutional models that exhibit internal synergies and have precedents in existing organizations: 1) a Commission on Frontier AI that facilitates expert consensus on opportunities and risks from advanced AI, 2) an Advanced AI Governance Organization that sets international standards to manage global threats from advanced models, supports their implementation, and possibly monitors compliance with a future governance regime, 3) a Frontier AI Collaborative that promotes access to cutting-edge AI, and 4) an AI Safety Project that brings together leading researchers and engineers to further AI safety research. We explore the utility of these models and identify open questions about their viability.
Software that monitors students during tests perpetuates inequality and violates their privacy
Click to have a preview.
The coronavirus pandemic created a surge in demand for exam proctoring tools. Here’s why universities should stop using them.
The Global Governance of Artificial Intelligence: Next Steps for Empirical and Normative Research
Click to have a preview.
Artificial intelligence (AI) represents a technological upheaval with the potential to change human society. Because of its transformative potential, AI is increasingly becoming subject to regulatory initiatives at the global level. Yet, so far, scholarship in political science and international relations has focused more on AI applications than on the emerging architecture of global AI regulation. The purpose of this article is to outline an agenda for research into the global governance of AI. The article distinguishes between two broad perspectives: an empirical approach, aimed at mapping and explaining global AI governance; and a normative approach, aimed at developing and applying standards for appropriate global AI governance. The two approaches offer questions, concepts, and theories that are helpful in gaining an understanding of the emerging global governance of AI. Conversely, exploring AI as a regulatory issue offers a critical opportunity to refine existing general approaches to the study of global governance.
The role of artificial intelligence in achieving the Sustainable Development Goals
Click to have a preview.
The emergence of artificial intelligence (AI) and its progressively wider impact on many sectors requires an assessment of its effect on the achievement of the Sustainable Development Goals. Using a consensus-based expert elicitation process, we find that AI can enable the accomplishment of 134 targets across all the goals, but it may also inhibit 59 targets. However, current research foci overlook important aspects. The fast development of AI needs to be supported by the necessary regulatory insight and oversight for AI-based technologies to enable sustainable development. Failure to do so could result in gaps in transparency, safety, and ethical standards.
Manage Opportunities in AI
The opportunities created by current AI development are not distributed equally, which may cause enduring digital inequality between regions and harm the sustainability of AI development.
AI Regulation through an Intergenerational Lens
Click to have a preview.
Artificial intelligence (AI) intersects with society across age groups, geographies and generations. Current debates on AI and AI ethics focus primarily on AI’s impacts on today’s populations (e.g., the fairness of AI used in predictive technologies used in employment, housing, finance, education and quality of life decisions). In some cases, AI excludes many elderly people and it can often obscure and neglect the agency and rights of younger people who might be targeted and negatively impacted by algorithmic and AI profiling. Further, data infrastructures of AI create incredible power differentials and disparities, which include uneven control and influence over people’s data, issues of personal data and surveillance, data and rights, data and accountability and data inheritance.
Committing to bridging the digital divide in least developed countries
Click to have a preview.
The crises of recent years–from the COVID-19 pandemic and political instabilities to climate challenges–have reversed global development progress in almost every country. Highly vulnerable least developed countries (LDCs) are often disproportionately impacted and bear the brunt of these cascading shocks.
International Institutions for Advanced AI
Click to have a preview.
International institutions may have an important role to play in ensuring advanced AI systems benefit humanity. International collaborations can unlock AI’s ability to further sustainable development, and coordination of regulatory efforts can reduce obstacles to innovation and the spread of benefits. Conversely, the potential dangerous capabilities of powerful and general-purpose AI systems create global externalities in their development and deployment, and international efforts to further responsible AI practices could help manage the risks they pose. This paper identifies a set of governance functions that could be performed at an international level to address these challenges, ranging from supporting access to frontier AI systems to setting international safety standards. It groups these functions into four institutional models that exhibit internal synergies and have precedents in existing organizations: 1) a Commission on Frontier AI that facilitates expert consensus on opportunities and risks from advanced AI, 2) an Advanced AI Governance Organization that sets international standards to manage global threats from advanced models, supports their implementation, and possibly monitors compliance with a future governance regime, 3) a Frontier AI Collaborative that promotes access to cutting-edge AI, and 4) an AI Safety Project that brings together leading researchers and engineers to further AI safety research. We explore the utility of these models and identify open questions about their viability.
The Global Governance of Artificial Intelligence: Next Steps for Empirical and Normative Research
Click to have a preview.
Artificial intelligence (AI) represents a technological upheaval with the potential to change human society. Because of its transformative potential, AI is increasingly becoming subject to regulatory initiatives at the global level. Yet, so far, scholarship in political science and international relations has focused more on AI applications than on the emerging architecture of global AI regulation. The purpose of this article is to outline an agenda for research into the global governance of AI. The article distinguishes between two broad perspectives: an empirical approach, aimed at mapping and explaining global AI governance; and a normative approach, aimed at developing and applying standards for appropriate global AI governance. The two approaches offer questions, concepts, and theories that are helpful in gaining an understanding of the emerging global governance of AI. Conversely, exploring AI as a regulatory issue offers a critical opportunity to refine existing general approaches to the study of global governance.
Why we must consider the intergenerational impacts of AI
Click to have a preview.
Although our future remains unwritten, each day we shape its foundation through our collective efforts, just as our past has paved the way to our present moment. Current debates on artificial intelligence (AI) and AI ethics focus primarily on the impact it is having on today’s populations – the fairness of using AI in recruitment, finance, and education, for example. But what about the equally important intergenerational impacts of today’s AI systems on future generations? Policymakers and regulators should pay attention to these issues.
Historical and Present Efforts
Before the surge of AI technology, the international community had laid down frameworks in line with cooperative regulation of influential technologies and critical matters.
International Governance of Civilian AI: A Jurisdictional Certification Approach (Trager et al., 2023)
A reality check and a way forward for the global governance of artificial intelligence
Click to have a preview.
Global governance of artificial intelligence (AI) must grapple with four monumental challenges. AI is a tough problem to govern given the speed, scale, and uncertainty of its progress. Various aspects of the AI problem require governing because of the range of benefits, risks, and impacts on other global issues. Multilateral efforts on AI are nascent, as is national-level policy. And the multilateral system is under immense pressure from institutional gridlock, fragmentation, and geopolitical competition. No one global governance model for AI is perfect, or desirable. Instead, policymakers must pursue several governance models, each starting in a targeted and focused manner before evolving. They must make clear what policy outcomes are being sought and which institutional functions are needed to reach those outcomes. AI governance within regional and multilateral issue-based groupings would commit nations to action and test models for governing AI globally. And national champions will be critical to success. This pragmatic yet optimistic path will allow humanity to maximize the benefits of artificial intelligence applications and distribute them as widely as possible, while mitigating harms and reducing risks as effectively as possible.
G20 AI Principles
Click to have a preview.
The G20 member nations recommended addressing existing concerns around data protection, biases, appropriate human oversight and ethics to ensure the responsible use and development of AI.
International Governance of Civilian AI: A Jurisdictional Certification Approach
Click to have a preview.
This report describes trade-offs in the design of international governance arrangements for civilian artificial intelligence (AI) and presents one approach in detail. This approach represents the extension of a standards, licensing, and liability regime to the global level. We propose that states establish an International AI Organization (IAIO) to certify state jurisdictions (not firms or AI projects) for compliance with international oversight standards. States can give force to these international standards by adopting regulations prohibiting the import of goods whose supply chains embody AI from non-IAIO-certified jurisdictions. This borrows attributes from models of existing international organizations, such as the International Civilian Aviation Organization (ICAO), the International Maritime Organization (IMO), and the Financial Action Task Force (FATF). States can also adopt multilateral controls on the export of AI product inputs, such as specialized hardware, to non-certified jurisdictions. Indeed, both the import and export standards could be required for certification. As international actors reach consensus on risks of and minimum standards for advanced AI, a jurisdictional certification regime could mitigate a broad range of potential harms, including threats to public safety.
International Institutions for Advanced AI
Click to have a preview.
International institutions may have an important role to play in ensuring advanced AI systems benefit humanity. International collaborations can unlock AI’s ability to further sustainable development, and coordination of regulatory efforts can reduce obstacles to innovation and the spread of benefits. Conversely, the potential dangerous capabilities of powerful and general-purpose AI systems create global externalities in their development and deployment, and international efforts to further responsible AI practices could help manage the risks they pose. This paper identifies a set of governance functions that could be performed at an international level to address these challenges, ranging from supporting access to frontier AI systems to setting international safety standards. It groups these functions into four institutional models that exhibit internal synergies and have precedents in existing organizations: 1) a Commission on Frontier AI that facilitates expert consensus on opportunities and risks from advanced AI, 2) an Advanced AI Governance Organization that sets international standards to manage global threats from advanced models, supports their implementation, and possibly monitors compliance with a future governance regime, 3) a Frontier AI Collaborative that promotes access to cutting-edge AI, and 4) an AI Safety Project that brings together leading researchers and engineers to further AI safety research. We explore the utility of these models and identify open questions about their viability.
OECD Principles on Artificial Intelligence
Click to have a preview.
The OECD AI Principles promote use of AI that is innovative and trustworthy and that respects human rights and democratic values. Adopted in May 2019, they set standards for AI that are practical and flexible enough to stand the test of time.
Recommendation on the Ethics of Artificial Intelligence
Click to have a preview.
This Recommendation addresses ethical issues related to the domain of Artificial Intelligence to the extent that they are within UNESCO’s mandate. It approaches AI ethics as a systematic normative reflection, based on a holistic, comprehensive, multicultural and evolving framework of interdependent values, principles and actions that can guide societies in dealing responsibly with the known and unknown impacts of AI technologies on human beings, societies and the environment and ecosystems, and offers them a basis to accept or reject AI technologies. It considers ethics as a dynamic basis for the normative evaluation and guidance of AI technologies, referring to human dignity, well-being and the prevention of harm as a compass and as rooted in the ethics of science and technology.
THE IEEE Global Initiative On Ethics On Autonomous And Intelligent Systems
Click to have a preview.
The IEEE Global Initiative’s mission is, “To ensure every stakeholder involved in the design and development of autonomous and intelligent systems is educated, trained, and empowered to prioritize ethical considerations so that these technologies are advanced for the benefit of humanity.”
Open-source Governance
The debate over the open sourcing of contemporary AI models is contentious in AI governance, particularly as these models gain increased potency.
Arguments for Open-sourcing
The view that supports the open-sourcing of existing models suggests that this method can mitigate the security risks inherent in these models in several ways:
Potentially Bolster Model’s Security
Recommended Papers List
Llama 2: Open foundation and fine-tuned chat models
Click to have a preview.
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.
Why We Released Grover
Click to have a preview.
Grover is a strong detector of neural fake news precisely because it is simultaneously a state-of-the-art generator of neural fake news. And it is precisely because of its strong – but not quite humanlike – generation and discrimination capabilities that we have made the model publically available. We’ve open-sourced the code and publicly released the model weights for the smaller Grover models. We’ve shared the model weights for the largest (and most powerful) Grover model, Grover-Mega, to over 30 research teams who have applied.
Foster the Decentralization of Power and Control
Recommended Papers List
Democratizing AI, Stable Diffusion & Generative Models
Click to have a preview.
The speaker discusses the democratization of AI and the importance of diversity in data sets to ensure aligned artificial intelligence. They argue for the need to build smaller, more democratized models that impact a broader set of people and allow for adaptation to social issues. The speaker also emphasizes the importance of transparency in the development of AI models and the need for human feedback in reinforcement learning.
Open-Sourcing Highly Capable Foundation Models
Click to have a preview.
Recent decisions by leading AI labs to either open-source their models or to restrict access to their models has sparked debate about whether, and how, increasingly capable AI models should be shared. Open-sourcing in AI typically refers to making model architecture and weights freely and publicly accessible for anyone to modify, study, build on, and use. This offers advantages such as enabling external oversight, accelerating progress, and decentralizing control over AI development and use. However, it also presents a growing potential for misuse and unintended consequences. This paper offers an examination of the risks and benefits of open-sourcing highly capable foundation models. While open-sourcing has historically provided substantial net benefits for most software and AI development processes, we argue that for some highly capable foundation models likely to be developed in the near future, open-sourcing may pose sufficiently extreme risks to outweigh the benefits. In such a case, highly capable foundation models should not be open-sourced, at least not initially. Alternative strategies, including non-open-source model sharing options, are explored. The paper concludes with recommendations for developers, standard-setting bodies, and governments for establishing safe and responsible model sharing practices and preserving open-source benefits where safe.
AI Safety and the Age of Dislightenment
Click to have a preview.
Proposals for stringent AI model licensing and surveillance will likely be ineffective or counterproductive, concentrating power in unsustainable ways, and potentially rolling back the societal gains of the Enlightenment. The balance between defending society and empowering society to defend itself is delicate. We should advocate for openness, humility and broad consultation to develop better responses aligned with our principles and values — responses that can evolve as we learn more about this technology with the potential to transform society for good or ill.
Arguments against Open-sourcing
Critics of open-source models assess the potential for misuse from the following viewpoints:
Potentially Be Fine-Tuned into Detrimental Instances
Recommended Papers List
AI Safety and the Age of Dislightenment
Click to have a preview.
Proposals for stringent AI model licensing and surveillance will likely be ineffective or counterproductive, concentrating power in unsustainable ways, and potentially rolling back the societal gains of the Enlightenment. The balance between defending society and empowering society to defend itself is delicate. We should advocate for openness, humility and broad consultation to develop better responses aligned with our principles and values — responses that can evolve as we learn more about this technology with the potential to transform society for good or ill.
Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools
Click to have a preview.
As advancements in artificial intelligence propel progress in the life sciences, they may also enable the weaponisation and misuse of biological agents. This article differentiates two classes of AI tools that pose such biosecurity risks: large language models (LLMs) and biological design tools (BDTs). LLMs, such as GPT-4, are already able to provide dual-use information that could have enabled historical biological weapons efforts to succeed. As LLMs are turned into lab assistants and autonomous science tools, this will further increase their ability to support research. Thus, LLMs will in particular lower barriers to biological misuse. In contrast, BDTs will expand the capabilities of sophisticated actors. Concretely, BDTs may enable the creation of pandemic pathogens substantially worse than anything seen to date and could enable forms of more predictable and targeted biological weapons. In combination, LLMs and BDTs could raise the ceiling of harm from biological agents and could make them broadly accessible. The differing risk profiles of LLMs and BDTs have important implications for risk mitigation. LLM risks require urgent action and might be effectively mitigated by controlling access to dangerous capabilities. Mandatory pre-release evaluations could be critical to ensure that developers eliminate dangerous capabilities. Science-specific AI tools demand differentiated strategies to allow access to legitimate users while preventing misuse. Meanwhile, risks from BDTs are less defined and require monitoring by developers and policymakers. Key to reducing these risks will be enhanced screening of gene synthesis, interventions to deter biological misuse by sophisticated actors, and exploration of specific controls of BDTs.
Dual use of artificial-intelligence-powered drug discovery
Click to have a preview.
An international security conference explored how artificial intelligence (AI) technologies for drug discovery could be misused for de novo design of biochemical weapons. A thought experiment evolved into a computational proof.
Generative language models and automated influence operations: Emerging threats and potential mitigations
Click to have a preview.
Generative language models have improved drastically, and can now produce realistic text outputs that are difficult to distinguish from human-written content. For malicious actors, these language models bring the promise of automating the creation of convincing and misleading text for use in influence operations. This report assesses how language models might change influence operations in the future, and what steps can be taken to mitigate this threat. We lay out possible changes to the actors, behaviors, and content of online influence operations, and provide a framework for stages of the language model-to-influence operations pipeline that mitigations could target (model construction, model access, content dissemination, and belief formation). While no reasonable mitigation can be expected to fully prevent the threat of AI-enabled influence operations, a combination of multiple mitigations may make an important difference.
Inadvertently Encourage System Jailbreaks
Recommended Papers List
Universal and transferable adversarial attacks on aligned language models
Click to have a preview.
Because" out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures–so-called" jailbreaks" against LLMs–these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors …
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Click to have a preview.
We introduce Vicuna-13B, an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as a judge shows Vicuna-13B achieves more than 90%* quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90%* of cases. The cost of training Vicuna-13B is around $300. The code and weights, along with an online demo, are publicly available for non-commercial use.