1 Introduction 1 引言

Artificial Intelligence (AI) systems are increasingly used to make automated decisions that impact people to a significant extent. As the use of AI for automated decision-making increases, so do concerns over its harmful social consequences, including the undermining of democratic rule of law and the infringement of basic human rights to dignity and self-determination (e.g. Chiusi et al., 2020; Crawford et al., 2019). A way to counteract such harmful automated decision-making is through contestability. Contestable AI systems are open and responsive to human intervention throughout their lifecycle: not only after an automated decision has been made, but also during its design and development.
人工智能（AI）系统越来越被用于做出对人们产生重大影响的自动化决策。随着将 AI 用于自动化决策的增加，人们对其可能带来的有害社会后果也越来越担忧，包括破坏民主法治和侵犯基本人权尊严和自决权（例如 Chiusi 等，2020 年；Crawford 等，2019 年）。对抗这种有害的自动化决策的一种方式是通过可争议性。可争议的 AI 系统在其整个生命周期中都是开放的，并对人类干预做出响应：不仅在自动化决策做出后，而且在其设计和开发过程中。

A small but growing body of research explores the concept of contestable AI (Almada, 2019; Henin & Le Métayer, 2021; Hirsch et al., 2017; Lyons et al., 2021; Sarra, 2020; Vaccaro et al., 2019, 2021). However, although many do make practical recommendations, very little of this research is presented in a format readily usable in design practice. One such form of “intermediate-level generative design knowledge” (Höök & Löwgren, 2012; Löwgren et al., 2013) are design frameworks.
一小部分但不断增长的研究探讨了可争议人工智能的概念（Almada, 2019; Henin & Le Métayer, 2021; Hirsch 等，2017; Lyons 等，2021; Sarra, 2020; Vaccaro 等，2019, 2021）。然而，尽管许多人提出了实际建议，但很少有这些研究以设计实践可直接使用的格式呈现。一种“中级生成设计知识”形式（Höök & Löwgren, 2012; Löwgren 等，2013）是设计框架。

In this contribution we use qualitative interpretative methods supported by visual mapping techniques to develop a preliminary design framework that synthesizes elements identified through a systematic literature review, that contribute to contestability of AI systems. This preliminary framework serves as a starting point for subsequent testing and validation in specific application contexts.
在这项研究中，我们运用定性解释方法，并结合视觉映射技术，制定了一个初步设计框架，综合了通过系统文献综述确定的有助于人工智能系统可争议性的要素。这个初步框架作为后续在特定应用背景中进行测试和验证的起点。

Our framework consists of five system features and six development practices that contribute to contestable AI. The features are: 1. built-in safeguards against harmful behavior; 2. interactive control over automated decisions; 3. explanations of system behavior; 4. human review and intervention requests; and 5. tools for scrutiny by subjects or third parties. The practices are: 1. ex-ante safeguards; 2. agonistic approaches to machine learning (ML) development; 3. quality assurance during development; 4. quality assurance after deployment; 5. risk mitigation strategies; and 6. third-party oversight. We also offer a diagram for each set, capturing how features relate to various actors in a typical AI system, and how practices relate to typical AI system lifecycle stages.
我们的框架由五个系统特征和六种开发实践组成，这些特征和实践有助于可争议的人工智能。这些特征包括：1. 防止有害行为的内置保障措施；2. 对自动决策的交互式控制；3. 对系统行为的解释；4. 人工审查和干预请求；5. 提供供受试者或第三方审查的工具。这些实践包括：1. 前置保障措施；2. 对机器学习（ML）开发采取对抗性方法；3. 开发过程中的质量保证；4. 部署后的质量保证；5. 风险缓解策略；6. 第三方监督。我们还为每组提供了一个图表，展示了特征与典型人工智能系统中各种参与者的关系，以及实践与典型人工智能系统生命周期阶段的关系。

This paper is structured as follows: First we discuss why contestability is a necessary quality of AI systems used for automated decision-making. Then we situate our efforts in the larger field of responsible design for AI. We subsequently frame design frameworks as generative, intermediate-level knowledge. We then describe our method of constructing the design framework. Following this, we describe the literature review, and the elements we have identified in the included sources. Finally, we discuss the synthesis of these elements into our proposed design framework. We end with some concluding remarks.
本文结构如下：首先我们讨论了为什么竞争性是用于自动决策的人工智能系统的必要品质。然后我们将我们的努力置于更大的 AI 负责任设计领域中。我们随后将设计框架定位为生成的中级知识。然后我们描述了构建设计框架的方法。在此之后，我们描述了文献综述以及我们在所包含来源中确定的元素。最后，我们讨论了这些元素如何融合到我们提出的设计框架中。最后，我们以一些结论性的话结束。

2 Contestability in Automated Decision-Making
自动决策中的竞争性

The main focus of our effort is to ensure AI systems are open and responsive to contestation by those people directly or indirectly impacted throughout the system lifecycle. We define AI broadly, following Suchman (2018): “[a] cover term for a range of techniques for data analysis and processing, the relevant parameters of which can be adjusted according to either internally or externally generated feedback”.
我们努力的主要重点是确保人工智能系统对系统生命周期中直接或间接受影响的人们的质疑是开放的和响应的。我们广泛定义人工智能，遵循 Suchman（2018）的定义：“[一种涵盖数据分析和处理技术范围的术语，其相关参数可以根据内部或外部生成的反馈进行调整]”。

A growing number of scholars argue for contestability of AI systems in general, and in automated decision-making specifically (Almada, 2019; Hirsch et al., 2017; Sarra, 2020; Vaccaro et al., 2019).
越来越多的学者主张 AI 系统的可争议性，特别是在自动决策方面（Almada, 2019; Hirsch 等，2017; Sarra, 2020; Vaccaro 等，2019）。

Hirsch et al. (2017) describe contestability as “humans challenging machine predictions”. They claim models are and will continue to be fallible. In many cases, the cost of “getting it wrong” can be quite high for decision subjects, and those human controllers held responsible for AI system performance. Contestability ensures such failures are avoided by allowing human controllers to intervene before machine decisions are put into force.
Hirsch 等人（2017 年）将可争议性描述为“人类挑战机器预测”。他们声称模型是并将继续是可犯错误的。在许多情况下，“出错”的代价对于决策主体来说可能相当高，那些人类控制者要对 AI 系统的表现负责。可争议性确保通过允许人类控制者在机器决策实施之前进行干预来避免这种失败。

Vaccaro et al. (2019) argue that contestability can surface values, align design practice with context of use, and increase the perceived legitimacy of AI systems. Contestability is a “deep system property”, representing a coming together of human and machine to jointly make decisions. It aids iteration on decision-making processes and can be aimed at human controllers (“experts”) but also decision subjects. Contestability is a form of procedural justice, a way of giving voice to decision subjects, which increases perceptions of fairness, in particular for marginalized or disempowered populations.
Vaccaro 等人（2019 年）认为，可争议性可以凸显价值观，使设计实践与使用背景保持一致，并增加人工智能系统的感知合法性。可争议性是一种“深层系统属性”，代表人类和机器共同做出决策。它有助于对决策过程进行迭代，并可以针对人类控制者（“专家”）以及决策对象。可争议性是一种程序公正形式，是一种给予决策对象发声的方式，增加了对公平性的感知，特别是对边缘化或无权力的人群。

Almada (2019) argues that contestability protects decision subjects against flawed machine predictions, by enabling human intervention. Such human intervention can take place not only post-hoc, in response to an individual decision, but also ex-ante, as part of AI system development processes (Kamarinou et al., 2016). Ex-ante contestability allows for an “agonistic debate”, both internal and external, about data and modeling choices made to represent decision subjects, ensuring decisions comply with scientific, legal and democratic standards and values (Hildebrandt, 2017). Thus, contestability protects human self-determination and ensures human control over automated systems. Significant decisions do not only happen once a system is in operation and acting on subjects. Decisions are made throughout the system lifecycle. Contestability should therefore be part of the entire AI system development process: the practice of “contestability by design”.
Almada（2019）认为，可争议性保护决策主体免受有缺陷的机器预测的影响，通过实现人类干预。这种人类干预不仅可以在事后进行，作为对个体决策的回应，还可以在事前进行，作为人工智能系统开发过程的一部分（Kamarinou 等，2016）。事前的可争议性允许进行关于用于代表决策主体的数据和建模选择的“对抗性辩论”，确保决策符合科学、法律和民主标准和价值观（Hildebrandt，2017）。因此，可争议性保护人类自主权，并确保人类对自动化系统的控制。重要决策不仅发生在系统运行并对主体采取行动时。决策贯穿整个系统生命周期。因此，可争议性应成为整个人工智能系统开发过程的一部分：实践“设计中的可争议性”。

Finally, for Sarra (2020) contestability includes, but also exceeds, mere human intervention. Furthermore, it is distinct from simple opposition to automated decision-making. Instead, to contest is to engage with the substance of decisions themselves. It is more than voicing ones opinion. It requires an “articulate act of defense”. Such a defense requires arguments, and arguments need information. In this case, an explanation of the decision made. This must include both a description of the “how” and a justification of the “why”. Therefore, contestability demands explainability, and insofar as such explanations must include a justification specific to the case at hand, contestability also increases accountability. Most notably, contestability requires a “procedural relationship”. A “human in the loop” is insufficient if there is no possibility of a “dialectical exchange” between decision subject and human controller. Without such dialogue, there can be no exchange of arguments specific to the case at hand.
最后，对于 Sarra（2020 年）来说，可争议性包括，但也超越了仅仅的人为干预。此外，它与简单反对自动决策是不同的。相反，争议是与决策本身的实质进行互动。这不仅仅是发表意见。它需要一种“表达的辩护行为”。这样的辩护需要论据，而论据需要信息。在这种情况下，需要解释所做的决定。这必须包括对“如何”和“为什么”的描述。因此，可争议性要求可解释性，因为这样的解释必须包括特定于手头案例的理由，所以可争议性也增加了问责制。值得注意的是，可争议性要求“程序关系”。如果没有决策主体和人类控制者之间进行“辩证交流”的可能性，“人在回路”是不够的。没有这样的对话，就不可能交换特定于手头案例的论据。

In summary, contestability helps to protect against fallible, unaccountable, illegitimate, and unjust automated decision-making, by ensuring the possibility of human intervention as part of a procedural relationship between decision subjects and human controllers. The aim of this contribution is to develop a proposal for a framework for contestability both as an AI system quality (contestability features), and an AI system development practice (“contestability by design”).
总的来说，可争议性有助于防范易出错、不负责任、非法和不公正的自动决策，通过确保人类干预的可能性作为决策主体和人类控制者之间程序关系的一部分。本文的目的是提出一个关于可争议性框架的建议，既作为人工智能系统质量（可争议性特征），又作为人工智能系统开发实践（“设计中的可争议性”）。

3 Responsible Design for AI
3 AI 的负责任设计

As the adoption of AI continues to increase, so do concerns over its shortcomings, including lack of fairness, legitimacy and accountability. Such concerns cannot be met by purely technical solutions. They require a consideration of social and technical aspects in conjunction. This sociotechnical view emphasizes technical and social dimensions are entangled, producing specific outcomes irreducible to constitutive components (Franssen, 2015; Kroes et al., 2006). What is more, AI systems are distinct from “traditional” sociotechnical systems because they include “artificial agents” and humans interacting in a dynamic evolving environment (van de Poel, 2020). As a result, AI systems contain a particularly high degree of uncertainty and unpredictability.
随着人工智能的应用不断增加，人们对其缺陷，包括缺乏公平性、合法性和问责性的担忧也在增加。这些担忧不能仅通过技术解决方案来解决。它们需要综合考虑社会和技术方面。这种社会技术观强调技术和社会维度是纠缠在一起的，产生的具体结果无法简化为构成要素（Franssen, 2015; Kroes 等，2006）。此外，人工智能系统与“传统”社会技术系统不同，因为它们包括“人工代理”和人类在动态演变的环境中互动（van de Poel, 2020）。因此，人工智能系统包含特别高程度的不确定性和不可预测性。

Design, human–computer interaction (HCI) design in particular, is uniquely equipped to tackle such sociotechnical challenges, because it draws on both computer science and social science, joining positivist and interpretive traditions (Dourish, 2004; Katell et al., 2020; Tonkinwise, 2016). This allows interaction design to more adequately “see” AI systems. By virtue of its roots in traditional design, HCI design has the capacity to act in the face of complexity and ambiguity, by co-evolving problem and solution space in tandem (Dorst & Cross, 2001; Norman & Stappers, 2015).
设计，尤其是人机交互（HCI）设计，独特地能够应对这些社会技术挑战，因为它融合了计算机科学和社会科学，结合了实证主义和解释学传统（Dourish，2004 年；Katell 等，2020 年；Tonkinwise，2016 年）。这使得交互设计能够更充分地“看到”人工智能系统。由于其根源于传统设计，HCI 设计有能力在复杂性和模糊性面前行动，通过共同演化问题和解决方案空间（Dorst & Cross，2001 年；Norman & Stappers，2015 年）。

However, current design knowledge aimed at “responsible” and “ethical” AI is often of a high level of abstraction, and not connected to specific application domains. A lot of work is left for designers to translate such knowledge to their own practice. To illustrate this point we briefly summarize a number of prominent systematic reviews and meta-analyses drawn from across disciplines (Jobin et al., 2019; Morley et al., 2019; Shneiderman, 2020).
然而，目前针对“负责任”和“道德”的人工智能的设计知识往往具有很高的抽象水平，并且与特定的应用领域没有联系。设计师们需要大量工作来将这些知识转化为他们自己的实践。为了阐明这一观点，我们简要总结了一些来自不同学科的著名系统性审查和元分析（Jobin 等，2019 年；Morley 等，2019 年；Shneiderman，2020 年）。

Jobin et al. (2019) identify eleven overarching ethical values and principles. These are, in order of frequency of the number of sources featuring them:
Jobin 等人（2019 年）确定了十一项首要的道德价值观和原则。按照出现频率的顺序，这些价值观和原则如下：

transparency, justice and fairness, non-maleficence, responsibility, privacy, beneficence, freedom and autonomy, trust, dignity, sustainability, and solidarity.
透明度、公正和公平、非伤害、责任、隐私、利益、自由和自治、信任、尊严、可持续性和团结。

The first five principles are mentioned in over half of the sources. Importantly, Jobin et al note that, although there is convergence on the level of principles, the sources surveyed do diverge significantly in: 1. how they are interpreted; 2. why they are considered important; 3. what they should be applied to; and 4. how they should be implemented.
前五个原则在一半以上的来源中提到。重要的是，Jobin 等指出，尽管在原则的层面上存在一定的一致性，但调查的来源在以下方面存在显著分歧：1. 它们如何被解释；2. 为什么它们被认为重要；3. 它们应该应用于什么；以及 4. 它们应该如何被实施。

Morley et al. (2019) offer a more condensed set of themes, which together “define” ethically-aligned ML as:
Morley 等人（2019 年）提供了一组更为简洁的主题，这些主题共同“定义”了道德对齐的机器学习，即：

(a) beneficial to, and respectful of, people and the environment (beneficence); (b) robust and secure (non-maleficence); (c) respectful of human values (autonomy); (d) fair (justice); and (e) explainable, accountable and understandable (explicability).
(a) 对人和环境有益且尊重的（善行）；（b）强大且安全的（非伤害）；（c）尊重人类价值观的（自主权）；（d）公平的（正义）；以及（e）可解释、可追溯和易理解的（可解释性）。

Morley et al argue that principles are insufficient for changing actual AI systems design, and ethics scholars must do the hard work of translating the “what” of principles into the “how” of practices. By mapping principles to AI system lifecycle phases, they show current efforts are unevenly distributed, and where coverage exists, available solutions lack variety.
Morley 等人认为，原则对于改变实际的人工智能系统设计是不够的，伦理学者必须努力将原则的“是什么”转化为实践的“如何”。通过将原则映射到人工智能系统生命周期阶段，他们展示了当前的努力分布不均，而且在覆盖范围存在的地方，可用的解决方案缺乏多样性。

Finally, Shneiderman (2020) also notes there is a gap between principles and practice when it comes to “human-centered AI”. They offer 15 recommendations organized in a “a three-layer governance structure:”
最后，Shneiderman（2020）还指出，在涉及“以人为中心的人工智能”时，原则与实践之间存在差距。他们提出了 15 条建议，组织成“三层治理结构：”。

(1) reliable systems based on sound software engineering practices, (2) safety culture through proven business management strategies, and (3) trustworthy certification by independent oversight.
(1) 基于可靠的软件工程实践的系统，(2) 通过经过验证的商业管理策略建立安全文化，以及(3) 由独立监督机构进行的可信认证。

Shneiderman also points out it is necessary to move beyond general statements, towards support for specific social practices.
Shneiderman 还指出，有必要超越一般性陈述，支持特定的社会实践。

In short, currently available knowledge related to responsible and ethical AI is often of a high level of abstraction. Furthermore, scholars surveying the field agree it is necessary to translate principles into practices. Our aim is therefore to create knowledge of a more intermediate level, situated between theory and specific instances, in the form of a design framework.
简而言之，目前关于负责任和道德人工智能的相关知识往往具有较高的抽象水平。此外，对该领域进行调查的学者们一致认为有必要将原则转化为实践。因此，我们的目标是创造一种更中间层次的知识，位于理论和具体实例之间，以设计框架的形式呈现。

We focus on the principle of contestability in the context of automated decision-making. This principle stresses the sociotechnical character of AI systems: Contestability is about humans challenging machine decisions. It helps to surface values embedded in AI systems, aligning design with context of use. Contestability is a deep system property, linking humans and machines in joint decision-making. It enables agonistic debate about how models are made to represent the world in a particular way. Because human and AI decisions happen throughout the system lifecycle, what is needed is contestability by design.
我们关注自动决策背景下的可争议性原则。这一原则强调人工智能系统的社会技术特性：可争议性是指人类挑战机器决策。它有助于凸显嵌入在人工智能系统中的价值观，将设计与使用背景相一致。可争议性是一种深层系统属性，将人类和机器联系在一起进行共同决策。它促使人们就如何使模型以特定方式代表世界展开辩论。由于人类和人工智能的决策贯穿整个系统生命周期，所需的是设计上的可争议性。

In this paper we take the first step towards a design framework for contestable AI by summarizing ideas and mechanisms collated from previous work. Such mechanisms should align with the sociotechnical view, taking into account AI systems’ entangled and volatile nature. Future efforts may then make ready use of the resulting provisional framework, for purposes of testing and validation in specific application contexts.
在本文中，我们通过总结从先前工作中整理出的想法和机制，迈出了朝着可竞争的人工智能设计框架的第一步。这些机制应与社会技术观点保持一致，考虑到人工智能系统错综复杂且不稳定的特性。未来的努力可以利用由此产生的临时框架，用于在特定应用环境中进行测试和验证。

4 Design Frameworks as Generative Intermediate-Level Design Knowledge
4 个设计框架作为生成中级设计知识

We seek to construct a framework for the design of contestable AI systems. We conceive of a design framework as a form of “generative intermediate-level design knowledge” (Löwgren et al., 2013). Generative means it offers the seed for a design solution with particular qualities without fully prescribing its shape. Intermediate-level means it occupies a space between specific instances of designed artifacts, and generalized knowledge (theory). The design knowledge we seek to create describes particular sociotechnical system properties operationalizing the principle of contestability. We ground our framework in current knowledge on contestable AI. The purpose of the framework is to aid in the creation of designed artifacts. Following Stolterman and Wiberg (2010), we understand such design artifacts to be either in the service of improving a use situation, or in service of embodying new ideas (concepts) and theories. Our definition of “design framework” is aligned with Obrenović (2011). It should describe “the characteristics that a design solution should have to achieve a particular set of goals in a particular context”, where our goal is contestable AI in the context of automated decision-making.
我们旨在构建可争议人工智能系统设计的框架。我们将设计框架构想为一种“生成中间层设计知识”（Löwgren 等，2013）。生成意味着它提供了具有特定特质的设计解决方案的种子，而不完全规定其形状。中间层意味着它占据了设计制品的具体实例和广义知识（理论）之间的空间。我们试图创造的设计知识描述了操作可争议性原则的特定社会技术系统属性。我们将我们的框架基于当前关于可争议人工智能的知识。框架的目的是帮助创造设计制品。根据 Stolterman 和 Wiberg（2010），我们理解这种设计制品要么是为了改善使用情境，要么是为了体现新的想法（概念）和理论。我们对“设计框架”的定义与 Obrenović（2011）一致。 它应描述“设计解决方案应具备的特征，以在特定背景下实现特定目标”，我们的目标是在自动决策背景下具有争议性的人工智能。

5 Method of Design Framework Construction
设计框架构建的 5 种方法

We performed the following steps to construct our framework: We used a systematic review to collect sources discussing contestable AI. We then used reflexive thematic analysis to construct from the literature a number of elements contributing to contestable AI. Finally, we used visual mapping techniques to synthesize these elements into framework diagrams.
我们执行了以下步骤来构建我们的框架：我们使用系统性审查收集讨论有争议的人工智能的来源。然后，我们使用反思性主题分析从文献中构建了一些有助于有争议的人工智能的要素。最后，我们使用视觉映射技术将这些要素综合成框架图。

5.1 Data Collection 5.1 数据收集

Our data-collection procedure broadly follows Moher et al. (2009). Using Scopus, we searched for journal articles and conference papers published between 2016 and 2021 mentioning in their title, abstract or keywords “AI”, “contestability” and “design”. Synonyms for contestability were selected from the Merriam-Webster thesaurus entry for “contestation”Footnote 1. We used our best judgment to decide on related terms for AI. See Table 1 for an overview of search terms used. The exact Scopus search is as follows:
我们的数据收集程序大体遵循 Moher 等人（2009 年）的方法。使用 Scopus，我们搜索了在标题、摘要或关键词中提及“人工智能”、“可争议性”和“设计”的 2016 年至 2021 年间发表的期刊文章和会议论文。可争议性的同义词是从 Merriam-Webster 词典中“争议”一词的词库中选择的 Footnote 1 。我们凭借自己的判断力确定了与人工智能相关的术语。有关使用的搜索词的概述，请参见表 1。具体的 Scopus 搜索如下：

TITLE-ABS-KEY( (design*) AND (contest* OR controvers* OR debat* OR disagree* OR disput* OR dissen*) AND ("artificial intelligence" OR "AI" OR "machine learning" OR "ML" OR algorithm* OR "automated decision making") ) AND (PUBYEAR> 2015) AND (PUBYEAR< 2022) AND (LIMIT-TO(DOCTYPE, "cp") OR LIMIT-TO (DOCTYPE, "ar"))
标题-摘要-关键词((设计*) AND (竞赛* OR 争议* OR 辩论* OR 不同意* OR 争执* OR 分歧*) AND ("人工智能" OR "AI" OR "机器学习" OR "ML" OR 算法* OR "自动决策")) AND (出版年> 2015) AND (出版年< 2022) AND (限制为(文档类型, "cp") OR 限制为(文档类型, "ar"))

We collated the results, and first removed duplicates. Then, using Rayyan (Ouzzani et al., 2016), we manually screened records’ titles and abstracts for actually referring to contestability (rather than e.g. “contest” in the sense of a competition). The resulting set was assessed for eligibility on the basis of the full text. Here our criterion was whether papers did indeed discuss actionable sociotechnical system properties contributing to contestability. Once an initial set of inclusions was identified, we used Scopus to also screen 1. their references (i.e. “backward snowball”), and 2. all items referring to our inclusions (i.e. “forward snowball”). The resulting inclusions were once again assessed for eligibility. We then performed one final round of snowballing, screening, and qualitative assessment on the new inclusions. Figure 1 shows the stages of our systematic review.
我们整理了结果，首先去除了重复项。然后，使用 Rayyan（Ouzzani 等，2016 年），我们手动筛选记录的标题和摘要，确保实际涉及到可争议性（而不是例如“竞赛”之类的竞争意义）。根据全文对结果集进行了资格评估。我们的标准是论文是否确实讨论了对可争议性有贡献的可操作的社会技术系统属性。一旦确定了初始的包含集，我们还使用 Scopus 来筛选 1.它们的参考文献（即“向后滚雪球”），以及 2.所有涉及我们的包含内容的项目（即“向前滚雪球”）。结果的包含再次进行了资格评估。然后我们对新的包含内容进行了最后一轮滚雪球、筛选和定性评估。图 1 显示了我们系统评审的阶段。

Table 1 Search terms used
表 1 使用的搜索词
Full size table 完整尺寸表
Fig. 1 图 1
figure 1

Flow of information through the different phases of the systematic review
系统评价不同阶段的信息流动

Full size image 全尺寸图像

5.2 Analysis & Synthesis 5.2 分析与综合

Our approach to analysis and synthesis is adapted from reflexive thematic analysis as described by Braun and Clarke (2006). Our procedure was as follows: Analysis was done in Atlas.ti (version 22 on MacOS). We read the included sources and selected those passages discussing what we might call “active ingredients”: actionable sociotechnical system properties contributing to contestability. We grouped similar passages together, and assigned a label to each grouping capturing the essence of the property it represents. We then took the resulting list of properties, and looked for hierarchical and lateral relationships. In this step we relied heavily on visual mapping techniques, and used existing diagrams as a foundation. Once we had our preliminary framework, we checked the result against the selected passages, and against an end-to-end read-through of the source literature, to verify the framework properly covers and reflects it.
我们对分析和综合的方法是根据 Braun 和 Clarke（2006）描述的反思性主题分析进行调整的。我们的程序如下：分析是在 Atlas.ti（在 MacOS 上的 22 版本）中完成的。我们阅读了包含的来源，并选择了那些讨论我们可能称之为“活性成分”的段落：对可争议性有贡献的可操作的社会技术系统属性。我们将类似的段落分组在一起，并为每个分组分配一个标签，捕捉它所代表的属性的本质。然后，我们拿到了属性列表，寻找了层次和横向关系。在这一步骤中，我们大量依赖视觉映射技术，并将现有的图表作为基础。一旦我们有了初步的框架，我们将结果与选定的段落以及对源文献的端到端阅读进行核对，以验证框架是否正确覆盖和反映了它。

6 Elements in Extant Literature Contributing to Contestable AI
现有文献中有助于有争议的人工智能的 6 个要素

This section describes the elements we have identified in the literature. We have categorized them as either features or practices. They are summarized in Tables 2, 3 and 4, and are described in detail in the following sections.
本节描述了我们在文献中确定的元素。我们将它们分类为特征或实践。它们总结在表 2、表 3 和表 4 中，并在接下来的各节中详细描述。

6.1 Features 6.1 特点

6.1.1 Built-in Safeguards Against Harmful Behavior
6.1.1 防止有害行为的内置保障措施

This feature introduces procedural safeguards limiting what AI systems can do unilaterally. One such safeguard is to make the automated decision-making process itself adversarial. This can be achieved by introducing a second automated system external to the controlling organization, which machine decisions are run through. If disagreement between both systems occurs, decision can be flagged for human review, or automated dispute resolution mechanisms can take over. Such adversarial procedures could occur on an ongoing basis, or at the request of human controllers or decision subjects. An additional benefit of a second (possibly public) system that decisions need to pass through is the creation of a record of all decisions made, which can aid outside scrutiny (Almada, 2019; Elkin-Koren, 2020; Lyons et al., 2021).
该功能引入了程序保障措施，限制人工智能系统可以单方面执行的操作。其中一项保障措施是使自动决策过程本身具有对抗性。这可以通过引入一个第二个自动化系统来实现，该系统不属于控制组织，而是用于运行机器决策。如果两个系统之间发生分歧，决策可以被标记为人工审查，或者自动争议解决机制可以接管。这种对抗性程序可以持续进行，或者根据人类控制者或决策主体的要求进行。第二个（可能是公开的）系统的额外好处是，决策需要通过该系统，从而创建所有决策的记录，这有助于外部审查（Almada, 2019; Elkin-Koren, 2020; Lyons 等，2021）。

In some cases, it may be necessary and possible to implement formal constraints on system behavior. These would protect against undesired actions, and demonstrate compliance with standards and legislation (Aler Tubella et al., 2020).
在某些情况下，可能需要并且可能实施对系统行为的正式约束。这些约束将防止不良行为，并展示符合标准和法规（Aler Tubella 等，2020 年）。

6.1.2 Interactive Control Over Automated Decisions
6.1.2 自动决策的交互式控制

This feature is primarily aimed at human controllers, although in some cases it may also be made available to decision subjects. It enables direct intervention in machine decisions. In HCI, the concept of mixed-initiative interaction refers to shared control between intelligent systems and system users. Such an approach may also be employed in the case of decision-support or semi-automated decisions. The final decision would be the result of a “negotiation” between system and user (Kluttz & Mulligan, 2019; Novick & Sutton, 1997 in Vaccaro et al., 2019) In some cases it may be possible to allow users to correct or override a system decision. This is of particular importance in a decision-support setting, where such corrections may also function as a feedback loop for further system learning (Bayamlıoğlu, 2021; Hirsch et al., 2017; Vaccaro et al., 2019, 2020). Where direct override is not a possibility, some form of control can be offered in an indirect manner by allowing users to supplement the data a decision is based on with additional contextual information (Hirsch et al., 2017; Jewell, 2018).
该功能主要面向人类控制者，尽管在某些情况下也可能向决策主体开放。它使直接干预机器决策成为可能。在人机交互中，混合倡议交互的概念指的是智能系统和系统用户之间的共享控制。这种方法也可以在决策支持或半自动化决策的情况下使用。最终决策将是系统和用户之间的“协商”结果（Kluttz & Mulligan, 2019; Novick & Sutton, 1997 in Vaccaro 等人，2019）。在某些情况下，用户可能有可能纠正或覆盖系统决策。这在决策支持环境中尤为重要，这些纠正也可以作为进一步系统学习的反馈循环（Bayamlıoğlu, 2021; Hirsch 等人，2017; Vaccaro 等人，2019, 2020）。在无法直接覆盖的情况下，可以通过允许用户提供额外的上下文信息来以间接方式提供某种形式的控制（Hirsch 等人，2017; Jewell, 2018）。

6.1.3 Explanations of System Behavior
6.1.3 系统行为的解释

This feature is primarily aimed at decision subjects but can also be of use to human controllers. It helps actors understand the decisions made by AI systems. A decision subject should know a decision has been made, that there is a means of contesting, and be provided with an explanation of the decision (Lyons et al., 2021). Explanations should contain the information necessary for a decision subject to exercise their rights to human intervention and contestation (Bayamlıoğlu, 2021; Lyons et al., 2021; Ploug & Holm, 2020).
该功能主要针对决策主体，但也可供人类控制者使用。它帮助行动者理解人工智能系统所做的决定。决策主体应该知道已经做出了决定，有争议的手段，并且应该提供决定的解释（Lyons 等，2021 年）。解释应包含决策主体行使其人工干预和争议权利所需的信息（Bayamlıoğlu，2021 年；Lyons 等，2021 年；Ploug & Holm，2020 年）。

Individual decisions should be reproducible and traceable. It should be possible to verify the compliance of individual decisions with norms. This requires version control, and thorough record-keeping (Aler Tubella et al., 2020). Simply keeping an internal log could already be a huge improvement. These records should include the state of the model, the inputs, and decision rules at the time of producing a specific outcome (Bayamlıoğlu, 2021). The norms decisions should adhere to should be elicited and specified ex ante (Aler Tubella et al., 2020).
个人决策应当是可重现和可追溯的。应当可以验证个人决策与规范的一致性。这需要版本控制和彻底的记录保存（Aler Tubella 等，2020 年）。简单地保留内部日志已经是一个巨大的改进。这些记录应包括模型状态、输入和决策规则在产生特定结果时的状态（Bayamlıoğlu，2021 年）。决策应遵循的规范应在事前被引出和明确规定（Aler Tubella 等，2020 年）。

Explanations should not simply be a technical account of how a model’s output relates to its input. It should also include the organizational, social and legal context of the decision. In other words, the emphasis shifts from explaining the computational rules to the decision rules, offering a behavioral model of the AI system as a whole, from a sociotechnical perspective (Aler Tubella et al., 2020; Almada, 2019; Brkan, 2019; Crawford, 2016; Hirsch et al., 2017). This behavioral approach accounts for the limitations of transparency efforts focusing on “the algorithm” in isolation (Ananny & Crawford, 2018 in Henin & Le Métayer, 2021). It also seeks to strike a balance between usability and comprehensiveness, in an effort to avoid the “transparency paradox” (Nissenbaum, 2011 in Crawford, 2016).
解释不应仅仅是关于模型输出与输入之间关系的技术描述。它还应包括决策的组织、社会和法律背景。换句话说，重点从解释计算规则转向决策规则，提供一个从社会技术角度看整个人工智能系统的行为模型（Aler Tubella 等，2020 年；Almada，2019 年；Brkan，2019 年；Crawford，2016 年；Hirsch 等，2017 年）。这种行为方法考虑了透明度努力集中于孤立的“算法”所面临的限制（Ananny 和 Crawford，2018 年，见 Henin 和 Le Métayer，2021 年）。它还试图在可用性和全面性之间取得平衡，以避免“透明度悖论”（Nissenbaum，2011 年，见 Crawford，2016 年）。

These requirements should be satisfiable even for models that are opaque due to their technical nature. Nevertheless, it may be desirable to reduce model complexity, e.g. by limiting the number of features under consideration, or by using fundamentally more intelligible methods (e.g. decision trees vs. deep neural networks) (Bayamlıoğlu, 2021).
这些要求应该是可以满足的，即使对于由于其技术性质而不透明的模型。然而，减少模型复杂性可能是可取的，例如通过限制考虑的特征数量，或者使用基本更易理解的方法（例如，决策树与深度神经网络）（Bayamlıoğlu，2021）。

Although explanations may be of a static form, if deep understanding and exploration of counterfactual scenarios is desired, “sandboxing” or “black box in a glass box” approaches are worth considering. Using these approaches, users are able to manipulate inputs and see how these affect outputs. These techniques can work without needing to fully describe decision rules, which may be useful for cases where these cannot or will not be disclosed (Höök et al., 1998 in Hirsch et al., 2017). By offering explanations that include confidence levels, human controllers can direct their focus to those decisions warranting closer scrutiny (Hirsch et al., 2017; Vaccaro et al., 2019).
尽管解释可能是静态形式的，但如果希望深入理解和探索反事实情景，“沙盒化”或“玻璃箱中的黑匣子”方法值得考虑。使用这些方法，用户能够操纵输入并观察这些输入如何影响输出。这些技术可以在无需完全描述决策规则的情况下运作，这对于那些无法或不愿透露这些规则的情况可能是有用的（Höök 等人，1998 年，引自 Hirsch 等人，2017 年）。通过提供包括置信水平的解释，人类控制者可以将注意力集中在那些需要更仔细审查的决策上（Hirsch 等人，2017 年；Vaccaro 等人，2019 年）。

Another way to deal with model opacity (due to their proprietary or sensitive nature) is to generate local approximations using techniques such as “model inversion”. However, once again we emphasize not to fixate on the technical components of AI systems in isolation (Hirsch et al., 2017; Leahu, 2016; Mahendran & Vedaldi, 2015; Ribeiro et al., 2016; Tickle et al., 1998 in Edwards & Veale, 2018).
另一种处理模型不透明性（由于其专有或敏感性质）的方法是使用诸如“模型反演”之类的技术生成局部近似。然而，我们再次强调不要孤立地专注于人工智能系统的技术组成部分（Hirsch 等，2017 年；Leahu，2016 年；Mahendran 和 Vedaldi，2015 年；Ribeiro 等，2016 年；Tickle 等，1998 年，见 Edwards 和 Veale，2018 年）。

Explanations in the service of contestability should not simply describe why a decision was made, but also why the decision is considered good. In other words, decision subjects should receive a justification as well. This avoids the self-production of norms (Rouvroy, 2012 in Henin & Le Métayer, 2021).
在可争议性服务中的解释不应仅仅描述为什么做出了决定，还应该解释为什么认为这个决定是好的。换句话说，决策主体也应该得到理由。这样可以避免规范的自我生成（Rouvroy，2012 年，见 Henin & Le Métayer，2021 年）。

6.1.4 Human Review and Intervention Requests
6.1.4 人工审核和干预请求

This feature is aimed at decision subjects, and third parties acting on behalf of decision subject individuals and groups. It gives subjects the ability to “ask questions and record disagreements”, both on the individual and the aggregate scale (Hirsch et al., 2017; Ploug & Holm, 2020; Vaccaro et al., 2019).
该功能旨在决策主体以及代表决策主体个人和团体行事的第三方。它赋予主体“提出问题和记录异议”的能力，无论是在个体还是整体规模上（Hirsch 等，2017 年；Ploug 和 Holm，2020 年；Vaccaro 等，2019 年）。

Human controllers and decision subjects should not be mere passive recipients of automated decisions. They should be put in dialogue with AI systems. Reliance on out-of-system mechanisms for contestation is insufficient (Kluttz et al., 2019 in Henin & Le Métayer, 2021).
人类控制者和决策主体不应只是自动决策的被动接受者。他们应与人工智能系统进行对话。依赖系统外机制进行争议是不足够的（Kluttz 等人，2019 年，见 Henin 和 Le Métayer，2021 年）。

A commonly recommended mechanism for responding to post-hoc contestation is human review and intervention (Lyons et al., 2021). Requests for human intervention are necessarily post-hoc, since they happen in response to discrete decisions, when a subject feels a decision has harmed or otherwise impacted their rights, freedoms or interests (Almada, 2019). Such intervention requests could be facilitated through auxiliary platforms, or be part of the system itself (Almada, 2019; Bayamlıoğlu, 2021). Although existing internal or external review procedures are sometimes considered sufficient, in many cases new mechanisms for contestation will be required. Due process mechanisms should be designed into the AI systems itself (Lyons et al., 2021).
对于事后争议的常见建议机制是人工审查和干预（Lyons 等，2021 年）。对人工干预的请求必然是事后的，因为它们是针对离散决定而发生的，当主体感到某个决定损害或以其他方式影响了他们的权利、自由或利益时（Almada，2019 年）。这种干预请求可以通过辅助平台进行促进，或者成为系统本身的一部分（Almada，2019 年；Bayamlıoğlu，2021 年）。尽管现有的内部或外部审查程序有时被认为是足够的，但在许多情况下，将需要新的争议机制。应该将正当程序机制设计到 AI 系统本身中（Lyons 等，2021 年）。

Human review is seen as an antidote to machine error. Human controllers can use tacit knowledge, intuition, and access to contextual information to identify and correct harmful automated decisions. In this way, allowing for human intervention is a form of quality control (Almada, 2019; Walmsley, 2021).
人工审查被视为对机器错误的解药。人类控制者可以利用隐性知识、直觉和获取背景信息的能力来识别和纠正有害的自动化决策。因此，允许人类干预是一种质量控制形式（Almada, 2019; Walmsley, 2021）。

In the context of GDPR the right to human intervention is tied to fully automated decision-making only (Brkan, 2019). In practice, such a distinction may not be so clear-cut. From a sociotechnical perspective humans are always part of the decision chain leading up to a machine decision, in the role of designers, developers and operators. Furthermore, the mere presence of a human at the very end of the chain (the so-called “human in the loop”) may not be a sufficient safeguard against machine error if human controllers do not have the authority or ability to base their final decision on more information than what was provided to them by the AI system (Almada, 2019). By extension, human controllers who respond to intervention request should have the authority and capability to actually change previous decisions (Brkan, 2019).
在 GDPR 的背景下，人类干预权利仅与完全自动化决策相关联（Brkan，2019）。实际上，这样的区分可能并不那么清晰。从社会技术的角度来看，人类始终是决策链中的一部分，扮演着设计师、开发人员和操作者的角色，导致机器决策的产生。此外，即使在链的最后出现人类（所谓的“人在环中”），如果人类控制者没有权威或能力基于比 AI 系统提供给他们的更多信息做出最终决定，那么人类的存在也可能无法有效防范机器错误（Almada，2019）。因此，对干预请求做出响应的人类控制者应具有权威和能力实际改变先前的决定（Brkan，2019）。

It is of course entirely possible for human intervention to be biased, leading to worse outcomes compared to a fully automated decision. This should be guarded against by introducing comparative measures of the performance of human-controlled and fully automated procedures (Almada, 2019). AI system controllers must make room within their organizations for receiving, evaluating and responding to disputes (Sarra, 2020).
人类干预存在偏见的可能性，可能导致比完全自动化决策更糟糕的结果。应通过引入对人控制和完全自动化程序性能的比较措施来防范这种情况（Almada，2019）。人工智能系统控制者必须在其组织内留出接收、评估和回应争议的空间（Sarra，2020）。

Channels for contestation should be clear, accessible, affordable and efficient so that further harm to subjects is minimized (Lyons et al., 2021; Vaccaro et al., 2021). Mechanisms for requesting human intervention should provide “scaffolding for learning” (Applebee & Langer, 1983; Salehi et al., 2017 in Vaccaro et al., 2020). Documentation of the decision-making procedures should be integrated with the appeal procedure and communicated in alternative formats to ease comprehension (Vaccaro et al., 2020) and to help subjects in formulating their argument (Lyons et al., 2021; Vaccaro et al., 2021)
争议渠道应明确、易于访问、价格合理且高效，以最大程度减少对对象的进一步伤害（Lyons 等，2021 年；Vaccaro 等，2021 年）。请求人类干预的机制应提供“学习的支架”（Applebee 和 Langer，1983 年；Salehi 等，2017 年，见 Vaccaro 等，2020 年）。决策程序的文档应与上诉程序整合，并以替代格式传达，以便易于理解（Vaccaro 等，2020 年），并帮助对象构建自己的论点（Lyons 等，2021 年；Vaccaro 等，2021 年）。

A risk of appeal procedures is that burdens are shifted to individual subjects. Ways of addressing this include allowing for synchronous communication with decision makers (Vaccaro et al., 2021), or to have third parties represent subjects (Bayamlıoğlu, 2021; Edwards & Veale, 2018; Lyons et al., 2021; Vaccaro et al., 2020).
上诉程序的风险之一是将负担转移到个体主体身上。解决这个问题的方法包括允许与决策者进行同步沟通（Vaccaro 等，2021 年），或让第三方代表主体（Bayamlıoğlu，2021 年；Edwards & Veale，2018 年；Lyons 等，2021 年；Vaccaro 等，2020 年）。

Another limitation of current appeal procedures is that they handle decisions individually (Vaccaro et al., 2019). Groups should be able to acquire explanations of decisions collectively. Developers should not only consider individual impacts, but also group impacts (Edwards & Veale, 2018). Mechanisms for contestability should allow for collective action, because harms can be connected to group membership (Lyons et al., 2021). One way to aid collective action would be to publicize individual appeals cases so subjects can compare their treatment to those of others, and identify fellow sufferers (Matias et al., 2015; Myers West 2018; Sandvig et al., 2014 in Vaccaro et al., 2020). Subjects should be supported in connecting to those who share their fate (Vaccaro et al., 2021).
当前上诉程序的另一个局限性是它们单独处理决定（Vaccaro 等人，2019 年）。团体应该能够集体获得决定的解释。开发者不仅应考虑个体影响，还应考虑团体影响（Edwards 和 Veale，2018 年）。争议机制应允许集体行动，因为伤害可能与团体成员身份相关联（Lyons 等人，2021 年）。促进集体行动的一种方式是公开个别上诉案例，以便受试者可以将自己的待遇与他人进行比较，并识别同病相怜者（Matias 等人，2015 年；Myers West，2018 年；Sandvig 等人，2014 年，见 Vaccaro 等人，2020 年）。受试者应得到支持，以便与那些分享同命运的人联系（Vaccaro 等人，2021 年）。

Any kind of human intervention in response to decision subjects’ appeals may not qualify as actual contestation. Decision subjects should be able to express their point of view, if only to provide additional information based on which a decision may be reconsidered (Bayamlıoğlu, 2021). For true contestation to be the case, not only should the subject be allowed to express their point of view, but there should also be a dialectical exchange between subject and controller (Mendoza & Bygrave, 2017 in Brkan 2019). Therefore, contestation includes human intervention, but should not be reduced to it. Care should also be taken to avoid contestability becomes merely a way for subjects to complain about their plight. This means contestations of these kinds cannot be handled in a fully automated fashion, because a dialectic exchange is not possible in a meaningful sense between humans and machines. Computational logic can only offer an answer to the “how”, whereas a proper response to a contestation must also address the “why” of a given decision (Sarra, 2020). Contestability should include a right to a new decision, compensation of harm inflicted, or reversal (Lyons et al., 2021).
任何对决策主体诉求的人为干预都可能不符合实际争议。决策主体应该能够表达他们的观点，即使只是提供基于这些观点可以重新考虑决定的额外信息（Bayamlıoğlu，2021）。为了确保真正的争议存在，主体不仅应被允许表达他们的观点，还应该存在主体和控制者之间的辩证交流（Mendoza & Bygrave，2017 in Brkan 2019）。因此，争议包括人为干预，但不应被简化为此。还应注意避免争议仅仅成为主体抱怨困境的一种方式。这意味着这类争议无法完全以自动化方式处理，因为在人类和机器之间在有意义的意义上进行辩证交流是不可能的。计算逻辑只能回答“如何”，而对争议的适当回应还必须解决给定决定的“为什么”（Sarra，2020）。争议应包括获得新决定的权利、赔偿受到的伤害或撤销（Lyons 等，2021）。

6.1.5 Tools for Scrutiny by Subjects or Third Parties
6.1.5 主体或第三方审查工具

This feature supports scrutiny by outside actors (decision subjects, indirect stakeholders, third parties) of AI systems, separate from individual decisions. These tools for scrutiny mainly take the form of a range of information resources.
该功能支持外部参与者（决策主体、间接利益相关者、第三方）对人工智能系统进行审查，与个体决策分开。这些审查工具主要采取各种信息资源的形式。

These should contribute to the contestability of the sociotechnical system in its entirety (Lyons et al., 2021). The aim is to justify the system as a whole (i.e. “globally”), rather than individual decisions (“locally”). This requires the demonstration of a clear link between high-level objectives (norms external to the technical system) and its implemention. Compliance is established by tracing this link through requirements, specifications, and the code itself.
这些应有助于整个社会技术系统的可争议性（Lyons 等，2021 年）。目标是证明系统作为一个整体（即“全局”），而不是个别决策（“局部”）的合理性。这需要展示高层目标（技术系统外部的规范）与其实施之间的明确联系。通过跟踪这一联系，可以建立合规性，包括需求、规范和代码本身。

Documentation should describe the technical composition of the system (Vaccaro et al., 2020). Such documentation may include up-to-date system performance indicators, in particular related to training data and models. Further documentation should describe how the system was constructed (i.e. documentation of the design and development process) (Selbst & Barocas, 2018 in Almada 2019), the role of human decision-makers, group or systemic impacts and how they are safeguarded against (Lyons et al., 2021). Mitchell et al. (2019) and Gebru et al. (2020) offer examples of possible documentation approaches.
文档应描述系统的技术构成（Vaccaro 等，2020 年）。这样的文档可能包括最新的系统性能指标，特别是与训练数据和模型相关的指标。进一步的文档应描述系统是如何构建的（即设计和开发过程的文档）（Selbst＆Barocas，2018 年在 Almada 2019 年），人类决策者的角色，群体或系统影响以及如何防范（Lyons 等，2021 年）。Mitchell 等人（2019 年）和 Gebru 等人（2020 年）提供了可能的文档编制方法的示例。

Formal proof of compliance may be possible when a system specification can be described unambiguously, and its implementation can be verified (semi-)automatically. However, ML-based systems cannot be described using formal logic. Their performance is better assessed through statistical means (Henin & Le Métayer, 2021).
当系统规范能够明确描述，并且其实施可以（半）自动验证时，合规性的正式证明可能是可能的。然而，基于机器学习的系统无法用形式逻辑描述。它们的性能更好地通过统计手段进行评估（Henin & Le Métayer, 2021）。

If a system makes a fully automated decision, it is recommended to include a means of comparing its performance to an equivalent decision-making procedure made by humans (Cowgill & Tucker, 2017 in Almada 2019).
如果一个系统做出完全自动化的决策，建议包括一种比较其性能与人类做出的等效决策程序的手段（Cowgill & Tucker, 2017 in Almada 2019）。

If confidential or sensitive information must be protected that would aid in the assessment of proper system performance, it may be possible to employ “zero-knowledge proofs” in order to provide so-called opaque assurances (Kroll et al., 2016 in Almada 2019).
如果必须保护机密或敏感信息以帮助评估系统性能，可能可以采用“零知识证明”来提供所谓的不透明保证（Kroll 等人，2016 年，Almada 2019 年）。

6.2 Practices 6.2 实践

6.2.1 Ex-ante Safeguards 6.2.1 前置保障措施

This practice focuses on the earliest stages of the AI system lifecycle, during the business and use-case development phase. It aims to put in place policy-level constraints protecting against potential harms. Developers should make an effort to anticipate the impacts of their system in advance (Brkan, 2019; Henin & Le Métayer, 2021; Sarra, 2020), and pay close attention to how the system may “mediate” new and existing social practices (Verbeek 2015 in Hirsch et al., 2017). If after an initial exploration it becomes clear impacts are potentially significant or severe, a more thorough and formalized impact assessment should be performed (e.g. Data Protection Impact Assessments (DPIA)) (Edwards & Veale, 2018; Lyons et al., 2021). Such assessments can also enforce production of extensive technical documentation in service of transparency, and by extension contestability (Bayamlıoğlu, 2021). Any insights from this act of anticipation should feed into the subsequent phases of the AI system lifecycle. Considering AI system development tends to be cyclical and ongoing, anticipation should be revisited with every proposed change (Schot & Rip, 1997 in Kariotis and Mir 2020). If system decisions are found to impact individuals or groups to a significant extent, contestability should be made a requirement (Henin & Le Métayer, 2021). A fairly obvious intervention would be to make contestability part of a system’s acceptance criteria. This would include the features identified in our framework, first and foremost means of acquiring explanation and human intervention (Almada, 2019; Brkan, 2019; Walmsley, 2021). Questions that must be answered at this point include what can be contested, who can contest, who is accountable, and what type of review is necessary (Lyons et al., 2021).
该实践侧重于人工智能系统生命周期的最早阶段，即业务和用例开发阶段。它旨在制定政策级别的约束，以防范潜在危害。开发人员应努力预见其系统的影响（Brkan，2019 年；Henin 和 Le Métayer，2021 年；Sarra，2020 年），并密切关注系统可能如何“调解”新的和现有的社会实践（Verbeek 2015 年，引自 Hirsch 等人，2017 年）。如果在初步探索后发现影响可能显著或严重，应进行更彻底和正式的影响评估（例如数据保护影响评估（DPIA））（Edwards 和 Veale，2018 年；Lyons 等人，2021 年）。这种评估还可以强制制定广泛的技术文档，以促进透明度，并在延伸上进行争议（Bayamlıoğlu，2021 年）。从这种预见行为中获得的任何见解都应反映在人工智能系统生命周期的后续阶段中。考虑到人工智能系统开发往往是循环和持续的，预见应随着每一项拟议的变更而重新审视（Schot 和 Rip，1997 年，引自 Kariotis 和 Mir，2020 年）。 如果发现系统决策对个人或群体产生重大影响，应将可争议性作为要求（Henin & Le Métayer, 2021）。一个相当明显的干预措施是将可争议性纳入系统的验收标准。这将包括我们框架中确定的特征，首要是获取解释和人类干预的手段（Almada, 2019; Brkan, 2019; Walmsley, 2021）。在这一点上必须回答的问题包括什么可以争议，谁可以争议，谁负责，以及需要什么类型的审查（Lyons 等，2021）。

A final type of ex-ante safeguard is certification. This can be applied to the AI system as a software object, by either specifying aspects of its technological design directly, or by requiring certain outputs that enable monitoring and evaluation. It may also be applied to the controlling organization as a whole, which from a sociotechnical perspective is the more desirable option, seeing as how automated decisions cannot be reduced to an AI system’s data and model. However, certificates and seals are typically run in a for-profit manner and depend on voluntary participation by organizations. As such they struggle with enforcement. Furthermore, there is little evidence that certificates and seals lead to increased trust on behalf of subjects (Bayamlıoğlu, 2021; Edwards & Veale, 2018).
一种最终的前置保障类型是认证。这可以应用于人工智能系统作为软件对象，要么直接指定其技术设计的方面，要么要求产生某些输出以实现监测和评估。它也可以应用于整个控制组织，从社会技术角度来看，这是更理想的选择，因为自动化决策不能简化为人工智能系统的数据和模型。然而，证书和印章通常以盈利方式运作，并依赖组织的自愿参与。因此，它们在执法方面存在困难。此外，很少有证据表明证书和印章会增加受试者的信任（Bayamlıoğlu, 2021; Edwards & Veale, 2018）。

6.2.2 Agonistic Approaches to ML Development
6.2.2 机器学习开发的对抗性方法

This practice relates to the early lifecycle phases of an AI system: business and use-case development, design, and training and test data procurement. The aim of this practice is to support ways for stakeholders to “explore and enable alternative ways of datafying and modeling the same event, person or action” (Hildebrandt, 2017 in Almada 2019). An agonistic approach to ML development allows for decision subjects, third parties, and indirect stakeholders to “co-construct the decision-making process” (Vaccaro et al., 2019). The choices of values embedded in systems should be subject to broad debate facilitated by elicitation of the, potentially conflicting, norms at stake (Henin & Le Métayer, 2021). This approach stands in contrast to ex-post mechanisms for contestation, which can only go so far in protecting against harmful automated decisions because they are necessarily reactive in nature (Almada, 2019; Edwards & Veale, 2018). In HCI, a well-established means of involving stakeholders in the development of technological systems is participatory design (Davis, 2009 in Almada 2019). By getting people involved in the early stages of the AI lifecycle, potential issues can be flagged before they manifest themselves through harmful actions (Almada, 2019). Participants should come from those groups directly or indirectly affected by the specific AI systems under consideration. Due to the scale at which many AI systems operate, direct engagement with all stakeholders might be hard or impossible. In such cases, representative sampling techniques should be employed, or collaboration should be sought with third parties representing the interests of stakeholder groups (Almada, 2019). Representation can be very direct (similar to “jury duty”). Or more indirect (volunteer or elected representatives forming a board or focus group) (Vaccaro et al., 2021).
这种做法涉及人工智能系统的早期生命周期阶段：业务和用例开发、设计以及培训和测试数据采购。这种做法的目的是支持利益相关者“探索和启用数据化和建模相同事件、人物或行为的替代方式”（Hildebrandt，2017，Almada 2019）。对机器学习开发采取对抗性方法允许决策主体、第三方和间接利益相关者“共同构建决策过程”（Vaccaro 等，2019）。系统中嵌入的价值选择应该受到广泛辩论的影响，通过引发潜在冲突的利益相关方规范（Henin & Le Métayer，2021）来促进。这种方法与事后争议机制形成对比，后者在保护免受有害自动决策方面只能走得那么远，因为它们本质上是被动的（Almada，2019；Edwards & Veale，2018）。在人机交互领域，一种既定的让利益相关者参与技术系统开发的方法是参与式设计（Davis，2009，Almada 2019）。 通过让人们参与到人工智能生命周期的早期阶段，潜在问题可以在它们通过有害行为表现出来之前被发现（Almada, 2019）。参与者应该来自直接或间接受到特定人工智能系统影响的群体。由于许多人工智能系统运作的规模，直接与所有利益相关者进行接触可能很困难或不可能。在这种情况下，应采用代表性抽样技术，或寻求与代表利益相关者群体利益的第三方合作（Almada, 2019）。代表性可以是非常直接的（类似于“陪审团职责”）。或更间接的（志愿者或选举产生的代表组成委员会或焦点小组）（Vaccaro 等，2021）。

Power differentials may limit the degree to which stakeholders can actually affect development choices. Methods should be used that ensure participants are made aware of and deal with power differentials (Geuens et al., 2018; Johnson, 2003 in Kariotis and Mir 2020).
权力差异可能限制利益相关者实际影响发展选择的程度。应采用确保参与者意识到并处理权力差异的方法（Geuens 等，2018 年；Johnson，2003 年，见 Kariotis 和 Mir，2020 年）。

One-off consultation efforts are unlikely to be sufficient, and run the risk of being reduced to mere “participation theater” or a ticking-the-box exercise. Participation, in the agonistic sense, implies an ongoing adversarial dialogue between developers and decision subjects (Kariotis & Mir, 2020).Footnote 2 AI systems, like all designed artifacts, embody particular political values (Winner, 1980 in Crawford 2016). A participatory, agonistic approach should be aimed at laying bare these values, and to create an arena in which design choices supporting one value over an other can be debated and resolved (although such resolutions should always be considered provisional and subject to change) (Kariotis & Mir, 2020). König and Wenzelburger (2021) offer an outline of one possible way of structuring such a process.
一次性的咨询努力不太可能足够，存在被简化为纯粹的“参与戏剧”或仅仅是打勾的风险。参与，在对抗性意义上，意味着开发者和决策主体之间持续的对抗性对话（Kariotis & Mir, 2020）。AI 系统，像所有设计的工件一样，体现特定的政治价值（Winner, 1980 in Crawford 2016）。一个参与式、对抗性的方法应该旨在揭示这些价值，并创造一个设计选择支持一种价值而不是另一种价值的辩论和解决的舞台（尽管这样的解决方案应始终被视为暂时的并可能会发生变化）（Kariotis & Mir, 2020）。König 和 Wenzelburger（2021）提供了一个可能的结构化这一过程的概要。

6.2.3 Quality Assurance During Development
6.2.3 开发过程中的质量保证

This practice ensures safe system performance during the development phases of the AI system lifecycle. This includes collection of data and training of models, programming, and testing before deployment. A tried and true approach is to ensure the various stakeholder rights, values and interests guide development decisions. Contestability should not be an afterthought, a “patch” added to a system once it has been deployed. Instead developers should ensure the system as a whole will be receptive and responsive to contestations. Care should also be taken to understand the needs and capabilities of human controllers so they will be willing and able to meaningfully intervene when necessary (Kluttz et al., 2018; Kluttz and Mulligan 2019; Leydens & Lucena, 2018 in Almada, 2019; Kariotis & Mir, 2020; Hirsch et al., 2017). Before deploying a system, it can be tested, e.g. for potential bias, by applying the model to datasets with relevant differences (Ploug & Holm, 2020). Given the experimental nature of some AI systems, it may be very challenging to foresee all potential impacts beforehand, on the basis of tests in lab-like settings alone. In such cases, it may be useful to evaluate system performance in the wild using a “living lab” approach (Kariotis & Mir, 2020). In any case, development should be set up in such a way that feedback from stakeholders is collected before actual deployment, and time and resources are available to perform multiple rounds of improvement before proceeding to deployment (Hirsch et al., 2017; Vaccaro et al., 2019, 2020). Developers should seek feedback from stakeholders both with respect to system accuracy, and ethical dimensions (e.g. fairness, justice) (Walmsley, 2021).
这种做法确保了人工智能系统生命周期开发阶段的安全性能。这包括数据收集和模型训练、编程和部署前的测试。一种经过验证的方法是确保各利益相关者的权利、价值观和利益指导开发决策。可争议性不应该是事后的想法，一种“补丁”是在系统部署后才添加的。相反，开发人员应确保整个系统能够接受和响应争议。还应该注意理解人类控制者的需求和能力，以便他们愿意并能够在必要时进行有意义的干预（Kluttz 等，2018 年；Kluttz 和 Mulligan，2019 年；Leydens 和 Lucena，2018 年在 Almada，2019 年；Kariotis 和 Mir，2020 年；Hirsch 等，2017 年）。在部署系统之前，可以通过将模型应用于具有相关差异的数据集来测试其潜在偏见（Ploug 和 Holm，2020 年）。鉴于一些人工智能系统的实验性质，仅仅依靠实验室环境中的测试可能非常具有挑战性，难以事先预见所有潜在影响。 在这种情况下，使用“实验室”方法评估系统在实际环境中的性能可能是有用的（Kariotis & Mir, 2020）。无论如何，开发应该以一种收集利益相关者反馈意见的方式设置，以便在实际部署之前收集反馈意见，并且有时间和资源进行多轮改进后再进行部署（Hirsch 等，2017；Vaccaro 等，2019，2020）。开发人员应该寻求利益相关者的反馈意见，无论是关于系统准确性，还是伦理维度（例如公平性，正义）（Walmsley, 2021）。

6.2.4 Quality Assurance After Deployment
6.2.4 部署后的质量保证

This practice relates to the AI system lifecycle phases following deployment. It is aimed at monitoring performance and creating a feedback loop to enable ongoing improvements. The design concept “procedural regularity” captures the idea that one should be able to determine if a system actually does what it is declared to be doing by its developers. In particular when models cannot be simplified, additional measures are required to demonstrate procedural regularity, including monitoring (Bayamlıoğlu, 2021). System operators should continuously monitor system performance for unfair outcomes both on individuals, and in the aggregate, on communities. To this end, mathematical models can be used to determine if a given model is biased against individuals or groups (Goodman, 2016 in Almada 2019). Monitoring should also be done for potential misuse of the system. Corrections, appeals, and additional contextual information from human controllers and decision subjects can be used as feedback signals for the decision-making process as a whole (Hirsch et al., 2017; Vaccaro et al., 2020). In some cases, feedback loops back to training can be created by means of “reinforcement learning”, where contestations are connected to reward functions. In decision-support settings, such signals can also be derived from occurrences where human controllers reject system predictions (Walmsley, 2021).
这种做法涉及到 AI 系统在部署后的生命周期阶段。其目的在于监测性能并创建反馈循环，以实现持续改进。设计概念“程序规律性”捕捉到了这样一个观念，即人们应该能够确定系统是否确实按照开发者所声明的方式运行。特别是当模型无法简化时，需要采取额外措施来证明程序规律性，包括监测（Bayamlıoğlu, 2021）。系统操作员应该持续监测系统性能，以防止对个人和整个社区产生不公平的结果。为此，数学模型可以用来确定某个模型是否对个人或群体存在偏见（Goodman, 2016 in Almada 2019）。监测还应该针对系统潜在的滥用进行。来自人类控制者和决策主体的纠正、上诉和额外的背景信息可以作为整个决策过程的反馈信号（Hirsch 等，2017；Vaccaro 等，2020）。 在某些情况下，反馈回到培训可以通过“强化学习”来创建，其中争议与奖励功能相连。在决策支持设置中，这些信号也可以来自人类控制器拒绝系统预测的发生（Walmsley，2021）。

6.2.5 Risk Mitigation Strategies
6.2.5 风险缓解策略

This practice relates to all phases of the AI system lifecycle. The aim is to intervene in the broader context in which systems operate, rather than to change aspects of what is commonly considered systems themselves. One strategy is to educate system users on the workings of the systems they operate or are subject to. Such training and education efforts should focus on making sure users understand how systems work, and what their strengths and limitations are. Improving users’ understanding of systems may: 1. discourage inappropriate use and encourage adoption of desirable behavior; 2. prevent erroneous interpretation of model predictions; 3. create a shared understanding for the purposes of resolving disputes; and 4. ensure system operators along decision chains are aware of risks and responsibilities (Hirsch et al., 2017; Lyons et al., 2021; Ploug & Holm, 2020; Vaccaro et al., 2019, 2020).
这种做法涉及人工智能系统生命周期的所有阶段。其目的是干预系统运行的更广泛背景，而不是改变通常被认为是系统本身的方面。一种策略是教育系统用户了解他们操作或受其约束的系统的运作方式。这种培训和教育工作应侧重于确保用户了解系统的工作原理以及其优势和局限性。提高用户对系统的理解可能会：1. 阻止不当使用并鼓励采纳良好行为；2. 防止对模型预测的错误解释；3. 为解决争议创造共同理解；以及 4. 确保决策链上的系统操作人员了解风险和责任（Hirsch 等，2017 年；Lyons 等，2021 年；Ploug 和 Holm，2020 年；Vaccaro 等，2019 年，2020 年）。

6.2.6 Third-Party Oversight
6.2.6 第三方监督

This practice relates to all phases of the AI system lifecycle. Its purpose is to strengthen the supervising role of trusted third party actors such as government agencies, civil society groups, and NGOs. As automated decision-making happens at an increasingly large scale, it will be necessary to establish new forms of ongoing outside scrutiny (Bayamlıoğlu, 2021; Edwards & Veale, 2018; Elkin-Koren, 2020; Vaccaro et al., 2019). System operators may be obligated to implement model-centric tools for ongoing auditing of systems’ overall compliance with rules and regulations (Bayamlıoğlu, 2021). Companies may resist opening up proprietary data and models for fear of losing their competitive edge and users “gaming the system” (Crawford, 2016). Where system operators have a legitimate claim to secrecy, third parties can act as trusted intermediaries to whom sensitive information is disclosed, both for ex-ante inspection of systems overall and post-hoc contestation of individual decisions (Bayamlıoğlu, 2021). Such efforts can be complemented with the use of technological solutions including secure environments which function as depositories for proprietary or sensitive data and models (Edwards & Veale, 2018).
这一实践涉及人工智能系统生命周期的所有阶段。其目的是加强政府机构、民间社会团体和非政府组织等可信第三方行为者的监督作用。随着自动决策规模不断扩大，建立新形式的持续外部审查将是必要的（Bayamlıoğlu, 2021; Edwards & Veale, 2018; Elkin-Koren, 2020; Vaccaro 等，2019）。系统运营商可能有义务实施以模型为中心的工具，持续审计系统整体遵守规则和法规（Bayamlıoğlu, 2021）。公司可能会抵制开放专有数据和模型，担心失去竞争优势和用户“操纵系统”（Crawford, 2016）。在系统运营商有合法保密要求的情况下，第三方可以充当可信中介，向其披露敏感信息，既用于系统整体的前期检查，也用于对个别决策的事后争议（Bayamlıoğlu, 2021）。 这些努力可以通过使用技术解决方案来补充，其中安全环境可作为专有或敏感数据和模型的存储库（Edwards & Veale, 2018）。

6.3 Contestable AI by Design: Towards a Framework
6.3 通过设计可争议的人工智能：走向一个框架

We have mapped the identified features in relation to the main actors mentioned in the literature (Fig. 2): System developers create built-in safeguards to constrain the behavior of AI systems. Human controllers use interactive controls to correct or override AI system decisions. Decision subjects use interactive controls, explanations, intervention requests, and tools for scrutiny to contest AI system decisions. Third parties also use tools for scrutiny and intervention requests for oversight and contestation on behalf of individuals and groups.
我们已将文献中提到的主要参与者与识别出的特征进行了映射（图 2）：系统开发人员创建内置保障措施来限制人工智能系统的行为。人类控制者使用交互式控制来纠正或覆盖人工智能系统的决策。决策主体使用交互式控制、解释、干预请求和审查工具来质疑人工智能系统的决策。第三方也使用审查工具和干预请求，代表个人和团体进行监督和争议。

Fig. 2 图 2
figure 2

Features contributing to contestable AI
促进可争议人工智能的特征

Full size image 全尺寸图像

We have mapped the identified practices to the AI lifecycle phases of the Information Commissioner’s Office (ICO)’s auditing framework (Binns & Gallo, 2019) (Fig. 3). These practices are primarily performed by system developers. During business and use-case development, ex-ante safeguards are put in place to protect against potential harms. During design and procurement of training and test data, agonistic development approaches enable stakeholder participation, making room for and leveraging conflict towards continuous improvement. During building and testing, quality assurance measures are used to ensure stakeholder interests are centered and progress towards shared goals is tracked. During deployment and monitoring, further quality assurance measures ensure system performance is tracked on an ongoing basis, and the feedback loop with future system development is closed. Finally, throughout, risk mitigation intervenes in the system context to reduce the odds of failure, and third party oversight strengthens the role of external reviewers to enable ongoing outside scrutiny.
我们已将识别出的实践映射到信息专员办公室（ICO）审计框架的人工智能生命周期阶段（Binns & Gallo, 2019）（图 3）。这些实践主要由系统开发人员执行。在业务和用例开发期间，预防性保障措施被采取以防范潜在危害。在设计和采购培训和测试数据期间，对抗性开发方法使利益相关者参与其中，为持续改进创造空间并利用冲突。在构建和测试期间，质量保证措施被用来确保以利益相关者为中心，并跟踪朝着共同目标的进展。在部署和监控期间，进一步的质量保证措施确保系统性能得到持续跟踪，并与未来系统开发的反馈循环闭合。最后，在整个过程中，风险缓解介入系统环境以降低失败的几率，第三方监督加强外部审阅者的作用，以实现持续的外部审查。

Fig. 3 图 3
figure 3

Practices contributing to contestable AI
促进可争议人工智能的实践

Full size image 全尺寸图像

7 Discussion 7 讨论

Using a systematic review and qualitative analysis of literature on the design of contestable AI, we have identified five system features and six development practices contributing to AI system contestability. The features are: 1. built-in safeguards against harmful behavior; 2. interactive control over automated decisions; 3. explanations of system behavior; 4. human review and intervention requests; and 5. tools for scrutiny by subjects or third parties. The practices are: 1. ex-ante safeguards; 2. agonistic approaches to ML development; 3. quality assurance during development; 4. quality assurance after deployment; 5. strategies for risk mitigation; and 6. third-party oversight. We used diagrams to capture how features relate to various actors in typical AI systems, and how practices relate to typical AI system lifecycle stages. These features and practices are a step towards more intermediate-level design knowledge for contestable AI. It represents our attempt to take the general principle of contestability as “open and responsive to dispute” and articulate potential ways in which AI systems, and the practices constituting them, can be changed or amended to support it, with a particular focus on interventions cutting across social and technical dimensions.
通过对可争议人工智能设计文献的系统性审查和定性分析，我们确定了对人工智能系统的五个系统特征和六种开发实践有助于提高系统的可争议性。这些特征包括：1. 防止有害行为的内置保障措施；2. 对自动决策的交互式控制；3. 对系统行为的解释；4. 人类审查和干预请求；5. 提供供受试者或第三方审查的工具。这些实践包括：1. 前置保障措施；2. 对机器学习开发采取对抗性方法；3. 开发过程中的质量保证；4. 部署后的质量保证；5. 风险缓解策略；6. 第三方监督。我们使用图表来捕捉特征与典型人工智能系统中各种参与者的关系，以及实践与典型人工智能系统生命周期阶段的关系。这些特征和实践是朝着更具中级设计知识的方向迈出的一步，以实现可争议人工智能的设计。 这代表了我们试图将可争议性的一般原则定义为“开放且对争议做出回应”，并阐明人工智能系统以及构成它们的实践可以如何改变或修改以支持这一原则的潜在方式，特别关注跨越社会和技术维度的干预措施。

Our framework takes a sociotechnical perspective by focusing many of its recommendations on the entangled and volatile nature of AI systems. For example, interactive control enables negotiation between artificial and human agents; explanations account for the behavior of automated decision-making systems as a whole, not just technical models; intervention requests enable a dialectical process between decision subjects and human controllers in close coupling with artificial agents; and tools for scrutiny require documentation of not just technical systems but also how they are constructed. Furthermore, ex ante safeguards include certification of entire organizations, not just technical systems in isolation; agonistic design approaches lay bare how values are embedded in specific sociotechnical arrangements, creating arenas for stakeholders to co-construct decision-making processes; QA during development addresses system volatility through iterative building and testing, possibly in a living lab setting; QA after deployment focuses on traceable decision chains across human and artificial agents; and risk mitigation educates human controllers and decision subjects on responsible and effective ways of relating to AI system.
我们的框架采取了社会技术视角，将其许多建议集中在人工智能系统错综复杂和不稳定的特性上。例如，交互式控制使人工和人类代理之间能够进行协商；解释说明了自动决策系统整体行为，而不仅仅是技术模型；干预请求使决策主体与人类控制者之间的辩证过程与人工代理密切耦合；审查工具要求不仅记录技术系统，还要记录它们的构建方式。 此外，前置保障措施包括对整个组织进行认证，而不仅仅是孤立的技术系统；对抗性设计方法揭示了价值观如何嵌入特定的社会技术安排中，为利益相关者创造了共同构建决策过程的场所；开发过程中的质量保证通过迭代构建和测试，可能在实验室环境中进行，以应对系统的不稳定性；部署后的质量保证侧重于跟踪人类和人工代理之间的可追溯决策链；风险缓解教育人类控制者和决策主体如何负责任且有效地与人工智能系统互动。

The framework has been developed based on a small sample of academic papers. This approach has obvious limitations. There may be gaps caused by lack of coverage in source papers. The papers included approach the subject of contestability from specific fields (e.g. ethics of technology, computer science, law). Many of these papers are not based on empirically validated interventions. While our framework tries to make the translation to practice, most of the papers on which the content of our framework is based are still “context-free”. We have developed a framework ready to be tested (and validated) in practice, in specific application contexts. The validation itself was not part of this paper.
该框架是基于少量学术论文开发的。这种方法显然存在局限性。由于源论文覆盖不足，可能存在空白。所包含的论文从特定领域（如技术伦理学、计算机科学、法律）探讨了可争议性主题。这些论文中许多并非基于经验验证的干预措施。虽然我们的框架试图将理论转化为实践，但我们框架内容的大部分基础论文仍然是“无上下文”的。我们已经开发了一个准备在具体应用环境中进行测试（和验证）的框架。验证本身不是本文的一部分。

Morley et al. (2019) note that many AI ethics tools lack usability in the sense that they are not actionable and do not come with guidance on how they may be put to use in practice. The usability of our own offering here is still limited: We offer diagrams, which are one step up from lists in terms of conceptual richness. The recommendations are on the level of practices and features rather than general principles, making them more actionable. However, we do not offer directions for the use of the framework to actually design contestable AI. Future work should seek to apply the framework in design activities towards the improvement of use situations, or the creation of artifacts embodying the idea of contestable AI for the purpose of further knowledge development.
Morley 等人（2019 年）指出，许多人工智能伦理工具在可用性方面存在不足，因为它们缺乏可操作性，并且没有指导如何在实践中使用它们。我们自己提供的可用性仍然有限：我们提供图表，从概念丰富度来看比列表高一步。建议是关于实践和特征的水平，而不是一般原则，使其更具可操作性。然而，我们并未提供关于如何使用框架来实际设计可争议的人工智能的指导。未来的工作应该致力于将该框架应用于设计活动，以改善使用情况，或者创造体现可争议人工智能理念的工件，以进一步发展知识。

Many of the themes captured by our framework have also been explored in the literature related to AI accountability. Future efforts may seek to compare our proposed framework to more generic ethical, responsible and accountable AI frameworks (e.g. Cobbe et al., 2021; Hutchinson et al., 2021; Mohseni 2019; Raji et al., 2020).
我们的框架捕捉到的许多主题也在与人工智能问责相关的文献中得到探讨。未来的努力可以尝试将我们提出的框架与更通用的道德、负责任和可问责的人工智能框架进行比较（例如 Cobbe 等，2021 年；Hutchinson 等，2021 年；Mohseni，2019 年；Raji 等，2020 年）。

Our framework assumes no context, or in any case assumes a generic “automated decision-making” setting. It assumes some things are at stake in the decision-making process, typically captured by the phrase “significant impact” on individuals or groups. This covers quite a broad range, but likely does preclude extreme high stakes contexts one finds in e.g. lethal autonomous weapons. Similarly, our framework assumes contexts where time-sensitivity of human intervention is relatively low. That is to say, this framework probably does not cover cases such as shared control of autonomous vehicles. A related research field more focused on these high-stakes and time sensitive scenarios is meaningful human control (for which see e.g. Methnani et al., 2021; de Sio & van den Hoven, 2018; Umbrello, 2021; Braun et al., 2021; Verdiesen et al., 2021; Wyatt & Galliott, 2021; Cavalcante Siebert et al., 2022).
我们的框架假设没有上下文，或者在任何情况下假设一个通用的“自动决策”设置。它假设在决策过程中有一些事情处于危险之中，通常由“对个人或群体的重大影响”这一短语捕捉。这涵盖了相当广泛的范围，但很可能排除了例如致命自主武器中发现的极高风险的情境。同样，我们的框架假设人类干预的时间敏感性相对较低的情境。也就是说，这个框架可能不涵盖共享控制自主车辆等情况。一个相关的研究领域更专注于这些高风险和时间敏感的场景，即有意义的人类控制（例如见 Methnani 等，2021 年；de Sio 和 van den Hoven，2018 年；Umbrello，2021 年；Braun 等，2021 年；Verdiesen 等，2021 年；Wyatt 和 Galliott，2021 年；Cavalcante Siebert 等，2022 年）。

Much of our own empirical work is situated in (local) government public services in OECD countries. Some distinctive features of such settings include distribution of system components across public and private organizations; the duty of care government organizations have towards citizens; and the (at least nominal) democratic control of citizens over public organizations. We expect this framework to hold up quite well in such settings.
我们自己的大部分实证工作都是在经合组织国家的（地方）政府公共服务领域进行的。这些环境的一些独特特征包括系统组件在公共和私人组织之间的分配；政府组织对公民的关怀义务；以及公民对公共组织的（至少名义上的）民主控制。我们预计这一框架在这样的环境中会表现得相当不错。

A pattern running through all identified features and practices is to avoid attempts to at all cost resolve disputes up front before they arise using some form of compromise or consensus-seeking. Instead, we accept that controversy is at times inevitable, and in fact may even be desirable as a means of spurring continuous improvement. We propose to set up procedural, agonistic mechanisms through which disputes can be identified and resolved. Stakeholders do not need to agree on every decision that goes into the design of a system, or indeed every decision a system makes. However, stakeholders do need to agree on procedures by which such disagreements will be resolved. A risk, of course, is that this procedural and adversarial approach is abused to cover for negligence on the part of system designers. This, however, can be addressed by making sure these adversarial procedures include an obligation to account for any decisions leading up to the disagreement under consideration (i.e. ensure decision chains are traceable). This adversarial approach should be an effective way to curb the administrative logic of efficiency, and to instead center democratic values of inclusion, plurality, and justice.
所有已确定的特征和实践中的一个模式是要尽一切努力避免在事前就尝试通过某种妥协或寻求共识的形式解决争端。相反，我们接受争议有时是不可避免的，实际上甚至可能是促进持续改进的手段。我们建议建立程序性、对抗性机制，通过这些机制可以识别和解决争端。利益相关者不需要在设计系统的每个决定上达成一致意见，或者说实际上不需要在系统做出的每个决定上达成一致意见。然而，利益相关者需要就解决这类分歧的程序达成一致意见。当然，一个风险是这种程序性和对抗性方法可能被滥用，以掩盖系统设计者的疏忽。然而，可以通过确保这些对抗性程序包括对导致考虑中的分歧的任何决定负责的义务来解决这个问题（即确保决策链可追溯）。 这种对抗性方法应该是遏制行政效率逻辑的有效方式，而是将民主价值观包容、多元和正义置于中心。

8 Concluding Remarks 8 结论

Subjects of automated decisions have the right to human intervention throughout the AI system lifecycle. Contestable AI by design is an approach that ensures systems respect this right. Most contestable AI knowledge produced thus far lacks adaptability to a design context. Design frameworks are an effective form of knowledge because they are generative and of an intermediate level of abstraction. We analyzed extant literature on contestable AI for system properties enabling contestation. Using visual mapping techniques we synthesized these elements into a design framework. Our framework offers five features and six practices contributing to contestable AI. By thinking in terms of contestability, we close the loop between ex-ante agonistic and participatory forms of anticipation with post-hoc mechanisms for opposition, dissent and debate. In this way, contestability leverages conflict for continuous system improvement.
自动决策的主体在整个人工智能系统生命周期中有权进行人工干预。通过设计具有争议性的人工智能是一种确保系统尊重这一权利的方法。迄今为止产生的大多数具有争议性的人工智能知识缺乏适应设计背景的能力。设计框架是一种有效的知识形式，因为它们具有生成性和中间抽象级别。我们分析了关于启动争议的人工智能的现有文献，以确定促使争议的系统属性。利用视觉映射技术，我们将这些元素综合成一个设计框架。我们的框架提供了五个特征和六种实践，有助于实现具有争议性的人工智能。通过以争议性思考，我们在前期对抗性和参与性预期形式与后期反对、异议和辩论机制之间闭环。通过这种方式，争议性利用冲突促进持续系统改进。