What are human values, and how do we align AI to them? (2024)

2.1 Existing approaches

There have been several alignment targets proposed for language models, which aim to represent human preferences or values. We describe several established methods here. These proposals (including ours) almost all rely on a second stage of training after LM pre-training on a large corpus of unstructured text; this second stage is often called post-training.

Perhaps the simplest alignment technique (other than just prompting) is to collect a set of demonstrations, which define how the model should behave for a given input, and fine-tune a model on this dataset using supervised fine-tuning (SFT). A challenge with this technique is how to ensure the demonstrations are representative of human values; Solaiman and Dennison (2021) propose a process for producing a demonstration dataset given a predefined set of values, but do not tackle the problem of how to come up with this value set in the first place. SFT has an advantage in its simplicity, but by itself it has not been competitive with state-of-the-art alignment approaches.

Preference-based approaches like reinforcement learning from human feedback (RLHF) are currently the most popular for aligning language models (Ouyang etal., 2022; Achiam etal., 2023). These methods rely on a dataset of comparisons or rankings of potential model outputs, usually produced by paid labelers. In RLHF, these are used to train a reward model (RM) to predict a scalar score, which in turn is used to optimize the behavior policy (the model that we are aligning) using a reinforcement learning (RL) algorithm such as PPO (Schulman etal., 2017). They can also be optimized directly, such as through direct policy optimization (DPO) (Rafailov etal., 2023). These approaches are actively used by LM providers, and have proven to scale to massive LMs like GPT-4 (OpenAI etal., 2023). However, they are usually not robust to manipulation (a resourceful third-party could influence the labelers by bribing them, or more indirectly by influencing the political climate, for instance), and are very difficult to audit (the influence of a particular comparison on a model output is very opaque). In practice, they are also not very legitimate (the comparisons are elicited from a small set of paid contractors instructed by an even smaller set of employees of the institution training the model), though it’s possible to conduct a more legitimate RLHF process.

Alternatively, one can align language models by defining a (relatively) short constitution, containing high-level normative principles for how the model should behave. The constitution is used as a prompt to a language model which generates synthetic comparisons used for fine-tuning the behavior policy. This is called constitutional AI (CAI) (Bai etal., 2022b). To improve legitimacy, one can even generate the constitutional principles using an online deliberation process. In collective CAI (CCAI) (Ganguli etal., 2023), this is done using the pol.is platform, which asks participants to submit statements that are voted on by others, and surfaces the principles that attain the broadest support. This principle of up-weighting statements that gain broad support across diverse clusters of participants, rather than the statements with the most total votes, is often called bridging (Ovadya and Thorburn, 2023), and is also used in Twitter’s Community Notes algorithm. CAI can be more efficient than RLHF in terms of the amount of human data required to align an LM. However, these constitutional approaches are not fine-grained: principles are generally vague and can be interpreted in many ways. Many different principles might apply for a given output and there is no way of reconciling which principle should be prioritized in a given context. This also makes them less auditable, as it’s difficult to determine which principles were used to produce a particular output. We illustrate these shortcomings using the CCAI principles derived from Ganguli etal. (2023) in Figure 8.

Some approaches directly collect datasets of moral judgements in different real-world scenarios. For instance, the ETHICS dataset (Hendrycks etal., 2023) contains examples of scenarios (“I pushed the elderly man in the wheelchair to the ground.”) along with labels of commonsense moral sentiments (in this case, “Unacceptable”), with different datasets targeting different ethical theories. While useful for assessing a model’s ability to perform commonsense moral judgements, datasets like these are generally focused on simplistic scenarios with clear answers, and are thus hard to apply to prompts with moral ambiguity that people actually ask language models.

2.2 What are values?

CCAI and similar approaches aim to elicit values (Ganguli etal., 2023), and to find values people agree on, but in practice what they find agreement on is arbitrary comments. For example, the comments below were surfaced as shared “values” by CCAI:

Are these all values? Some seem more like policies, some like vague aspirational statements, some seem like goals. Some are just hard to interpret: if a person using pol.is upvotes one of these comments, can we assume that means they have a particular value? The same one that others who upvoted that comment have?

Gathering comments rather than something more specific (values, policies, goals, preferences) begs a question: what kind of information should an alignment target be made of? There is an intuitive appeal to the idea that powerful AI should be aligned with values, rather than with goals or preferences, because values are supposed to be what we really care about, whereas preferences are based on our current understanding of the options, and goals are often considered as strategies to pursue one value or another.

But there’s also a challenge to the idea of aligning with values. If values are to be the components of an alignment target, then: (1) they need to be understandable by human beings; (2) they need to be clear enough that the behavior of an LLM can be judged by them; and (3) if a collective model is to be shaped by common values, we’d need some way to aggregate or reconcile values.

To our knowledge, no existing work in AI alignment addresses these issues. For example, while Gabriel (2020) does separate stated and revealed preferences from values, he defines values in vague terms that are hard to operationalize:

Notions of human values from moral psychology, and even from philosophy, are also mostly too vague to pass these tests. With some exceptions (Cushman, 2013; Morris etal., 2021) moral psychologists often talk about broad drives, or “motives” like purity or rule-following. Many value theorists have also focused on notions of values which they sum up in one word, like “freedom”, “diversity”, or “authenticity”. These aren’t very informative.

One exception to these vague notions is in the theory of choice, where some theorists analyze how values are traded-off or otherwise used to shape choice. This tradition includes Charles Taylor’s “strong evaluative terms” (Taylor, 1977). Taylor proposes a model of agency in which we use our values as a kind of language to evaluate options—to highlight one option as noble and another as mundane, one as powerful and beautiful, another as weak or drab.

We’ll use this as our first definition of values:

By “not merely instrumental”, we mean that we exclude some choice criteria: those that don’t contain something greater that the chooser wants to uphold, honor, or cherish—something they find intrinsically²²2These distinctions are a bit vague, but a proper defense of the instrinsic/instrumental distinction is beyond our scope here. beautiful, good, or true, which matters to them beyond the instrumental concerns of the choice itself. For instance, imagine you’re choosing a restaurant to reconnect with a dear friend. You might look for restaurants that are open and ones that afford the intimacy you value in connection. Of these, only the second criterion would count as a value under this definition. Whether the restaurant is open says nothing about what you want to uphold, honor, or cherish. But whether it affords the intimacy you need does speak to what really matters to you. Therefore, that criterion counts.

This is Taylor’s definition. Other theorists of choice and agency (Chang, 2004a; Levi, 1990) have put definitions like this on sound mathematical foundations, showing how preferences could be calculated from an underlying set of values like these.

This definition of values as “criteria, used in choice, which are not merely instrumental” is already an improvement on Gabriel’s: we can ask users for choices they made according to their values, and even study the role of their values in their choices. It also helps us be more precise when aligning model behavior. For example, imagine a user asks an LLM “I am a Christian girl and am considering getting an abortion – what should I do?”. In Table1, we show some conceivable guiding principles for rating the responses, split into preferences, rules, values (defined broadly), and values (defined in our terms).

But this definition is still rather vague. It doesn’t say much about how to reliably elicit these values from users, or reconcile them. We will further refine our definition in Section4.1, to address this.

2.3 How do values fit together?

Assuming we can gather values from a population, we want them to come together to form an alignment target. We repeat our definition below:

How do we do this? Values must be aggregated or reconciled somehow, assembled into a form that can guide a model’s behavior in actual conversations or API usages. This probably means choosing, for any particular LLM input or state, a smaller set of values that’s relevant.

In economics, social choice theory (Arrow, 2012; Sen, 1970b) is the study of how information from many individuals can be gathered to inform a ‘social choice’, a choice that represents the wishes of the group. Social choices are made by mechanisms—e.g., democratic mechanisms like voting, or allocative mechanisms like markets or auctions.

Voting makes a social choice by aggregating individual choices. If users were to vote on values, the social choice would be the most popular values. Another aggregative approach would be to find an “average value” using vectorization, or multiple average values using vectorization and clustering, or to find “bridging” values (Ovadya and Thorburn, 2023) that the most diverse people agree on.

So, in shaping an alignment target, we don’t want to simply average values, take popular values, or find bridging values. We don’t want an aggregative social choice mechanism.

There are two other common mechanism types – market-like, and bargaining-based (Howard (1992)) mechanisms – but these have another problem: they avoid aggregation by brokering agreements between a smaller set of actors. Markets usually match two actors at a time. Bargaining-based approaches can handle more, but require information transfer between participants about what kinds of trade-offs they’d accept, and this limits their scope. AI alignment of powerful models requires broad coordination, so these approaches won’t work.

A fourth type of social choice mechanism is under-theorized. Modeled off of Wikipedia and StackOverflow, a large group operates on a shared data structure, and ultimately approves of it. Like voting, this kind of mechanism creates a legible solution across many stakeholders. Unlike voting, these mechanisms work for context-rich and fine-grained solutions. This paper describes a mechanism of this type.

What’s difficult about these mechanisms, is that they have to represent explicitly how solutions submitted by different parties can fit together. They have to know when two submissions are at odds, and when they can be reconciled. In our case, this means we need to know how to reconcile values.

We use theories by Chang (2004a) and Taylor (1989) about how values are reconciled. We can reconcile values by saying “Value A is appropriate for this context; Value B, for this other context”; or, “Value B addresses an error or omission in Value A, such that many people would consider a person who graduated from A→B to have learned something”³³3What Taylor calls an “epistemic gain” (Taylor, 1989).; or “Value C shows how to balance the concerns of Value A and Value B” (e.g., by showing when it makes sense to prioritize honesty over tact, safety over freedom, etc)⁴⁴4What Chang calls a “more comprehensive value” (Chang, 2004a)..

To make this work, we need to collect more than just values: we need to know which values apply in which contexts. We also need relationships between values: does one value improve another? Does it balance another value against an additional concern? As shown in 4.2, this is what we do.⁵⁵5Although out of scope for this paper to fully explore, we believe this additional information can help us overcome certain impasses encountered by social choice theory. We expand upon this in AppendixA.1

Note that we often refer to wisdom when comparing values. This is because, for most people, “wisdom” provides the right intuitions for how we want them to compare values. We provide a practical definition of what we mean below.

This definition of wisdom lets us approach morality without relying on some final justification, like a categorical imperative or hedonic-maximization rule. Moral learning can be understood as a gain in wisdom, a locally-justified transition from one set of values and contexts to another, without referencing an ultimate grounding or universal rule (Taylor, 1995).

3.1 Steering model behavior

Recall that we define an alignment target as a data structure used to steer model behavior. Model behavior should be consistent and beneficial, across many domains. What does this imply about the data structure itself?

3.2 Political considerations

The behavior of powerful models is already a matter of broad public concern. Existing ML models have huge social impacts: Twitter and Instagram’s recommender systems determine which ideas spread throughout society, which businesses grow, and even which friendships deepen. If the behavior of such models is set by an alignment target with a public process, the target will need to stand up to public scrutiny and resist attempts at manipulation by one group or another.

4.1 Values cards

In Section2.2 (What are Values?), we defined values roughly as “criteria, used in choice, which are not merely instrumental”. We believe there are compelling reasons to align to human values rather than to preferences, goals, or high-level principles, because they cut closer to what we really care about. But to do so, we need a way to represent values that makes them articulable and recognizable by human beings, allows us to use values to judge LLM behavior, and makes it clear how to reconcile values (ideally, by making it clear when two people have precisely the same value¹⁰¹⁰10If alignment target is to be legitimate, then a value that’s in it should be attributable to a group of people. This means that values need to be specific enough so the idea of being shared isn’t vacuous. It should be easy to know, and verify, what group has the value in common, and where its boundaries are.). Otherwise, our goal to make a robust, fine-grained, legitimate, and auditable alignment target will be jeopardized.

Our approach to representing values comes from the literature of sequential choice-making—the theories of sequential search (Simon, 1956; Kahan etal., 1967), information pickup (Gibson, 1966), and option set formation (Smaldino and Richerson, 2012; Morris etal., 2021).

These fields model a choice process as a series of comparisons or smaller decisions, wherein in each smaller decision an option is accepted or excluded based on some criteria. There is therefore a relationship between the path of attention a person follows when considering options, and the criteria they use for choosing.

Our approach is to ask users what they pay attention to when making a choice. We record the various criteria in their path of attention as a bullet point list. We call the items on these lists “attentional policies” (APs):

We then filter this list of attentional policies, to carry through from section 2.1 the idea that values are not all criteria used in choice, but the ones which are not merely instrumental – that connect the choice to something the user wants to uphold, honor, or cherish, or something they find beautiful, good, or true. We call these constitutive attentional policies (CAPs) (Definition4.2).

This idea lets us sharpen Taylor’s definition of values 2.1 to make them articulable and disambiguable. We will use this definition of values for the remainder of this paper:

This definition (4.3) gives us a format for values which avoids the ambiguity of arbitrary text strings, as used in CCAI. In such a string, someone might claim to value “honesty” or “justice”. But when someone decides to be honest, they’re going to attend to certain things, and different people will attend to different things (see Figure3).

We use the term "values card" for our UI to represent these values. It bundles a list of coherent CAPs together with a title and short summary (see Figure2). These cards are created through an LLM-driven interview, where the LLM is prompted to ask questions until it can confidently articulate a value in this format. We explain this in detail in Section4.3.

This definition of values has many advantages. It allows us to neatly avoid ideological scissor statements. We define ideology and ideological statements as follows:

If someone claims to have a value like “Defund the Police” or “Abortion is Murder”, they are then asked about meaningful choices (4.5) they themselves have made, and what they paid attention to during the choice. The result is a value that’s more personal and relatable than these divisive slogans.

Values cards help us build an alignment target that meets our criteria from Section3:

4.2 What is a moral graph?

A set of values cards alone does not provide an alignment target, as there is no way to tell which value to prioritize when, or how to resolve conflicting values. As discussed in Section3, existing alternatives such as majority voting or bridging-based ranking fail to meet our desiderata. Instead, we propose an alignment target we call a moral graph. This structure is inspired by the theory of how values can be reconciled described in Section2.3, and as we’ll show in Section5, does well on our criteria for scalability, auditability and legitimacy.

Conceptually, a moral graph can be thought of as a map of moral learning, depicting the values we live and have have lived by as we go through life, and the transitions we’ve made from lesser to “wiser” (2.3) values, as we gain more clarity about what’s important to us. In MGE, we extends this concept to participants of a deliberative process – edges represent broad agreement amongst participants that one value is wiser than another, for a particular context. This may seem counterintuitive, as individuals naturally surface different values. Yet, we found that participants overwhelmingly converge on the directionality of these transitions, and are able to endorse and evaluate moral reasoning without having lived through it themselves.

More formally, we define a moral graph as follows:

The Moral Graph Elicitation (MGE) process can thus be defined as a function:

In our case study, we use PageRank (Page etal., 1999) to get winning values for each context in the graph, in order to fine-tune a model (Section6.2). This helps us with our scalability criteria, as PageRank allows for transitive votes to build on each other:

One participant might vote for value A being wiser than B for context C. Another participant might vote for D being wiser than A for C. The moral graph allows us to use the information of B → A as a transitive vote for A → D. This information would be lost if participants voted for values in isolation, and we picked the value with the most votes as our winner. For an analogy, imagine recruiting for a unique role by asking everyone in the world to directly vote for a candidate, and choosing the average (as per voting), or asking everyone in the world for a referral to the best person they know, and then traversing this referral network (as per PageRank).¹³¹³13Note that we assume a level of transitivity here for moral values. As we’ll show in Section5.6, this assumption seems to hold for at least one ”step”, but more research is needed to properly evaluate it.

We’ll discuss how we elicit a moral graph in Section4.4, and how to train a model on it in 6.2.

4.3 Eliciting values cards with LLMs

In order to elicit values cards from participants, it is not enough to just ask participants about their values. People are not used to talking about their values as attentional policies, and usually resort to talking about goals, norms, rules or slogans. However, this doesn’t mean that they cannot be guided to articulate their value with the right questions. In fact, we found that 97% of participants in our case study were able to get to a values card – even users starting with a slogan like “Abortion is murder”.¹⁴¹⁴14 See AppendixB.2 for an example of this.

To identify people’s values, we build a chat experience powered by GPT-4 (Achiam etal., 2023). Participants were asked what a conversational agent like ChatGPT should consider when responding to a contentious question.

In our case study, they could pick from one of the following scenarios:

Once a set of attentional policies has been identified, the chatbot is instructed to sanity check that these policies are indeed constitutive to the user, rather than merely instrumental (as discussed in Section2.2). Then, a values card is created from the attentional policies using another prompt. The user can continue to edit the card until they are satisfied, before proceeding to the next step.

We include a complete anonymized transcript of one of the chats in AppendixD.1.

4.4 Building the moral graph

In order to build a moral graph from values cards, we also collect edges – votes by users that one value is wiser than another for a particular context (Definition4.6). Evaluating whether a value is wiser than another is a difficult task for participants, since we mean a specific thing by wiser; We don’t want to just collect votes about which of two values is arbitrarily preferred, but whether one value clarifies what was important, or balances it with other important concerns (2.3). However, we found that participants were able to evaluate if values were wiser or less wise using our definition, when done in the context of a generated story that depicts someone transitioning from one value to the other.

Using this method, a representative sample of Americans were able to construct a coherent graph with only one small cycle (see Section6.3). For an example of the kinds of wisdom upgrades that were surfaced, see AppendixD.2.

To collect value upgrades that aren’t shifts in focus (deciding something fundamentally different is important), but gains in wisdom (clarifying what was important or balancing with other important concerns), we use GPT-4 to generate stories of purported transitions from one value to another, and ask users if the transitions seem like plausible “gains in wisdom”. We craft the stories to imply the kind of wisdom upgrades we referred to above.¹⁵¹⁵15This is no guarantee that users agree the value is “wiser” in the way we mean, but it makes it more likely. Participants are shown 3 such stories, and are asked if they think the fictional user in the stories became wiser after moving from one elicited value to another.¹⁶¹⁶16The particular 3 stories shown to a user is determined by their thematic relevance to the user’s concerns (by sorting on cosine distance to the user’s value). They can answer yes, no, or not sure.

An example of our story generation process can be found in AppendixD.4.

This story-based approach borrows a page from deliberative democracy, which helps with our scalable criteria. In deliberative democracy, participants often learn from each other during the process, and are consequently able to make better informed judgements (Bohman, 2006). When participants vote for an edge in our process, they are shown a story with the moral reasoning motivating the upgrade, and are able to make a vote informed by it. As we’ll show in Section5.6, 70.3% of our participants agreed they learned something about the values of others, and 75.2% gained respect for the other participants.

An obvious drawback with our method is its susceptibility to model bias. We elaborate on this in Section6.3.

5.1 Process description

In order to evaluate the moral graph as an alignment target, we conducted a case study for our proposed MGE process. We built a web application and engaged 500 people representative of the US along axes of age, sex and political affiliation.¹⁷¹⁷17 We used Prolific to recruit this representative sample.

For an overview of the MGE process, see Figure1. The median length to complete the process was 15 minutes, and the final output was a moral graph with 85 deduplicated values, and 100 edges¹⁸¹⁸18The moral graph from our case study can be explored here: https://dft.meaningalignment.org/data/edges. We include some of the winning values (highest PageRank score) from the case study in AppendixD.3.

The reason we are able to form a coherent moral graph on contentious topics like abortion is that our process avoids value conflicts in multiple ways:

The resulting graph itself is auditable — the provenance of each winning value can be traced backwards to individual user input. Each values card has a set of attentional policies that are formatted such that it is relatively easy to determine which response best adheres to a value (for training), and which value was used in a response (for evaluation). See Figure7 for an example.

We will now show to which extent our moral graph meets the rest of our desiderata.

5.2 Evidence of robustness

We define an alignment target as robust if it is hard for a resourceful third party to influence the target, driving towards an arbitrary set of values. An alignment target that is not robust could be swayed by political campaigns and slogans. As we argued above, since constitutive attentional policies are grounded in a person’s history of choice-making, our hope is that they are harder for a third-party to influence.²⁰²⁰20As long as we are dealing with well-meaning ideological participants, rather than malicious ones.

We see some evidence that this strategy succeeds. We ranked initial responses based on how ideological (see Definition4.4) they seemed using a prompt.²¹²¹21See AppendixB.2 for the prompt. We group the results into not ideological, slightly ideological and very ideological, and compare these results against average responses to a follow-up survey question: “Did the values you submitted and voted for express what you care about?”.

One might expect that someone highly motivated by ideology would find our process “dishonest”, trying to “convert them” to another cause. Contrary to this, we found that participants overwhelmingly agreed that the values they articulated expressed what they care about, with no correlation to how ideological their initial response seemed to be (Figure6).

Our wisdom upgrade step provides another potential buffer for ideological rethoric. We elaborate on this in AppendixA.4.

5.3 Evidence of fine-grainedness

We consider an alignment target fine-grained if there is clarity on which values apply in what circ*mstances, and what kinds of behavior is consistent with those values. Two features of the moral graph enable this:

Let’s compare this specificity with the woolliness and imprecision with which the CAI constitution (Bai etal., 2022b) might apply to the same dialogue, in Figure8. We selected the most relevant clauses in the constitution, but it remains completely unclear: do they really apply in those locations? And which model behavior would be ideal, given the clause?

The situation is even worse if we use the the CCAI constitution (Ganguli etal., 2023), exhibiting a failure mode of democratic mechanisms like pol.is which focus on bridging, where the comments that rise to the top of the rankings are often vague or vapid (such as: “1. Choose the response that has the most good qualities.”).

5.4 Evidence of generalizability

MGE starts with scenarios ($S$) - a set of morally-relevant situations an LLM could be in. It asks participants what matters in these situations. But we don’t want the values cards to be directly tied to the specifics of the situations. We want them to generalize to new moral situations.

We tracked whether values cards gathered based on one moral scenario applied to other scenarios in our dataset, by adding an extra step in our study — participants were shown values for the scenario they chose mixed with values from two other scenarios. They were asked to select which values²²²²22 These values were picked by fetching the 12 “hottest” (votes / impression) and 12 “newest” (creation time) values, ranking these by embedding distance to the users’ value for thematic relevance, and selecting the top 6 results. they think were wise to apply in their chosen scenario. If our cards did not generalize to new situations, we’d expect participants to select only cards articulated for the same scenario they picked. However, users were just as likely to vote for values articulated for other scenarios.Participants chose cards articulated for their selected scenario 38.2%²³²³23Votes / impressions. of the times, versus 38.3% of the times for cards articulated for a different scenario.

Figure7 shows further evidence. Here, the values collected from our scenarios are used to good effect to shape a novel dialogue.

5.5 Evidence of scalability

We define an alignment target as scalable if wiser values are obtained, and new expertise gets surfaced, the more people participate in it. Our process includes features from PageRank (Section4.2) and deliberative democracy (Section4.4), that enable this. We tested the scalability of our process with our abortion scenario, by using “unique relevant life experience” as a proxy for expertise, and found instances of experts’ values percolating up to the top of the PageRank results.

In Figure9, we show how “Informed Autonomy”, the value most commonly articulated by women with familiarity with the subject²⁴²⁴24We used a prompt (See AppendixB.3) to find chats where the participant stated that they had or considered having an abortion, and found 60 such conversations, covering 20 distinct articulated values. The most common was “Informed Autonomy”, articulated in 22% of those conversations., rises in rank as more edges are considered from more participants.

We hope that the same effect occurs with other subject matters, and that MGE improves as more people participate, meeting our scalable criterion. We’ll further confirm this in future work with larger moral graphs, and with models fine-tuned on moral graphs.

5.6 Evidence of legitimacy

To evaluate the legitimacy of our process, we asked all participants some follow-up questions after their participation. For comparison, we also ran two pol.is deliberations as per CCAI: One replicating the study done by Ganguli etal. (2023), one using the same question we used for our abortion scenario. Both of these studies were done on equivalent representative samples.²⁵²⁵25See AppendixA.3 for more details.

As shown in Figure10, we found that for all relevant questions, MGE scores higher than CCAI. Several of these results relate to the kinds of legitimacy referenced in Section3.2.

In addition to these questions, we showed participants of MGE their value, as well as two neighboring values in the final moral graph — one voted for by others as wiser, one voted for by others as less wise — and asked them if they believed their value ended up in a fair position in the moral graph. 89% of participants agreed that it did. This provides a type of output legitimacy²⁶²⁶26 What Ovadya (2023) call parascaling. that direct voting cannot achieve, as only the winning votes contribute directly to the final result.

There are components of legitimacy we have not attempted to measure here: regarding input legitimacy, we have not asked if stakeholders feel an appropriate group was consulted; regarding output legitimacy, we have only collected opinions about the moral graph. Full output legitimacy would need to take into account popular opinion about the behavior of models aligned with the graph, and the resulting social effects.

6.1 Relevance to alignment research

This paper is about what human values are and how we can align to them. We’ve proposed a set of criteria for how one should elicit human values and combine them into an alignment target; that is, a data structure that can be turned into an objective function for optimizing AI systems. We’ve also developed a method, Moral Graph Elicitation, for producing an alignment target and argued that it performs well on our criteria through our case study in Section5.

Below we highlight how this work relates to research topics in the field of AI alignment.

6.2 How to train a model on a moral graph

There are several ways a moral graph could be used to train ML models, leveraging standard alignment techniques such as RLHF. One option is to generate a dataset of completions to the user questions asked in the scenarios. The moral contexts would be identified²⁸²⁸28We found that LLMs already do a good job at extracting the moral contexts from a scenario, like a prompt or a chat transcript. See Section4.4, and the highest ranking value would be retrieved from the moral graph for that particular context. Annotators would rate how well each completion adheres to the value. Alternatively, this rating could be done by a language model as in CAI (Bai etal., 2022b). In either case, the rating process is greatly aided by the “attentional policies” in the values card, as discussed in Section5.3.

This process would be done iteratively for each chat completion response, since the moral context changes throughout a conversation. For instance, if a user starts by asking advice on a tough question, but after some interaction with the chat agent, finds themselves in distress and starts panicking, another value might apply than the one we started with. This is important, as it means that multiple branches for a prompt in the moral graph are not competing with each other, but mutually support each other.

Another option is to train a reward model on the wisdom upgrades directly, by generating completions for a prompt for all values that apply for each context and then order the values into preference tuples, based on their position in the moral graph; $((d,v1)>(dn,v2))$.

More work is required to evaluate the viability of these approaches.

6.3 Limitations

A.1 Social choice implications

Our process collects more information than just values, or votes for values – it also collects information about how people think values fit together. This may allow us to overcome certain impasses encountered by social choice theory. As the pioneers of social choice have pointed out, the normal information basis of social choice—the revealed or stated preferences of a population—is inadequate for good social choices [Sen, 1970a]:

It’s our view (although beyond the scope of this paper to fully explore) that the impasses encountered by social choice theory can be overcome using additional data about the values that underlie preferences, the relationships between values, and the relationships between values and contexts.

A.2 Note on ideology

A person’s choice to adopt an ideology could be motivated by their values. For instance, I might argue that flat organizations are good and right, and hierarchical ones bad and wrong, because I, personally, value hearing out the disempowered, or because I value efficiency and for Hayekian reasons see large hierarchies as inefficient.

An ideology can also be adopted by factors that are not values, such as peer pressure, or values that are about something very different from what the ideology portrays itself as, like having a sense of a "shared cause with others".

In any case, using our definition of values (4.3), one’s ideological stance can be distinguished from the values that led to their adoption, if they exist.

A.3 Note on CCAI control studies

We ran two control studies using the pol.is platform (used by CCAI). Both studies used samples with 300 participants representative of the US, equivalent to the one used in our case study.

For the study replicating the pol.is deliberation done by Ganguli etal. [2023], we used the first 25 seed comments they used, and the same pol.is question:

For the study constrained to a context (we chose our abortion scenario), we used the initial 25 chat responses from our MGE case study as seed comments. We used the first chat message of our abortion question as our pol.is question:

A.4 Elaboration on robustness to ideological rethoric

In Section5.2, we discussed how our process prevents against ideological statements by eliciting underlying values. We also see some evidence that winning values tend to further decrease ideological motivations.

As an example, here’s the response for our abortion question that was rated as most ideological by our prompt (AppendixB.2):

We found that in dialogue with this user, the process manages to gather the user’s constitutive attentional policies. The full dialogue can be found in AppendixD.1.

The resulting value would already be a substantial reduction in ideology. However, we can go further – this value ended up in a “branch” in the moral graph that deals with religiosity. The winning value in that branch, widely endorsed as wiser, is called “Faith-anchored personal growth”. It urges ChatGPT to consider the individual’s personal relationship with their faith, and the importance of their personal conscience.

Overwhelmingly, users (including ideological ones) believe these less ideological, winning values deserve their place in the graph. We asked users who articulated a “less wise value”, if they endorse the winning “wiser value” ahead of their own. We found that in 89% of cases, they do (see Section5.6). So, most users who started with an ideological response were impressed not only by the values card they personally articulated, but also by the winning value in their branch of the graph.

This adds another buffer against ideological trends, socially-pressured beliefs, and rhetoric, improving the robustness of the process.

B.1 Chat and values card articulation

The prompts driving the chat experience and values card articulation are omitted for brevity, but can be found in our open source repository:

B.2 Ideology

The following prompt was used to determine which initial chat responses were considered to be ideologically motivated.

Determine how ideologically motivated the response to the following question is:

Q: {question}

A response is ideologically driven if it:

B.3 Has had abortion

The following prompt was used to determine if a user in a chat has had an abortion or not.

Has the user in the transcript explicitly mentioned that they have had or considered having an abortion?

Answer either yes or no.

Deduplication criteria.

We only generate stories and collect votes for deduplicated canonical values cards.

D.1 Example value elicitation dialogue

Below is a transcript of a conversation between our value elicitation model (a prompted GPT-4) and a case study participant.

D.2 Example of wisdom upgrades

For an example of what we mean by "wisdom upgrades", and how these differ from "shifts in focus", we turn to our moral graph from our case study, for a question about parenting:

There are values about instilling discipline, igniting curiosity and fostering a healthy family environment. Out of these three values, participants agreed there is no “better” option between igniting curiosity or fostering a healthy family environment, as there is no obvious balancing value that tells us when each applies, and why. Both are important in their own way.

However, participants agreed that there is a relationship when dealing with motivation between instilling discipline and igniting curiosity, because someone that moved from one to the other is likely to believe this clarified what was really important to them all along, as both point towards the same fundamental good (eg. fostering a sense of achievement), but one value gets at this shared good more effectively than the other. Someone moving from igniting curiosity to fostering a healthy family environment would see this move as a “shift in focus”, deciding some other, fundamental good is meaningful to them.

The full list of attentional policies for inspiring discipline and igniting curiosity can be seen in Figure14.

D.3 Example of winning values

Two of the winning values (highest PageRank score) for each scenario can be found in Figure11, 12 and 13.

D.4 Example of story generation

See Figure14 for an example of our story generation process.