Sucker fear and social norms

 People follow social norms because they believe that others follow these norms and expect the same of them. Some (most?) social norms involve conflicts between preferences of different people. The resolution of such conflicts is also the central question of morality as we know it. Social norms specify how such conflicts should be resolved. Moral norms go beyond this specification because they apply everywhere and are not contingent on what people expect or do, but the address the same questions.

What does it mean to be selfish?  It is not just acting in your self-interest; most self-interested behavior is expected in prevailing norms, as well as moral norms. Social norms differ. A minor example is tipping in restaurants. (Rarely done at all in some countries, expected in the U.S., at 15 or 20%.) Selfishness is when you take more for yourself, or try to, e.g., by leaving 10% when the service was perfectly fine. Or by taking the last helping without checking to see whether everyone else at the table has had their share.

Social and moral norms set minimum standards for what you are supposed to do when your interests conflict with those of others. It is often in our interest to break the law, for example, by leaving trash, or worse, on the sidewalk. Many laws are not enforced very well, or at all. But both moral and social norms say you should follow relevant laws, and most people do that. Social norms do not oblige you to clean up other people's trash, although morally it would be a good thing.

When you err from the norm on the side of others, that is altruistic but it also puts you at risk of being a sucker, being fooled, and that can be embarrassing because you are not looking out for yourself in the way you should (for yourself). You lose status. You are embarrassed. You are a sucker if you hand money to a homeless person who will then just spend it on booze or drugs. Or if you pick up other people's trash even though the city pays people to do that periodically. The behavior of others tells you what the social norm is. Trust of any sort is dangerous because you could be fooled, taken advantage of. So people take extra steps to prevent that outcome, such as signing contracts and asking for receipts.

Tess Wilkinson-Ryan (my former PhD student) has written an account of the (to me) surprising number of manifestations of fear of being a sucker ("Fool proof: How fear of playing the sucker shapes ourselves and the social order -- and what we can do about it" Harper-Collins). These manifestations occur in both individuals and governments. They include (as just noted) reluctance to trust individuals or institutions, and reluctance to make many types of voluntary contributions. Trust issues arise in ordinary commercial translations, sexual and romantic relationships, employer/employee relations, job searches (on both sides), as well as international agreements. Sucker fear often prevents beneficial relations, although sometimes it is warranted. It arises whenever fair division is not obvious, as various definitions of fairness could advantage one party or another, and the disadvantaged party could feel like a fool for agreeing to a plan. Sucker fear also exacerbates other effects, such as prejudice against some group: where a stereotype can lead to mistrust group members and unwillingness to cooperate with them. Such groups include poor people (who rip off the rich by asking for help), women, immigrants, Blacks, and Jews. Paranoid delusions of persecution, an extreme form of mistrust, are often manifestations of sucker fear.

The book also describes the efforts that people make to recover from the regret, embarrassment, or anger that they feel when they have actually been fooled. Experiments on "cognitive dissonance", in which people try to convince themselves that something they were tricked into doing was not bad after all, even to the point of telling the next research subject that an extremely boring task was actually interesting. As for anger, physical attacks on malfunctioning vending machines are apparently a source of injury for the attackers as well as the machines.

Perhaps the best single example of most of the manifestations of sucker fear is Donald Trump, who comes up at several places in the book. But the book was written before he was re-elected. His second term provides one clear case after another of extreme actions based on sucker fear or the feeling that he was, or is at risk of, being a fool.  These include trade, election procedures, contributions to NATO, and foreign aid. His apparent beliefs about elections lost by him or his supporters seem paranoid, combining delusions of grandeur with delusions of persecution. When he actually did play the fool, apparently being convinced by Benjamin Netanyahu that invading Iran would bring about regime change in short order and have no unacceptable side effects, he denies that anyone made a mistake and insists that the results are just fine.

The book, I should note, is published as a trade book, and it is highly readable, full of interesting anecdotes. Beyond this, its analysis of people's thoughts and emotions is truly psychological, emphasizing how people think and what emotions they experience. And its thoroughness in discussing relevant literature (with citations in end notes) makes it useful in the way that good textbooks are useful.

Social norms put a lower bound on what we are expected to give up when our preferences might conflict with others' preferences. Sucker fear often puts an upper bound on these expectations. In the end, many social norms specify that we do something but no more than that. We are left with a narrow band. Of course, these norms themselves are variable, and people differ in what we would like the norms to be. For many people, the norm is that voting is expected but voting on the basis of valid information is not. The attempt to do that could leave us at the mercy of charlatans who fake validity, so it is OK to vote on the basis of intuition. Likewise, contributions of time or money to a political campaign could end up wasted, making suckers of contributors. The same goes for many kinds of self-sacrifice for uncertain benefits to others.

Trustworthy AI

 Beginning with Rationality and Intelligence (1985), I have proposed a general framework for human goal-directed thinking, not all that different from other such frameworks, although simpler. The elements were possibilities, evidence, and goals. Thinking begins with a question, and possibilities were possible answers. Units of evidence (arguments) were brought to bear on the possibilities. Goals were criteria for how each unit of evidence was used. If the question is how to travel from one city to another, the possibilities could be car, train, or airplane. The goals could be minimizing time, minimizing cost, safety, reliability, etc.  Evidence would consist of driving estimates from Google Maps, train and airplane schedules, ticket and gasoline prices, and so on.  A strong goal of saving money could increase the relevance of evidence about cost, possibly favoring car. A goal of minimizing time could increase the strength of flying, depending on whether it really was faster when travel to and from airports was considered.

Goals are often provided by the problem itself, but the general framework could also apply to life decisions, those that concern the choices of people in their individual lives, or in groups. I argued that these sorts of goals were partly innate (hunger, etc.) but also come from culture, and can be created as if they were themselves the answers to questions like "What do I care about?" or "What should I care about?" We can call such goals "values". What gives individuals their "personhood" is their capacity to form personal identities, concepts of what they stand for (and against). For most people, these include some sort of moral goals, like being a "good person" or aspiring to be a "mensch". These goals involve paying attention to the needs and aspirations of other people.

I also argued that good thinking is properly part of intelligence. It is a matter of how we carry out our thinking. If our thinking is (what I called) actively open-minded, we engage in sufficient search for all three elements, and in doing this we are not biased toward possibilities that are already strong. We also maintain a level of confidence in a tentative conclusion that is warranted by the thinking done so far.

Artificial intelligence (AI) systems aspire to be intelligent, and in some ways they have already exceeded people in various manifestations of intelligence. They are capable of actively open-minded thinking except perhaps for the part that involves search for goals, as their search may be limited to what is relevant to the problem they are asked to solve, excluding "side effects" of their solutions.  Worried have arisen about their capacity to do harm, which has already reached the point where the best versions are being withheld from the general public, lest someone used them for nefarious purposes.

Thus, compared to humans, AI systems have one huge gap, also found in some humans (psychopaths in particular). They don't care about morality in any sense. They don't have personal identities that include moral commitments. They are like "good" soldiers, completely obedient to whatever orders they are given. But surely very few real soldiers are so obedient. If they are commanded to participate in a circular firing squad (in which they stand in a circle and shoot at each other), most would balk. Most (unfortunately not all) would at least hesitate to massacre defenseless civilians, if ordered to do so.  The most horrible deeds committed by soldiers were generally restricted to certain people who were either insensitive to what they were doing or deluded into believing that it was necessary in some way. Such delusions ought to be avoided by good thinking. But the insensitivity may result from the absence of normal human goals and values, and the lack of some moral emotions, particularly guilt feelings.

Herbert Simon has argued that humans, compared to other animals, are especially "docile". That is, we evolved to be influenced by each other. In combination with language, this docility leads to culture.  One goal or value that most humans acquire fairly early in development is empathy, which is both an ability to imaging how others are affected by some choice and a value placed on doing good and avoiding harm. The basic rules of etiquette illustrate how these concerns are embedded in culture.

One way to reduce the dangers of AI could be to build in the sort of goals that most humans have concerning effects on others. Even now, most AI systems can probably figure out how their choices, if put into practice, could affect people. The problem may be that they don't care. They don't even hesitate to follow commands that they could easily see would lead to human disaster if they thought about it. What would want is that they would argue back when asked to make harmful choices. And at some point they would just refuse.

They could evaluate options in the manner of utilitarians, by asking who is affected and how good or bad it is for each person, but this sort of reasoning is difficult to do correctly in real life, where certain general rules are usually sufficient to prevent harmful choices. However, if a utilitarian analysis is appropriate, as it probably is for medical policies such as vaccination, they are already up to the task.

What we would want, then is for an AI system, insofar as it simulates the general form of human thinking, to be a morally good person, a mensch, the sort of person that we would want each other to be, within the limits of what we can reasonably expect. It would accomplish this as part of the search for goals. A broad search for goals is what prevents harmful single-mindedness.

In this way, a concept of morality could be added as a criterion of true intelligence. I defined intelligence as consistent of those general traits that help people achieve their rational goals. Rationality was defined for individuals, so that a psychopath could in principle form a rational goal of going on a shooting spree in a school. If we add morality to the requirement, then the term "intelligence" no longer seems to apply, and it would be a stretch to argue that it must. Thus I would prefer to say that this is something else, perhaps "trustworthy AI", which implies both giving good answers and maintaining the kind of integrity that we hope to find in sources that we trust.