Geoffrey Irving

Reflections on AISI

Fri, 15 May 2026 00:00:00 +0000

I left DeepMind to join the UK AI Security Institute (AISI) as a Research Director in December 2023 and officially started in April 2024 (I spent the intervening time on family travel, wandering around London, and formalising interval arithmetic). The past two years have cemented AISI as my favourite job to date: AISI has done a tremendous amount to sharpen the conversation around AI risks within the UK government, partner governments, and with the public, and it’s the first time I’ve been at an org so uniformly filled with people taking AI seriously and trying to make the world a better place.

However, for family reasons, I will be leaving AISI soon to move back to the Bay Area. I will be starting a new nonprofit alignment research org (more to come).

I have mixed feelings about this! It is the right decision on balance, and I am excited for the work I’ll be doing next, but I love AISI and will miss this place a ton. The reasons I joined AISI continue to hold: progress on technical safety research is too slow, coordination across AI developers and society is under-resourced, and governments have a huge role to play in that coordination.

AISI is an extremely impactful place to work, especially for Research Director-level candidates! We have a deep bench of excellent, driven talent; the ability to drive research on catastrophic risks or large-scale societal impacts that translates unusually directly into policy and action; and lots of access to very senior government decision makers who want to understand AI and influence its course. Please reach out if interested! I will continue on in an advisory capacity at AISI and am eager to pitch interested folk.

Broadening out from safety cases over time

Throughout my time at AISI as Research Director and then Chief Scientist, I advised across a mixture of technical research and policy discussions. Some of my favourite experiences were meetings with a mixture of experts across machine learning, civil service, law, and policy, mapping out how the government should best engage with AI. On the technical advising side I started out roughly 80% narrow (safety cases) and 20% broad (everything from details of statistics and capability elicitation for evaluations to timelines and AI trajectories), and gradually shifted to 20% narrow (alignment) and 80% broad (splitting my time across most of the research teams at AISI, with a focus on catastrophic risk).

We started the Safety Case Team shortly after I joined, then split it into multiple teams to reflect a fundamental trend across AI and the third-party oversight ecosystem. As one moves along a spectrum of risks from “present today” to “has yet to appear”, the type of oversight work changes. Where AI developers agree on threat-model details and deployed defences exist, such as with jailbreak safeguards to prevent human misuse for biological risks, AISI can focus on breaking those defences. Loss of control to AI systems is more subtle: there is wide disagreement about the plausibility of the threat and its details. When we started AISI’s research on AI control, no developer had mitigations in place, so all empirical work used red team/blue team games to simulate defences and then attack them (happily this has changed, and both OpenAI and Anthropic have deployed asynchronous control monitors). Alignment is subtler still.

This spectrum reflects the general evidence dilemma the world faces with AI: there are a mix of risks, ranging from already-here to…maybe? Ideally we find the wisdom to work on both kinds.

Getting to touch a bunch of different areas is tremendous fun. While I’ll be focusing on alignment as the next thing, it will be a portfolio of different bets within alignment (we don’t have any single bet that is all that likely to work).

Independent research matters!

When I left DeepMind, I thought of the move as setting aside technical safety research in favour of advising policy from the technical side. My perspective then was that the bulk of the safety work I would have wanted to do fit better within an AI-developer context, due to deeper access to frontier models and scale.

This was a bad take! There is way more scope for independent research than I had accounted for at the time (with, uncharitably, my AI lab blinders on). There is a certain slice of research that is smoothest within an AI company, such as work requiring full-scale RL or full whitebox access to the capability frontier, and it is easy to settle into a view that this slice is the important slice overall. Due to a combination of this comparative-advantage effect, cultural preferences, and a dose of groupthink, safety efforts within AI companies tend to fall within a fairly narrow space of available strategies. There are other strategies! It is not at all clear that the narrow set is going to do the job, and having a thriving ecosystem of third-party safety research orgs might fill in the strategies that work.

A corollary of “independent research matters” is that sometimes AISI work is good because it is within the UK government, and sometimes it is good because AISI is an independent research org full of talented people.

Governments matter!

But I did join AISI because it was in the UK government!

When I joined AISI I wrote “it is much easier to avoid risky actions if there is a large space of safe, beneficial actions to choose instead, and AISI has enormous levers to increase the size of this space and take such actions”. I continue to believe this!

Over the years since that post the space seemed to have contracted, but recently it has expanded enormously. The Vonnegut plot went down and then back up. This makes me more bullish on the impact of working in government!
AISI is well positioned to take these actions, including by being part of the UK government specifically!

I still firmly believe that governments have a critical role to play in the development and deployment of AI. In my time at AISI, in conversations both within AISI and with other governments, I’ve seen many examples where technical research and reasoning flows through to inform key policy choices. Some of this work is visible: In the very first few days after I got here, I was asked to input into the wording of the Seoul Commitments to ensure all the technical details were right. More recently, AISI’s Mythos results have received wide pick-up in the UK government, other governments, and the outside community. A lot of the activity is behind the scenes as well (and I assure you it is happening).

I also think what government friction there is at AISI has in many ways reduced over time. There is something self-reinforcing about success here: turn operating freedom or “doing things differently” in government into results, and people tend to give you further freedoms.

You matter!

There is a lot more to be done. I think now is an especially good time to be working in government for many people doing technical work on AI governance as the public and government conversation around AI is shifting. If you are a person earlier in your career, there are few better places where I think you can “have it all” on driving public mission while doing cutting-edge technical safety work, and being mentored by world-class research leaders.

And if you are an experienced Research Director-level person, I would love for you to come and do my job! Please reach out to me or to AISI leadership if you are interested in exploring further!

Thank you for holding my duck

Mon, 08 Mar 2021 00:00:00 +0000

There’s a story I like to tell, which I vaguely remembered as originating at Bell Labs or Xerox PARC. A researcher had a rubber duck in his office. When he found himself stumped on a problem, he would pick up the duck, walk over to a colleague, and ask them to hold the duck. He would proceed to explain the problem, often realizing the solution himself in the middle of the explanation. Then he would say, “Thank you for holding my duck”, and leave.

I love this story. Years ago, Tamar Shinar and I agreed that the “Thank you for holding my duck” expression is better if it is understood to not mean the colleague didn’t contribute, as then it can be used in boundary cases without slight. But finally, someone asked for the source, and I looked for one and failed.

I did find something, but it was worse than my version: Wikipedia’s page on Rubber duck debugging has a programmer explaining their problem to the duck. This may work, but misses the social aspect: explaining the problem to a person uses a different part of the speaker’s brain. I asked some people at Google; only Martin Wicke had heard of my version, and his source was me. I did learn from the inventor of the Burrows-Wheeler transform that rubber duck debugging is sometimes called “brickwalling” in the UK, but again that’s the worse version. Finally, with Tamar’s help I found my direct source: Bill Polson, back when we were both at Pixar.

Pixar, then the mists of time

Here is the story as far back as I can trace it. I got it from Bill Polson, who got it from Leo Hourvitz, who got it from a story about Xerox PARC. Leo, who worked at Apple and NeXT prior to Pixar, has “tried to search for an authoritative source many times, but any search involving PARC inevitably devolves into results about Steve and the Mac”. Bill’s words, slightly corrected by Leo:

According to Leo, there was a duck at one point at PARC, and it was involved in some sort of protocol. Probably you had to be holding it to speak in a standup or something like that. Or maybe it started by explaining yourself to the duck, but it just didn’t work unless you had a person there, so you gave the duck to someone to hold…

However it happened (losts in the mists of time) it did evolve into a meaning where the duck was a virtual duck, and “holding the duck” meant listening (only listening, not commenting) as someone worked through the problem in their own mind. In practice this worked this way:

I come into Geoffrey’s office. I need help. “Geoffrey, I’m trying to figure out how to diagonalize the defrobulator in sub-logarithmic time. I’m considering algorithm A, B, and C.” And I talk and talk and at some point, I say “Oh right, B is clearly the answer. Thanks!”

Geoffrey then says, “Glad to help!”

–

Ultimately, the term was used in this way: “Hey Geoffrey, I need you to hold my duck.” (Thus signalling: shut up, I need to work this out.)

And then: “Thanks for holding my duck.”

Bill continues:

In the interests of absolute clarity, I would say that there was a small group at Pixar, centered around Leo and the TS2 FX team, for whom this was absolutely the norm. Anybody who worked on that team, or who worked with someone who worked on that team, used this expression almost daily.

So to be clear, I doubt any animators ever used this.

But I took this to Pixar University, and then to the Shorts program, and then to the Tools Group. I would bet that Larry Cutler took this to Dreamworks and etc.

And now I can point people to this post when asked for a source!

Is it originally from Xerox PARC?

I would of course love to trace the story back to Xerox PARC. Please let me know if you have any leads!

Lessons from Lyndon Johnson

Wed, 25 Sep 2019 10:18:33 +0000

I’m in the middle of the third book in Robert Caro’s biography of Lyndon Johnson. The books are amazing; I can’t thank David Luan enough for recommending them. In brief, Caro’s thesis is

Lyndon Johnson cares only about power, not about issues. He is essentially amoral in pursuit of that power.
Lyndon Johnson is spectacularly skilled at politics.

It’s unclear how completely (1) holds; Caro provides plenty of evidence for it, but given Johnson’s facility with lies and pragmatism it’s hard to rule out underlying political views. The case for (2) is solid.

Moreover, (2) holds in a strong sense: Johnson is not simply skilled at politics, but far more skilled than nearly everyone around him. As a result, Johnson’s life is an example of asymmetric play in a theoretically symmetric game, and a beautiful illustration of how such asymmetric play is equivalent to the game itself having asymmetric rules. Much of the time, Johnson is simply playing a different game than the other political agents around him: he knows more of the state, has more resources, knows the rules better, and moves faster (literally running when others are walking).

Asymmetric play is equivalent to an asymmetric game

We have a multiagent game, with Johnson and the other political actors as agents. For the most part, we can treat the intrinsic game as symmetric: everyone is playing by the same rules. But relative to Johnson, the other players are impoverished in terms of skill, knowledge, resources, and knowledge of the rules themselves.

The intuitive point I want to make is that if we start with a symmetric game with asymmetric play (different players with different skill levels), and then coarsen, the coarsened game is asymmetric. Here’s the purest example of this phenomenon I’ve found in the books so far:

And then, all at once, Lyndon Johnson, standing next to his desk as he managed the bill under the unanimous consent agreement he had negotiated, noticed something. Under that agreement, two hours had been allocated to discussion of the Smith Amendment. The Republican arguments in favor of it had been completed, but the Democratic hour was just beginning. Not expecting a vote for an hour, senators had begun wandering on and off the floor. All at once, although there were still a substantial number of senators on the floor, that number did not include most of the liberals who opposed the Labor subcommittee bill—or most of the conservatives who opposed the bill. By coincidence, at that moment the bill’s strongest opponents all happened to be gone at the same time, leaving on the floor mostly moderates who were willing to settle for an unamended bill–no broadening of coverage but an increase to one dollar in the wage—in the form the Labor Committee had reported.

Johnson wants to pass a bill, but the vote count is against him. As Senate Majority Leader, he’s arranged to have a lot of power over exactly when voting takes place (previous Leaders had some of this power but did not take advantage of it). Senators are walking randomly on and off of the Senate floor, and when Johnson notices that one of these random fluctuations has shifted the count in his favor, he immediately calls the vote. We can illustrate the situation pictorially: The rest of the players see a random point of the random fluctuation, or its mean, or perhaps a settled value. Johnson sees, and acts, on the entire fluctuating curve. If we coarsen the game with respect to this behavior, Johnson’s reward in the coarsened game will be shifted up relative to the fluctuating mean.

The books are full of examples of this kind of asymmetry, with results essentially as if Johnson was playing a game with highly lopsided rules in Johnson’s favor. Let’s go through some examples.

Asymmetric resources

Johnson’s campaigns so far (I’m up to his 1948 entrance to the Senate) have been campaign spending outliers, with Johnson spending far more money than his opponents and indeed anyone in history for the corresponding Texas election. He had this money due to ties with various Texas businessmen which he managed to enrich via policy and favorable contracts. In the 1948 Senate race, this extra money allowed Johnson to

Blanket the state in media (radio, billboards, etc.), drowning out his opponent.
Campaign via helicopter, enabling far more campaign stops than his opponent, and drawing large crowds of people who had never seen a helicopter.
Buy tens of thousands of votes.

The media blanket allowed Johnson to win even with obvious lies: a key to his victory was claiming that his opponent, Coke Stevenson, was in the pocket of labor. This was sufficiently absurd that Stevenson refused to dignify it with a crisp denial, instead referring to his stauchly conservative record. When Stevenson finally realized the lie was working and issued a clear statement, he lacked funds to get the statement into the media at sufficient scale, and Johnson continued to broadcast his lie.

Dodging transparency

Johnson was careful to avoid being tied down to particular policy views in public. In private, especially in 1-1 conversations, he was then able to claim whatever the other person wanted to hear. Liberals got the sense that he was secretly liberal, voting conservatively only to satisfy his Texas base. In the Senate, conservatives felt he was deeply conservative, catering to liberals only to unify the Democrats and advance towards the presidency. Indeed, he would often tell mutually contractory stories in the same room, such as in the Senate Democrat cloakroom (deeply divided between liberals and conservatives).

Although there was some awareness of this strategy, much of his success would have evaporated had his various statements been common knowledge. In the Senate, he frequently managed to pass or block legislation while creating the impression that his goal had been the opposite. The Bricker Amendment would have sharply limited presidential power. Bricker was anathema to Johnson since it would have limited his own power once he achieved the presidency, but was strongly supported by many of his Texas funders. He managed to kill it in 1954 via the complicated strategy of encouraging another senator to introduce a milder amendment to draw support away from the original bill, arranging the vote counts to initially favor the new amendment, then killing it by a single vote in a second pass. Johnson himself voted for Bricker in each case, and word that he was actually against Bricker never reached his funders.

Knowing all the details

In the Senate, my impression from Caro’s book is that Johnson has a detailed model of every senator, including how they will vote and how they can be convinced otherwise. He knows all the rules of the Senate, and can take advantage of them in novel ways. None of this knowledge is magical: it was gained by talking to everyone, reading all of the rules himself or via staff, spending hours watching the function of the Senate, and being intelligent enough to hold all of this state in his head.

As the only person who knew the details of everything going on in the Senate, senators who wanted to know how to get a measure passed or blocked learned to consult Johnson. His status as an information hub allowed him to accrue actual power despite having little recognized power according to the rules, such as during his time as Assistant Majority Leader in the Senate (a role no one prior to Johnson wanted). Although he occasionally used this hub status to communicate lies, it was effective even when telling the truth: people who didn’t help Johnson were exiled from the information source.

Asymmetric speed

Johnson spent a lot of time literally running. (In the traditionally dignified Senate, he would sometimes run almost to his destination but slow to a dignified walk for effect.) He was more effective at using staff than other politicians, and drove his staff much harder, allowing him to accomplish tasks faster and in parallel. One can imagine a go game where one player gets to make twice as many moves. We can view the situation as a game that starts symmetric with continuous time, then coarsens to an asymmetric game with discrete time where Johnson gets to play more often.

But he still did a lot of good!

Despite Johnson’s amorality and the asymmetry of the game he was playing, he still ended up doing a lot of good. I haven’t gotten to the Civil Rights Act and the Great Society, but at my current point in the book he just managed to pass a massive affordable housing bill and a $\frac{4}{3} \times$ minimum wage jump to the distinct surprise of everyone else in the Senate (using procedural tricks such as the sudden vote call above). (The Vietnam War is less good.)

This could be explained by Johnson acquiring power through asymmetric play and then revealing his secretly altruistic preferences. Caro doesn’t have enough evidence to rule this out, but I think he does have enough evidence that Johnson’s positive accomplishments can be explained by the search for power. Prior to Johnson, the prevailing view was that no Southerner could become president. His shift towards the liberal side in the Senate let him work around this problem by unifying the liberal and conservative sides of the Democratic party, while preserving the support of conservative Southerners (who expected to get a conservative Southerner in the White House).

AI safety optimism

Overall I think this is hopeful story: even a woefully asymmetric game managed to align Johnson towards positive things. Moreover, the features that caused the asymmetry are not subtle: money, speed, knowledge, lack of transparency. In the AI safety case we get to design our own game. We can use balanced agents for symmetry, with matched resources and speed, full common knowledge to reveal inconsistencies, Ideally, interpretability methods will let us look inside agents and see what they are actually thinking. Hopefully we can design a game better than the one Johnson was playing.

Morality does not come from within (retracted)

Sun, 30 Dec 2018 07:21:40 +0000

Retraction (15 April 2022): Greg Egan has kindly explained on Twitter that I was misinterpreting the narrator’s statements, and specifically that from the “from within” part means that morality is in part a result of human internal mental processes but that those processes of course condition on the external world. I am happy to stand corrected!

Post prior to retraction:

Greg Egan’s short story “Silver Fire” is about people falling back from secular values. It’s the near future, and organized religion is fading away but “the saccharine poison of spirituaity” is taking its place. The main character is a medical researcher, and most of the plot deals with spirituality in conflict with reliable science. In the background, the reseacher worries about her daughter, who thinks science is boring and much prefers alchemy.

I won’t go into the plot further. Here’s the important bit, where Egan goes wrong:

We thought we were passing on everything that mattered to our children: science, history, literature, art. Vast libraries of information lay at their fingertips. But we hadn’t fought hard enough to pass on the hardest-won truth of all: Morality comes only from within. Meaning comes only from within. Outside our own skulls, the universe is indifferent.

No, morality does not come from within. We choose it to some extent, but that choice is shaped by the world. Morality is evolved, an iteratively tested system for making human interactions go well. The idea that morality “comes from within” disconnects it from the world, and morality disconnected from the world is a weird, barren thing.

So of course people would run away from such a notion. It is not just that “morality from within” sounds lonely and isolated, it violates basic common sense. And it violates that common sense not because common sense is wrong, but because morality comes from the world, from all of us together, interacting through time. Partly from within, mostly from without. And it is only because it comes from the world that morality is a thing which can be reasoned about, understood, improved over time.

I’m not sure if Egan intends “morality comes from within” to be a true statement outside the context of the story, or merely a thought by a fictional character. I doubt he would disagree that morality is shaped by the world, so upon reflection he might agree that the statement is wrong. But my guess is that he wrote it thinking it was true, that it was a reasonable way of describing the world.

Towards the end, the narrator expresses her worries about her daughter this way:

I missed them both already, more than I’d anticipated–but I wasn’t sure how I’d manage when I finally made it home, to a daughter who was turning away from reason, and a husband who took it for granted that any bright adolescent would recapitulate five thousand years of intellectual progress in six months.

That recapituation will be a lot easier if we choose secular slogans that are true, rather than actively misleading.

A constructive critique of Sapiens and Homo Deus

Fri, 28 Jul 2017 09:21:05 -1000

Thanks to a recommendation from Dandelion Mané, I recently read “Sapiens” and “Homo Deus” by Yuval Noah Harari. Both books are wonderful breaths of fresh air and perspective. “Sapiens” is organized as a history of the species Homo Sapiens, tracing from our evolutionary separation from other primates through the cognitive revolution, the agricultural revolution, through the rest of history to the present. From this historical background, “Homo Deus” attempts to extrapolate into the future, in particular asking how our morality and goals will evolve with technology.

The core theme of both books is the separation between three types of reality:

Objective reality, which exists independent of our believing in it. Physics, biology, evolution, etc.
Subjective reality, which exists only in the mind of the believer.
Intersubjective reality, which exists in the mind of many believers at once.

Intersubjective reality includes such ideas as cities, money, religion, and companies. Money is an imagined construct whose meaning vanishes if everyone stops believing in it, but it does not vanish if only I stop believing in (though it does slightly decrease in value).

Harari’s notion of religion includes not only Buddhism, Islam, Christianity, Hinduism, Judaism, etc., but also Communism, Liberal Humanism, Nazism, and Science. This is true. The idea of Science as not religion is roughly the Aristotelian view: the idea that we can derive the structure of the world from pure logic, with no assumptions required. This is not the case: one needs religious assumptions such as “objective reality exists”, “the world is explainable in simple terms”, “induction is correct”, and so on.

So we need belief systems – religion – and religions are not objective. But religions are influenced by the objective world, and change to the objective world forces change in religions. The major classical religions have one by one acknowledged evolution and been changed by it. The same applies to the religion of Liberal Humanism, which enshrines individualism, free will, and self determination. The solidity of these concepts is being eroded by neuroscience, behavioral economics, and the increasing technical power of advertising and marketing.

Harari is worried about the future. “Homo Deus” asks what comes after Liberal Humanism once the intersubjective notion of free will dissolves. He does not have a good answer, and he is rightfully concerned that society is not focusing on the question.

So “Sapiens” lays out the history of our species, and “Homo Deus” worries that we are heading blindly into a future of changing morality. It’s a good question to ask. Harari doesn’t know the answer. Unfortunately, Harari misses a huge part of the story, which prevents him from talking about the space of solutions in depth.

What Harari is missing: derived morality

Here’s how I think about morality, using Harari’s language:

To start, you need a religion.
The religion let’s you derive various principles about the world, and about how one should behave.
One uses the derived principles to make decisions.

Unfortunately, Harari has a poor understanding of the boundary between religion and derived principles. Conveniently, Harari has a single sentence which perfectly illustrates the problem:

No amount of data and no mathematically wizardry can prove that it is wrong to murder, yet human society cannot survive without such value judgements.

This is stated as a fact; decoupled, he means two things:

You can’t derive that murder is wrong from theory.
If humans did not believe murder was wrong, society would not survive.

He’s identifying “derive” with the Aristotelian notion of proving something from pure logic. This is a meaningless notion: one always needs axioms, and the useful notion of derive is to produce one statement from another. With the right notion, his statement is itself a derivation of the “wrongness of murder” from the “wrongness of society not surviving”.

Is that last bit objectively true? No: it’s a religion. But it’s potentially a simpler religion, and the history of the scientific revolution is in part a replacement of complex religions with simpler religions.

Does this answer his existential crisis? No: it doesn’t say what the religion should be, and there are many mistakes one can make in thinking about simplicity of religion (see below). However, it does highlight the problem: Harari doesn’t seem to realize that the intersubjective is composed of many levels, with factual or semifactual arguments relating the different levels. In this example, our religion (one level of the intersubjective) is “society should survive”. “Murder is wrong” is also part of our religion, and is usually taken as an axiom (a commandment, if you will). The fact that “murder is wrong” is part of the religion, however, does not mean that it can’t be approximately derived from other parts.

To summarize, Harari’s paraphrased position is:

You need a religion. The current religions fall apart at our current and upcoming level of technology, and it’s not clear what will replace them. We should think hard about this.

This is exactly true but misses the blur between religion and derived religion, in large part because historically it was a lot blurrier than it is today (though many people today also don’t get it).

It’s interesting to contrast this with Sam Harris’ position from “The Moral Landscape”:

Science can be used to derive most of morality, and increasingly most over time. Not everything, but most. Also my religion is that morality is about improving human well being and if you don’t agree with my religion you are unreasonable and incompetent.

Harris’ position is only partially true, but it contains the problem with Harari’s. The right position is a mixture of Harari’s views and some of Harris’: Harris is right that morality can be derived, but wrong that “human well being” is the one true religion.

In any case, missing this critical piece of the story, Harari’s discussion of possible futures contains quite a bit of nonsense. The subtlety is that some of this nonsense is Harari’s and some of it is legitimately shared by the religious adherents he is describing.

The dangers of simple religions

Before we get into the details of where “Homo Deus” goes wrong, it’s important to point out some perils of derived religion. Derived religion is about turning simple religions plus facts about the world into complicated religions. This can go wrong in a few ways:

One can start to have, as a religion, “religion should be as simple as possible”. This is one of the guidelines of my religion, for instance, but it’s not an absolute. There’s a reason the quote is “simple as possible, but no simpler”.
One can imagine that one has a simple religion but misinterpret the actual underlying goals. For example, libertarians claim the goal is freedom, often to the point of religion. To justify this goal, they often say things like “freedom makes humans happier”, or “freedom let’s people thrive”. Well, which is it? Freedom or thriving? What if we have evidence that thriving is sometimes inconsistent with maximal freedom? If we act based only on the simple religion, we may fail to achieve our true goals.
One can make incorrect inferences from the simple religion to a complicated derived principle. Harari takes it as given that “don’t murder” is important for civilization surviving. Certain Evolutionary Humanists (Nazis) instead concluded that civilization surviving required killing all the Jews. The former inference is correct, the latter is not. In general (not in that case), the process of inference can be complicated and subtle, since it requires understanding emergent phenomena for which we usually can’t do controlled experiments.

To elaborate on the difficulty of inference: we can sometimes do it, but usually only when the surrounding context is taken as fixed. Abortion is a good example: advocates can say things like “abortion doesn’t appear to lead to bulk devaluation of adult life” or even sometimes “countries that allow abortions have fewer abortions”. This seems conclusive, but there is almost always an escape hatch reachable by (1) widening the context and (2) appealing to a bit of religion: “yes it seems like abortion doesn’t have bad consequences, but it weakens the moral fabric and takes one further from God’s laws”. To combat this, one also needs a bit of religion, though a bit I agree with: “entropy is a common result of complex mixing operations, so without evidence to the contrary the short term disappearance of signal is evidence there will be no long term effect” and that “we can find a consistent religion to replace Christianity even if abortion weakens it”.

Help, my religion is dissolving

Harari’s worry in “Homo Deus” is that the modern dominant religion, Liberal Humanism, is dissolving under the forces of science and technology. Different religions have been dissolving for tens of thousands of years; this time is different because of the speed and because of our newfound ability to destroy ourselves if we get the transition wrong.

Like all religions, Liberal Humanism is a composite religion with a bunch of different beliefs, some of them derived from others. Some of these derivations were made consciously by humans, some happened by cultural evolution. Since Harari misses the concept of derived religion, his discussion of Liberal Humanism misses some historical and game theoretic context. It also misses some of the solution space: if we understand how different parts of Liberal Humanism derive from others, we may be able to preserve some parts even as others dissolve.

In particular, Harari spends a lot of time discussing how Liberal Humanism dissolves once algorithms know us better than ourselves. Liberal Humanism supposedly says one should look inside oneself for answers. First, this is not what Liberal Humanists said, at least not to the extreme Harari is taking it. He repeatedly says the Liberal Humanist ideal is to go to the ballot box, look deep inside at your emotions, and pick the candidate which resonates the best. But this idea, while certainly a large part of how people vote in practice, would have horrified the Founders. Their ideal was an informed (also white, male, land owning) citizen that would look at the facts of how policies would affect them, and act according to a combination of their and their societies actual interest.

The issue is that “look inside oneself” is a derived value, caused by two principles:

An outsider does not know you as well as you do.
An outsider does not share your interests.

(1) is a factual statement which is dissolving. (2) is a combination of a factual statement and a religious reference to “your interests”. These interests are left blurry, but probably religiously include Life, Liberty, Happiness, etc. In any case, once (1) dissolves the entirety of (2) still holds, both the misalignment of values and the religious reference to interests.

Can we preserve (2)? Well, it would require having a picture of the what values are, but is also requires engineering the world to avoid massive information asymmetry and power asymmetry. We’re not doing a great job of this engineering yet, but it’s a natural inference if one tries to preserve as much of Humanism as possible. Harari does not discuss information asymmetry as a derived religious value, because he takes it as given that at least at first Google shares our interests.

His discussion of free will is similar. The underlying trend is that science is proving free will a suspect concept. However, the reason free will was a useful concept historically, and is still useful if not as useful today, is that free will is what you get when you can’t predict behavior from outside. That is, it’s a derived or emergent concept that emerges once you coarsen enough to not know the details of someone’s thoughts.

There are a couple inferences from this. One is that free will is still a useful concept for more complex entities that are too complex for other entities to know their thoughts. The superintelligences of the future, if they exist, will still have free will or a partial equivalent as a concept, since the laws of physics prevent one place from having all the information. Second, free will is a useful concept even in asymmetric situations where Alice knows enough to see through Bob’s free will, but Bob does not know enough to see through his own free will. For example, the disappearance of free will in terms of cognitive neuroscience does not immediately invalidate the use of free will as a concept in the criminal justice system: if people think they themselves have free will because they can’t model themselves very well, then free will can rationally be used as an argument for punitive responses to crime in terms of its preventative effects. If you want to argue against punitive imprisonment, you can’t just say “free will is a myth”; you need something real like “punishment doesn’t prevent crime very well”.

Actually, the Bible wasn’t written by God. QED.

Another strange section of “Homo Deus” is his discussion of the factual claims that underlie religions. Harari states that, while much of the purpose of religion is moral statements about the world, all religions make assumptions about objective reality. As I noted above, science assumes objective reality exists, induction works, etc. Christianity assumes the Bible was written by God. If you want to argue against a religion, he suggests targeting the factual claims it makes, and presenting evidence against them.

But Harari basically stops there, with “Christianity assumes the Bible was written by God”. He then goes off on an interesting but largely irrelevant historical discussion of the evidence that, in fact…drumroll…the Bible was not written by God. For example, linguistic analysis shows that different parts were written at different times, by different people. Take that, Christianity!

Of course those are the wrong facts to target. Harari is taking “The Bible is written by God” and the closely related “The Bible is literally true” as atomic statements. Once you’ve assumed the whole Bible is true, everything else follows. For example, I know the Bible is literally true, and the Bible says homosexuality is bad, so homosexuality is bad. But if you actually ask a fundamentalist Christian why homosexuality is bad they won’t stop at this argument. They’ll say gay people have more mental illnesses. They’ll say they’re more likely to be pedophiles. They’ll say children with gay parents grow up warped. These are the factual claims one should be targeting.

Again, the issue is that he’s missing the concept of derived religion. Sure, Christianity says the Bible is true, but it also makes a bunch of factual claims about derivations from one set of moral precepts to another. Since it’s a patchwork of claims, they can be peeled off semi-independently. The core “the Bible is written by God” claim is actually quite robust, since it can be easily modified without breaking the whole derivation structure.

It’s strawman time

Religions dissolve in pieces. There are still a lot of Catholics, even though the part about humans being created in one go has dissolved into an acceptance of evolution. Indeed, the first sentence of the relevant Wikipedia page is

Since the publication of Charles Darwin’s On the Origin of Species in 1859, the attitude of the Catholic Church on the theory of evolution has slowly been refined.

I would like Liberal Humanism to dissolve slowly too. The advantage of slowness is that we don’t have a complete answer for what to replace it with, so slower is safer. And the way towards a slow dissolve is to understand how the different parts of Liberal Humanism derive from one another, so we can understand which parts to keep.

Or, absent a concept of derived religion, we could talk about strawmen.

The weirdest part of “Homo Deus” are Harari’s hypothetical “what comes next” religions. They’re not pretty, and they’re not meant to be: Harari is worried about the future, and he’s worried because he doesn’t have a good answer for what replaces Liberal Humanism. However, his candidate replacements aren’t just bad: they fail the same tests that are killing off the current religions.

Take Dataism. Dataism, according to Harari, is the belief that the goal is more data processing. The problem is that he is proposing this as a replacement for Humanism as the concept of individualism dissolves, but his Dataist concepts were nonsense from Shannon’s first paper on information theory. What does data processing mean? Well, it probably means one needs a definition of “data”, but the whole insight of Shannon is that you get coherent definitions only once you assume a separation between data and noise. A random process is not necessarily data: it might be noise, or it might not be, and the only way to know is to measure how well it can be used to predict another data source. Presumably one should use interesting data sources for this comparison, but then you get an infinite regress only resolvable by injecting some religion which chooses the interesting data sources. And you can’t just say “any data”: the time symmetry of physics means the total information of the universe is preserved with time, which is not very interesting as a goal. You can’t propose Dataism as a religion to replace the dissolving Humanism if Dataism comes predissolved.

In fairness to Harari, some of this silliness is intentional, since some of the confusion about Dataism and the other strawman replacement religions is shared by actual adherents. Harari is right to be frightened by Dataism; indeed, it bares a striking resemblance to Paperclipism (“the goal is more paperclips”). But having spent a whole book talking about dissolving religions, not mentioning the predissilution of Dataism is a major lapse.

Change religion slowly, and change objective reality

I am more optimistic about the future than Harari. Not optimistic enough to not worry about it – I join an AI Safety Team on Monday – but hopeful. And fundamentally, the reason I am more hopeful than Harari is that I think large chunks of our current religion (our morality) can survive intact across the next period of rapid technological change. This is not to say our current religion is ideal, but I don’t have anything better to replace it with, so it will have to do.

So how can we let religion dissolve slowly? There are two ways:

Break it into pieces, analyze how different pieces derive from other pieces. Figure out which pieces survive in our changing objective reality, and which we do actually want to keep. Try to keep them, let the others fall away.
Sometimes, adjust objective reality so that a piece that we want to keep but is in danger of dissolving doesn’t dissolve.

Most of this post has been about (1), so I won’t say much more. I’m not saying (1) is easy, and the answers are not going to be simple. A good approximation to human morality achievable today will be moderately high dimensional. Sometimes we can do the derivations and then pick the simple axioms, but even in the “murder is wrong” case the terms require definitions, and there are a lot of cases and caveats. This is not to say the simple low dimensional principle doesn’t exist, but due to the pitfalls discussed above I don’t think will find it time, safely.

What about changing reality? Take free will. As discussed above, free will is a meaningful emergent concept in situations where an agent can’t be externally predicted. This is important because the outsider attempting prediction generally has different goals. The free will approximation breaks down when information is highly asymmetric: secretive government surveillance, advanced machine learning-based advertising, etc. But information asymmetry can be fought: EFF, journalists, others. Usually it’s better to argue for this in terms of explicit consequences, but it also preserves the environment that makes free will a meaningful emergent concept, so it’s part of the story of how religion dissolves.

Against long term thinking

Sat, 17 Dec 2016 14:39:14 -0800

The Long Now Foundation is a wonderful organization advocating for long term thinking. Specifically, by long term they mean the next ten thousand years:

The Long Now Foundation was established in 01996 to develop the Clock and Library projects, as well as to become the seed of a very long-term cultural institution. The Long Now Foundation hopes to provide a counterpoint to today’s accelerating culture and help make long-term thinking more common. We hope to foster responsibility in the framework of the next 10,000 years.

The 01996 gimic is to remind people of the $10^4$ year timescale. While I agree that our society focuses too much on the short term, the Long Now’s timescale is off by a factor of 100-200. Instead of short or long term thinking, we need what I’ll call “medium term thinking”, meaning 10 to 50 or 100 years out. Here’s why:

A detour into kalah

Kalah is a game where stones are picked up and moved around a small board to score or capture other stones. A typical game lasts 20-30 moves. A game ends when one player runs out of stones on their side of the board; that player then wins all stones on the opponent’s side as points.

Trivially, long term thinking is the right policy if one has sufficient resources: you should think ahead to the end of the game, and take into account everything that happens throughout. This includes both points scored in the middle of the game, and the possible windfall at the end.

I wrote a program to play kalah in high school. It was a simple brute force search engine. At first, it could see ahead only 4 or 5 moves, and was easy to beat: all one had to do was plan for the end of the game. The computer would rack up a few tactical victories on the way, but the windfall would go to the human at the end, and the human would win. A clear victory for long term vs. short term thinking.

But then I optimized the program, and it started to see 7 or 8 moves ahead. It still wasn’t long term thinking: remember that a typical game lasts 20-30 moves. Despite that, the computer would win every game. Indeed, it would win by a landslide: it would rack up a bunch of points along the way, and also get the windfall at the end.

It matters who has control

What was happening? When the computer was looking 8 moves ahead, it was still only taking into account points scored within those 8 moves. However, points scored towards the end of the 8 move window tend to go to the player who has tactical control over the board: if I have a lot of safe options at move 5, I’m more likely to win points in moves 6-8. Thus points scored by move 8 is a proxy metric for tactical control over the board.

And then! This 8 move plan is not used to make 8 moves: only the first such move is actually played. The next time the computer moves, it uses a new 8 move window, looking ahead a total of 10 moves into the game. And so on. At all times, the computer is optimizing a proxy metric for tactical control, as a byproduct of scoring a few points along the way. Eventually, when the 8 move window sees to the end of the game, the computer already has tactical control, and it can use this control to grab the windfall.

How to choose a timescale

Now back to the real world. We would like to choose a timescale (or range of timescales) around which to structure our thoughts. The lesson from the kalah example is that we shouldn’t think too short term, but we also shouldn’t think too long term if control is decided in the middle.

Consider what’s going to happen in the next 10-100 years. If civilization survives, we’re going to get strong AI. A good chunk of global warming will have already happened or not, along with a much better understanding of mitigating factors (geoengineering). We may have a self sufficient space presence. We’ll be able to print custom human genomes from scratch.

Even just the first of these, strong AI, almost entirely determines control. A few other dimensions matter over a long timescale, such as climate change and other environment disasters, but these also matter in the short and medium terms! To argue against medium term thinking in favor of long term thinking, you need a dimension that affects only the long term, or the long term in a different way than it affects the short term; climate change does not apply.

Thus, the next time someone adds a zero to a date: dissent. A hundred years ought to be enough for anyone.

Increasingly bizarre typos?

Mon, 28 Dec 2015 05:48:51 +0000

I make weird typos when writing. Sometimes I substitute an entirely different word in place of the correct one; otherwise times I simply a word. Both kind of typos are more common than misspelling a word, indicating that the typo mechanism is operating at a higher level than the spelling or typing itself.

This parallels some of the intuition people have about deep neural networks, which is backed up by pretty pictures of what different neurons see. According to the intuition, a deep neural network for classifying images starts with low level, local features of images (gradients, edge detectors) and moves layer by layer towards high level features (biological vs. inorganic, fur vs. hair, golden retriever vs. labrador retriever).

A neural network trained to generate images rather than classify them operates in reverse: it starts with high level features and moves towards lower and lower level features until it spits out pixels at the end. As a consequence, the kind of errors produced by such a network depend on the layer at which the error first occurred: an error near the pixel level will be localized and fairly boring; an error at a higher level might create an extra eye in the middle of a limb (as in Deep Dream).

Humans are similar, at least phenomenologically: we can make typos at a variety of scales. I can mispell words, skip words, say grammatically correct sentences with small logical errors, write entire programs based on faulty assumptions, harbor incorrect political views for decades, and so on. There’s error correction, so most of the mistakes are caught quickly. Not all, though, and some of the mistakes that are not caught quickly are subject to positive error correction, and remain mistakes for years or decades.

At the highest levels, many of our biggest mistakes are due to invalid or imprecise connections between different ideas. For example, the U.S. political system has unified the ideas of fiscal conservative, religiousness, and uncontrolled gun ownership into one party, which means that mistakes in one of those issues blur over into mistakes in the others. This is a typo at a very high level, both in terms of ideas and numbers of people.

This blurring of somewhat but not exactly related concepts is one of the keys to intelligence, so it’s going to remain with us forever. One thousand years from now, the superintelligent AIs of the future will still have implicit associations between only vaguely related ideas. Indeed they will be much more creative they we are, so they will likely have more implicit associations between even less related ideas than we do. They will also have much deeper thought processes, in the sense of having more layers with a greater gap between low level and high level features.

Will they have even weirder typos? No.

Error correction

Error correction is perhaps the single most important idea in computer science. For example, it is more important than the idea of a bit, since without error correction it would be impossible to approximate a discrete choice like a bit on top of the analog physical world. The most important fact about error correction is

A reasonably small amount of error correction is the same as an infinite amount (almost).

In other words, if you’re building a computer system out of a huge number of little pieces, you need to have error correction for each of the pieces, but you do not need more error correction because you have a lot of them. That’s not exactly true: a computer with a lot of pieces needs a little more, but most of what you need is already needed by a very simple computer. For example, say we have a physical transistor which makes errors 10% of the time. That’s a lot of errors, so it might take quite a few transistors in a little error correction circuit to drive the errors down to 1% of the time, and then 0.1% of the time. However, once you’re there, adding more copies of the error correction machinery will drive the error rate down exponentially, so it doesn’t take very much to get it down so far that no errors will ever be detected (a practically infinite amount of error correction).

This principle is true for classical computers as proved by von Neumann, and it is true for quantum computers as proved by Ben-Or and Aharonov. In both cases, there is a finite threshold $\epsilon \gt 0$ such that if the error rates of each gate can be pushed below $\epsilon$, error correction can push it exponentially close to 0 with a moderate amount of extra work.

Unfortunately, this is not true of humans. We have some error correction, but it is unevenly distributed and in many areas we don’t have nearly enough to hit the threshold. However, there’s no reason to believe a self-designed AI wouldn’t take advantage of the threshold theorem(s), so whenever it desired it would make no errors (with arbitrarily high probability).

There are two main caveats to that statement which need to be explored: the definition of an “error” and “whenever it desired”. First, when considering fuzzy, human type intelligence, the concept of “error” is pretty fuzzy, and in general defining “error” is the same as being able to formally specify AI. However, since $\mathrm{P} \ne \mathrm{NP}$, recognizing an error in a thought is often much easier than having the thought in the first place; whenever this is the case error correction can work. The really problematic errors (such as not recognizing climate change) are caused not by failure to define the error (by scientists) but a failure of our mechanisms for propagating information through the rest of the system (politics). That latter bit is what error correction fixes.

Second, error correction is in apparent conflict with our previously discussed mechanism of intelligence: implicit associations between slightly related ideas. To some degree I think this is true: there are presumably valuable creative thoughts which cannot be well translated into a formalized setting. However, I think in practice the conflict is illusory, because the ideal level of error hovers on the boundary between chaos and rigor.

Just the right amount of error correction

If there is a lot of error, long trains of thought are impossible: any interesting though will decay into noise in a few steps. Too little error is more subtle, since by the principle above it doesn’t take too many extra resources to have almost no errors at all. The actual problem with too little error is that isn’t very “nimble”: interesting thoughts that aren’t the most interesting thought might decay away before they can be explored and amplified. For a formal analogy, we can imagine a process that can either converge to zero or explode to infinity depending on a scale parameter; the interesting behavior is in the thin middle regime where it neither explodes nor converges. Another analogy is a PID controller (for a robot or heater, say): a very high damping coefficient means errors are hard to introduce but response is molasses-like, a very low damping coefficient causes oscillations, and a damping coefficient near the critical value gives the fastest possible response to both errors and new settings.

For a less formal analogy, consider an artist or musician making a new piece. Many of the good artists know quite a bit of theory, and can apply that theory to avoid low level “mistakes” in the work. However, depending too much on theory can interfere with creativity, and even where the theory is useful it may be best applied by first internalizing it into semi-conscious reflexes and then applying the reflexes.

An interesting aspect about a level of error correction that hovers between too little and too much (between error-prone creativity and plodding rigor) is that with a little bit of error it is possible to push it to one side. That is, if we have a thought using the critical value, but can revisit and replay the thought process, more work should let us push the thought into rigorous, explainable territory, at which point any remaining errors can be detected and driven to zero. This is the real reason why I think AI will essentially immune to bad errors: if we accept that the ideal error rate is near the threshold value, there is no reason not to go the rest of the way for important decisions.

Of course, humans can do this too: an art critic can push a little further to explain aspects of a creative work using art theory, and a mathematician can push a little further to turn intuitive conjectures into proofs. We just don’t do it enough. We’ve evolved something pretty close to a critical error rate, but since it’s evolved it’s uneven, and our processes for correcting errors at higher levels are similarly uneven. Hopefully we get closer to the threshold before we explode.

You can't always get what you want

Mon, 11 May 2015 01:06:20 +0000

(This is an expanded version of a Facebook comment, because Jeremy asked.)

Recently I came across an article about opposition to housing development in San Francisco. The headline positions of everyone involved are uninteresting: housing advocates want more affordable housing, housing developers want less. The really interesting bit is more subtle: at one point the developer says they’re still trying to figure out what the community wants and is immediately booed. What he means is that they are trying to figure out how the community would prefer to allocate a fixed amount of housing affordability across different levels of affordability: 50% of market, 80% of market, etc.

I think this is one of the most problematic failures of modern political discourse: not admitting that a desired quantity is bounded or has to be traded off against other desired quantities. The same failure applies to security vs. privacy, taxes (who do you tax vs. how much), libertarianism (freedom vs. externalities), rationality (optimal policy making vs. democratic fairness), etc., etc.

It’s important that this is a failure of discourse, not just policy by itself. No two people will have a common ordering for the importance of all desired quantities: unless each person is willing to suspend their ordering by freezing some of the quantities and consider how the others change, constructive conversation is impossible. Even if everyone did agree on the relative importance of issues, freezing some of the quantities is a vast simplification that can lead to useful insights about the best solution even if the final solution does not freeze these quantities.

Here’s a recent experience that highlights how easy it is to talk past one another even if everyone fundamentally agrees. A friend and I were discussing different algorithms for picking a set of representatives that accurately captured a group’s views. I kept using the preamble “fix a number $r$ of representatives”. My friend pointed out that this was an absurd assumption: obviously the best number of representatives is problem dependent, and sometimes we need more representatives to capture the range of opinions. I responded that if we let the number vary freely, the best answer is clearly to pick everyone as a representative, which doesn’t achieve our goal of picking representatives at all. My response came across as an attack on a weird strawman. It took us surprisingly long to figure out what the other was talking about, since from different perspectives we were both correct: the ideal representative set size varies, but freezing the set is a useful simplifying assumption to start out.

“If we fix A, then …”

The above rambling can be summarized into the following rule for communication:

Rule 1: If someone says, “If we fix $A$, then $B$”, they are not saying that $A$ should be fixed. Listen to the rest of the sentence.

For example, if I say “If we fix total taxation, …” and you start yelling before I’ve reached the end of the sentence because you strongly believe taxes should go up or down, all hope is lost. What I want to talk about is contained in the “…”. Even if you think that the level of taxes is the most important thing, you should still listen to the “…”: it might be something we both agree on.

There is an equally important flip side to Rule 1:

Rule 2: If someone follows Rule 1, it does not mean they have accepted that $A$ should be fixed.

Rule 2 is important because it is one reason why Rule 1 isn’t followed in practice. Following Rule 1 means you have to join the other party in a hypothetical world which may not conform to your views, or even to reality. If joining the conversation counts as an admission of defeat, following Rule 1 is bad. Sound bite culture is a problem here, since a quote pulled out of a weird hypothetical world may reflect negatively on the speaker when taken out of context. To use a mathematical analogy: it would be like pulling a sentence out of the middle of a proof by contradiction and yelling, “Look what they’re crazy enough to believe!”

Let’s try to apply these rules to the political examples above.

Security vs. privacy

This one is really about security vs. law enforcement access vs. privacy vs. complexity, and part of the reason everyone keeps talking past each other is the refusal (for at least one side (I am biased)) to have subconversations with only 2 or 3 dimensions at a time. Examples are trotted out in support of the view that more law enforcement access means more security, then shot down by those on the privacy side. But obviously more law enforcement access means more security in a limited sense: if someone on the privacy side doesn’t admit that, they will look like extremists to anyone on the security side. The question is how much access helps, what kind helps, when, etc.

The security side generally ignores the issue of protocol complexity, arguing for tricky split key policies allowing privacy against anyone without a court order. It’s obvious that ignoring complexity is silly, but beware: theoretical CS is on the side of the authorities. In 10 or 20 years, the complexity issue will vanish as automated proof eliminates the bug penalty for linking together multiple protocols. Moreover, there’s presumably a finite amount of money, probably less than a billion dollars, which would make a complicated split key protocol more secure than most existing software. “Complexity is bad!” is not a sufficient argument: the question is how bad, and whether the cost is worth it when traded off against the other qualities.

In any case, the complexity argument is unnecessary: the “we don’t trust the authorities” argument is future proof and strong given the existence of other countries.

Taxes: who vs. how much

Given that U.S. politics is strongly polarized between “No more taxes!” and “More taxes!”, why on Earth don’t we pass bills that freeze the total level of taxation but shift around the relative amounts? Pretty much everyone agrees that jobs are good and pollution is bad. Taxing something means people will do less of it, and a sane political system would act accordingly.

A lot of this falls to Rule 2. Politicians who have gotten elected on simplistic “No new taxes!” stances are unwilling to support bills which do not cut the overall level, because doing that would count somehow as admitting defeat. Interestingly, Norquist’s Taxpayer Protection pledge claims to not be this simplistic, explicitly allowing proposals that move taxes around without increasing the total level. The link from this claim to the pledge itself is hilariously a broken link, but once you get through their safeguards the pledge itself does seem to back up this claim:

I, NAME, pledge to the taxpayers of the state of STATE, and to the American people that I will (1) oppose any and all efforts to increase the marginal income tax rates for individuals and/or businesses; and (2) oppose any net reduction or elimination of deductions and credits, unless matched dollar for dollar by further reducing tax rates.

On the other hand, a literal interpretation of the pledge suffers from a ratcheting phenomenon where it is possible to shift tax revenue from anti-labor to anti-pollution but impossible to shift it back, since the pledge specifically targets the marginal individual tax rate. Even if this ratcheting is in a good direction, it bans experiments, changes in circumstances, and all kinds of subtlety. What if we shift taxes towards pollution but then eliminated pollution for other reasons: would taxes be allowed to shift back? Is it compatible to sign a bill that shifts taxes between income and pollution adaptively? Probably not, in which case someone who wants taxes to go up would have trouble collaborating with a pledge-taker on a tax shifting bill.

Libertarianism: freedom vs. externalities

A while ago I saw an appalling statement in a Facebook conversation: someone said they used to believe more strongly in libertarian principles before deciding the situation was more complicated, and someone else said something like “It must be relaxing to have given up consistency.”

Obviously freedom is good. But freedom is also multidimensional: there are over 7 billion dimensions of freedom even if you only count humans today and have only one dimension per human. Property rights do not cleanly choose whose freedom matters more in every situation, nor does the idea of violence vs. non-violence. The world is fuzzy, and believing in fuzziness is not inconsistency.

But obviously freedom is good, which means that if there is a way to achieve more freedom without losing too much in return, we should do that. If we could achieve a higher quality heath care system for most with less regulation, but the cost was that $N$ people had no health care, should we do it? Yes, depending on $N$, and the threshold is not zero.

Rationality

This one is the most fun. One of the things you should be willing to trade off vs. other things is rationality itself. These other things include time, simplicity, and inclusiveness. It’s irrational to spend too much time trying to be rational, or to never use faster slightly irrational heuristics. Less rational simpler models are often better: they have fewer possibly broken moving parts, are less vulnerable to overfitting, and are easier to communicate to others. A political system that weeded out irrational views would be necessarily less inclusive, which has downsides even if the weeding process was perfect: those left out would be less satisfied and more prone to revolt, etc.

Rule 0: If you do not accept that your desired quantities cannot be optimized without trading off something else, you are doomed to talk nonsense.

No more experts

Mon, 07 Oct 2013 06:07:46 +0000

I am writing this in Mac OS X, having momentarily given up getting Linux satisfactorily configured on my laptop. So, in the spirit of escapist fantasy and cracking nuts using sledgehammers, I am going to write about what a world with strong AI would be like. Warning: I am in a very lazy, rambling mood.

Say we get strong AI. This means we understand intelligence sufficiently to be able to replicate it digitally. We’re going to completely ignore any potential speed advantages: pretend that this new strong AI has exactly the same effective intelligence as a normal human when running on conventional hardware. However, like everything digital, intelligence is now repeatable, shareable, and mixable.

What does repeatable, shareable, mixable intelligence mean? Basically, it will be the end of experts. If you’ve spent years accumulated specialized knowledge and skills, anyone else can absorb these skills in a tiny fraction of the time. Sure, they need your permission, but presumably you aren’t the only expert in your field, and someone else will be more than willing to give away their skills (either free or for a price). There are privacy and security concerns, but these are resolvable: we need a few seed experts willing to forgo all privacy to build a reasonably secure AI simulation platform, which can then bootstrap into safely simulating even more experts. The seed experts will either plentiful or spectacular heroes, so there should be no difficulty finding them.

Initially the simulated experts might be only consultants in boxes, available to anyone to ask questions but not directly integrated into one’s consciousness. However, that won’t last: digital intelligence is easy to experiment on, and we’ll quickly get better at sandboxing and separating out the knowledge and expertise we want and integrating it into other minds. The risks involved are fairly small: if you need to combine your expertise with several different kinds to solve some task, just put the combination behind a firewall and inspect the results carefully. Yes, some people will balk at simulating and then destroying copies of their own minds; these people are going to have trouble competing.

What does the world look like without experts? Here are a few examples:

Government

Why take what politicians say on faith when you can simply copy a policy expert, merge with them, and check? Sure, some people will refuse to think reasonably even if it takes no effort; these people will be a tiny, irrelevant minority. The vast majority of voters will simply ignore anyone who doesn’t share the reasons behind their political statements. The idea of representative democracy starts to look a bit silly when the voters know just as much as the representatives, and liquid democracy (or something better) is easy if everyone has optional expertise in game theory and security.

Privacy and security

Secure communication is already possible if you have the relevant expertise. So now it is easy for everyone. If you don’t want to think about it, just buy an extra computer and simulate you plus security expert checking all of your Facebook posts. This makes the NSA’s job impossible: now all the terrorists are security experts too. Luckily, every average citizen is suddenly an optional expert in counterterrorism (not to mention probably backed up, which makes terrorism fairly difficult).

Computer science

To end the bickering between people building system software in C and functional language theorem proving advocates, someone breaks down and simulates a million aggregate years of combined hardware hacker / systems programmer / formal methods / mathematician time and produces a proof that a (slightly modified) Linux system is correct. It is suddenly very difficult to get anyone to download software which doesn’t have some level of safety proof; all legacy software is relegated to formally checked sandboxes. All new low level code now comes with matching high level code, plus a proof (courtesy of a constructive logic expert if necessary) that they do the same thing. Parallelism bugs go away, and all new parallel hardware comes complete with simulatable parallel hardware optimization expert (you only need one!). There is no longer any reason why Linux trackpad support needs to be so inferior to OS X.

Automation

The concept of “automated” starts to blur, is it a self driving car if you are simulating a widely shared driving expert with part of your mind? Is it is an automated factory if the automation is a thousand copies of one guy pumped full of simulated anti-boredom drugs?

Economics

Entrepreneurs no longer need to assemble a team of expert employees to get work done, and the employees are expert entrepreneurs anyways. Thus, reasons for different people to work together boil down to either fun or resource sharing (everyone is now an expert capitalist).

Conclusion

I’m going to end the post here, since I really have no idea what such a world would be like (other than fun!), and have satisfied my desire for rambling. I’m fairly confident we’ll get there as long as climate change and energy shortages don’t destroy us, so here’s hoping.

Rootstrikers conference summary

Sun, 21 Apr 2013 21:45:18 +0000

I went to the Rootstrikers conference yesterday, which consisted of a few panel debates/discussions plus questions from the audience. I also got to hang out in a bar at a table with Lawrence Lessig for a half hour or so after the conference, which was pretty cool. I’ll summarize the conference here, and include links for anyone who wants to follow the movement or get actively involved.

Stepping back: there are two core ideas behind Rootstrikers: (1) representative democracy in the United States is being corrupted by the influence of money and money-connected lobbying, and (2) even if this corruption isn’t the most important problem to solve, it is the FIRST problem to solve, since it is blocking satisfactory progress on nearly every other issue (climate change, tax reform, health care costs, etc.). Lessig has a variety of talks laying this out; the most recent one at TED is particularly well done (I hadn’t seen it until yesterday), and I highly recommend it:

Lawrence Lessig, TED 2013

The main question is what to do if you believe this argument. I’m going to focus on what to do with a small amount of time. There are a few different efforts attacking the problem, and each one could use a small amount of time in different ways. The common thread in all of these is (1) provide public funding for elections and (2) block lobbying and large donor funding from swamping the public funding:

The American Anti-Corruption Act

I just read the full text of it online here. The plan is to get one million “citizen co-sponsors”, then seek co-sponsors in Congress and attempt to pass the bill. It consists of a variety of explicit “no corruption” provisions, the best of which forces Congresspeople to recuse themselves from legislation regulating their significant donors. It also gives each citizen a $\$100$ tax rebate which they can donate to any candidate who agrees to fairly strict per-donor limits ($$500$ / person, in particular); more discussion of this below. All the provisions seem good, except possibly banning Congresspeople from raising money while their house is in session, which may be unworkable and/or unnecessary given the other provisions.

If you agree with the provisions, please co-sponsor it even if you believe it is vanishingly unlikely to pass directly! If a sufficient number of people get on board, it will generate a lot of media attention even if Congress ignores it, which could further boost support for the general movement. Moreover (I asked Lessig about this at the bar), if it fails at the federal level it can be used as a model for legislation at the state level.

Signing on is easy, though you should read either the summary or hopefully the full text first. Presumably it’ll generate a small amount of email spam, which you can largely ignore until it moves to the next stage of attempted introduction to Congress. In the best case I can imagine a SOPA/PIPA-ish spamming of Congress by the entire internet when this happens; the main point of tolerating the email spam would be to know when to participate in such (call/email representatives, etc.).

State-level efforts

Several states have passed publicly financed election laws with tremendous effect. In several, the vast majority of candidates sign on to public financing, which requires them to accept money only in small portions from individuals which are then matched by the state. Common Cause has a summary of which states provide public financing.

Other states are very close. In particular, in New York both the governor and the majority of the legislature is on board with the idea. The Brennan Center has a description of one proposed plan. If you live in New York, please email or call your representative and ask them to support this effort! Passing public financing in one of the largest states would be huge.

In California, there is a proposed bill called the California Disclose Act, which would expand disclosure requirements in a few different ways. For example, political television ads would be required to clearly state their three largest donors. You can help this effort by contacting your state representative and asking them to support it.

Other states may have similar efforts: look them up!

Constitutional Amendments

There are two main organizations focused on a constitutional amendment for federally funded elections, Wolf PAC and Move to Amend. There are two arguments for a constitutional amendment: (1) the Supreme Court has started to block campaign finance reform as unconstitutionally infringing on the first amendment, and (2) Congress is so horrifically corrupt that there no chance of Congress-based reform. The downside is that constitutional amendments are very hard.

Wolf PAC is lobbying state legislatures to pass a resolution calling for an Article 5 Constitutional Convention, with the goal of an amendment that (1) bans corporate personhood and (2) implements federal public funding of elections. Move to Amend is similar. The corporate personhood ban makes me a bit angry, and I was rather turned off by how demagogic Move to Amend’s David Cobb sounded when he spoke at the conference. Huge individual donors (who often own corporations) seem just as bad as corporations to me, and it turns out only 11% of SuperPAC (I think) money in the last presidential election came from corporations; the rest was from individuals.

However, Wolf PAC has made some interesting progress pushing on state legislatures to call a convention, and they seem a good route if you want to contribute a small portion of time. Cenk Uygur said that often it only takes a handful of calls from constituents (say, 5 people) to turn a state legislator into a co-sponsor of their plan. The convention itself is fairly safe: once called it has the power only to propose the amendment, which must then be ratified. Moreover, there is historical precedent for Congress freaking out if this kind of thing seems likely to happen and passing amendments themselves, which could speed up the process.

Thus, with the proviso that corporate personhood may be an irrelevant emotional push-button, Wolf PAC (and possibly Move to Amend) may be a good use of a small amount of time, either to call one’s own representatives or to call people in key states to get them to call their representatives in turn.

Or just Rootstrikers

If you’re unsure which option is best but want to follow the movement, a good default is to just sign up with the Rootstrikers mailing list, which will keep you informed of how both of it and related projects are developing.

More conference summary

The conference itself was organized into the following four sessions:

Different mechanisms for public financing

There was a panel discussion debating the merits of three different mechanisms for public funding: matching funds, vouchers, and tax rebates. All three in order to be meaningful would disallow candidates accepting public financing from raising additional campaign money, and would limit individual donations to a small amount (between $\$100$ and $$500$, typically). Matching funds would multiply the individual donation by some amount (say 6x), and vouchers and tax rebates would both provide a small amount of free money per citizen to donate to the candidate of their choice. The vouchers and tax rebates have the obvious benefit that extremely poor people can still participate. Tax rebates also fit perfectly into the historical narrative of America’s anti-tax streak: the Republican advocating for tax rebates expressed this beautifully as “No taxation without representation”, meaning no taxation without representation in the funding part of elections.

One suggested advantage of matching funds is that small amounts of money could be provided to start candidates off, but this fits in just as well to the tax rebate scheme. In fact, if you believed as an individual citizen that a tax rebate scheme didn’t do enough of this, you could simply donate your rebate to a general pool of small candidate helper money established by an authorized 3rd party, which would then donate to an appropriate candidate, solving this problem without need for special provisions.

Thus, tax rebates seem like the best scheme. They are also the one included in the American Anti-Corruption Act, which is great.

Campaign finance reform as a civil rights issue

The next discussion consisted of various people arguing that this sort of political corruption and campaign finance reform can be understood as fitting into the narrative of civil rights, since African Americans, Latinos, etc. are disproportionally not included in the tiny number of significant donors. This seems quite true, and articulating it could build support for campaign finance reform from those focused on civil rights issues. Unfortunately, due to the extremely unfortunate polarized nature of the civil rights debate, linking the two issues has the potential downside of turning away many people on the right.

Lessig expressed this concern in an interesting way in his keynote speech at the end. The founders certainly lacked a modern understanding of the dangers of discrimination based on race, gender, etc. However, one thing they did understand was class, and the constitution was explicitly intended to prevent takeover of the government by some sort of aristocracy. This has clearly failed, and the advantage of interpreting the corruption issue in this context is that it fits into a narrative that existing in a reasonably correct form from the beginning of the country.

Luckily, we can do both, so not really a conflict.

Constitutional amendments and corporate personhood

This session was more of an actual debate, between three people advocating for a constitutional amendment as the sole focus and one person arguing for a focus on state efforts (in particular the strong possibility of public financing in New York). As mentioned, I was a bit turned off by the pro-amendment side’s emphasis on corporate personhood as opposed to the actual solution of public financing of elections. The more important details are above.

Lessig’s keynote

Lessig then played his TED talk and then gave a related in-person talk, both of which were quite good. The TED talk is great even if you’ve seen similar talks of his, so definitely worth watching.

The one charity theorem

Wed, 13 Mar 2013 05:11:10 +0000

The normal scheme for donating to charities is to divide money up among several different charities. The following argument shows why this strategy is often wrong. Both the statement and the proof will be extremely informal:

One charity theorem: Assume we have a fixed amount of money to divide between $n$ charities. Assume that utility is a smooth function of the performances of the charities, which in turn depend smoothly on the amount of money each receives. In the limit of a small amount of money, it is optimal to give to only one charity. Conversely, with overwhelming probability, it is never optimal to give to more than one charity.

Proof: Let the utility function be $u(X) = u(x_1, \ldots, x_n)$, where $x_1 + \cdots + x_n = T$ are the amounts of money given to each charity. Since $u$ is smooth and $T$ is small, we can linearize to get

$$u(X) \approx u(0) + \nabla u \cdot X$$

where $\nabla u$ is the gradient of $u$. This linearized version is maximized by giving all money to the charity which maximizes $du/dx_i$. Moreover, as long as this maximum value is distinct, which occurs with overwhelming probability if we imagine that $u$ is a bit noisy, the maximum is unique.

Practical matters

It’s important to discuss when this result applies in practice and when it doesn’t. First, uncertainty does not matter, including uncertainty about “the values of $du/dx_i$”. All kinds of uncertainly are simply folded into the utility function $u$, which results in more smoothness rather than less. Thus, the result applies even if you don’t know what your preferences really are; in this case, just make a good guess and give all the money to that charity.

Similarly, multiplexing in time also does not matter to some extent: even if we don’t know about the future, we can simply list “charities in the future” as one of the entries and apply the theorem. In particular, saving up money and donating it in larger chunks may be better than numerous small donations over time if one expects to have more knowledge (and therefore more accurate utility) in the future.

There are two valid reasons the theorem may not apply: failure of smoothness and failure of smallness. The smoothness is easy: two charities need a small amount of money to meet a certain goal, donating a small amount of money to both may be optimal. However, this applies only if the discontinuity can be accurately predicted. For example, Kickstarter projects have a threshold amount which must be reached to take effect, but an individual donating a small amount of money will still have a smooth utility function due to the unknown amount of other people’s donations.

The more important condition is smallness, which turns utility into a fully nonlinear function and in particular increases the likelihood of discontinuities. Smallness can arise either because the charity itself is small, or because utility depends on something intrinsic to the donation rather than the performance of the charity. If you derive personal satisfaction or reputation based on the number of charities you donate to, independent of the amount, the result does not apply (and also you are part of a problem). If the charities themselves are small, so that smaller donations to several have both a significant effect and significant diminishing returns, great. This is one of the reasons why microfinance is such a great idea.

On the other hand, even if there are too many charities for the theorem to apply for each individually, it’s possible that it does apply to entire classes of charities. It may not be rational to donate to both microfinance and any other charity, for example.

Also note that same argument applies when money is replaced with donations of time, though it’s much easier for time to have personal utility terms which break smallness.

Rootstrikers

This argument is one of the main reasons why Rootstrikers is a good idea in principle: given a small amount of influence on a large system such as government, it is highly likely that focusing on one problem to the exclusion of all else is the only rational course. This applies to those with fulltime politically related jobs (e.g., Lessig), or to donations of money and (again to a lesser extent) donations of small amounts of time, thought, etc.

While nothing in the argument says that Rootstrikers is the best way to go about this task, or that Rootstrikers is the best organization with this plan, remember that uncertainty does not invalidate the theorem. If you are small, pick one. Do not hedge charities.

Would anarchy work?

Tue, 11 Dec 2012 04:50:54 +0000

In the scale free government post, one of the completely unresolved issues was what to do about the federalism axis. There are two scale free extremes to choose from: completely uniform democracy and pure libertarianism (i.e., anarchy). This post will ramble about the anarchy option without getting anywhere very useful.

Anarchy would only work if the universe is such that the middle ground can be efficiently simulated by ad-hoc coordinated groups. Recall that the goal isn’t actual anarchy, which is absurd, but a system with as few foundational rules as possible.

Here’s a This American Life episode describing a typical sham treaty between the U.S. and Dakota Indians in Minnesota which later turned into a war. I haven’t listened the whole thing yet since one part already struck me as illustrative: the initial treaty was negotiated by mistranslated a key portion from English to Dakota, so that the Dakota didn’t realize what they were going to sign. Then, during the actual signing, the Dakota were asked to sign an extra document (they thought it was another copy of the same treaty) giving away most of the settlement money for “debt” purposes.

Thus, the first prerequisite for anarchy to work is for all sides to have a fairly similar level of legal/game theoretic/political knowledge. At the moment, this isn’t remotely true at the level of individuals; the easiest example are the software license agreements we all scroll idly by. It’s also not true in mass: if it were, political advertising would have only informational effects. However, at least at the level of information, it’s possible to imagine technological or cultural improvements that could improve the situation. Cryptography is a good case (as mentioned here); a cryptographic defender can be exponentially weaker than an attacker and still remain secure.

Back to the Dakota: after the treaty, they were giving a 20 mile strip of land around a river. This was sufficient for farming but not for their traditional hunting lifestyle. Imagine they had kept enough land for hunting. This (cough, arguably) would go against the societal interest of the surrounding (new) majority population, since hunting is not an efficient use of land in terms of population density. However, Dakota on sufficient land for hunting were presumably quite self sufficient. As it happened they didn’t play the game well enough, fell into debt, and were tricked into the treaty. If they were smarter and kept clear of the surrounding economy, the only external pressure available would have been military. Let’s set that aside for a minute.

Thus, the next prerequisite for anarchy to work is some effective form of externally available pressure. There’s not much point if the external pressure has to be military (this basically reduces to normal government). The next best thing is probably boycotts or their variants (sanctions, tariffs, etc.), which large groups of people would have to jointly agree to. Setting aside the difficulties in setting up such an agreement, boycotts would not have been effective in the case of the Dakota; they would simply have laughed them off and gone back to hunting. The modern economy isn’t even close to self sufficiency for two reasons: (1) the massive amount of capital required for such high technology projects such as semiconductors and (2) comparative advantage. It would be nice if (1) would go away for fragility’s sake, but it’s also possible that (2) could soften, especially if transportation costs increase, energy and food become more local, the internet remains free, etc. These are great, but would also reduce the amount of external pressure available to combat carbon and other pollutants (imagine if the self sufficient Dakota were spewing tons of CO$_2$ into the sky).

And then, of course, the Dakota were crushed by industrial military power. The main game theoretic problems that military power would pose to an anarchist structure are (1) economies of scale and centralization and (2) the imbalance between defense and attack. The effectiveness of guerrilla warfare on home turf suggests that (1) may not be a problem. If a widely dispersed and self organizing military works, it would fit right in. I don’t have anything useful to say about (2). If (1) favors guerrillas and (2) isn’t too biased towards attack, the same techniques used to structure other forms of pressure seem like they should work fine for the military.

Back to the radio…

Toothpaste and amortized complexity

Mon, 03 Dec 2012 04:08:51 +0000

A past girlfriend and I would occasionally (cheerfully) quibble over the optimal strategy for extracting toothpaste. It occurred to me recently that the disagreement was fundamentally about amortized vs. worst case complexity.

Being lazy, I tend to squeeze the toothpaste out of the front of the tube, optimizing the time spent in the moment and reducing the degree of control required since pressure is exerted near the toothbrush. She would carefully squeeze the tube from the back, maintaining a flat region that would slowly grow as the toothpaste emptied. The main advantage of her strategy is that toothpaste is always at hand, and every iteration is fast. In contrast, squeezing from the front pushes toothpaste both out of the tube and backwards towards the other end. Occasionally, this must be fixed by rolling the back of the tube forwards. Each fix up step takes much longer than a normal toothpaste extraction, but the average time spent might be lower since a normal squeeze is faster. We never did the experiment, so I’m not sure which strategy actually wins on average.

However, the real problem is heterogeneous parallel computation: if two people squeeze a toothpaste tube from the front, their thresholds for when to perform a fix up step will almost certainly differ (I had significantly more finger strength). The result is unfair: the person with the lower threshold will end up doing most of the work. Relatedly, in a parallel context the time lost during a fix up step can amplify, since more than one agent can stall waiting for fix up on a shared resource to complete. Incidentally, in contrast to the computational case, for toothpaste it might be even worse if the fix up step falls to the second person.

Some of these points definitely came up, but I don’t remember the details. The important thing is that I immediately caved. :)

Not everything happens for a reason

Wed, 21 Nov 2012 05:42:40 +0000

The phrase “everything happens for a reason” came up in a couple contexts recently (conversation with a friend, Radiolab, etc.). It’s a good example of an obviously false statement that contains plenty of useful insight, and is interesting to think about in that context.

We’ll get the pedantic out of the way first: “everything happens for a reason” is literally true in the sense that the future happens for the reason that is the past. What people are usually implying is “everything happens because of a simple event in the future”. It isn’t worth wasting time tearing apart that absurdity.

What’s more interesting is why people say such a thing. Here are some possible related statements:

I choose not to regret the past.
Every event contains benefits.
We’ll make it work.

I was imagining writing more, but I think that sums it up. You do not need to be irrational to be optimistic and positive.

Exponentially harder isn't hard enough yet

Tue, 03 Jul 2012 03:04:36 +0000

In well designed cryptographic security systems, the attacker needs to do exponentially more work than the defender in order to read a secret, forge a message, etc., subject to appropriate hardness assumptions. Maybe this is true for many non-computer security-ish systems as well, like choosing good representatives in a voting system or avoiding overpaying for advertised merchandise, and we simply haven’t reached the level of intelligence as defenders for the exponential effort of attackers to be prohibitive. Extra intelligence is a ways off, but faster and simpler access to information is closer, and may have similar effects. In any case, a hopeful thought.