I left DeepMind to join the UK AI Security Institute (AISI) as a Research Director in December 2023 and officially started in April 2024 (I spent the intervening time on family travel, wandering around London, and formalising interval arithmetic). The past two years have cemented AISI as my favourite job to date: AISI has done a tremendous amount to sharpen the conversation around AI risks within the UK government, partner governments, and with the public, and it’s the first time I’ve been at an org so uniformly filled with people taking AI seriously and trying to make the world a better place.
However, for family reasons, I will be leaving AISI soon to move back to the Bay Area. I will be starting a new nonprofit alignment research org (more to come).
I have mixed feelings about this! It is the right decision on balance, and I am excited for the work I’ll be doing next, but I love AISI and will miss this place a ton. The reasons I joined AISI continue to hold: progress on technical safety research is too slow, coordination across AI developers and society is under-resourced, and governments have a huge role to play in that coordination.
AISI is an extremely impactful place to work, especially for Research Director-level candidates! We have a deep bench of excellent, driven talent; the ability to drive research on catastrophic risks or large-scale societal impacts that translates unusually directly into policy and action; and lots of access to very senior government decision makers who want to understand AI and influence its course. Please reach out if interested! I will continue on in an advisory capacity at AISI and am eager to pitch interested folk.
Broadening out from safety cases over time
Throughout my time at AISI as Research Director and then Chief Scientist, I advised across a mixture of technical research and policy discussions. Some of my favourite experiences were meetings with a mixture of experts across machine learning, civil service, law, and policy, mapping out how the government should best engage with AI. On the technical advising side I started out roughly 80% narrow (safety cases) and 20% broad (everything from details of statistics and capability elicitation for evaluations to timelines and AI trajectories), and gradually shifted to 20% narrow (alignment) and 80% broad (splitting my time across most of the research teams at AISI, with a focus on catastrophic risk).
We started the Safety Case Team shortly after I joined, then split it into multiple teams to reflect a fundamental trend across AI and the third-party oversight ecosystem. As one moves along a spectrum of risks from “present today” to “has yet to appear”, the type of oversight work changes. Where AI developers agree on threat-model details and deployed defences exist, such as with jailbreak safeguards to prevent human misuse for biological risks, AISI can focus on breaking those defences. Loss of control to AI systems is more subtle: there is wide disagreement about the plausibility of the threat and its details. When we started AISI’s research on AI control, no developer had mitigations in place, so all empirical work used red team/blue team games to simulate defences and then attack them (happily this has changed, and both OpenAI and Anthropic have deployed asynchronous control monitors). Alignment is subtler still.
This spectrum reflects the general evidence dilemma the world faces with AI: there are a mix of risks, ranging from already-here to…maybe? Ideally we find the wisdom to work on both kinds.
Getting to touch a bunch of different areas is tremendous fun. While I’ll be focusing on alignment as the next thing, it will be a portfolio of different bets within alignment (we don’t have any single bet that is all that likely to work).
Independent research matters!
When I left DeepMind, I thought of the move as setting aside technical safety research in favour of advising policy from the technical side. My perspective then was that the bulk of the safety work I would have wanted to do fit better within an AI-developer context, due to deeper access to frontier models and scale.
This was a bad take! There is way more scope for independent research than I had accounted for at the time (with, uncharitably, my AI lab blinders on). There is a certain slice of research that is smoothest within an AI company, such as work requiring full-scale RL or full whitebox access to the capability frontier, and it is easy to settle into a view that this slice is the important slice overall. Due to a combination of this comparative-advantage effect, cultural preferences, and a dose of groupthink, safety efforts within AI companies tend to fall within a fairly narrow space of available strategies. There are other strategies! It is not at all clear that the narrow set is going to do the job, and having a thriving ecosystem of third-party safety research orgs might fill in the strategies that work.
A corollary of “independent research matters” is that sometimes AISI work is good because it is within the UK government, and sometimes it is good because AISI is an independent research org full of talented people.
Governments matter!
But I did join AISI because it was in the UK government!
When I joined AISI I wrote “it is much easier to avoid risky actions if there is a large space of safe, beneficial actions to choose instead, and AISI has enormous levers to increase the size of this space and take such actions”. I continue to believe this!
- Over the years since that post the space seemed to have contracted, but recently it has expanded enormously. The Vonnegut plot went down and then back up. This makes me more bullish on the impact of working in government!
- AISI is well positioned to take these actions, including by being part of the UK government specifically!
I still firmly believe that governments have a critical role to play in the development and deployment of AI. In my time at AISI, in conversations both within AISI and with other governments, I’ve seen many examples where technical research and reasoning flows through to inform key policy choices. Some of this work is visible: In the very first few days after I got here, I was asked to input into the wording of the Seoul Commitments to ensure all the technical details were right. More recently, AISI’s Mythos results have received wide pick-up in the UK government, other governments, and the outside community. A lot of the activity is behind the scenes as well (and I assure you it is happening).
I also think what government friction there is at AISI has in many ways reduced over time. There is something self-reinforcing about success here: turn operating freedom or “doing things differently” in government into results, and people tend to give you further freedoms.
You matter!
There is a lot more to be done. I think now is an especially good time to be working in government for many people doing technical work on AI governance as the public and government conversation around AI is shifting. If you are a person earlier in your career, there are few better places where I think you can “have it all” on driving public mission while doing cutting-edge technical safety work, and being mentored by world-class research leaders.
And if you are an experienced Research Director-level person, I would love for you to come and do my job! Please reach out to me or to AISI leadership if you are interested in exploring further!