AI Safety and Governance: A Guide to High Impact Careers

The last several years have seen a rapid increase in the capability of artificial intelligence (AI) systems. Over the coming decades, there are likely to be even more significant advances in this technology. This could drastically change society—both for the better and for the worse. 

This path profile will cover two related paths focused on mitigating these possible risks: AI safety and governance. Together, these paths encompass both technical work focused on ensuring AI systems behave as intended, as well as non-technical work focused on influencing the development, adoption, and governance of AI in socially responsible ways. 

We think these paths could be highly promising for those who are a great fit. However, just how promising depends on some very uncertain predictions about how AI technology might develop, and how serious the associated risks are. This is also a path where personal fit seems to be especially important.

What do we mean by artificial intelligence safety and governance?

In this career profile, we’ll explore two of the main career paths to help mitigate potential risks from AI: technical AI safety, and AI governance.

Technical AI safety 

The field of AI safety is a technical discipline aimed at ensuring AI systems are aligned with human values and goals, reducing more speculative risks posed by powerful, more advanced AI systems we could see in the future.

There’s quite a few key questions and different approaches within technical AI safety. One problem that receives a lot of attention is interpretability, i.e. our ability to decipher how AI models really work so that we can supervise them properly. Another strand of research focuses on how AI systems could be vulnerable to “adversarial” attacks, whereby users trick AI systems into displaying unintended behaviors, or even use them for potentially harmful purposes. These are just a couple of examples of approaches to AI safety—there are many others you can look into.

In terms of the paths you can take within technical AI safety, it’s worth making a distinction (following others) between empirical and theoretical safety work. 

Empirical AI safety workers dive into the nuts-and-bolts of existing AI systems. They both design and run experiments in order to extrapolate relevant insights which might generalize across AI systems, including more advanced AI systems in the future. 

Many people in this path are engineers with backgrounds in machine learning and computer science, and are capable of working with sophisticated technology stacks and managing large software projects. Typically, people within empirical AI safety also have experience in machine learning. People on the empirical side of AI safety work in AI companies, research nonprofits and think tanks, academia, and even government.

On the other side, those working in theoretical AI safety tackle more high-level, abstract questions about potential risks from advanced artificial intelligence, often drawing from disciplines like mathematics and philosophy. A couple of examples of theoretical work include conceptualizing the nature of “agency”—in other words, what it means to have autonomy in performing actions—or determining how future AI systems might make decisions in order to guide our safety strategies.

Though theoretical safety researchers can work in many of the same organizations as those in empirical research, there may be fewer theoretical roles available, and a heavier emphasis on academia.

AI governance

Another career trajectory within the field of AI Safety is that of AI governance. In short, professionals in AI governance work at the intersection of technology, law, and policy to identify and implement strategies to guide the safe development of future AI systems. 

This includes work to steer (and where necessary, constrain) the development of advanced AI systems, as well as prevent their misuse and ensure that society is equipped to handle the transformative effects that advanced AI systems might bring about. 

Governance work is typically a combination of research, strategy, advocacy, and policy. It’s less focused on the nuts-and-bolts of AI systems themselves, and more on the regulatory frameworks that can help shape the future of AI in a way that encourages benefits while minimizing risks.

To make this a little more concrete, a few pieces of recent work in AI governance has included trying to form policies that ensure AI companies don’t scale their AI models beyond their ability to handle them safely, creating grading criteria for AI companies’ safety frameworks, and advocating for governments to exclude AI systems from the nuclear weapons control centres. This is nowhere close to being an exhaustive list of governance efforts, but it’s illustrative of the kind of work produced in the field. 

In terms of organizations, most professional AI governance and policy roles are typically found in government, research nonprofits and think tanks, as well as academia.

How promising are careers in AI safety and governance?

As we’ve already discussed, the field of AI safety involves a lot of uncertainty. Because of this, the potential impact of pursuing a career in this space will depend a lot on the scale, severity, and likelihood of associated risks—which even experts disagree about.

Because AI could be so transformative for human society, getting AI right is incredibly important in terms of scale. If it goes right, AI could bring about significant benefits, like accelerating scientific progress and assisting sustainable development.

However, the potential downside risks of AI are also enormous. As we cover in our introduction to AI safety, some autonomous algorithmic systems are already having negative effects, for instance through displaying biases in hiring practices or prejudicing black defendants in parole hearings. 

Though the capabilities of existing systems are still fairly limited, in the near future, biased decision-making could become even further entrenched as these systems become more widespread. A host of new risks are likely to open up, like the greater spread of disinformation and the easier creation of biological weapons.

Furthermore, the possibility of highly advanced superintelligent AI systems introduces some even more dramatic risks. For instance, they could be intentionally used to spread and protect authoritarian regimes, or we might lose control of powerful AIs who don’t share human values. Many working in AI are concerned that these advanced AIs could cause massive and irreparable damage to civilization, posing an existential risk. However, there’s sharp disagreement on the plausibility of these outcomes.

In fact, there’s a huge amount of uncertainty and disagreement surrounding all facets of AI technology. For one, there’s substantial disagreement among experts about when, or even if, advanced forms of artificial intelligence (significantly beyond what we have now) will be created. A 2023 survey of AI experts gave a median estimate of a 50% chance of the advent of AI that can surpass human ability in every task by 2047, though many experts gave much shorter, or much longer, timelines. 

Where you stand on these uncertain questions will likely make a big difference to how promising you view AI safety and governance as high-impact careers. The more confident you are that advanced AI will be created, and that this could lead to an existential catastrophe, the more impactful these careers will look. We’ll recommend several resources at the end of this article for those who want to develop an informed view on this topic.

However, even for those worried about the potential dangers of AI, there still may be reasons to prioritize other cause areas and career paths instead. 

For one, though AI technical safety challenges range in how difficult they may be to overcome, some of the more fundamental issues within AI safety may be very difficult to solve. In fact, some in the field have become skeptical that good solutions are feasible (though this is far from consensus).

Effective governance efforts are also likely to be difficult to implement. Policy changes require persuading high-ranking officials (as well as the public) across the world. And with the huge amounts of money being poured into AI companies, there are strong financial incentives to quell regulation efforts. Having said that, policy change in AI isn’t necessarily much more difficult than it is in other cause areas—to a significant extent, these barriers are present among most cause areas. 

Furthermore, there have been recent indicators that policymakers may be sympathetic to mitigating risks from AI. For instance, in 2024, a bill proposing tighter regulations on AI development companies was considered by the California state legislature. Though the bill was vetoed, it received significant domestic support from policymakers and the public, showing that AI governance work can get traction. Many governments are now also explicitly concerned with these risks, with some investing significant resources. For instance, the UK government established the prestigious AI Safety Institute.

Overall, though, we suspect that making a positive difference within AI safety and governance is much harder than many other careers we’ve covered. Despite this, it’s likely still a very promising option (at least, in expectation), for people who could excel in either of these fields.

We don’t have a clear view on whether technical AI safety or AI governance might be more promising to work on if you want to work in this area. It’s likely that this will depend in large part on your personal fit for the different types of work, as well as your background and skillset. 

It’s important to note that, as with all careers, your personal fit for AI safety and governance roles will play a big part in how promising they might look to you. However, our sense is that personal fit is probably even more important for these roles than most, since making real change—particularly in AI technical safety—will often depend on your ability to be at the forefront of a complex new technology.

Getting into the field

Here are a few pointers for those looking to make their first steps in AI Safety and Governance:

• Fellowships, internships, and boot camps are available for people exploring careers in both AI technical safety and governance. These might be helpful both for developing the required skills for these careers, as well as testing if you’d be a good fit. We’ll highlight some particularly good opportunities at the end of the article, but this list is also a great place to start looking.
• Relevant university degrees – Many people in this path have advanced degrees, and often PhDs, in related subjects like machine learning and computer science, especially technical research leads. On the governance side, relevant subjects may also include law, international relations, or strategic studies. Postgraduate qualifications seem like a very common background for people across AI safety and governance, and are essential for academic roles.
• Prior work experience – For roles in technical safety a very common route is to first work as a software engineer, especially in roles that involve some engagement with machine learning. 
• Independent research – A few people manage to work in AI safety and governance as independent researchers. Various sources of funding are available for potentially promising individuals to potentially pursue independent research, or to train to switch careers into AI safety and governance. This list of funders is a good (though non-exhaustive) place to start for funding in technical AI safety, and there’s a similar list in this ‘starter pack’ for AI safety. Having said this, we’d caution that succeeding independent research can be difficult, not least because it’s harder to access good feedback.