AI Incident Monitoring: A Brief Analysis

People are making important decisions about artificial intelligence every day, ranging from whether and how to use it to what policies should be passed regarding its development, without substantial awareness of the possibilities and risks. This is a significant problem, acknowledged by Founders Pledge’s Tom Barnes in his report Navigating Risks from Advanced Artificial Intelligence (page 94). Given limited awareness, one research area I believe could lead to effective catastrophic risk mitigation is artificial intelligence incident monitoring. Incident monitoring exclusively deals with the risks associated with AI by cataloging times where intelligent systems have caused problems in the real world. Other sectors have been doing incident monitoring for a long time, such as the FAA with air transport services. Whether it's the airline industry or artificial intelligence, the goal here is simple: to capture, organize, and provide access to information about unwanted outcomes so that humanity can learn from them and mitigate future harm. Open access to accurate information is essential for AI research, journalism, and lawmaking, among other fields. Constituents across these areas benefit from AI incident databases for tasks such as risk trend analysis, reporting that shapes public opinion, and making sound policy decisions.

Let’s break down AI incident monitoring into three areas of activity, which can help clarify where and how philanthropy might have an impact. We can begin with INPUTS, the information drawn into a monitoring system. In order to be effective, a repository should comprehensively capture relevant episodes. If a repository is only capturing incidents that make it into mainstream news, for example, the database would likely be missing critical information. The organization behind this incident monitoring must be equipped with the resources necessary to identify incidents and track down relevant information (staff, connections, legal permissions/protections, etc.). One way they might do this is an proactive approach, wherein the organization seeks firsthand incident reporting from AI users by regularly checking if they have had relevant experiences such as chatbot-based confabulations/hallucinations.

The third entry point is OUTPUTS and OUTREACH, effectively how the repository connects to the world. The repository must make the outputs accessible and appealing to those who wish to use it. This may sound redundant with operations, but outputs rather deals with the way that this information is packaged. One model that current AI incident repositories are using is a weekly newsletter with top stories in order to boost engagement. This is a good start but there may be opportunities to go further, such as putting together a more comprehensive but less frequent report on artificial intelligence incidents and safety (something like Stanford HAI’s annual AI Index Report). Monitoring organizations must also raise awareness with relevant potential users and organizations who could benefit from the data. Regarding the field of AI research, for example, monitoring and repository organizations might attend or host events such as conferences in order to make themselves better known. This “outputs and outreach” work requires thoughtful leadership and further capital investment.

A second area of activity concerns OPERATIONS, what an incident monitoring organization does with the information it collects. On an internal level, this deals with the ease of inputting new incidents and organizing them with existing ones for repository employees and contributors. On a public level, this deals with the approachability and navigability of the repository’s data. Both are important to the repository working as it should, since people are more effectively and efficiently able to extract value if UI/UX is well optimized. Successful operations management requires the monitoring organization to have the capital and ability to do this work internally or to outsource it to others.

In sum, several organizations are already operating in this space. An initial look suggests that one of them, AIID, may have meaningful potential for increasing its impact with the help of additional philanthropy.

There are three reasons that this is an important issue to address: scale, tractability, and neglectedness (as adapted by Founder’s Pledge from William MacAskill, p. 307). First is scale. There is an extremely wide range of AI risk awareness and a concerning number of AI decision makers are already under-informed. This is despite the fact that AI itself seems poised to touch nearly everyone’s life, with some non-trivial risk of catastrophic outcomes. AI incident monitoring is important because being more aware of the past makes people more resilient to face similar challenges in the future. A repository would be particularly useful to individuals and organizations researching AI, influencing public opinion, and shaping our governmental policies.

Second is tractability. The three areas of activity described above (inputs, operations, and outputs/outreach) are potential entry points where minimal capital injection could make a meaningful difference in AI risk mitigation.

Third is neglectedness. AI safety is already deeply neglected in comparison to AI research, making up just 3% of papers (Center for AI Safety). AI incident monitoring is even further neglected. Of the four organizations mentioned above, two have limited resources. The other two are well-resourced, yet both have critical shortcomings that suggest they aren’t a big priority for their parent organizations. This leads us to the second aspect of neglectedness, assessing it from an institutional point of view. Current organizational methodology is sporadic: one repository might be over-reliant on news articles and miss some inputs, another might organize their data in a confusing way that repels users, a third might fail to reach out to different audiences and grow their user base. Ultimately this work may be best done by a single international eminent authority for a few key reasons. First, consolidation would avoid redundancy and confusion with potential users who currently don’t know which incident database is best for their purposes. Second, funding by a broad set of international sources would minimize the risk of bias. Third, AI incidents are currently occurring at a rate where decent monitoring is currently manageable by a handful of people per organization, but this is almost certainly going to intensify in the near future (OECD reports their rate of incidents cataloged increasing dramatically over the last two years, visualized in this chart).

Source: OECD

This initial analysis suggests promise in the intervention of AI incident monitoring. A closer examination may reveal whether there is value in supporting one of the current players (such as AIID) or in looking for opportunities to align, coordinate, and perhaps consolidate organizations to face the challenges and opportunities that lie ahead.