Articial
Intelligence
Index Report
2024
Articial Intelligence
Index Report 2024
2
Introduction to the
AI Index Report 2024
Welcome to the seventh edition of the AI Index report. The 2024 Index is our most comprehensive to date and
arrives at an important moment when AI’s inuence on society has never been more pronounced. This year,
we have broadened our scope to more extensively cover essential trends such as technical advancements
in AI, public perceptions of the technology, and the geopolitical dynamics surrounding its development.
Featuring more original data than ever before, this edition introduces new estimates on AI training costs,
detailed analyses of the responsible AI landscape, and an entirely new chapter dedicated to AI’s impact on
science and medicine.
The AI Index report tracks, collates, distills, and visualizes data related to articial intelligence (AI). Our
mission is to provide unbiased, rigorously vetted, broadly sourced data in order for policymakers, researchers,
executives, journalists, and the general public to develop a more thorough and nuanced understanding of the
complex eld of AI.
The AI Index is recognized globally as one of the most credible and authoritative sources for data and insights
on articial intelligence. Previous editions have been cited in major newspapers, including the The New York
Times, Bloomberg, and The Guardian, have amassed hundreds of academic citations, and been referenced
by high-level policymakers in the United States, the United Kingdom, and the European Union, among
other places. This year’s edition surpasses all previous ones in size, scale, and scope, reecting the growing
signicance that AI is coming to hold in all of our lives.
Articial Intelligence
Index Report 2024
3
Message From
the Co-directors
A decade ago, the best AI systems in the world were unable to classify objects in images at a human level. AI
struggled with language comprehension and could not solve math problems. Today, AI systems routinely exceed
human performance on standard benchmarks.
Progress accelerated in 2023. New state-of-the-art systems like GPT-4, Gemini, and Claude 3 are impressively
multimodal: They can generate uent text in dozens of languages, process audio, and even explain memes. As AI
has improved, it has increasingly forced its way into our lives. Companies are racing to build AI-based products,
and AI is increasingly being used by the general public. But current AI technology still has signicant problems. It
cannot reliably deal with facts, perform complex reasoning, or explain its conclusions.
AI faces two interrelated futures. First, technology continues to improve and is increasingly used, having major
consequences for productivity and employment. It can be put to both good and bad uses. In the second future,
the adoption of AI is constrained by the limitations of the technology. Regardless of which future unfolds,
governments are increasingly concerned. They are stepping in to encourage the upside, such as funding university
R&D and incentivizing private investment. Governments are also aiming to manage the potential downsides, such
as impacts on employment, privacy concerns, misinformation, and intellectual property rights.
As AI rapidly evolves, the AI Index aims to help the AI community, policymakers, business leaders, journalists, and
the general public navigate this complex landscape. It provides ongoing, objective snapshots tracking several
key areas: technical progress in AI capabilities, the community and investments driving AI development and
deployment, public opinion on current and potential future impacts, and policy measures taken to stimulate AI
innovation while managing its risks and challenges. By comprehensively monitoring the AI ecosystem, the Index
serves as an important resource for understanding this transformative technological force.
On the technical front, this year’s AI Index reports that the number of new large language models released
worldwide in 2023 doubled over the previous year. Two-thirds were open-source, but the highest-performing
models came from industry players with closed systems. Gemini Ultra became the rst LLM to reach human-
level performance on the Massive Multitask Language Understanding (MMLU) benchmark; performance on the
benchmark has improved by 15 percentage points since last year. Additionally, GPT-4 achieved an impressive 0.96
mean win rate score on the comprehensive Holistic Evaluation of Language Models (HELM) benchmark, which
includes MMLU among other evaluations.
Articial Intelligence
Index Report 2024
4
Although global private investment in AI decreased for the second consecutive year, investment in generative
AI skyrocketed. More Fortune 500 earnings calls mentioned AI than ever before, and new studies show that AI
tangibly boosts worker productivity. On the policymaking front, global mentions of AI in legislative proceedings
have never been higher. U.S. regulators passed more AI-related regulations in 2023 than ever before. Still, many
expressed concerns about AI’s ability to generate deepfakes and impact elections. The public became more
aware of AI, and studies suggest that they responded with nervousness.
Ray Perrault and Jack Clark
Co-directors, AI Index
Message From the
Co-directors (cont’d)
Articial Intelligence
Index Report 2024
5
Top 10 Takeaways
1. AI beats humans on some tasks, but not on all. AI has surpassed human performance on several
benchmarks, including some in image classication, visual reasoning, and English understanding. Yet it trails
behind on more complex tasks like competition-level mathematics, visual commonsense reasoning and planning.
2. Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable
machine learning models, while academia contributed only 15. There were also 21 notable models resulting from
industry-academia collaborations in 2023, a new high.
3. Frontier models get way more expensive. According to AI Index estimates, the training costs
of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an
estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.
4. The United States leads China, the EU, and the U.K. as the leading source of top AI
models.
In 2023, 61 notable AI models originated from U.S.-based institutions, far outpacing the European
Union’s 21 and China’s 15.
5. Robust and standardized evaluations for LLM responsibility are seriously lacking.
New research from the AI Index reveals a signicant lack of standardization in responsible AI reporting.
Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against dierent
responsible AI benchmarks. This practice complicates eorts to systematically compare the risks and
limitations of top AI models.
6. Generative AI investment skyrockets. Despite a decline in overall AI private investment last
year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in
the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inection, reported substantial
fundraising rounds.
7. The data is in: AI makes workers more productive and leads to higher quality work. In
2023, several studies assessed AI’s impact on labor, suggesting that AI enables workers to complete tasks more
quickly and to improve the quality of their output. These studies also demonstrated AI’s potential to bridge
the skill gap between low- and high-skilled workers. Still, other studies caution that using AI without proper
oversight can lead to diminished performance.
5
