5 Scale of Ambition
If there was one person who could be credited with first establishing OpenAI’s scaling ethos, it was its cofounder Ilya Sutskever. Sutskever had long had a paramount belief in deep learning, one that began soon after he showed up unannounced one day, only seventeen years old, at Geoffrey Hinton’s office. At the time, Sutskever was still an undergraduate studying math at the University of Toronto and working the french fry station at a local joint to pay the bills. He knocked urgently on Hinton’s door and declared that he was eager to join the professor’s lab. Hinton told him to schedule a meeting. “Okay,” Sutskever said, unbudging. “How about now?”
Sutskever absorbed the principles of connectionism quickly. He stunned Hinton with his intuitive grasp of research problems and uncanny ability to identify elegant and effective solutions. Sutskever also brought his own dramatic flair to the research. At times he grew so excited by new ideas, he did handstand pushups in the middle of his shared apartment. He had a penchant for making unflinching, categorical pronouncements. “One doesn’t bet against deep learning,” he would say. “Success is guaranteed.”
It was this level of instinct, combined with Alex Krizhevsky’s programming abilities, that Hinton credits for producing the 2012 breakout results on ImageNet. At the time, because deep learning was already demonstrating its potential in speech recognition, Hinton didn’t think much about the significance of applying it to computer vision. Sutskever pushed for the idea. To Ilya, “it was obvious that it was going to work, and it was obvious that would be a big deal,” Hinton says. “He saw that very clearly, and he was right.”
Sutskever brought his die-hard belief in deep learning to OpenAI at a time when the field’s confidence in the paradigm was just beginning to falter and critics like Gary Marcus were pushing for new thinking. Sutskever did not falter. His faith rested on the simple hypothesis that underpinned connectionism: that the artificial nodes in a neural network were sufficient approximators of the real neurons in a biological brain. Each took in inputs and transformed them to produce an output; it was enough of a similarity, such thinking believed, to assume that nodes, just like neurons, could be used to construct highly complex information-processing systems. As OpenAI’s founding research director and a widely respected AI visionary, Sutskever had full rein of the lab’s direction. He had won the equivalent of a scientific lottery. He had little competition among his peers and an abundance of resources to advance his ideas. “Anything non–deep learning wasn’t even remotely considered,” recalls Pieter Abbeel, the UC Berkeley professor, of the lab’s early years.
Just as firm as Sutskever’s belief in deep learning was his view on scaling it. It was Sutskever who held the extreme position for the time that further advancements in AI didn’t need the invention of more complex neural networks or new innovative techniques. The intelligence of different species was correlated with the size of their biological brains, he’d say. Thus, if nodes were like neurons, he argued, advancements in digital intelligence should emerge by scaling simple neural networks to have more and more nodes.
Like a professor running a lab, Sutskever advised others based on these ideas for which projects were most worth pursuing. Many scientists joined OpenAI to seek his mentorship and guidance. “Ilya can see ten years into the future,” an OpenAI researcher says, echoing others who have worked with Sutskever. “He’s like a philosopher,” another says. “If you give him a bunch of ideas, he’ll tell you which ideas are philosophically right.”
Sutskever didn’t often program himself. At times his hands-off approach to technical work bothered some employees. One engineer admitted to thinking at first that he was largely useless; he seemed only to march around the office and pop into meetings repeating the same message: scale, scale, scale! But the person later came to appreciate Sutskever’s conviction in rallying people around a single focus: one that would ultimately allow OpenAI—then an underdog—to beat Google and DeepMind at their own game.
It wasn’t that Sutskever was particularly persuasive. If Altman was the politician, Sutskever was the opposite. He never minced his words or massaged his language to potentially land better with his audience. He simply delivered his opinions with a raw sincerity and outrageous confidence that people either resonated with and found inspiring or did not. After OpenAI reversed course from openly sharing its research, Sutskever wouldn’t sugarcoat the reasoning behind the decision. “Flat out, we were wrong,” he said simply to AI reporter James Vincent at The Verge, of the company’s original commitment to transparency. “If you believe, as we do, that at some point, AI—AGI—is going to be extremely, unbelievably potent, then it just does not make sense to open-source.” There were commercial considerations as well, he plainly noted. “GPT-4 is not easy to develop. It took pretty much all of OpenAI working together for a very long time to produce this thing. And there are many many companies who want to do the same thing.”
As OpenAI grew and Sutskever’s profile rose, his lack of a filter would at times turn into a liability. He was no longer speaking to just researchers; his audience had expanded to the general public. But ever the same, he didn’t adapt his messaging. He made statements using his signature bullish confidence, now lacking significant context for any layperson listening. “it may be that today’s large neural networks are slightly conscious,” he tweeted in 2022, even as other researchers warned that such rhetoric could fan popular misunderstandings of the technology. One DeepMind scientist specialized in the study of cognition and consciousness replied in the comments, “…in the same sense that it may be that a large field of wheat is slightly pasta.” The following year, Sutskever would induce panic by proclaiming at a conference that AGI would eventually disappear all jobs. That fall, he would declare, without scientific backing, on X, “In the future…we will have wildly effective and dirt cheap AI therapy,” after an OpenAI leader triggered online controversy for casually comparing talking to ChatGPT with professionally licensed therapy.
What drew people to follow Sutskever was his reputation and his seniority. Many employees at OpenAI were well aware of his earlier contributions to the field; some saw him as something of a prophet. Over time, as OpenAI grew more successful, Sutskever would act more and more like one. At all-hands meetings, he would get up in front of the company, take a deep breath, and walk back and forth for dramatic effect before delivering vague motivational messages. During one virtual meeting in September 2020, his eyes glazed over as he stared in the distance and painted a science fiction–like vision of the future that was possible. Outside in the Bay Area, the sky had turned orange from nearby forest fires. “It was surreal because it already felt like the apocalypse,” a researcher remembers.
Shortly after ChatGPT’s release in late 2022, OpenAI would host a holiday party at the California Academy of Sciences. Sutskever would get up in front of the crowd, wearing an OpenAI shirt and black blazer, to give some short remarks with Brockman. At the end of it, Sutskever, still wiry as ever and now balding, delivered what had become his new mantra. “Feel the AGI,” he said. “Feel the AGI.”
Following Sutskever’s philosophy of scaling simple neural networks, the question in the early days of OpenAI became: Scale which one? Different researchers proposed and tinkered with different options, but none of the neural networks that had gained widespread traction within the field seemed to fit the bill.
In August 2017, that changed with Google’s invention of a new type of neural network known as the Transformer. Transformers excel at picking up long-range patterns. Think back to the limited predictive text capabilities on iPhones in the early days and the memes they spawned for producing babbling, incoherent sentences. These were the product of short-range pattern analysis— the neural network looking at each word only in relation to the words directly around it. Transformers can ingest large volumes of text and consider each word, sentence, and paragraph in a significantly larger context. Google saw the Transformer as a way to improve its search engine and Google Translate as well as its other services based heavily on language processing. Sutskever saw it for something else. Transformers are simple and scalable neural networks, an example of what he was looking for. He began evangelizing them around the office.
Sutskever’s push struck some researchers as odd. “It felt like a wack idea,” remembers Yilun Du, an MIT researcher who started at OpenAI as a fellow around this time. “Transformers felt like a niche architecture.” But Sutskever, who had focused his PhD thesis with Hinton on the predecessor to Transformers, recognized their potential for taking deep learning to the next level. Others at OpenAI were just as excited. A smattering of researchers began testing it out, including Alec Radford, a dropout from Olin College of Engineering in the greater Boston area with brilliant technical abilities. He began hacking away on his laptop, often late into the night, to scale Transformers just a little and observe what happened.
Radford trained Google’s neural network on a dataset of over seven thousand unpublished English-language books ranging from romance to adventure, which he pulled from a dataset that other AI researchers had previously compiled and open-sourced for a different project. While experimenting, he made a fateful decision to change the task that the Transformer had to learn. Instead of translating languages, as Google had been using Transformers for, he switched it to learn text generation by predicting the most probable next word in a sentence. Early on, OpenAI researchers had hypothesized that generative models would be an important step to reaching AGI. The company explained in a blog post in heavily anthropomorphized terms that the situation was akin to a famous quote from theoretical physicist Richard Feynman: “What I cannot create, I do not understand.” Sutskever had a different way of framing it internally: Training a model to generate something convincing would force it to compress data about the world into its essence. “Intelligence is compression,” Sutskever would say, elaborating in a 2016 memo his strong belief that compression was in fact the only thing needed to achieve artificial general intelligence. In more concrete terms, Radford discovered that giving the algorithm the simple goal of producing convincing text through next-word-prediction did indeed make it pick up the nuances and structure of English at a deeper level.
During one of Musk’s visits to the office, Radford demoed early progress on his work. The model was generating poor-quality text, and Musk was wholly unimpressed. At first, Radford felt deflated. But after pursuing it further, he was surprised by the results. The Transformer had improved quickly and performed much better on a range of language processing tasks, such as summarizing or answering questions about a document, than anything else he had tried before.
In 2018, OpenAI released the first version of that model, called Generative Pre-Trained Transformer, later nicknamed GPT-1. The second word in the name —pre-trained—is a technical term within AI research that refers to training a model on a generic pool of data as a prerequisite for it to learn more specific tasks later. GPT-1, in other words, had been trained on a generic pool of English to create a rough approximation of how the language worked. The model could then be “fine-tuned,” or specialized, later by training it on a much more tailored dataset—say, Shakespeare plays to teach it how to generate Shakespeare-esque prose. GPT-1 barely received any attention. But this was only the beginning. Radford had validated the idea enough to continue pursuing it. The next step was more scale.
Radford was given more of the company’s most precious resource: compute. His work dovetailed with a new project Amodei was overseeing in AI safety, in line with what Nick Bostrom’s Superintelligence had suggested. In 2017, one of Amodei’s teams began to explore a new technique for aligning AI systems to human preferences. They started with a toy problem, teaching an AI agent to do backflips in a virtual video game–like environment. The agent was a simulation of a T-shaped stick, with three joints along the shaft. Instead of giving it the objective of learning backflips directly, the team taught the agent by giving it feedback: They hired contractors to watch the agent as it randomly twisted and turned about the environment; periodically, the contractors would then be asked to compare two video clips of the agent’s actions and select which one better resembled a backflip. Around nine hundred comparisons later, the T-shaped stick was successfully bunching up at its joints and flipping over. OpenAI touted the technique in a blog post as a way to get AI models to follow difficult-to-specify directions. The researchers on the team called it “reinforcement learning from human feedback.”
Amodei wanted to move beyond the toy environment, and Radford’s work with GPT-1 made language models seem like a good option. But GPT-1 was too limited. “We want a language model that humans can give feedback on and interact with,” Amodei told me in 2019, where “the language model is strong enough that we can really have a meaningful conversation about human values and preferences.”
Radford and Amodei joined forces. As Radford collected a bigger and more diverse dataset, Amodei and other AI safety researchers trained up progressively larger models. They set their sights on a final model with 1.5 billion parameters, or variables, at the time one of the largest models in the industry. The work further confirmed the utility of Transformers, as well as an idea that another one of Amodei’s teams had begun to develop after their work on OpenAI’s Law. There wasn’t just one empirical law but many. His team called them collectively “scaling laws.”
Where OpenAI’s Law described the pace at which the field had previously expanded its resources to advance AI performance, scaling laws described the relationship between the performance of a deep learning model and three key inputs: the volume of a model’s training data, the amount of compute it was trained on, and the number of its parameters. Previously, AI researchers had generally understood that increasing these inputs somewhat proportionally to one another could also lead to a somewhat proportional improvement in a model’s capabilities. Amodei and his team’s surprising observation was that the relationship between each of these inputs as well as the model’s performance on a specific, measurable task, such as next-word-prediction, could be described by a smooth curve. In other words, it was possible to estimate with high accuracy how much data, how much compute, and how many parameters to use to produce a model with a desired level of performance on a discrete capability tightly correlated with next-word-prediction—say, fluency in text generation. For capabilities less but still somewhat correlated, increasing these inputs should also lead to better performance.
The cluster of models that OpenAI trained leading up to the final 1.5-billion-parameter version illustrated this relationship. Each one fell neatly on a curve of increasing capability. So it was little surprise when the largest one, which they named GPT-2, markedly improved over the juvenile text generation of GPT-1 to produce lengthy and coherent-enough prose to be confused with a human’s. Compared with today’s models, the text was clunky and often descended into gibberish. But for the very first time, it was suddenly possible to automate writing at scale.
What was a darker surprise to the team was the content that GPT-2 was producing with its new coherence. Fed a few words like Hillary Clinton or George Soros, the chattier language model could quickly veer into conspiracy theories. Small amounts of neo-Nazi propaganda swept up in its training data could surface in horrible ways. The model’s unexpected poor behavior disturbed AI safety researchers, who saw it as foreshadowing of the future abuses and risks that could come from more powerful misaligned AI. After GPT-2 generated a tirade against recycling (“Recycling is NOT good for the world. It is bad for the environment, it is bad for our health, and it is bad for our economy.”), one AI safety researcher printed out a copy and posted it, part joke, part warning, above the recycling bins in the office.
In another instance, someone prompted GPT-2 to create a reward scale for small children for finishing homework and doing their chores. When GPT-2 suggested using candy, it once again disturbed some AI safety people who remarked that this was a tactic of pedophiles. A European employee was taken aback by the association. “My mom definitely did this. Sundays in the summer was ice cream if you do your chores,” he remembers. He wondered if the hypersensitivity was somehow an American thing. It was one of many moments that made him question the basic premise of OpenAI’s lofty goals: How could it benefit all of humanity when it lacked meaningful global representation? Even as a European coming from a highly overlapping culture to the US, he often felt alienated by the overwhelming bias in AI safety and other discussions toward American values and American norms.
GPT-2 started a debate within the company. Had OpenAI reached the point when it was time to start withholding research? The charter had accommodated for this possibility. Amodei, who had by then been promoted to director of research, and Jack Clark, who headed policy and worried in his own way about existential and other dangerous risks, took point on deciding a way forward. They ran an internal survey and held several “information hazard” meetings to discuss possible abuses of the technology. If GPT-2 fell into the hands of terrorists, dictators, or clickbait farms, they reasoned, the model could be used for nefarious purposes. And though it didn’t seem existentially risky this time, future models would only grow more powerful, and that likelihood would get higher. It was better to set a precedent for withholding research early. OpenAI, they decided, should not release the full version.
Jack Clark, a former journalist, had been the director of OpenAI’s strategy and communications before transitioning fully in late 2018 to cultivating its budding policy presence. He took regular trips to Washington and relished being the go-to AI guy for policymakers. He’d tell them, “I’m like AI Wikipedia,” and would introduce his “bias,” as he called it, coming from OpenAI: “We want a stable policymaking environment for advanced tech that operates over multiple political administrations because the mission we have is not going to get done in a presidential cycle.”
After recounting this to me, he added, “We’ve been very lucky that policymakers give us quite a lot of time, because I think it’s clear that basically for stuff to go well, we just want them to have more information, and we also want them to have more means to generate their own information.”
Clark began a media offensive in February 2019, broadcasting widely to various publications that OpenAI had created a dangerous technology, and therefore was not releasing it. Instead it would release only a smaller version, with 8 percent of the full-fledged model’s parameters, to give the public a taste of its capabilities. He, Amodei, and several others coauthored a blog post with examples of GPT-2’s outputs to illustrate its full potential. “It’s very clear that if this technology matures—and I’d give it one or two years—it could be used for disinformation or propaganda,” he said to my then colleague at MIT Technology Review Will Knight. Clark sidestepped the fact that OpenAI was the one leading the push to mature the technology on that timeline. “We’re trying to get ahead of this,” he said.
OpenAI’s move sparked intense blowback from external researchers, who adhered strictly to the idea that open science was the bedrock of the field. Any organization that didn’t participate should be viewed suspiciously. More so if they were publicly boasting about the decision. Many also viewed OpenAI’s alarmism about what was essentially powerful auto-complete software as poorly calibrated and ridiculous. GPT-2 was not nearly advanced enough to be a serious threat; and if it were, why tell everyone about it and then preclude it from public scrutiny? The whole thing felt disingenuous and like a self-aggrandizing publicity stunt. At Stanford, after Radford gave a talk about GPT-2, a well-established natural language processing professor would raise his hand to ask the last question. “So, is it dangerous?” he taunted. The room burst out in laughter. “Alec looked so sad,” remembers a Stanford researcher in the room. “Stanford had so much contempt for OpenAI.”
Within OpenAI, many researchers also chafed against Amodei and Clark’s decision. For those who didn’t share the pair’s views on catastrophic risks, both their ruling and the subsequent media circus felt somewhat baffling. Even for those who did, some still questioned the soundness of the pair’s judgment. “It was a mistake to make such a big deal out of it,” one AI safety researcher told me. “It felt like crying wolf.”
In the immediate aftermath of the blowback, Clark paced up and down the office with manic energy on call after call, working to regain control of the situation. He brushed off the controversy. “We’re breaking with norms, and that creates a lot of different views,” he later told me during my office visit. Sooner or later all organizations conducting cutting-edge AI research would have to be more selective about what to publish, he said. OpenAI was taking the lead in trialing what that process could look like to not be caught flat-footed. “If we’re right, and it is possible to build AGI,” Clark said, “we sure as shit need really good information-hazard procedures.”
And where researchers may not have liked OpenAI’s maneuver, policymakers did, he added. Many DC types viewed the open culture in AI research as threatening. OpenAI’s willingness to go against the grain had gained it more trust in Washington.
But behind the scenes, the leadership team also understood that the animosity from the research community wasn’t viable in the long run. The lab was struggling with compounding reputational challenges. What with its wild claims about AGI, over-the-top approach to GPT-2 and other marketing, and now, in early 2019, its newly announced Frankenstein structure, it was being criticized left and right, and being viewed with more and more skepticism from top researchers in the field. Combined with the fact that its equity didn’t yet mean anything, it was still having trouble hiring and retaining talent. Employees wondered whether external candidates were securing offers from OpenAI simply to use as leverage for negotiating higher offers with Google or DeepMind. OpenAI needed to find a way to legitimize itself as a research organization.
This was frequently discussed at lunches and in company meetings, as well as in an internal document called “Research Community Outreach Brainstorming.” Under a section titled “Strategy,” it read, “Explicitly treat the ML community as a comms stakeholder,” using the abbreviation for machine learning. “Change our tone and external messaging such that we only antagonize them when we intentionally choose to.” The document also acknowledged how a poor research reputation would ultimately undermine OpenAI’s influence in Washington. “In order to have government-level policy influence, we need to be viewed as the most trusted source on ML research and AGI,” it read under “Policy.” “Widespread support and backing from the research community is not only necessary to gain such a reputation, but will amplify our message.”
Clark’s team formulated a new plan: a staged release. Instead of withholding GPT-2 permanently, OpenAI would publish the progressively larger models that it had developed at staggered intervals and then, if all went well, release the full 1.5-billion-parameter version. This would allow OpenAI and others to gradually observe and address any emerging consequences, the team said, as well as give the lab time to partner with other organizations to research the risks between stages.
Clark emphasized to his team the importance of building an ecosystem through those partnerships. Working with high-profile institutions would help foster more cooperation between industry and academia for addressing AI safety risks. It would also get broader buy-in into OpenAI’s efforts to shift research release norms and simultaneously help burnish the lab’s reputation. His team reached out to AI and security researchers at a select few organizations and gave them early access to the full version of GPT-2 to test its potential for harmful applications. Clark instructed his team to get “the strongest endorsement” they could from each researcher’s organization so OpenAI could name not just the individuals but also their institutions as partners. The team then prepared a white paper touting its release strategy and highlighting those partnerships. They sought to frame OpenAI as a leader by listing examples of organizations that had also deviated from immediately releasing their research after GPT-2.
The work paid off. Before long, it had seeded conversations across industry groups and policy think tanks about withholding research as a responsible approach to managing AI safety risks. In late 2020, Clark would be among the people who would break off from OpenAI with the Amodei siblings to cofound Anthropic. Until that time, his work at OpenAI would help establish its influence and lay the groundwork for its sprawling policy ambitions.
OpenAI began to keep a road map to systemize its research. Amodei treated it like an investor: He called it having “a portfolio of bets.” He and other researchers kept tabs on different ideas within the field, born out of different philosophies about how to achieve artificial general intelligence, and advanced each one through small-scale experimentation. Those that seemed promising, OpenAI would continue. Those that didn’t pan out, it would abandon.
The project to win the Dota 2 video game championship was one area that Amodei believed no longer had much utility. The Dota 2 team had beat its opponents and achieved its goal in April and helped secure Microsoft’s investment. It had also helped some people gain new confidence in the company’s scaling strategy. The project, as he saw it, had run its course. The Dota 2 team disbanded.
Where Amodei did see continued promise was in GPT-2. It represented a bet known in the field as the “pure language” hypothesis. Language, the theory goes, is the primary medium through which humans communicate, meaning all of the world’s knowledge must at some point be documented in text. It follows then that AGI should be able to emerge from training an algorithm on massive amounts of language and nothing else. This idea is in contrast to the “grounding” hypothesis, which asserts that the physical world and our ability as humans to perceive and interact with it is just as crucial an ingredient to our intelligence. AGI would then only be able to emerge from the combination of language and perception, like computer-vision, as well as interaction, such as through a physical or virtual agent taking actions in the real world.
In company documents, researchers weighed the merits of the different approaches, with AI safety staff at one point debating the virtues of the “pure language” hypothesis by drawing repugnant analogies to people with disabilities.
The discussions revealed how quickly measures of intelligence could veer into disturbing assessments of which groups of people had superior or inferior intelligence.
“Language of some form is the difference between a feral human and human in society. Example, Helen Keller,” read the document under the heading “Some initial arguments for the centrality of language.” In the margins, AI safety researchers continued their arguments for and against “pure language” through threaded comments.
“Also blind people are about as capable as sighted people,” wrote one researcher, as evidence that “grounding” through vision seemed unnecessary.
“Blind people seem at a significant economic disadvantage,” replied another, citing statistics from the National Federation of the Blind that over 70 percent of vision-impaired adults did not work full-time.
“Blind people are still way more capable than chimpanzees,” replied a third. “There exist very impressive blind people.”
Many at OpenAI had been pure language skeptics, but GPT-2 made them reconsider. Training the model to predict the next word with more and more accuracy had gone quite far in advancing the model’s performance on other seemingly loosely related language processing tasks. It seemed possible, even plausible, that a GPT model could develop a broader set of capabilities by continuing down this path: pushing its training and improving the accuracy of its next-word-prediction still further. Amodei began viewing scaling language models as—though likely not the only thing necessary to reach AGI—perhaps the fastest path toward it. It didn’t help that the robotics team was constantly running into hardware issues with its robotic hand, which made for the worst combination: costly yet slow progress.
But there was a problem: If OpenAI continued to scale up language models, it could exacerbate the possible dangers it had warned about with GPT-2. Amodei argued to the rest of the company—and Altman agreed—that this did not mean it should shy away from the task. The conclusion was in fact the opposite: OpenAI should scale its language model as fast as possible, Amodei said, but not immediately release it. GPT-2 had demonstrated how easy it would be for other actors to obtain more powerful AI capabilities; in fact, two graduate students had already created an open-source version of GPT-2 before OpenAI had released its own full version. It was only a matter of time before other people would start scaling up language models further. That meant the best way to ensure beneficial AGI was for OpenAI to leap ahead and, with the internal lead time, figure out how to make its scaled model safer. Once it was time to reveal the model, its extra polish and refinement would help establish AI safety norms, in the same way the initial withholding of GPT-2 shifted norms for releasing research.
With a version of GPT-2 now out in the world, there was also evidence that the dangers of pure language models weren’t all that bad. As far as OpenAI knew, it hadn’t been used in coordinated mass disinformation campaigns—and such campaigns were certainly better than the potential existential risks of AGI.
“Obviously misuse is not good,” Amodei told me. “But a language model is a lot less powerful than an AGI. I’m very worried about language models being weaponized for disinformation and this sort—that is very scary to me—but at the same time, it’s a relatively singular and clear and defined concern.”
From Amodei’s view, in other words, scaling GPT-2 was not only potentially the fastest path to advance to AGI but also one whose possible risks along the way would be relatively contained to those he viewed as manageable —mis- and disinformation, as opposed to catastrophe. It would give OpenAI a safer testing ground to experiment with a powerful, but not so powerful, AI system, and work out various kinks, including with releasing it.
“What is AGI? What does AGI look like?” Amodei said. “Well, you know, we’re in the awkward position of, we don’t know what it looks like. We don’t know when it’s going to happen. So we look for things that aren’t AGI but that present at least some of the opportunities and difficulties of AGI. And the hope is if we can handle those things well, then we’re kind of, like, ready for the bigger leagues.”
It was a logic that worked under a specific assumption: that AGI, despite being amorphous and unknowable, was also inevitable. OpenAI would repeatedly justify its behaviors against variations of the same argument for years after. Under the specter of AGI’s unstoppable arrival, the company needed to keep developing more and more powerful models to prepare itself and to prepare society. Even if those models carried with them their own risks, the experience they offered to prevent or face possible AI apocalypse made those risks bearable.
As ChatGPT swept the world by storm in early 2023, a Chinese AI researcher would share with me a clear-eyed analysis that unraveled OpenAI’s inevitability argument. What OpenAI did never could have happened anywhere but Silicon Valley, he said. In China, which rivals the US in AI talent, no team of researchers and engineers, no matter how impressive, would get $1 billion, let alone ten times more, to develop a massively expensive technology without an articulated vision of exactly what it would look like and what it would be good for. Only after ChatGPT’s release did Chinese companies and investors begin funding the development of gargantuan models with gusto, having now seen enough evidence that they could recoup their investments through commercial applications.
Through the course of my reporting, I would come to conclude something even more startling. Not even in Silicon Valley did other companies and investors move until after ChatGPT to funnel unqualified sums into scaling. That included Google and DeepMind, OpenAI’s original rival. It was specifically OpenAI, with its billionaire origins, unique ideological bent, and Altman’s singular drive, network, and fundraising talent, that created a ripe combination for its particular vision to emerge and take over. “I get the sense that Sam is the most ambitious person on the planet,” a former employee says. In other words, everything OpenAI did was the opposite of inevitable; the explosive global costs of its massive deep learning models, and the perilous race it sparked across the industry to scale such models to planetary limits, could only have ever arisen from the one place it actually did.
For the Gates Demo in April 2019, OpenAI had already scaled up GPT-2 into something modestly larger. But Amodei wasn’t interested in a modest expansion. If the goal was to increase OpenAI’s lead time, GPT-3 needed to be as big as possible. Microsoft was about to deliver a new supercomputer to OpenAI as part of its investment, with ten thousand Nvidia V100s, what were then the world’s most powerful GPUs for training deep learning models. (The V was for Italian chemist and physicist Alessandro Volta.) Amodei wanted to use all of those chips, all at once, to create the new large language model.
The idea seemed to many nothing short of absurdity. Before then, models were already considered large-scale if trained on a few dozen chips. In top academic labs at MIT and Stanford, PhD students considered it a luxury to have ten chips. In universities outside the US, such as in India, students were lucky to share a single chip with multiple peers, making do with a fraction of a GPU for their research.
Many OpenAI researchers were skeptical that Amodei’s idea would even work. Some also argued that a more gradual scaling approach would be more measured, scientific, and predictable. But Amodei was adamant about his proposal and had the backing of other executives. Sutskever was keen to play out his hypothesis of scaling Transformers; Brockman wanted to continue raising the company’s profile; Altman was pushing to take the biggest swing possible. Soon after, Amodei was promoted to a VP of research.
Behind the scenes, Altman was also attuned to another factor: Microsoft’s $1 billion investment came with $1 billion expectations; OpenAI was on the clock to deliver something that would justify the expense. Where Amodei saw a larger language model as a necessary prerequisite for AI safety research, Altman saw its potential for fulfilling OpenAI’s promise to Microsoft.
In the coming months, Amodei and Altman would clash over how and when to release GPT-3; Altman would win out, pushing the model into the world on an accelerated timeline. Years before ChatGPT, these two decisions—the one to explode GPT-3’s size and the one to quickly release it—would change the course of AI development. It would set off a rapid acceleration of AI advancement, sparking fierce competition between companies and countries. It would fuel an unprecedented expansion of surveillance capitalism and labor exploitation. It would, by virtue of the sheer resources required, consolidate the development of the technology to a degree never seen before, locking out the rest of the world from participating. It would accelerate the vicious cycle of universities, unable to compete, losing PhD students and professors to industry, atrophying independent academic research, and spelling the beginning of the end of accountability. It would amplify the environmental impacts of AI to an extent that, in the absence of transparency or regulation, neither external experts nor governments have been able to fully tabulate to this day.
But all this was yet to pass. In the fall of 2019, Amodei assembled a team, called Nest, of mostly other AI safety researchers, intent on keeping careful control of GPT-3’s development within the company. With that, the team began its aggressive push to scale.
GPT-3 was effectively the same model as GPT-2, fed massively more data and compute to be so much bigger that the outcome would appear to many as beyond a difference of degree to a difference in kind. But using ten thousand chips posed new problems. There was always a small probability that any chip might crash in the middle of training, the same way a laptop might crash when there are too many windows open. If one chip crashed, everything did, meaning training would need to start all over. The probability of a single chip crashing compounded significantly across ten thousand GPUs. Such an error would be enormously costly—in both money and time—when the Nest team expected training to take several months at a minimum.
To fix the problem, the team needed a way to make sure model training could restart exactly where it left off after any disruptions. It also needed to determine a strategy for how to spread the training across all ten thousand chips, a process known as sharding: Was it better, say, to chop up the model into tens, hundreds, or thousands of pieces, with each piece training on separate clusters of GPUs before being merged?
Then there was a challenge with the data. To get the best performance, the size of the dataset needed to grow proportionally with the number of parameters and the amount of compute. If there were too many parameters and not enough data, the model could start regurgitating word for word the lines in its training data, effectively rendering it useless. For GPT-2, Radford had been selective about what made it into the data. He scraped the text from articles and websites that had been shared on Reddit and received at least three upvotes on the platform. This had produced a forty-gigabyte trove of some eight million documents, which he named WebText.
That wasn’t nearly enough for GPT-3. So Nest expanded the data by adding an even broader scrape of links shared on Reddit as well as a scrape of English-language Wikipedia and a mysterious dataset called Books2, details of which OpenAI has never disclosed, but which two people with knowledge of the dataset told me contained published books ripped from Library Genesis, an online shadow repository of torrented books and scholarly articles. In 2023, the Authors Guild and seventeen authors, including George R. R. Martin and Jodi Picoult, would sue OpenAI and Microsoft alleging mass copyright infringement. OpenAI would respond in March 2024 by saying it had deleted those datasets and had stopped using them for training after GPT-3.5, which by that time had already been deprecated.
This was still not enough data. So Nest turned finally to a publicly available dataset known as Common Crawl, a sprawling data dump with petabytes, or millions of gigabytes, of text, regularly scraped from all over the web—a source Radford had purposely avoided because it was such poor quality. In an effort to tame the trash in the data, the Nest team trained a machine-learning model to find the samples within Common Crawl that looked most like articles on Wikipedia. If it looked like Wikipedia, the idea was, it would be more likely to match Wikipedia quality. They also included some samples in languages other than English, though they ultimately accounted for only 7 percent of the data. Still, when training the model, the researchers weighted the filtered Common Crawl data as the lowest priority. GPT-2, in other words, had been peak data quality; it declined from there.
When it came time to assemble the data for GPT-4, released two years later, the pressure for quantity eroded quality even further. The filter was removed from the Common Crawl data and most of it poured in. Through its partnership with Microsoft, OpenAI also received a full download of GitHub, the Microsoft-owned online code repository. When this still wasn’t enough, OpenAI employees also gathered whatever they could find on the internet, scraping links shared on Twitter, transcribing YouTube videos, and cobbling together a long tail of other content, including from niche blogs, existing online data dumps, and a text storage site called Pastebin. Anything that didn’t have an explicit warning against scraping was treated as available for the taking.
Within Google, some researchers lamented OpenAI’s willingness to take legal risks to gather data as giving them a major advantage. Google was a lot more conservative about data access and usage and had a rigorous protocol for complying with regulations including Europe’s data privacy law, colloquially known as the GDPR. Google’s commitment to compliance, ironically, gave OpenAI easier access to Google’s data than Google itself. Where OpenAI readily scraped and transcribed videos from Google-owned YouTube, Google researchers had to maneuver through significant internal red tape to abide by YouTube’s restrictive license on its user-uploaded content. OpenAI was unconcerned—or in tech startup terms, “unburdened”—by this compliance. It was a classic mindset in Silicon Valley, where founders and investors espouse the mantra that startups could and should move into legal gray areas (think Airbnb, Uber, or Coinbase) to disrupt and revolutionize industries.
The decision to lower quality barriers—and then effectively drop them altogether—would have sweeping downstream effects on the human labor behind AI systems. For years, the tech industry had relied on poorly paid workers in precarious economic conditions to perform essential data preparation tasks for its AI models, such as categorizing text and labeling images. Soon after GPT-3 normalized the use of giant, poorer quality datasets, the demands for the work shifted from the handling of largely benign content to frequently disturbing content, including for the purposes of content moderation, much like social media before it. Such moderation was necessary to prevent generative AI systems from reproducing the most vile parts of their all-encompassing datasets —descriptions and depictions of violence, sexual abuse, or self-harm—to hundreds of millions of users.
“There’s a big paradigm shift in how you control the output of these models,” says Ryan Kolln, the CEO and managing director of Appen, a platform for connecting Silicon Valley companies with data workers. “In a traditional AI sense, you control the output by constraining the inputs”—the kinds of data filtering that Radford’s team did—“because it only learns from the examples that you are giving it. The challenge with generative AI is the inputs are the entire corpus of humanity. So you need to control the outputs.”
In a 2023 paper, Abeba Birhane and her coauthors would introduce the concept of “hate scaling laws” to critique the premise of training deep learning models on unfiltered data, or what they called “data-swamps.” They analyzed two publicly available image-and-text datasets used to train open-source image generators, LAION-400M and LAION-2B-en, both pulled from Common Crawl, with four hundred million and two billion images, respectively. They showed that the amount of hateful and abusive content scaled with the size of the dataset and exacerbated the discriminatory behaviors of the models trained on them. Models trained on the two billion images, for example, were five times more likely than models trained on the four hundred million images to label Black male faces as criminals. Later that year, a Stanford study analyzing LAION-5B, a dataset with five billion images used to train Stable Diffusion, would discover it contained thousands of images of verified and suspected child sexual abuse.
Among its tactics to control the outputs, OpenAI would hire workers in Kenya for on average less than two dollars an hour to build an automated content-moderation filter, a revelation first reported by Time magazine correspondent Billy Perrigo. It would also employ over a thousand other contractors globally to perform reinforcement learning from human feedback, or RLHF, the technique it had developed to teach an AI agent backflips, on its language models, including prompting the models repeatedly and scoring the answers, in an effort to tame the model as much as possible.
Hito Steyerl, a German artist and filmmaker who produced a documentary on Syrian refugees who perform data work, echoed Birhane’s critique in the observations she shared with me. Psychologically harmful material accumulates when mass surveillance is the basis for data collection, she said. To fix the problem, we have to return to its root: questioning what is really in the data, questioning the whole premise of its wide-scale, indiscriminate seizure.