Can we substantially increase the size of the pretraining corpus in specific domains?
Note from Caspar: I gave my models this task, suggested the two approaches to make Fermi estimates and then did a bunch of pruning of what the models wrote. I didn’t check any of the sources or calculations myself.
Suppose we want language models to reason better in a specific domain, such as AI safety, moral philosophy, or the foundations of decision theory. One important part of training LLMs is pretraining, in which a language model learns from a large text collection (say, all text available on the Internet, scanned books, newspapers, …). It may matter especially in conceptual domains. Answers in these domains are hard to check, so developers get little ground-truth feedback for training, and, at least for now, human experts produce the best available reasoning in them. Pretraining also seems relatively tolerant of wrong and confused material, so the bar for new pretraining data is much lower than for data used in later training stages, in which models learn to imitate or prefer specific answers.
This raises a question: can we substantially increase the amount of text about such a domain that is available for pretraining, for example by paying experts to record their conversations and publishing the transcripts? Below, I estimate how much text such a program could produce and compare it with the existing written corpora of two domains, moral philosophy and AI safety.
In gross text volume, the answer is yes, especially for small domains. I estimate that 500 philosophers recording three hours a week could produce 0.4–0.8 billion transcript tokens (units of text, often words or word fragments) per year. Each year’s output would equal roughly 13–27% of my central estimate for the English-language academic literature of moral philosophy, about 3 billion tokens. The written corpus of AI safety research is much smaller, roughly 0.1–0.3 billion tokens by my estimate, so a program in which 100 researchers record three hours a week would add the equivalent of roughly 30–160% of it each year. Relative to a whole training run, these additions are tiny: 0.4–0.8 billion tokens, sampled once each, would be about 0.003–0.005% of the 15 trillion used to train Meta’s Llama 3 (Grattafiori et al. 2024).
How much text a recording program could produce
I assume a conservative 150 spoken words per minute; Yuan, Liberman, and Cieri 2006 measured averages of 193 and 214 words per minute in two telephone-conversation corpora. With OpenAI’s rule of thumb of 100 tokens per 75 English words, one audio hour yields about 12,000 tokens.
| Program | Assumptions | Gross transcript tokens per year | Estimated annual cost |
|---|---|---|---|
| Small | 100 experts, 3 hours per week, 45 weeks | 81–162 million | USD 0.7–1.8 million |
| Medium | 500 experts, 3 hours per week, 45 weeks | 0.4–0.8 billion | USD 3.5–9 million |
Lower ends assume participants talk in pairs, so two participant-hours yield one audio hour; upper ends assume each participant talks with an unpaid outside interlocutor, whose speech the count includes. Costs cover participant pay of USD 40–100 per hour, consent and rights handling, and coordination. I assume automatic transcription only; speech recognition costs at most about USD 1 per audio hour at OpenAI’s prices but makes errors, especially on names, technical terms, and overlapping speech. Manual correction would add roughly USD 15–60 per audio hour; keeping the audio instead allows re-transcription as the technology improves. Recruiting 500 participants for three hours a week could prove harder than the arithmetic suggests.
Recording existing events avoids paying participants but needs the agreement of everyone recorded. In the United States, recording is generally lawful if participants know about it and continue talking (Reporters Committee for Freedom of the Press 2021). In the EU, recording and publishing what identifiable people say requires a legal basis under the General Data Protection Regulation, and consent is the safest one. Speakers presenting unpublished work may decline, and questioners, especially students and junior faculty, may decline or speak less freely. US universities generally treat class recordings in which identifiable students speak as education records under FERPA, a student-privacy law that requires written consent or de-identification before the university shares them beyond the class (MSU Denver 2022). So recording, for example, colloquia and seminars at universities might be difficult: seminars because of FERPA, and colloquia because every speaker and questioner has to agree.
Gross tokens might overstate the added conceptual content. Edited papers contain more argument per token than speech; conversations repeat published positions and spend time on setup and social coordination. On the other hand, conversations can also contain valuable data that papers omit. Perhaps most importantly, recorded conversations might show (in appropriate chronological/causal order) the intermediate reasoning steps involved in getting to a conclusion.
Example: moral philosophy
The philosophy bibliography PhilPapers listed 3,125,704 entries in September 2026, with about 260,000 category assignments in normative ethics (52,191), metaethics (23,075), and applied ethics (183,109). Entries are bibliographic records, and PhilPapers assigns each paper to one to three leaf categories, so distinct works are fewer. My estimate of the English-language literature assumes roughly 110,000–200,000 distinct articles and chapters of 5,000–10,000 words and 8,000–30,000 relevant books of 60,000–120,000 words. On these assumptions, the literature comes to 1.4–7.5 billion tokens, with a central estimate (the geometric mean) of about 3 billion; normative ethics and metaethics alone come to 0.3–2 billion.
I assume that developers train, or will train, on everything available in principle, including paywalled papers and books, so I compare new text with the whole literature estimated above. Some developers already acquire such text: Informa announced a data-access agreement with Microsoft covering Taylor & Francis content, with an initial fee above USD 10 million (Informa 2024), and a 2025 court order describes Anthropic buying and scanning millions of print books (Bartz v. Anthropic 2025). One year of the medium program, if its participants were moral philosophers, would equal roughly 13–27% of the central estimate.
Example: AI safety
By AI safety I mean research on preventing catastrophic outcomes from advanced AI, including alignment: making AI systems act in line with human values and intentions. The Alignment Forum, a central online forum for alignment research, had received about 2,000 posts and 9,000 comments by 2021 (Ruby and Habryka 2021), and about 4,200 posts by mid-2025, according to a scrape of LessWrong that flags Alignment Forum posts. Assuming about 5,000 posts today, averaging 1,500–3,000 words, and 20,000 comments of 100–200 words, I estimate that the forum now holds 13–25 million tokens. LessWrong, which shares a database with the Alignment Forum and automatically carries all of its posts and comments, also hosts discussion of many other topics. A May 2026 crawl of LessWrong found about 46,000 posts and 906,000 comments, excluding short-form posts, together about 210 million tokens. Adding the AI-related part of LessWrong (perhaps 50–130 million tokens), alignment papers (20–80 million), and blogs and podcast transcripts (40–90 million), and counting cross-posted material once, I estimate an AI safety corpus of roughly 0.1–0.3 billion tokens, 3–10% of my central estimate for moral philosophy. A broader definition that includes machine-learning research on robustness and fairness would give a larger corpus.
The field is small: McAleese 2025 estimates about 600 full-time equivalents working on technical AI safety and 500 on non-technical work such as governance. The small program in the table, with 100 AI safety researchers, would produce 81–162 million tokens a year, several times the Alignment Forum’s total and roughly 30–160% of my estimate for the broader corpus. It would need roughly a tenth of the field, and researchers at AI companies may be hard to recruit at USD 40–100 an hour. Organizations could also release some of the AI safety discussion that already exists in unpublished writing, such as internal documents and chat channels, to the extent confidentiality allows.
Another type of Fermi estimate
Most of what researchers say and write about their work never becomes public. Consider a researcher who publishes two 10,000-word papers a year, 20,000 words in total. University students in one study spoke about 16,000 words a day on average (Mehl et al. 2007), nearly 6 million words a year. If a tenth of the researcher’s speech concerns their research, that alone comes to about 600,000 words a year, 30 times their published output. In one survey, employees of small businesses reported writing about 110 emails a week (Slack 2023). In a public corpus of emails from the energy company Enron, sent emails average about 140 words after automated removal of most quoted and forwarded text (Styler 2011). If the researcher writes 100 such emails a week for 46 weeks, that comes to about 640,000 words a year. If a quarter of this text concerns research, the emails add about 160,000 research-related words. Drafts add more. In one small study that logged how graduate students wrote five computer-science papers over four months, the logged authors deleted about 63,000 words while adding about 119,000, so the deleted text roughly matched the text that remained (Le et al. 2026). I assume the same here: text cut or rewritten during drafting matches the length of the published papers. On these assumptions, the published papers make up about 2.5% of what the researcher says and writes about their research. Recording conversations would make part of the rest available for pretraining.