1. Can an AI Agent Study a Codebase?

Humans can turn reading a textbook and actively studying its material into deep knowledge and even expertise. Why can’t AI agents do the same?

That was the question posed in June 2026 by Machine Studying. Today, we largely make agents more capable by scaling inference-time compute, giving them better access to external information, or training them through task-specific feedback and reinforcement learning environments. But these approaches are quite different from something humans do routinely: study a body of material before knowing exactly how that knowledge will later be tested.

Give a coding agent a repository it has never seen before and ask it a difficult question about the codebase. Its strategy is usually: search for a promising symbol, open a few files, reason over whatever fits in context, and produce an answer. Ask it another question, and much of that process begins again.

That can work. It is also a strange way to imagine expertise.

An expert has a rich internal model of a domain. That knowledge changes how they recognize problems, what they pay attention to, and how efficiently they arrive at good answers. In other words, expertise should shift the entire relationship between performance and the amount of computation required to achieve it.

Machine Studying quantifies this relationship into one number it calls Expertise. It defines Expertise as the weighted area under an agent’s performance versus inference-compute curve:

\[\mathcal{E}(\Sigma; \mathbf{D}) = \int p_{\Sigma,\mathbf{D}}(x)\,w(x)\,dx\]

Here, $p(x)$ is performance at position $x$ on a log-token axis, while $w(x)$ represents how important that inference budget is. Cheaper budgets can be given more weight, so an agent that reaches a good answer efficiently counts as more expert than one that reaches the same answer only after extensive search.

Humans often build that kind of expertise by studying. Before an exam, we study the available corpus, such as the textbook, lecture notes, practice problems, or other preparation material, without knowing the exact questions we will eventually face. We spend time building a mental model of the domain first, then apply that model when the questions arrive.

StudyLoop asks whether a small AI agent can benefit from a similarly ordinary act: studying before it knows the exam questions.

StudyLoop does not attempt to solve expertise outright. Instead, it is a small experimental artifact aimed at understanding what happens when an agent is given the opportunity to study a corpus before evaluation: what strategies help, what fails, what constraints get in the way, and whether studying can produce measurable gains in downstream expertise.

The experiment builds on StudyBench, the benchmark released alongside Machine Studying. As the paper puts it: “StudyBench is our attempt to create a concrete hill for us all to climb toward agents that develop expertise in new domains from nothing but a corpus.”

Think of StudyLoop as one attempt at climbing that hill while paying close attention to what makes the climb difficult, which approaches actually move us upward, and what it might take to scale these methods further.

To make that exploration useful beyond this experiment, StudyLoop is fully open source, including the evaluation rig, study methods, cheatsheet baseline, trajectories, and experimental setup. This allows others to reproduce the results, test their own ideas, and continue exploring this direction.

2. Three Approaches to Studying a Codebase

The StudyBench benchmark includes a subset called Studying-DSPy. This subset includes questions on the popular Python framework for building AI systems. I tested three conditions on the 30 StudyBench questions about the DSPy codebase.

• Base: no precomputed guide (control).
• Cheatsheet: inspected the repository once and produced a conventional guide.
• StudyLoop: followed a longer process: survey the repository, build a concept map, generate eight self-questions, answer them from source, critique weak or unsupported claims, and revise a final guide.

Below you’ll find the full processes behind Cheatsheet and StudyLoop. The guides are reproduced as model-generated experimental artifacts.

Cheatsheet: the procedure and exact frozen guide

Cheatsheet was one source-grounded study session. Gemma was instructed to inspect the repository broadly and produce a concise developer guide covering important architecture, public APIs, execution flows, interactions, and practical pitfalls. It could make at most 20 repository calls, one per turn, but it was explicitly told not to quiz itself, critique an earlier guide, or revise iteratively.

In the final run, Cheatsheet made seven repository calls: one broad file listing followed by six explicit file reads. It read dspy/clients/lm.py, dspy/predict/__init__.py, dspy/predict/react.py, dspy/primitives/module.py, dspy/signatures/signature.py, and dspy/teleprompt/bootstrap.py. It then produced this 849-token guide:

DSPy Developer Guide

This guide provides an overview of the DSPy framework architecture, core components, and execution flows based on the repository structure.

Core Architecture Overview

DSPy is designed as a modular framework for building and optimizing LLM-based programs. It moves away from manual prompt engineering toward a “programming” paradigm where models are treated as modules that can be composed, swapped, and optimized systematically.

1. Signatures (The Interface)

Signatures define the inputs, outputs, and instructions for an LLM task. They act as the contract between different components of a program.

• Definition: You can subclass dspy.Signature to define specific fields using InputField and OutputField.
• Dynamic Creation: Signatures can be created from strings (e.g., "question -> answer") or via the make_signature function.
• Source Reference: dspy/signatures/signature.py

2. Modules (The Building Blocks)

Modules are the primary units of composition in a DSPy program. They inherit from dspy.primitives.module.Module.

• Execution Flow: Every module implements a forward method. When called, it manages the flow of data through various predictors and sub-modules.
• State Management: Modules maintain a history of LM calls and a list of callbacks for monitoring. The ProgramMeta metaclass ensures that these attributes are initialized correctly even if a subclass forgets to call super().__init__().
• Source Reference: dspy/primitives/module.py

3. Predictors (The Logic)

Predictors implement specific reasoning patterns or interaction styles with the LLM. They are typically used as components within a Module.

• Common Patterns:
  • Predict: Basic completion.
  • ChainOfThought: Encourages step-by-step reasoning.
  • ReAct: An agentic pattern where the model reasons and acts (calls tools) iteratively. It manages a trajectory of thoughts, tool calls, and observations.
  • Refine: Iteratively improves an output based on feedback.
• Source Reference: dspy/predict/__init__.py

4. Clients & Providers (The Infrastructure)

DSPy abstracts the underlying LLM providers using a unified client interface.

• LM Class: The primary entry point for interacting with models. It handles parameters like temperature, max_tokens, and num_retries.
• Caching: Supports request caching to reduce costs and latency during optimization rounds.
• Source Reference: dspy/clients/lm.py

Execution Flow & Optimization

1. Program Definition: Define a Signature for the task and compose it into a Module using Predictors.
2. Execution: Call the module with inputs. The module executes its forward logic, invoking predictors which interact with the LM client.
3. Teleprompting (Optimization): Use Teleprompters to optimize the program.
   • BootstrapFewShot: A common optimizer that generates high-quality demonstrations from a small set of labeled examples by “bootstrapping” them through the model and selecting successful traces based on a metric.
   • Source Reference: dspy/teleprompt/bootstrap.py

Practical Pitfalls & Tips

• Signature Polymorphism: Use ensure_signature to handle cases where inputs might be strings or complex objects, ensuring compatibility across different predictors.
• Rollout IDs: When using optimizers like BootstrapFewShot, the framework uses rollout_id and temperature=1.0 to bypass caches and ensure diverse traces are gathered during the bootstrapping phase.
• Tool Handling: In agentic patterns like ReAct, tools are automatically wrapped into a standard format, including a special “finish” tool to signal task completion.

StudyLoop: the procedure and exact frozen guide

StudyLoop divided preparation into six fixed stages. Each later stage received the artifacts produced by the earlier stages:

1. Survey the repository, with a budget of five tool calls.
2. Build a concept map from the survey, without repository tools.
3. Generate exactly eight self-questions from the survey and concept map.
4. Produce source-grounded answers to all eight questions, with a budget of twelve tool calls.
5. Critique the accumulated understanding, using up to three tool calls to check the most consequential doubts.
6. Revise everything into one concise final guide, without repository tools.

The final run used four repository calls during the survey, nine while answering its self-questions, and one during critique: fourteen calls in total. It generated these eight questions:

1. How does the dspy.primitives.Module class use a metaclass to manage its internal state, and what specific attributes are initialized this way?
2. Explain the distinction between a Signature and a Predictor in terms of their responsibilities within a DSPy program.
3. Describe the execution flow that occurs when a predictor is called, specifically how it handles input arguments in relation to the Signature.
4. How does the framework achieve modularity and systematic optimization by treating LLM pipelines similarly to neural network layers?
5. In the context of Module Composition, how is the history attribute propagated through a nested tree of modules during a single forward pass?
6. What are the primary inputs required for the dspy.teleprompt suite, and what is the intended output of this optimization process?
7. How does the framework abstract the differences between various LLM providers (e.g., OpenAI vs. local models) and RAG components like retrievers?
8. Identify a potential complexity or area requiring deeper inspection regarding how dspy.predict.parallel handles multi-threaded execution across different modules.

After answering and critiquing those questions, StudyLoop produced this 762-token guide:

DSPy Developer Guide

Architecture Overview

DSPy treats LLM pipelines as modular, optimizable programs analogous to neural networks. Instead of manual prompt engineering, it uses a structured approach where components (Modules) are composed into trees, and logic is defined by Signatures.

Core Building Blocks

• dspy.primitives.Module: The fundamental base class for all program components. It manages state, execution history, and the forward() pass.
  • Initialization: Uses a metaclass (ProgramMeta) to ensure consistent initialization of history and callbacks. While _base_init explicitly sets these attributes, the metaclass acts as a fail-safe to guarantee their existence even if subclasses fail to call super().__init__().
  • History Management: Unlike simple recursive propagation, DSPy manages execution context using thread-local overrides. The history attribute records LM calls for each module, while the hierarchy of active modules is tracked via a thread-local caller_modules setting. This ensures that nested modules can correctly associate their history with the parent call stack.
• dspy.signatures.Signature: Defines the “what” of a task (schema). It uses Pydantic models to enforce type safety for inputs, outputs, and instructions. Signatures provide the structure that Predictors use to format prompts.
• Predictors: Define the “how” (reasoning strategy). Examples include ChainOfThought, ReAct, and Refine.
  • Execution Flow: A predictor’s forward method receives input arguments, unpacks them into the fields defined by its associated Signature, generates a prompt for the LLM client, and returns a structured Prediction object.

Execution & Optimization Flows

Module Composition

Programs are built by nesting Modules. When a parent module is called, it executes the .forward() methods of its children. The framework handles the propagation of state and history across this tree using the thread-local context mentioned above.

Teleprompting (dspy.teleprompt)

This is the “training” loop for DSPy programs. It optimizes prompts and parameters systematically.

• Inputs: A composed Program and a small set of examples (loaded via dspy.datasets).
• Optimizers: Includes strategies like Bootstrap, GRPO, and Simba.
• Output: Optimized weights/prompts for the modules within the program, allowing the pipeline to adapt to specific task requirements without manual rewriting.

Parallelization (dspy.predict.parallel)

The framework supports concurrent execution across different modules using a ParallelExecutor and threading.Lock.

• Key Complexity: Because history and callbacks rely on thread-local settings, the primary architectural challenge is maintaining isolated states for parallel branches to prevent history leakage between concurrent threads.

Supporting Infrastructure

• Clients (dspy.clients): Abstracted interfaces for various LLM providers (e.g., OpenAI, local models) and embedding providers. This allows switching backends without changing program logic.
• Retrievers (dspy.retrievers): Components for RAG workflows that abstract context-fetching logic (e.g., embeddings, databricks_rm).

Repository Reference Paths

• dspy/primitives/module.py
• dspy/signatures/signature.py
• dspy/predict/predict.py
• dspy/predict/parallel.py

I then varied the resources available while the model answered each question. In Direct, Base received only the question, while Cheatsheet and StudyLoop received the question plus their guide. This was our control with no tools. Search-5 and Search-20 allowed up to five or twenty calls to local repository tools for listing files, searching text, and reading source. Each of those settings was run twice: once with evaluation-time thinking disabled and once with it enabled.

That produced the following grid:

30 questions × 3 study conditions (Base, Cheatsheet, StudyLoop) × 3 evaluation budgets (Direct, Search-5, Search-20) × 2 thinking modes (Off, On) = 540 planned trajectories

Both guides were created before any benchmark question was loaded. During that stage, the model could not see the questions, gold answers, rubrics, benchmark evidence, topic labels, or anything derived from the exam. Each guide also had to be grounded in the repository: the model had to read at least six distinct source files and cite at least four paths it had actually read. Once a guide passed those checks, it was hashed, frozen, and reused unchanged throughout its condition. The experiment does not update the model’s weights, and the model does not remember one evaluation question while answering the next.

Gemma had no internet access. Its only source of external information was the pinned local DSPy repository. The same gemma4:12b-it-qat model performed the studying, repository searches, and final answering. A separate gpt-5.6-sol model graded only the completed answer against StudyBench’s grading material; it did not see the study condition or Gemma’s private thinking, and Gemma never saw the grading data.

The grader produced two measures like Machine Studying’s work with Qwen3.5-9B. The lenient score awarded credit for each weighted rubric claim the answer satisfied. The strict score retained that credit only if every core claim was present; otherwise it became zero. The first measure captures partial progress. The second asks whether the answer was complete.

So the claim here is deliberately narrow and restrictive in-scope: can a structured study process create a more useful preparation artifact?

3. Preparation Helps Most When Inference Is Constrained

Of the 540 planned trajectories, 525 completed with a valid answer and grade. The successful runs were evenly distributed, leaving 175 in each study condition.

When I aggregated across every search and thinking setting, StudyLoop had the highest mean lenient score:

Condition	Completed	Mean lenient score	Mean strict score	Nonzero strict scores
Base	175	12.8	2.07	4
Cheatsheet	175	13.4	1.00	2
StudyLoop	175	15.3	2.44	5

StudyLoop finished 2.5 raw points above Base and 1.9 above Cheatsheet. That was the direction I expected, but the uncertainty around those differences is too large to call it a clear win. In a paired analysis clustered by question, StudyLoop’s estimated advantage over Base was 2.8 points, with a bootstrap 95% interval from -0.2 to 6.0. Its advantage over Cheatsheet was 1.6 points, with an interval from -2.6 to 5.5. Both intervals include zero.

The aggregate result is therefore suggestive and not decisive. StudyLoop finished ahead, but this experiment does not establish that its active study process is generally better than either alternative.

More importantly, averaging everything together obscures how much the result depended on the resources available at answer time:

Evaluation setting	Base	Cheatsheet	StudyLoop
Direct, thinking off	5.9	9.4	8.9
Direct, thinking on	6.4	6.8	8.4
Search-5, thinking off	5.6	18.6	15.1
Search-5, thinking on	18.4	11.0	18.3
Search-20, thinking off	11.9	16.8	15.6
Search-20, thinking on	30.3	18.5	26.5

The Cheatsheet condition led whenever evaluation-time thinking was disabled. Its largest advantage appeared at Search-5, where it scored 18.6 against 15.1 for StudyLoop and 5.6 for Base. In those constrained settings, arriving with a compact map of the repository appears to have helped. More on this later.

Once thinking was enabled, the pattern changed. Base improved by 12.8 paired points at Search-5 and 18.0 at Search-20, while its Direct score moved by only 0.5. At Search-20 with thinking enabled, Base produced the highest mean of the entire experiment: 30.3.

This pattern is consistent with a guide being most useful when the model cannot spend much effort reconstructing the answer from source. When both search and reasoning are available, the model can build question-specific context on demand, and the advantage of a generic guide may shrink.

Anecdotally, it seems that a model’s built-in reasoning may be stronger on its own than when constrained by a specific process early on (e.g., StudyLoop). Processes like StudyLoop may inadvertently pigeonhole the model into a particular reasoning strategy. However, there is not enough conclusive evidence to support this hypothesis.

4. More Preparation Is Not Automatically Better

StudyLoop used substantially more preparation than Cheatsheet. It consumed about 1.9 times as many study input tokens, four times as many study output tokens, twice as many repository calls, and 2.8 times the runtime. Despite that extra work, it did not reliably produce better downstream answers than the one-pass guide in our small experiment.

The same lesson appeared during evaluation: more inference was not automatically more useful. Search-20 with thinking enabled produced the highest cell means, but its trajectories were expensive. The average successful Base run in that setting took 196 seconds and processed 204,553 input and output tokens. StudyLoop took 196 seconds and processed 206,251. Cheatsheet took 229 seconds and processed 277,544, yet scored below both of them at 18.5.

Going more into the logistics and reproducibility of this experiment: it ran sequentially on one Runpod Nvidia RTX 5090 for about 15 hours and 40 minutes. The successful trajectory runtimes themselves summed to 9.65 hours; failed attempts, retries, grading, checkpointing, and orchestration account for the remaining wall time.

Those figures matter because a nominal tool budget does not fully describe the compute an agent uses. Two Search-20 trajectories can accumulate very different amounts of context and generation. If the goal of studying is to make later inference more efficient, performance has to be considered alongside actual tokens, runtime, and reliability.

5. What These Results Do and Do Not Show

A few patterns can be taken from these results, although the experiment cannot firmly establish why they occurred.

The Cheatsheet appears to act as an orientation layer. It provides broad, reusable context before the question arrives, and it led all three settings in which evaluation-time thinking was disabled. Its relative value then shrank when Gemma could combine repository search with additional reasoning. One interpretation is that the model could reconstruct more relevant, question-specific context from source and therefore relied less on its generic preparation.

A frozen guide may also become a constraint of its own. Because both guides were written without knowing the eventual question, they emphasized general concepts rather than every implementation detail a particular answer might require. They may have oriented the model toward useful abstractions, but they may also have anchored it on information that was too broad. The current experiment did not isolate or directly test that mechanism.

StudyLoop performed more intermediate work: self-questioning, grounded answering, critique, and revision. But this did not translate into a decisive advantage over the simpler Cheatsheet. Additional preparation stages can consume more compute without ensuring that the final artifact preserves the information later questions need. Machine Studying separately observed agents settling on plausible solutions early and then engineering around them. The same kind of early commitment is possible here, but I did not conduct a systematic trajectory analysis that could support that explanation.

There are several limitations to keep in view regarding this experiment as well:

• The experiment tested one 12B Gemma 4 model in a Q4_0 QAT quantization, one repository, and 30 questions. The 540-cell grid explores interactions within that setting; it does not make 30 questions a large or representative sample of domains.
• DSPy has existed since December 2022, though its popularity increased rapidly after 2024. Google reports a January 2025 cutoff for Gemma 4’s pretraining data, while the pinned DSPy snapshot used here was from March 2026. Gemma may therefore have had partial or stale DSPy priors that helped some answers and conflicted with others. More broadly in the subject of continual learning, one could make the argument that overriding priors is essential and therefore good to test in an experiment. However, StudyLoop did not isolate prior knowledge or audit enough failures to estimate that effect.
• Cheatsheet and StudyLoop each produced one separate frozen guide. Reusing each guide unchanged made its condition controlled and resembles giving many worker agents the same .md file. It also means the experiment measures these particular artifacts, not the distribution of guides each method might produce. Another generation could change the ranking.
• Fifteen trajectories remained incomplete, with twelve concentrated in Search-20. The missingness is therefore not safely assumed to be random.
• The experiment records performance and actual compute, but it has not yet calculated the weighted expertise score defined by Machine Studying. More on this in possible next steps.

With those limits, the results support four cautious takeaways. Of those:

• StudyLoop had the highest aggregate lenient score, but its paired advantages over Base and Cheatsheet were uncertain.
• A simple Cheatsheet was consistently useful when evaluation-time thinking was disabled.
• Base benefited most from adding thinking when repository search was available, matching or beating both guide conditions at Search-5 and Search-20.
• Larger inference budgets raised the best observed scores but also increased token use, runtime, and protocol failures.

6. Toward Agents That Can Acquire Expertise

The most direct next step is to repeat the experiment across more StudyBench domains. Studying-OpenClaw would be a stronger test of adaptation to a post-cutoff corpus, where prior familiarity with the repository is less plausible. The Studying-Literature configuration was not available in the pinned public Hugging Face revision used for this experiment, perhaps it’s too large. If it becomes available, it would test a different problem: finding and recognizing relevant evidence in a corpus too large to treat as a single context window.

I would also measure expertise directly rather than treating the aggregate score as a proxy. Direct is the cheapest point on a performance-compute curve, not the curve itself. A follow-up should evaluate each fixed agent configuration at several actual generation-token budgets, include a forced-search setting, preregister the importance function $w(x)$, and then calculate the weighted area under the curve. This would allow me to say whether a method produced a net expertise gain, rather than only whether it raised average performance across the six settings I happened to run.

There is a second curve worth measuring. For each amount of study compute, I could first calculate the agent’s downstream expertise and then plot expertise against the preparation budget. Machine Studying calls the weighted area under that second curve studying intelligence. Extending StudyBench in this direction across several unfamiliar domains could help distinguish a model that starts strong from a system that is especially efficient at acquiring expertise.

StudyLoop also points toward a more ambitious training experiment. In this project, the agent generated its own questions, answered them from the corpus, critiqued the answers, and compressed the result into a guide. But the questions never became a training environment: there was no learned reward, no on-policy sampling, and no update to Gemma’s weights.

A next version could turn that intermediate material into source-grounded practice. The agent would generate questions and rubrics from the corpus, produce several answer rollouts, verify them against source, and score them for correctness, grounding, and inference cost. Those scores could then train a small adapter or policy on-policy, while the held-out StudyBench questions remained completely hidden. The comparison would be between the original model, the frozen-guide methods, and the trained model at the same downstream inference budgets. The central question would not be whether the model memorized its own synthetic answers, but whether training moved its full performance-compute curve upward and to the left.

That direction introduces two new risks. Self-generated questions may emphasize what the model already understands rather than its real weaknesses, and a self-generated reward may not align with the unseen evaluation. The existing StudyLoop artifacts—survey, questions, grounded answers, critique, and guide—provide a starting pipeline, but the next experiment would need independent source checks and strict separation from the benchmark to keep synthetic practice from becoming synthetic self-confirmation.

And finally, a natural next step is to develop ExpertiseBench, a benchmark for measuring not just how much expertise a model already has, but how effectively it can acquire expertise in a previously unfamiliar domain. Given a fixed corpus and study-compute budget, models or agents would be evaluated before and after studying, allowing us to measure both total expertise gained and the efficiency with which that expertise is acquired. Across multiple novel domains, this could help distinguish models that are simply strong at baseline from systems that are particularly good at learning.

7. Acknowledgments

I’m deeply grateful to Remzi Arpaci-Dusseau for believing in me and other young researchers coming out of the Wisconsin Computer Sciences program, and to Braden Hancock and the Laude Institute for pushing me toward this research direction and encouraging me to pursue open-ended research.

I’d especially like to thank Omar Khattab, Jacob Li, and Rick Battle for Machine Studying and StudyBench, which provided both the motivation and experimental foundation for this project. More broadly, I’m grateful to MIT OASYS and its collaborators for consistently doing ambitious research in the open. Work like Machine Studying, Recursive Language Models (RLMs), Pedagogical RL, and GEPA played a large role in getting me interested in open research and in the questions around how agents can learn, adapt, and improve.

8. Reproducibility, Code, and Links

Everything used to run StudyLoop is open source. The repository contains the studying methods, prompts, evaluation harness, frozen-guide machinery, trajectory logging, experiment runners, tests, and documentation needed to reproduce or extend the experiment.

• StudyLoop — GitHub repository — complete source code and entry point for the project.
• Experiment Protocol — experimental design, conditions, controls, and evaluation procedure.
• Reproducing StudyLoop — environment setup, hardware, execution, recovery, and reproduction instructions.
• Results Data — documentation for the released trajectories, grades, guides, and experimental artifacts.
• StudyLoop Prompts — the versioned study, answering, and grading prompts used by the harness.

If you use StudyLoop or its experimental harness in your own work, please cite this project as:

Samad Syed. “StudyLoop: Can an AI Agent Study a Codebase?” 2026. https://samadsyed.com/blog/2026/studyloop/