An end-to-end auto-research system takes a research question through hypothesis generation, experimentation, interpretation, and communication. The hard part is maintaining a reliable connection between those stages. An idea must become an executable test; a result must support a claim; and a failed experiment must inform the next attempt.
The systems collected in Awesome-Auto-Research-Tools tackle this problem with different degrees of autonomy. Their designs become easier to understand through four questions: What can the agent change? How does it choose the next experiment? What does it remember? Who decides whether the result is credible?
An executable research problem
Karpathy’s autoresearch reduces autonomous research to a tight experimental loop. The agent edits a training script, runs it for a fixed five-minute budget, and keeps the change when the validation metric improves. The research object is the script: a proposed architecture, optimizer, or training change becomes a code edit that can be tested. Git records the successive versions, while program.md lets a human adjust the agent’s research instructions.
This loop works because it has an executable task and a comparable outcome. The fixed time budget makes experiments compete under the same resource constraint. It also illustrates the boundary of a metric-driven system: the metric can guide search only when it represents the research goal well.
The AI Scientist expands the loop from experiments to ideas, paper writing, and automated review. Its templates supply a codebase, data, and an evaluation protocol. A template gives the agent a place to run ideas and a rule for comparing them. AI-Scientist-v2 removes the preset template and starts from a topic description. That expands the space of possible ideas and makes experiment design a larger part of the agent’s work.
These systems show why the experimental environment matters. An agent can generate many hypotheses, but autonomous research depends on turning each one into a test whose output can be interpreted.
Choosing the next experiment
A single chain of experiments is easy to execute and difficult to redirect after a weak early choice. AI-Scientist-v2 uses best-first tree search: experimental states form branches, and the agent expands a promising state while retaining other paths for later exploration or debugging. This makes alternatives explicit.
DeepScientist frames the choice as an exploration–exploitation problem. Its reviewer estimates a candidate’s usefulness, quality, and exploration value; an upper-confidence-bound-style rule selects candidates for expensive evaluation. The key design decision is to spend experimental compute on ideas selected from a broader set, while using earlier findings to inform selection.
RD-Agent makes a different choice in data-intensive research. It searches historical factor and model hypotheses in a Knowledge Forest and uses a contextual bandit to decide which kind of optimization to prioritize. Its objective is a better factor or model. EvoScientist searches both ideas and implementations as trees and ranks ideas through pairwise Elo comparisons. These mechanisms address distinct decisions: which hypothesis to pursue, which implementation to try, and where to allocate the next experiment.
Carrying results forward
An experiment has little cumulative value if the next run cannot use its outcome. DeepScientist stores untested ideas, implemented findings, and results that surpass a baseline as different types of findings. Failed paths remain available, so later searches can avoid the same dead end.
EvoScientist keeps separate memories for research directions and experimental strategies. Its Evolution Manager updates both from successes and failures across runs. InternAgent stores task outcomes, connects similar ideas in an IdeaGraph, and revises its idea-generation prompt based on experience. AutoResearchClaw extracts lessons from pipeline failures as reusable skills.
These are different forms of learning from research history. A record of past metrics helps compare experiments. A graph of ideas exposes repeated directions. A reusable strategy or skill changes how future experiments are designed. Memory becomes useful when it changes a subsequent decision.
Extending beyond one experimental setting
A general research pipeline still needs domain-specific tools. AutoResearchClaw routes experiments to specialized execution agents for areas such as high-energy physics, biology, and statistics. Biomni takes the tool environment as its central problem: it builds a biomedical action space from tools, software packages, databases, and workflows found in the literature. Its agent then plans and executes biomedical computational tasks using that environment.
InternAgent uses a domain-agnostic agent framework with task-specific prompts and evaluation checklists. Its FlowSearch represents a literature problem as a changing graph of dependent subquestions. These approaches place domain knowledge in different locations: specialist executors, a curated action space, or task context and research planning. All three give an agent concrete operations beyond generating text.
Connecting evidence to claims
A completed experiment does not automatically justify a paper claim. AI-Researcher addresses one source of error by explicitly mapping mathematical concepts to code implementations. That mapping helps ensure that an experiment tests the intended method. Its Documentation Agent then organizes experimental artifacts into a paper.
ARIS separates the executor from a reviewer who reads artifacts in a fresh context. Its audit stages check experimental code, the connection from results to claims, reported figures, citations, and counterarguments. This puts checks between producing an artifact and accepting the claims drawn from it.
claude-scholar keeps researchers in charge of that promotion. Its evidence-gated workflow moves material from source annotations to reviewed knowledge and then writing. Idea2Paper tackles an earlier point in the same chain: it extracts reusable methodological units from papers into an offline knowledge graph and uses peer-review signals to guide retrieval and refinement. One approach manages the evidentiary status of claims; the other structures the source material from which ideas are formed.
The role of the researcher
Full autonomy is one design choice. Agent Laboratory offers a co-pilot mode that pauses for input at key decisions. AutoResearchClaw provides several intervention modes, while EvoScientist lets humans set goals and review final outputs. claude-scholar makes human judgment central throughout the work.
The practical question is which decisions can be delegated. A reliable metric can support autonomous iteration on a training script. A literature interpretation, a scientific claim, or a choice of research direction may need a different review process. End-to-end systems are strongest when their search, execution environment, memory, and review method fit the kind of knowledge they aim to produce.
For a closer comparison of search and learning mechanisms, see A Methodological Review of Auto-Research Systems.
modifications:
- Oct 1, 2026: rewrite in English.