Karpathy's Auto-Research + Claude Code

That's not science fiction anymore. Andrej Karpathy — one of the most respected names in AI research — just open-sourced a project called autoresearch that does exactly that. And when you pair it with Claude Code, you can apply the same self-improving loop to almost any business metric you care about.

No, this isn't hype. This is a real GitHub repo with real practical applications. Let's break down what it actually does, how the pattern works, and how to set it up for your business.

What Is Karpathy's Auto-Research Pattern?

Karpathy built autoresearch while training his own language model. His idea: instead of manually tweaking hyperparameters and waiting hours for results, why not let the AI do the experimentation itself?

Here's the core concept, straight from the repo:

The agent runs a tight loop: form a hypothesis → run the experiment → measure a metric → keep what works → iterate. In Karpathy's case, the metric was validation loss (how well the model predicts text). But the pattern is universal.

Why This Changes Everything for Business

Here's the thing — you don't need to be training language models to use this pattern. The loop applies to anything with three ingredients:

Once you have those three things, you can build an optimization loop that runs 24 hours a day — without you being in the loop at all.

Real-World Use Cases Beyond Machine Learning

What You'll Need

• Claude Code — Any IDE that runs Claude Code (VS Code, Cursor, or OpenClaw on a VPS)
• GitHub account — For hosting the repo and running GitHub Actions
• An API for your platform — Instantly, Webflow, Meta Ads, etc.
• Anthropic API key — The orchestrator uses Claude to generate new variants
• A clear metric you can query — If your platform doesn't have an API, Chrome DevTools MCP can help
• ~30 minutes — Most of the setup is handled by Claude Code

Step 1: Clone the Auto-Research Repo

Open Claude Code in a new project folder and give it this command:

Clone the autoresearch repo from https://github.com/karpathy/autoresearch
into the current working directory.

Claude Code will clone the repo, read the documentation, and load all the context it needs before asking what you want to build. The repo includes:

• program.md — The instructions the agent follows to run experiments
• train.py — The mutable file the agent is allowed to modify (you'll replace this with your own equivalent)
• Research logs — Where the agent records what it learned from each experiment

Step 2: Define Your Goal, Metric & Test Method

This is the most important step. Once Claude Code has the repo context, describe exactly what you want to optimize. Here's the prompt structure that works best:

Use the context in the auto-research folder to help me build a similar
pipeline, except instead of testing for validation loss and iterating on
a machine learning model, I want to do this for [YOUR USE CASE].

The metric I want to optimize is: [YOUR METRIC]
The platform I'm using is: [YOUR PLATFORM]
The thing that changes between experiments: [YOUR VARIABLE]
API credentials will be provided in a moment.

Put this on the cloud using GitHub Actions, running every [X] hours.

Step 3: What Claude Code Builds for You

Claude Code will scaffold the entire system. Here's what it typically generates:

Step 4: Deploy to GitHub Actions (Run 24/7)

Once Claude Code has generated all the files, push to GitHub and configure your secrets:

git init
git add .
git commit -m "feat: auto-research optimizer"
git remote add origin https://github.com/YOUR_USERNAME/YOUR_REPO.git
git push -u origin main

Then in your GitHub repo, go to Settings → Secrets and variables → Actions and add:

• ANTHROPIC_API_KEY — Your Claude API key
• PLATFORM_API_KEY — Your platform API key (e.g. Instantly)
• Any other credentials Claude Code identified

The GitHub Actions workflow will look like this:

name: Auto-Research Optimizer

on:
  schedule:
    - cron: '0 */4 * * *'  # Run every 4 hours
  workflow_dispatch:        # Allow manual runs

jobs:
  optimize:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r requirements.txt
      - run: python orchestrator.py
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          PLATFORM_API_KEY: ${{ secrets.PLATFORM_API_KEY }}

Step 5: Monitor with Slack Notifications

Since the loop runs completely autonomously, you'll want visibility into what's happening. Ask Claude Code to add a Slack webhook integration:

Add a Slack webhook notification to the orchestrator. 
Send a message every time:
- A new challenger variant is created (include the copy)
- A harvest completes (include which variant won and the metric scores)

Slack webhook URL: [YOUR_SLACK_WEBHOOK_URL]

You'll get Slack pings like:

What Good Metrics Look Like

Not all metrics are created equal. The auto-research pattern works best when your metric is:

• Objective — A number, not an opinion. "Reply rate" is objective. "Feels more engaging" is not.
• Fast to measure — The tighter the feedback loop, the faster you improve. Cold email takes 4–24 hours. ML training takes 5 minutes. Choose accordingly.
• API-accessible — The agent needs to query the result programmatically. If there's no API, Chrome DevTools MCP can bridge the gap for browser-based platforms.

When Auto-Research Doesn't Work

Be honest about these limitations before you start:

• No API access — If you can't query the metric or deploy changes via code, the loop breaks. Chrome DevTools MCP is a workaround but adds complexity.
• Very slow feedback loops — If results take weeks (e.g. SEO), the iteration cycle is too slow to be useful for automated optimization.
• Compliance-sensitive platforms — Some ad platforms have strict rules about automated account modifications. Check their ToS before running automated experiments.
• Insufficient volume — You need enough data per experiment to get statistically meaningful results. If you're getting 10 email opens per day, your metrics are too noisy.

The Bigger Picture

This is what every major AI lab in the world is already doing — running thousands of experiments overnight to make their models better. Karpathy just open-sourced the pattern and made it accessible to everyone.

The compounding effect is the point. In early experiments, most challengers lose to the baseline. But over hundreds of runs, the agent builds a resource.md of accumulated knowledge — "shorter subject lines with personal openers outperform generic intros by 40%" — and starts generating genuinely better variants.

Run this for a year, and you end up with an optimization system that's orders of magnitude better than anything you could build manually.