This AI Reasoning Harness Crushed Putnam 2026 Meet Nomos

This AI Reasoning Harness Crushed Putnam 2026 Meet Nomos

What if your AI could obsess over math problems until it got them right? Not just try once and give up like every other LLM you've used—but actually iterate, self-critique, and tournament-select its way to competition-level solutions?

Here's the brutal truth: most AI reasoning tools fail at hard mathematics because they treat proof-writing like chat completion. One shot. One answer. No verification loop. Meanwhile, Nomos—a new open-source reasoning harness from NousResearch—just scored 87/120 on the notoriously difficult Putnam Mathematical Competition 2026, demolishing Qwen3's 24/120 under identical conditions. That's not incremental improvement. That's a different species of tool entirely.

If you're building AI systems for theorem proving, automated verification, or competitive mathematics, you need to understand why everyone in the mechanized reasoning community is suddenly talking about Nomos. This isn't another wrapper around GPT-4 with fancy prompting. It's a self-improving, tournament-based reasoning architecture that treats mathematical problem-solving as a stochastic optimization process—with parallel workers, pairwise elimination, and intelligent consolidation.

Ready to see how it works? Let's dissect the system that's making traditional "chain-of-thought" look like a toy.

What is Nomos?

Nomos is a reasoning harness for mathematical problem-solving and proof-writing in natural language, developed by the research collective NousResearch. The name evokes the Greek concept of "law" or "custom"—fitting for a system built to discover mathematical truth through structured, repeatable processes.

Unlike conventional LLM inference where a model generates a single response and hopes for the best, Nomos operates as an outer optimization loop around any OpenAI-compatible API endpoint. It was created by a team led by Roger Jin, Jeffrey Quesnelle, Dakota Mahan, and collaborators including the mysterious contributor "massiveaxe" (whose handle appears in both the README's argument documentation and the citation—talk about earned recognition).

Why it's trending now: The release coincides with a watershed moment in AI mathematics. The Putnam Competition—Harvard's annual undergraduate math olympiad that routinely stumps PhD candidates—has become the new ImageNet for reasoning systems. Nomos's 87/120 score represents one of the highest autonomous performances ever recorded. For context, the median Putnam score among actual human competitors is often 0 or 1 out of 120.

The harness is model-agnostic at its core. While NousResearch provides their own Nomos-1 model on Hugging Face, the architecture works with any OpenAI-compatible endpoint. This flexibility, combined with its MIT license, has made it instantly attractive to researchers who want to benchmark their own models or build proprietary reasoning pipelines.

The philosophical shift Nomos represents is subtle but profound: from generating answers to running experiments. Each "submission" is a hypothesis. Each "judge" evaluation is an experiment. The system doesn't trust single outputs—it builds statistical confidence through repetition and structured elimination.

Key Features That Separate Nomos from the Pack

Nomos isn't a monolithic solver. It's a distributed reasoning orchestrator with several architectural innovations worth studying:

Parallel Worker Architecture with Priority Scheduling

Nomos launches up to max_concurrent parallel workers (default: 32) that dynamically prioritize problems based on deficiency scoring. Problems with the fewest perfect scores get attention first, with round-robin tiebreaking. This isn't random exploration—it's adaptive resource allocation that ensures hard problems get disproportionate compute.

Multi-Phase Self-Critique Pipeline

The system implements a rigorous three-tier evaluation:

• Point-scoring (0-7): Each submission receives granular feedback
• Consolidation by logical conclusion: Groups equivalent proofs, keeping the "correct" group (not majority voting!)
• Pairwise tournament: Single-elimination bracket with random tiebreaking for final selection

This tournament structure is particularly clever. Rather than trusting a single judge's absolute score, Nomos uses comparative judgment—which psychological research shows is more reliable than absolute rating scales.

Intelligent Termination Conditions

Nomos doesn't run forever. It stops when either:

• All problems achieve target_perfect_scores (default: 4) independent 7/7 ratings
• The time_limit_hours expires (default: 3 hours)

The finalization phase triggers proactively—15 minutes before timeout for normal runs, or at 50% elapsed time for short runs. This ensures sufficient time for consolidation and tournament selection.

Full Prompt Customization

Every stage of the pipeline uses configurable prompt files:

• prompts/score.md for judging
• prompts/consolidation.md for grouping equivalent solutions
• prompts/pairwise.md for tournament comparisons
• Optional custom solve_prompt for domain-specific reasoning strategies

This makes Nomos extensible to non-mathematical reasoning domains—legal analysis, scientific hypothesis generation, or code verification.

OpenAI-Compatible API Flexibility

The --base_url parameter defaults to http://localhost:30000/v1 but accepts any OpenAI-compatible endpoint. Run local models via vLLM, connect to commercial APIs, or mix models—use nomos-1 for solving and a stronger model for judging, for instance.

Real-World Use Cases Where Nomos Dominates

1. Competitive Mathematics Benchmarking

The obvious application: Putnam, IMO, and other olympiad problems. But the deeper value is systematic capability evaluation. Nomos doesn't just give you a score—it gives you a distribution of attempts, convergence rates, and per-problem difficulty curves. Research labs can use this to identify exactly where their models fail.

2. Automated Theorem Proving Assistance

Interactive theorem provers like Lean or Coq require formal specification. Nomos bridges the gap: natural language proof sketching with automated verification. Mathematicians can explore conjectures in prose, let Nomos iterate toward rigorous arguments, then formalize the successful paths.

3. Mathematical Education at Scale

Imagine homework systems that don't just check answers but generate multiple solution attempts, critique them, and present the best. Nomos's tournament output format—showing original problem, selected solution, and implicitly the rejected alternatives—creates teachable moments about proof quality.

4. Research Hypothesis Generation

Scientific reasoning often proceeds by conjecture and refinement. Nomos's architecture adapts naturally: pose a theoretical question, let parallel workers propose approaches, judge feasibility, and consolidate toward promising directions. The time-bound nature prevents infinite exploration of dead ends.

5. Formal Verification of Informal Arguments

In security-critical domains—cryptographic protocol analysis, distributed system proofs—teams write natural language arguments before formalization. Nomos can stress-test these: generate attacks on the reasoning, find edge cases the authors missed, and quantify confidence through repeated evaluation.

Step-by-Step Installation & Setup Guide

Getting Nomos running takes minutes if you have Python^{↗ Bright Coding Blog} and an API endpoint ready.

Prerequisites

• Python 3.8+
• An OpenAI-compatible API server running locally or remote access
• For local deployment: GPU with sufficient VRAM for Nomos-1 (check Hugging Face for specifications)

Installation

Clone the repository and install dependencies:

# Clone the repository
git clone https://github.com/NousResearch/nomos.git
cd nomos

# Install dependencies
pip install -r requirements.txt

The requirements.txt isn't shown in detail, but expect standard ML inference dependencies: openai client, aiohttp for parallel requests, and likely tiktoken or similar for token management.

Directory Structure Setup

Create your problem directory with .md files:

mkdir my_problems
echo '# Problem 1

Prove that for any prime $p > 3$, the number $p^2 - 1$ is divisible by 24.' > my_problems/problem_01.md

Each problem file should contain the problem statement in markdown^{↗ Smart Converter}. The harness will parse these and generate structured output.

Configure Your API Endpoint

If running Nomos-1 locally via vLLM:

# Example vLLM launch (adjust model path and parameters)
python -m vllm.entrypoints.openai.api_server \
  --model NousResearch/nomos-1 \
  --port 30000 \
  --tensor-parallel-size 2  # Adjust for your GPU count

For remote endpoints, set --base_url accordingly.

Basic Execution

# Minimal run with defaults
python solve_agent.py my_problems

# Production run with full configuration
python solve_agent.py my_problems \
  --submissions_dir results/my_competition \
  --time_limit_hours 6.0 \
  --max_concurrent 64 \
  --target_perfect_scores 6 \
  --model nomos-1 \
  --judge_model nomos-1 \
  --base_url http://localhost:30000/v1

Output Inspection

After completion, check your submissions directory:

ls submissions/my_problems-2026*/
# Each problem gets its own markdown file with problem restatement and selected solution
cat submissions/my_problems-*/problem_01.md

REAL Code Examples from the Repository

Let's examine the actual implementation patterns from NousResearch/nomos, with detailed commentary on how each component functions.

Example 1: Basic Execution Command

The simplest entry point, straight from the README:

python solve_agent.py <problems_dir> [options]

This is deceptively simple. The solve_agent.py script encapsulates the entire orchestration logic—parallel worker management, priority queue maintenance, API rate limiting, and graceful shutdown. The <problems_dir> positional argument is strictly required: the harness expects a directory containing .md files, not individual files. This design choice enables batch processing of entire competition problem sets.

The [options] bracket indicates substantial configurability. In practice, you'll almost always specify at least --base_url and likely tune --max_concurrent based on your API rate limits. The default 32 concurrent workers assumes a local vLLM deployment that can handle the load; commercial APIs may require reduction to avoid throttling.

Example 2: Full Flag Configuration

Here's the complete options table implemented as a practical command:

python solve_agent.py putnam_2026_a \
  --submissions_dir submissions/putnam_a_final \
  --judge_prompt prompts/score.md \
  --solve_prompt prompts/custom_math.txt \
  --consolidation_prompt prompts/consolidation.md \
  --pairwise_prompt prompts/pairwise.md \
  --time_limit_hours 3.0 \
  --max_concurrent 32 \
  --target_perfect_scores 4 \
  --model nomos-1 \
  --judge_model nomos-1 \
  --base_url http://localhost:30000/v1

Critical insight: The --solve_prompt defaults to None, meaning the system uses the model's base behavior. For specialized domains—say, geometric proofs requiring specific diagram conventions—a custom solve prompt can dramatically improve convergence rates. The separation of model and judge_model enables adversarial configurations: use a smaller, faster model for exploration and a larger model for verification, optimizing cost-quality tradeoffs.

The target_perfect_scores=4 parameter deserves attention. It means Nomos demands four independent 7/7 scores before considering a problem "solved." This isn't arbitrary conservatism—it's statistical robustness. A single lucky 7/7 happens; four consecutive perfect scores from independent attempts indicates genuine mastery. The harness will keep exploring a problem even if it has three perfect scores, which explains why hard Putnam problems consumed disproportionate compute.

Example 3: Runbook Execution for Putnam Benchmarking

The repository includes pre-configured runbooks for reproducible benchmarking:

./runbooks/run_putnam_2025_b_nomos-1.sh   # Putnam 2025 A problems
./runbooks/run_putnam_2025_b_nomos-1.sh   # Putnam 2025 B problems

Note: The README contains a likely typo—both lines reference run_putnam_2025_b_nomos-1.sh but comment them as A and B problems respectively. In practice, you'd expect run_putnam_2025_a_nomos-1.sh for the A division. These shell scripts encode the exact configuration that produced the published 87/120 score, ensuring reproducible research.

Runbooks are essential for scientific validity. They capture not just hyperparameters but environmental assumptions—API endpoint state, model quantization, hardware configuration. When you read the paper (forthcoming, presumably), these scripts let you verify every claim.

Example 4: Output Format Structure

The generated submission files follow a strict schema:

# problem_id

## Problem

[original problem text]

## Submission

[selected solution]

This isn't mere presentation formatting—it's provenance preservation. By restating the original problem, the output becomes self-contained for human review. The tournament-selected solution appears without commentary, forcing the model to produce standalone, comprehensible proofs rather than fragmented reasoning traces.

The absence of rejected alternatives in the final output is a deliberate design choice. The consolidation and tournament phases happen opaquely; what matters is the confidence in the selected solution, not the path there. For debugging, you'd inspect intermediate submission files in the working directory.

Example 5: Citation Block

@misc{nomos2025,
  title        = {Nomos},
  author       = {Jin, Roger and Quesnelle, Jeffrey and Mahan, Dakota and Guang, Chen and Teknium, Ryan and Park, Jun and Ustelbay, Ibrakhim and Kim, Samuel and Yurkevich, Miron and Zauytkhan, Adilet and Amankos, Rinat and Andreyev, Alex and Nurlanov, Damir and Abuov, Abuzer and massiveaxe, Askar},
  year         = {2025},
  howpublished = {\url{https://github.com/NousResearch/nomos}},
  note         = {GitHub repository},
}

The author list reveals NousResearch's collaborative, distributed nature—fifteen contributors spanning what appears to be multiple continents and institutional affiliations. The inclusion of a pseudonymous contributor ("massiveaxe") alongside traditionally-credited researchers reflects the open-source ethos. For academic use, this BibTeX entry ensures proper attribution when building upon or comparing against Nomos.

Advanced Usage & Best Practices

Optimize Your Concurrency Sweet Spot

The default max_concurrent=32 assumes local inference. For cloud APIs, start with 8-16 and monitor rate limit headers. The priority queue means reduced concurrency doesn't harm solution quality—just throughput. For cost optimization, run with lower concurrency overnight.

Craft Domain-Specific Prompts

The default prompts work for general competition math, but specialized domains benefit from custom injection. For algebraic geometry, your solve_prompt might emphasize scheme-theoretic thinking. For combinatorics, emphasize bijective proof strategies. The judge prompt is equally important—tune scoring rubrics to your domain's values.

Exploit Model Asymmetry

Run experiments with model=gpt-4o-mini and judge_model=gpt-4o. The cheap model explores; the expensive model verifies. This can reduce costs 10x while maintaining quality, though convergence may require adjusted target_perfect_scores.

Monitor Intermediate Outputs

The working directory contains rich diagnostic data: all submissions with scores, consolidation groupings, tournament brackets. Analyze these to understand where your model fails. Are scores bimodal (clear correct/incorrect) or uniformly mediocre? This reveals model calibration.

Time Limit Calibration

For known problem difficulties, use shorter time limits with higher target_perfect_scores to force efficient exploration. For unknown problems, the 3-hour default provides good coverage. The 15-minute/50% finalization trigger is fixed—plan your total time accordingly.

Comparison with Alternatives

Why choose Nomos? It occupies a unique position: more rigorous than raw LLM inference, more flexible than formal provers, and more scalable than human-in-the-loop approaches. The tournament-based selection specifically addresses a failure mode of other methods: judge inconsistency. By comparing solutions pairwise rather than scoring absolutely, Nomos reduces variance in final selection.

FAQ

What hardware do I need to run Nomos locally?

For the full Nomos-1 model, you'll need multi-GPU setup or substantial VRAM via quantization. However, Nomos works with any OpenAI-compatible endpoint—cloud APIs, smaller local models, or even proxy services. Start with whatever you have and scale up.

Can I use Nomos for non-mathematical reasoning tasks?

Absolutely. While optimized for proofs, the harness architecture is domain-agnostic. Replace the problem .md files with legal analysis questions, scientific hypotheses, or code verification tasks. You'll need to customize the prompt files for your domain's evaluation criteria.

How does Nomos differ from chain-of-thought prompting?

Chain-of-thought is a single reasoning trace. Nomos is a population-based optimization: it generates many independent attempts, scores them, groups equivalent solutions, and tournaments the best. It's the difference between one person thinking aloud and a research team iterating with peer review.

Is the 87/120 Putnam score reproducible?

Yes, using the provided runbooks and Nomos-1 model. Exact reproduction requires identical model weights, API server configuration, and random seed handling. The runbooks capture the hyperparameters; environmental details may require additional documentation.

What does "consolidation by conclusion" mean practically?

Nomos groups submissions that reach the same mathematical conclusion, not those with similar proof text. Two proofs using entirely different techniques (induction vs. generating functions) that prove the same result get grouped together. The system then selects what it judges the "correct" group—using structural features of the proofs, not mere majority voting.

How do I interpret the 0-7 scoring rubric?

The default prompts/score.md implements this, but typically: 7 = complete correct proof, 5-6 = correct with minor gaps, 3-4 = significant progress but critical flaw, 0-2 = irrelevant or fundamentally wrong. The granularity enables nuanced priority scheduling beyond binary correct/incorrect.

Can I contribute to Nomos development?

The repository is MIT-licensed and actively maintained by NousResearch. Issues, pull requests, and community extensions are welcome. The modular prompt architecture makes it particularly accessible for contributions—new domain prompts don't require code changes.

Conclusion

Nomos represents a paradigm shift in AI-assisted reasoning: from hoping your model gets lucky to engineering confidence through structured iteration. The 87/120 Putnam score isn't just a number—it's proof that systematic self-critique, parallel exploration, and tournament selection can extract capabilities from language models that single-shot inference never reveals.

For researchers, Nomos provides a reproducible benchmarking framework. For practitioners, it's a production-ready harness for hard reasoning tasks. For the curious, it's a glimpse at how AI systems might eventually match human mathematical creativity—not through bigger models alone, but through smaller architectures that think longer and critique harder.

The field is moving fast. Qwen3's 24/120 under identical conditions shows that model scale without reasoning architecture hits a wall. Nomos shows the path through.

Ready to run your own mathematical experiments? Clone Nomos from GitHub, download the Nomos-1 model from Hugging Face, and start generating proofs that would make Erdős proud. The future of automated mathematics isn't just about smarter models—it's about smarter ways to use them. Nomos is that smarter way.

Found this analysis valuable? Star the Nomos repository and share your own benchmarking results with the community.