TL;DR — DEEPSYNTH is a benchmark of 120 expert-curated tasks across 7 domains and 67 countries that evaluates LLM agents on multi-step web information synthesis. State-of-the-art systems reach only 8.97 F1 / 17.5 LLM-Judge — leaving plenty of headroom.

• Leaderboard
• Quickstart
• Dataset
• Evaluation
• Submitting to the Leaderboard
• Baselines
• Citation

Live, interactive leaderboard: huggingface.co/spaces/DeepSynthesisTeam/deepsynth-leaderboard

🔒 closed model · 🔓 open-weights. Ranked by F1; LLM Judge used as tiebreaker.

Self-reported numbers on the public dev split. Anyone can score themselves locally with the gold answers released alongside the questions.

📈 Submit your agent → Submit tab on the leaderboard or open a PR to this repo.

# 1. Clone
git clone https://github.com/agentdeepsynthesis/deepsynth-bench
cd deepsynth-bench

# 2. Install
pip install -r requirements.txt

# 3. Download the dev set from Hugging Face
python -c "
from huggingface_hub import hf_hub_download
import json

dev_path = hf_hub_download(
    repo_id='DeepSynthesisTeam/deepsynth-bench',
    filename='DEEPSYNTH_lite.json',
    repo_type='dataset',
)
with open(dev_path) as f:
    tasks = json.load(f)
print(f'Loaded {len(tasks)} dev tasks')
"

# 4. Evaluate a sample prediction file
python scripts/evaluation/eval_static_score.py \
    --predictions examples/sample_predictions.json \
    --split dev

DEEPSYNTH is hosted on the Hugging Face Hub: DeepSynthesisTeam/deepsynth-bench

The benchmark ships as 120 expert-curated tasks in two splits:

from huggingface_hub import hf_hub_download
import json

path = hf_hub_download(
    repo_id="DeepSynthesisTeam/deepsynth-bench",
    filename="DEEPSYNTH_lite.json",
    repo_type="dataset",
)
tasks = json.load(open(path))

DEEPSYNTH reports five complementary metrics:

Run evaluation locally:

python scripts/evaluation/eval_static_score.py \
    --predictions your_predictions.json \
    --split dev

{
  "001": {"Sweden": 1.2, "Finland": 0.8},
  "002": {"Brunei": -0.67, "Singapore": -0.34}
}

For leaderboard submissions (test split), wrap predictions in the full submission schema — see below.

We accept submissions two ways — pick whichever is easier:

Fastest path. Go to the Submit tab on the Hugging Face Space, fill in your agent's metadata, upload your predictions JSON, and submit. A maintainer reviews and scores it within about a week.

1. Produce a submission JSON conforming to scripts/evaluation/submission_schema.json — metadata block + predictions map.
2. Validate locally:

   python scripts/evaluation/validate_submission.py my_submission.json --strict

3. Fork this repo, add your file under submissions/YYYY-MM-DD-org-agentname.json, and open a PR.
4. CI validates the schema; a maintainer reviews and merges.

• Public code_url that reproduces your numbers. Submissions without reproducible code won't be accepted.
• Honest metadata. Misreporting scaffold or tool access is grounds for retraction.
• We may request a run trace for spot-check verification.

Email the contact on the submission. Corrections and retractions are logged transparently in the leaderboard history.

Runnable baseline agents live in scripts/baselines/:

• vanilla/ — LLM-only, no tools.
• react/ — ReAct with web search and Python.
• codeact/ — CodeAct style with full Python sandbox.
• plan_and_execute/ — Two-stage planner + executor.

Each baseline has its own README.md with setup, API key requirements, and an end-to-end run command.

If you use DEEPSYNTH in your research, please cite:

@inproceedings{paul2026deepsynth,
  title     = {{DEEPSYNTH}: A Benchmark for Deep Information Synthesis},
  author    = {Debjit Paul and Daniel Murphy and Milan Gritta and Ronald Cardenas and Victor Prokhorov
               and Lena Sophia Bolliger and Aysim Toker and Roy Miles and Andreea-Maria Oncescu and
               Jasivan Alex Sivakumar and Philipp Borchert and Ismail Elezi and Meiru Zhang and
               Ka Yiu Lee and Guchun Zhang and Jun Wang and Gerasimos Lampouras},
  booktitle = {The Fourteenth International Conference on Learning Representations (ICLR)},
  year      = {2026},
  url       = {https://openreview.net/forum?id=0Dhpt9aY3n}
}

A CITATION.cff file is provided for GitHub's "Cite this repository" button.

Apache License 2.0. See LICENSE.

Developed across Huawei Noah's Ark Lab, Imperial College London, UCL Centre for AI, University of Zurich, University of Sheffield, and University of Cambridge.

Disclaimer: This open-source project is not an official Huawei product; Huawei is not expected to provide support.