Scale Labs (@ScaleAILabs) / X

ICML 2026 is almost here, and we're headed to Seoul with 7 accepted papers. 🇰🇷 Our research spans agent training, scientific reasoning, and evaluations advancing the next generation of AI systems. Here's an early look at the work we'll be presenting 🧵

2.2K

Scale Labs

@ScaleAILabs

Jul 2

Replying to @ScaleAILabs

RubricRobustness: Evaluating the Sensitivity of Rubrics-Based Benchmarks to Simple Perturbations openreview.net/forum?id=2Y3I3…

Connect with us at ICML: scale.com/events/icml-20…

Less than a year ago, we launched the Remote Labor Index with @CAIS to measure how well AI agents can complete paid freelance work. Since then, the best automation rate has jumped from less than 3% to 16%. We've also added results for three frontier models to the leaderboard:

00:00

Full leaderboard:

Remote Labor Index (RLI)

AI is transforming drug discovery, but until now there hasn't been an independent benchmark for the computational tasks behind early-stage research. Together with @phylo_bio, we built DrugDiscoveryBench to evaluate how today's leading AI models perform on the core tasks

00:00

More on what we learned: scale.com/blog/drugdisco…

392

Scale Labs reposted

Mohit Raghavendra (@ ACL)

@mohit_r9a

Jun 30

📝 New research from @scale_AI Frontier SWE benchmarks are usually single-turn, one-shot tasks: the agent gets a detailed spec upfront, then implements autonomously. That is not how most real coding-agent workflows feel. Introducing SWE-Interact. 🧵

GLM 5.2 is now live across all three SWE Atlas leaderboards: Codebase QnA, Test Writing, and Refactoring. When we launched SWE Atlas a few months ago, open models were significantly behind frontier closed models, clustering at the bottom. GLM 5.2 has leapfrogged this gap, and is

More results across SWE Atlas leaderboards:

AI Model Leaderboards & Benchmarks

From labs.scale.com

1.3K

Scale Labs reposted

MohammadHossein Rezaei

@mhrezaeics

Jun 12

Rubrics are becoming the standard way to train/evaluate LLMs on open-ended tasks. But rubric-RL has a bottleneck: every rollout needs to be graded by an LLM verifier. That’s expensive, slow, and is prone to reward hacking. At the same time, the field is moving toward on-policy

Rubrics have emerged as an alternative to RLVR in open-ended domains where a single groundtruth final answer is not available. Existing rubric-based training methods rely on an LLM verifier
that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory
signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student.
RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge
from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based
GRPO while using one on-poli

Excited to share a new @ScaleAILabs research in collaboration with @phylo_bio on coding agents for drug-discovery research! 💊 We ran Claude Code, Codex, and Gemini on 60+ expert-curated drug-discovery tasks inside a shared Biomni-powered biomedical research environment and the

How do you turn agent traces into an improvement flywheel? Excited to share Insights Generator (IG) — new @scale_AI / @ScaleAILabs research that finds behavioral patterns and bugs in agent traces. Engineers & coding agents using IG achieved 30+% gains on agent benchmarks. 🧵

Today we're releasing HiL-Dynamics, the first open-source tool that measures how production agents actually collaborate with humans under uncertainty. Not just whether they got the answer. Now you can measure exactly when your agent asks for help, when it makes assumptions, and

Replying to @ScaleAILabs

Selective escalation remains one of the biggest challenges for reliable human-in-the-loop AI. We hope HiL-Dynamics helps users find the right setup for their workflows and gives model builders clearer signals for building agents that collaborate with humans more effectively.

HiL-Dynamics: github.com/melfeki-11/HiL… Blog: labs.scale.com/blog/hil-dynam…

GitHub - melfeki-11/HiL-Dynamics: Does your coding agent know when it doesn't know? HiL-Dynamics...

From github.com

389