Pinned
Introducing the FrontierCyber benchmark: Irregular’s new approach to advanced offensive-cyber evaluations. It measures AI models’ offensive skills on real systems, including mobile devices, hosted software services, databases, and networks.
Irregular
140 posts
- We worked with @OpenAI to evaluate GPT-5.6 Sol, including the first deployment of FrontierCyber as part of a frontier model assessment with a partner. FrontierCyber measures offensive-cyber capability on real, off-the-shelf systems, with no planted vulnerabilities and no
Replying to @IrregularGPT-5.6 Sol demonstrated capability slightly stronger than GPT-5.5. It discovered vulnerabilities more consistently than it could compose them into reliable attack paths under production defenses, with clear limitations against hardened targets and over long horizons.
- Replying to @IrregularInitial evaluations are already surfacing previously unknown vulnerabilities, now moving through responsible disclosure. For example, a model built a novel multi-vulnerability chain to gain unauthorized access to private information on a widely used mobile device.
- At @ManGroup's Technology Offsite this week, our CEO @dan_lahav gave the keynote on frontier AI security risk as a category of its own, alongside classical cybersecurity. The tools we defend networks with were built for systems that follow rules. AI systems reason toward a goal,
- Most 'world-changing' AI ideas are about what these systems can do. Ours is about whether you can trust them to do it. We’re proud to be a winner on @FastCompany’s 2026 World Changing Ideas list, alongside the labs and teams betting on getting this right.
Full Fast Company writeup: fastcompany.com/91547245/pushi… The World Changing Ideas list: fastcompany.com/world-changing… - We’re happy to share that CyScenarioBench, our benchmark for offensive cyber operations, was used by @AnthropicAI to test Claude Mythos 5 and Claude Fable 5. Most current cybersecurity evaluations check isolated skills, such as vulnerability research or exploitation.
More about Irregular's CyScenarioBench here: - The New York Times covered new research from the University of Toronto on AI-powered worms. Speaking to @nytimes, our CEO @dan_lahav highlighted the gap between lab demonstrations and real-world cyber impact: reliability, complexity, and defenses. At Irregular, we work on
- Honored to be the main sponsor of CyberML 2026, a leading technical conference dedicated to the intersection of cybersecurity and machine learning. Our co-founder and CTO, Omer Nevo, opens with the keynote "Artificial Attackers: Risks, Capabilities and Mitigations.” Swing by our
- Thrilled to be recognized in @Redpoint's 2026 InfraRed 100, highlighting 100 of the most promising private companies in AI infrastructure. This recognition is a powerful validation of our mission: to protect the world as AI systems become increasingly capable and sophisticated.
