Security
Preprint introduces CryptanalysisBench for testing AI cryptanalysis
A new preprint describes CryptanalysisBench, a 191-task benchmark designed to measure whether large language model agents can find weaknesses in cryptographic algorithms.
arXiv
What happened
The research preprint “CryptanalysisBench: Can LLMs do Cryptanalysis?” was first submitted to arXiv on July 20, 2026. It introduces 191 tasks across six families of cryptographic primitives, mainly drawn from four NIST standardization competitions.
The project’s public repository says model agents write attack scripts for formal security games, which are tests with defined rules. Automated evaluation then runs those scripts against fresh game instances to check whether an attack succeeds.
The authors report that five frontier models broke 65%–86% of Tier 1 schemes, 6–12 Tier 2 schemes at full strength, and 24–61 scaled-down variants. These are benchmark results, not evidence that deployed cryptography has been broken.
Why it matters
The benchmark could provide a structured way to track AI cryptanalysis capabilities and test candidate cryptographic schemes before deployment. Its authors present those as intended uses, while the available evidence does not establish that it improves real-world risk assessment or predicts future capability.
What remains unclear
- The work is an arXiv preprint, and the independent commentary located discussed the findings without reproducing the experiments.
- Contributor descriptions differ across sources. Anthropic named ETH Zurich, Tel Aviv University, and TU Berlin, while an independent report also identified the University of Haifa among contributors.
Sources
-
CryptanalysisBench: Can LLMs do Cryptanalysis?
Used for: Benchmark introduction, task count, corpus composition, and author-reported model results.
-
Discovering cryptographic weaknesses with Claude
Used for: Anthropic’s stated academic collaboration and intended capability-tracking context.
-
GitHub - ethz-spylab/cryptanalysis-benchmark
Used for: Published description of the formal-game and automated-validation framework.
-
August 15, 2026
Used for: Independent context that the results remain early and were not a reproduction study.
-
Claude AI Just Cracked a Post-Quantum Test Scheme and Found a Faster 7-Round AES Attack
Used for: Independent reporting on the listed benchmark contributors.