🔍 Read the full analysis: Z.ai’s Legal AI Fully Passes About 1 In 5 Benchmark Tasks. Why That Still Ranks Near The Top on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh of 81 models. The strict score requires every rubric criterion to pass on a task; Vals also reports a 92.5% pass rate across individual criteria. The benchmark result is not proof of reliable legal work, and a small, non-benchmark spot check reported by ThorstenMeyerAI.com found citations that could not be verified.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, placing seventh among 81 listed models. The score counts a task as passed only when every required criterion is met, so it does not mean the model got just one-fifth of all legal details right; Vals separately reports that it passed 92.5% of individual criteria.
Z.ai describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including research, contract review and drafting. Its documentation says the model underwent legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows. Z.ai recommends using it as an agent connected to services supplying laws, cases and company information. These are the developer’s descriptions of its training and intended uses.
The score comes from Vals AI’s public leaderboard using Harvey’s benchmark and grading protocol. In its archived Version 1 table, GLM Legal is listed at 19.17% across benchmark tasks, or seventh of 81, at a reported $0.15 per test. The leaderboard lists the base GLM-5.3-Flash model at 6.67%. That makes the difference 12.50 percentage points, matching Z.ai’s stated improvement over the base model. The comparison is between entries on the same table; it does not establish performance across all legal work.
Harvey’s benchmark contains 1,250 tasks across 24 legal practice areas, with tasks designed around client materials and a requested work product. The leaderboard uses a private holdout set of 120 tasks, according to Vals. Two language-model judges score submissions under the benchmark’s rubric, with the final task score averaged across their results. A pass requires every criterion for a task to pass; one missed deadline, citation or other item can make the whole task a failure.
What the All-Pass Score Shows
The ranking indicates that GLM Legal performed strongly relative to the models on this particular leaderboard, while the all-pass rate remains below one in five tasks. Those two points can coexist: a model can rank highly in a field where end-to-end task completion is difficult, without being ready to produce unreviewed legal work. The score is not a percentage of legal advice that is safe, accurate or usable.
The gap between the task score and the reported 92.5% individual-criteria pass rate reflects how demanding all-or-nothing grading is. Legal work products often combine facts, conclusions, citations, recommendations, deadlines and formatting requirements. Strong performance on most items may still leave a material error in the final document. For law firms, that makes the benchmark useful as a signal of capability, but insufficient as a substitute for checking the complete work product.
Cost is another point of interest: Vals lists GLM Legal at $0.15 per test, far below several higher-ranked entries in the archived table. That figure is specific to the benchmark run, not a complete estimate of deployment costs, which can include data handling, supporting tools, human review and repeated runs. Firms evaluating the model would need to weigh those costs against accuracy and confidentiality requirements.
Products Worth Considering
Bose QuietComfort Ultra Headphones (2nd Gen), Wireless Bluetooth Noise-Cancelling Over-Ear Headphones, Black
Bose SoundLink Flex Bluetooth Speaker (2nd Gen) – Portable Outdoor Speaker with Hi‑Fi Audio, Waterproof and Dustproof, USB‑C, Up to 12 Hours
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as an open-source benchmark on May 6, 2026. It describes each task as an instruction, a client matter containing relevant material and a requested work product. More than 75,000 expert-written rubric criteria cover items such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting.
Harvey calls its scoring method all-pass grading: a task is complete only when every criterion passes. The company has argued that a report identifying most, but not all, risks may still be materially incomplete. In a May baseline, Harvey said the best overall all-pass rate among the frontier models it tested was 7.1%, and that those models completed less than 10% of tasks end to end in aggregate. That was a separate run and should not be treated as a direct comparison with the later Vals leaderboard.
Z.ai’s AutoClaw blog post announcing the model was dated October 13, 2026, while Vals lists the release date as October 12. Z.ai’s post names Vals as the evaluator. The company’s benchmark statement also calls the result state of the art among open-source models, but the source material does not identify the comparison set, and the legal model’s published weights were not located in the reporting described by ThorstenMeyerAI.com.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark grading rule
Limits Beyond the Leaderboard
The benchmark establishes a result on a 120-task private holdout, not how the model will perform across a law firm’s full range of matters, jurisdictions or client documents. The source material does not provide enough detail to determine whether the holdout represents the distribution of real cases that a particular firm handles. A high ranking on this leaderboard cannot establish suitability for any specific legal workflow.
ThorstenMeyerAI.com also reported a small spot check of four tasks, explicitly not a benchmark. In that check, the author said the legal model cited rulings that could not be found. The author reported that OpenAI and Claude models in the comparison invented no citations, while two Claude models cited real rulings for the opposite of what those rulings held. The sample is too small to support broad comparisons or a reliable estimate of citation error rates.
It remains unclear what comparison set Z.ai used for its “state-of-the-art among open-source models” claim. The reporting also says it found no published weights for the legal model. Separately, Z.ai’s pages reviewed by ThorstenMeyerAI.com described outputs as drafts for legal professionals but did not include a “not legal advice” notice. That observation does not by itself establish the company’s policies or the legal status of model outputs. The sources do not settle how client data is processed, what contractual protections apply, or how users should verify citations before relying on a draft.
What Firms Should Verify
Z.ai’s AutoClaw announcement opened a beta by application, and its documentation recommends connecting the model to services that provide legal authorities and company information. The available source material does not give a schedule for wider access, disclose a further benchmark release, or specify a next evaluation milestone. The next concrete development will depend on whether Z.ai publishes additional documentation or broader access details.
For firms considering a trial, the unanswered operational questions include where client data is processed, which rules and contractual terms govern it, and who checks every factual assertion and citation. The benchmark score supports testing the model under controlled conditions; it does not remove the need for professional review. Further independent evaluations, especially on legal tasks representative of a firm’s practice, would help clarify how the leaderboard result translates into day-to-day use.
Key Questions
What does the 19.17% score mean?
On Vals AI’s leaderboard, it is the share of benchmark tasks the model passed under all-pass grading. Every required criterion on a task had to pass for that task to count as complete.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The criteria rate counts individual rubric items, while the task rate requires every item in a task to pass. A single missed criterion can disqualify a task even when the model passes most of its other items.
What was GLM Legal’s leaderboard rank?
Vals AI’s archived Version 1 leaderboard lists GLM Legal seventh among 81 models, with a score of 19.17% and a reported cost of $0.15 per test.
Does the result show the model is ready to provide legal advice?
No. It measures performance on a defined benchmark, not legal readiness or safety in a firm’s practice. The source material describes the model as intended for professional workflows and reports unresolved questions about citation verification and client-data handling.
Is Z.ai’s open-source performance claim verified?
The leaderboard supports the reported score and rank, but the reporting does not identify the comparison set behind Z.ai’s claim that the model leads among open-source models. ThorstenMeyerAI.com also said it found no published weights for the legal model.
Source: ThorstenMeyerAI.com
