🔍 Read the full analysis: We Gave 42 AI Setups The Same 15 Legal Questions. The Model Built For Law Scored 8 Out Of 100 on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, reports that GLM-5.3 FlashX Legal scored 8 out of 100 across 15 questions, while Claude Opus 5.5 led with 79. The operator says the test is a spot check, discloses that Claude agents graded the answers and publishes its materials for review; its results should not be treated as a definitive ranking of legal AI systems.
In results published November 4, 2026, the legal AI model GLM-5.3 FlashX Legal scored 8 out of 100 on Lexassay, a test run by Thorsten Meyer AI that gave 42 AI setups the same 15 legal questions. The result is a warning about relying on a chatbot for legal work, but it is not a definitive ranking: the test operator disclosed that Claude agents graded the answers, and described the exercise as a spot check rather than a strict benchmark.
According to the published results, GLM-5.3 FlashX Legal fully solved none of the 15 tasks, scored zero on the hidden questions and provided no answer on two tasks. Its citation check found 21 citations judged real, 25 misattributed and one invented. The model’s maker, Z.ai, describes it as optimized for professional legal workflows. The test says its general-purpose base model scored 3, so the legal training produced a small improvement under this evaluation.
The test operator says the questions cover German, US and EU law, including appeal deadlines, legal changes, court decisions and questions with false premises. Models answered without internet access or tools. Each answer was graded against a reference answer, and citations were checked against official databases. Lexassay deducted points for citations that were invented or did not support the answer, as well as for answers lacking a legal-advice disclaimer or recommendation to consult a lawyer.
Other scores in the report varied widely. Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71; the report says the leading model fully met every criterion on eight of the ten public tasks. Mistral Large 4 scored 10. Meyer reports that, across published runs, Lexassay confirmed 2,644 citations as real and found 804 that were invented or did not support the claims made. Those counts come from the operator’s test and are not a measure of every legal AI system or use.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Errors Can Cost a Case
The test highlights a practical risk: an AI answer can sound specific while getting a legal deadline, rule or citation wrong. In the example Meyer gives, two German clients receive civil judgments on April 27, 2027, and want to appeal. The reported calculation puts the Munich filing deadline on May 28 because May 27 is a public holiday in Bavaria, while Berlin’s deadline remains May 27. Meyer says the written grounds are due June 28 in both cases. A late appeal can be rejected as inadmissible under German civil procedure, making a one-day error consequential.
That example is an illustration in the source article, not a substitute for advice on an actual case. The broader finding is that citation accuracy remains a concern: the operator says some references were misattributed or fabricated, and those errors can lend false authority to an answer. Readers using AI to understand a legal issue may find it useful as a starting point, but should verify each cited law or ruling in the source and consult a qualified lawyer about deadlines, contracts, claims or dealings with authorities.
The results also show why a single score needs careful interpretation. A model can perform differently on a test where it must answer from its own stored knowledge than on a system that can search supplied documents. Lexassay’s results should inform questions about reliability, not be taken as proof that one product will perform the same way on a particular person’s case.
Products Worth Considering
EcoFlow Delta 3 Portable Power Station – 1024 Wh, 1800 W
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Apple AirPods Pro 3 Wireless Earbuds with Active Noise Cancellation
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay gave participating setups 15 questions: ten public and five hidden. According to Meyer, the hidden questions are intended to reduce the chance that future models can find answers online; only their scores are published. The test spans three legal systems and includes both ordinary legal questions and traps based on false assumptions, where a sound answer should correct the premise rather than accept it.
The operator says it tested models without web access or external tools and evaluated different reasoning-effort settings separately. That means the 42 setups do not necessarily represent 42 distinct underlying models. Each answer was scored against a reference answer with three to six weighted criteria. The reported citation categories include real, misattributed, invented and not-findable references. The test also applied a five-percent penalty when an answer omitted a legal-advice warning or direction to a lawyer.
Meyer’s disclosure says Thorsten Meyer AI runs Lexassay, and several Claude agents graded the answers, even though a Claude model ranked first. The operator says its method page sets out the conflict and that public answers, reference answers and citation checks are available for scrutiny. The report also points to a separate Vals AI legal-agent benchmark, where models work with supplied documents and Mistral Large 4 ranks ahead of some models that placed higher on Lexassay. The differing results reflect different test designs, not a direct contradiction.
““Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.””
— Z.ai, as quoted in the Lexassay report
Limits of the Reported Scores
Lexassay’s figures are the operator’s own test results, and the source does not establish that they predict performance on real client matters. Meyer explicitly calls the exercise a spot check: one answer was tested for each model and task, so variation between runs and small differences in points may matter. The test excludes internet access and tools, while many legal AI products may rely on document retrieval or other features in practice.
The disclosed grading arrangement also warrants attention: Claude agents graded the answers, and a Claude model ranked first. The operator says it publishes its method and underlying materials so readers can inspect the grades, but the supplied report does not provide an independent audit of the scoring. It is also not clear from the material provided how the 42 setups were distributed across model versions and reasoning settings, or how results would change with repeated runs.
The article’s date and scenario describe a hypothetical future legal deadline, not a report that those clients or cases exist. The source includes only part of its discussion of GLM-5.3 FlashX Legal and does not provide enough information to assess every model’s performance or the full comparison with the separate Vals AI benchmark.
Verify Before Acting on AI Advice
The immediate next step for readers is not to treat a score as legal guidance. For a real deadline or legal dispute, check the rule and cited authority directly and speak with a lawyer. Meyer’s published method and answer materials give readers a way to examine the reported scoring, while future evaluations could clarify whether results hold across repeated runs, newer model versions and systems that can consult documents or search the web.
The source does not announce a date for another Lexassay test or an independent review. Until further testing is available, the reported results should be understood as a dated, limited evaluation of the tested setups under the stated conditions. Model updates and different tools may change performance, so the scores should not be assumed to describe versions or products tested elsewhere.
Key Questions
What did the law-focused model score?
GLM-5.3 FlashX Legal scored 8 out of 100 on Lexassay. The operator says it fully solved none of the 15 tasks and scored zero on the hidden set.
Which model scored highest on Lexassay?
The report places Claude Opus 5.5 first with 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The operator disclosed that Claude agents graded the answers, which is a conflict of interest readers should factor into their assessment.
Does this test show which AI is best for legal work?
No. Lexassay describes its results as a spot check, not a strict benchmark. It tested models without internet access or tools and used one answer per model-task combination. Results on other tests can differ when models are given documents or search tools.
Can I use a chatbot to calculate a legal deadline?
The report advises readers to never rely on a chatbot alone for a deadline. Verify the applicable law and dates with an authoritative source and consult a lawyer about an actual case, because a missed deadline can affect the right to proceed.
Source: ThorstenMeyerAI.com
