📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM was built from scratch with extensive native-language data but underperformed on Italian academic benchmarks. This challenges assumptions about scale and investment in sovereign-language models.
Italy’s Minerva-3B, a large language model trained entirely from scratch on 2.5 trillion tokens with about 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite its extensive native-language training.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, has been recognized for its ambitious scale and open approach. It trained models ranging from 350 million up to 7 billion parameters, with the 7B version trained on over a trillion Italian tokens, making it one of the largest native-language LLM efforts in Europe.
While Minerva’s technical performance on benchmarks surpasses many multilingual models in Italian tasks, its performance on the INVALSI exams—an established measure of Italian academic language—remains strikingly low. The 3B model’s 4.9% score is near chance levels, indicating a significant gap between technical capability and real-world language understanding. Researchers attribute this to the broader finding that dataset size and parameter count are more critical than language-specific data alone for complex language tasks, a lesson that complicates the narrative that more native-language data alone guarantees higher performance.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language AI Development
The results from Minerva suggest that simply scaling up native-language data and model size may not be sufficient to achieve meaningful language comprehension in specialized contexts such as education. This raises important questions for European countries investing heavily in sovereign-language AI: what scale of investment is truly necessary to develop models capable of understanding and performing complex language tasks? The findings challenge assumptions that more data and larger models automatically lead to better real-world performance, emphasizing the need for strategic focus on data quality, task-specific training, and possibly new architectural approaches.
For policymakers and researchers, these insights highlight the importance of realistic benchmarks and performance metrics that reflect actual language use and understanding, not just technical scores. The European sovereign-LLM movement may need to reassess its scaling strategies and consider more nuanced approaches to achieve country-specific language mastery.
European Sovereign-Language Model Strategies and Challenges
Italy’s Minerva project emerged as a significant effort to develop a European sovereign LLM from scratch, aiming to demonstrate that large-scale native-language models are feasible and effective. Funded through Italy’s national AI strategy and utilizing CINECA’s supercomputing infrastructure, Minerva trained on 2.5 trillion tokens with half Italian content, resulting in high technical benchmarks but disappointing results on real-world academic tests.
In contrast, the Portuguese AMÁLIA model took a different approach by extending a multilingual foundation with a smaller proportion of European Portuguese data, raising debates about the effectiveness of continuation pre-training versus scratch training. Both projects exemplify the broader European debate on how best to develop sovereign-language AI, with Minerva representing the ‘from scratch’ camp and AMÁLIA exemplifying the ‘layered specialization’ approach.
Recent empirical results, especially Minerva’s low exam score, challenge the assumption that larger native-language datasets automatically produce deeper country-specific knowledge, prompting a reevaluation of strategic priorities across Europe.
Unresolved Questions About Model Performance and Scaling
It remains unclear whether further scaling of Minerva—either through increasing parameters, data quality, or specialized training—can improve its performance on complex language tasks such as academic assessments. Additionally, the extent to which these findings generalize to other languages and domains is still being studied. The ongoing development of Minerva and related models will clarify whether the low exam score is an anomaly or indicative of a broader limitation in current scaling strategies.
Next Steps for European Sovereign-Language AI Projects
The Minerva team is continuing to iterate on training methodologies, with recent experiments in continual training and fine-tuning. Future evaluations will assess whether targeted approaches can bridge the gap between technical benchmarks and real-world language understanding. Policymakers and researchers are likely to reassess investment strategies, emphasizing not just scale but also data quality and task-specific training. Further public benchmarks and real-world tests will inform the strategic direction of Europe’s sovereign-LM development efforts.
Key Questions
Why did Minerva perform poorly on Italian school exams despite large-scale training?
The low performance suggests that dataset size and parameters alone are insufficient; effective understanding may require more targeted training, higher data quality, or architectural innovations. It indicates that scale does not automatically translate into real-world language proficiency.
Does this mean European sovereign LLMs are not worth pursuing?
Not necessarily. The results highlight challenges and areas for improvement but do not negate the strategic importance of developing native-language models. They suggest that scale must be complemented with other strategies for meaningful progress.
What are the implications for other languages and domains?
The findings from Minerva may generalize to other languages with less training data or more complex linguistic structures, emphasizing the need for tailored approaches beyond just increasing data and model size.
Will further scaling improve Minerva’s performance?
It remains an open question. Ongoing research aims to determine whether additional scale, data refinement, or architectural changes can enhance performance on complex tasks.
Source: ThorstenMeyerAI.com
