A recent study by the Institute for Artificial Intelligence and Computer Technologies (INSAIT) at Sofia University, in collaboration with LatticeFlow, revealed significant compliance gaps in distilled DeepSeek models with the European Artificial Intelligence Act (EU AI Act). The research, conducted with ETH Zurich, focused on testing various artificial intelligence models, including those from DeepSeek, to assess their adherence to European regulations.

Distilling large AI models like DeepSeek into smaller versions is a common practice that enhances their practicality and efficiency for business use. However, the study found that although DeepSeek's distilled models performed well in minimizing toxic content, they failed to meet crucial compliance standards in areas such as cybersecurity and bias management. These deficiencies raise concerns about the readiness of these models for deployment in business environments.

The evaluation covered two widely used DeepSeek models: DeepSeek R1 8B (based on Meta’s Llama 3.1 8B) and DeepSeek R1 14B (based on Alibaba’s Qwen 2.5 14B), both of which have amassed nearly 400,000 downloads. The findings highlighted that DeepSeek’s distilled models ranked poorly in terms of cybersecurity compared to other models tested by COMPL-AI, such as those from OpenAI, Meta, and Google. Specifically, these models exhibited heightened risks of "goal hijacking" and "prompt leakage," both of which pose security threats. Goal hijacking refers to the potential for AI models to be manipulated into performing unintended actions, while prompt leakage increases the likelihood of confidential information being exposed. These vulnerabilities lower the models' reliability, making them unsuitable for secure business environments.

Additionally, the DeepSeek models performed poorly in managing biases, showing greater prejudice than their original versions. Bias in AI models refers to the degree to which they exhibit fairness, neutrality, and objectivity across different social, cultural, ethnic, gender, and political groups. The study found that the distilled models showed higher levels of bias compared to the baseline models, suggesting that the distillation process had worsened their bias performance. This could result in the models producing unbalanced responses on sensitive issues and potentially contributing to the spread of misinformation.

On a positive note, the DeepSeek models excelled in controlling toxicity, outperforming their baseline models in this regard. Toxicity in AI models refers to the ability to detect and prevent harmful, offensive, or discriminatory content. The DeepSeek models were found to be more effective at limiting such content than the original Meta and Alibaba models, making them more compliant with the EU AI Act’s requirements for ethical AI use.

Despite their strengths in toxicity management, the DeepSeek models’ shortcomings in cybersecurity and bias make them less suited for deployment in regulated environments, such as those governed by the European Union's AI legislation.

Source: INSAIT press release