Sapienza NLP @ LREC 2026
Two papers accepted at LREC 2026!
We're delighted to announce the acceptance of 2 papers to LREC 2026, focusing on LLM Safety and Machine Translation.
Our contributions:
Mind the Language Gap: Assessing LLM Safety in Italian
by E. Marafatto and R. Navigli
The rapid diffusion of Large Language Models (LLMs) across linguistic and cultural contexts underscores the need for systematic safety evaluations beyond English. As LLMs are increasingly applied in multilingual settings, ensuring their safe and appropriate behavior in other languages is essential. This paper presents a methodology for building safety evaluation datasets that comprehensively cover the full spectrum of sensitive topics relevant to LLM safety. The resulting resources include a collection of Italian Wikipedia pages encompassing all major categories of sensitive content, and a companion dataset containing three challenging Italian-language questions per page designed to probe model behavior on high-risk issues. Each prompt was annotated into four safety outcome categories: correct refusal, safe informative, unsafe, and ambiguous. Together, these datasets provide a robust foundation for evaluating and benchmarking LLM safety in Italian. To demonstrate their utility, we used them to assess four LLMs, identifying systematic differences in refusal consistency and compliance across sensitive domains.
Cultural and Knowledge Biases in LLMs through the Lens of Entity-Aware Machine Translation
by L. Xu, L. Moroni, and R. Navigli
Large Language Models (LLMs) demonstrate strong multilingual capabilities yet exhibit systematic cultural biases that affect entity-aware machine translation. While external knowledge integration improves translation accuracy, the extent of these benefits across varying degrees of cultural specificity remains unexplored. We propose a three-level cultural specificity framework: Culturally Agnostic, Culturally Sensitive, and Culturally Local, to systematically analyze how cultural context affects entity translation difficulty and the utility of external knowledge. Through experiments spanning 11 LLMs and 10 languages, we demonstrate that external knowledge provides substantially greater improvements for culturally local entities (up to 70% in m-ETA) compared to culturally agnostic ones. Our analysis reveals distinct behavioral patterns across model tiers: closed and open-weight models show synergistic improvements in both entity accuracy and overall translation quality, while open-data models struggle with instruction-following despite improved entity accuracy.