Sapienza NLP @ ACL 2023
8 papers at ACL!
We are happy to announce that our group has 8 publications at ACL 2023!
Our works cover a set of diverse topics such as descriptive language modeling, the meaning of superhuman performances in NLP semantic parsing, book summarization, relation extraction, and AMR alignment .
- DMLM: Descriptive Masked Language Modeling
- What's the Meaning of Superhuman Performance in Today's NLU?
- Exploring Non-Verbal Predicates in Semantic Role Labeling
- REDFM: a Filtered and Multilingual Relation Extraction Dataset
- Echoes from Alexandria: A Large Resource for Multilingual Book Summarization
- Incorporating Graph Information in Transformer-based AMR Parsing
- AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing
- Cross-lingual AMR Aligner: Paying Attention to Cross-Attention
DMLM: Descriptive Masked Language Modeling
by E. Barba, N. Campolungo, R. Navigli
Over the last few years, Masked Language Modeling (MLM) pre-training has resulted in remarkable advancements in many Natural Language Understanding (NLU) tasks, which has sparked an interest in researching alternatives and extensions to the MLM objective. In this paper, we tackle the absence of explicit semantic grounding in MLM and propose Descriptive Masked Language Modeling (DMLM), a knowledge-enhanced reading comprehension objective, where the model is required to predict the most likely word in a context, being provided with the word’s definition. For instance, given the sentence “I was going to the _”, if we provided as definition “financial institution”, the model would have to predict the word “bank”; if, instead, we provided “sandy seashore”, the model should predict “beach”. Our evaluation highlights the effectiveness of DMLM in comparison with standard MLM, showing improvements on a number of well-established NLU benchmarks, as well as other semantics-focused tasks, e.g., Semantic Role Labeling. Furthermore, we also demonstrate how it is possible to take full advantage of DMLM to embed explicit semantics in downstream tasks, explore several properties of DMLM-based contextual representations and suggest a number of future directions to investigate.
What's the Meaning of Superhuman Performance in Today's NLU?
by S. Tedeschi, J. Bos, T. Declerck, J. Hajic, D. Hershcovich, E. H. Hovy, A. Koller, S. Krek, S. Schockaert, R. Sennrich, Ekaterina S., R. Navigli
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.
Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities
by R. Orlando, S. Conia, R. Navigli
Although we have witnessed impressive progress in Semantic Role Labeling (SRL), most of the research in the area is carried out assuming that the majority of predicates are verbs. Conversely, predicates can also be expressed using other parts of speech, e.g., nouns and adjectives. However, non-verbal predicates appear in the benchmarks we commonly use to measure progress in SRL less frequently than in some real-world settings – newspaper headlines, dialogues, and tweets, among others. In this paper, we put forward a new PropBank dataset which boasts wide coverage of multiple predicate types. Thanks to it, we demonstrate empirically that standard benchmarks do not provide an accurate picture of the current situation in SRL and that state-of-the-art systems are still incapable of transferring knowledge across different predicate types. Having observed these issues, we also present a novel, manually-annotated challenge set designed to give equal importance to verbal, nominal, and adjectival predicate-argument structures. We use such dataset to investigate whether we can leverage different linguistic resources to promote knowledge transfer. In conclusion, we claim that SRL is far from “solved”, and its integration with other semantic tasks might enable significant improvements in the future, especially for the long tail of non-verbal predicates, thereby facilitating further research on SRL for non-verbal predicates. We release our software and datasets at https://github. com/sapienzanlp/exploring-srl .
REDFM: a Filtered and Multilingual Relation Extraction Dataset
by P. H. Cabot, S. Tedeschi, A. Ngonga, R. Navigli
Relation Extraction (RE) is a task that identifies relationships between entities in a text, enabling the acquisition of relational facts and bridging the gap between natural language and structured knowledge. However, current RE models often rely on small datasets with low coverage of relation types, particularly when working with languages other than English. In this paper, we address the above issue and provide two new resources that enable the training and evaluation of multilingual RE systems. First, we present SREDFM, an automatically annotated dataset covering 18 languages, 400 relation types, 13 entity types, totaling more than 40 million triplet instances. Second, we propose REDFM, a smaller, human-revised dataset for seven languages that allows for the evaluation of multilingual RE systems. To demonstrate the utility of these novel datasets, we experiment with the first end-to-end multilingual RE model, mREBEL, that extracts triplets, including entity types, in multiple languages. We release our resources and model checkpoints at https://www.github.com/babelscape/rebel.
Echoes from Alexandria: A Large Resource for Multilingual Book Summarization
by A. Scirè, S. Conia, R. Navigli
In recent years, research in text summarization has mainly focused on the news domain, where texts are typically short and have strong layout features. The task of full-book summarization presents additional challenges which are hard to tackle with current resources, due to their limited size and availability in English only. To overcome these limitations, we present "Echoes from Alexandria", or in shortened form, "Echoes", a large resource for multilingual book summarization. Echoes features three novel datasets: i) Echo-Wiki, for multilingual book summarization, ii) Echo-XSum, for extremely-compressive multilingual book summarization, and iii) Echo-FairySum, for extractive book summarization. To the best of our knowledge, Echoes, with its thousands of books and summaries, is the largest resource, and the first to be multilingual, featuring 5 languages and 25 language pairs. In addition to Echoes, we also introduce a new extractive-then-abstractive baseline, and, supported by our experimental results and manual analysis of the summaries generated, we argue that this baseline is more suitable for book summarization than purely-abstractive approaches. We release our resource and software at https://github.com/Babelscape/echoes-from-alexandria in the hope of fostering innovative research in multilingual book summarization.
Incorporating Graph Information in Transformer-based AMR Parsing
by P. Vasylenko, P. H. Cabot, A. C. Martínez Lorenzo, R. Navigli
Abstract Meaning Representation (AMR) is a
Semantic Parsing formalism that aims at providing a semantic graph abstraction representing
a given text. Current approaches are based on
autoregressive language models such as BART
or T5, fine-tuned through Teacher Forcing to
obtain a linearized version of the AMR graph
from a sentence. In this paper, we present
LeakDistill, a model and method that explores
a modification to the Transformer architecture,
using structural adapters to explicitly incorporate graph information into the learned representations and improve AMR parsing performance. Our experiments show how, by
employing word-to-node alignment to embed
graph structural information into the encoder
at training time, we can obtain state-of-the-art
AMR parsing through self-knowledge distillation, even without the use of additional data.
We release the code at http://www.github.com/sapienzanlp/LeakDistill
AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing
by A. C. Martínez Lorenzo, P. H. Cabot, R. Navigli
In this paper, we examine the current state-ofthe-art in AMR parsing, which relies on ensemble strategies by merging multiple graph predictions. Our analysis reveals that the present models often violate AMR structural constraints. To address this issue, we develop a validation method, and show how ensemble models can exploit SMATCH metric weaknesses to obtain higher scores, but sometimes result in corrupted graphs. Additionally, we highlight the demanding need to compute the SMATCH score among all possible predictions. To overcome these challenges, we propose two novel ensemble strategies based on Transformer models, improving robustness to structural constraints, while also reducing the computational time. Our methods provide new insights for enhancing AMR parsers and metrics. Our code is available at github.com/babelscape/AMRsAssemble .
Cross-lingual AMR Aligner: Paying Attention to Cross-Attention
by A. C. Martínez Lorenzo, P. H. Cabot, R. Navigli
This paper introduces a novel aligner for Abstract Meaning Representation (AMR) graphs that can scale cross-lingually, and is thus capable of aligning units and spans in sentences of different languages. Our approach leverages modern Transformer-based parsers, which inherently encode alignment information in their cross-attention weights, allowing us to extract this information during parsing. This eliminates the need for English-specific rules or the Expectation Maximization (EM) algorithm that have been used in previous approaches. In addition, we propose a guided supervised method using alignment to further enhance the performance of our aligner. We achieve state-of-the-art results in the benchmarks for AMR alignment and demonstrate our aligner’s ability to obtain them across multiple languages. Our code will be available at github.com/SapienzaNLP/amr-alignment.