A critique of word similarity as a method for evaluating distributional semantic models

Batchkarov, Miroslav, Kober, Thomas, Reffin, Jeremy, Weeds, Julie and Weir, David (2016) A critique of word similarity as a method for evaluating distributional semantic models. In: The First Workshop on Evaluating Vector Space Representations for NLP, 12th August 2016, Berlin.

[img] PDF - Accepted Version
Download (202kB)


This paper aims to re-think the role of the word similarity task in distributional semantics research. We argue while it is a valuable tool, it should be used with care because it provides only an approximate measure of the quality of a distributional model. Word similarity evaluations assume there exists a single notion of similarity that is independent of a particular application. Further, the small size and low inter-annotator agreement of existing data sets makes it challenging to find significant differences between models.

Item Type: Conference or Workshop Item (Paper)
Keywords: distributional semantics, evaluation
Schools and Departments: School of Engineering and Informatics > Informatics
Subjects: P Language and Literature > P Philology. Linguistics > P0098 Computational linguistics. Natural language processing
Q Science > QA Mathematics > QA0075 Electronic computers. Computer science
Depositing User: Miroslav Batchkarov
Date Deposited: 19 Jul 2016 10:31
Last Modified: 03 Nov 2017 10:58
URI: http://srodev.sussex.ac.uk/id/eprint/62044

View download statistics for this item

📧 Request an update
Project NameSussex Project NumberFunderFunder Ref
A Unified Model of Compositional and Distributional Semantics: Theory and ApplicationsG0853EPSRC-ENGINEERING & PHYSICAL SCIENCES RESEARCH COUNCILEP/IO37458/1