3 2009 13th European Conference on Digital Libraries (ECDL 2009) Corfu, Greece 09/2009 In digital libraries semantic techniques are often deployed to reduce the expensive manual overhead for indexing documents and maintaining meta-data, as well as later information searches. However, using such techniques may cause a decrease in a collection?s quality due to their statistical nature. Since da-ta quality is a major concern in digital libraries, it is important to be able to measure the (loss of) quality of metadata automatically generated by semantic techniques. In this paper we present a user study based on a typical semantic technique used for automatic metadata creation, namely taxonomies of author keywords and tag clouds. We observed experts assessing typical relations be-tween keywords and documents over a small corpus in the field of chemistry. Based on the evaluation of this experiment we focused on communalities be-tween the experts? perception and thus draw a first roadmap on how to evaluate semantic techniques by proposing some preliminary metrics.