Kai Eckert, Magnus Pfeffer and Heiner Stuckenschmidt

JCDL '08: Proceedings of the 2008 conference on Digital libraries, June 16-20 2008, Pittsburgh, PA, USA

This deliverable investigates methods and results for benchmarking annotation tools. We define first some criteria for benchmarking, including both performance and usability issues, and examine those factors which are particularly important for a user in an industrial setting to be able to determine which is the most suitable tool for their use. We also look particularly at two issues: scalability of the tool, which is most important if the tools are to be used in a real industrial setting rather than just as research prototypes; and the best way to evaluate performance. We then perform a series of experiments on the annotation tools, and discuss the results, finally drawing some conclusions about the future of annotation tools.

Download