Diana Maynard, Stamatia Dasiopoulou, Stefania Costache, Kai Eckert, Heiner Stuckenschmidt, Martin Dzbor and Siegfried Handschuh

Knowledge Web Project, 2007

This deliverable investigates methods and results for benchmarking annotation tools. We define first some criteria for benchmarking, including both performance and usability issues, and examine those factors which are particularly important for a user in an industrial setting to be able to determine which is the most suitable tool for their use. We also look particularly at two issues: scalability of the tool, which is most important if the tools are to be used in a real industrial setting rather than just as research prototypes; and the best way to evaluate performance. We then perform a series of experiments on the annotation tools, and discuss the results, finally drawing some conclusions about the future of annotation tools.

Download