View source for Talk:Symbolic Melodic

===Open points===

* Copyright.  Any opinions on whether we need to contact RISM? If so, what would be the best way?

== My two cents ==

It is great to have the ground truth set up. Thank you for the painstaking efforts! And I hope more queries can be added in the future. 

For the second evaluation, we can try the pool method in TREC: let experts judge on items retreved by all participant teams, items not retrieved by any team are assumed to be irrelevant. It is fair to all participants, but need experts' judgment. 

For the first evaluation, I agree we will need to consider those items ignored in building the ground truth. Again, experts' judgment is needed. Can we ask music librarian or music students for help? 

Thank you!