Difference between revisions of "2008:Query-by-Singing/Humming Results"
IMIRSELBot (talk | contribs) m (Bot: Fixing image links) |
(→Team ID) |
||
| (2 intermediate revisions by 2 users not shown) | |||
| Line 11: | Line 11: | ||
====Team ID==== | ====Team ID==== | ||
| − | '''JL1''' = [https://www.music-ir.org/mirex/ | + | '''JL1''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 1]<br /> |
| − | '''JL2''' = [https://www.music-ir.org/mirex/ | + | '''JL2''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 2]<br /> |
| − | '''JL3''' = [https://www.music-ir.org/mirex/ | + | '''JL3''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 3]<br /> |
| − | '''JL4''' = [https://www.music-ir.org/mirex/ | + | '''JL4''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 4]<br /> |
| − | '''RK''' = [https://www.music-ir.org/mirex/ | + | '''RK''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_ryynanen.pdf M. Ryynänen, A. Klapuri]<br /> |
| − | '''LW1''' = [https://www.music-ir.org/mirex/ | + | '''LW1''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 1]<br /> |
| − | '''LW2''' = [https://www.music-ir.org/mirex/ | + | '''LW2''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 2]<br /> |
| − | ''' | + | '''LW3''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 3]<br /> |
| − | '''WL1''' = [https://www.music-ir.org/mirex/ | + | '''WL1''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_wu.pdf X. Wu, M. Li 1]<br /> |
| − | '''WL2''' = [https://www.music-ir.org/mirex/ | + | '''WL2''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_wu.pdf X. Wu, M. Li 2]<br /> |
===Task 1 Results=== | ===Task 1 Results=== | ||
Latest revision as of 17:26, 23 July 2010
Contents
Introduction
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2008:Query by Singing/Humming page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The query database consists of 2797 sung queries.
Task 2 Goto Task 2 Results: The second subtask is the same as last year too. In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
General Legend
Team ID
JL1 = J-S. R. Jang 1
JL2 = J-S. R. Jang 2
JL3 = J-S. R. Jang 3
JL4 = J-S. R. Jang 4
RK = M. Ryynänen, A. Klapuri
LW1 = L. Wang 1
LW2 = L. Wang 2
LW3 = L. Wang 3
WL1 = X. Wu, M. Li 1
WL2 = X. Wu, M. Li 2
Task 1 Results
Task 1 Overall Results
| songID | JL1 | JL2 | JL3 | JL4 | LW1 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|
| Avg. MRR | 0.81 | 0.69 | 0.82 | 0.72 | 0.93 | 0.93 | 0.92 | 0.91 |
Task 1 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| JL1 | JL2 | 0.2791 | 1.7708 | 3.2626 | TRUE |
| JL1 | JL3 | -2.1063 | -0.6146 | 0.8772 | FALSE |
| JL1 | JL4 | -0.3147 | 1.1771 | 2.6688 | FALSE |
| JL1 | LW1 | -4.5334 | -3.0417 | -1.5499 | TRUE |
| JL1 | RK | -4.3980 | -2.9062 | -1.4145 | TRUE |
| JL1 | WL1 | -4.5751 | -3.0833 | -1.5916 | TRUE |
| JL1 | WL2 | -3.6272 | -2.1354 | -0.6437 | TRUE |
| JL2 | JL3 | -3.8772 | -2.3854 | -0.8937 | TRUE |
| JL2 | JL4 | -2.0855 | -0.5938 | 0.8980 | FALSE |
| JL2 | LW1 | -6.3043 | -4.8125 | -3.3207 | TRUE |
| JL2 | RK | -6.1688 | -4.6771 | -3.1853 | TRUE |
| JL2 | WL1 | -6.3459 | -4.8542 | -3.3624 | TRUE |
| JL2 | WL2 | -5.3980 | -3.9063 | -2.4145 | TRUE |
| JL3 | JL4 | 0.2999 | 1.7917 | 3.2834 | TRUE |
| JL3 | LW1 | -3.9188 | -2.4271 | -0.9353 | TRUE |
| JL3 | RK | -3.7834 | -2.2917 | -0.7999 | TRUE |
| JL3 | WL1 | -3.9605 | -2.4688 | -0.9770 | TRUE |
| JL3 | WL2 | -3.0126 | -1.5208 | -0.0291 | TRUE |
| JL4 | LW1 | -5.7105 | -4.2188 | -2.7270 | TRUE |
| JL4 | RK | -5.5751 | -4.0833 | -2.5916 | TRUE |
| JL4 | WL1 | -5.7522 | -4.2604 | -2.7687 | TRUE |
| JL4 | WL2 | -4.8043 | -3.3125 | -1.8207 | TRUE |
| LW1 | RK | -1.3563 | 0.1354 | 1.6272 | FALSE |
| LW1 | WL1 | -1.5334 | -0.0417 | 1.4501 | FALSE |
| LW1 | WL2 | -0.5855 | 0.9063 | 2.3980 | FALSE |
| RK | WL1 | -1.6688 | -0.1771 | 1.3147 | FALSE |
| RK | WL2 | -0.7209 | 0.7708 | 2.2626 | FALSE |
| WL1 | WL2 | -0.5438 | 0.9479 | 2.4397 | FALSE |
Task 1 Summary Results by Query Group
| Query Group | JL1 | JL2 | JL3 | JL4 | LW1 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|
| 1 | 0.82 | 0.67 | 0.94 | 0.78 | 1.00 | 0.93 | 1.00 | 1.00 |
| 2 | 0.78 | 0.64 | 0.65 | 0.59 | 0.90 | 0.69 | 0.45 | 0.44 |
| 3 | 0.75 | 0.75 | 0.88 | 0.77 | 0.88 | 1.00 | 0.88 | 0.88 |
| 4 | 1.00 | 0.94 | 0.85 | 0.81 | 1.00 | 1.00 | 1.00 | 0.94 |
| 5 | 0.56 | 0.22 | 1.00 | 0.56 | 1.00 | 0.90 | 1.00 | 0.92 |
| 6 | 1.00 | 0.90 | 1.00 | 1.00 | 1.00 | 0.75 | 1.00 | 1.00 |
| 7 | 0.94 | 0.81 | 1.00 | 0.93 | 1.00 | 0.89 | 0.94 | 0.90 |
| 8 | 0.56 | 0.63 | 0.83 | 0.94 | 1.00 | 1.00 | 1.00 | 1.00 |
| 9 | 0.88 | 0.81 | 1.00 | 0.88 | 1.00 | 1.00 | 1.00 | 1.00 |
| 10 | 0.80 | 1.00 | 0.90 | 0.95 | 1.00 | 1.00 | 1.00 | 1.00 |
| 11 | 0.85 | 0.75 | 0.84 | 0.78 | 0.95 | 0.92 | 0.93 | 0.92 |
| 12 | 0.85 | 0.76 | 0.87 | 0.86 | 0.94 | 0.93 | 0.93 | 0.93 |
| 13 | 0.84 | 0.76 | 0.86 | 0.78 | 0.93 | 0.95 | 0.96 | 0.95 |
| 14 | 0.82 | 0.64 | 0.66 | 0.43 | 0.90 | 0.94 | 0.94 | 0.92 |
| 15 | 0.77 | 0.79 | 0.87 | 0.87 | 0.84 | 0.87 | 0.83 | 0.83 |
| 16 | 0.82 | 0.78 | 0.86 | 0.81 | 0.95 | 0.87 | 0.94 | 0.92 |
| 17 | 0.74 | 0.48 | 0.75 | 0.59 | 0.96 | 0.85 | 0.96 | 0.94 |
| 18 | 0.87 | 0.69 | 0.87 | 0.79 | 0.99 | 0.99 | 1.00 | 0.99 |
| 19 | 0.92 | 0.81 | 0.92 | 0.83 | 0.97 | 0.95 | 0.95 | 0.94 |
| 20 | 0.89 | 0.78 | 0.86 | 0.85 | 0.99 | 0.97 | 0.97 | 0.96 |
| 21 | 0.65 | 0.56 | 0.63 | 0.56 | 0.64 | 0.77 | 0.66 | 0.66 |
| 22 | 0.89 | 0.87 | 0.90 | 0.86 | 0.98 | 0.97 | 0.99 | 0.99 |
| 23 | 0.84 | 0.68 | 0.91 | 0.81 | 0.95 | 0.97 | 0.96 | 0.94 |
| 24 | 0.76 | 0.65 | 0.64 | 0.64 | 0.72 | 0.88 | 0.79 | 0.77 |
| 25 | 0.46 | 0.30 | 0.64 | 0.45 | 0.60 | 0.78 | 0.71 | 0.69 |
| 26 | 0.88 | 0.74 | 0.90 | 0.86 | 0.95 | 0.90 | 0.90 | 0.88 |
| 27 | 0.84 | 0.65 | 0.84 | 0.70 | 0.96 | 0.91 | 0.97 | 0.97 |
| 28 | 0.41 | 0.22 | 0.40 | 0.18 | 0.93 | 0.67 | 0.84 | 0.81 |
| 29 | 0.79 | 0.73 | 0.72 | 0.64 | 0.96 | 0.97 | 0.96 | 0.94 |
| 30 | 0.83 | 0.65 | 0.87 | 0.65 | 0.98 | 0.94 | 0.94 | 0.91 |
| 31 | 0.86 | 0.81 | 0.70 | 0.66 | 0.97 | 0.99 | 1.00 | 0.97 |
| 32 | 0.86 | 0.68 | 0.90 | 0.79 | 0.98 | 0.98 | 0.97 | 0.95 |
| 33 | 0.70 | 0.53 | 0.91 | 0.71 | 0.94 | 0.96 | 0.72 | 0.65 |
| 34 | 0.79 | 0.66 | 0.87 | 0.69 | 0.97 | 0.93 | 0.93 | 0.90 |
| 35 | 0.92 | 0.75 | 0.85 | 0.70 | 0.97 | 0.95 | 0.92 | 0.89 |
| 36 | 0.85 | 0.81 | 0.88 | 0.71 | 0.88 | 1.00 | 0.96 | 0.92 |
| 37 | 0.74 | 0.53 | 0.80 | 0.68 | 0.93 | 0.90 | 0.95 | 0.93 |
| 38 | 0.87 | 0.87 | 0.84 | 0.79 | 0.93 | 0.96 | 0.91 | 0.93 |
| 39 | 0.81 | 0.71 | 0.85 | 0.76 | 0.76 | 0.85 | 0.89 | 0.91 |
| 40 | 0.73 | 0.60 | 0.82 | 0.73 | 0.91 | 0.91 | 0.90 | 0.88 |
| 41 | 0.76 | 0.69 | 0.74 | 0.64 | 0.94 | 0.99 | 0.96 | 0.96 |
| 42 | 0.84 | 0.70 | 0.89 | 0.73 | 0.95 | 0.93 | 0.95 | 0.92 |
| 43 | 0.89 | 0.58 | 0.88 | 0.67 | 0.98 | 0.94 | 0.97 | 0.95 |
| 44 | 0.82 | 0.78 | 0.94 | 0.92 | 1.00 | 0.98 | 0.96 | 0.96 |
| 45 | 0.82 | 0.76 | 0.87 | 0.82 | 0.92 | 0.86 | 0.93 | 0.90 |
| 46 | 0.80 | 0.64 | 0.85 | 0.80 | 0.92 | 0.99 | 0.96 | 0.95 |
| 47 | 0.78 | 0.65 | 0.86 | 0.75 | 0.93 | 1.00 | 0.97 | 0.97 |
| 48 | 0.80 | 0.75 | 0.78 | 0.70 | 0.88 | 0.94 | 0.94 | 0.96 |
Task 2 Results
In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
Task 2 Overall Results
| songID | JL1 | JL2 | JL3 | JL4 | LW2 | LW3 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|---|
| Avg. MRR | 0.32 | 0.53 | 0.32 | 0.59 | 0.92 | 0.94 | 0.89 | 0.94 | 0.92 |
Task 2 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| JL1 | JL2 | -1.9291 | -0.9198 | 0.0895 | FALSE |
| JL1 | JL3 | -1.0093 | 0.0000 | 1.0093 | FALSE |
| JL1 | JL4 | -2.5282 | -1.5189 | -0.5096 | TRUE |
| JL1 | LW2 | -4.9574 | -3.9481 | -2.9388 | TRUE |
| JL1 | LW3 | -5.1650 | -4.1557 | -3.1463 | TRUE |
| JL1 | RK | -4.7169 | -3.7075 | -2.6982 | TRUE |
| JL1 | WL1 | -5.2027 | -4.1934 | -3.1841 | TRUE |
| JL1 | WL2 | -4.8961 | -3.8868 | -2.8775 | TRUE |
| JL2 | JL3 | -0.0895 | 0.9198 | 1.9291 | FALSE |
| JL2 | JL4 | -1.6084 | -0.5991 | 0.4103 | FALSE |
| JL2 | LW2 | -4.0376 | -3.0283 | -2.0190 | TRUE |
| JL2 | LW3 | -4.2452 | -3.2358 | -2.2265 | TRUE |
| JL2 | RK | -3.7970 | -2.7877 | -1.7784 | TRUE |
| JL2 | WL1 | -4.2829 | -3.2736 | -2.2643 | TRUE |
| JL2 | WL2 | -3.9763 | -2.9670 | -1.9577 | TRUE |
| JL3 | JL4 | -2.5282 | -1.5189 | -0.5096 | TRUE |
| JL3 | LW2 | -4.9574 | -3.9481 | -2.9388 | TRUE |
| JL3 | LW3 | -5.1650 | -4.1557 | -3.1463 | TRUE |
| JL3 | RK | -4.7169 | -3.7075 | -2.6982 | TRUE |
| JL3 | WL1 | -5.2027 | -4.1934 | -3.1841 | TRUE |
| JL3 | WL2 | -4.8961 | -3.8868 | -2.8775 | TRUE |
| JL4 | LW2 | -3.4386 | -2.4292 | -1.4199 | TRUE |
| JL4 | LW3 | -3.6461 | -2.6368 | -1.6275 | TRUE |
| JL4 | RK | -3.1980 | -2.1887 | -1.1794 | TRUE |
| JL4 | WL1 | -3.6838 | -2.6745 | -1.6652 | TRUE |
| JL4 | WL2 | -3.3772 | -2.3679 | -1.3586 | TRUE |
| LW2 | LW3 | -1.2169 | -0.2075 | 0.8018 | FALSE |
| LW2 | RK | -0.7687 | 0.2406 | 1.2499 | FALSE |
| LW2 | WL1 | -1.2546 | -0.2453 | 0.7640 | FALSE |
| LW2 | WL2 | -0.9480 | 0.0613 | 1.0706 | FALSE |
| LW3 | RK | -0.5612 | 0.4481 | 1.4574 | FALSE |
| LW3 | WL1 | -1.0470 | -0.0377 | 0.9716 | FALSE |
| LW3 | WL2 | -0.7404 | 0.2689 | 1.2782 | FALSE |
| RK | WL1 | -1.4952 | -0.4858 | 0.5235 | FALSE |
| RK | WL2 | -1.1886 | -0.1792 | 0.8301 | FALSE |
| WL1 | WL2 | -0.7027 | 0.3066 | 1.3159 | FALSE |
Task 2 Summary Results by Query Group
| Query Group | JL1 | JL2 | JL3 | JL4 | LW2 | LW3 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.32 | 0.74 | 0.28 | 0.62 | 1.00 | 1.00 | 0.91 | 0.95 | 0.93 |
| 2 | 0.60 | 0.70 | 0.40 | 0.61 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 3 | 0.00 | 0.00 | 0.00 | 0.50 | 0.11 | 0.00 | 1.00 | 0.25 | 0.11 |
| 4 | 0.00 | 0.50 | 0.25 | 0.75 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 |
| 5 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 |
| 6 | 0.00 | 0.56 | 0.00 | 0.11 | 0.67 | 0.68 | 0.83 | 0.69 | 0.68 |
| 7 | 0.83 | 0.83 | 0.68 | 0.45 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 8 | 0.04 | 0.00 | 0.11 | 0.05 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 |
| 9 | 0.00 | 0.60 | 0.00 | 0.70 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 10 | 0.00 | 0.50 | 0.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 11 | 0.00 | 0.22 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.75 | 0.63 |
| 12 | 0.33 | 0.11 | 0.00 | 0.07 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 13 | 0.03 | 0.60 | 0.23 | 0.60 | 1.00 | 1.00 | 0.80 | 1.00 | 1.00 |
| 14 | 0.00 | 0.63 | 0.00 | 0.56 | 1.00 | 1.00 | 0.50 | 1.00 | 1.00 |
| 15 | 0.00 | 0.17 | 0.00 | 0.00 | 0.67 | 1.00 | 0.13 | 1.00 | 0.60 |
| 16 | 0.38 | 0.38 | 0.31 | 0.39 | 1.00 | 1.00 | 0.75 | 1.00 | 1.00 |
| 17 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 18 | 0.00 | 0.63 | 0.25 | 0.17 | 0.50 | 0.50 | 0.50 | 1.00 | 1.00 |
| 19 | 0.00 | 1.00 | 0.00 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 20 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.75 | 0.42 | 0.33 |
| 21 | 0.16 | 0.03 | 0.03 | 0.60 | 0.75 | 1.00 | 0.63 | 0.55 | 0.53 |
| 22 | 0.39 | 0.58 | 0.44 | 0.62 | 0.74 | 0.81 | 1.00 | 0.92 | 0.94 |
| 23 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 24 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 25 | 0.50 | 0.67 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 26 | 0.42 | 0.50 | 0.30 | 0.60 | 1.00 | 1.00 | 0.92 | 1.00 | 1.00 |
| 27 | 0.06 | 0.75 | 0.25 | 0.65 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 28 | 0.50 | 0.58 | 0.33 | 0.80 | 1.00 | 1.00 | 0.63 | 1.00 | 1.00 |
| 29 | 0.33 | 0.05 | 1.00 | 1.00 | 1.00 | 1.00 | 0.25 | 1.00 | 1.00 |
| 30 | 1.00 | 1.00 | 0.75 | 0.58 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 31 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 32 | 0.00 | 0.50 | 0.00 | 0.71 | 1.00 | 1.00 | 0.86 | 1.00 | 1.00 |
| 33 | 0.00 | 0.80 | 0.00 | 0.80 | 0.63 | 0.67 | 1.00 | 0.72 | 0.66 |
| 34 | 0.00 | 0.25 | 0.50 | 0.00 | 0.50 | 0.50 | 0.58 | 0.50 | 0.50 |
| 35 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 36 | 0.00 | 0.67 | 0.00 | 0.52 | 0.82 | 0.80 | 0.87 | 0.87 | 0.80 |
| 37 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 0.11 | 0.06 | 0.00 |
| 38 | 0.50 | 0.55 | 0.50 | 1.00 | 0.63 | 1.00 | 1.00 | 1.00 | 1.00 |
| 39 | 0.36 | 0.73 | 0.43 | 1.00 | 1.00 | 1.00 | 0.90 | 1.00 | 1.00 |
| 40 | 0.54 | 0.50 | 1.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 41 | 0.25 | 0.57 | 0.06 | 0.56 | 0.78 | 0.81 | 0.88 | 0.88 | 0.88 |
| 42 | 0.63 | 0.38 | 0.59 | 0.68 | 1.00 | 1.00 | 0.92 | 1.00 | 0.88 |
| 43 | 0.48 | 0.35 | 0.67 | 0.54 | 0.73 | 0.85 | 0.81 | 0.83 | 0.76 |
| 44 | 0.48 | 0.68 | 0.56 | 0.83 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 45 | 0.69 | 0.67 | 1.00 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 46 | 0.00 | 0.25 | 0.00 | 0.25 | 1.00 | 1.00 | 0.29 | 1.00 | 1.00 |
| 47 | 0.75 | 0.63 | 0.50 | 0.51 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 48 | 0.07 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 49 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 51 | 0.05 | 0.25 | 0.25 | 0.26 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 52 | 0.08 | 0.54 | 0.25 | 0.54 | 1.00 | 1.00 | 0.77 | 1.00 | 1.00 |
| 53 | 0.33 | 0.55 | 0.33 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 54 | 0.20 | 0.66 | 0.13 | 0.60 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 55 | 0.00 | 0.20 | 0.00 | 0.63 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 56 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.58 | 0.50 | 0.50 |
| 57 | 0.90 | 0.90 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 58 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 59 | 0.00 | 0.73 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 60 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 61 | 0.00 | 0.50 | 0.00 | 0.08 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 |
| 62 | 0.08 | 0.67 | 0.33 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 63 | 0.63 | 0.53 | 0.53 | 0.57 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 64 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 65 | 0.00 | 0.08 | 0.00 | 0.17 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 66 | 0.50 | 0.75 | 0.50 | 0.55 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 67 | 0.00 | 0.83 | 0.00 | 1.00 | 1.00 | 1.00 | 0.67 | 1.00 | 1.00 |
| 68 | 1.00 | 0.54 | 0.55 | 0.54 | 0.75 | 1.00 | 0.50 | 0.67 | 0.67 |
| 69 | 0.00 | 0.03 | 0.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 70 | 1.00 | 0.88 | 0.81 | 0.79 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 71 | 0.00 | 0.37 | 0.00 | 0.67 | 0.75 | 1.00 | 1.00 | 0.42 | 0.42 |
| 72 | 1.00 | 0.71 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 73 | 0.75 | 0.75 | 0.65 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 74 | 1.00 | 0.54 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 75 | 0.17 | 0.35 | 0.33 | 0.38 | 1.00 | 1.00 | 0.83 | 1.00 | 1.00 |
| 76 | 0.00 | 0.30 | 0.00 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 0.55 |
| 77 | 1.00 | 0.80 | 0.64 | 0.45 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 78 | 0.00 | 0.33 | 0.07 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 79 | 0.50 | 0.50 | 0.50 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 80 | 0.33 | 0.73 | 0.00 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 81 | 0.17 | 0.60 | 0.17 | 0.58 | 0.92 | 0.92 | 1.00 | 1.00 | 0.92 |
| 82 | 0.00 | 0.75 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 83 | 0.00 | 0.05 | 0.02 | 0.22 | 0.36 | 0.50 | 1.00 | 1.00 | 1.00 |
| 84 | 0.05 | 0.02 | 0.00 | 0.00 | 0.27 | 0.34 | 0.54 | 0.75 | 0.75 |
| 85 | 0.50 | 1.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 86 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 87 | 0.00 | 1.00 | 0.00 | 0.45 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 88 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.17 | 1.00 | 1.00 |
| 89 | 0.02 | 0.16 | 0.43 | 0.36 | 0.93 | 0.93 | 0.93 | 0.86 | 0.86 |
| 90 | 0.00 | 0.11 | 0.00 | 0.60 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 91 | 0.00 | 0.25 | 0.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 92 | 0.50 | 0.54 | 0.50 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 93 | 0.25 | 0.17 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 94 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 95 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 96 | 0.00 | 0.00 | 1.00 | 0.05 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 |
| 97 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 98 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 |
| 99 | 0.55 | 0.03 | 0.38 | 0.15 | 1.00 | 1.00 | 0.63 | 1.00 | 0.88 |
| 100 | 0.33 | 1.00 | 0.33 | 1.00 | 0.68 | 0.67 | 1.00 | 1.00 | 1.00 |
| 101 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 102 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 103 | 0.27 | 0.39 | 0.26 | 0.38 | 0.94 | 1.00 | 1.00 | 1.00 | 1.00 |
| 104 | 0.00 | 0.67 | 0.00 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 105 | 1.00 | 0.00 | 0.17 | 0.08 | 0.00 | 0.00 | 0.11 | 0.14 | 0.05 |
| 106 | 0.00 | 0.36 | 0.00 | 0.35 | 1.00 | 1.00 | 0.61 | 1.00 | 0.69 |
Runtime Results
| Participant | Task | Runtime (min'sec") | Machine |
|---|---|---|---|
| JL1 | 1 | 20'10" | FAST |
| JL2 | 1 | 417'36" | FAST |
| JL3 | 1 | 54'25" | FAST |
| JL4 | 1 | 297'32" | FAST2 |
| LW1 | 1 | 143'40" | FAST2 |
| RK | 1 | 71'52" | cluster |
| WL1 | 1 | 31'03" | FAST |
| WL2 | 1 | 28'19" | FAST |
| JL1 | 2 | 2'08" | FAST2 |
| JL2 | 2 | 54'05" | FAST2 |
| JL3 | 2 | 7'32" | FAST2 |
| JL4 | 2 | 59'23" | FAST2 |
| LW2 | 2 | 561'42" | FAST2 |
| LW3 | 2 | 62'39" | FAST2 |
| RK | 2 | 19'36" | cluster |
| WL1 | 2 | 8'30" | FAST |
| WL2 | 2 | 6'03" | FAST |

