Difference between revisions of "2008:Query-by-Singing/Humming Results"
From MIREX Wiki
(→General Legend) |
(→Team ID) |
||
| (9 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
==Introduction== | ==Introduction== | ||
| − | These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[Query by Singing/Humming]] page. | + | These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the [[2008:Query by Singing/Humming]] page. |
===Task Descriptions=== | ===Task Descriptions=== | ||
| Line 11: | Line 11: | ||
====Team ID==== | ====Team ID==== | ||
| − | '''JL1''' = [https://www.music-ir.org/mirex/2008/ | + | '''JL1''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 1]<br /> |
| − | '''JL2''' = [https://www.music-ir.org/mirex/2008/ | + | '''JL2''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 2]<br /> |
| − | '''JL3''' = [https://www.music-ir.org/mirex/2008/ | + | '''JL3''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 3]<br /> |
| − | '''JL4''' = [https://www.music-ir.org/mirex/2008/ | + | '''JL4''' = [https://www.music-ir.org/mirex/abstracts/2008/MIREX2008_QBSH_Davidson_abstract.pdf J-S. R. Jang 4]<br /> |
| − | '''RK''' = [https://www.music-ir.org/mirex/2008 | + | '''RK''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_ryynanen.pdf M. Ryynänen, A. Klapuri]<br /> |
| − | '''LW1''' = [https://www.music-ir.org/mirex/2008/ | + | '''LW1''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 1]<br /> |
| − | '''LW2''' = [https://www.music-ir.org/mirex/2008/ | + | '''LW2''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 2]<br /> |
| − | ''' | + | '''LW3''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_leiwang.pdf L. Wang 3]<br /> |
| − | '''WL1''' = [https://www.music-ir.org/mirex/2008/ | + | '''WL1''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_wu.pdf X. Wu, M. Li 1]<br /> |
| − | '''WL2''' = [https://www.music-ir.org/mirex/2008/ | + | '''WL2''' = [https://www.music-ir.org/mirex/abstracts/2008/QBSH_wu.pdf X. Wu, M. Li 2]<br /> |
===Task 1 Results=== | ===Task 1 Results=== | ||
=====Task 1 Overall Results===== | =====Task 1 Overall Results===== | ||
| − | <csv>qbsh/qbsh_task1_summary.csv</csv> | + | <csv>2008/qbsh/qbsh_task1_summary.csv</csv> |
====Task 1 Friedman's Test for Significant Differences==== | ====Task 1 Friedman's Test for Significant Differences==== | ||
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. | The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. | ||
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
| − | <csv>qbsh/qbsh.task1.friedman_detailed.csv</csv> | + | <csv>2008/qbsh/qbsh.task1.friedman_detailed.csv</csv> |
| − | [[Image: | + | [[Image:2008_qbsh.task1.friedman.small.png]] |
====Task 1 Summary Results by Query Group==== | ====Task 1 Summary Results by Query Group==== | ||
| − | <csv>qbsh/qbsh.task1.res.byQueryGroup.csv</csv> | + | <csv>2008/qbsh/qbsh.task1.res.byQueryGroup.csv</csv> |
===Task 2 Results=== | ===Task 2 Results=== | ||
| Line 41: | Line 41: | ||
=====Task 2 Overall Results===== | =====Task 2 Overall Results===== | ||
| − | <csv>qbsh/qbsh.task2.summary.csv</csv> | + | <csv>2008/qbsh/qbsh.task2.summary.csv</csv> |
====Task 2 Friedman's Test for Significant Differences==== | ====Task 2 Friedman's Test for Significant Differences==== | ||
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. | The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. | ||
Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05); | ||
| − | <csv>qbsh/qbsh.task2.friedman_detailed.csv</csv> | + | <csv>2008/qbsh/qbsh.task2.friedman_detailed.csv</csv> |
| − | [[Image: | + | [[Image:2008_qbsh.task2.friedman.s.png]] |
====Task 2 Summary Results by Query Group==== | ====Task 2 Summary Results by Query Group==== | ||
| − | <csv>qbsh/qbsh_task2_res_byQueryGroup.csv</csv> | + | <csv>2008/qbsh/qbsh_task2_res_byQueryGroup.csv</csv> |
===Runtime Results=== | ===Runtime Results=== | ||
| − | <csv>qbsh.runtime.csv</csv> | + | <csv>2008/qbsh.runtime.csv</csv> |
[[Category: Results]] | [[Category: Results]] | ||
Latest revision as of 16:26, 23 July 2010
Contents
Introduction
These are the results for the 2008 running of the Query-by-Singing/Humming task. For background information about this task set please refer to the 2008:Query by Singing/Humming page.
Task Descriptions
Task 1 Goto Task 1 Results: The first subtask is the same as last year. In this subtask, submitted systems take a sung query as input and return a list of songs from the test database. Mean reciprocal rank (MRR) of the ground truth is calculated over the top 20 returns. The test database consists of 48 ground-truth MIDIs + 2000 Essen Collection MIDI noise files. See ESAC Data Homepage for more information about the Essen Collection. The query database consists of 2797 sung queries.
Task 2 Goto Task 2 Results: The second subtask is the same as last year too. In the second subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
General Legend
Team ID
JL1 = J-S. R. Jang 1
JL2 = J-S. R. Jang 2
JL3 = J-S. R. Jang 3
JL4 = J-S. R. Jang 4
RK = M. Ryynänen, A. Klapuri
LW1 = L. Wang 1
LW2 = L. Wang 2
LW3 = L. Wang 3
WL1 = X. Wu, M. Li 1
WL2 = X. Wu, M. Li 2
Task 1 Results
Task 1 Overall Results
| songID | JL1 | JL2 | JL3 | JL4 | LW1 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|
| Avg. MRR | 0.81 | 0.69 | 0.82 | 0.72 | 0.93 | 0.93 | 0.92 | 0.91 |
Task 1 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| JL1 | JL2 | 0.2791 | 1.7708 | 3.2626 | TRUE |
| JL1 | JL3 | -2.1063 | -0.6146 | 0.8772 | FALSE |
| JL1 | JL4 | -0.3147 | 1.1771 | 2.6688 | FALSE |
| JL1 | LW1 | -4.5334 | -3.0417 | -1.5499 | TRUE |
| JL1 | RK | -4.3980 | -2.9062 | -1.4145 | TRUE |
| JL1 | WL1 | -4.5751 | -3.0833 | -1.5916 | TRUE |
| JL1 | WL2 | -3.6272 | -2.1354 | -0.6437 | TRUE |
| JL2 | JL3 | -3.8772 | -2.3854 | -0.8937 | TRUE |
| JL2 | JL4 | -2.0855 | -0.5938 | 0.8980 | FALSE |
| JL2 | LW1 | -6.3043 | -4.8125 | -3.3207 | TRUE |
| JL2 | RK | -6.1688 | -4.6771 | -3.1853 | TRUE |
| JL2 | WL1 | -6.3459 | -4.8542 | -3.3624 | TRUE |
| JL2 | WL2 | -5.3980 | -3.9063 | -2.4145 | TRUE |
| JL3 | JL4 | 0.2999 | 1.7917 | 3.2834 | TRUE |
| JL3 | LW1 | -3.9188 | -2.4271 | -0.9353 | TRUE |
| JL3 | RK | -3.7834 | -2.2917 | -0.7999 | TRUE |
| JL3 | WL1 | -3.9605 | -2.4688 | -0.9770 | TRUE |
| JL3 | WL2 | -3.0126 | -1.5208 | -0.0291 | TRUE |
| JL4 | LW1 | -5.7105 | -4.2188 | -2.7270 | TRUE |
| JL4 | RK | -5.5751 | -4.0833 | -2.5916 | TRUE |
| JL4 | WL1 | -5.7522 | -4.2604 | -2.7687 | TRUE |
| JL4 | WL2 | -4.8043 | -3.3125 | -1.8207 | TRUE |
| LW1 | RK | -1.3563 | 0.1354 | 1.6272 | FALSE |
| LW1 | WL1 | -1.5334 | -0.0417 | 1.4501 | FALSE |
| LW1 | WL2 | -0.5855 | 0.9063 | 2.3980 | FALSE |
| RK | WL1 | -1.6688 | -0.1771 | 1.3147 | FALSE |
| RK | WL2 | -0.7209 | 0.7708 | 2.2626 | FALSE |
| WL1 | WL2 | -0.5438 | 0.9479 | 2.4397 | FALSE |
Task 1 Summary Results by Query Group
| Query Group | JL1 | JL2 | JL3 | JL4 | LW1 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|
| 1 | 0.82 | 0.67 | 0.94 | 0.78 | 1.00 | 0.93 | 1.00 | 1.00 |
| 2 | 0.78 | 0.64 | 0.65 | 0.59 | 0.90 | 0.69 | 0.45 | 0.44 |
| 3 | 0.75 | 0.75 | 0.88 | 0.77 | 0.88 | 1.00 | 0.88 | 0.88 |
| 4 | 1.00 | 0.94 | 0.85 | 0.81 | 1.00 | 1.00 | 1.00 | 0.94 |
| 5 | 0.56 | 0.22 | 1.00 | 0.56 | 1.00 | 0.90 | 1.00 | 0.92 |
| 6 | 1.00 | 0.90 | 1.00 | 1.00 | 1.00 | 0.75 | 1.00 | 1.00 |
| 7 | 0.94 | 0.81 | 1.00 | 0.93 | 1.00 | 0.89 | 0.94 | 0.90 |
| 8 | 0.56 | 0.63 | 0.83 | 0.94 | 1.00 | 1.00 | 1.00 | 1.00 |
| 9 | 0.88 | 0.81 | 1.00 | 0.88 | 1.00 | 1.00 | 1.00 | 1.00 |
| 10 | 0.80 | 1.00 | 0.90 | 0.95 | 1.00 | 1.00 | 1.00 | 1.00 |
| 11 | 0.85 | 0.75 | 0.84 | 0.78 | 0.95 | 0.92 | 0.93 | 0.92 |
| 12 | 0.85 | 0.76 | 0.87 | 0.86 | 0.94 | 0.93 | 0.93 | 0.93 |
| 13 | 0.84 | 0.76 | 0.86 | 0.78 | 0.93 | 0.95 | 0.96 | 0.95 |
| 14 | 0.82 | 0.64 | 0.66 | 0.43 | 0.90 | 0.94 | 0.94 | 0.92 |
| 15 | 0.77 | 0.79 | 0.87 | 0.87 | 0.84 | 0.87 | 0.83 | 0.83 |
| 16 | 0.82 | 0.78 | 0.86 | 0.81 | 0.95 | 0.87 | 0.94 | 0.92 |
| 17 | 0.74 | 0.48 | 0.75 | 0.59 | 0.96 | 0.85 | 0.96 | 0.94 |
| 18 | 0.87 | 0.69 | 0.87 | 0.79 | 0.99 | 0.99 | 1.00 | 0.99 |
| 19 | 0.92 | 0.81 | 0.92 | 0.83 | 0.97 | 0.95 | 0.95 | 0.94 |
| 20 | 0.89 | 0.78 | 0.86 | 0.85 | 0.99 | 0.97 | 0.97 | 0.96 |
| 21 | 0.65 | 0.56 | 0.63 | 0.56 | 0.64 | 0.77 | 0.66 | 0.66 |
| 22 | 0.89 | 0.87 | 0.90 | 0.86 | 0.98 | 0.97 | 0.99 | 0.99 |
| 23 | 0.84 | 0.68 | 0.91 | 0.81 | 0.95 | 0.97 | 0.96 | 0.94 |
| 24 | 0.76 | 0.65 | 0.64 | 0.64 | 0.72 | 0.88 | 0.79 | 0.77 |
| 25 | 0.46 | 0.30 | 0.64 | 0.45 | 0.60 | 0.78 | 0.71 | 0.69 |
| 26 | 0.88 | 0.74 | 0.90 | 0.86 | 0.95 | 0.90 | 0.90 | 0.88 |
| 27 | 0.84 | 0.65 | 0.84 | 0.70 | 0.96 | 0.91 | 0.97 | 0.97 |
| 28 | 0.41 | 0.22 | 0.40 | 0.18 | 0.93 | 0.67 | 0.84 | 0.81 |
| 29 | 0.79 | 0.73 | 0.72 | 0.64 | 0.96 | 0.97 | 0.96 | 0.94 |
| 30 | 0.83 | 0.65 | 0.87 | 0.65 | 0.98 | 0.94 | 0.94 | 0.91 |
| 31 | 0.86 | 0.81 | 0.70 | 0.66 | 0.97 | 0.99 | 1.00 | 0.97 |
| 32 | 0.86 | 0.68 | 0.90 | 0.79 | 0.98 | 0.98 | 0.97 | 0.95 |
| 33 | 0.70 | 0.53 | 0.91 | 0.71 | 0.94 | 0.96 | 0.72 | 0.65 |
| 34 | 0.79 | 0.66 | 0.87 | 0.69 | 0.97 | 0.93 | 0.93 | 0.90 |
| 35 | 0.92 | 0.75 | 0.85 | 0.70 | 0.97 | 0.95 | 0.92 | 0.89 |
| 36 | 0.85 | 0.81 | 0.88 | 0.71 | 0.88 | 1.00 | 0.96 | 0.92 |
| 37 | 0.74 | 0.53 | 0.80 | 0.68 | 0.93 | 0.90 | 0.95 | 0.93 |
| 38 | 0.87 | 0.87 | 0.84 | 0.79 | 0.93 | 0.96 | 0.91 | 0.93 |
| 39 | 0.81 | 0.71 | 0.85 | 0.76 | 0.76 | 0.85 | 0.89 | 0.91 |
| 40 | 0.73 | 0.60 | 0.82 | 0.73 | 0.91 | 0.91 | 0.90 | 0.88 |
| 41 | 0.76 | 0.69 | 0.74 | 0.64 | 0.94 | 0.99 | 0.96 | 0.96 |
| 42 | 0.84 | 0.70 | 0.89 | 0.73 | 0.95 | 0.93 | 0.95 | 0.92 |
| 43 | 0.89 | 0.58 | 0.88 | 0.67 | 0.98 | 0.94 | 0.97 | 0.95 |
| 44 | 0.82 | 0.78 | 0.94 | 0.92 | 1.00 | 0.98 | 0.96 | 0.96 |
| 45 | 0.82 | 0.76 | 0.87 | 0.82 | 0.92 | 0.86 | 0.93 | 0.90 |
| 46 | 0.80 | 0.64 | 0.85 | 0.80 | 0.92 | 0.99 | 0.96 | 0.95 |
| 47 | 0.78 | 0.65 | 0.86 | 0.75 | 0.93 | 1.00 | 0.97 | 0.97 |
| 48 | 0.80 | 0.75 | 0.78 | 0.70 | 0.88 | 0.94 | 0.94 | 0.96 |
Task 2 Results
In this subtask, the same setup as the first subtask used with combination of different transcribers and matchers. The test databases consists of 106 ground-truth MIDIS + 2000 Essen Collection MIDI noise files. The query databases consists of 355 sung queries.
Task 2 Overall Results
| songID | JL1 | JL2 | JL3 | JL4 | LW2 | LW3 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|---|
| Avg. MRR | 0.32 | 0.53 | 0.32 | 0.59 | 0.92 | 0.94 | 0.89 | 0.94 | 0.92 |
Task 2 Friedman's Test for Significant Differences
The Friedman test was run in MATLAB against the QBSH Task 1 MRR data over the 48 ground truth song groups. Command: [c,m,h,gnames] = multcompare(stats, 'ctype', 'tukey-kramer','estimate', 'friedman', 'alpha', 0.05);
| TeamID | TeamID | Lowerbound | Mean | Upperbound | Significance |
|---|---|---|---|---|---|
| JL1 | JL2 | -1.9291 | -0.9198 | 0.0895 | FALSE |
| JL1 | JL3 | -1.0093 | 0.0000 | 1.0093 | FALSE |
| JL1 | JL4 | -2.5282 | -1.5189 | -0.5096 | TRUE |
| JL1 | LW2 | -4.9574 | -3.9481 | -2.9388 | TRUE |
| JL1 | LW3 | -5.1650 | -4.1557 | -3.1463 | TRUE |
| JL1 | RK | -4.7169 | -3.7075 | -2.6982 | TRUE |
| JL1 | WL1 | -5.2027 | -4.1934 | -3.1841 | TRUE |
| JL1 | WL2 | -4.8961 | -3.8868 | -2.8775 | TRUE |
| JL2 | JL3 | -0.0895 | 0.9198 | 1.9291 | FALSE |
| JL2 | JL4 | -1.6084 | -0.5991 | 0.4103 | FALSE |
| JL2 | LW2 | -4.0376 | -3.0283 | -2.0190 | TRUE |
| JL2 | LW3 | -4.2452 | -3.2358 | -2.2265 | TRUE |
| JL2 | RK | -3.7970 | -2.7877 | -1.7784 | TRUE |
| JL2 | WL1 | -4.2829 | -3.2736 | -2.2643 | TRUE |
| JL2 | WL2 | -3.9763 | -2.9670 | -1.9577 | TRUE |
| JL3 | JL4 | -2.5282 | -1.5189 | -0.5096 | TRUE |
| JL3 | LW2 | -4.9574 | -3.9481 | -2.9388 | TRUE |
| JL3 | LW3 | -5.1650 | -4.1557 | -3.1463 | TRUE |
| JL3 | RK | -4.7169 | -3.7075 | -2.6982 | TRUE |
| JL3 | WL1 | -5.2027 | -4.1934 | -3.1841 | TRUE |
| JL3 | WL2 | -4.8961 | -3.8868 | -2.8775 | TRUE |
| JL4 | LW2 | -3.4386 | -2.4292 | -1.4199 | TRUE |
| JL4 | LW3 | -3.6461 | -2.6368 | -1.6275 | TRUE |
| JL4 | RK | -3.1980 | -2.1887 | -1.1794 | TRUE |
| JL4 | WL1 | -3.6838 | -2.6745 | -1.6652 | TRUE |
| JL4 | WL2 | -3.3772 | -2.3679 | -1.3586 | TRUE |
| LW2 | LW3 | -1.2169 | -0.2075 | 0.8018 | FALSE |
| LW2 | RK | -0.7687 | 0.2406 | 1.2499 | FALSE |
| LW2 | WL1 | -1.2546 | -0.2453 | 0.7640 | FALSE |
| LW2 | WL2 | -0.9480 | 0.0613 | 1.0706 | FALSE |
| LW3 | RK | -0.5612 | 0.4481 | 1.4574 | FALSE |
| LW3 | WL1 | -1.0470 | -0.0377 | 0.9716 | FALSE |
| LW3 | WL2 | -0.7404 | 0.2689 | 1.2782 | FALSE |
| RK | WL1 | -1.4952 | -0.4858 | 0.5235 | FALSE |
| RK | WL2 | -1.1886 | -0.1792 | 0.8301 | FALSE |
| WL1 | WL2 | -0.7027 | 0.3066 | 1.3159 | FALSE |
Task 2 Summary Results by Query Group
| Query Group | JL1 | JL2 | JL3 | JL4 | LW2 | LW3 | RK | WL1 | WL2 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.32 | 0.74 | 0.28 | 0.62 | 1.00 | 1.00 | 0.91 | 0.95 | 0.93 |
| 2 | 0.60 | 0.70 | 0.40 | 0.61 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 3 | 0.00 | 0.00 | 0.00 | 0.50 | 0.11 | 0.00 | 1.00 | 0.25 | 0.11 |
| 4 | 0.00 | 0.50 | 0.25 | 0.75 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 |
| 5 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 |
| 6 | 0.00 | 0.56 | 0.00 | 0.11 | 0.67 | 0.68 | 0.83 | 0.69 | 0.68 |
| 7 | 0.83 | 0.83 | 0.68 | 0.45 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 8 | 0.04 | 0.00 | 0.11 | 0.05 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 |
| 9 | 0.00 | 0.60 | 0.00 | 0.70 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 10 | 0.00 | 0.50 | 0.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 11 | 0.00 | 0.22 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.75 | 0.63 |
| 12 | 0.33 | 0.11 | 0.00 | 0.07 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 13 | 0.03 | 0.60 | 0.23 | 0.60 | 1.00 | 1.00 | 0.80 | 1.00 | 1.00 |
| 14 | 0.00 | 0.63 | 0.00 | 0.56 | 1.00 | 1.00 | 0.50 | 1.00 | 1.00 |
| 15 | 0.00 | 0.17 | 0.00 | 0.00 | 0.67 | 1.00 | 0.13 | 1.00 | 0.60 |
| 16 | 0.38 | 0.38 | 0.31 | 0.39 | 1.00 | 1.00 | 0.75 | 1.00 | 1.00 |
| 17 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 18 | 0.00 | 0.63 | 0.25 | 0.17 | 0.50 | 0.50 | 0.50 | 1.00 | 1.00 |
| 19 | 0.00 | 1.00 | 0.00 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 20 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.75 | 0.42 | 0.33 |
| 21 | 0.16 | 0.03 | 0.03 | 0.60 | 0.75 | 1.00 | 0.63 | 0.55 | 0.53 |
| 22 | 0.39 | 0.58 | 0.44 | 0.62 | 0.74 | 0.81 | 1.00 | 0.92 | 0.94 |
| 23 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 24 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 25 | 0.50 | 0.67 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 26 | 0.42 | 0.50 | 0.30 | 0.60 | 1.00 | 1.00 | 0.92 | 1.00 | 1.00 |
| 27 | 0.06 | 0.75 | 0.25 | 0.65 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 28 | 0.50 | 0.58 | 0.33 | 0.80 | 1.00 | 1.00 | 0.63 | 1.00 | 1.00 |
| 29 | 0.33 | 0.05 | 1.00 | 1.00 | 1.00 | 1.00 | 0.25 | 1.00 | 1.00 |
| 30 | 1.00 | 1.00 | 0.75 | 0.58 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 31 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 32 | 0.00 | 0.50 | 0.00 | 0.71 | 1.00 | 1.00 | 0.86 | 1.00 | 1.00 |
| 33 | 0.00 | 0.80 | 0.00 | 0.80 | 0.63 | 0.67 | 1.00 | 0.72 | 0.66 |
| 34 | 0.00 | 0.25 | 0.50 | 0.00 | 0.50 | 0.50 | 0.58 | 0.50 | 0.50 |
| 35 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 36 | 0.00 | 0.67 | 0.00 | 0.52 | 0.82 | 0.80 | 0.87 | 0.87 | 0.80 |
| 37 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 0.11 | 0.06 | 0.00 |
| 38 | 0.50 | 0.55 | 0.50 | 1.00 | 0.63 | 1.00 | 1.00 | 1.00 | 1.00 |
| 39 | 0.36 | 0.73 | 0.43 | 1.00 | 1.00 | 1.00 | 0.90 | 1.00 | 1.00 |
| 40 | 0.54 | 0.50 | 1.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 41 | 0.25 | 0.57 | 0.06 | 0.56 | 0.78 | 0.81 | 0.88 | 0.88 | 0.88 |
| 42 | 0.63 | 0.38 | 0.59 | 0.68 | 1.00 | 1.00 | 0.92 | 1.00 | 0.88 |
| 43 | 0.48 | 0.35 | 0.67 | 0.54 | 0.73 | 0.85 | 0.81 | 0.83 | 0.76 |
| 44 | 0.48 | 0.68 | 0.56 | 0.83 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 45 | 0.69 | 0.67 | 1.00 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 46 | 0.00 | 0.25 | 0.00 | 0.25 | 1.00 | 1.00 | 0.29 | 1.00 | 1.00 |
| 47 | 0.75 | 0.63 | 0.50 | 0.51 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 48 | 0.07 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 49 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 51 | 0.05 | 0.25 | 0.25 | 0.26 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 52 | 0.08 | 0.54 | 0.25 | 0.54 | 1.00 | 1.00 | 0.77 | 1.00 | 1.00 |
| 53 | 0.33 | 0.55 | 0.33 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 54 | 0.20 | 0.66 | 0.13 | 0.60 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 55 | 0.00 | 0.20 | 0.00 | 0.63 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 56 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.58 | 0.50 | 0.50 |
| 57 | 0.90 | 0.90 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 58 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 59 | 0.00 | 0.73 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 60 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 61 | 0.00 | 0.50 | 0.00 | 0.08 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 |
| 62 | 0.08 | 0.67 | 0.33 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 63 | 0.63 | 0.53 | 0.53 | 0.57 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 64 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 65 | 0.00 | 0.08 | 0.00 | 0.17 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 66 | 0.50 | 0.75 | 0.50 | 0.55 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 67 | 0.00 | 0.83 | 0.00 | 1.00 | 1.00 | 1.00 | 0.67 | 1.00 | 1.00 |
| 68 | 1.00 | 0.54 | 0.55 | 0.54 | 0.75 | 1.00 | 0.50 | 0.67 | 0.67 |
| 69 | 0.00 | 0.03 | 0.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 70 | 1.00 | 0.88 | 0.81 | 0.79 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 71 | 0.00 | 0.37 | 0.00 | 0.67 | 0.75 | 1.00 | 1.00 | 0.42 | 0.42 |
| 72 | 1.00 | 0.71 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 73 | 0.75 | 0.75 | 0.65 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 74 | 1.00 | 0.54 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 75 | 0.17 | 0.35 | 0.33 | 0.38 | 1.00 | 1.00 | 0.83 | 1.00 | 1.00 |
| 76 | 0.00 | 0.30 | 0.00 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 | 0.55 |
| 77 | 1.00 | 0.80 | 0.64 | 0.45 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 78 | 0.00 | 0.33 | 0.07 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 79 | 0.50 | 0.50 | 0.50 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 80 | 0.33 | 0.73 | 0.00 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 81 | 0.17 | 0.60 | 0.17 | 0.58 | 0.92 | 0.92 | 1.00 | 1.00 | 0.92 |
| 82 | 0.00 | 0.75 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 83 | 0.00 | 0.05 | 0.02 | 0.22 | 0.36 | 0.50 | 1.00 | 1.00 | 1.00 |
| 84 | 0.05 | 0.02 | 0.00 | 0.00 | 0.27 | 0.34 | 0.54 | 0.75 | 0.75 |
| 85 | 0.50 | 1.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 86 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 87 | 0.00 | 1.00 | 0.00 | 0.45 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 88 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.17 | 1.00 | 1.00 |
| 89 | 0.02 | 0.16 | 0.43 | 0.36 | 0.93 | 0.93 | 0.93 | 0.86 | 0.86 |
| 90 | 0.00 | 0.11 | 0.00 | 0.60 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 91 | 0.00 | 0.25 | 0.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 92 | 0.50 | 0.54 | 0.50 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 93 | 0.25 | 0.17 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 94 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 95 | 0.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 96 | 0.00 | 0.00 | 1.00 | 0.05 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 |
| 97 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 98 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 |
| 99 | 0.55 | 0.03 | 0.38 | 0.15 | 1.00 | 1.00 | 0.63 | 1.00 | 0.88 |
| 100 | 0.33 | 1.00 | 0.33 | 1.00 | 0.68 | 0.67 | 1.00 | 1.00 | 1.00 |
| 101 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| 102 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 103 | 0.27 | 0.39 | 0.26 | 0.38 | 0.94 | 1.00 | 1.00 | 1.00 | 1.00 |
| 104 | 0.00 | 0.67 | 0.00 | 0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| 105 | 1.00 | 0.00 | 0.17 | 0.08 | 0.00 | 0.00 | 0.11 | 0.14 | 0.05 |
| 106 | 0.00 | 0.36 | 0.00 | 0.35 | 1.00 | 1.00 | 0.61 | 1.00 | 0.69 |
Runtime Results
| Participant | Task | Runtime (min'sec") | Machine |
|---|---|---|---|
| JL1 | 1 | 20'10" | FAST |
| JL2 | 1 | 417'36" | FAST |
| JL3 | 1 | 54'25" | FAST |
| JL4 | 1 | 297'32" | FAST2 |
| LW1 | 1 | 143'40" | FAST2 |
| RK | 1 | 71'52" | cluster |
| WL1 | 1 | 31'03" | FAST |
| WL2 | 1 | 28'19" | FAST |
| JL1 | 2 | 2'08" | FAST2 |
| JL2 | 2 | 54'05" | FAST2 |
| JL3 | 2 | 7'32" | FAST2 |
| JL4 | 2 | 59'23" | FAST2 |
| LW2 | 2 | 561'42" | FAST2 |
| LW3 | 2 | 62'39" | FAST2 |
| RK | 2 | 19'36" | cluster |
| WL1 | 2 | 8'30" | FAST |
| WL2 | 2 | 6'03" | FAST |
