Dear all,

as announced, ranking is made among teams. This is my proposal of the definition of teams.

$teams = [ 'AProVE' => ['AProVE'], 'iRankFinder' => ['iRankFinder'], 'LoAT' => ['LoAT'], 'Matchbox' => ['matchbox'], 'MU-TERM' => ['MuTerm'], 'MultumNonMulta' => ['MnM'], 'NaTT' => ['NaTT'], 'NTI+cTI' => ['NTI', 'NTI+cTI'], 'SOL' => ['SOL'], 'Tyrolean Tools' => ['TTT2', 'TcT'], 'Ultimate' => ['Ultimate'], 'Wanda' => ['Wanda'], ];

To reserve some excitement for the final run, I will not show the ranking in the first run.

Best, Akihisa

On 7/19/2022 3:05 PM, YAMADA, Akihisa wrote:

Dear all,

FLoC will give us five medals. We distribute them as follows:

Gold, Silver, and Bronze medals go to the top three teams according to a competition-wide ranking. (*)

Two special medals go to the best two teams in advancing the state of the art. (**)

The unit "team" is chosen to ensure medals go to different places. When the registration is closed, I'll suggest a definition of teams. Please raise comments on it if necessary. It will ultimately be decided by SC minus conflict of interest.

The detail of the ranking is as follows:

Recall that for each benchmark, each claim (Termination, Nontermination, Upper-bound, Lower-bound, and their CERTIFIED versions) yields score in range [0,1].

The virtual best solver (VBS) records the best (consistent) score for each claim collected at least since 2018.

(*) Teams are ranked by the Euclidean norms of the normalized score vector. Each component of the vector is the score of the team in a category, divided by the score of the VBS in the category.

(**) If a team gets higher score for a benchmark and a claim than previous year's VBS, then the team gets the difference as a special score, and teams are ranked by sums of these scores. In short, if you claim YES/NO while no tool in the past claimed so, then you get special score 1.

Best, Akihisa