Hi,
I am writing to you today as I have been utilizing your Termination-Competition dataset, which is available at the following link: [ https://github.com/TermCOMP/TPDB https://github.com/TermCOMP/TPDB ], for my research purposes. Then I'm in trouble because expected answers corresponding to the benchmarks included in the dataset cannot be found.
Thanks for your interest in that data. It is the result of hard work by many researchers over many years. It's nice to see it used in supporting ongoing research.
Your question was answered at
https://lists.rwth-aachen.de/hyperkitty/list/termtools@lists.rwth-aachen.de/...
"ground truth" is not in the benchmarks, and can only be obtained from results tables of previous competitions. For un-certified categories, some faith is needed.
For the moment, the VBS column (virtual best solver) gives the best practical approximation.
I support the idea of collecting the status _somewhere_.
If we write it _in_ the benchmark, then we don't need an extra storage system/location (good) but each status change needs a fresh commit (bad?)
Note that VBS extends over recent years only, and there might be cases of a benchmark solved earlier, but the solution forgotten later. If you know of such instances, you might post here. E.g., for string rewriting, there were participants TPA (Adam Koprowski), Teparla (J. van der Wulp), Jambox (Jörg Endrullis) that had unique proof search methods (IIRC - semantic labeling with infinite domain, recursive semlab with (small) finite domain, efficient automata completion for matchbounds) not replicated (fully) by recent participants. Some years ago, I collected historical competition results here https://gitlab.imn.htwk-leipzig.de/waldmann/termcomp but currently there is no unified method of access.
An extension of the idea (to know the absolute truth) is to accept termination certificates that were obtained outside competition, e.g., using more resources (time). Perhaps still accept and publish at time of competition only, in order to not draw attention away from termcomp (the event).
- Johannes.