Dear Termination-Portal Organization.
I hope this email finds you well. My name is Yurika Hirobe, and I am currently a master's student at the University of Tokyo.
I am writing to you today as I have been utilizing your Termination-Competition dataset, which is available at the following link: [ https://github.com/TermCOMP/TPDB ], for my research purposes. Then I'm in trouble because expected answers corresponding to the benchmarks included in the dataset cannot be found.
From my understanding, I believe that there should be XML files that provide comprehensive information, including the expected answers for each benchmark. Unfortunately, I have encountered some difficulties in locating such files within the dataset. It is crucial for my research to have access to this information to ensure the accuracy and completeness of my analysis.
Hence, I kindly request your guidance in this matter. Could you please direct me to the specific location or resources where I can find the expected answers for the dataset? Your assistance in this regard would be of immense help to my research project.
Thank you for your time and consideration. I look forward to hearing from you.
Best regards, Yutika Hirobe Master's Student University of Tokyo yurikahirobe@gmail.com
Hi,
I am writing to you today as I have been utilizing your Termination-Competition dataset, which is available at the following link: [ https://github.com/TermCOMP/TPDB https://github.com/TermCOMP/TPDB ], for my research purposes. Then I'm in trouble because expected answers corresponding to the benchmarks included in the dataset cannot be found.
Thanks for your interest in that data. It is the result of hard work by many researchers over many years. It's nice to see it used in supporting ongoing research.
Your question was answered at
https://lists.rwth-aachen.de/hyperkitty/list/termtools@lists.rwth-aachen.de/...
"ground truth" is not in the benchmarks, and can only be obtained from results tables of previous competitions. For un-certified categories, some faith is needed.
For the moment, the VBS column (virtual best solver) gives the best practical approximation.
I support the idea of collecting the status _somewhere_.
If we write it _in_ the benchmark, then we don't need an extra storage system/location (good) but each status change needs a fresh commit (bad?)
Note that VBS extends over recent years only, and there might be cases of a benchmark solved earlier, but the solution forgotten later. If you know of such instances, you might post here. E.g., for string rewriting, there were participants TPA (Adam Koprowski), Teparla (J. van der Wulp), Jambox (Jörg Endrullis) that had unique proof search methods (IIRC - semantic labeling with infinite domain, recursive semlab with (small) finite domain, efficient automata completion for matchbounds) not replicated (fully) by recent participants. Some years ago, I collected historical competition results here https://gitlab.imn.htwk-leipzig.de/waldmann/termcomp but currently there is no unified method of access.
An extension of the idea (to know the absolute truth) is to accept termination certificates that were obtained outside competition, e.g., using more resources (time). Perhaps still accept and publish at time of competition only, in order to not draw attention away from termcomp (the event).
- Johannes.
Hi,
thank you for the interest, indeed.
For the moment, the VBS column (virtual best solver) gives the best practical approximation.
I support the idea of collecting the status _somewhere_.
I just pushed the VBSs from 2023 here: https://github.com/TermCOMP/starexec-master/tree/master/Y2023
But this location is just a result of my ad-lib implementation of VBSs. If this should be made somewhat official, probably it's nicer to make a new repository for past records.
If we write it _in_ the benchmark, then we don't need an extra storage system/location (good) but each status change needs a fresh commit (bad?)
A bigger bad is that rendering the result tables will be complicated, as one must parse a benchmark for every row.
An extension of the idea (to know the absolute truth) is to accept termination certificates that were obtained outside competition, e.g., using more resources (time). Perhaps still accept and publish at time of competition only, in order to not draw attention away from termcomp (the event).
Currently, the VBS only collect YES or NO or complexity bounds. At least CERTIFIED flags should be useful. This extension would mean to allow more flags.
Best, Akihisa
termtools@lists.rwth-aachen.de
-
Johannes Waldmann -
YAMADA, Akihisa -
Yurika Hirobe