Have you ever complained that you cannot reproduce an experiment from an RTSS paper?
Have you ever wished you could have someone else validate your work?
Have you ever tried to validate someone else’s experiment?
AE is intended to help validate experimental results from the accepted publications (see more information in the links below), and not to prove formally that the artifact is correct. This is akin to the paper reviewing process, where qualified experts give feedback to the authors, and papers/artifacts are kept confidential and under control of the authors. Notifications of “passing” evaluation will be sent to the authors only 4 weeks after artifact is submitted.
This process is relatively straightforward. Participation in the AE is optional by submitting all related research material (a guide/README, simulators, tools, benchmarks, data sets, configuration files, …) necessary to validate claims and results of their accepted paper. The guide is a README-style file describes the artifacts and experimental procedure for their artifact. Two reviewers will follow your guide, install your software, rerun your experiments, recreate your graphs, and send you a report with the overall assessment of your artifact. Specific information about how to submit can be found here.
The window of submission is very wide, to allow authors to include a note in their camera-ready papers if they so wish. The AE webpage will recognize all the artifacts that passed evaluation. The discussion period is to enable authors to improve their artifacts for eventual sharing, which is the goal of this exercise. The AE chairs will mediate all communication to keep the reviewers identities confidential. The chairs also offer help in packaging the artifact to the first 5 authors that request help.
Important Dates
Flexible Submission Window: July 13 to Sept 13, 2016, but must submit by July 26 to get notification before camera-ready paper is due. Submissions must be made at: https://www.softconf.com/g/ae2016.
Feedback step/Discussion period: In the 3 weeks following submission, the evaluators and authors will potentially engage in a helpful discussion. Evaluators give initial feedback to authors about any problems in the first few days, let the authors fix them, and then resume the evaluation. This is to make sure reviewers can unpackage and run the artifact before doing a deeper evaluation.
Notification: 4 weeks after artifact submission.
AE Committee
Bruce Childers, Univ of Pittsburgh (co-chair)
Daniel Mosse, Univ of Pittsburgh (co-chair)
Other members TBD