SAS 2024 Artifacts

Call for Artifacts

Deadlines extended! Check out the new dates!

As in previous years, we encourage authors to submit any artifacts and evaluations presented in the paper. The goal of the artifact submissions is to strengthen our field’s scientific approach to evaluations and reproducibility of results. Artifact submissions also serve as a stepping stone for future research, allowing researchers to rapidly build upon existing tools, and better evaluate or contrast existing work.

Imporant Dates:

• Artifact Submission: May 12th 2024, May 22nd 2024 AoE (Extended Deadline)
• Author Response Period: June 17th - June 19th, June 27th - July 3rd, 2024 AoE
• Artifact Notification: July 7th, July 16th, 2024 AoE

Criteria

This year, as last year, each artifact can be awarded up to three badges:

• Validated

Artifacts which can reasonably reproduce the experimental claims in the paper. This is the minimum expectation from a submitted artifact.

• Extensible

Artifacts which allow easy addition of new capabilities. To be eligible for an Extensible badge, the artifact must be bundled with the source code and pass the requirements for the Validated badge. Reviewers may attempt to tweak the source code and recompile, in order to evaluate how easy it is to build upon the tool. Please note that the artifact submission is optional. Submitting source code with the artifact is also optional. However, artifacts without source code will only be eligible for the Validated badge.

• Available

Artifacts of accepted papers will obtain the Available badge during the camera-ready period if the authors upload the submitted version of the artifact to the Zenodo SAS 2024 public repository. After the final notification of acceptance of your paper submitted to the main track of SAS24, you can add your artifact to Zenodo SAS 2024 community by publishing your artifact here. The title and set of authors of your Zenodo submission must match with your SAS submission. The files you need to add are your artifact (virtual machine or container image) and the step-by-step instructions. After you publish your artifact, we will accept it to the community if the submission is OK, and, during the camera-ready perdiod, we will award you the Available badge. Please note that in case your artifact is not eligible for the Validated and Extensible badges, or in case, for any reason, you choose not to submit your artifact to the SAS’24 artifact track (thus not undergoing review by the AEC), you can still be awarded the Available badge.

Packaging

Authors can package their artifact either as a virtual machine image or a Docker container image, and provide step-by-step instructions for loading/running the artifact.

• Virtual machine image: The VM image must be bootable and contain all the necessary libraries installed. Please ensure that the VM image can be processed with VirtualBox. While preparing your artifact, please make it as lightweight as possible. Consider installing ssh in the VM to make it reachable even when virtual screen isn’t working (common problem when changing host hardware).

• Container image: Authors can provide a zipped version of a container image containing the experiments, along with all dependencies. Please ensure that the zipped container image can be loaded using the docker load command. Please opt for container images only if your experiments can be executed and validated from the command line.

• Step-by-step instructions: This should clearly explain how to reproduce the results that support your paper’s conclusions. We encourage the authors to include easy-to-run scripts. Note that a reviewer may be evaluating your artifact on weaker hardware compared to your setup, so try to parameterize your scripts to allow running smaller versions of your experiments. Also, you should explain how to interpret the outputs of the artifact. If possible, please provide an estimate of the execution times for the experiments. Lastly, if you are providing the source code, try to provide an outline of how the code is structured.

Submission

Please follow the instructions below to submit your artifact:

• Package the VM/container image and the instruction document into a single compressed archive file using zip or gzip. Use your paper number for the name of the archive file.
• Upload the archive file to well-known storage service such as Dropbox or Google Drive and get the sharable link of it.
• Run a checksum function on the archive file.
• Submit the link and the checksum via the specific submission page on EasyChair. You may put the link, checksum and possible extra informations in the abstract text field (in which case, please upload the paper submission as the file), or upload a pdf/txt containing the checksum/link/extra info (and then, the abstract can be empty.

All artifacts will follow a double-blind process: author names and affiliations must be hidden in every file (paper, readme, source code, …). This does not hold for artifacts related to NEAT papers: they will follow a single-blind review process (namely, no need to anonymize your artifact).

Submission link: https://easychair.org/conferences/?conf=sas24

Badges

The SAS Artifact Evaluation badges were designed by Arpita Biswas and Suvam Mukherjee, and are available for download from GitHub.

This year at SAS’24 the Radhia Cousot Award will be announced at the conference! All young researchers author of a paper accepted at SAS24 are invited to join SAS24 and register at the conference!