Green turtle, Chelonia mydas, Hawaii, Photo by Brocken Inaglory, GNU Free Documentation/Creative Commons Attribution license, via Wikimedia
Experiments and Analysis Papers focus on the experimental evaluation of existing algorithms, data structures, and systems. Papers proposing new techniques should continue to be submitted to the regular research track. The primary contribution of Experiments and Analysis papers is performance evaluation through analytical modelling, simulation, and/or experiments. Suitable papers can fit in different categories:
- Experimental Surveys: papers that compare a wide spectrum of approaches to a problem and, through extensive experiments, provide a comprehensive perspective on the results available and how they compare to each other.
- Result Verification: papers that verify or refute results published in the past and that, through a renewed performance evaluation, help to advance the state of the art.
- Problem Analysis: papers that focus on relevant problems or phenomena and through analysis and/or experimentation provide insights on the nature or characteristics of these phenomena.
Overall, the track places a great emphasis on comprehensive and detailed performance evaluation, appropriate methodology, and a fair, unemotional assessment of the strength and weaknesses of ideas published already in previous work. The scientific contribution of an E&A-track paper lies in providing new insights into the strengths and weaknesses of existing methods rather than providing new methods.
How to write an E&A Paper?
If E&A papers are to evaluate the approaches in published papers, they must repeat at least one central experiment of each original paper they address unchanged. In addition, they must extend the original performance evaluation by either: using other datasets, different parameters, different workloads, and/or different systems. We favor papers that reimplement published methods rather than papers that just use the original implementation. However, both variants are possible and may make sense depending on the context, especially for larger pieces of software, e.g., systems, that are hard or impossible to reimplement. E&A papers can also focus on evaluating implemented systems.
A major motivation for these requirements is not solely to identify scenarios where a particular method does not perform well (negative results and/or rebuttals), but in particular to extend the scope of the original performance evaluation. All performance evaluations are limited in one way or the other – simply through the possible effort/time/costs or space constraints authors are able to invest into it. Thus, E&A papers is also an opportunity to better understand the applicability and usefulness of published methods on unexplained terrain.
How to submit?
The submission guidelines, submission process, revisions, extended journal articles, and resubmission rule for the E&A papers are the same as all other Research track submissions for VLDB 2015. Except that for E&A papers we allow you to submit two different types of papers:
- Full Paper: a 12 page E&A research paper, just as in previous years
- Mini Paper: a 2-4 page paper solely containing an experimental evaluation.
What is a Mini Paper? This type of paper must not contain an Introduction, nor a Motivation, nor a renewed description of algorithms, nor related work. We expect a Mini Paper to be fully understandable if it is read together with the method description of the original paper. A Mini Paper solely presents a new performance evaluation of one or more published papers. In other words: the entire paper is equivalent to a performance evaluation.
Notice that the write-up of a Mini Paper does not require a justification in any way. Repeating and extending other work in order to increase the confidence in that work is already enough of a motivation. Also notice that these kinds of papers are somewhat easier to write than a full paper. Thus, they may be an interesting entry point for grad students. However, the same quality standards as for full papers apply to Mini Papers as well.
You submit both Full and Mini E&A papers in the Conference Management Tool by selecting the research track for the correct month and indicating “X: Experiments and Analysis Track Full” or “X: Experiments and Analysis Track Mini” as the primary subject area of your paper.
Additionally, please place the string “[Experiments and Analysis Paper]” or “[Experiments and Analysis Mini-Paper]” at the end of your paper title, to doubly ensure that a reviewer does not mistakenly evaluate it as a research paper. This string is required only during the review process, and should be removed when you prepare your camera-ready copy.
Papers submitted to this track will undergo the same review process as other research track papers. Reviewers will be drawn from a common review board, but there are two Associate Editors specifically responsible for this track.
What happens to Accepted Papers?
All accepted papers to the Experiments and Analysis track will be published in PVLDB. All full papers in this track will be presented at the VLDB conference in a regular paper slot. All mini papers will have at least a poster slot, and may have a presentation slot in addition.
We encourage authors of accepted papers, at the time of the publication, to make available all the experimental data and, whenever possible, the related software. For papers that identify negative or contradictory results for published results by third parties, the Program Committee may ask the third party to comment on the submission and even request a short rebuttal/explanation to be published along the submission in the event of acceptance.