Reproducible Research in Signal Processing - What, why, and how

Abstract

Have you ever tried to reproduce the results presented in a research paper? For many of our current publications, this would unfortunately be a challenging task. For a computational algorithm, details such as the exact data set, initialization or termination procedures, and precise parameter values are often omitted in the publication for various reasons, such as a lack of space, a lack of self-discipline, or an apparent lack of interest to the readers, to name a few. This makes it difficult, if not impossible, for someone else to obtain the same results. In our experience, it is often even worse as even we are not always able to reproduce our own experiments, making it difficult to answer questions from colleagues about details. Following are some examples of e-mails we have received: "I just read your paper X. It is very completely described, however I am confused by Y. Could you provide the implementation code to me for reference if possible?" "Hi! I am also working on a project related to X. I have implemented your algorithm but cannot get the same results as described in your paper. Which values should I use for parameters Y and Z?"

Reference

Vandewalle, Patrick and Kovacevic, Jelena and Vetterli, Martin (2009) Reproducible Research in Signal Processing - What, why, and how. IEEE Signal Processing Magazine, 26 (3). pp. 37-47. [BibTeX]

Full Text

Preview

PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader Copyright Notice: Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE. 523Kb

Supporting Figures

	HTML (Reviewers for reproducibility study.) 1847b
	HTML (Additional figures: Scores from reproducible research study versus number of citations.) 5Kb

Acknowledgements

We are grateful to Jon Claerbout for getting the reproducible research efforts started and to Mauro Barni and Fernando Perez-Gonzalez for starting the discussions in our signal processing community. We would like to thank the 90 reviewers for their great help in our reproducibility study. We also thank the editorial board members and the other reviewers of this article for the valuable discussions and feedback on early versions of the manuscript.

Code/Data

Here you can find the code and data to reproduce all images and tables from the paper. If you use this code in your research and publications, please also put a reference to this paper. Thank you!

HTML (Info about dataset.) 469b

Further Details