Reproducible Research Repository

Reproducible Research in Signal Processing - What, why, and how

This item supports Reproducible Research Add your evaluation Leave comments

Vandewalle, Patrick and Kovacevic, Jelena and Vetterli, Martin

Abstract

Have you ever tried to reproduce the results presented in a research paper? For many of our current publications, this would unfortunately be a challenging task. For a computational algorithm, details such as the exact data set, initialization or termination procedures, and precise parameter values are often omitted in the publication for various reasons, such as a lack of space, a lack of self-discipline, or an apparent lack of interest to the readers, to name a few. This makes it difficult, if not impossible, for someone else to obtain the same results. In our experience, it is often even worse as even we are not always able to reproduce our own experiments, making it difficult to answer questions from colleagues about details. Following are some examples of e-mails we have received: "I just read your paper X. It is very completely described, however I am confused by Y. Could you provide the implementation code to me for reference if possible?" "Hi! I am also working on a project related to X. I have implemented your algorithm but cannot get the same results as described in your paper. Which values should I use for parameters Y and Z?"

Reference

Vandewalle, Patrick and Kovacevic, Jelena and Vetterli, Martin (2009) Reproducible Research in Signal Processing - What, why, and how. IEEE Signal Processing Magazine, 26 (3). pp. 37-47. [BibTeX]

Full Text

[img]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Copyright Notice: Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
523Kb

Supporting Figures

[img]HTML (Reviewers for reproducibility study.)
1847b
[img]HTML (Additional figures: Scores from reproducible research study versus number of citations.)
5Kb

Acknowledgements

We are grateful to Jon Claerbout for getting the reproducible research efforts started and to Mauro Barni and Fernando Perez-Gonzalez for starting the discussions in our signal processing community. We would like to thank the 90 reviewers for their great help in our reproducibility study. We also thank the editorial board members and the other reviewers of this article for the valuable discussions and feedback on early versions of the manuscript.

Code/Data

Here you can find the code and data to reproduce all images and tables from the paper. If you use this code in your research and publications, please also put a reference to this paper. Thank you!

[img]HTML (Info about dataset.)
469b

Further Details

Item Type:Article
Subjects:UNSPECIFIED
ID Code:17
Deposited By:Patrick Vandewalle
Deposited On:20 Jun 2008 22:17
Last Modified:28 Jun 2009 08:40

User evaluations

0 people have verified that this code works

0 people have not been able to make it work

0 people have verified that this code reproduces the results presented in the paper

0 people have not been able to reproduce the results

Submit your evaluation

I have tested this code and it works
I have tested this code and it does not work (on my computer)
I have tested this code and was able to reproduce the results from the paper
I have tested this code and was unable to reproduce the results from the paper

User comments

Posted by Patrick on 28 Jun 2009 08:40
I just discovered a paper that should have been on our reference list too: K. Price, Anything You Can Do, I Can Do Better (No You Can’t)..., Computer Vision, Graphics, and Image Processing, Vol. 36, pp. 387-391, 1986, doi:10.1016/0734-189X(86)90083-6.

Leave a comment

Your name:
Your comment:

Repository Staff Only: item control page