Scholarly Activities & Output

2018

Juliana Freire and Fernando Chirigati. Provenance and the Different Flavors of Computational Reproducibility. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, pages 12, March 2018. URL: http://sites.computer.org/debull/A18mar/p15.pdf.
[abstract▼] [BibTeX▼]

While reproducibility has been a requirement in natural sciences for centuries, computational experiments have not followed the same standard. Often, there is insufﬁcient information to reproduce computational results described in publications, and in the recent past, this has led to many retractions. Although scientists are aware of the numerous beneﬁts of reproducibility, the perceived amount of work to make results reproducible is a signiﬁcant disincentive. Fortunately, much of the information needed to reproduce an experiment can be obtained by systematically capturing its provenance. In this paper, we give an overview of different types of provenance and how they can be used to support reproducibility. We also describe a representative set of provenance tools and approaches that make it easy to create reproducible experiments.
```
@article{freire_provenance_nodate,
    author = "Freire, Juliana and Chirigati, Fernando",
    language = "en",
    url = "http://sites.computer.org/debull/A18mar/p15.pdf",
    title = "Provenance and the {Different} {Flavors} of {Computational} {Reproducibility}",
    month = "March",
    journal = "Bulletin of the IEEE Computer Society Technical Committee on Data Engineering",
    pages = "12",
    year = "2018"
}
```
REANA: Reusable research data analysis platform. 2018. URL: https://github.com/reanahub/reana.
[abstract▼] [BibTeX▼]

REANA is a reusable and reproducible research data analysis platform. It helps researchers to structure their input data, analysis code, containerised environments and computational workflows so that the analysis can be instantiated and run on remote compute clouds. REANA was born to target the use case of particle physics analyses, but is applicable to any scientific discipline. The system paves the way towards reusing and reinterpreting preserved data analyses even several years after the original publication.
```
@misc{noauthor_reana:_nodate,
    url = "https://github.com/reanahub/reana",
    title = "{REANA}: {Reusable} research data analysis platform",
    publisher = "CERN",
    year = "2018"
}
```

2017

Vicky Steeves, Rémi Rampin, and Fernando Chirigati. Using ReproZip for Reproducibility and Library Services. IASSIST Quarterly, 42:14–14, December 2017. URL: https://iassistquarterly.com/index.php/iassist/article/view/18, doi:10.29173/iq18.
[abstract▼] [BibTeX▼]

Achieving research reproducibility is challenging in many ways: there are social and cultural obstacles as well as a constantly changing technical landscape that makes replicating and reproducing research difficult. Users face challenges in reproducing research across different operating systems, in using different versions of software across long projects and among collaborations, and in using publicly available work. The dependencies required to reproduce the computational environments in which research happens can be exceptionally hard to track – in many cases, these dependencies are hidden or nested too deeply to discover, and thus impossible to install on a new machine, which means adoption remains low. In this paper, we present ReproZip , an open source tool to help overcome the technical difficulties involved in preserving and replicating research, applications, databases, software, and more. We will examine the current use cases of ReproZip , ranging from digital humanities to machine learning. We also explore potential library use cases for ReproZip, particularly in digital libraries and archives, liaison librarianship, and other library services. We believe that libraries and archives can leverage ReproZip to deliver more robust reproducibility services, repository services, as well as enhanced discoverability and preservation of research materials, applications, software, and computational environments.
```
@article{steeves_using_2018,
    author = "Steeves, Vicky and Rampin, Rémi and Chirigati, Fernando",
    language = "en",
    url = "https://iassistquarterly.com/index.php/iassist/article/view/18",
    title = "Using {ReproZip} for {Reproducibility} and {Library} {Services}",
    month = "December",
    journal = "IASSIST Quarterly",
    pages = "14--14",
    volume = "42",
    issn = "0739-1137",
    year = "2017",
    doi = "10.29173/iq18"
}
```

Vicky Steeves. Reproducibility Librarianship. Collaborative Librarianship, July 2017. URL: https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss2/4.
[BibTeX▼]

@article{steeves_reproducibility_2017,
    author = "Steeves, Vicky",
    url = "https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss2/4",
    title = "Reproducibility {Librarianship}",
    month = "July",
    journal = "Collaborative Librarianship",
    volume = "9",
    issn = "1943-7528",
    year = "2017",
    number = "2"
}

Matthew Spitzer, Vicky Steeves, and Cynthia Hudson-Vitale. Open, Public Goods Infrastructure for Research Management & Discovery. IASSIST 2017, May 2017. URL: https://osf.io/sy2zf/.
[abstract▼] [BibTeX▼]

Presentations on open technical infrastructure that supports research reproducibility, open access mandates, and data management and sharing requirements. Panel on Thursday, May 25th, 2017 at 4pm.
```
@unpublished{spitzer_open_2017,
    author = "Spitzer, Matthew and Steeves, Vicky and Hudson-Vitale, Cynthia",
    language = "en",
    url = "https://osf.io/sy2zf/",
    title = "Open, {Public} {Goods} {Infrastructure} for {Research} {Management} \\& {Discovery}",
    month = "May",
    address = "Lawrence, Kansas",
    note = "IASSIST 2017",
    year = "2017",
    copyright = "CC 1.0 Universal"
}
```

Vicky Steeves and Rémi Rampin. ReproZip for Reproducible Research. PresQT, April 2017. URL: https://osf.io/s6fw9/.
[BibTeX▼]

@unpublished{steeves_reprozip_2017,
    author = "Steeves, Vicky and Rampin, Rémi",
    language = "en",
    url = "https://osf.io/s6fw9/",
    title = "{ReproZip} for {Reproducible} {Research}",
    month = "April",
    address = "South Bend, Indiana",
    note = "PresQT",
    year = "2017",
    copyright = "CC-By Attribution 4.0 International"
}

Vicky Steeves, Rémi Rampin, and Daniel Nüst. Reproducible computational research in the publication cycle. European Geosciences Union General Assembly, April 2017. URL: https://osf.io/umy6g/.
[abstract▼] [BibTeX▼]

Materials for the short course "Reproducible computational research in the publication cycle" at the European Geosciences Union General Assembly 2017. Short course website: http://meetingorganizer.copernicus.org/EGU2017/session/25726. Hosted on the Open Science Framework
```
@unpublished{steeves_reproducible_2017,
    author = "Steeves, Vicky and Rampin, Rémi and Nüst, Daniel",
    language = "en",
    url = "https://osf.io/umy6g/",
    title = "Reproducible computational research in the publication cycle",
    month = "April",
    address = "Vienna, Austria",
    note = "European Geosciences Union General Assembly",
    year = "2017",
    copyright = "CC-By Attribution 4.0 International"
}
```
Vicky Steeves. Reproducibility: the What, Why, & How. ACRL, September 2017. URL: https://dx.doi.org/10.17605/OSF.IO/CYR3N.
[abstract▼] [BibTeX▼]

Vicky Steeves' presentation from the panel 'Open Science: Understanding Modern Research Practices' presented with Robin Champieux, Jeff Leek, Brett Davidson (organizer), Eka Grguric (organizer).
```
@unpublished{steeves_reproducibility:_2017,
    author = "Steeves, Vicky",
    shorttitle = "Reproducibility",
    url = "https://dx.doi.org/10.17605/OSF.IO/CYR3N",
    title = "Reproducibility: the {What}, {Why}, \\& {How}",
    month = "September",
    address = "Baltimore, Maryland",
    note = "ACRL",
    year = "2017"
}
```

Vicky Steeves and Rémi Rampin. Reproducing and Preserving Research with ReproZip. IASSIST 2017, May 2017. URL: https://vickysteeves.gitlab.io/2017-IASSIST-ReproZip/#/.
[BibTeX▼]

@unpublished{steeves_reproducing_2017,
    author = "Steeves, Vicky and Rampin, Rémi",
    url = "https://vickysteeves.gitlab.io/2017-IASSIST-ReproZip/\#/",
    title = "Reproducing and {Preserving} {Research} with {ReproZip}",
    month = "May",
    address = "Lawrence, Kansas",
    note = "IASSIST 2017",
    year = "2017",
    copyright = "CC-By Attribution 4.0 International"
}

Vicky Steeves and Rémi Rampin. Creating Reproducible Experiments with ReproZip. SciPy, July 2017. URL: https://vickysteeves.gitlab.io/2017-SciPy/#/.
[abstract▼] [BibTeX▼]

Reproducibility is a core component of the scientific process: it helps researchers all around the world to verify research results and also to build on them, allowing science to move forward. Unfortunately, computational reproducibility can be very painful. We’ll present an open source tool for computational reproducibility, ReproZip. ReproZip is written in Python, and was designed to simplify the process of making an experiment reproducible across platforms. ReproZip creates self-contained, reproducible packages by automatically tracking, identifying, and capturing all its required dependencies: programs, libraries, data, and configuration files. The original user can share the package with others, who can then use ReproZip to unpack and rerun the experiment on their favorite operating system.
```
@unpublished{steeves_creating_2017,
    author = "Steeves, Vicky and Rampin, Rémi",
    language = "en",
    url = "https://vickysteeves.gitlab.io/2017-SciPy/\#/",
    title = "Creating {Reproducible} {Experiments} with {ReproZip}",
    month = "July",
    address = "Austin, Texas",
    note = "SciPy",
    year = "2017",
    copyright = "CC-By Attribution 4.0 International"
}
```
Vicky Steeves. How GitLab can help in research reproducibility. August 2017. URL: https://about.gitlab.com/2017/08/25/gitlab-and-reproducibility/.
[abstract▼] [BibTeX▼]

NYU reproducibility librarian Vicky Steeves shares why GitLab is her choice for ongoing collaborative research, and how it can help overcome challenges with sharing code in academia.
```
@misc{steeves_how_2017,
    author = "Steeves, Vicky",
    language = "en",
    url = "https://about.gitlab.com/2017/08/25/gitlab-and-reproducibility/",
    title = "How {GitLab} can help in research reproducibility",
    type = "Blog",
    journal = "GitLab",
    year = "2017",
    month = "August"
}
```

Rémi Rampin. ReproServer. May 2017. URL: https://github.com/ViDA-NYU/reproserver.
[abstract▼] [BibTeX▼]

@misc{rampin_reproserver_2018,
    author = "Rampin, Rémi",
    shorttitle = "reproserver",
    url = "https://github.com/ViDA-NYU/reproserver",
    title = "{ReproServer}",
    publisher = "ViDA-NYU",
    month = "May",
    year = "2017",
    collaborator = "Steeves, Vicky and Freire, Juliana and Chirigati, Fernando",
    copyright = "BSD-3-Clause"
}

Fernando Chirigati and Juliana Freire. Provenance and Reproducibility. In Encyclopedia of Database Systems. Springer, May 2017. URL: https://link.springer.com/referenceworkentry/10.1007/978-1-4899-7993-3_80747-1, doi:10.1007/978-1-4899-7993-3_80747-1.
[abstract▼] [BibTeX▼]

Computational reproducibility A computational experiment composed by a sequence of steps S created at time T, on environment (hardware and operating system) E, using data D is reproducibleif it can...
```
@incollection{chirigati_provenance_2017,
    author = "Chirigati, Fernando and Freire, Juliana",
    language = "en",
    isbn = "978-1-4899-7993-3",
    title = "Provenance and {Reproducibility}",
    publisher = "Springer",
    month = "May",
    url = "https://link.springer.com/referenceworkentry/10.1007/978-1-4899-7993-3\_80747-1",
    year = "2017",
    doi = "10.1007/978-1-4899-7993-3\_80747-1",
    booktitle = "Encyclopedia of {Database} {Systems}"
}
```
Rémi Rampin, Tommy Ellqvist, David Koop, Fernando Chirigati, Juliana Freire, and Claudio Silva. VisTrails. June 2017. URL: https://github.com/VisTrails/VisTrails.
[abstract▼] [BibTeX▼]

VisTrails is an open-source data analysis and visualization tool. It provides a comprehensive provenance infrastructure that maintains detailed history information about the steps followed and data..
```
@misc{rampin_vistrails_2018,
    author = "Rampin, Rémi and Ellqvist, Tommy and Koop, David and Chirigati, Fernando and Freire, Juliana and Silva, Claudio",
    month = "June",
    url = "https://github.com/VisTrails/VisTrails",
    title = "{VisTrails}",
    publisher = "NYU",
    year = "2017",
    copyright = "BSD-3-Clause"
}
```
Katherine Boss and Meredith Broussard. Challenges of archiving and preserving born-digital news applications , Challenges of archiving and preserving born-digital news applications. IFLA Journal, 43(2):150–157, June 2017. URL: https://doi.org/10.1177/0340035216686355, doi:10.1177/0340035216686355.
[abstract▼] [BibTeX▼]

Born-digital news content is increasingly becoming the format of the first draft of history. Archiving and preserving this history is of paramount importance to the future of scholarly research, but many technical, legal, financial, and logistical challenges stand in the way of these efforts. This is especially true for news applications, or custom-built websites that comprise some of the most sophisticated journalism stories today, such as the “Dollars for Docs” project by ProPublica. Many news applications are standalone pieces of software that query a database, and this significant subset of apps cannot be archived in the same way as text-based news stories, or fully captured by web archiving tools such as Archive-It. As such, they are currently disappearing. This paper will outline the various challenges facing the archiving and preservation of born-digital news applications, as well as outline suggestions for how to approach this important work., Born-digital news content is increasingly becoming the format of the first draft of history. Archiving and preserving this history is of paramount importance to the future of scholarly research, but many technical, legal, financial, and logistical challenges stand in the way of these efforts. This is especially true for news applications, or custom-built websites that comprise some of the most sophisticated journalism stories today, such as the “Dollars for Docs” project by ProPublica. Many news applications are standalone pieces of software that query a database, and this significant subset of apps cannot be archived in the same way as text-based news stories, or fully captured by web archiving tools such as Archive-It. As such, they are currently disappearing. This paper will outline the various challenges facing the archiving and preservation of born-digital news applications, as well as outline suggestions for how to approach this important work.
```
@article{boss_challenges_2017,
    author = "Boss, Katherine and Broussard, Meredith",
    language = "en",
    url = "https://doi.org/10.1177/0340035216686355",
    title = "Challenges of archiving and preserving born-digital news applications , {Challenges} of archiving and preserving born-digital news applications",
    pages = "150--157",
    year = "2017",
    doi = "10.1177/0340035216686355",
    month = "June",
    journal = "IFLA Journal",
    volume = "43",
    issn = "0340-0352",
    number = "2"
}
```
Kyle Cranmer and Lukas Heinrich. Yadage and Packtivity - analysis preservation using parametrized workflows. Journal of Physics: Conference Series, 898:102019, October 2017. arXiv: 1706.01878. URL: http://arxiv.org/abs/1706.01878, doi:10.1088/1742-6596/898/10/102019.
[abstract▼] [BibTeX▼]

Preserving data analyses produced by the collaborations at LHC in a parametrized fashion is crucial in order to maintain reproducibility and re-usability. We argue for a declarative description in terms of individual processing steps - packtivities - linked through a dynamic directed acyclic graph (DAG) and present an initial set of JSON schemas for such a description and an implementation - yadage - capable of executing workflows of analysis preserved via Linux containers.
```
@article{cranmer_yadage_2017,
    author = "Cranmer, Kyle and Heinrich, Lukas",
    url = "http://arxiv.org/abs/1706.01878",
    title = "Yadage and {Packtivity} - analysis preservation using parametrized workflows",
    pages = "102019",
    keywords = "High Energy Physics - Experiment, Physics - Data Analysis, Statistics and Probability",
    year = "2017",
    doi = "10.1088/1742-6596/898/10/102019",
    note = "arXiv: 1706.01878",
    month = "October",
    journal = "Journal of Physics: Conference Series",
    volume = "898",
    issn = "1742-6588, 1742-6596"
}
```
Lukas Heinrich and Kyle Cranmer. Analysis Preservation and Systematic Reinterpretation within the ATLAS Experiment. ACAT 2017, August 2017. URL: https://indico.cern.ch/event/567550/contributions/2638695/.
[abstract▼] [BibTeX▼]

The LHC data analysis software used in order to derive and publish experimental results is an important asset that is necessary to preserve in order to fully exploit the scientific potential of a given measurement. Among others, important use cases of analysis preservation are the reproducibility of the original results and the reusability of the analysis procedure in the context of new scientific studies. A prominent use-case for the latter is the systematic reinterpretation of searches for new Physics in terms of signal models that not studied in the original publication (RECAST). This paper presents the usage of the graph-based workflow description language yadage to drive the reinterpretation of preserved HEP analyses. The analysis software for individual states in the analysis is preserved using Docker containers, while the workflow structure is preserved using plain JSON documents. This allows the re-execution of complex analysis workflows on industry standard container-based distributed computing clusters (Kubernetes via OpenStack Magnum) We present re-interpretations of ATLAS analyses based on both the original ATLAS analysis code and third-party re-implementations such as CheckMATE and integrations with other analysis preservation efforts such as the CERN Analysis Preservation Portal.
```
@unpublished{heinrich_analysis_2017,
    author = "Heinrich, Lukas and Cranmer, Kyle",
    language = "en",
    url = "https://indico.cern.ch/event/567550/contributions/2638695/",
    title = "Analysis {Preservation} and {Systematic} {Reinterpretation} within the {ATLAS} {Experiment}",
    month = "August",
    address = "CERN",
    note = "ACAT 2017",
    year = "2017"
}
```

Lukas Heinrich. ATLAS software and containers. March 2017.
[BibTeX▼]

@misc{heinrich_atlas_2017,
    author = "Heinrich, Lukas",
    language = "en",
    title = "{ATLAS} software and containers",
    month = "March",
    year = "2017",
    address = "CERN"
}

Kyle Cranmer and Lukas Heinrich. CERN Analysis Preservation/DASPOS/RECAST workshop. DASPOS @ CERN, February 2017. URL: https://indico.cern.ch/event/611389/.
[abstract▼] [BibTeX▼]

The aims of this mini workshop are: 1) getting community members together to give feedback on the first prototype of CERN Analysis Preservation (CAP) and its connection with RECAST and DASPOS work, 2) special emphasis will be given to the three high-level use cases of CAP: describe, capture and reuse a physics analysis. Dedicated sessions will allow a more detailed discussion of functionalities and expectations from the community to inform the roadmap of the services/tools/integrations. Based on the demonstration of the recent prototype most of the day will be spent on a discussion of the next steps. Examples include, but are not limited to: Analysis submission, are we missing out on any types of analysis, What is needed (beyond RECAST) for reuse/reinterpretation, what is needed or interesting to do on top of CAP, how can tools be connected (for example connected to CAP), search terms of interest, etc.
```
@unpublished{cranmer_cern_2017,
    author = "Cranmer, Kyle and Heinrich, Lukas",
    url = "https://indico.cern.ch/event/611389/",
    title = "{CERN} {Analysis} {Preservation}/{DASPOS}/{RECAST} workshop",
    month = "February",
    address = "CERN",
    note = "DASPOS @ CERN",
    year = "2017"
}
```

2016

Rémi Rampin, Fernando Chirigati, Dennis Shasha, Juliana Freire, and Vicky Steeves. ReproZip: The Reproducibility Packer. The Journal of Open Source Software, 1(8):107, December 2016. URL: http://joss.theoj.org/papers/10.21105/joss.00107, doi:10.21105/joss.00107.
[BibTeX▼]

@article{rampin_reprozip:_2016,
    author = "Rampin, Rémi and Chirigati, Fernando and Shasha, Dennis and Freire, Juliana and Steeves, Vicky",
    shorttitle = "{ReproZip}",
    url = "http://joss.theoj.org/papers/10.21105/joss.00107",
    title = "{ReproZip}: {The} {Reproducibility} {Packer}",
    month = "December",
    journal = "The Journal of Open Source Software",
    pages = "107",
    volume = "1",
    year = "2016",
    doi = "10.21105/joss.00107",
    number = "8",
    language = "en"
}

Vicky Steeves and Rémi Rampin. Workshop Use Case 2: ReproZip. DASPOS Workshop 1, March 2016. URL: https://osf.io/aqztf/.
[abstract▼] [BibTeX▼]

@unpublished{steeves_workshop_2016,
    author = "Steeves, Vicky and Rampin, Rémi",
    shorttitle = "Workshop {Use} {Case} 2",
    url = "https://osf.io/aqztf/",
    title = "Workshop {Use} {Case} 2: {ReproZip}",
    month = "March",
    address = "South Bend, Indiana",
    note = "DASPOS Workshop 1",
    year = "2016",
    language = "en"
}

Vicky Steeves, Rémi Rampin, and Fernando Chirigati. ReproZip Examples. March 2016. URL: https://examples.reprozip.org/.
[abstract▼] [BibTeX▼]

This site holds some examples and use cases from different domains that showcase the ReproZip software packing tool. You’ll find more information about each particular example in its associated README when you click the "REPRODUCE THIS" button below. Instructions for reproducing the examples use the vagrant and the docker unpackers. However, any of the available unpackers can be used.
```
@misc{steeves_reprozip_nodate,
    author = "Steeves, Vicky and Rampin, Rémi and Chirigati, Fernando",
    language = "en",
    url = "https://examples.reprozip.org/",
    title = "{ReproZip} {Examples}",
    month = "March",
    journal = "ReproZip Examples",
    year = "2016"
}
```

Fernando Chirigati, Rebecca Capone, Rémi Rampin, Juliana Freire, and Dennis Shasha. A collaborative approach to computational reproducibility. Information Systems, 59:95–97, July 2016. URL: http://linkinghub.elsevier.com/retrieve/pii/S0306437916300813, doi:10.1016/j.is.2016.03.002.
[BibTeX▼]

@article{chirigati_collaborative_2016,
    author = "Chirigati, Fernando and Capone, Rebecca and Rampin, Rémi and Freire, Juliana and Shasha, Dennis",
    language = "en",
    url = "http://linkinghub.elsevier.com/retrieve/pii/S0306437916300813",
    title = "A collaborative approach to computational reproducibility",
    month = "July",
    journal = "Information Systems",
    pages = "95--97",
    volume = "59",
    issn = "03064379",
    year = "2016",
    doi = "10.1016/j.is.2016.03.002"
}

Andreas Wolke, Martin Bichler, Fernando Chirigati, and Vicky Steeves. Reproducible experiments on dynamic resource allocation in cloud data centers. Information Systems, 59:98–101, July 2016. URL: http://www.sciencedirect.com/science/article/pii/S0306437915301113, doi:10.1016/j.is.2015.12.004.
[abstract▼] [BibTeX▼]

In Wolke et al. [1] we compare the efficiency of different resource allocation strategies experimentally. We focused on dynamic environments where virtual machines need to be allocated and deallocated to servers over time. In this companion paper, we describe the simulation framework and how to run simulations to replicate experiments or run new experiments within the framework.
```
@article{wolke_reproducible_2016,
    author = "Wolke, Andreas and Bichler, Martin and Chirigati, Fernando and Steeves, Vicky",
    url = "http://www.sciencedirect.com/science/article/pii/S0306437915301113",
    title = "Reproducible experiments on dynamic resource allocation in cloud data centers",
    month = "July",
    journal = "Information Systems",
    pages = "98--101",
    keywords = "Cloud computing, Dynamic resource allocation, Reproducibility",
    volume = "59",
    issn = "0306-4379",
    year = "2016",
    doi = "10.1016/j.is.2015.12.004"
}
```
Fernando Chirigati, Rémi Rampin, Dennis Shasha, and Juliana Freire. ReproZip: Computational Reproducibility With Ease. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD '16, 2085–2088. ACM, June 2016. URL: http://doi.acm.org/10.1145/2882903.2899401, doi:10.1145/2882903.2899401.
[abstract▼] [BibTeX▼]

We present ReproZip, the recommended packaging tool for the SIGMOD Reproducibility Review. ReproZip was designed to simplify the process of making an existing computational experiment reproducible across platforms, even when the experiment was put together without reproducibility in mind. The tool creates a self-contained package for an experiment by automatically tracking and identifying all its required dependencies. The researcher can share the package with others, who can then use ReproZip to unpack the experiment, reproduce the findings on their favorite operating system, as well as modify the original experiment for reuse in new research, all with little effort. The demo will consist of examples of non-trivial experiments, showing how these can be packed in a Linux machine and reproduced on different machines and operating systems. Demo visitors will also be able to pack and reproduce their own experiments.
```
@inproceedings{chirigati_reprozip:_2016,
    author = "Chirigati, Fernando and Rampin, Rémi and Shasha, Dennis and Freire, Juliana",
    shorttitle = "{ReproZip}",
    isbn = "978-1-4503-3531-7",
    title = "{ReproZip}: {Computational} {Reproducibility} {With} {Ease}",
    pages = "2085--2088",
    url = "http://doi.acm.org/10.1145/2882903.2899401",
    year = "2016",
    doi = "10.1145/2882903.2899401",
    month = "June",
    publisher = "ACM",
    keywords = "computational reproducibility, provenance, reprozip",
    series = "{SIGMOD} '16",
    booktitle = "Proceedings of the 2016 {International} {Conference} on {Management} of {Data}"
}
```
Juliana Freire, Norbert Fuhr, and Andreas Rauber. Reproducibility of Data-Oriented Experiments in e-Science. Dagstuhl Reports, 6(1):108–159, 2016. URL: http://drops.dagstuhl.de/opus/volltexte/2016/5817, doi:10.4230/DagRep.6.1.108.
[abstract▼] [BibTeX▼]

This report documents the program and the outcomes of Dagstuhl Seminar 16041 "Reproducibility of Data-Oriented Experiments in e-Science". In many subfields of computer science, experiments play an important role. Besides theoretic properties of algorithms or methods, their effectiveness and performance often can only be validated via experimentation. In most of these cases, the experimental results depend on the input data, settings for input parameters, and potentially on characteristics of the computational environment where the experiments were designed and run. Unfortunately, most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available. This has serious implications. Scientific discoveries do not happen in isolation. Important advances are often the result of sequences of smaller, less significant steps. In the absence of results that are fully documented, reproducible, and generalizable, it becomes hard to re-use and extend these results. Besides hindering the ability of others to leverage our work, and consequently limiting the impact of our field, the absence of reproducibility experiments also puts our reputation at stake, since reliability and validity of empiric results are basic scientific principles. This seminar brought together experts from various sub-fields of computer science to create a joint understanding of the problems of reproducibility of experiments, discussing existing solutions and impediments, and proposing ways to overcome current limitations.
```
@article{freire_reproducibility_2016,
    author = "Freire, Juliana and Fuhr, Norbert and Rauber, Andreas",
    url = "http://drops.dagstuhl.de/opus/volltexte/2016/5817",
    title = "Reproducibility of {Data}-{Oriented} {Experiments} in e-{Science}",
    month = "",
    journal = "Dagstuhl Reports",
    pages = "108--159",
    volume = "6",
    issn = "2192-5283",
    year = "2016",
    doi = "10.4230/DagRep.6.1.108",
    number = "1"
}
```
Troy Kohwalter, Thiago Oliveira, Juliana Freire, Esteban Clua, and Leonardo Murta. Prov Viewer: A Graph-Based Visualization Tool for Interactive Exploration of Provenance Data. In Provenance and Annotation of Data and Processes, Lecture Notes in Computer Science, 71–82. Springer, Cham, June 2016. URL: https://link.springer.com/chapter/10.1007/978-3-319-40593-3_6, doi:10.1007/978-3-319-40593-3_6.
[abstract▼] [BibTeX▼]

The analysis of provenance data for an experiment is often crucial to understand the achieved results. For long-running experiments or when provenance is captured at a low granularity, this analysis process can be overwhelming to the user due to the large volume of provenance data. In this paper we introduce, Prov Viewer, a provenance visualization tool that enables users to interactively explore provenance data. Among the visualization and exploratory features, we can cite zooming, filtering, and coloring. Moreover, we use of other properties such as shape and size to distinguish visual elements. These exploratory features are linked to the provenance semantics to ease the comprehension process. We also introduce collapsing and filtering strategies, allowing different levels of granularity exploration and analysis. We describe case studies that show how Prov Viewer has been successfully used to explore provenance in different domains, including games and urban data.
```
@inproceedings{kohwalter_prov_2016,
    author = "Kohwalter, Troy and Oliveira, Thiago and Freire, Juliana and Clua, Esteban and Murta, Leonardo",
    shorttitle = "Prov {Viewer}",
    isbn = "978-3-319-40592-6 978-3-319-40593-3",
    title = "Prov {Viewer}: {A} {Graph}-{Based} {Visualization} {Tool} for {Interactive} {Exploration} of {Provenance} {Data}",
    pages = "71--82",
    url = "https://link.springer.com/chapter/10.1007/978-3-319-40593-3\_6",
    year = "2016",
    doi = "10.1007/978-3-319-40593-3\_6",
    language = "en",
    month = "June",
    publisher = "Springer, Cham",
    series = "Lecture {Notes} in {Computer} {Science}",
    booktitle = "Provenance and {Annotation} of {Data} and {Processes}"
}
```
João Felipe Pimentel, Juliana Freire, Vanessa Braganholo, and Leonardo Murta. Tracking and Analyzing the Evolution of Provenance from Scripts. In Provenance and Annotation of Data and Processes, Lecture Notes in Computer Science, 16–28. Springer, Cham, June 2016. URL: https://link.springer.com/chapter/10.1007/978-3-319-40593-3_2, doi:10.1007/978-3-319-40593-3_2.
[abstract▼] [BibTeX▼]

Script languages are powerful tools for scientists. Scientists use them to process data, invoke programs, and link program outputs/inputs. During the life cycle of scientific experiments, scientists compose scripts, execute them, and perform analysis on the results. Depending on the results, they modify their script to get more data to confirm the original hypothesis or to test a new hypothesis, evolving the experiment. While some tools capture provenance from the execution of scripts, most approaches focus on a single execution, leaving out the possibility to analyze the provenance evolution of the experiment as a whole. This work enables tracking and analyzing the provenance evolution gathered from scripts. Tracking the provenance evolution also helps to reconstruct the environment of previous executions for reproduction. Provenance evolution analysis allows comparison of executions to understand what has changed and supports the decision of which execution provides better results.
```
@inproceedings{pimentel_tracking_2016,
    author = "Pimentel, João Felipe and Freire, Juliana and Braganholo, Vanessa and Murta, Leonardo",
    language = "en",
    isbn = "978-3-319-40592-6 978-3-319-40593-3",
    title = "Tracking and {Analyzing} the {Evolution} of {Provenance} from {Scripts}",
    pages = "16--28",
    url = "https://link.springer.com/chapter/10.1007/978-3-319-40593-3\_2",
    year = "2016",
    doi = "10.1007/978-3-319-40593-3\_2",
    month = "June",
    publisher = "Springer, Cham",
    series = "Lecture {Notes} in {Computer} {Science}",
    booktitle = "Provenance and {Annotation} of {Data} and {Processes}"
}
```

Lukas Heinrich. Yadage. June 2016. URL: https://github.com/diana-hep/yadage.
[abstract▼] [BibTeX▼]

@misc{heinrich_yadage_2018,
    author = "Heinrich, Lukas",
    shorttitle = "yadage",
    url = "https://github.com/diana-hep/yadage",
    title = "yadage",
    publisher = "diana-hep",
    month = "June",
    year = "2016",
    copyright = "MIT"
}

João Felipe Pimentel, Juliana Freire, Leonardo Murta, and Vanessa Braganholo. Fine-Grained Provenance Collection over Scripts Through Program Slicing. In Marta Mattoso and Boris Glavic, editors, Provenance and Annotation of Data and Processes, volume 9672, pages 199–203. Springer International Publishing, Cham, 2016. URL: http://link.springer.com/10.1007/978-3-319-40593-3_21, doi:10.1007/978-3-319-40593-3_21.
[abstract▼] [BibTeX▼]

Collecting provenance from scripts is often useful for scientists to explain and reproduce their scientiﬁc experiments. However, most existing automatic approaches capture provenance at coarse-grain, for example, the trace of user-deﬁned functions. These approaches lack information of variable dependencies. Without this information, users may struggle to identify which functions really inﬂuenced the results, leading to the creation of false-positive provenance links. To address this problem, we propose an approach that uses dynamic program slicing for gathering provenance of Python scripts. By capturing dependencies among variables, it is possible to expose execution paths inside functions and, consequently, to create a provenance graph that accurately represents the function activations and the results they affect.
```
@incollection{mattoso_fine-grained_2016,
    author = "Pimentel, João Felipe and Freire, Juliana and Murta, Leonardo and Braganholo, Vanessa",
    editor = "Mattoso, Marta and Glavic, Boris",
    language = "en",
    isbn = "978-3-319-40592-6 978-3-319-40593-3",
    title = "Fine-{Grained} {Provenance} {Collection} over {Scripts} {Through} {Program} {Slicing}",
    pages = "199--203",
    address = "Cham",
    url = "http://link.springer.com/10.1007/978-3-319-40593-3\_21",
    year = "2016",
    doi = "10.1007/978-3-319-40593-3\_21",
    publisher = "Springer International Publishing",
    volume = "9672",
    booktitle = "Provenance and {Annotation} of {Data} and {Processes}"
}
```
Lukas Heinrich. Packtivity. June 2016. URL: https://github.com/diana-hep/packtivity.
[abstract▼] [BibTeX▼]

PROV-like activities with schema and bindings. This package aims to collect implementations of both synchronous and asynchronous execution of preserved, but parametrized scientific computational tasks that come with batteries included, i.e. with a full specification of their software dependencies. In that sense they are packaged activities -- packtivities.
```
@misc{heinrich_packtivity_2018,
    author = "Heinrich, Lukas",
    shorttitle = "packtivity",
    url = "https://github.com/diana-hep/packtivity",
    title = "packtivity",
    publisher = "diana-hep",
    month = "June",
    year = "2016"
}
```
Lukas Heinrich and Kyle Cranmer. WG 1: Simplified models / MC / RECASTing and reinterpretation for LLPs. Workshop of LHC LLP Community, April 2016. URL: https://indico.cern.ch/event/607314/.
[abstract▼] [BibTeX▼]

Following the success of the LHC Long-Lived Particle (LLP) Mini-Workshop in May of 2016, the LHC LLP Community -- composed of members of the CMS, LHCb, and ATLAS collaborations as well as theorists, phenomenologists and those interested in LLP searches with auxiliary LHC detectors -- convenes again to address the status and future of LLP searches at the LHC. This workshop will be one of two workshops devoted to producing an LHC LLP white paper that will be a snapshot of the status of LLP...
```
@unpublished{heinrich_wg_2016,
    author = "Heinrich, Lukas and Cranmer, Kyle",
    url = "https://indico.cern.ch/event/607314/",
    title = "{WG} 1: {Simplified} models / {MC} / {RECASTing} and reinterpretation for {LLPs}",
    month = "April",
    note = "Workshop of LHC LLP Community",
    year = "2016"
}
```

2015

Vicky Steeves, Erik Mitchell, and Jennifer Muilenburg. Organizational Implications of Data Science Environments in Education, Research, and Research Management in Libraries. CNI Fall Meeting, December 2015. URL: https://osf.io/rhgq3/.
[abstract▼] [BibTeX▼]

@unpublished{steeves_organizational_2015,
    author = "Steeves, Vicky and Mitchell, Erik and Muilenburg, Jennifer",
    language = "en",
    url = "https://osf.io/rhgq3/",
    title = "Organizational {Implications} of {Data} {Science} {Environments} in {Education}, {Research}, and {Research} {Management} in {Libraries}",
    month = "December",
    address = "Washington, D.C",
    note = "CNI Fall Meeting",
    year = "2015"
}

NYU Reproducibility Working Group. Reproducible Science @ NYU. August 2015. URL: https://nyu.reproduciblescience.org.
[abstract▼] [BibTeX▼]

@misc{group_reproducible_2017,
    author = "Group, NYU Reproducibility Working",
    language = "en",
    url = "https://nyu.reproduciblescience.org",
    title = "Reproducible {Science} @ {NYU}",
    month = "August",
    journal = "Reproducible Science @ NYU",
    year = "2015"
}

NYU Reproducibility Working Group. Reproducible Science. August 2015. URL: https://reproduciblescience.org/.
[abstract▼] [BibTeX▼]

On this site, the moderators have curated sources of various types discussing reproducibility in the directory. You can find academic papers, blog posts, popular media articles, talks, tools, and more.
```
@misc{group_reproducible_2018,
    author = "Group, NYU Reproducibility Working",
    language = "en",
    url = "https://reproduciblescience.org/",
    title = "Reproducible {Science}",
    month = "August",
    journal = "Reproducible Science",
    year = "2015"
}
```

Vicky Steeves and Rémi Rampin. Reproducibility-news. September 2015. URL: https://github.com/ViDA-NYU/reproducibility-news.
[abstract▼] [BibTeX▼]

@misc{steeves_reproducibility-news,
    author = "Steeves, Vicky and Rampin, Rémi",
    shorttitle = "reproducibility-news",
    url = "https://github.com/ViDA-NYU/reproducibility-news",
    title = "reproducibility-news",
    publisher = "ViDA-NYU",
    month = "September",
    year = "2015"
}

Rémi Rampin, Fernando Chirigati, and Juliana Freire. ReproZip. September 2015. URL: https://github.com/ViDA-NYU/reprozip.
[abstract▼] [BibTeX▼]

@misc{rampin_reprozip_2018,
    author = "Rampin, Rémi and Chirigati, Fernando and Freire, Juliana",
    shorttitle = "reprozip",
    url = "https://github.com/ViDA-NYU/reprozip",
    title = "{ReproZip}",
    publisher = "ViDA-NYU",
    month = "September",
    year = "2015",
    collaborator = "Steeves, Vicky",
    copyright = "BSD-3-Clause"
}

Timothy McPhillips, Tianhong Song, Tyler Kolisnik, Steve Aulenbach, Khalid Belhajjame, Kyle Bocinsky, Yang Cao, Fernando Chirigati, Saumen Dey, Juliana Freire, Deborah Huntzinger, Christopher Jones, David Koop, Paolo Missier, Mark Schildhauer, Christopher Schwalm, Yaxing Wei, James Cheney, Mark Bieda, and Bertram Ludaescher. YesWorkflow: A User-Oriented, Language-Independent Tool for Recovering Workflow Information from Scripts. arXiv:1502.02403 [cs], February 2015. arXiv: 1502.02403. URL: http://arxiv.org/abs/1502.02403.
[abstract▼] [BibTeX▼]

Scientific workflow management systems offer features for composing complex computational pipelines from modular building blocks, for executing the resulting automated workflows, and for recording the provenance of data products resulting from workflow runs. Despite the advantages such features provide, many automated workflows continue to be implemented and executed outside of scientific workflow systems due to the convenience and familiarity of scripting languages (such as Perl, Python, R, and MATLAB), and to the high productivity many scientists experience when using these languages. YesWorkflow is a set of software tools that aim to provide such users of scripting languages with many of the benefits of scientific workflow systems. YesWorkflow requires neither the use of a workflow engine nor the overhead of adapting code to run effectively in such a system. Instead, YesWorkflow enables scientists to annotate existing scripts with special comments that reveal the computational modules and dataflows otherwise implicit in these scripts. YesWorkflow tools extract and analyze these comments, represent the scripts in terms of entities based on the typical scientific workflow model, and provide graphical renderings of this workflow-like view of the scripts. Future versions of YesWorkflow also will allow the prospective provenance of the data products of these scripts to be queried in ways similar to those available to users of scientific workflow systems.
```
@article{mcphillips_yesworkflow:_2015,
    author = "McPhillips, Timothy and Song, Tianhong and Kolisnik, Tyler and Aulenbach, Steve and Belhajjame, Khalid and Bocinsky, Kyle and Cao, Yang and Chirigati, Fernando and Dey, Saumen and Freire, Juliana and Huntzinger, Deborah and Jones, Christopher and Koop, David and Missier, Paolo and Schildhauer, Mark and Schwalm, Christopher and Wei, Yaxing and Cheney, James and Bieda, Mark and Ludaescher, Bertram",
    shorttitle = "{YesWorkflow}",
    url = "http://arxiv.org/abs/1502.02403",
    title = "{YesWorkflow}: {A} {User}-{Oriented}, {Language}-{Independent} {Tool} for {Recovering} {Workflow} {Information} from {Scripts}",
    month = "February",
    journal = "arXiv:1502.02403 [cs]",
    note = "arXiv: 1502.02403",
    year = "2015"
}
```

Juliana Freire. Reproducibility Made Easy. In Proceedings of EuroRV3. 2015.
[BibTeX▼]

@inproceedings{freire_reproducibility_2015,
    author = "Freire, Juliana",
    language = "en",
    title = "Reproducibility {Made} {Easy}",
    booktitle = "Proceedings of {EuroRV}3",
    year = "2015"
}

Meredith Broussard. The Irony of Writing About Digital Preservation. The Atlantic, November 2015. URL: https://www.theatlantic.com/technology/archive/2015/11/the-irony-of-writing-about-digital-preservation/416184/.
[abstract▼] [BibTeX▼]

Last month, The Atlantic published a lengthy article about information that is lost on the web. That story itself is in jeopardy.
```
@article{broussard_irony_2015,
    author = "Broussard, Meredith",
    language = "en",
    url = "https://www.theatlantic.com/technology/archive/2015/11/the-irony-of-writing-about-digital-preservation/416184/",
    title = "The {Irony} of {Writing} {About} {Digital} {Preservation}",
    month = "November",
    journal = "The Atlantic",
    year = "2015"
}
```
Meredith Broussard. Preserving news apps present huge challenges. Newspaper Research Journal, 36(3):299–313, September 2015. URL: http://journals.sagepub.com/doi/10.1177/0739532915600742, doi:10.1177/0739532915600742.
[abstract▼] [BibTeX▼]

Currently the digital archives of newspapers are not archiving news apps, the interactive database-driven, multimedia projects. Because of the multiple elements required to access a news app, conversion of the dynamic news app into static HTML pages is one possible avenue for future archiving.
```
@article{broussard_preserving_2015,
    author = "Broussard, Meredith",
    language = "en",
    url = "http://journals.sagepub.com/doi/10.1177/0739532915600742",
    title = "Preserving news apps present huge challenges",
    pages = "299--313",
    year = "2015",
    doi = "10.1177/0739532915600742",
    month = "September",
    journal = "Newspaper Research Journal",
    volume = "36",
    issn = "0739-5329, 2376-4791",
    number = "3"
}
```
Juliana Freire. Managing and Reusing Provenance as a Critical Capability for Data Scientists. In American Association for the Advancement of Science. February 2015.
[abstract▼] [BibTeX▼]

Provenance provides documentation that is key to preserve data, to determine the datas quality and authorship, to understand, reproduce, as well as validate results. In this talk, I will review some of the state-of-the-art techniques, and discuss benefits of provenance that go beyond reproducibility, including the support for collaborative data exploration and visualization. I will also show how provenance can aid in teaching and enable reproducible publications.
```
@inproceedings{freire_managing_2015,
    author = "Freire, Juliana",
    language = "en",
    title = "Managing and {Reusing} {Provenance} as a {Critical} {Capability} for {Data} {Scientists}",
    month = "February",
    year = "2015",
    booktitle = "American Association for the Advancement of Science"
}
```
Meredith Broussard. Big Data in Practice: Enabling computational journalism through code-sharing and reproducible research methods. Digital Journalism, 2015. URL: https://www.tandfonline.com/doi/full/10.1080/21670811.2015.1074863, doi:10.1080/21670811.2015.1074863.
[abstract▼] [BibTeX▼]

Applied research in computational journalism offers opportunities for scholars to engage with data and platforms by developing original code-based scholarship and research methods. However, scholars who write code need a place to publish about it. This paper suggests new article forms and academic publishing conventions that will allow the field of computational journalism to expand. It explores logistical details associated with sharing and publishing code, and describes reproducible research methods in use in other disciplines.
```
@article{broussard_big_nodate,
    author = "Broussard, Meredith",
    language = "en",
    url = "https://www.tandfonline.com/doi/full/10.1080/21670811.2015.1074863",
    title = "Big {Data} in {Practice}: {Enabling} computational journalism through code-sharing and reproducible research methods",
    journal = "Digital Journalism",
    volume = "4",
    issn = "2167-082X",
    year = "2015",
    doi = "10.1080/21670811.2015.1074863",
    number = "2"
}
```

2014

Leonardo Murta, Vanessa Braganholo, Fernando Chirigati, David Koop, and Juliana Freire. noWorkflow: Capturing and Analyzing Provenance of Scripts. In Provenance and Annotation of Data and Processes, Lecture Notes in Computer Science, 71–83. Springer, Cham, June 2014. URL: https://link.springer.com/chapter/10.1007/978-3-319-16462-5_6, doi:10.1007/978-3-319-16462-5_6.
[abstract▼] [BibTeX▼]

We propose noWorkflow, a tool that transparently captures provenance of scripts and enables reproducibility. Unlike existing approaches, noWorkflow is non-intrusive and does not require users to change the way they work – users need not wrap their experiments in scientific workflow systems, install version control systems, or instrument their scripts. The tool leverages Software Engineering techniques, such as abstract syntax tree analysis, reflection, and profiling, to collect different types of provenance, including detailed information about the underlying libraries. We describe how noWorkflow captures multiple kinds of provenance and the different classes of analyses it supports: graph-based visualization; differencing over provenance trails; and inference queries.
```
@inproceedings{murta_noworkflow:_2014,
    author = "Murta, Leonardo and Braganholo, Vanessa and Chirigati, Fernando and Koop, David and Freire, Juliana",
    shorttitle = "noWorkflow",
    isbn = "978-3-319-16461-8",
    title = "{noWorkflow}: {Capturing} and {Analyzing} {Provenance} of {Scripts}",
    pages = "71--83",
    url = "https://link.springer.com/chapter/10.1007/978-3-319-16462-5\_6",
    year = "2014",
    doi = "10.1007/978-3-319-16462-5\_6",
    language = "en",
    month = "June",
    publisher = "Springer, Cham",
    series = "Lecture {Notes} in {Computer} {Science}",
    booktitle = "{Provenance} and {Annotation} of {Data} and {Processes}"
}
```
Juliana Freire, D. Koop, F. Chirigati, and Claudio Silva. Reproducibility using VisTrails. In Implementing Reproducible Research. Chapman and Hall/CRC, 2014. URL: https://nyuscholars.nyu.edu/en/publications/reproducibility-using-vistrails.
[abstract▼] [BibTeX▼]

In computational science, reproducibility requires that researchers make code and data available to others so that the data can be analyzed in a similar manner as in the original publication. Code must be available to be distributed, data must be accessible in a readable format, and a platform must be available for widely distributing the data and code. In addition, both data and code need to be licensed permissively enough so that others can reproduce the work without a substantial legal burden.
```
@incollection{freire_vt,
    author = "Freire, Juliana and Koop, D. and Chirigati, F. and Silva, Claudio",
    language = "English (US)",
    isbn = "978-1-4665-6159-5",
    title = "Reproducibility using {VisTrails}",
    publisher = "Chapman and Hall/CRC",
    url = "https://nyuscholars.nyu.edu/en/publications/reproducibility-using-vistrails",
    year = "2014",
    booktitle = "Implementing {Reproducible} {Research}"
}
```

2013

Tommy Ellqvist, Fernando Chirigati, Juliana Freire, and Dennis Shasha. ReproMatch. March 2013. URL: http://repromatch.poly.edu/tools/search/.
[abstract▼] [BibTeX▼]

Many tools have been developed to support researchers in the practice of reproducible research. However, given the many requirements across disciplines, identifying the right tool or a set of tools can be a daunting task. With the help of developers of reproducibility tools, we will create a comprehensive catalog of tools. Using ReproMatch, users will be able to search for tools that match their reproducibility needs.
```
@misc{rampin_repromatch_nodate,
    author = "Ellqvist, Tommy and Chirigati, Fernando and Freire, Juliana and Shasha, Dennis",
    url = "http://repromatch.poly.edu/tools/search/",
    title = "{ReproMatch}",
    month = "March",
    journal = "ReproMatch",
    year = "2013"
}
```

João Felipe Pimentel, Juliana Freire, Leonardo Murta, and Vanessa Braganholo. noWorkflow. September 2013. URL: https://github.com/gems-uff/noworkflow.
[abstract▼] [BibTeX▼]

@misc{pimentel_noworkflow_2018,
    author = "Pimentel, João Felipe and Freire, Juliana and Murta, Leonardo and Braganholo, Vanessa",
    shorttitle = "noworkflow",
    url = "https://github.com/gems-uff/noworkflow",
    title = "{noWorkflow}",
    publisher = "Grupo de Evolução e Manutenção de Software (GEMS)",
    month = "September",
    year = "2013",
    collaborator = "Koop, David and Chirigati, Fernando and Missier, Paolo",
    copyright = "MIT"
}