Automated Discovery of Container Executables
References
- Ratliff J. Docker: Accelerated, Containerized Application Development.
https://www.docker.com/ , 2022. Accessed: 2022-10-25. - Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. May 2017; 12:
e0177459 . DOI: 10.1371/journal.pone.0177459 - Gamblin T, LeGendre M, Collette MR, Lee GL, Moody A, De Supinski BR, Futral S. The Spack package manager: bringing order to HPC software chaos. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 2015; 1–12. DOI: 10.1145/2807591.2807623
- Hoste K, Timmerman J, Georges A, De Weirdt S.
Easybuild: Building software with ease . In: 2012 SC Companion: High Performance Computing, Networking Storage and Analysis. IEEE. 2012; 572–582. DOI: 10.1109/SC.Companion.2012.81 - Furlani JL.
Modules: Providing a flexible user environment . In: Proceedings of the fifth large installation systems administration conference (LISA V). 1991; 141–152. - Lmod: A New Environment Module System — Lmod 8.7.13 documentation. 2022.
https://lmod.readthedocs.io/en/latest/ . Accessed: 2022-10-25. - Sochat V, Scott A. Collaborative Container Modules with Singularity Registry HPC. Journal of Open Source Software. 2021; 6(63): 3311. DOI: 10.21105/joss.03311
- YAML Specification Index. 2022.
https://yaml.org/spec/ . Accessed: 2022-10-25. - Savanth V. Docker images: Name vs. tag vs. digest. Sept. 2021.
https://hackernoon.com/docker-images-name-vs-tag-vs-digest . Accessed: 2022-11-4. - shpc-registry: Testing a remote registry for Singularity Registry HPC. 2022.
https://github.com/singularityhub/shpc-registry . - da Veiga Leprevost F, Grüning BA, Alves Aflitos S, Röst HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Vera Alvarez R, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. Aug. 2017; 33: 2580–2582. DOI: 10.1093/bioinformatics/btx192
- pipelib: a library for creating pipelines for parsing, filtering, and sorting iterables. 2022.
https://github.com/vsoch/pipelib . - guts: Actions and client to derive container guts! 2022.
https://github.com/singularityhub/guts . - shpc-guts: Singularity Registry HPC… container guts! 2022.
https://github.com/singularityhub/shpc-guts . - shpc-registry-cache: A cache of commands (currently for biocontainers). 2022.
https://github.com/singularityhub/shpc-registry-cache . - opencontainers/image-spec manifest. 2022.
https://github.com/opencontainers/image-spec/blob/main/manifest.md . - Wikipedia contributors. diff. Oct. 2022.
https://en.?wikipedia.org/w/index.php?title=Diff&oldid=1118832554 . Accessed: 2022-12-8. - Docker hub HTTP API V2. Nov. 2020.
https://docs.docker.com/registry/spec/api/ . Accessed: 2020-11-23. - container-executable-discovery action: GitHub action to assist in creating a cache of container executables. 2022.
https://github.com/singularityhub/container-executable-discovery/ . Accessed: 2022-11-29. - Afgan E, Baker D, Batut B, Van Den Beek M, Bouvier D, Čech M, Chilton J, Clements D, Coraor N, Grüning BA, et al. The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2018 update. Nucleic acids research. 2018; 46(W1): W537–W544. DOI: 10.1093/nar/gky379
- shpc-registry workflow to update Biocontainers. 2022.
https://github.com/singularityhub/shpc-registry/blob/main/.github/workflows/update-biocontainers.yaml . Accessed: 2022-11-29. - looking-at-counts.ipynb at shpc-registrycache repository. 2022.
https://github.com/singularityhub/shpc-registry-cache/blob/main/looking-at-counts.ipynb . Accessed: 2023-03-16. - Usage limits, billing, and administration. 2022.
https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration . - hashlib — secure hashes and message digests — python 3.11.0 documentation. 2022.
https://docs.python.org/3/library/hashlib.html . Accessed: 2022-11-4. - split-list-action: Simple GitHub action to evenly split (and present a subset) of items based on random selection or the day of the month! 2022.
https://github.com/vsoch/split-list-action . Accessed: 2022-11-8. - Pawsey Supercomputing Research Centre. 2022.
https://pawsey.org.au . Accessed: 2022-11-8. - Australian BioCommons. 2022.
https://biocommons.org.au . Accessed: 2022-11-8. - CernVM File System. 2022.
https://cernvm.cern.ch/fs/ . Accessed: 2022-11-8. - biocontainer-match.py at main shpc repository. 2022.
https://github.com/singularityhub/singularity-hpc/blob/main/example/biocontainer-match.py . Accessed: 2022-11-16. - A proof of concept of RNA-seq pipeline, written in Nextflow. 2022.
https://github.com/nextflow-io/rnaseq-nf . - Demo/toy RNAseq pipeline, to test interplay of Nextflow with SHPC container modules. 2022.
https://github.com/marcodelapierre/demo-shpc-nf . - Walsh D. The Pod Manager tool (podman). 2022.
https://podman.io/ . Accessed: 2022-11-26. - Singularity registry hpc – documentation. 2022.
https://singularity-hpc.readthedocs.io . Accessed: 2022-12-8. - Singularity registry hpc – developer tutorial. 2022.
https://singularity-hpc.readthedocs.io/en/latest/getting_started/developer-guide.html#developer-tutorial . Accessed: 2022-12-14. - container-executable-discovery cache creation action: GitHub action to assist in creating a cache of container executables. 2022.
https://github.com/singularityhub/shpc-registry-cache/blob/main/.github/workflows/update-cache.yaml . Accessed: 2022-11-29. - Conda: Package, dependency and environment management for any language. 2022.
https://docs.conda.io/ . - Barker M, Chue Hong NP, Katz DS, Lamprecht A-L, Martinez-Ortiz C, Psomopoulos F, Harrow J, Castro LJ, Gruenpeter M, Martinez PA, Honeyman T. Introducing the FAIR Principles for research software. Scientific Data. Oct. 2022; 9: 622. DOI: 10.1038/s41597-022-01710-x
- Bray S, Chilton J, Bernt M, Soranzo N, van den Beek M, Batut B, Rasche H, Čech M, Cock PJA, Grüning B, Nekrutenko A. The Planemo toolkit for developing, deploying, and executing scientific data analyses in Galaxy and beyond. Genome Research. Jan. 2023; 33: 261–268. DOI: 10.1101/gr.276963.122
DOI: https://doi.org/10.5334/jors.451 | Journal eISSN: 2049-9647
Language: English
Submitted on: Dec 16, 2022
Accepted on: Apr 12, 2023
Published on: Apr 24, 2023
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year
Keywords:
© 2023 Vanessa Sochat, Matthieu Muffato, Audrey Stott, Marco De La Pierre, Georgia Stuart, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.