Skip to main content

Taskfarm: A Client/Server Framework for Supporting Massive Embarrassingly Parallel Workloads Cover

Taskfarm: A Client/Server Framework for Supporting Massive Embarrassingly Parallel Workloads

Journal of Open Research Software

Volume 11 (2023): Issue 1

By: Magnus Hagdorn and Noel Gourmelen

Open Access

|Jan 2023

Abstract

Taskfarm is a client/server framework that can be used to keep track of massive embarrassingly parallel workloads. The system is split up into two packages: (1) a flask server that hands out new tasks via HTTP and (2) a python client that requests and updates tasks. The server stores task progress in a database. This system has been designed to manage a satellite data processing workflow with hundreds of thousands of tasks with variable compute costs. It can be used for any problem that can be solved using a task farm.

References

Aldinucci M, Danelutto M. Stream parallel skeleton optimization. In Proceedings of the 11th IASTED International Conference on Parallel and Distributed Computing and Systems, MIT. 1999; 955–962. IASTED/ACTA press.
Search in Google Scholar Back to article
Amundsen M, Ruby S, Richardson L. RESTful Web APIs. O’Reilly; 2013.
Search in Google Scholar Back to article
Dowd K, Severance C. High Performance Computing. O’Reilly, 2nd edition; 1998.
Search in Google Scholar Back to article
Foresta L, Gourmelen N, Pálsson F, Nienow P, Björnsson H, Shepherd A. Surface elevation change and mass balance of icelandic ice caps derived from swath mode cryosat-2 altimetry. Geophysical Research Letters. 2016; 43. ISSN 19448007. DOI: 10.1002/2016GL071485
Open DOI Search in Google Scholar Back to article
Gourmelen N, Escorihuela MJ, Shepherd A, Foresta L, Muir A, Garcia-Mondéjar A, Roca M, Baker SG, Drinkwater MR. Cryosat-2 swath interferometric altimetry for mapping ice elevation and elevation change. Advances in Space Research. 2018; 62. ISSN 18791948. DOI: 10.1016/j.asr.2017.11.014
Open DOI Search in Google Scholar Back to article
Jakob L, Gourmelen N, Ewart M, Plummer S. Spatially and temporally resolved ice loss in high mountain asia and the gulf of alaska observed by cryosat-2 swath altimetry between 2010 and 2019. Cryosphere. 2021; 15. ISSN 19940424. DOI: 10.5194/tc-15-1845-2021
Open DOI Search in Google Scholar Back to article
Shepherd A, Ivins E, Rignot E, Smith B, Broeke MVD, Velicogna I, Whitehouse P, Briggs K, Joughin I, Krinner G, Nowicki S, Payne T, Scambos T, Schlegel N, Geruo A, Agosta C, Ahlstrøm A, Babonis G, Barletta V, Blazquez A, Bonin J, Csatho B, Cullather R, Felikson D, Fettweis X, Forsberg R, Gallee H, Gardner A, Gilbert L, Groh A, Gunter B, Hanna E, Harig C, Helm V, Horvath A, Horwath M, Khan S, Kjeldsen KK, Konrad H, Langen P, Lecavalier B, Loomis B, Luthcke S, McMillan M, Melini D, Mernild S, Mohajerani Y, Moore P, Mouginot J, Moyano G, Muir A, Nagler T, Nield G, Nilsson J, Noel B, Otosaka I, Pattle ME, Peltier WR, Pie N, Rietbroek R, Rott H, Sandberg-Sørensen L, Sasgen I, Save H, Scheuchl B, Schrama E, Schröder L, Seo KW, Simonsen S, Slaters T, Spada G, Sutterley T, Talpe M, Tarasov L, Berg WJVD, Wal WVD, Wessem MV, Vishwakarma BD, Wiese D, Wouters B. Mass balance of the antarctic ice sheet from 1992 to 2017. Nature. 2018; 558. ISSN 14764687. DOI: 10.1038/s41586-018-0179-y
Open DOI Search in Google Scholar Back to article
Shepherd A, Ivins E, Rignot E, Smith B, van den Broeke M, Velicogna I, Whitehouse P, Briggs K, Joughin I, Krinner G, Nowicki S, Payne T, Scambos T, Schlegel N, Geruo A, Agosta C, Ahlstrøm A, Babonis G, Barletta VR, Bjørk AA, Blazquez A, Bonin J, Colgan W, Csatho B, Cullather R, Engdahl ME, Felikson D, Fettweis X, Forsberg R, Hogg AE, Gallee H, Gardner A, Gilbert L, Gourmelen N, Groh A, Gunter B, Hanna E, Harig C, Helm V, Horvath A, Horwath M, Khan S, Kjeldsen KK, Konrad H, Langen PL, Lecavalier B, Loomis B, Luthcke S, McMillan M, Melini D, Mernild S, Mohajerani Y, Moore P, Mottram R, Mouginot J, Moyano G, Muir A, Nagler T, Nield G, Nilsson J, Noël B, Otosaka I, Pattle ME, Peltier WR, Pie N, Rietbroek R, Rott H, Sandberg Sørensen L, Sasgen I, Save H, Scheuchl B, Schrama E, Schröder L, Seo KW, Simonsen SB, Slater T, Spada G, Sutterley T, Talpe M, Tarasov L, van de Berg WJ, van der Wal W, van Wessem M, Vishwakarma BD, Wiese D, Wilton D, Wagner T, Wouters B, Wuite J. Mass balance of the greenland ice sheet from 1992 to 2018. Nature. 2020; 579. ISSN 14764687. DOI: 10.1038/s41586-019-1855-2
Open DOI Search in Google Scholar Back to article
Wingham DJ, Francis CR, Baker S, Bouzinac C, Brockley D, Cullen R, de Chateau-Thierry P, Laxon SW, Mallow U, Mavrocordatos C, Phalippou L, Ratier G, Rey L, Rostan F, Viau P, Wallis DW. Cryosat: A mission to determine the fluctuations in earth’s land and marine ice fields. Advances in Space Research. 2006; 37. ISSN 02731177. DOI: 10.1016/j.asr.2005.07.027
Open DOI Search in Google Scholar Back to article

Figures & tables

Articles in this issue

DOI: https://doi.org/10.5334/jors.393 | Journal eISSN: 2049-9647

Journal RSS Feed

Language: English

Submitted on: Sep 6, 2021

|

Accepted on: Nov 29, 2022

|

Published on: Jan 12, 2023

Published by: Ubiquity Press

In partnership with: Paradigm Publishing Services

Publication frequency: 1 issue per year

Keywords:

high performance computing,

client/server framework,

embarrassingly parallel workload

© 2023 Magnus Hagdorn, Noel Gourmelen, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 11 (2023): Issue 1

Previous article