Pack'em ESMs! Tools for (un-)archiving Earth system model data

by Daniel Heydebreck (DKRZ), Karl-Hermann Wieners

Tuesday 27 Oct 2020, 15:15 → 16:15 Europe/Berlin

Recording on Youtube (external site)

Archiving large amounts of data files via pftp to the HPSS (tape system) is time consuming and takes some manual work. A new set of command line tools called "packems" was released for all DKRZ users with this Tech Talk. These tools simplify the archival and retrieval of data files to/from the HPSS – including the creation of tar balls and saving information on where which files are stored.

Archiving large amounts of data files via pftp to the HPSS (tape system) is time consuming and takes some manual work: (a) users have to aggregate small files in tar balls of 10 GB – 100 GB size; (b) operations with pftp are not handy to automatize. Additionally, the retrieval of small individual files, which have been archived via tar balls in the HPSS, is problematic if users do not remember which files were put into which tar ball. Many users might have experienced these issues.


That's why a new set of command line tools called "packems" was made available for all DKRZ users. These tools will simplify the archival and retrieval of data files to/from the HPSS – including the creation of tar balls and saving information on where which files are stored. The "packems" tool set is a joint development of the MPI-M and the DKRZ.

In this TechTalk, Karl-Hermann Wieners (MPI-M) and Daniel Heydebreck (DKRZ) present this new tool which is now available on DKRZ's mistral system directly via a module.