ProIO: An event-based I/O stream format for protobuf messages

Published: 25 April 2019| Version 1 | DOI: 10.17632/mfxsg2d5x5.1
Contributors:
D. Blyth,
J. Alcaraz,
S. Binet,
S. V. Chekanov

Description

ProIO is a new event-oriented streaming data format which utilizes Google’s Protocol Buffers (protobuf) to be flexible and highly language-neutral. The ProIO concept is described here along with its software implementations. The performance of the ProIO concept for a dataset with Monte-Carlo event records used in high-energy physics was benchmarked and compared/contrasted with ROOT I/O. Various combinations of general-purpose compression and variable-length integer encoding available in protobuf were used to investigate the relationship between I/O performance and size-on-disk in a few key scenarios.

Files

Categories

Computational Physics

License