Skip to content

Resubmission: Save output dataframe as HDF5, not Pickle

Aaron Graubert requested to merge github/fork/julianhess/hdf5 into master

Created by: julianhess

I was having issues with reading Pickles from a buffer — as updated in ee641937, in order to accommodate multiple backends, the output Pickle is now read via a transport, rather than with a simple pd.read_pickle("filename") call.

HDF5 is more amenable to being read/written via buffers, so I've switched over. Also, HDF5 ought to be a more stable serialization format — people have reported that the Pickle spec often introduces breaking changes.

Merge request reports