How can I append to an existing bundle?

#1

Hi,

I have reasonably deeply researched how could I append new update data to an existing zipline bundle (i.e. make an initial ingest, and on a daily basis append new data), and have come to the conclussion that bcolz would need to be reconsidered as a database framework for storing bundle data. The reasong that drives to obtain that conclusion is that the attrs file inside the daily_equities.bcolz folder holds row numbers for the starting position for the different securities, and apparently you may only use a single starting position, hence enabling only 1 time data ingestion. So basically I believe all the persistence approach would need to be reformulated.

Is there any ongoing effort or zipline fork trying to attain that. I find zipline a marvelous tool (also true that haven’t researched others). But this is an important drawback. If handling a relative small universe, it’s fine, but if managing universe sizes in the order of thousands of securities, this work procedure becomes unfeasible.

Any hints or comments are greatly appreciated.

Regards,

Carlos.

#2

Hi,
Sqlite and bcolz have their intrinsic pros and cons.
Zipline is built so that every “ingestion” creates a new sqlite “database”.
I can only assume that, maybe one of the reasons the original developers did that, is to more easily find/avoid data revisions induced inconsistencies.
Using a more robust database like postgresql wouldn’t be too hard, but it would still require some time effort and I am not yet sure if there is a sufficiently large community that would be able to work with a more demanding database.
Cheers,
MBounouar