I have reasonably deeply researched how could I append new update data to an existing zipline bundle (i.e. make an initial ingest, and on a daily basis append new data), and have come to the conclussion that bcolz would need to be reconsidered as a database framework for storing bundle data. The reasong that drives to obtain that conclusion is that the attrs file inside the daily_equities.bcolz folder holds row numbers for the starting position for the different securities, and apparently you may only use a single starting position, hence enabling only 1 time data ingestion. So basically I believe all the persistence approach would need to be reformulated.
Is there any ongoing effort or zipline fork trying to attain that. I find zipline a marvelous tool (also true that haven’t researched others). But this is an important drawback. If handling a relative small universe, it’s fine, but if managing universe sizes in the order of thousands of securities, this work procedure becomes unfeasible.
Any hints or comments are greatly appreciated.