[Yum-devel] Is providing sql text files instead of binary sqlite db files feasible?

[Hedayat Vatankhah]

On ۰۹/۱۱/۰۸  05:43, Seth Vidal wrote:
>
>
> On Sat, 7 Nov 2009, Hedayat Vatankhah wrote:
>
>> Hi,
>> Today I was testing different configurations for yum metadata files 
>> (using different compression formats such as xz, with and without 
>> requires table); I noticed something interesting:
>> Dumping Fedora 11's primary.sqlite file (using .dump command of 
>> sqlite3) in a text file and compressing that with bzip2 results in a 
>> 4.4MB (4.1MB using xz) file, comparing with compressing that db 
>> directly using bzip2 which results in a 11MB file (8.7MB using xz).
>> For Fedora 11 Updates primary database using bzip2, the size reduces 
>> from 5.6MB to 2.1MB.
>>
>> These savings are higher than the values I get when I drop requires 
>> table from the databases and create new databases from them. Also, 
>> they need the least changes in the code.
>>
>> Just thought that it might be interesting for you!
>
> Now - how long does it take to go from the .dump file back into a 
> sqlite db you can use? I suspect that populating the filelists in 
> particular is pretty expensive in terms of time.
On my system (1.2G Core 2 Due Centerino), it take about 10 seconds to 
read Fedora 11 repository's filelist data. I don't know how much time is 
expected as "expensive", it depends on one's bandwidth. BTW, considering 
that such metadata is not downloaded often, this time might be reasonable.

But if spending such time is considered unacceptable, yum might provide 
two operating modes: low bandwidth and high bandwidth modes. In low 
bandwidth mode, it'll download the compressed dump file and import it to 
the sqlite database. In high bandwidth mode it'll download compressed 
sqlite db files. The mode could be configured in yum config files, or it 
could even determine the appropriate mode of operation considering its 
download speed.

Using this approach, delta metadata updates can be generated as a set of 
SQL insert/delete/update commands.

How is this?

Thanks,
Hedayat
>
> -sv
>
> _______________________________________________
> Yum-devel mailing list
> Yum-devel at lists.baseurl.org
> http://lists.baseurl.org/mailman/listinfo/yum-devel