[Israel.pm] Handling huge data-structures?
yuval at windax.com
Sun Aug 29 13:12:37 PDT 2004
> SQLite uses a single file to store a single database. The database
> *code* is embedded in your application (DBD::SQLite in this case), not
> the actual data.
> I suggested CDBI for a familiar (and scalable) interface, not for speed.
> I expect (but don't take my word for it) that straight SQLite is going
> to be one of your fastest options that don't require you to implement
> the store yourself.
I love CDBI, but it would give me performance problems, which I am trying to
Either I've been misusing CDBI or it's really slow.
> The advantage with DB_File is that it uses perl's tie interface. I have
> no idea if it works for large files though and indeed how it performs.
> Whatever the bioinformatics guys were talking about, it probably used
> tie for the magic.
As Offer wrote, it doesn't work with huge files, as it loads them to memory.
> Note that each solution here will still have different performance
> characteristics than a natural Perl hash, even if it looks like one.
> Some operations may be more expensive than you think. Also, tuning a
> large database in the low levels can dramatically impact performance,
> if you know how. But you will need to research this.
Is there any solution in the form of a hash?
I should look at bioinformatics code :)
I would also see if SQLite/CDBI solve my problem.
More information about the Perl