[Israel.pm] Handling huge data-structures?
elem at compugen.co.il
Mon Aug 30 05:15:42 PDT 2004
you can try use BerkeleyDB module
its the interface for the new version of use BerkeleyDB (3.* and above)
and its have hash interface
$db = tie %hash, 'BerkeleyDB::Hash', [OPTIONS] ;
$db = new BerkeleyDB::Hash [OPTIONS] ;
$db = tie %hash, 'BerkeleyDB::Btree', [OPTIONS] ;
$db = new BerkeleyDB::Btree [OPTIONS] ;
From: Yuval Yaari [mailto:yuval at windax.com]
Sent: Sunday, August 29, 2004 11:13 PM
To: Perl in Israel
Subject: Re: [Israel.pm] Handling huge data-structures?
> SQLite uses a single file to store a single database. The database
> *code* is embedded in your application (DBD::SQLite in this case), not
> the actual data.
> I suggested CDBI for a familiar (and scalable) interface, not for speed.
> I expect (but don't take my word for it) that straight SQLite is going
> to be one of your fastest options that don't require you to implement
> the store yourself.
I love CDBI, but it would give me performance problems, which I am trying to
Either I've been misusing CDBI or it's really slow.
> The advantage with DB_File is that it uses perl's tie interface. I have
> no idea if it works for large files though and indeed how it performs.
> Whatever the bioinformatics guys were talking about, it probably used
> tie for the magic.
As Offer wrote, it doesn't work with huge files, as it loads them to memory.
> Note that each solution here will still have different performance
> characteristics than a natural Perl hash, even if it looks like one.
> Some operations may be more expensive than you think. Also, tuning a
> large database in the low levels can dramatically impact performance,
> if you know how. But you will need to research this.
Is there any solution in the form of a hash?
I should look at bioinformatics code :)
I would also see if SQLite/CDBI solve my problem.
Perl mailing list
Perl at perl.org.il
More information about the Perl