[Israel.pm] Handling huge data-structures?

[Itamar Elem]

hi
you can try use BerkeleyDB module
its the interface for the new version of use BerkeleyDB (3.* and above)
and its have hash interface

         use BerkeleyDB;

         $db  = tie %hash, 'BerkeleyDB::Hash', [OPTIONS] ;
         $db  = new BerkeleyDB::Hash [OPTIONS] ;

         $db  = tie %hash, 'BerkeleyDB::Btree', [OPTIONS] ;
         $db  = new BerkeleyDB::Btree [OPTIONS] ;


and more

itamar

-----Original Message-----
From: Yuval Yaari [mailto:yuval at windax.com]
Sent: Sunday, August 29, 2004 11:13 PM
To: Perl in Israel
Subject: Re: [Israel.pm] Handling huge data-structures?


[SNIPPED]
> SQLite uses a single file to store a single database. The database
> *code* is embedded in your application (DBD::SQLite in this case), not
> the actual data.

My mistake...

> I suggested CDBI for a familiar (and scalable) interface, not for speed.
> I expect (but don't take my word for it) that straight SQLite is going
> to be one of your fastest options that don't require you to implement
> the store yourself.

I love CDBI, but it would give me performance problems, which I am trying to 
solve.
Either I've been misusing CDBI or it's really slow.

> The advantage with DB_File is that it uses perl's tie interface. I have
> no idea if it works for large files though and indeed how it performs.
> Whatever the bioinformatics guys were talking about, it probably used
> tie for the magic.

As Offer wrote, it doesn't work with huge files, as it loads them to memory.

> Note that each solution here will still have different performance
> characteristics than a natural Perl hash, even if it looks like one.
> Some operations may be more expensive than you think. Also, tuning a
> large database in the low levels can dramatically impact performance,
> if you know how. But you will need to research this.

Is there any solution in the form of a hash?
I should look at bioinformatics code :)
I would also see if SQLite/CDBI solve my problem.

  --Yuval 


_______________________________________________
Perl mailing list
Perl at perl.org.il
http://perl.org.il/mailman/listinfo/perl