[Israel.pm] Handling huge data-structures?

Gaal Yahas gaal at forum2.org
Sun Aug 29 08:46:41 PDT 2004


On Sun, Aug 29, 2004 at 06:30:02PM +0300, Offer Kaye wrote:
> > I have a <1GB file that I want to keep in a hash, and work with that hash
> > like a normal hash, just keep it out of my memory (on the HDD, probably).
> > 
> > What solutions exist (except for buying more RAM) and what are their
> > limitations?
> 
> Array based rather than hash based solution, but good for large data
> sets and works pretty much as you described above:
> Tie::File

If by "works" you mean only "has the same semantics as", then yes; but
anything that's going to insert or delete -- or indeed, change the length
of an existing record -- is going to be very expensive.

Yuval, if I were you I'd do some research on DB_File and maybe SQLite
via Class::DBI. Or just chuck the semantic equivalence requirement and
bite the SQL bullet: sufficiently different problems tolerate different
solutions.

-- 
Gaal Yahas <gaal at forum2.org>
http://gaal.livejournal.com/



More information about the Perl mailing list