[Israel.pm] Scraping data via Perl from ASP websites

Yuval Kogman nothingmuch at woobling.org
Thu Aug 28 12:42:40 PDT 2008


On Thu, Aug 28, 2008 at 04:08:17 -0700, Yossi Klein wrote:
> I have had much success scraping data from Perl websites using the
> HTTP modules (HTTP:Request, HTTP::Response, etc.). However, I now
> have a need to use Perl to scrape data off of ASP sites and am not
> having much success.

Did you try WWW::Mechanize? Web::Scraper? They are higher level
wrappers over LWP, maybe they can take care of whatever weird
redirection is happenning.

Also, I can't believe i'm saying this, but Evan Carroll's module
sounds related:

	http://search.cpan.org/~ecarroll/HTML-TreeBuilderX-ASP_NET-0.07/

He's sometimes difficult to deal with so take this with a grain of
salt.

Apparently some ASP.NET pages rely on javascript to function
correctly.

-- 
  Yuval Kogman <nothingmuch at woobling.org>
http://nothingmuch.woobling.org  0xEBD27418




More information about the Perl mailing list