[Israel.pm] Scraping data via Perl from ASP websites

Eli Billauer eli at billauer.co.il
Thu Aug 28 08:24:54 PDT 2008


Hello,


It looks like you need to write a crawler.


Perl has excellent modules for that. The internet is full with examples, 
but if you want some homegrown stuff, I suggest you had a look on the 
examples from a lightning talk session, which was held in Haifux.


By the way, it sounds like the sites you're accessing are expecting 
proper cookies handling. But that's just a wild guess.


http://www.haifux.org/lectures/183/


Have fun (it is!)

    Eli


Yossi Klein wrote:

> I have had much success scraping data from Perl websites using the HTTP modules (HTTP:Request, HTTP::Response, etc.). However, I now have a need to use Perl to scrape data off of ASP sites and am not having much success.
>
> Without going into too much detail, I've always used the Live HTTPHeaders add-on to Firefox to see the HTTP requests and responses and I use that information to help me emulate the same in a Perl program. This method doesn't work for ASP sites. I see my original request and the response from the website. But after that I see my browser making a request to a completely unrelated site and I can't figure out how that request gets initiated. If anyone can help me figure this out or knows of a tool (preferrably in Perl and preferrably one with they'd had success) that can help me, it would be greatly appreciated.
>
> (I know that this doesn't sound like a Perl question, but I've taken on this assignment and will only have time to do it if I have a Perl solution. The request for a tool even if it's not Perl, is a favor to the person who asked me to do this so that at least he has a head-start for whoever takes over for me if I can't do it in Perl).
>
> Thanks,
> Yossi
>
>
>       
> _______________________________________________
> Perl mailing list
> Perl at perl.org.il
> http://perl.org.il/mailman/listinfo/perl
>
>
>   




More information about the Perl mailing list