[Israel.pm] Site Crawler

Issac Goldstand margol at beamartyr.net
Sun Nov 14 13:00:24 PST 2004


I gave a talk on something similar at Jerusalem.pm (and may give it as a 
lightning talk at YAPC if I can't think of a better lightning talk) - you 
can look at the slides at http://www.beamartyr.net/jerusalem.pm/crawler/

  Yitzchak

PS - The basic technique is LWP + HTML::SimpleLinkExtor

----- Original Message ----- 
From: "Guy Malachi" <guy at ucmore.com>
To: <perl at perl.org.il>
Sent: Sunday, November 14, 2004 7:05 PM
Subject: [Israel.pm] Site Crawler


> Hey,
> Anybody have any tips on how I can create a site crawler that will
> extract all the links on a remote site and see if the site links to my
> site?
> Basically I have a list of urls that I want to check for each url if
> somewhere on the site (extracting all links and following onsite links
> recursively) there is a link to my site.
>
> Oh yea, it must run on Windows.
>
> TIA,
> Guy
>
> _______________________________________________
> Perl mailing list
> Perl at perl.org.il
> http://perl.org.il/mailman/listinfo/perl
> 




More information about the Perl mailing list