Friday, January 28, 2011

Perl: Retrieve URLs with LWP and LWP::Simple

With Perl there are many ways to make requests over the web. One method is to use the LWP module. Below is an example of using it grab the contents of a web page:

use Carp;
use LWP;

my $url = 'http://prystash.blogspot.com';
my $contents = get_contents_from($url);

print $contents;

sub get_contents_from {
my ($url) = @_;

my $agent = LWP::UserAgent->new;
my $request = HTTP::Request->new(GET => $url);
my $response = $agent->request($request);

if (!$response->is_success) {
croak "Could not get URL '$url'";
}

return $response->content
}
Another simpler method is to use the LWP::Simple module:
use Carp;
use LWP::Simple;

my $url = 'http://prystash.blogspot.com';
my $contents = get_contents_from($url);

sub get_contents_from {
my ($url) = @_;
my $contents = get($url) or croak "Could not get URL '$url'";
return $contents;
}

2 comments:

  1. plz can u tell how to refresh or reload the page

    ReplyDelete
  2. I'm not so sure you can, but wouldn't a refresh simply be another GET of the page?

    There's a little cookbook here, and I don't see anything regrading refreshing.

    ReplyDelete