.\" Automatically generated by Pod::Man 4.11 (Pod::Simple 3.35)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
.    ds C`
.    ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
.    if \nF \{\
.        de IX
.        tm Index:\\$1\t\\n%\t"\\$2"
..
.        if !\nF==2 \{\
.            nr % 0
.            nr F 2
.        \}
.    \}
.\}
.rr rF
.\" ========================================================================
.\"
.IX Title "LWP::Parallel::RobotUA 3"
.TH LWP::Parallel::RobotUA 3 "2016-05-29" "perl v5.26.3" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
LWP::Parallel::RobotUA \- A class for Parallel Web Robots
.SH "SYNOPSIS"
.IX Header "SYNOPSIS"
.Vb 7
\&  require LWP::Parallel::RobotUA;
\&  $ua = new LWP::Parallel::RobotUA \*(Aqmy\-robot/0.1\*(Aq, \*(Aqme@foo.com\*(Aq;
\&  $ua\->delay(0.5);  # in minutes!
\&  ...
\&  # just use it just like a normal LWP::Parallel::UserAgent
\&  $ua\->register ($request, \e&callback, 4096); # or
\&  $ua\->wait ( $timeout );
.Ve
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
This class implements a user agent that is suitable for robot
applications.  Robots should be nice to the servers they visit.  They
should consult the \fI/robots.txt\fR file to ensure that they are welcomed
and they should not make requests too frequently.
.PP
But, before you consider writing a robot take a look at
<URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.
.PP
When you use a \fILWP::Parallel::RobotUA\fR as your user agent, then you do not
really have to think about these things yourself.  Just send requests
as you do when you are using a normal \fILWP::Parallel::UserAgent\fR and this
special agent will make sure you are nice.
.SH "METHODS"
.IX Header "METHODS"
The LWP::Parallel::RobotUA is a sub-class of LWP::Parallel::UserAgent
and LWP::RobotUA and implements a mix of their methods.
.PP
In addition to LWP::Parallel::UserAgent, these methods are provided:
.ie n .SS "$ua = LWP::Parallel::RobotUA\->new($agent_name, $from, [$rules])"
.el .SS "\f(CW$ua\fP = LWP::Parallel::RobotUA\->new($agent_name, \f(CW$from\fP, [$rules])"
.IX Subsection "$ua = LWP::Parallel::RobotUA->new($agent_name, $from, [$rules])"
Your robot's name and the mail address of the human responsible for
the robot (i.e. you) are required by the constructor.
.PP
Optionally it allows you to specify the \fIWWW::RobotRules\fR object to
use. (See WWW::RobotRules::AnyDBM_File for persistent caching of
robot rules in a local file)
.ie n .SS "$ua\->delay([$minutes])"
.el .SS "\f(CW$ua\fP\->delay([$minutes])"
.IX Subsection "$ua->delay([$minutes])"
Set/Get the minimum delay between requests to the same server.  The
default is 1 minute.
.PP
Note: Previous versions of \s-1LWP\s0 Parallel-Robot used \fISeconds\fR instead of 
      \fIMinutes\fR! This is now compatible with \s-1LWP\s0 Robot.
.ie n .SS "$ua\->host_wait($netloc)"
.el .SS "\f(CW$ua\fP\->host_wait($netloc)"
.IX Subsection "$ua->host_wait($netloc)"
Returns the number of seconds you must wait before you can make a new
request to this server. This method keeps track of all of the robots
connection, and enforces the delay constraint specified via the delay
method above for each server individually.
.PP
Note: Although it says 'host', it really means 'netloc/server',
i.e. it differentiates between individual servers running on different
ports, even though they might be on the same machine ('host'). This
function is mostly used internally, where RobotUA calls it to find out
when to send the next request to a certain server.
.ie n .SS "$ua\->as_string"
.el .SS "\f(CW$ua\fP\->as_string"
.IX Subsection "$ua->as_string"
Returns a string that describes the state of the \s-1UA.\s0
Mainly useful for debugging.
.SH "SEE ALSO"
.IX Header "SEE ALSO"
LWP::Parallel::UserAgent, LWP::RobotUA, WWW::RobotRules
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright 1997\-2004 Marc Langheinrich <marclang@cpan.org>
.PP
This library is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.