The R Project SVN R-packages

<?xml version="1.0"?>
<!-- <?xml-stylesheet type="text/xsl" href="../../../Docs/XSL/Rexample.xsl" ?> -->
<?xml-stylesheet type="text/xsl" href="http://www.omegahat.net/XSL/Rexample.xsl" ?>


<article xmlns:r="http://www.r-project.org"
           xmlns:s="http://cm.bell-labs.com/stat/S4"
           xmlns:c="http://www.C.org">
<package name="RCurl"/>
<title>Multiple Concurrent Downloads using <r:package>RCurl</r:package></title>

<section>
<title>Overview</title>
In this example, we look at how we can send multiple HTTP requests and
process them concurrently.  The basic idea is this.  We specify a
collection of URIs to download when establishing the HTTP requests, but
don't send any of the requests until they are all constructed.  Then, we
send the requests and harvest the results.  
When the last one finishes, we return control to the caller.

<para/>
This is different from processing the requests in the background.  In
that case, having dispatched the requests, control would be returned
to the caller and R would be able to do other things.  Notification of
pending content from the request would be done via the event
loop. This is feasible (at least on Unix), but different
from the example we are describing here.
</section>

<section>
<title>The Basics</title>

The implementation requires using the multi interface for libcurl.  We
create a multi handle and then we create a regular curl handle for
each individual request, i.e. for each URI to be fetched.  We add each
of these regular/easy curl handles to the multi handler and then call
<s:func>curlMultiPerform</s:func> until it terminates.  Terminating
means either an error or that each of the requests has completed.

<s:function>
<![CDATA[
getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
  content = list()
  curls = list()

  for(i in uris) {
    curl = getCurlHandle()
    content[[i]] = basicTextGatherer()
    opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)    
    curlSetOpt(.opts = opts, curl = curl)
    multiHandle = push(multiHandle, curl)
  }

  if(.perform) {
     perform(multiHandle)
     lapply(content, function(x) x$value())
   } else {
     return(list(multiHandle = multiHandle, content = content))
   }
}
]]>
</s:function>


To test this, we can fetch pages from the Omegahat Web site:
<s:expr>
getURIs(c("http://www.omegahat.net/index.html", "http://www.omegahat.net/RecentActivities.html"))
</s:expr>

To see what effect we have on timing, let's run the asynchronous
version and the serial version and compute their times.
<r:code>
<![CDATA[
uris = c("http://www.omegahat.net/index.html", "http://www.omegahat.net/RecentActivities.html")

atime = system.time(z <- getURIs(uris))
stime = system.time(zz <- lapply(uris, getURL))
]]>
</r:code>

How does this vary with larger numbers of URIs?

<r:code>
<![CDATA[
uris = c("http://www.omegahat.net/index.html", 
         "http://www.omegahat.net/RecentActivities.html",
         "http://www.omegahat.net/RCurl/index.html", 
         "http://www.omegahat.net/RCurl/philosophy.xml",
         "http://www.omegahat.net/RCurl/philosophy.html")

atime = system.time(z <- getURIs(uris))
stime = system.time(zz <- lapply(uris, getURL))
]]>
</r:code>


The key thing to note is that the user and system CPU times are much
lower for the serialized downloads, but the overall elapsed time is
significantly higher.  This is quite reasonable when one thinks about
it.  The asynchronous version spends time switching between the
downloads, checking if there is anything to be read on each. The
serialized version does not have to do this switching but can focus on
each single download.  The switch is done in a call to
<c:routine>select</c:routine>.  This can be arranged not to consume
CPU time, if I recall correctly, but it will do so if a timeout is
specified.

<para/>
This tradeoff between elapsed time and CPU time is a convenient
one. The user can chose which is important for their particular situation.

<para/>
Being statisticians, we should repeat the measurement process
and get data about the variability of the times taken in each
approach.

<!-- old approach
<r:code>
atimes = sapply(1:40, function(i) system.time(getURIs(uris)))
stimes = sapply(1:40, function(i) system.time(lapply(uris, getURI)))
</r:code>
-->


<r:code>
atimes = sapply(1:40, function(i) system.time(getURI(uris)))
stimes = sapply(1:40, function(i) system.time(getURI(uris, async = FALSE)))
</r:code>

<r:code>
times = data.frame(user = c(atimes[1,], stimes[1,]),
                   system = c(atimes[2,], stimes[2,]),
                   elapsed = c(atimes[3,], stimes[3,]),
                   style = factor(c(rep("Asynchronous", 40), rep("Serial", 40))))
</r:code>


<s:expr>
by(times, times$style, summary)
</s:expr>
<s:output>
times$style: Asynchronous
      user           system          elapsed               style   
 Min.   :0.340   Min.   :0.0300   Min.   :0.430   Asynchronous:40  
 1st Qu.:1.665   1st Qu.:0.1275   1st Qu.:2.085   Serial      : 0  
 Median :2.135   Median :0.1700   Median :2.865                    
 Mean   :2.319   Mean   :0.1790   Mean   :3.091                    
 3rd Qu.:2.740   3rd Qu.:0.2025   3rd Qu.:3.683                    
 Max.   :5.780   Max.   :0.5400   Max.   :7.410                    
------------------------------------------------------------ 
times$style: Serial
      user            system           elapsed                style   
 Min.   :0.0500   Min.   :0.00000   Min.   : 1.780   Asynchronous: 0  
 1st Qu.:0.0875   1st Qu.:0.00750   1st Qu.: 3.788   Serial      :40  
 Median :0.1050   Median :0.01000   Median : 5.320                    
 Mean   :0.1168   Mean   :0.01325   Mean   : 5.973                    
 3rd Qu.:0.1225   3rd Qu.:0.02000   3rd Qu.: 7.875                    
 Max.   :0.2500   Max.   :0.04000   Max.   :14.020                    
</s:output>

As usual, a picture makes things clearer.  A <s:func
package="lattice">stripplot</s:func> of elapsed time shows us the
different distributions for the serial and asynchronous transfer.
<inlinemediaobject>
<imageobject>
<imagedata fileref="elapsed.png"/>
</imageobject>
</inlinemediaobject>
Clearly, the download times for the asynchronous
download are significantly shorter than for the serial
download. And the variability is also much larger.

<para/>

Stripplots for user and system times show the
difference between the two modes of downloading also
and that the asynchronous method makes more use
of system resources.

<inlinemediaobject>
<imageobject>
<imagedata fileref="user.png"/>
</imageobject>
</inlinemediaobject>

<inlinemediaobject>
<imageobject>
<imagedata fileref="system.png"/>
</imageobject>
</inlinemediaobject>


</section>


<r:init>
library(RCurl)
</r:init>

</article>