Implement queuing and pipelining for HTTP requests over a persistent connection.

1 files changed, 153 insertions, 6 deletions

diff --git a/doc/http.n b/doc/http.n
index 40ced23..2dae77e 100644
--- a/doc/http.n
+++ b/doc/http.n
@@ -6,14 +6,14 @@
 '\" See the file "license.terms" for information on usage and redistribution
 '\" of this file, and for a DISCLAIMER OF ALL WARRANTIES.
 '\"
-.TH "http" n 2.7 http "Tcl Bundled Packages"
+.TH "http" n 2.8 http "Tcl Bundled Packages"
 .so man.macros
 .BS
 '\" Note:  do not modify the .SH NAME line immediately below!
 .SH NAME
 http \- Client-side implementation of the HTTP/1.1 protocol
 .SH SYNOPSIS
-\fBpackage require http ?2.7?\fR
+\fBpackage require http ?2.8?\fR
 .\" See Also -useragent option documentation in body!
 .sp
 \fB::http::config ?\fI\-option value\fR ...?
@@ -49,7 +49,7 @@ http \- Client-side implementation of the HTTP/1.1 protocol
 .SH DESCRIPTION
 .PP
 The \fBhttp\fR package provides the client side of the HTTP/1.1
-protocol, as defined in RFC 2616.
+protocol, as defined in RFC 7230 to RFC 7235, which supersede RFC 2616.
 The package implements the GET, POST, and HEAD operations
 of HTTP/1.1.  It allows configuration of a proxy host to get through
 firewalls.  The package is compatible with the \fBSafesock\fR security
@@ -95,6 +95,19 @@ comma-separated list of mime type patterns that you are
 willing to receive.  For example,
 .QW "image/gif, image/jpeg, text/*" .
 .TP
+\fB\-pipeline\fR \fIboolean\fR
+.
+Specifies whether HTTP/1.1 transactions on a persistent socket will be
+pipelined.  See the \fBPERSISTENT SOCKETS\fR section for details. The default
+is 1.
+.TP
+\fB\-postfresh\fR \fIboolean\fR
+.
+Specifies whether requests that use the \fBPOST\fR method will always use a
+fresh socket, overriding the \fB-keepalive\fR option of
+command \fBhttp::geturl\fR.  See the \fBPERSISTENT SOCKETS\fR section for details.
+The default is 0.
+.TP
 \fB\-proxyhost\fR \fIhostname\fR
 .
 The name of the proxy host, if any.  If this value is the
@@ -116,6 +129,18 @@ an empty list.  The default filter returns the values of the
 \fB\-proxyhost\fR and \fB\-proxyport\fR settings if they are
 non-empty.
 .TP
+\fB\-repost\fR \fIboolean\fR
+.
+Specifies what to do if a POST request over a persistent connection fails
+because the server has half-closed the connection.  If boolean \fBtrue\fR, the
+request
+will be automatically retried; if boolean \fBfalse\fR it will not, and the
+application
+that uses \fBhttp::geturl\fR is expected to seek user confirmation before
+retrying the POST.  The value \fBtrue\fR should be used only under certain
+conditions. See the \fBPERSISTENT SOCKETS\fR section for details. The
+default is 0.
+.TP
 \fB\-urlencoding\fR \fIencoding\fR
 .
 The \fIencoding\fR used for creating the x-url-encoded URLs with
@@ -128,8 +153,22 @@ characters.
 .TP
 \fB\-useragent\fR \fIstring\fR
 .
-The value of the User-Agent header in the HTTP request.  The default is
-.QW "\fBTcl http client package 2.7\fR" .
+The value of the User-Agent header in the HTTP request.  In an unsafe
+interpreter, the default value depends upon the operating system, and
+the version numbers of \fBhttp\fR and \fBTcl\fR, and is (for example)
+.QW "\fBMozilla/5.0 (Windows; U; Windows NT 10.0) http/2.8.12 Tcl/8.6.8\fR" .
+A safe interpreter cannot determine its operating system, and so the default
+in a safe interpreter is to use a Windows 10 value with the current version
+numbers of \fBhttp\fR and \fBTcl\fR.
+.TP
+\fB\-zip\fR \fIboolean\fR
+.
+If the value is boolean \fBtrue\fR, then by default requests will send a header
+.QW "\fBAccept-Encoding: gzip,deflate,compress\fR" .
+If the value is boolean \fBfalse\fR, then by default this header will not be sent.
+In either case the default can be overridden for an individual request by
+supplying a custom \fBAccept-Encoding\fR header in the \fB-headers\fR option
+of \fBhttp::geturl\fR. The default is 1.
 .RE
 .TP
 \fB::http::geturl\fR \fIurl\fR ?\fIoptions\fR?
@@ -227,7 +266,7 @@ Pragma: no-cache
 .TP
 \fB\-keepalive\fR \fIboolean\fR
 .
-If true, attempt to keep the connection open for servicing
+If boolean \fBtrue\fR, attempt to keep the connection open for servicing
 multiple requests.  Default is 0.
 .TP
 \fB\-method\fR \fItype\fR
@@ -504,6 +543,14 @@ The following elements of
 the array are supported:
 .RS
 .TP
+\fBbinary\fR
+.
+This is boolean \fBtrue\fR if (after decoding any compression specified
+by the
+.QW "Content-Encoding"
+response header) the HTTP response is binary.  It is boolean \fBfalse\fR
+if the HTTP response is text.
+.TP
 \fBbody\fR
 .
 The contents of the URL.  This will be empty if the \fB\-channel\fR
@@ -602,6 +649,106 @@ A copy of the \fBContent-Type\fR meta-data value.
 .
 The requested URL.
 .RE
+.SH "PERSISTENT CONNECTIONS"
+.PP
+.SS "BASICS"
+.PP
+See RFC 7230 Sec 6, which supersedes RFC 2616 Sec 8.1.
+.PP
+A persistent connection allows multiple HTTP/1.1 transactions to be
+carried over the same TCP connection.  Pipelining allows a
+client to make multiple requests over a persistent connection without
+waiting for each response.  The server sends responses in the same order
+that the requests were received.
+.PP
+If a POST request fails to complete, typically user confirmation is
+needed before sending the request again.  The user may wish to verify
+whether the server was modified by the failed POST request, before
+sending the same request again.
+.PP
+A HTTP request will use a persistent socket if the call to
+\fBhttp::geturl\fR has the option \fB-keepalive true\fR. It will use
+pipelining where permitted if the \fBhttp::config\fR option
+\fB-pipeline\fR is boolean \fBtrue\fR (its default value).
+.PP
+The http package maintains no more than one persistent connection to each
+server (i.e. each value of
+.QW "domain:port" ).
+If \fBhttp::geturl\fR is called to make a request over a persistent
+connection while the connection is busy with another request, the new
+request will be held in a queue until the connection is free.
+.PP
+The http package does not support HTTP/1.0 persistent connections
+controlled by the \fBKeep-Alive\fR header.
+.SS "SPECIAL CASES"
+.PP
+This subsection discusses issues related to closure of the
+persistent connection by the server, automatic retry of failed requests,
+the special treatment necessary for POST requests, and the options for
+dealing with these cases.
+.PP
+In accordance with RFC 7230, \fBhttp::geturl\fR does not pipeline
+requests that use the POST method.  If a POST uses a persistent
+connection and is not the first request on that connection,
+\fBhttp::geturl\fR waits until it has received the response for the previous
+request; or (if \fBhttp::config\fR option \fB-postfresh\fR is boolean \fBtrue\fR) it
+uses a new connection for each POST.
+.PP
+If the server is processing a number of pipelined requests, and sends a
+response header
+.QW "\fBConnection: close\fR"
+with one of the responses (other than the last), then subsequent responses
+are unfulfilled. \fBhttp::geturl\fR will send the unfulfilled requests again
+over a new connection.
+.PP
+A difficulty arises when a HTTP client sends a request over a persistent
+connection that has been idle for a while.  The HTTP server may
+half-close an apparently idle connection while the client is sending a
+request, but before the request arrives at the server: in this case (an
+.QW "asynchronous close event" )
+the request will fail.  The difficulty arises because the client cannot
+be certain whether the POST modified the state of the server.  For HEAD or
+GET requests, \fBhttp::geturl\fR opens another connection and retransmits
+the failed request. However, if the request was a POST, RFC 7230 forbids
+automatic retry by default, suggesting either user confirmation, or
+confirmation by user-agent software that has semantic understanding of
+the application.  The \fBhttp::config\fR option \fB-repost\fR allows for
+either possibility.
+.PP
+Asynchronous close events can occur only in a short interval of time.  The
+\fBhttp\fR package monitors each persistent connection for closure by the
+server.  Upon detection, the connection is also closed at the client end,
+and subsequent requests will use a fresh connection.
+.PP
+If the \fBhttp::geturl\fR command is called with option \fB-keepalive true\fR,
+then it will both try to use an existing persistent connection
+(if one is available), and it will send the server a
+.QW "\fBConnection: keep-alive\fR"
+request header asking to keep the connection open for future requests.
+.PP
+The \fBhttp::config\fR options \fB-pipeline\fR, \fB-postfresh\fR, and
+\fB-repost\fR relate to persistent connections.
+.PP
+Option \fB-pipeline\fR, if boolean \fBtrue\fR, will pipeline GET and HEAD requests
+made
+over a persistent connection.  POST requests will not be pipelined - if the
+POST is not the first transaction on the connection, its request will not
+be sent until the previous response has finished.  GET and HEAD requests
+made after a POST will not be sent until the POST response has been
+delivered, and will not be sent if the POST fails.
+.PP
+Option \fB-postfresh\fR, if boolean \fBtrue\fR, will override the \fBhttp::geturl\fR option
+\fB-keepalive\fR, and always open a fresh connection for a POST request.
+.PP
+Option \fB-repost\fR, if \fBtrue\fR, permits automatic retry of a POST request
+that fails because it uses a persistent connection that the server has
+half-closed (an 
+.QW "asynchronous close event" ).
+Subsequent GET and HEAD requests in a failed pipeline will also be retried.
+\fIThe -repost option should be used only if the application understands
+that the retry is appropriate\fR - specifically, the application must know
+that if the failed POST successfully modified the state of the server, a repeat POST
+would have no adverse effect.
 .SH EXAMPLE
 .PP
 This example creates a procedure to copy a URL to a file while printing a