Minor bugfixes and refactoring of new code. Handle more errors in OpenSocket. Rename some keys of dict returned by ::http::responseInfo, and add more keys. Improve error reporting from ::http::OpenSocket and ::http::socket, especially when using threads. Adapt http(n) to emphasize using http::* commands rather than peeking at internal state array.

1 files changed, 535 insertions, 239 deletions

diff --git a/doc/http.n b/doc/http.n
index 8a9c35b..5ba4813 100644
--- a/doc/http.n
+++ b/doc/http.n
@@ -13,7 +13,7 @@
 .SH NAME
 http \- Client-side implementation of the HTTP/1.1 protocol
 .SH SYNOPSIS
-\fBpackage require http\fI ?\fB2.10\fR?
+\fBpackage require http\fR ?\fB2.10\fR?
 .\" See Also -useragent option documentation in body!
 .sp
 \fB::http::config\fR ?\fI\-option value\fR ...?
@@ -80,7 +80,8 @@ Namespace \fBhttp\fR exports the commands \fBconfig\fR, \fBformatQuery\fR,
 \fBregister\fR,
 \fBregisterError\fR, \fBrequestHeaders\fR, \fBrequestHeaderValue\fR,
 \fBrequestLine\fR, \fBresponseBody\fR, \fBresponseCode\fR,
-\fBresponseHeaders\fR, \fBresponseHeaderValue\fR, \fBresponseInfo\fR, \fBresponseLine\fR,
+\fBresponseHeaders\fR, \fBresponseHeaderValue\fR, \fBresponseInfo\fR,
+\fBresponseLine\fR,
 \fBreset\fR, \fBunregister\fR, and \fBwait\fR.
 .PP
 It does not export the commands \fBcleanup\fR, \fBcode\fR, \fBdata\fR,
@@ -90,7 +91,8 @@ It does not export the commands \fBcleanup\fR, \fBcode\fR, \fBdata\fR,
 .SH DESCRIPTION
 .PP
 The \fBhttp\fR package provides the client side of the HTTP/1.1
-protocol, as defined in RFC 7230 to RFC 7235, which supersede RFC 2616.
+protocol, as defined in RFC 9110 to 9112, which supersede RFC 7230
+to RFC 7235, which in turn supersede RFC 2616.
 The package implements the GET, POST, and HEAD operations
 of HTTP/1.1.  It allows configuration of a proxy host to get through
 firewalls.  The package is compatible with the \fBSafesock\fR security
@@ -103,14 +105,13 @@ The \fB::http::geturl\fR procedure does a HTTP transaction.
 Its \fIoptions \fR determine whether a GET, POST, or HEAD transaction
 is performed.
 The return value of \fB::http::geturl\fR is a token for the transaction.
-The value is also the name of an array in the ::http namespace
-that contains state information about the transaction.  The elements
-of this array are described in the \fBSTATE ARRAY\fR section.
+The token can be supplied as an argument to other commands, to manage the
+transaction and examine its results.
 .PP
 If the \fB\-command\fR option is specified, then
 the HTTP operation is done in the background.
 \fB::http::geturl\fR returns immediately after generating the
-HTTP request and the callback is invoked
+HTTP request and the \fB\-command\fR callback is invoked
 when the transaction completes.  For this to work, the Tcl event loop
 must be active.  In Tk applications this is always true.  For pure-Tcl
 applications, the caller can use \fB::http::wait\fR after calling
@@ -119,6 +120,15 @@ applications, the caller can use \fB::http::wait\fR after calling
 \fBNote:\fR The event queue is even used without the \fB\-command\fR option.
 As a side effect, arbitrary commands may be processed while \fBhttp::geturl\fR
 is running.
+.PP
+When the HTTP server has replied to the request, call the command
+\fB::http::responseInfo\fR, which
+returns a \fBdict\fR of metadata that is essential for identifying a
+successful transaction and making use of the response.  See
+section \fBMETADATA\fR for details of the information returned.
+The response itself is returned by command \fB::http::responseBody\fR,
+unless it has been redirected to a file by the \fI\-channel\fR option
+of \fB::http::geturl\fR.
 .SH COMMANDS
 .TP
 \fB::http::config\fR ?\fIoptions\fR?
@@ -204,7 +214,8 @@ default is 0.
 .TP
 \fB\-threadlevel\fR \fIlevel\fR
 .
-Specifies whether and how to use the \fBThread\fR package.  Possible values of \fIlevel\fR are 0, 1 or 2.
+Specifies whether and how to use the \fBThread\fR package.  Possible values
+of \fIlevel\fR are 0, 1 or 2.
 .RS
 .PP
 .DS
@@ -212,7 +223,11 @@ Specifies whether and how to use the \fBThread\fR package.  Possible values of \
 1 - use Thread if it is available, do not use it if it is unavailable
 2 - use Thread if it is available, raise an error if it is unavailable
 .DE
-The Tcl \fBsocket -async\fR command can block in adverse cases (e.g. a slow DNS lookup).  Using the Thread package works around this problem, for both HTTP and HTTPS transactions.  Values of \fIlevel\fR other than 0 are available only to the main interpreter in each thread.  See section \fBTHREADS\fR for more information.
+The Tcl \fBsocket -async\fR command can block in adverse cases (e.g. a slow
+DNS lookup).  Using the Thread package works around this problem, for both
+HTTP and HTTPS transactions.  Values of \fIlevel\fR other than 0 are
+available only to the main interpreter in each thread.  See
+section \fBTHREADS\fR for more information.
 .RE
 .TP
 \fB\-urlencoding\fR \fIencoding\fR
@@ -235,20 +250,21 @@ numbers of \fBhttp\fR and \fBTcl\fR.
 .
 If the value is boolean \fBtrue\fR, then by default requests will send a header
 .QW "\fBAccept-Encoding: gzip,deflate\fR" .
-If the value is boolean \fBfalse\fR, then by default this header will not be
-sent.  In either case the default can be overridden for an individual request by
+If the value is boolean \fBfalse\fR, then by default requests will send a header
+.QW "\fBAccept-Encoding: identity\fR" .
+In either case the default can be overridden for an individual request by
 supplying a custom \fBAccept-Encoding\fR header in the \fB\-headers\fR option
-of \fBhttp::geturl\fR. The default is 1.
+of \fBhttp::geturl\fR. The default value is 1.
 .RE
 .TP
 \fB::http::geturl\fR \fIurl\fR ?\fIoptions\fR?
 .
 The \fB::http::geturl\fR command is the main procedure in the package.
-The \fB\-query\fR option causes a POST operation and
+The \fB\-query\fR or \fB\-querychannel\fR option causes a POST operation and
 the \fB\-validate\fR option causes a HEAD operation;
 otherwise, a GET operation is performed.  The \fB::http::geturl\fR command
-returns a \fItoken\fR value that can be used to get
-information about the transaction.  See the \fBSTATE ARRAY\fR and
+returns a \fItoken\fR value that can be passed as an argument to other commands
+to get information about the transaction.  See the \fBMETADATA\fR and
 \fBERRORS\fR section for
 details.  The \fB::http::geturl\fR command blocks until the operation
 completes, unless the \fB\-command\fR option specifies a callback
@@ -272,13 +288,14 @@ At most \fIsize\fR bytes are read at once.  After each block, a call to the
 \fB\-channel\fR \fIname\fR
 .
 Copy the URL contents to channel \fIname\fR instead of saving it in
-\fBstate(body)\fR.
+a Tcl variable for retrieval by \fB::http::responseBody\fR.
 .TP
 \fB\-command\fR \fIcallback\fR
 .
-Invoke \fIcallback\fR after the HTTP transaction completes.
-This option causes \fB::http::geturl\fR to return immediately.
-The \fIcallback\fR gets an additional argument that is the \fItoken\fR returned
+The presence of this option causes \fB::http::geturl\fR to return immediately.
+After the HTTP transaction completes, the value of \fIcallback\fR is expanded,
+an additional argument is added, and the resulting command is evaluated.
+The additional argument is the \fItoken\fR returned
 from \fB::http::geturl\fR. This token is the name of an array that is
 described in the \fBSTATE ARRAY\fR section.  Here is a template for the
 callback:
@@ -286,8 +303,10 @@ callback:
 .PP
 .CS
 proc httpCallback {token} {
-    upvar #0 $token state
-    # Access state as a Tcl array
+    upvar 0 $token state
+    # Access state as a Tcl array defined in this proc
+    ...
+    return
 }
 .CE
 .PP
@@ -299,22 +318,28 @@ details.
 .TP
 \fB\-guesstype\fR \fIboolean\fR
 .
-Attempt to guess the Content-Type and character set when a misconfigured
+Attempt to guess the \fBContent-Type\fR and character set when a misconfigured
 server provides no information.  The default value is \fIfalse\fR (do
 nothing).  If boolean \fItrue\fR then, if the server does not send a
-"Content-Type" header, or if it sends the value "application/octet-stream",
+\fBContent-Type\fR header, or if it sends the value "application/octet-stream",
 \fBhttp::geturl\fR will attempt to guess appropriate values.  This is not
 intended to become a general-purpose tool, and currently it is limited to
 detecting XML documents that begin with an XML declaration.  In this case
-the Content-Type is changed to "application/xml", and the character set to
+the \fBContent-Type\fR is changed to "application/xml", the binary flag
+state(binary) is changed to 0, and the character set is changed to
 the one specified by the "encoding" tag of the XML line, or to utf-8 if no
-encoding is specified.
+encoding is specified.  Not used if a \fI\-channel\fR is specified.
 .TP
 \fB\-handler\fR \fIcallback\fR
 .
-Invoke \fIcallback\fR whenever HTTP data is available; if present, nothing
-else will be done with the HTTP data.  This procedure gets two additional
-arguments: the socket for the HTTP data and the \fItoken\fR returned from
+If this option is absent, \fBhttp::geturl\fR processes incoming data itself,
+either appending it to the state(body) variable or writing it to the -channel.
+But if the \fB\-handler\fR option is present, \fBhttp::geturl\fR does not do
+this processing and instead calls \fIcallback\fR.
+Whenever HTTP data is available, the value of \fIcallback\fR is expanded, an
+additional two arguments are added, and the resulting command is evaluated.
+The two additional
+arguments are: the socket for the HTTP data and the \fItoken\fR returned from
 \fB::http::geturl\fR.  The token is the name of a global array that is
 described in the \fBSTATE ARRAY\fR section.  The procedure is expected
 to return the number of bytes read from the socket.  Here is a
@@ -323,8 +348,8 @@ template for the callback:
 .PP
 .CS
 proc httpHandlerCallback {socket token} {
-    upvar #0 $token state
-    # Access socket, and state as a Tcl array
+    upvar 0 $token state
+    # Access socket, and state as a Tcl array defined in this proc
     # For example...
     ...
     set data [read $socket 1000]
@@ -337,8 +362,9 @@ proc httpHandlerCallback {socket token} {
 The \fBhttp::geturl\fR code for the \fB\-handler\fR option is not compatible
 with either compression or chunked transfer-encoding.  If \fB\-handler\fR is
 specified, then to work around these issues \fBhttp::geturl\fR will reduce the
-HTTP protocol to 1.0, and override the \fB\-zip\fR option (i.e. it will not
-send the header "\fBAccept-Encoding: gzip,deflate\fR").
+HTTP protocol to 1.0, and override the \fB\-zip\fR option (i.e. it will
+send the header \fBAccept-Encoding: identity\fR instead
+of \fBAccept-Encoding: gzip,deflate\fR).
 .PP
 If options \fB\-handler\fR and \fB\-channel\fR are used together, the handler
 is responsible for copying the data from the HTTP socket to the specified
@@ -384,7 +410,10 @@ It is the caller's responsibility to ensure that the headers and request body
 (if any) conform to the requirements of the request method.  For example, if
 using \fB\-method\fR \fIPOST\fR to send a POST with an empty request body, the
 caller must also supply the option
-.QW "\-headers {Content-Length 0}" .
+.PP
+.CS
+\-headers {Content-Length 0}
+.CE
 .RE
 .TP
 \fB\-myaddr\fR \fIaddress\fR
@@ -394,18 +423,26 @@ multiple interfaces are available.
 .TP
 \fB\-progress\fR \fIcallback\fR
 .
-The \fIcallback\fR is made after each transfer of data from the URL.
-The callback gets three additional arguments: the \fItoken\fR from
+If the \fB\-progress\fR option is present, 
+then the \fIcallback\fR is made after each transfer of data from the URL.
+The value of \fIcallback\fR is expanded, an additional three arguments are
+added, and the resulting command is evaluated.
+The three additional arguments are: the \fItoken\fR returned from
 \fB::http::geturl\fR, the expected total size of the contents from the
-\fBContent-Length\fR metadata, and the current number of bytes
-transferred so far.  The expected total size may be unknown, in which
+\fBContent-Length\fR response header, and the current number of bytes
+transferred so far.  The token is the name of a global array that is
+described in the \fBSTATE ARRAY\fR section.  The expected total size may
+be unknown, in which
 case zero is passed to the callback.  Here is a template for the
 progress callback:
 .RS
 .PP
 .CS
 proc httpProgress {token total current} {
-    upvar #0 $token state
+    upvar 0 $token state
+    # Access state as a Tcl array defined in this proc
+    ...
+    return
 }
 .CE
 .RE
@@ -449,20 +486,24 @@ This flag causes \fB::http::geturl\fR to do a POST request that passes the
 data contained in \fIchannelID\fR to the server. The data contained in
 \fIchannelID\fR must be an x-url-encoding
 formatted query unless the \fB\-type\fR option below is used.
-If a Content-Length header is not specified via the \fB\-headers\fR options,
-\fB::http::geturl\fR attempts to determine the size of the post data
+If a \fBContent-Length\fR header is not specified via the \fB\-headers\fR
+options, \fB::http::geturl\fR attempts to determine the size of the post data
 in order to create that header.  If it is
 unable to determine the size, it returns an error.
 .TP
 \fB\-queryprogress\fR \fIcallback\fR
 .
-The \fIcallback\fR is made after each transfer of data to the URL
-(i.e. POST) and acts exactly like the \fB\-progress\fR option (the
-callback format is the same).
+If the \fB\-queryprogress\fR option is present, 
+then the \fIcallback\fR is made after each transfer of data to the URL
+in a POST request (i.e. a call to \fB::http::geturl\fR with
+option \fB\-query\fR or \fB\-querychannel\fR) and acts exactly like
+the \fB\-progress\fR option (the callback format is the same).
 .TP
 \fB\-strict\fR \fIboolean\fR
 .
-Whether to enforce RFC 3986 URL validation on the request.  Default is 1.
+If true then the command will test that the URL complies with RFC 3986, i.e.
+that it has no characters that should be "x-url-encoded" (e.g. a space should
+be encoded to "%20").  Default value is 1.
 .TP
 \fB\-timeout\fR \fImilliseconds\fR
 .
@@ -470,7 +511,8 @@ If \fImilliseconds\fR is non-zero, then \fB::http::geturl\fR sets up a timeout
 to occur after the specified number of milliseconds.
 A timeout results in a call to \fB::http::reset\fR and to
 the \fB\-command\fR callback, if specified.
-The return value of \fB::http::status\fR is \fBtimeout\fR
+The return value of \fB::http::status\fR (and the value of the \fIstatus\fR key
+in the dictionary returned by \fB::http::responseInfo\fR) is \fBtimeout\fR
 after a timeout has occurred.
 .TP
 \fB\-type\fR \fImime-type\fR
@@ -482,10 +524,11 @@ POST operation.
 \fB\-validate\fR \fIboolean\fR
 .
 If \fIboolean\fR is non-zero, then \fB::http::geturl\fR does an HTTP HEAD
-request.  This request returns meta information about the URL, but the
-contents are not returned.  The meta information is available in the
-\fBstate(meta) \fR variable after the transaction.  See the
-\fBSTATE ARRAY\fR section for details.
+request.  This server returns the same status line and response headers as it
+would for a HTTP GET request, but omits the response entity
+(the URL "contents").  The response headers are available after the
+transaction using command \fB::http::responseHeaders\fR or, for selected
+information, \fB::http::responseInfo\fR.
 .RE
 .TP
 \fB::http::formatQuery\fR \fIkey value\fR ?\fIkey value\fR ...?
@@ -509,7 +552,7 @@ This sets the \fBstate(status)\fR value to \fIwhy\fR, which defaults to
 .TP
 \fB::http::wait\fR \fItoken\fR
 .
-This is a convenience procedure that blocks and waits for the
+This command blocks and waits for the
 transaction to complete.  This only works in trusted code because it
 uses \fBvwait\fR.  Also, it is not useful for the case where
 \fB::http::geturl\fR is called \fIwithout\fR the \fB\-command\fR option
@@ -519,29 +562,44 @@ wait for.
 .TP
 \fB::http::status\fR \fItoken\fR
 .
-This is a convenience procedure that returns the \fBstatus\fR element of
-the state array.
+This command returns a description of the status of the HTTP transaction.
+The return value is the empty string until the HTTP transaction is
+completed; after completion it has one of the values ok, eof, error,
+timeout, and reset.  The meaning of these values is described in the
+section \fBERRORS\fR (below).
+.PP
+.RS
+The name "status" is not related to the terms "status line" and
+"status code" that are defined for a HTTP response.
+.RE
 .TP
 \fB::http::size\fR \fItoken\fR
 .
-This is a convenience procedure that returns the \fBcurrentsize\fR
-element of the state array, which represents the number of bytes
-received from the URL in the \fB::http::geturl\fR call.
+This command returns the number of bytes
+received so far from the URL in the \fB::http::geturl\fR call.
 .TP
 \fB::http::error\fR \fItoken\fR
 .
-This is a convenience procedure that returns the \fBerror\fR element
-of the state array.
+This command returns the error information if the HTTP transaction failed,
+or the empty string if there was no error.  The information is a Tcl list of
+the error message, stack trace, and error code.
 .TP
 \fB::http::postError\fR \fItoken\fR
 .
-A POST request is a call to \fB::http::geturl\fR with either the \fB\-query\fR or \fB\-querychannel\fR option.  The \fB::http::postError\fR command returns the error string generated when a HTTP POST request sends its request-body to the server; or the empty string if there was no error.  When this type of error occurs, the \fB::http::geturl\fR command continues the transaction and attempts to receive a response from the server.
+A POST request is a call to \fB::http::geturl\fR with either
+the \fB\-query\fR or \fB\-querychannel\fR option.
+The \fB::http::postError\fR command returns the error information generated
+when a HTTP POST request sends its request-body to the server; or the empty
+string if there was no error.  The information is a Tcl list of the error
+message, stack trace, and error code.  When this type of error occurs,
+the \fB::http::geturl\fR command continues the transaction and attempts to
+receive a response from the server.
 .TP
 \fB::http::cleanup\fR \fItoken\fR
 .
 This procedure cleans up the state associated with the connection
 identified by \fItoken\fR.  After this call, the procedures
-like \fB::http::data\fR cannot be used to get information
+like \fB::http::responseBody\fR cannot be used to get information
 about the operation.  It is \fIstrongly\fR recommended that you call
 this function after you are done with a given HTTP request.  Not doing
 so will result in memory not being freed, and if your app calls
@@ -565,13 +623,12 @@ POST /forms/order.html HTTP/1.1
 .TP
 \fB::http::requestHeaders\fR \fItoken\fR ?\fIheaderName\fR?
 .
-This command returns a list of the HTTP request header names and values, in the
-order that they were sent to the server: a Tcl list of the form
+This command returns the HTTP request header names and values, in the
+order that they were sent to the server, as a Tcl list of the form
 ?name value ...?  Header names are case-insensitive and are converted to lower
 case.  The return value is not a \fBdict\fR because some header names may occur
 more than once.  If one argument is supplied, all request headers
-are returned: the value is that of the \fBrequestHeaders\fR element
-of the state array (described below).  If two arguments are supplied, the
+are returned.  If two arguments are supplied, the
 second provides the value of a header name.  Only headers with the requested
 name (converted to lower case) are returned.  If no such headers are found,
 an empty list is returned.
@@ -587,9 +644,8 @@ preserving their order.
 .TP
 \fB::http::responseLine\fR \fItoken\fR
 .
-This command returns the "status line" of the server response (which is stored
-as element \fBhttp\fR of the state array).
-The "status line" is the first line of a HTTP server response, and has three
+This command returns the first line of the server response: the
+HTTP "status line".  The "status line" has three
 elements separated by spaces: the HTTP version, a three-digit numerical
 "status code", and a "reason phrase".  Only the reason phrase may contain
 spaces.  Examples:
@@ -619,7 +675,7 @@ command \fB::http::reasonPhrase\fR.
 .
 This command returns the "status code" (200, 404, etc.) of the server
 "status line".  If a three-digit code cannot be found, the full status
-line is returned.  See command \fB::http::code\fR for more information
+line is returned.  See command \fB::http::responseLine\fR for more information
 on the "status line".
 .TP
 \fB::http::reasonPhrase\fR \fIcode\fR
@@ -636,8 +692,9 @@ HEAD, GET, or POST request method.
 The "reason phrase" returned by a HTTP server may differ from the recommended
 value, without affecting the HTTP protocol.  The value returned by
 \fB::http::geturl\fR can be obtained by calling either command
-\fB::http::code\fR (which returns the full status line) or command
-\fB::http::ncode\fR (for the status code only).
+\fB::http::responseLine\fR (which returns the full status line) or command
+\fB::http::responseInfo\fR (which returns a dictionary, with
+the "reason phrase" stored in key \fIreasonPhrase\fR).
 .PP
 A registry of valid status codes is maintained at
 https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml
@@ -645,16 +702,18 @@ https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml
 .TP
 \fB::http::responseHeaders\fR \fItoken\fR ?\fIheaderName\fR?
 .
-This command returns a list of HTTP response header names and values, in the
-order that they were received from the server: a Tcl list of the form
+The response from a HTTP server includes metadata headers that describe the
+response body and the transaction itself.
+This command returns the HTTP response header names and values, in the
+order that they were received from the server, as a Tcl list of the form
 ?name value ...?  Header names are case-insensitive and are converted to lower
 case.  The return value is not a \fBdict\fR because some header names may occur
-more than once, notably \fIset-cookie\fR.  If one argument is supplied, all
-response headers are returned: the value is that of the \fBmeta\fR element
-of the state array (described below).  If two arguments are supplied, the
-second provides the value of a header name.  Only headers with the requested
-name (converted to lower case) are returned.  If no such headers are found,
-an empty list is returned.
+more than once, notably \fBSet-Cookie\fR.  If the second argument is not
+supplied, all response headers are returned.  If the second argument is
+supplied, it provides the value of a header name.  Only headers with the
+requested name (converted to lower case) are returned.  If no such headers
+are found, an empty list is returned.  See section \fBMETADATA\fR for more
+information.
 .TP
 \fB::http::responseHeaderValue\fR \fItoken\fR \fIheaderName\fR
 .
@@ -664,17 +723,32 @@ lower case.  If no such header exists, the return value is the empty string.
 If there are multiple headers named \fIheaderName\fR, the result is obtained
 by joining the individual values with the string ", " (comma and space),
 preserving their order.  Multiple headers with the same name may be processed
-in this manner, except \fIset-cookie\fR which does not conform to the
+in this manner, except \fBSet-Cookie\fR which does not conform to the
 comma-separated-list syntax and cannot be combined into a single value.
-Each \fIset-cookie\fR header must be treated individually, e.g. by processing
-the return value of \fB::http::responseHeaders\fR \fItoken\fR \fIset-cookie\fR.
+Each \fBSet-Cookie\fR header must be treated individually, e.g. by processing
+the return value of \fB::http::responseHeaders\fR \fItoken\fR \fBSet-Cookie\fR.
 .TP
 \fB::http::responseInfo\fR \fItoken\fR
 .
-This command returns a \fBdict\fR of selected response metadata that are essential for identifying a successful transaction and making use of the response, along with other metadata that are informational.  The keys of the \fBdict\fR are \fIstage\fR, \fIstatus\fR, \fIncode\fR, \fIreason\fR, \fItype\fR, \fIbinary\fR, \fIredirection\fR, \fIcharset\fR, \fIcoding\fR, \fIhttpRequest\fR, \fIhttpResponse\fR, \fIurl\fR, \fIconnRequest\fR, \fIconnResponse\fR, \fIconnection\fR, \fItransfer\fR, \fItotalsize\fR, and \fIcurrentsize\fR.  The meaning of these keys is described in the chapter \fBMETADATA\fR below.
+This command returns a \fBdict\fR of selected response metadata that are
+essential for identifying a successful transaction and making use of the
+response, along with other metadata that are informational.  The keys of
+the \fBdict\fR are \fIstage\fR, \fIstatus\fR, \fIresponseCode\fR,
+\fIreasonPhrase\fR, \fIcontentType\fR, \fIbinary\fR, \fIredirection\fR,
+\fIupgrade\fR, \fIerror\fR, \fIpostError\fR, \fImethod\fR, \fIcharset\fR,
+\fIcompression\fR, \fIhttpRequest\fR, \fIhttpResponse\fR, \fIurl\fR,
+\fIconnectionRequest\fR, \fIconnectionResponse\fR, \fIconnectionActual\fR,
+\fItransferEncoding\fR, \fItotalPost\fR, \fIcurrentPost\fR, \fItotalSize\fR,
+and \fIcurrentSize\fR.  The meaning of these keys is described in the
+section \fBMETADATA\fR below.
 .RS
 .PP
-It is always worth checking the value of \fIbinary\fR after a HTTP transaction, to determine whether a misconfigured server has caused http to interpret a text resource as a binary, or vice versa.
+It is always worth checking the value of \fIbinary\fR after a HTTP transaction,
+to determine whether a misconfigured server has caused http to interpret a
+text resource as a binary, or vice versa.
+.PP
+After a POST transaction, check the value of \fIpostError\fR to verify that
+the request body was uploaded without error.
 .RE
 .TP
 \fB::http::responseBody\fR \fItoken\fR
@@ -687,7 +761,7 @@ channel, and the command returns the empty string).
 Other terms for
 "entity", with varying precision, include "representation of resource",
 "resource", "response body after decoding", "payload",
-"message body after decoding", "content", and "file".
+"message body after decoding", "content(s)", and "file".
 .RE
 .TP
 \fB::http::register\fR \fIproto port command\fR
@@ -738,25 +812,21 @@ An alternative name for the command \fB::http::responseBody\fR.
 .
 An alternative name for the command \fB::http::responseHeaders\fR
 .TP
-\fB::http::metaValue\fR \fItoken\fR \fIheaderName\fR
-.
-An alternative name for the command \fB::http::responseHeaderValue\fR
-.TP
 \fB::http::ncode\fR \fItoken\fR
 .
 An alternative name for the command \fB::http::responseCode\fR
 .SH ERRORS
 The \fB::http::geturl\fR procedure will raise errors in the following cases:
 invalid command line options,
-an invalid URL,
-a URL on a non-existent host,
-or a URL at a bad port on an existing host.
+or an invalid URL.
 These errors mean that it
 cannot even start the network transaction.
-It will also raise an error if it gets an I/O error while
-writing out the HTTP request header.
 For synchronous \fB::http::geturl\fR calls (where \fB\-command\fR is
-not specified), it will raise an error if it gets an I/O error while
+not specified), it will raise an error if
+the URL is on a non-existent host
+or at a bad port on an existing host.
+It will also raise an error for any I/O errors while
+writing out the HTTP request line and headers, or
 reading the HTTP reply headers or data.  Because \fB::http::geturl\fR
 does not return a token in these cases, it does all the required
 cleanup and there is no issue of your app having to call
@@ -768,13 +838,12 @@ HTTP reply headers or data, no exception is thrown.  This is because
 after writing the HTTP headers, \fB::http::geturl\fR returns, and the
 rest of the HTTP transaction occurs in the background.  The command
 callback can check if any error occurred during the read by calling
-\fB::http::status\fR to check the status and if its \fIerror\fR,
-calling \fB::http::error\fR to get the error message.
+\fB::http::responseInfo\fR to check the transaction status.
 .PP
 Alternatively, if the main program flow reaches a point where it needs
 to know the result of the asynchronous HTTP request, it can call
 \fB::http::wait\fR and then check status and error, just as the
-callback does.
+synchronous call does.
 .PP
 The \fB::http::geturl\fR command runs the \fB\-command\fR, \fB\-handler\fR,
 and \fB\-proxyfilter\fR callbacks inside a \fBcatch\fR command.  Therefore
@@ -788,15 +857,17 @@ In any case, you must still call
 \fB::http::cleanup\fR to delete the state array when you are done.
 .PP
 There are other possible results of the HTTP transaction
-determined by examining the status from \fB::http::status\fR.
+determined by examining the status from \fB::http::status\fR (or the value
+of the \fIstatus\fR key in the dictionary returned
+by \fB::http::responseInfo\fR).
 These are described below.
 .TP
 \fBok\fR
 .
 If the HTTP transaction completes entirely, then status will be \fBok\fR.
-However, you should still check the \fB::http::code\fR value to get
-the HTTP status.  The \fB::http::ncode\fR procedure provides just
-the numeric error (e.g., 200, 404 or 500) while the \fB::http::code\fR
+However, you should still check the \fB::http::responseLine\fR value to get
+the HTTP status.  The \fB::http::responseCode\fR procedure provides just
+the numeric error (e.g., 200, 404 or 500) while the \fB::http::responseLine\fR
 procedure returns a value like
 .QW "HTTP 404 File not found" .
 .TP
@@ -807,32 +878,47 @@ is raised, but the status of the transaction will be \fBeof\fR.
 .TP
 \fBerror\fR
 .
-The error message will also be stored in the \fBerror\fR status
-array element, accessible via \fB::http::error\fR.
+The error message, stack trace, and error code are accessible
+via \fB::http::error\fR.  The error message is also provided by the value of
+the \fIerror\fR key in the dictionary returned by \fB::http::responseInfo\fR.
 .TP
 \fBtimeout\fR
 .
-A timeout occurred before the transaction could complete
+A timeout occurred before the transaction could complete.
 .TP
 \fBreset\fR
 .
-user-reset
-.PP
-Another error possibility is that \fB::http::geturl\fR is unable to
-write all the post query data to the server before the server
-responds and closes the socket.
-The error message is saved in the \fBposterror\fR status array
-element and then  \fB::http::geturl\fR attempts to complete the
-transaction.
-If it can read the server's response
-it will end up with an \fBok\fR status, otherwise it will have
-an \fBeof\fR status.
+The user has called \fB::http::reset\fR.
+.TP
+\fB""\fR
+.
+(empty string) The transaction has not yet finished.
+.PP
+Another error possibility is that \fB::http::geturl\fR failed to
+write the whole of the POST request body (\fB-query\fR or \fB-querychannel\fR
+data) to the server.  \fB::http::geturl\fR stores the error message for later
+retrieval by the \fB::http::postError\fR or \fB::http::responseInfo\fR
+commands, and then attempts to complete the transaction.
+If it can read the server's response the status will be \fBok\fR, but it is
+important to call \fB::http::postError\fR or \fB::http::responseInfo\fR after
+every POST to check that the data was sent in full.
+If the server has closed the connection the status will be \fBeof\fR.
 .SH "METADATA"
-When a HTTP server responds to a request, it supplies not only the entity requested, but also metadata.  This is provided by the first line (the "status line") of the response, and by a number of HTTP headers.  Further metadata relates to how \fB::http::geturl\fR has processed the response from the server.
+.PP
+.SS "MOST USEFUL METADATA"
+When a HTTP server responds to a request, it supplies not only the entity
+requested, but also metadata.  This is provided by the first line (the
+"status line") of the response, and by a number of HTTP headers.  Further
+metadata relates to how \fB::http::geturl\fR has processed the response
+from the server.
 .PP
 The most important metadata can be accessed with the command
 \fB::http::responseInfo\fR.
-This command returns a \fBdict\fR of metadata that are essential for identifying a successful transaction and making use of the response, along with other metadata that are informational.  The keys of the \fBdict\fR are:
+This command returns a \fBdict\fR of metadata that are essential for
+identifying a successful transaction and making use of the response,
+along with other metadata that are informational.  The keys of
+the \fBdict\fR are:
+.PP
 .RS
 .RS
 \fB===== Essential Values =====\fR
@@ -841,220 +927,398 @@ This command returns a \fBdict\fR of metadata that are essential for identifying
 .TP
 \fBstage\fR
 .
-This value, set by \fB::http::geturl\fR, describes the stage that the transaction has reached. Values, in order of the transaction lifecycle, are: "created", "connecting", "header", "body", and "complete".  Other \fBdict\fR keys are available when the value of stage is "body" or "complete".  The key \fBconnection\fR has its final value only when \fBstage\fR is "complete".
+This value, set by \fB::http::geturl\fR, describes the stage that the
+transaction has reached. Values, in order of the transaction lifecycle,
+are: "created", "connecting", "header", "body", and "complete".  The
+other \fBdict\fR keys will not be available until the value of \fBstage\fR
+is "body" or "complete".  The key \fBcurrentSize\fR has its final value only
+when \fBstage\fR is "complete".
 .TP
 \fBstatus\fR
 .
-This value, set by \fB::http::geturl\fR, is "ok" for a successful transaction; "eof", "error", "timeout", or "reset" for an unsuccessful transaction; or "" if the transaction is still in progress.  In the last case the values for other dictionary keys may not be available.  The meaning of these values is described in the chapter \fBERRORS\fR (above).
+This value, set by \fB::http::geturl\fR, is "ok" for a successful transaction;
+"eof", "error", "timeout", or "reset" for an unsuccessful transaction; or ""
+if the transaction is still in progress.  The value is the same as that
+returned by command \fB::http::status\fR. The meaning of these values is
+described in the section \fBERRORS\fR (above).
 .TP
-\fBncode\fR
+\fBresponseCode\fR
 .
-The "HTTP status code" for the response.
+The "HTTP status code" sent by the server in the first line (the "status line")
+of the response.  If the value cannot be extracted from the status line, the
+full status line is returned.
 .TP
-\fBreason\fR
+\fBreasonPhrase\fR
 .
-The "reason phrase" sent by the server.
+The "reason phrase" sent by the server as a description of the HTTP status code.
+If the value cannot be extracted from the status line, the full status
+line is returned.
 .TP
-\fBcontent-type\fR
+\fBcontentType\fR
 .
-The value of the \fBContent-Type\fR response header or, if the header was not supplied, the default value "application/octet-stream".
+The value of the \fBContent-Type\fR response header or, if the header was not
+supplied, the default value "application/octet-stream".
 .TP
 \fBbinary\fR
 .
-This boolean value, set by \fB::http::geturl\fR, describes how the command has interpreted the entity returned by the server (after decoding any compression specified
-by the
-.QW "Content-Encoding"
-response header).  This decoded entity is accessible as the return value of the command \fB::http::responseBody\fR.
-
-The value is \fBtrue\fR if http has interpreted the decoded entity as binary. The value returned by ::http::responseBody is a Tcl binary string.  This is a suitable format for image data, zip files, etc.
-\fB::http::geturl\fR chooses this value if the user has requested a binary interpretation by passing the option \fI\-binary\fR to the command, or if the server has supplied a binary content type in a Content-Type response header, or if the server has not supplied any Content-Type header.
-
-The value is \fBfalse\fR if http has interpreted the decoded entity as text. The text has been converted from the character set notified by the server into Tcl's internal Unicode format, and the value returned by ::http::responseBody is an ordinary Tcl string.
-
-It is always worth checking the value of "binary" after a HTTP transaction, to determine whether a misconfigured server has caused http to interpret a text resource as a binary, or vice versa.
+This boolean value, set by \fB::http::geturl\fR, describes how the command
+has interpreted the entity returned by the server (after decoding any
+compression specified by the \fBContent-Encoding\fR response header).
+This decoded entity is accessible as the return value of the
+command \fB::http::responseBody\fR.
+.PP
+.RS
+The value is \fBtrue\fR if http has interpreted the decoded entity as binary.
+The value returned by \fB::http::responseBody\fR is a Tcl binary string.
+This is a suitable format for image data, zip files, etc.
+\fB::http::geturl\fR chooses this value if the user has requested a binary
+interpretation by passing the option \fI\-binary\fR to the command, or if the
+server has supplied a binary content type in a \fBContent-Type\fR response
+header, or if the server has not supplied any \fBContent-Type\fR header.
+.PP
+The value is \fBfalse\fR in other cases, and this means that http has
+interpreted the decoded entity as text. The text has been converted, from the
+character set notified by the server, into Tcl's internal Unicode format;
+the value returned by \fB::http::responseBody\fR is an ordinary Tcl string.
+.PP
+It is always worth checking the value of "binary" after a HTTP transaction,
+to determine whether a misconfigured server has caused http to interpret a
+text resource as a binary, or vice versa.
+.RE
 .TP
 \fBredirection\fR
 .
-The URL that is the redirection target.  The value is that of the Location response header.  This header is sent when a response has status code 3XX (redirection).
+The URL that is the redirection target. The value is that of the \fBLocation\fR
+response header.  This header is sent when a response has status code
+3XX (redirection).
+.TP
+\fBupgrade\fR
+.
+If not empty, the value indicates the protocol(s) to which the server will
+switch after completion of this transaction, while continuing to use the
+same connection.  When the server intends to switch protocols, it will also
+send the value "101" as the status code (the \fBresponseCode\fR key), and the
+word "upgrade" as an element of the \fBConnection\fR response header (the
+\fBconnectionResponse\fR key), and it will not send a response body.
+See the section \fBPROTOCOL UPGRADES\fR for more information.
+.TP
+\fBerror\fR
+.
+The error message, if there is one.  Further information, including a stack
+trace and error code, are available from command \fB::http::error\fR.
+.TP
+\fBpostError\fR
+.
+The error message (if any) generated when a HTTP POST request sends its
+request-body to the server.  Further information, including a stack trace
+and error code, are available from command \fB::http::postError\fR.  A POST
+transaction may appear complete, according to the
+keys \fBstage\fR, \fBstatus\fR, and \fBresponseCode\fR, but it is important
+to check this \fBpostError\fR key in case an error occurred when uploading
+the request-body.
+.PP
 .RS
 .RS
 \fB===== Informational Values =====\fR
 .RE
 .RE
 .TP
+\fBmethod\fR
+.
+The HTTP method used in the request.
+.TP
 \fBcharset\fR
 .
-The value of the charset attribute of the \fBContent-Type\fR response header.  The charset value is used only for a text resource.  If the server did not specify a charset, the value defaults to that of the variable \fB::http::defaultCharset\fR, which unless it has been deliberately modified by the caller is \fBiso8859-1\fR.  Incoming text data is automatically converted from the character set defined by \fBcharset\fR to Tcl's internal Unicode representation, i.e. to a Tcl string.
+The value of the charset attribute of the \fBContent-Type\fR response header.
+The charset value is used only for a text resource.  If the server did not
+specify a charset, the value defaults to that of the
+variable \fB::http::defaultCharset\fR, which unless it has been deliberately
+modified by the caller is \fBiso8859-1\fR.  Incoming text data is automatically
+converted from the character set defined by \fBcharset\fR to Tcl's internal
+Unicode representation, i.e. to a Tcl string.
 .TP
-\fBcoding\fR
+\fBcompression\fR
 .
 A copy of the \fBContent-Encoding\fR response-header value.
 .TP
 \fBhttpRequest\fR
 .
 The version of HTTP specified in the request (i.e. sent in the request line).
+The value is that of the option \fB\-protocol\fR supplied
+to \fB::http::geturl\fR (default value "1.1"), unless the command reduced the
+value to "1.0" because it was passed the \fB\-handler\fR option.
 .TP
 \fBhttpResponse\fR
 .
-The version of HTTP used by the server (obtained from the response "status line").  The server uses this version of HTTP in its response, but ensures that this response is compatible with the HTTP version specified in the client's request.
+The version of HTTP used by the server (obtained from the response
+"status line").  The server uses this version of HTTP in its response, but
+ensures that this response is compatible with the HTTP version specified in the
+client's request.  If the value cannot be extracted from the status line, the
+full status line is returned.
 .TP
 \fBurl\fR
 .
-The requested URL, typically the URL supplied as an argument to \fB::http::geturl\fR but without its "fragment" (the final part of the URL beginning with "#".
+The requested URL, typically the URL supplied as an argument
+to \fB::http::geturl\fR but without its "fragment" (the final part of the URL
+beginning with "#").
 .TP
-\fBconnRequest\fR
+\fBconnectionRequest\fR
 .
-The value, if any, sent to the server in "Connection" request header(s).
+The value, if any, sent to the server in \fBConnection\fR request header(s).
 .TP
-\fBconnResponse\fR
+\fBconnectionResponse\fR
 .
-The value, if any, received from the server in "Connection" response header(s).
+The value, if any, received from the server in \fBConnection\fR response
+header(s).
 .TP
-\fBconnection\fR
+\fBconnectionActual\fR
 .
-This value, set by \fB::http::geturl\fR, reports whether the connection was closed after the transaction (value "close"), or left open (value "keep-alive").
+This value, set by \fB::http::geturl\fR, reports whether the connection was
+closed after the transaction (value "close"), or left open (value "keep-alive").
 .TP
-\fBtransfer\fR
+\fBtransferEncoding\fR
 .
-The value of the Transfer-Encoding response header, if it is present.  The value is either "chunked" (indicating HTTP/1.1 "chunked encoding") or the empty string.
+The value of the Transfer-Encoding response header, if it is present.
+The value is either "chunked" (indicating HTTP/1.1 "chunked encoding") or
+the empty string.
 .TP
-\fBquerylength\fR
+\fBtotalPost\fR
 .
 The total length of the request body in a POST request.
 .TP
-\fBqueryoffset\fR
+\fBcurrentPost\fR
 .
 The number of bytes of the POST request body sent to the server so far.
+The value is the same as that returned by command \fB::http::size\fR.
 .TP
-\fBtotalsize\fR
+\fBtotalSize\fR
 .
 A copy of the \fBContent-Length\fR response-header value.
-The number of bytes specified in a Content-Length header, if one was sent.  If none was sent, the value is 0.  A correctly configured server omits this header if the transfer-encoding is "chunked", or (for older servers) if the server closes the connection when it reaches the end of the resource.
+The number of bytes specified in a \fBContent-Length\fR header, if one
+was sent.  If none was sent, the value is 0.  A correctly configured server
+omits this header if the transfer-encoding is "chunked", or (for older
+servers) if the server closes the connection when it reaches the end of
+the resource.
 .TP
-\fBcurrentsize\fR
+\fBcurrentSize\fR
 .
 The number of bytes fetched from the server so far.
+.PP
+.SS "MORE METADATA"
+The dictionary returned by \fB::http::responseInfo\fR is the most useful
+subset of the available metadata.  Other metadata include:
+.PP
+1. The full "status line" of the response, available as the return value
+of command \fB::http::responseLine\fR.
+.PP
+2. The full response headers, available as the return value of
+command \fB::http::responseHeaders\fR.  This return value is a list of the
+response-header names and values, in the order that they were received from
+the server.
+.PP
+The return value is not a \fBdict\fR because some header names may
+occur more than once, notably \fBSet-Cookie\fR. If the value is read
+into a \fBdict\fR or into an array (using array set), only the last header
+with each name will be preserved.
+.PP
+.RS
+Some of the header names (metadata keys) are listed below, but the HTTP
+standard defines several more, and servers are free to add their own.
+When a dictionary key is mentioned below, this refers to the \fBdict\fR
+value returned by command \fB::http::responseInfo\fR.
+.TP
+\fBContent-Type\fR
+.
+The content type of the URL contents.  Examples include \fBtext/html\fR,
+\fBimage/gif,\fR \fBapplication/postscript\fR and
+\fBapplication/x-tcl\fR.  Text values typically specify a character set, e.g.
+\fBtext/html; charset=UTF-8\fR.  Dictionary key \fIcontentType\fR.
+.TP
+\fBContent-Length\fR
+.
+The advertised size in bytes of the contents, available as dictionary
+key \fItotalSize\fR.  The actual number of bytes read by \fB::http::geturl\fR
+so far is available as dictionary key \fBcurrentSize\fR.
+.TP
+\fBContent-Encoding\fR
+.
+The compression algorithm used for the contents.
+Examples include \fBgzip\fR, \fBdeflate\fR.
+Dictionary key \fIcontent\fR.
+.TP
+\fBLocation\fR
+.
+This header is sent when a response has status code 3XX (redirection).
+It provides the URL that is the redirection target.
+Dictionary key \fIredirection\fR.
+.TP
+\fBSet-Cookie\fR
+.
+This header is sent to offer a cookie to the client.  Cookie management is
+done by the \fB::http::config\fR option \fI\-cookiejar\fR, and so
+the \fBSet-Cookie\fR headers need not be parsed by user scripts.
+See section \fBCOOKIE JAR PROTOCOL\fR.
+.TP
+\fBConnection\fR
+.
+The value can be supplied as a comma-separated list, or by multiple headers.
+The list often has only one element, either "close" or "keep-alive".
+The value "upgrade" indicates a successful upgrade request and is typically
+combined with the status code 101, an \fBUpgrade\fR response header, and no
+response body.  Dictionary key \fIconnectionResponse\fR.
+.TP
+\fBUpgrade\fR
+.
+The value indicates the protocol(s) to which the server will switch
+immediately after the empty line that terminates the 101 response headers.
+Dictionary key \fIupgrade\fR.
+.RE
+.PP
+.SS "EVEN MORE METADATA"
+.PP
+1. Details of the HTTP request.  The request is determined by the options
+supplied to \fB::http::geturl\fR and \fB::http::config\fR.  However, it is
+sometimes helpful to examine what \fB::http::geturl\fR actually sent to the
+server, and this information is available through
+commands \fB::http::requestHeaders\fR and \fB::http::requestLine\fR.
+.PP
+2. The state array: the internal variables of \fB::http::geturl\fR.
+It may sometimes be helpful to examine this array.
+Details are given in the next section.
 .SH "STATE ARRAY"
-The \fB::http::geturl\fR procedure returns a \fItoken\fR that can be used to
-get to the state of the HTTP transaction in the form of a Tcl array.
-Use this construct to create an easy-to-use array variable:
+The \fB::http::geturl\fR procedure returns a \fItoken\fR that can be used
+as an argument to other \fB::http::*\fR commands, which examine and manage
+the state of the HTTP transaction.  For most purposes these commands are
+sufficient.  The \fItoken\fR can also be used to access
+the internal state of the transaction, which is stored in a Tcl array.
+This facility is most useful when writing callback commands for the
+options \fB\-command\fR, \fB\-handler\fR, \fB\-progress\fR,
+or \fB\-queryprogress\fR.
+Use the following command inside the proc to define an easy-to-use
+array \fIstate\fR as a local variable within the proc
 .PP
 .CS
-upvar #0 $token state
+upvar 0 $token state
 .CE
 .PP
 Once the data associated with the URL is no longer needed, the state
 array should be unset to free up storage.
 The \fB::http::cleanup\fR procedure is provided for that purpose.
-The following elements of
-the array are supported:
+.PP
+The following elements of the array are supported, and are the origin of the
+values returned by commands as described below.  When a dictionary key is
+mentioned below, this refers to the \fBdict\fR value returned by
+command \fB::http::responseInfo\fR.
 .RS
 .TP
 \fBbinary\fR
 .
-This is boolean \fBtrue\fR if (after decoding any compression specified
-by the
-.QW "Content-Encoding"
-response header) the HTTP response is binary.  It is boolean \fBfalse\fR
-if the HTTP response is text.
+For dictionary key \fIbinary\fR.
 .TP
 \fBbody\fR
 .
-The contents of the URL.  This will be empty if the \fB\-channel\fR
-option has been specified.  This value is returned by the \fB::http::data\fR
-command.
+For command \fB::http::responseBody\fR.
 .TP
 \fBcharset\fR
 .
-The value of the charset attribute from the \fBContent-Type\fR metadata
-value.  If none was specified, this defaults to the RFC standard
-\fBiso8859-1\fR, or the value of \fB$::http::defaultCharset\fR.  Incoming
-text data will be automatically converted from this charset to utf-8.
+For dictionary key \fIcharset\fR.
 .TP
 \fBcoding\fR
 .
-A copy of the \fBContent-Encoding\fR metadata value.
+For dictionary key \fIcompression\fR.
+.TP
+\fBconnection\fR
+.
+For dictionary key \fIconnectionActual\fR.
 .TP
 \fBcurrentsize\fR
 .
-The current number of bytes fetched from the URL.
-This value is returned by the \fB::http::size\fR command.
+For command \fB::http::size\fR; and for dictionary key \fIcurrentSize\fR.
 .TP
 \fBerror\fR
 .
-If defined, this is the error string seen when the HTTP transaction
-was aborted.
+For command \fB::http::error\fR; part is used in dictionary key \fIerror\fR.
 .TP
 \fBhttp\fR
 .
-The HTTP status reply from the server.  This value
-is returned by the \fB::http::code\fR command.  The format of this value is:
-.RS
-.PP
-.CS
-\fIHTTP/1.1 code string\fR
-.CE
-.PP
-The \fIcode\fR is a three-digit number defined in the HTTP standard.
-A code of 200 is OK.  Codes beginning with 4 or 5 indicate errors.
-Codes beginning with 3 are redirection errors.  In this case the
-\fBLocation\fR metadata specifies a new URL that contains the
-requested information.
-.RE
+For command \fB::http::responseLine\fR.
+.TP
+\fBhttpResponse\fR
+.
+For dictionary key \fIhttpResponse\fR.
 .TP
 \fBmeta\fR
 .
-The response from a HTTP server includes metadata headers that describe the
-response body and the message from the server.  The \fBmeta\fR element of the
-state array is a list of the keys (header names) and values (header values) of
-the metadata.  Header names are case-insensitive and are converted to lower
-case.  The value of meta is not a \fBdict\fR because some header names may
-occur more than once, notably "set-cookie".  If the value \fBmeta\fR is read
-into a dict or into an array (using array set), only the last header with each
-name will be preserved.
-.PP
-.RS
-Some of the metadata keys are listed below, but the HTTP standard defines
-more, and servers are free to add their own.
+For command \fB::http::responseHeaders\fR. Further discussion above in the
+section \fBMORE METADATA\fR.
 .TP
-\fBContent-Type\fR
+\fBmethod\fR
 .
-The type of the URL contents.  Examples include \fBtext/html\fR,
-\fBimage/gif,\fR \fBapplication/postscript\fR and
-\fBapplication/x-tcl\fR.
+For dictionary key \fImethod\fR.
 .TP
-\fBContent-Length\fR
+\fBposterror\fR
 .
-The advertised size of the contents.  The actual size obtained by
-\fB::http::geturl\fR is available as \fBstate(currentsize)\fR.
+For dictionary key \fIpostError\fR.
 .TP
-\fBLocation\fR
+\fBpostErrorFull\fR
 .
-An alternate URL that contains the requested data.
-.RE
+For command \fB::http::postError\fR.
 .TP
-\fBposterror\fR
+\fB\-protocol\fR
 .
-The error, if any, that occurred while writing
-the post query data to the server.
+For dictionary key \fIhttpRequest\fR.
+.TP
+\fBquerylength\fR
+.
+For dictionary key \fItotalPost\fR.
+.TP
+\fBqueryoffset\fR
+.
+For dictionary key \fIcurrentPost\fR.
+.TP
+\fBreasonPhrase\fR
+.
+For dictionary key \fIreasonPhrase\fR.
+.TP
+\fBrequestHeaders\fR
+.
+For command \fB::http::requestHeaders\fR.
+.TP
+\fBrequestLine\fR
+.
+For command \fB::http::requestLine\fR.
+.TP
+\fBresponseCode\fR
+.
+For dictionary key \fIresponseCode\fR.
+.TP
+\fBstate\fR
+.
+For dictionary key \fIstage\fR.
 .TP
 \fBstatus\fR
 .
-See description in the chapter \fBERRORS\fR above for a
-list and description of \fBstatus\fR.
-During the transaction this value is the empty string.
+For command \fB::http::status\fR; and for dictionary key \fIstatus\fR.
 .TP
 \fBtotalsize\fR
 .
-A copy of the \fBContent-Length\fR metadata value.
+For dictionary key \fItotalSize\fR.
+.TP
+\fBtransfer\fR
+.
+For dictionary key \fItransferEncoding\fR.
 .TP
 \fBtype\fR
 .
-A copy of the \fBContent-Type\fR metadata value.
+For dictionary key \fIcontentType\fR.
+.TP
+\fBupgrade\fR
+.
+For dictionary key \fIupgrade\fR.
 .TP
 \fBurl\fR
 .
-The requested URL.
+For dictionary key \fIurl\fR.
 .RE
 .SH "PERSISTENT CONNECTIONS"
 .PP
@@ -1153,7 +1417,7 @@ that fails because it uses a persistent connection that the server has
 half-closed (an
 .QW "asynchronous close event" ).
 Subsequent GET and HEAD requests in a failed pipeline will also be retried.
-\fIThe \-repost option should be used only if the application understands
+\fIThe \fB\-repost\fI option should be used only if the application understands
 that the retry is appropriate\fR - specifically, the application must know
 that if the failed POST successfully modified the state of the server, a repeat
 POST would have no adverse effect.
@@ -1261,22 +1525,25 @@ Other keys may always be ignored; they have no meaning in this protocol.
 .VE TIP406
 .SH "PROTOCOL UPGRADES"
 .PP
-The HTTP/1.1 \fBConnection\fR and \fBUpgrade\fR client headers inform the server
-that the client wishes to change the protocol used over the existing connection
-(RFC 7230).  This mechanism can be used to request a WebSocket (RFC 6455), a
+The HTTP/1.1 \fBConnection\fR and \fBUpgrade\fR request headers inform the
+server that the client wishes to change the protocol used over the existing
+connection (RFC 7230).
+This mechanism can be used to request a WebSocket (RFC 6455), a
 higher version of the HTTP protocol (HTTP 2), or TLS encryption.  If the
 server accepts the upgrade request, its response code will be 101.
 .PP
-To request a protocol upgrade when calling \fBhttp::geturl\fR, the \fB\-headers\fR
-option must supply appropriate values for \fBConnection\fR and \fBUpgrade\fR, and
+To request a protocol upgrade when calling \fBhttp::geturl\fR,
+the \fB\-headers\fR option must supply appropriate values for \fBConnection\fR
+and \fBUpgrade\fR, and
 the \fB\-command\fR option must supply a command that implements the requested
 protocol and can also handle the server response if the server refuses the
 protocol upgrade.  For upgrade requests \fBhttp::geturl\fR ignores the value of
 option \fB\-keepalive\fR, and always uses the value \fB0\fR so that the upgrade
-request is not made over a connection that is intended for multiple HTTP requests.
+request is not made over a connection that is intended for multiple HTTP
+requests.
 .PP
-The Tcllib library \fBwebsocket\fR implements WebSockets, and makes the necessary
-calls to commands in the \fBhttp\fR package.
+The Tcllib library \fBwebsocket\fR implements WebSockets, and makes the
+necessary calls to commands in the \fBhttp\fR package.
 .PP
 There is currently no native Tcl client library for HTTP/2.
 .PP
@@ -1287,30 +1554,59 @@ protocols such as Internet Printing Protocol (IPP) that are built on top of
 traffic.
 .PP
 In browsers, opportunistic encryption is instead implemented by the
-\fBUpgrade-Insecure-Requests\fR client header.  If a secure service is available,
-the server response code is a 307 redirect, and the response header
-\fBLocation\fR specifies the target URL.  The browser must call \fBhttp::geturl\fR
-again in order to fetch this URL.
+\fBUpgrade-Insecure-Requests\fR client header.  If a secure service is
+available, the server response code is a 307 redirect, and the response header
+\fBLocation\fR specifies the target URL.  The browser must
+call \fBhttp::geturl\fR again in order to fetch this URL.
 See https://w3c.github.io/webappsec-upgrade-insecure-requests/
 .PP
 .SH THREADS
 .PP
 .SS "PURPOSE"
 .PP
-Command \fB::http::geturl\fR uses the Tcl \fB::socket\fR command with the \-async option to connect to a remote server, but the return from this command can be delayed in adverse cases (e.g. a slow DNS lookup), preventing the event loop from processing other events.  This delay is avoided if the \fB::socket\fR command is evaluated in another thread.  The Thread package is not part of Tcl but is provided in "Batteries Included" distributions.  Instead of the \fB::socket\fR command, the http package uses \fB::http::socket\fR which makes connections in the manner specified by the value of \-threadlevel and the availability of package Thread.
+Command \fB::http::geturl\fR uses the Tcl \fB::socket\fR command with
+the \fI\-async\fR option to connect to a remote server, but the return from
+this command can be delayed in adverse cases (e.g. a slow DNS lookup),
+preventing the event loop from processing other events.
+This delay is avoided if the \fB::socket\fR command is evaluated in another
+thread.  The Thread package is not part of Tcl but is provided in
+"Batteries Included" distributions.  Instead of the \fB::socket\fR command,
+the http package uses \fB::http::socket\fR which makes connections in the
+manner specified by the value of \fI\-threadlevel\fR and the availability
+of package Thread.
 .PP
 .SS "WITH TLS (HTTPS)"
 .PP
-The same \-threadlevel configuration applies to both HTTP and HTTPS connections. HTTPS is enabled by using the \fBhttp::register\fR command, typically by specifying the \fB::tls::socket\fR command of the tls package to handle TLS cryptography.  The \fB::tls::socket\fR command connects to the remote server by using the command specified by the value of variable \fI::tls::socketCmd\fR, and this value defaults to "::socket".  If http::geturl finds that \fI::tls::socketCmd\fR has this value, it replaces it with the value "::http::socket".  If \fI::tls::socketCmd\fR has a value other than "::socket", i.e. if the script or the Tcl installation has replaced the value "::socket" with the name of a different command, then http does not change the value.  The script or installation that modified \fI::tls::socketCmd\fR is responsible for integrating \fR::http::socket\fR into its own replacement command.
+The same \fI\-threadlevel\fR configuration applies to both HTTP and HTTPS
+connections.
+HTTPS is enabled by using the \fBhttp::register\fR command, typically by
+specifying the \fB::tls::socket\fR command of the tls package to handle TLS
+cryptography.  The \fB::tls::socket\fR command connects to the remote server by
+using the command specified by the value of variable \fI::tls::socketCmd\fR, and
+this value defaults to "::socket".  If http::geturl finds
+that \fI::tls::socketCmd\fR has this value, it replaces it with the value
+"::http::socket".  If \fI::tls::socketCmd\fR has a value other than "::socket",
+i.e. if the script or the Tcl installation has replaced the value "::socket"
+with the name of a different command, then http does not change the value.
+The script or installation that modified \fI::tls::socketCmd\fR is responsible
+for integrating \fR::http::socket\fR into its own replacement command.
 .PP
 .SS "WITH A CHILD INTERPRETER"
 .PP
-The peer thread can transfer the socket only to the main interpreter of the script's thread.  Therefore the thread-based \fB::http::socket\fR works with non-zero \-threadlevel values only if the script runs in the main interpreter.  A child interpreter must use \-threadlevel 0 unless the parent interpreter has provided alternative facilities.  The main parent interpreter may grant full \-threadlevel facilities to a child interpreter, for example by aliasing, to \fB::http::socket\fR in the child, a command that runs \fBhttp::socket\fR in the parent, and then transfers the socket to the child.
+The peer thread can transfer the socket only to the main interpreter of the 
+script's thread.  Therefore the thread-based \fB::http::socket\fR works with
+non-zero \fI\-threadlevel\fR values only if the script runs in the main
+interpreter.  A child interpreter must use \fI\-threadlevel 0\fR unless the
+parent interpreter has provided alternative facilities.  The main parent
+interpreter may grant full \fI\-threadlevel\fR facilities to a child
+interpreter, for example by aliasing, to \fB::http::socket\fR in the child,
+a command that runs \fBhttp::socket\fR in the parent, and then transfers
+the socket to the child.
 .PP
 .SH EXAMPLE
 .PP
 This example creates a procedure to copy a URL to a file while printing a
-progress meter, and prints the metadata associated with the URL.
+progress meter, and prints the response headers associated with the URL.
 .PP
 .CS
 proc httpcopy { url file {chunk 4096} } {
@@ -1322,7 +1618,7 @@ proc httpcopy { url file {chunk 4096} } {
     # This ends the line started by httpCopyProgress
     puts stderr ""
 
-    upvar #0 $token state
+    upvar 0 $token state
     set max 0
     foreach {name value} $state(meta) {
         if {[string length $name] > $max} {