HTTP.request(method, url [, headers [, body]]; <keyword arguments>]) -> HTTP.Response

Send a HTTP Request Message and recieve a HTTP Response Message.

e.g.

r = HTTP.request("GET", "http://httpbin.org/ip")
println(r.status)
println(String(r.body))

headers can be any collection where [string(k) => string(v) for (k,v) in headers] yields Vector{Pair}. e.g. a Dict(), a Vector{Tuple}, a Vector{Pair} or an iterator.

body can take a number of forms:

• a String, a Vector{UInt8} or any T accepted by write(::IO, ::T)

• a collection of String or AbstractVector{UInt8} or IO streams or items of any type T accepted by write(::IO, ::T...)

• a readable IO stream or any IO-like type T for which eof(T) and readavailable(T) are defined.

The HTTP.Response struct contains:

• status::Int16 e.g. 200

• headers::Vector{Pair{String,String}} e.g. ["Server" => "Apache", "Content-Type" => "text/html"]

• body::Vector{UInt8}, the Response Body bytes (empty if a response_stream was specified in the request).

Functions HTTP.get, HTTP.put, HTTP.post and HTTP.head are defined as shorthand for HTTP.request("GET", ...), etc.

HTTP.request and HTTP.open also accept optional keyword parameters.

e.g.

HTTP.request("GET", "http://httpbin.org/ip"; retries=4, cookies=true)

HTTP.get("http://s3.us-east-1.amazonaws.com/"; aws_authorization=true)

conf = (readtimeout = 10,
        pipeline_limit = 4,
        retry = false,
        redirect = false)

HTTP.get("http://httpbin.org/ip"; conf..)
HTTP.put("http://httpbin.org/put", [], "Hello"; conf..)

Streaming options

• response_stream = nothing, a writeable IO stream or any IO-like type T for which write(T, AbstractVector{UInt8}) is defined.

• verbose = 0, set to 1 or 2 for extra message logging.

Connection Pool options

• connection_limit = 8, number of concurrent connections to each host:port.

• pipeline_limit = 16, number of concurrent requests per connection.

• reuse_limit = nolimit, number of times a connection is reused after the first request.

• socket_type = TCPSocket

Timeout options

• readtimeout = 60, close the connection if no data is recieved for this many seconds. Use readtimeout = 0 to disable.

Retry options

• retry = true, retry idempotent requests in case of error.

• retries = 4, number of times to retry.

• retry_non_idempotent = false, retry non-idempotent requests too. e.g. POST.

Redirect options

• redirect = true, follow 3xx redirect responses.

• redirect_limit = 3, number of times to redirect.

• forwardheaders = false, forward original headers on redirect.

Status Exception options

• statusexception = true, throw HTTP.StatusError for response status >= 300.

SSLContext options

• require_ssl_verification = false, pass MBEDTLS_SSL_VERIFY_REQUIRED to the mbed TLS library. "... peer must present a valid certificate, handshake is aborted if verification failed."

• sslconfig = SSLConfig(require_ssl_verification)`

Basic Authenticaiton options

• basicauthorization=false, add Authorization: Basic header using credentials from url userinfo.

AWS Authenticaiton options

• awsauthorization = false, enable AWS4 Authentication.

• aws_service = split(uri.host, ".")[1]

• aws_region = split(uri.host, ".")[2]

• aws_access_key_id = ENV["AWS_ACCESS_KEY_ID"]

• aws_secret_access_key = ENV["AWS_SECRET_ACCESS_KEY"]

• aws_session_token = get(ENV, "AWS_SESSION_TOKEN", "")

• body_sha256 = digest(MD_SHA256, body),

• body_md5 = digest(MD_MD5, body),

Cookie options

• cookies = false, enable cookies.

• cookiejar::Dict{String, Set{Cookie}}=default_cookiejar

Cananoincalization options

• canonicalizeheaders = false, rewrite request and response headers in Canonical-Camel-Dash-Format.

Request Body Examples

String body:

request("POST", "http://httpbin.org/post", [], "post body data")

Stream body from file:

io = open("post_data.txt", "r")
request("POST", "http://httpbin.org/post", [], io)

Generator body:

chunks = ("chunk$i" for i in 1:1000)
request("POST", "http://httpbin.org/post", [], chunks)

Collection body:

chunks = [preamble_chunk, data_chunk, checksum(data_chunk)]
request("POST", "http://httpbin.org/post", [], chunks)

open() do io body:

HTTP.open("POST", "http://httpbin.org/post") do io
    write(io, preamble_chunk)
    write(io, data_chunk)
    write(io, checksum(data_chunk))
end

Response Body Examples

String body:

r = request("GET", "http://httpbin.org/get")
println(String(r.body))

Stream body to file:

io = open("get_data.txt", "w")
r = request("GET", "http://httpbin.org/get", response_stream=io)
close(io)
println(read("get_data.txt"))

Stream body through buffer:

io = BufferStream()
@async while !eof(io)
    bytes = readavailable(io))
    println("GET data: $bytes")
end
r = request("GET", "http://httpbin.org/get", response_stream=io)
close(io)

Stream body through open() do io:

r = HTTP.open("GET", "http://httpbin.org/stream/10") do io
   while !eof(io)
       println(String(readavailable(io)))
   end
end

HTTP.open("GET", "https://tinyurl.com/bach-cello-suite-1-ogg") do http
    n = 0
    r = startread(http)
    l = parse(Int, header(r, "Content-Length"))
    open(`vlc -q --play-and-exit --intf dummy -`, "w") do vlc
        while !eof(http)
            bytes = readavailable(http)
            write(vlc, bytes)
            n += length(bytes)
            println("streamed $n-bytes $((100*n)÷l)%\u1b[1A")
        end
    end
end

Request and Response Body Examples

String bodies:

r = request("POST", "http://httpbin.org/post", [], "post body data")
println(String(r.body))

Stream bodies from and to files:

in = open("foo.png", "r")
out = open("foo.jpg", "w")
request("POST", "http://convert.com/png2jpg", [], in, response_stream=out)

Stream bodies through: open() do io:

HTTP.open("POST", "http://music.com/play") do io
    write(io, JSON.json([
        "auth" => "12345XXXX",
        "song_id" => 7,
    ]))
    r = readresponse(io)
    @show r.status
    while !eof(io)
        bytes = readavailable(io))
        play_audio(bytes)
    end
end

HTTP.open(method, url, [,headers]) do io
    write(io, body)
    [startread(io) -> HTTP.Response]
    while !eof(io)
        readavailable(io) -> AbstractVector{UInt8}
    end
end -> HTTP.Response

The HTTP.open API allows the Request Body to be written to (and/or the Response Body to be read from) an IO stream.

e.g. Streaming an audio file to the vlc player:

HTTP.open("GET", "https://tinyurl.com/bach-cello-suite-1-ogg") do http
    open(`vlc -q --play-and-exit --intf dummy -`, "w") do vlc
        write(vlc, http)
    end
end

HTTP.get(url [, headers]; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("GET", ...). See HTTP.request.

HTTP.put(url, headers, body; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("PUT", ...). See HTTP.request.

HTTP.post(url, headers, body; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("POST", ...). See HTTP.request.

HTTP.head(url; <keyword arguments>) -> HTTP.Response

Shorthand for HTTP.request("HEAD", ...). See HTTP.request.

The Response has a 4xx, 5xx or unrecognised status code.

Fields:

• status::Int16, the response status code.

• response the HTTP.Response

The [Parser] input was invalid.

Fields:

• code, internal @enum ParsingErrorCode.

• state, internal parsing state.

• status::Int32, HTTP response status.

• msg::String, error message.

The request terminated with due to an IO-related error.

Fields:

• e, the error.

HTTP.serve([server,] host::IPAddr, port::Int; verbose::Bool=true, kwargs...)

Start a server listening on the provided host and port. verbose indicates whether server activity should be logged. Optional keyword arguments allow construction of Server on the fly if the server argument isn't provided directly. See ?HTTP.Server for more details on server construction and supported keyword arguments. By default, HTTP.serve aims to "never die", catching and recovering from all internal errors. Two methods for stopping HTTP.serve include interrupting (ctrl/cmd+c) if blocking on the main task, or sending the kill signal via the server's in channel (put!(server.in, HTTP.KILL)).

Server(handler, logger::IO=STDOUT; kwargs...)

An http/https server. Supports listening on a host and port via the HTTP.serve(server, host, port) function. handler is a function of the form f(::Request, ::Response) -> HTTP.Response, i.e. it takes both a Request and pre-built Response objects as inputs and returns the, potentially modified, Response. logger indicates where logging output should be directed. When HTTP.serve is called, it aims to "never die", catching and recovering from all internal errors. To forcefully stop, one can obviously kill the julia process, interrupt (ctrl/cmd+c) if main task, or send the kill signal over a server in channel like: put!(server.in, HTTP.KILL).

Supported keyword arguments include:

• cert: if https, the cert file to use, as passed to HTTP.MbedTLS.SSLConfig(cert, key)

• key: if https, the key file to use, as passed to HTTP.MbedTLS.SSLConfig(cert, key)

• tlsconfig: pass in an already-constructed HTTP.MbedTLS.SSLConfig instance

• readtimeout: how long a client connection will be left open without receiving any bytes

• ratelimit: a Rational{Int} of the form 5//1 indicating how many messages//second should be allowed per client IP address; requests exceeding the rate limit will be dropped

• support100continue: a Bool indicating whether Expect: 100-continue headers should be supported for delayed request body sending; default = true

• logbody: whether the Response body should be logged when verbose=true logging is enabled; default = true

Abstract type representing an object that knows how to "handle" a server request.

Types of handlers include HandlerFunction (a julia function of the form f(request, response) and Router (which pattern matches request url paths to other specific Handler types).

HandlerFunction(f::Function)

A Function-wrapper type that is a subtype of Handler. Takes a single Function as an argument. The provided argument should be of the form f(request, response) => Response, i.e. it accepts both a Request and Response and returns a Response.

Router(h::Handler) Router(f::Function) Router()

An HTTP.Handler type that supports mapping request url paths to other HTTP.Handler types. Can accept a default Handler or Function that will be used in case no other handlers match; by default, a 404 response handler is used. Paths can be mapped to a handler via HTTP.register!(r::Router, path, handler), see ?HTTP.register! for more details.

HTTP.register!(r::Router, url, handler) HTTP.register!(r::Router, m::Union{HTTP.Method, String}, url, handler)

Function to map request urls matching url and an optional method m to another handler::HTTP.Handler. URLs are registered one at a time, and multiple urls can map to the same handler. Methods can be passed as a string "GET" or enum object directly HTTP.GET. The URL can be passed as a String or HTTP.URI object directly. Requests can be routed based on: method, scheme, hostname, or path. The following examples show how various urls will direct how a request is routed by a server:

• "http://*": match all HTTP requests, regardless of path

• "https://*": match all HTTPS requests, regardless of path

• "google": regardless of scheme, match requests to the hostname "google"

• "google/gmail": match requests to hostname "google", and path starting with "gmail"

• "/gmail": regardless of scheme or host, match any request with a path starting with "gmail"

• "/gmail/userId/*/inbox: match any request matching the path pattern, "*" is used as a wildcard that matches any value between the two "/"

HTTP.URL(host; userinfo="", path="", query="", fragment="", isconnect=false)
HTTP.URI(; scheme="", host="", port="", ...)
HTTP.URI(str; isconnect=false)
parse(HTTP.URI, str::String; isconnect=false)

A type representing a valid uri. Can be constructed from distinct parts using the various supported keyword arguments. With a raw, already-encoded uri string, use parse(HTTP.URI, str) to parse the HTTP.URI directly. The HTTP.URI constructors will automatically escape any provided query arguments, typically provided as "key"=>"value"::Pair or Dict("key"=>"value"). Note that multiple values for a single query key can provided like Dict("key"=>["value1", "value2"]).

For efficiency, the internal representation is stored as a set of offsets and lengths to the various uri components. To access and return these components as strings, use the various accessor methods:

• HTTP.scheme: returns the scheme (if any) associated with the uri

• HTTP.userinfo: returns the userinfo (if any) associated with the uri

• HTTP.host: returns the host only of the uri

• HTTP.port: returns the port of the uri; will return "80" or "443" by default if the scheme is "http" or "https", respectively

• HTTP.hostport: returns the "host:port" combination; if the port is not provided or is the default port for the uri scheme, it will be omitted

• HTTP.path: returns the path for a uri

• HTTP.query: returns the query for a uri

• HTTP.fragment: returns the fragment for a uri

• HTTP.resource: returns the path-query-fragment combination

percent-encode a string, dict, or pair for a uri

unescape a percent-encoded uri/url

Splits the path into components See: http://tools.ietf.org/html/rfc3986#section-3.3

checks if a HTTP.URI is valid

Cookie()
Cookie(; kwargs...)
Cookie(name, value; kwargs...)

A Cookie represents an HTTP cookie as sent in the Set-Cookie header of an HTTP response or the Cookie header of an HTTP request. Supported fields (which can be set using keyword arguments) include:

• name: name of the cookie

• value: value of the cookie

• path: applicable path for the cookie

• domain: applicable domain for the cookie

• expires: a Dates.DateTime representing when the cookie should expire

• maxage: maxage == 0 means no max age, maxage < 0 means delete cookie now, max age > 0 means the # of seconds until expiration

• secure::Bool: secure cookie attribute

• httponly::Bool: httponly cookie attribute

• hostonly::Bool: hostonly cookie attribute

See http:#tools.ietf.org/html/rfc6265 for details.

HTTP.sniff(content::Union{Vector{UInt8}, String, IO}) => String (mimetype)

HTTP.sniff will look at the first 512 bytes of content to try and determine a valid mimetype. If a mimetype can't be determined appropriately, "application/octet-stream" is returned.

Supports JSON detection through the HTTP.isjson(content) function.

escapeHTML(i::String)

Returns a string with special HTML characters escaped: &, <, >, ", '

Request Execution Stack

The Request Execution Stack is separated into composable layers.

Each layer is defined by a nested type Layer{Next} where the Next parameter defines the next layer in the stack. The request method for each layer takes a Layer{Next} type as its first argument and dispatches the request to the next layer using request(Next, ...).

The example below defines three layers and three stacks each with a different combination of layers.

abstract type Layer end
abstract type Layer1{Next <: Layer} <: Layer end
abstract type Layer2{Next <: Layer} <: Layer end
abstract type Layer3 <: Layer end

request(::Type{Layer1{Next}}, data) where Next = "L1", request(Next, data)
request(::Type{Layer2{Next}}, data) where Next = "L2", request(Next, data)
request(::Type{Layer3}, data) = "L3", data

const stack1 = Layer1{Layer2{Layer3}}
const stack2 = Layer2{Layer1{Layer3}}
const stack3 = Layer1{Layer3}

julia> request(stack1, "foo")
("L1", ("L2", ("L3", "foo")))

julia> request(stack2, "bar")
("L2", ("L1", ("L3", "bar")))

julia> request(stack3, "boo")
("L1", ("L3", "boo"))

This stack definition pattern gives the user flexibility in how layers are combined but still allows Julia to do whole-stack comiple time optimistations.

e.g. the request(stack1, "foo") call above is optimised down to a single function:

julia> code_typed(request, (Type{stack1}, String))[1].first
CodeInfo(:(begin
    return (Core.tuple)("L1", (Core.tuple)("L2", (Core.tuple)("L3", data)))
end))

The stack() function returns the default HTTP Layer-stack type. This type is passed as the first parameter to the HTTP.request function.

stack() accepts optional keyword arguments to enable/disable specific layers in the stack: request(method, args...; kw...) request(stack(;kw...), args...; kw...)

The minimal request execution stack is:

stack = MessageLayer{ConnectionPoolLayer{StreamLayer}}

The figure below illustrates the full request exection stack and its relationship with HTTP.Response, HTTP.Parser, HTTP.Stream and the HTTP.ConnectionPool.

 ┌────────────────────────────────────────────────────────────────────────────┐
 │                                            ┌───────────────────┐           │
 │  HTTP.jl Request Execution Stack           │ HTTP.ParsingError ├ ─ ─ ─ ─ ┐ │
 │                                            └───────────────────┘           │
 │                                            ┌───────────────────┐         │ │
 │                                            │ HTTP.IOError      ├ ─ ─ ─     │
 │                                            └───────────────────┘      │  │ │
 │                                            ┌───────────────────┐           │
 │                                            │ HTTP.StatusError  │─ ─   │  │ │
 │                                            └───────────────────┘   │       │
 │                                            ┌───────────────────┐      │  │ │
 │     request(method, uri, headers, body) -> │ HTTP.Response     │   │       │
 │             ──────────────────────────     └─────────▲─────────┘      │  │ │
 │                           ║                          ║             │       │
 │   ┌────────────────────────────────────────────────────────────┐      │  │ │
 │   │ request(RedirectLayer,     method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(BasicAuthLayer,    method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(CookieLayer,       method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(CanonicalizeLayer, method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(MessageLayer,      method, ::URI, ::Headers, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(AWS4AuthLayer,             ::URI, ::Request, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(RetryLayer,                ::URI, ::Request, body) │   │       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
 │   │ request(ExceptionLayer,            ::URI, ::Request, body) ├ ─ ┘       │
 │   ├────────────────────────────────────────────────────────────┤      │  │ │
┌┼───┤ request(ConnectionPoolLayer,       ::URI, ::Request, body) ├ ─ ─ ─     │
││   ├────────────────────────────────────────────────────────────┤         │ │
││   │ request(TimeoutLayer,              ::IO,  ::Request, body) │           │
││   ├────────────────────────────────────────────────────────────┤         │ │
││   │ request(StreamLayer,               ::IO,  ::Request, body) │           │
││   └──────────────┬───────────────────┬─────────────────────────┘         │ │
│└──────────────────┼────────║──────────┼───────────────║─────────────────────┘
│                   │        ║          │               ║                   │  
│┌──────────────────▼───────────────┐   │  ┌──────────────────────────────────┐
││ HTTP.Request                     │   │  │ HTTP.Response                  │ │
││                                  │   │  │                                  │
││ method::String                   ◀───┼──▶ status::Int                    │ │
││ uri::String                      │   │  │ headers::Vector{Pair}            │
││ headers::Vector{Pair}            │   │  │ body::Vector{UInt8}            │ │
││ body::Vector{UInt8}              │   │  │                                  │
│└──────────────────▲───────────────┘   │  └───────────────▲────────────────┼─┘
│┌──────────────────┴────────║──────────▼───────────────║──┴──────────────────┐
││ HTTP.Stream <:IO          ║           ╔══════╗       ║                   │ │
││   ┌───────────────────────────┐       ║   ┌──▼─────────────────────────┐   │
││   │ startwrite(::Stream)      │       ║   │ startread(::Stream)        │ │ │
││   │ write(::Stream, body)     │       ║   │ read(::Stream) -> body     │   │
││   │ ...                       │       ║   │ ...                        │ │ │
││   │ closewrite(::Stream)      │       ║   │ closeread(::Stream)        │   │
││   └───────────────────────────┘       ║   └────────────────────────────┘ │ │
│└───────────────────────────║────────┬──║──────║───────║──┬──────────────────┘
│┌──────────────────────────────────┐ │  ║ ┌────▼───────║──▼────────────────┴─┐
││ HTTP.Messages                    │ │  ║ │ HTTP.Parser                      │
││                                  │ │  ║ │                                  │
││ writestartline(::IO, ::Request)  │ │  ║ │ parseheaders(bytes) do h::Pair   │
││ writeheaders(::IO, ::Request)    │ │  ║ │ parsebody(bytes) -> bytes        │
│└──────────────────────────────────┘ │  ║ └──────────────────────────────────┘
│                            ║        │  ║                                     
│┌───────────────────────────║────────┼──║────────────────────────────────────┐
└▶ HTTP.ConnectionPool       ║        │  ║                                    │
 │                     ┌──────────────▼────────┐ ┌───────────────────────┐    │
 │ getconnection() ->  │ HTTP.Transaction <:IO │ │ HTTP.Transaction <:IO │    │
 │                     └───────────────────────┘ └───────────────────────┘    │
 │                           ║    ╲│╱    ║                  ╲│╱               │
 │                           ║     │     ║                   │                │
 │                     ┌───────────▼───────────┐ ┌───────────▼───────────┐    │
 │              pool: [│ HTTP.Connection       │,│ HTTP.Connection       │...]│
 │                     └───────────┬───────────┘ └───────────┬───────────┘    │
 │                           ║     │     ║                   │                │
 │                     ┌───────────▼───────────┐ ┌───────────▼───────────┐    │
 │                     │ Base.TCPSocket <:IO   │ │MbedTLS.SSLContext <:IO│    │
 │                     └───────────────────────┘ └───────────┬───────────┘    │
 │                           ║           ║                   │                │
 │                           ║           ║       ┌───────────▼───────────┐    │
 │                           ║           ║       │ Base.TCPSocket <:IO   │    │
 │                           ║           ║       └───────────────────────┘    │
 └───────────────────────────║───────────║────────────────────────────────────┘
                             ║           ║                                     
 ┌───────────────────────────║───────────║──────────────┐  ┏━━━━━━━━━━━━━━━━━━┓
 │ HTTP Server               ▼                          │  ┃ data flow: ════▶ ┃
 │                        Request     Response          │  ┃ reference: ────▶ ┃
 └──────────────────────────────────────────────────────┘  ┗━━━━━━━━━━━━━━━━━━┛

See docs/src/layers.monopic.

request(RedirectLayer, method, ::URI, headers, body) -> HTTP.Response

Redirects the request in the case of 3xx response status.

request(BasicAuthLayer, method, ::URI, headers, body) -> HTTP.Response

Add Authorization: Basic header using credentials from url userinfo.

request(CookieLayer, method, ::URI, headers, body) -> HTTP.Response

Add locally stored Cookies to the request headers. Store new Cookies found in the response headers.

request(CanonicalizeLayer, method, ::URI, headers, body) -> HTTP.Response

Rewrite request and response headers in Canonical-Camel-Dash-Format.

request(MessageLayer, method, ::URI, headers, body) -> HTTP.Response

Construct a Request object and set mandatory headers.

request(AWS4AuthLayer, ::URI, ::Request, body) -> HTTP.Response

Add a AWS Signature Version 4 Authorization header to a Request.

Credentials are read from environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.

request(RetryLayer, ::URI, ::Request, body) -> HTTP.Response

Retry the request if it throws a recoverable exception.

Base.retry and Base.ExponentialBackOff implement a randomised exponentially increasing delay is introduced between attempts to avoid exacerbating network congestion.

Methods of isrecoverable(e) define which exception types lead to a retry. e.g. HTTP.IOError, Base.DNSError, Base.EOFError and HTTP.StatusError (if status is `5xx).

request(ExceptionLayer, ::URI, ::Request, body) -> HTTP.Response

Throw a StatusError if the request returns an error response status.

request(ConnectionPoolLayer, ::URI, ::Request, body) -> HTTP.Response

Retrieve an IO connection from the ConnectionPool.

Close the connection if the request throws an exception. Otherwise leave it open so that it can be reused.

IO related exceptions from Base are wrapped in HTTP.IOError. See isioerror.

request(TimeoutLayer, ::IO, ::Request, body) -> HTTP.Response

Close IO if no data has been received for timeout seconds.

request(StreamLayer, ::IO, ::Request, body) -> HTTP.Response

Create a Stream to send a Request and body to an IO stream and read the response.

Sens the Request body in a background task and begins reading the response immediately so that the transmission can be aborted if the Response status indicates that the server does not wish to receive the message body. RFC7230 6.5.

The parser separates a raw HTTP Message into its component parts.

If the input data is invalid the Parser throws a ParsingError.

The parser processes a single HTTP Message. If the input stream contains multiple Messages the Parser stops at the end of the first Message. The parseheaders and parsebody functions return a SubArray containing the unuses portion of the input.

The Parser does not interpret the Message Headers except as needed to parse the Message Body. It is beyond the scope of the Parser to deal with repeated header fields, multi-line values, cookies or case normalization.

The Parser has no knowledge of the high-level Request and Response structs defined in Messages.jl. The Parser has it's own low level Message struct that represents both Request and Response Messages.

The Messages module defines structs that represent HTTP.Request and HTTP.Response Messages.

The Response struct has a request field that points to the corresponding Request; and the Request struct has a response field. The Request struct also has a parent field that points to a Response in the case of HTTP Redirect.

The Messages module defines IO read and write methods for Messages but it does not deal with URIs, creating connections, or executing requests. The

The read methods throw EOFError exceptions if input data is incomplete. and call parser functions that may throw HTTP.ParsingError exceptions. The read and write methods may also result in low level IO exceptions.

Sending Messages

Messages are formatted and written to an IO stream by Base.write(::IO,::HTTP.Messages.Message) and or HTTP.Messages.writeheaders.

Receiving Messages

Messages are parsed from IO stream data by HTTP.Messages.readheaders. This function calls HTTP.Messages.appendheader and HTTP.Messages.readstartline!.

The read methods rely on HTTP.IOExtras.unread! to push excess data back to the input stream.

Headers

Headers are represented by Vector{Pair{String,String}}. As compared to Dict{String,String} this allows repeated header fields and preservation of order.

Header values can be accessed by name using HTTP.Messages.header and HTTP.Messages.setheader (case-insensitive).

The HTTP.Messages.appendheader function handles combining multi-line values, repeated header fields and special handling of multiple Set-Cookie headers.

Bodies

The HTTP.Message structs represent the Message Body as Vector{UInt8}.

Streaming of request and response bodies is handled by the HTTP.StreamLayer and the HTTP.Stream <: IO stream.

Stream(::IO, ::Request, ::Parser)

Creates a HTTP.Stream that wraps an existing IO stream.

• startwrite(::Stream) sends the Request headers to the IO stream.

• write(::Stream, body) sends the body (or a chunk of the bocdy).

• closewrite(::Stream) sends the final 0 chunk (if needed) and calls closewrite on the IO stream. When the IO stream is a HTTP.ConnectionPool.Transaction, calling closewrite releases the HTTP.ConnectionPool.Connection back into the pool for use by the next pipelined request.

• startread(::Stream) calls startread on the IO stream then reads and parses the Response headers. When the IO stream is a HTTP.ConnectionPool.Transaction, calling startread waits for other pipelined responses to be read from the HTTP.ConnectionPool.Connection.

• eof(::Stream) and readavailable(::Stream) parse the body from the IO stream.

• closeread(::Stream) reads the trailers and calls closeread on the IO stream. When the IO stream is a HTTP.ConnectionPool.Transaction, calling closeread releases the readlock and allows the next pipelined response to be read by another Stream that is waiting in startread. If the Parser has not recieved a complete response, closeread throws an EOFError.

This module provides the getconnection function with support for:

• Opening TCP and SSL connections.

• Reusing connections for multiple Request/Response Messages,

• Pipelining Request/Response Messages. i.e. allowing a new Request to be sent before previous Responses have been read.

This module defines a Connection struct to manage pipelining and connection reuse and a Transaction<: IO struct to manage a single pipelined request. Methods are provided for eof, readavailable, unsafe_write and close. This allows the Transaction object to act as a proxy for the TCPSocket or SSLContext that it wraps.

The pool is a collection of open Connections. The request function calls getconnection to retrieve a connection from the pool. When the request function has written a Request Message it calls closewrite to signal that the Connection can be reused for writing (to send the next Request). When the request function has read the Response Message it calls closeread to signal that the Connection can be reused for reading.

• method::Method: internal parser @enum for HTTP method.

• major and minor: HTTP version

• url::String: request URL

• status::Int: response status

• upgrade::Bool: Connection should be upgraded to a different protocol. e.g. CONNECT or Connection: upgrade.

Parser()

Create an unconfigured Parser.

parseheaders(::Parser, bytes) do h::Pair{String,String} ... -> excess

Read headers from bytes, passing each field/value pair to f. Returns a SubArray containing bytes not parsed.

e.g.

excess = parseheaders(p, bytes) do (k,v)
    println("$k: $v")
end

parsebody(::Parser, bytes) -> data, excess

Parse body data from bytes. Returns decoded data and excess bytes not parsed.

reset!(::Parser)

Revert Parser to unconfigured state.

messagestarted(::Parser)

Has the Parser begun processng a Message?

headerscomplete(::Parser)

Has the Parser processed the entire Message Header?

headerscomplete(::Message)

Have the headers been read into this Message?

bodycomplete(::Parser)

Has the Parser processed the Message Body?

messagecomplete(::Parser)

Has the Parser processed the entire Message?

messagehastrailing(::Parser)

Is the Parser ready to process trailing headers?

waitingforeof(::Parser)

Is the Parser waiting for the peer to close the connection to signal the end of the Message Body?

seteof(::Parser)

Signal that the peer has closed the connection.

connectionclosed(::Parser)

Was "Connection: close" parsed?

setnobody(::Parser)

Tell the Parser not to look for a Message Body. e.g. for the Response to a HEAD Request.

Request <: Message

Represents a HTTP Request Message.

• method::String

• uri::String

• version::VersionNumber

• headers::Vector{Pair{String,String}}

• body::Vector{UInt8}

• response, the Response to this Request

• parent, the Response (if any) that led to this request (e.g. in the case of a redirect).

Response <: Message

Represents a HTTP Response Message.

• version::VersionNumber

• status::Int16

• headers::Vector{Pair{String,String}}

• body::Vector{UInt8}

• request, the Request that yielded this Response.

iserror(::Response)

Does this Response have an error status?

isredirect(::Response)

Does this Response have a redirect status?

ischunked(::Message)

Does the Message have a "Transfer-Encoding: chunked" header?

issafe(::Request)

https://tools.ietf.org/html/rfc7231#section-4.2.1

isidempotent(::Request)

https://tools.ietf.org/html/rfc7231#section-4.2.2

header(::Message, key [, default=""]) -> String

Get header value for key (case-insensitive).

hasheader(::Message, key) -> Bool

Does header value for key exist (case-insensitive)?

setheader(::Message, key => value)

Set header value for key (case-insensitive).

defaultheader(::Message, key => value)

Set header value for key if it is not already set.

appendheader(::Message, key => value)

Append a header value to message.headers.

If key is "" the value is appended to the value of the previous header.

If key is the same as the previous header, the vale is appended to the value of the previous header with a comma delimiter

Set-Cookie headers are not comma-combined because cookies often contain internal commas.

readheaders(::IO, ::Parser, ::Message)

Read headers (and startline) from an IO stream into a Message struct. Throw EOFError if input is incomplete.

readstartline!(::Parsers.Message, ::Message)

Read the start-line metadata from Parser into a ::Message struct.

headerscomplete(::Message)

Have the headers been read into this Message?

readtrailers(::IO, ::Parser, ::Message)

Read trailers from an IO stream into a Message struct.

writestartline(::IO, ::Message)

e.g. "GET /path HTTP/1.1\r\n" or "HTTP/1.1 200 OK\r\n"

writeheaders(::IO, ::Message)

Write Message start line and a line for each "name: value" pair and a trailing blank line.

write(::IO, ::Message)

Write start line, headers and body of HTTP Message.

This module defines extensions to the Base.IO interface to support:

• an unread! function for pushing excess bytes back into a stream,

• startwrite, closewrite, startread and closeread for streams with transactional semantics.

unread!(::IO, bytes)

Push bytes back into a connection (to be returned by the next read).

unread!(::Transaction, bytes)

Push bytes back into a connection's excess buffer (to be returned by the next read).

startwrite(::IO)
closewrite(::IO)
startread(::IO)
closeread(::IO)

Signal start/end of write or read operations.

isioerror(exception)

Is exception caused by a possibly recoverable IO error.

closebody(::Stream)

Write the final 0 chunk if needed.

isaborted(::Stream{Response})

Has the server signalled that it does not wish to receive the message body?

"If [the response] indicates the server does not wish to receive the message body and is closing the connection, the client SHOULD immediately cease transmitting the body and close the connection." RFC7230, 6.5

Connection{T <: IO}

A TCPSocket or SSLContext connection to a HTTP host and port.

Fields:

• host::String

• port::String, exactly as specified in the URI (i.e. may be empty).

• pipeline_linit, number of requests to send before waiting for responses.

• peerport, remote TCP port number (used for debug messages).

• localport, local TCP port number (used for debug messages).

• io::T, the TCPSocket or `SSLContext.

• excess::ByteView, left over bytes read from the connection after the end of a response message. These bytes are probably the start of the next response message.

• writebusy, is a Transaction busy writing to this Connection ?

• writecount, number of Request Messages that have been written.

• readcount, number of Response Messages that have been read.

• writelock, busy writing a Request to io.

• readlock, busy reading a Response from io.

• `timestamp, time data was last recieved.

• parser::Parser, reuse a Parser when this Connection is reused.

A single pipelined HTTP Request/Response transaction`.

Fields:

• c, the shared Connection used for this Transaction.

• sequence::Int, identifies this Transaction among the others that share c.

The pool is a collection of open Connections. The request function calls getconnection to retrieve a connection from the pool. When the request function has written a Request Message it calls closewrite to signal that the Connection can be reused for writing (to send the next Request). When the request function has read the Response Message it calls closeread to signal that the Connection can be reused for reading.

getconnection(type, host, port) -> Connection

Find a reusable Connection in the pool, or create a new Connection if required.

unread!(::Transaction, bytes)

Push bytes back into a connection's excess buffer (to be returned by the next read).

startwrite(::Transaction)

Set writebusy. Should only be called by the Transaction constructor because getconnection only creates new Transactions when a Connection is available for writing.

closewrite(::Transaction)

Signal that an entire Request Message has been written to the Transaction.

startread(::Transaction)

Wait for prior pending reads to complete, then lock the readlock.

closeread(::Transaction)

Signal that an entire Response Message has been read from the Transaction.

Increment readcount and wake up tasks waiting in startread by unlocking readlock.