The NGX_DONE value returned from ngx_http_upstream_cache_send() indicates
that upstream was already finalized in ngx_http_upstream_process_headers().
It was treated as a generic error which resulted in duplicate finalization.

Handled NGX_HTTP_UPSTREAM_INVALID_HEADER from ngx_http_upstream_cache_send().
Previously, it could return within ngx_http_upstream_finalize_request(), and
since it's below NGX_HTTP_SPECIAL_RESPONSE, a client connection could stuck.

When parsing of headers in a cache file fails, already parsed headers
need to be cleared, and protocol state needs to be reinitialized.  To do
so, u->request_sent is now set to ensure ngx_http_upstream_reinit() will
be called.

This change complements improvements in 46ddff109e72.

This slightly reduces cost of selecting a peer if all or almost all peers
failed, see ticket #1030.  There should be no measureable difference with
other workloads.

While this may result in non-ideal distribution of requests if nginx
won't be able to select a server in a reasonable number of attempts,
this still looks better than severe performance degradation observed
if there is no limit and there are many points configured (ticket #1030).
This is also in line with what we do for other hash balancing methods.

Previously, unix sockets were treated as AF_INET ones, and this may
result in buffer overread on Linux, where unbound unix sockets have
2-byte addresses.

Note that it is not correct to use just sun_path as a binary representation
for unix sockets.  This will result in an empty string for unbound unix
sockets, and thus behaviour of limit_req and limit_conn will change when
switching from $remote_addr to $binary_remote_addr.  As such, normal text
representation is used.

Reported by Stephan Dollberg.

The sync flag of HTTP/2 request body buffer is used when the size of request
body is unknown or bigger than configured "client_body_buffer_size".  In this
case the buffer points to body data inside the global receive buffer that is
used for reading all HTTP/2 connections in the worker process.  Thus, when the
sync flag is set, the buffer must be flushed to a temporary file, otherwise
the request body data can be overwritten.

Previously, the sync buffer wasn't flushed to a temporary file if the whole
body was received in one DATA frame with the END_STREAM flag and wasn't
copied into the HTTP/2 body preread buffer.  As a result, the request body
might be corrupted (ticket #1384).

Now, setting r->request_body_in_file_only enforces writing the sync buffer
to a temporary file in all cases.

When caching intercepted errors, previous behaviour was to use
proxy_cache_valid times specified, regardless of various cache control
headers present in the response.  Fix is to check u->cacheable and
use u->cache->valid_sec as set by various cache control response headers,
similar to how we do this in the normal caching code path.

If cache file is truncated, it is possible that u->process_header()
will return NGX_AGAIN.  Added appropriate handling of this case by
changing the error to NGX_HTTP_UPSTREAM_INVALID_HEADER.

Also, added appropriate logging of this and NGX_HTTP_UPSTREAM_INVALID_HEADER
cases at the "crit" level.  Note that this will result in duplicate logging
in case of NGX_HTTP_UPSTREAM_INVALID_HEADER.  While this is something better
to avoid, it is considered to be an overkill to implement cache-specific
error logging in u->process_header().

Additionally, u->buffer.start is now reset to be able to receive a new
response, and u->cache_status set to MISS to provide the value in the
$upstream_cache_status variable, much like it happens on other cache file
errors detected by ngx_http_file_cache_read(), instead of HIT, which is
believed to be misleading.

It is to be used as a bitmask with various bits set/reset when appropriate.
63b8b157b776 made a similar change to ngx_http_upstream_rr_peer_t.down and
ngx_stream_upstream_rr_peer_t.down.

The change in ac120e797d28 re-used the macro which was made obsolete
in adf25b8d0431.