When filter finalization is triggered when working with an upstream server,
and error_page redirects request processing to some simple handler,
ngx_http_request_finalize() triggers request termination when the response
is sent.  In particular, via the upstream cleanup handler, nginx will close
the upstream connection and the corresponding socket.

Still, this can happen to be with ngx_event_pipe() on stack.  While
the code will set p->downstream_error due to NGX_ERROR returned from the
output filter chain by filter finalization, otherwise the error will be
ignored till control returns to ngx_http_upstream_process_request().
And event pipe might try reading from the (already closed) socket, resulting
in "readv() failed (9: Bad file descriptor) while reading upstream" errors
(or even segfaults with SSL).

Such errors were seen with the following configuration:

    location /t2 {
        proxy_pass http://127.0.0.1:8080/big;

        image_filter_buffer 10m;
        image_filter   resize  150 100;
        error_page     415   = /empty;
    }

    location /empty {
        return 204;
    }

    location /big {
        # big enough static file
    }

Fix is to clear p->upstream in ngx_http_upstream_finalize_request(),
and ensure that p->upstream is checked in ngx_event_pipe_read_upstream()
and when handling events at ngx_event_pipe() exit.

Previous behaviour was to pass everything to the client, but this
seems to be suboptimal and causes issues (ticket #1695).  Fix is to
drop extra data instead, as it naturally happens in most clients.

This change covers generic buffered and unbuffered filters as used
in the scgi and uwsgi modules.  Appropriate input filter init
handlers are provided by the scgi and uwsgi modules to set corresponding
lengths.

Note that for responses to HEAD requests there is an exception:
we do allow any response length.  This is because responses to HEAD
requests might be actual full responses, and it is up to nginx
to remove the response body.  If caching is enabled, only full
responses matching the Content-Length header will be cached
(see b779728b180c).

In SSL connections, data can be buffered by the SSL layer, and it is
wrong to avoid doing c->recv_chain() if c->read->available is 0 and
c->read->pending_eof is set.  And tests show that the optimization in
question indeed can result in incorrect detection of premature connection
close if upstream closes the connection without sending a close notify
alert at the same time.  Fix is to disable c->read->available optimization
for SSL connections.

The ngx_event_pipe() function wasn't called on write events with
wev->delayed set.  As a result, threaded writing results weren't
properly collected in ngx_event_pipe_write_to_downstream() when a
write event was triggered for a completed write.

Further, this wasn't detected, as p->aio was reset by a thread completion
handler, and results were later collected in ngx_event_pipe_read_upstream()
instead of scheduling a new write of additional data.  If this happened
on the last reading from an upstream, last part of the response was never
written to the cache file.

Similar problems might also happen in case of timeouts when writing to
client, as this also results in ngx_event_pipe() not being called on write
events.  In this scenario socket leaks were observed.

Fix is to check if p->writing is set in ngx_event_pipe_read_upstream(), and
therefore collect results of previous write operations in case of read events
as well, similar to how we do so in ngx_event_pipe_write_downstream().
This is enough to fix the wev->delayed case.  Additionally, we now call
ngx_event_pipe() from ngx_http_upstream_process_request() if there are
uncollected write operations (p->writing and !p->aio).  This also fixes
the wev->timedout case.

This fixes a problem with aio threads and sendfile with aio_write switched
off, as observed with range requests after fc72784b1f52 (1.9.13).  Potential
problems with sendfile in threads were previously described in 9fd738b85fad,
and this seems to be one of them.

The problem occurred as file's thread_handler was set to NULL by event pipe
code after a sendfile thread task was scheduled.  As a result, no sendfile
completion code was executed, and the same buffer was additionally sent
using non-threaded sendfile.  Fix is to avoid modifying file's thread_handler
if aio_write is switched off.

Note that with "aio_write on" it is still possible that sendfile will use
thread_handler as set by event pipe.  This is believed to be safe though,
as handlers used are compatible.

When c->recv_chain() returns an error, it is possible that we already
have some data previously read, e.g., in preread buffer.  And in some
cases it may be even a complete response.  Changed c->recv_chain() error
handling to process the data, much like it is already done if kevent
reports about an error.

This change, in particular, fixes processing of small responses
when an upstream fails to properly close a connection with lingering and
therefore the connection is reset, but the response is already fully
obtained by nginx (see ticket #1037).

The "aio_write" directive is introduced, which enables use of aio
for writing.  Currently it is meaningful only with "aio threads".

Note that aio operations can be done by both event pipe and output
chain, so proper mapping between r->aio and p->aio is provided when
calling ngx_event_pipe() and in output filter.

In collaboration with Valentin Bartenev.

The change was missed in f69d1aab6a0f.