The ngx_event_pipe() function wasn't called on write events with
wev->delayed set.  As a result, threaded writing results weren't
properly collected in ngx_event_pipe_write_to_downstream() when a
write event was triggered for a completed write.

Further, this wasn't detected, as p->aio was reset by a thread completion
handler, and results were later collected in ngx_event_pipe_read_upstream()
instead of scheduling a new write of additional data.  If this happened
on the last reading from an upstream, last part of the response was never
written to the cache file.

Similar problems might also happen in case of timeouts when writing to
client, as this also results in ngx_event_pipe() not being called on write
events.  In this scenario socket leaks were observed.

Fix is to check if p->writing is set in ngx_event_pipe_read_upstream(), and
therefore collect results of previous write operations in case of read events
as well, similar to how we do so in ngx_event_pipe_write_downstream().
This is enough to fix the wev->delayed case.  Additionally, we now call
ngx_event_pipe() from ngx_http_upstream_process_request() if there are
uncollected write operations (p->writing and !p->aio).  This also fixes
the wev->timedout case.

The problem was introduced by 52bd8cc17f34.

A bug was introduced by 82efcedb310b that could lead to timing out of
responses or segmentation fault, when accept_mutex was enabled.

The output queue in HTTP/2 can contain frames from different streams.
When the queue is sent, all related write handlers need to be called.
In order to do so, the streams were added to the h2c->posted queue
after handling sent frames.  Then this queue was processed in
ngx_http_v2_write_handler().

If accept_mutex is enabled, the event's "ready" flag is set but its
handler is not called immediately.  Instead, the event is added to
the ngx_posted_events queue.  At the same time in this queue can be
events from upstream connections.  Such events can result in sending
output queue before ngx_http_v2_write_handler() is triggered.  And
at the time ngx_http_v2_write_handler() is called, the output queue
can be already empty with some streams added to h2c->posted.

But after 82efcedb310b, these streams weren't processed if all frames
have already been sent and the output queue was empty.  This might lead
to a situation when a number of streams were get stuck in h2c->posted
queue for a long time.  Eventually these streams might get closed by
the send timeout.

In the worst case this might also lead to a segmentation fault, if
already freed stream was left in the h2c->posted queue.  This could
happen if one of the streams was terminated but wasn't closed, due to
the HEADERS frame or a partially sent DATA frame left in the output
queue.  If this happened the ngx_http_v2_filter_cleanup() handler
removed the stream from the h2c->waiting or h2c->posted queue on
termination stage, before the frame has been sent, and the stream
was again added to the h2c->posted queue after the frame was sent.

In order to fix all these problems and simplify the code, write
events of fake stream connections are now added to ngx_posted_events
instead of using a custom h2c->posted queue.

Previously, a request body bigger than "client_body_buffer_size" wasn't written
into a temporary file if it has been pre-read entirely.  The preread buffer
is freed after processing, thus subsequent use of it might result in sending
corrupted body or cause a segfault.

Previously, while shutting down gracefully, the HTTP/2 connections were
closed in transition to idle state after all active streams have been
processed.  That might never happen if the client continued opening new
streams.

Now, nginx sends GOAWAY to all HTTP/2 connections and ignores further
attempts to open new streams.  A worker process will quit as soon as
processing of already opened streams is finished.

When the last_buf flag is cleared for add_after_body to append more data from a
subrequest, other filters may still have buffered data, which should be flushed
at this point.  For example, the sub_filter may have a partial match buffered,
which will only be flushed after the subrequest is done, ending up with
interleaved data in output.

Setting last_in_chain instead of last_buf flushes the data and fixes the order
of output buffers.

It fixes potential connection leak if some unsent data was left in the SSL
buffer.  Particularly, that could happen when a client canceled the stream
after the HEADERS frame has already been created.  In this case no other
frames might be produced and the HEADERS frame alone didn't flush the buffer.

Now it returns NGX_AGAIN if there's still data to be sent.

Checking for return value of c->send_chain() isn't sufficient since there
are data can be left in the SSL buffer.  Now the wew->ready flag is used
instead.

In particular, this fixed a connection leak in cases when all streams were
closed, but there's still some data to be sent in the SSL buffer and the
client forgot about the connection.

Particularly this fixes alerts on OS X and NetBSD systems when HTTP/2 is
configured over plain TCP sockets.

On these systems calling writev() with no data leads to EINVAL errors
being logged as "writev() failed (22: Invalid argument) while processing
HTTP/2 connection".