|
|
Created:
6 years, 7 months ago by haavardm Modified:
6 years, 1 month ago CC:
chromium-reviews, cbentzel+watch_chromium.org Base URL:
https://chromium.googlesource.com/chromium/src.git@master Visibility:
Public. |
DescriptionImplement zero-copy SSL buffers.
Use OpenSSL's non-copying interface to avoid allocating new buffers for
each read and write operation.
BUG=169885
Patch Set 1 #
Total comments: 6
Patch Set 2 : New IOBuffer version to reference count BIO objects #
Total comments: 8
Messages
Total messages: 13 (0 generated)
Had a go at http://crbug.com/169885. Ryan please review. I did not implement changing the OpenSSL buffers to the values set by setReceiveBufferSize()/SetSendBufferSize(). The reason is that it doesn't seem to be possible to change the buffer sizes of the bio pair after it has been init'ed.
not LGTM, but I appreciate you taking this on! I've tried to capture the subtle semantics here of net::IOBuffer, and why this can cause real problems (particularly on Windows), because we end up not respecting that contract. Hopefully this provides some guidance on how you might solve that - although I think any solution that ensures the API contract is kept will work. https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... File net/socket/ssl_client_socket_openssl.cc (right): https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... net/socket/ssl_client_socket_openssl.cc:1287: int available_read_bytes = BIO_nread0(transport_bio_, &bio_transport_buf); Note the comments in bss_bio.c WARNING: The non-copying interface is largely untested as of yet and may contain bugs. A little nervous, but hopefully unit tests will cover this. https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... net/socket/ssl_client_socket_openssl.cc:1290: CHECK_GE(static_cast<int>(max_read), available_read_bytes); Two things: 1) I'm a little nervous about this CHECK. A CHECK here would be in ensuring that OpenSSL doesn't violate it's contract, but your comment doesn't suggest it's that strong a guarantee from OpenSSL 2) I'm always nervous around size_t being cast to an int. That said, OpenSSL's API is terribad here (with respect to proper integer types), but I'm sure you can appreciate the concern. https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... net/socket/ssl_client_socket_openssl.cc:1295: available_read_bytes); BUG: So, the contract of IOBuffer is that, although it's ref-counted, the pointer must remain valid for the lifetime of the object, and it's legal for the consumer (in this case, the transport socket) to hold onto it beyond the object's destruction. A real world example of this is TCPClientSocketWin, which has TCPClientSocketWin::Core, which outlives the *SocketWin out of necessity, due to IO Completion Ports' behaviour that requires the buffers to be valid for the duration of the IOCP call. This is unsafe, because the buffer returned by BIO_nread0() only remains valid for the duration of transport_bio_, which is tied to the lifetime of this object. Thus you have a situation where SSLClientSocketOpenSSL is dtor'd, the transport_bio_ is released, transport_ is freed, but there is still an object that holds on to the IOBuffer and expects the buffer to remain valid (until the IOCP completes). We likely need to create a new IOBuffer type that can ensure that the transport_bio_ remains valid for as long as there are one or more IOBuffer's that still hold onto it. This also means that we need to consider the transfer semantics of the BIO* and the SSL* - if ownership of the BIO* is transferred to the SSL*, then we really have to keep the SSL* object alive as long as there are outstanding IOBuffers pointing into the BIO* buffer.
Thanks for reviewing and good comments. I knew it was a bit risky patch, so I guess this is starting point. I'll look into the buffer ownership issues and see if there are any ways to fix it. You mention in the bug report that it would save ~17kb per socket. But isn't the correct number closer to ~34Kb (both read and write buffer)? That is a bit of memory in "worst" case, so it would be nice to get this right. https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... File net/socket/ssl_client_socket_openssl.cc (right): https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... net/socket/ssl_client_socket_openssl.cc:1287: int available_read_bytes = BIO_nread0(transport_bio_, &bio_transport_buf); On 2014/04/29 18:57:02, Ryan Sleevi wrote: > Note the comments in bss_bio.c > > WARNING: The non-copying interface is largely untested as of yet and may contain Yes I saw that too and agree it's a bit risky, especially in the light of recent events regarding buffer handling in OpenSSL. Decided to have a go anyway. The API is not much code, so it should be possible get in control of it and understand the subtleness of it. The results of my analysis are in the comments here. > bugs. > > A little nervous, but hopefully unit tests will cover this. The unit tests runs fine on linux, but I would like more tests with different (especially smaller) buffer sizes in the tcp layer. https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... net/socket/ssl_client_socket_openssl.cc:1290: CHECK_GE(static_cast<int>(max_read), available_read_bytes); On 2014/04/29 18:57:02, Ryan Sleevi wrote: > Two things: > 1) I'm a little nervous about this CHECK. A CHECK here would be in ensuring that > OpenSSL doesn't violate it's contract, but your comment doesn't suggest it's > that strong a guarantee from OpenSSL Normally, if the read/write index is in the middle of the buffer, while there are free bytes in the beginning (end < index), then BIO_ctrl_pending() calculates the total number of free bytes including the chunk from start. BIO_read/BIO_write is able to wrap around and use all of the bytes when copying. However, this wrap around is not possible when the Socket object writes directly into the buffer. That is why the actual available bytes for non-copying interface might be less. I I remember correctly, the same issue exists for NSS's non-copying interface. According to this, the contract is that max > available_read_bytes. > > 2) I'm always nervous around size_t being cast to an int. That said, OpenSSL's > API is terribad here (with respect to proper integer types), but I'm sure you > can appreciate the concern. Absolutely. I do think it's safe in this case though. https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... net/socket/ssl_client_socket_openssl.cc:1295: available_read_bytes); On 2014/04/29 18:57:02, Ryan Sleevi wrote: > BUG: So, the contract of IOBuffer is that, although it's ref-counted, the > pointer must remain valid for the lifetime of the object, and it's legal for the > consumer (in this case, the transport socket) to hold onto it beyond the > object's destruction. > > A real world example of this is TCPClientSocketWin, which has > TCPClientSocketWin::Core, which outlives the *SocketWin out of necessity, due to > IO Completion Ports' behaviour that requires the buffers to be valid for the > duration of the IOCP call. > > This is unsafe, because the buffer returned by BIO_nread0() only remains valid > for the duration of transport_bio_, which is tied to the lifetime of this > object. > > Thus you have a situation where SSLClientSocketOpenSSL is dtor'd, the > transport_bio_ is released, transport_ is freed, but there is still an object > that holds on to the IOBuffer and expects the buffer to remain valid (until the > IOCP completes). Ah, right. I knew about the contract for IOBuffer but failed to realize that the OS could hold on to the buffer until read/write was done, even after destruction of the socket. I'm afraid my experience with Windows socket API is a bit limited. The documentation of Socket says: "If the operation is not completed immediately, the socket acquires a reference to the provided buffer until the callback is invoked or the socket is closed." I understood this as the socket would release the valid buffer requirement when the Socket object is closed/destructed by the owner. > > We likely need to create a new IOBuffer type that can ensure that the > transport_bio_ remains valid for as long as there are one or more IOBuffer's > that still hold onto it. > > This also means that we need to consider the transfer semantics of the BIO* and > the SSL* - if ownership of the BIO* is transferred to the SSL*, then we really > have to keep the SSL* object alive as long as there are outstanding IOBuffers > pointing into the BIO* buffer.
On 2014/04/29 20:52:56, haavardm wrote: > Thanks for reviewing and good comments. I knew it was a bit risky patch, so I > guess this is starting point. > I'll look into the buffer ownership issues and see if there are any ways to fix > it. > > You mention in the bug report that it would save ~17kb per socket. But isn't the > correct number closer to ~34Kb (both read and write buffer)? That is a bit of > memory in "worst" case, so it would be nice to get this right. > > https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... > File net/socket/ssl_client_socket_openssl.cc (right): > > https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... > net/socket/ssl_client_socket_openssl.cc:1287: int available_read_bytes = > BIO_nread0(transport_bio_, &bio_transport_buf); > On 2014/04/29 18:57:02, Ryan Sleevi wrote: > > Note the comments in bss_bio.c > > > > WARNING: The non-copying interface is largely untested as of yet and may > contain > > Yes I saw that too and agree it's a bit risky, especially in the light of recent > events regarding buffer handling in OpenSSL. Decided to have a go anyway. The > API is not much code, so it should be possible get in control of it and > understand the subtleness of it. The results of my analysis are in the comments > here. > > > bugs. > > > > A little nervous, but hopefully unit tests will cover this. > The unit tests runs fine on linux, but I would like more tests with different > (especially smaller) buffer sizes in the tcp layer. > > https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... > net/socket/ssl_client_socket_openssl.cc:1290: > CHECK_GE(static_cast<int>(max_read), available_read_bytes); > On 2014/04/29 18:57:02, Ryan Sleevi wrote: > > Two things: > > 1) I'm a little nervous about this CHECK. A CHECK here would be in ensuring > that > > OpenSSL doesn't violate it's contract, but your comment doesn't suggest it's > > that strong a guarantee from OpenSSL > > Normally, if the read/write index is in the middle of the buffer, while there > are free bytes in the beginning (end < index), then BIO_ctrl_pending() > calculates the total number of free bytes including the chunk from start. > BIO_read/BIO_write is able to wrap around and use all of the bytes when copying. > However, this wrap around is not possible when the Socket object writes directly > into the buffer. That is why the actual available bytes for non-copying > interface might be less. I I remember correctly, the same issue exists for NSS's > non-copying interface. > > According to this, the contract is that max > available_read_bytes. > > > > > 2) I'm always nervous around size_t being cast to an int. That said, OpenSSL's > > API is terribad here (with respect to proper integer types), but I'm sure you > > can appreciate the concern. > > Absolutely. I do think it's safe in this case though. > > https://codereview.chromium.org/255143002/diff/1/net/socket/ssl_client_socket... > net/socket/ssl_client_socket_openssl.cc:1295: available_read_bytes); > On 2014/04/29 18:57:02, Ryan Sleevi wrote: > > BUG: So, the contract of IOBuffer is that, although it's ref-counted, the > > pointer must remain valid for the lifetime of the object, and it's legal for > the > > consumer (in this case, the transport socket) to hold onto it beyond the > > object's destruction. > > > > A real world example of this is TCPClientSocketWin, which has > > TCPClientSocketWin::Core, which outlives the *SocketWin out of necessity, due > to > > IO Completion Ports' behaviour that requires the buffers to be valid for the > > duration of the IOCP call. > > > > This is unsafe, because the buffer returned by BIO_nread0() only remains valid > > for the duration of transport_bio_, which is tied to the lifetime of this > > object. > > > > Thus you have a situation where SSLClientSocketOpenSSL is dtor'd, the > > transport_bio_ is released, transport_ is freed, but there is still an object > > that holds on to the IOBuffer and expects the buffer to remain valid (until > the > > IOCP completes). > > Ah, right. I knew about the contract for IOBuffer but failed to realize that the > OS could hold on to the buffer until read/write was done, even after destruction > of the socket. I'm afraid my experience with Windows socket API is a bit > limited. > > The documentation of Socket says: > "If the operation is not completed immediately, the socket acquires a reference > to the provided buffer until the callback is invoked or the socket is closed." > > I understood this as the socket would release the valid buffer requirement when > the Socket object is closed/destructed by the owner. > Yeah, Socket is a bit weak in the documentation, but the net::IOBuffer documentation hopefully makes it more explicit, in https://code.google.com/p/chromium/codesearch#chromium/src/net/base/io_buffer...
I've created a new IOBuffer to keep the BIO buffer alive as long as the socket keeps a reference. PTAL. Since BIO uses reference counting, the ownership of transport_bio_ is never transferred as I can see. What is left (if you approve of this), is to make sure that OpenSSL behaves as intended. Note that if BIO_ctrl_get_write_guarantee() > 0 and BIO_ctrl_pending() then BIO_nwrite0()>0 or BIO_nwrite0()>0. Thus, I believe the flow will be as before, but we might risk in rare cases a few more calls to Read() and Write().
Promising! I need to dig into this a bit more, so don't feel rushed. What I need to look into is how this will behave with respect to the semantics of draining all available data. It'd be awesome if we could use scatter/gather IO, but that's inherently messy with SSL APIs, so it's not an option here. My concern (which I'll have to look more into, probably tomorrow) would be if we end up in a situation where we're making suboptimal read/writes (due to the 'free' position of the ring buffer shifting around, rather than being at the 'start' - which, in the copying interface, it's always reset to after draining the circle) as well as if there's any chance we can end up 'wedged' where data is hanging on in the beginning of the ring buffer but we either don't pick it up right away, or immediately after moving data to the network, we tell the caller we want more data. I'm more concerned about bugs - the optimizations can always come later - I just need to look into it a bit more and convince myself of all the state transitions. Other than that, I think this is right on path for the right implementation. https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... File net/socket/ssl_client_socket_openssl.cc (right): https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:47: explicit WrappedBIOAndIOBuffer(const char* bio_buffer, BIO* bio) nit: delete newline on 46 https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:1306: return 0; // Nothing pending in the OpenSSL write BIO. Is this still necessary, given that BIO_nread0 with a NULL buf (second arg) will return the size available for reading? Is this even needed to check, now that we follow a non-copying pattern? If available_read_bytes is 0, we can fall through, otherwise we can continue as normal. https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:1321: send_buffer_ = new DrainableIOBuffer( Huh. I'm not really sure why we use a DrainableIOBuffer for the OpenSSL case (we don't for NSS, which also uses a circular buffer).
Day off today but had a quick look anyway. I also think this is on the right track. Might be we need some more unit tests to really "fuzz up" the buffer sizes for testing the different situations that might happen, especially partial writes. https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... File net/socket/ssl_client_socket_openssl.cc (right): https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:1306: return 0; // Nothing pending in the OpenSSL write BIO. The problem with letting 0 fall through is that calling BIO_nread0 when there's no data triggers the retry flag and the request for one byte. This happens through the call to bio_read(bio, &dummy, 1) to avoid "code duplication " (see bss_bio.c:303). That's a side effect that doesn't happen today and to me this seems like something we want to avoid. On 2014/04/30 21:33:32, Ryan Sleevi wrote: > Is this still necessary, given that BIO_nread0 with a NULL buf (second arg) will > return the size available for reading? > > Is this even needed to check, now that we follow a non-copying pattern? If > available_read_bytes is 0, we can fall through, otherwise we can continue as > normal. https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:1321: send_buffer_ = new DrainableIOBuffer( On 2014/04/30 21:33:32, Ryan Sleevi wrote: > Huh. I'm not really sure why we use a DrainableIOBuffer for the OpenSSL case (we > don't for NSS, which also uses a circular buffer). I believe the drainable buffer is needed to handle partial write. It's useful to be able to consume data if TransportWriteComplete() gets called with only partial data sent. According to the socket documentation this may happen :"Note: data may be written partially." The calls BIO_nread() and Consume() in TransportWriteComplete() keeps the read and write indexes of transport_bio_ and send_buffer_ in sync.
Sorry for the delay in responding. https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... File net/socket/ssl_client_socket_openssl.cc (right): https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:1306: return 0; // Nothing pending in the OpenSSL write BIO. On 2014/05/01 16:55:19, haavardm wrote: > The problem with letting 0 fall through is that calling BIO_nread0 when there's > no data triggers the retry flag and the request for one byte. This happens > through the call to bio_read(bio, &dummy, 1) to avoid "code duplication " (see > bss_bio.c:303). That's a side effect that doesn't happen today and to me this > seems like something we want to avoid. Yeah, I suppose the alternative would be to reset the retry flag, but I agree, that seems less than ideal. https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:1321: send_buffer_ = new DrainableIOBuffer( On 2014/05/01 16:55:19, haavardm wrote: > On 2014/04/30 21:33:32, Ryan Sleevi wrote: > > Huh. I'm not really sure why we use a DrainableIOBuffer for the OpenSSL case > (we > > don't for NSS, which also uses a circular buffer). > > I believe the drainable buffer is needed to handle partial write. > > It's useful to be able to consume data if TransportWriteComplete() gets called > with only partial data sent. According to the socket documentation this may > happen :"Note: data may be written partially." Yeah, but why create a DrainableIOBuffer (and have custom logic for spinning the loop), as opposed to using BIO_nread0 to get the available, and then once data is written - partial or not - using BIO_nread to commit that many bytes (1421-1422) In the code prior to this, once we used BIO_read, we had committed to the BIO that we'd write that much data, so we had to use a DrainableIOBuffer to keep track of that. However, with the copy-less approach, I *think* it should be sufficient to BIO_nread0 the maximum, and then, based on what the transport wrote, just advance the circular buffer with the BIO_nread. eg: assume we have 10 bytes in the circular buffer BIO_ctrl_pending == 10 BIO_nread0(bio, buf) == 10 transport_->Write(..., 10) == 5 [only half the buffer was written] BIO_nread(bio, buf, 5) // what to do with ret here - currently dropping to the floor? Which will then cause BIO_ctrl_pending == 5 BIO_nread(bio, buf) == 5 transport_->Write(..., 5) == 5 [the remaining half] BIO_nread(bio, buf, 5); which will then cause BIO_ctrl_pending == 0 The DrainableIOBuffer was our poor-man's way of simulating that. > > The calls BIO_nread() and Consume() in TransportWriteComplete() keeps the read > and write indexes of transport_bio_ and send_buffer_ in sync.
How about the OpenSSL? Are you getting more confident about the non-copy API? What I could do is to write some more tests that will stress the API, by trying to trigger the different special cases that might happen. I know you already have stress tests, but I'm sure it's possible to add some for this case. It would also be nice to simulate and make sure all SSL code handles the keep-OutBuffer-alive-after-socket-close situation. I could also run those tests through Valgrind or Googles very own AddressSanitizer. https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... File net/socket/ssl_client_socket_openssl.cc (right): https://codereview.chromium.org/255143002/diff/20001/net/socket/ssl_client_so... net/socket/ssl_client_socket_openssl.cc:1321: send_buffer_ = new DrainableIOBuffer( On 2014/05/08 22:59:28, Ryan Sleevi wrote: > On 2014/05/01 16:55:19, haavardm wrote: > > On 2014/04/30 21:33:32, Ryan Sleevi wrote: > > > Huh. I'm not really sure why we use a DrainableIOBuffer for the OpenSSL case > > (we > > > don't for NSS, which also uses a circular buffer). > > > > I believe the drainable buffer is needed to handle partial write. > > > > It's useful to be able to consume data if TransportWriteComplete() gets called > > with only partial data sent. According to the socket documentation this may > > happen :"Note: data may be written partially." > > Yeah, but why create a DrainableIOBuffer (and have custom logic for spinning the > loop), as opposed to using BIO_nread0 to get the available, and then once data > is written - partial or not - using BIO_nread to commit that many bytes > (1421-1422) > > In the code prior to this, once we used BIO_read, we had committed to the BIO > that we'd write that much data, so we had to use a DrainableIOBuffer to keep > track of that. However, with the copy-less approach, I *think* it should be > sufficient to BIO_nread0 the maximum, and then, based on what the transport > wrote, just advance the circular buffer with the BIO_nread. > > eg: assume we have 10 bytes in the circular buffer > BIO_ctrl_pending == 10 > BIO_nread0(bio, buf) == 10 > transport_->Write(..., 10) == 5 [only half the buffer was written] > BIO_nread(bio, buf, 5) // what to do with ret here - currently dropping to the > floor? > > Which will then cause > BIO_ctrl_pending == 5 > BIO_nread(bio, buf) == 5 > transport_->Write(..., 5) == 5 [the remaining half] > BIO_nread(bio, buf, 5); > > which will then cause > BIO_ctrl_pending == 0 > > The DrainableIOBuffer was our poor-man's way of simulating that. > > > > > The calls BIO_nread() and Consume() in TransportWriteComplete() keeps the read > > and write indexes of transport_bio_ and send_buffer_ in sync. > Yes, I had a look into NSS yesterday and saw how you do it there. In NSS (as opposed to OpenSSL standard copy interface) you are able to report how much you have written. I agree we can remove the this for non-copy interface since OpenSSL takes care of the accounting. It will also simplify the code.
On 2014/05/09 07:23:41, haavardm wrote: > How about the OpenSSL? Are you getting more confident about the non-copy API? > > What I could do is to write some more tests that will stress the API, by trying > to trigger the different special cases that might happen. I know you already > have stress tests, but I'm sure it's possible to add some for this case. > > It would also be nice to simulate and make sure all SSL code handles the > keep-OutBuffer-alive-after-socket-close situation. > > I could also run those tests through Valgrind or Googles very own > AddressSanitizer. Agreed. If we can incorporate it in net_unittests and it will be covered by the valgrind bots and the msan bots. An SSLClientSocket takes a ClientSocketHandle. Similar to WrappedStreamSocket, we can create a ClientSocketHandle interposing layer that holds on to (and accesses) the IOBuffer after the fact, and ensure nothing crashes. This will simulate the Win sockets, but be usable cross platform - especially the Linux MSan/Asan bots. > <snip> > > Yes, I had a look into NSS yesterday and saw how you do it there. In NSS (as > opposed to OpenSSL standard copy interface) you are able to report how much you > have written. > > I agree we can remove the this for non-copy interface since OpenSSL takes care > of the accounting. It will also simplify the code. Yeah, it should definitely simplify things.
NACK. This CL appears to be resting, but I'd like to note that BIO_nread is likely to get removed from our OpenSSL in the near future. Removing the copies would be nice, but I suspect that a much better job can be done if not depending on the current OpenSSL API.
On 2014/06/09 18:00:44, agl wrote: > NACK. > > This CL appears to be resting, but I'd like to note that BIO_nread is likely to > get removed from our OpenSSL in the near future. > What do you mean by "our OpenSSL". Is it removed in Chromium only? > Removing the copies would be nice, but I suspect that a much better job can be > done if not depending on the current OpenSSL API. Yeah, I agree. Do you suggest to add a new zero copying API in Openssl?
On Tue, Jun 10, 2014 at 12:57 AM, <haavardm@opera.com> wrote: > What do you mean by "our OpenSSL". Is it removed in Chromium only? We'll be trimming some bits from the OpenSSL in Chromium. I think Sleevi will be emailing you about it, but had to sleep yesterday after a long fight with WebCrypto. > Yeah, I agree. Do you suggest to add a new zero copying API in Openssl? In the future, that's probably a good idea. Cheers AGL To unsubscribe from this group and stop receiving emails from it, send an email to chromium-reviews+unsubscribe@chromium.org. |