net: rework the NPN patch.

net/third_party/nss/ssl/ssl.h

Index: net/third_party/nss/ssl/ssl.h
diff --git a/net/third_party/nss/ssl/ssl.h b/net/third_party/nss/ssl/ssl.h
index 03535f3c6a940af276bd75b18a6841955c60c47a..232277524c149ff58d8e364c88415c43672d477b 100644
--- a/net/third_party/nss/ssl/ssl.h
+++ b/net/third_party/nss/ssl/ssl.h
@@ -157,17 +157,27 @@ SSL_IMPORT SECStatus SSL_OptionSetDefault(PRInt32 option, PRBool on);
 SSL_IMPORT SECStatus SSL_OptionGetDefault(PRInt32 option, PRBool *on);
 SSL_IMPORT SECStatus SSL_CertDBHandleSet(PRFileDesc *fd, CERTCertDBHandle *dbHandle);
 
-SSL_IMPORT SECStatus SSL_SetNextProtoNego(PRFileDesc *fd,
-					  const unsigned char *data,
-					  unsigned short length);
-SSL_IMPORT SECStatus SSL_GetNextProto(PRFileDesc *fd,
-				      int *state,
-				      unsigned char *buf,
-				      unsigned *length,
-				      unsigned buf_len);
-#define SSL_NEXT_PROTO_NO_SUPPORT	0 /* No peer support                */
-#define SSL_NEXT_PROTO_NEGOTIATED	1 /* Mutual agreement               */
-#define SSL_NEXT_PROTO_NO_OVERLAP	2 /* No protocol overlap found      */
+#define SSL_NEXT_PROTO_NEGOTIATION_SUPPORTED 1

[wtc, 2011/10/05 23:34:25]

Why do we need the SSL_NEXT_PROTO_NEGOTIATION_SUPPORTED
macro?

[agl, 2011/10/07 19:19:17]

Because we need to know that we don't have the NPN functions when building with
use_system_ssl. Previously we used SSL_NEXT_PROTO_NEGOTIATED for this.

+
+/* SSLNextProtoCallback is called, during the handshake, when the server has
+ * sent a Next Protocol Negotiation extension. |protos| and |protosLen| define
+ * a buffer which contains the server's advertisement. This data is guaranteed
+ * to be well formed per the NPN spec. |protoOut| is a buffer of length 255
+ * (the maximum allowed by the protocol) which, on successful return, must
+ * contain the protocol to be announced to the server. */
+typedef SECStatus (PR_CALLBACK *SSLNextProtoCallback)(
+    void *arg,
+    PRFileDesc *fd,
+    const unsigned char* protos,
+    unsigned short protosLen,
+    unsigned char* protoOut,
+    unsigned char* protoOutLen);

[wtc, 2011/10/05 23:34:25]

Should we use 'unsigned int' for the lengths to be future-proof?

[agl, 2011/10/07 19:19:17]

But the format of protos requires that the maximum protocol string be 255 bytes.
Changing this would require changes to the callback function itself, so making
the return length larger couldn't actually be used.

I'd be happy to make it an unsigned int just to make it clean that it's a
length. But, when I did that, it introduced the problem of what to do when the
value from the callback is too large. We would need to introduce a new error and
I wasn't sure if we can do that with the NSS/SSL split that we have on Linux.

[wtc, 2011/10/11 22:27:57]

We can consider this change when we submit this patch to
NSS upstream.
Yes, we can do that.  NSS has two error code spaces:
SEC_ERROR_xxx and SSL_ERROR_xxx.

We can add a new SSL_ERROR_xxx error code to
mozilla/security/nss/lib/ssl/sslerr.h for this condition,
but I think we can just use the existing error code
SEC_ERROR_INPUT_LEN.

+
+/* SSL_SetNextProtoCallback sets a callback function to handle Next Protocol
+ * Negotiation. It causes a client to advertise NPN. */
+SSL_IMPORT SECStatus SSL_SetNextProtoCallback(PRFileDesc *fd,
+                                              SSLNextProtoCallback callback,
+                                              void *arg);
 
 /*
 ** Control ciphers that SSL uses. If on is non-zero then the named cipher