crypto: disable NSS AES-NI support when AVX is disabled by OS.

crypto/nss_util.cc

Index: crypto/nss_util.cc
diff --git a/crypto/nss_util.cc b/crypto/nss_util.cc
index 3ecedf63a48fd51aa73652fdb0725a0e42c46fec..24a2d36e6d7edc3f3c3874221ebe584586a1b413 100644
--- a/crypto/nss_util.cc
+++ b/crypto/nss_util.cc
@@ -12,6 +12,7 @@
 #include <prinit.h>
 #include <prtime.h>
 #include <secmod.h>
+#include <stdlib.h>
 
 #if defined(OS_LINUX)
 #include <linux/nfs_fs.h>
@@ -23,6 +24,7 @@
 
 #include <vector>
 
+#include "base/cpu.h"
 #include "base/debug/alias.h"
 #include "base/debug/stack_trace.h"
 #include "base/environment.h"
@@ -432,6 +434,8 @@ class NSSInitSingleton {
     // other threads from accessing until the constructor is done.
     thread_checker_.DetachFromThread();
 
+    DisableAESNIIfNeeded();
+
     EnsureNSPRInit();
 
     // We *must* have NSS >= 3.14.3.
@@ -624,6 +628,18 @@ class NSSInitSingleton {
     return db_slot;
   }
 
+  static void DisableAESNIIfNeeded() {
+    base::CPU cpu;
+
+    if (cpu.has_avx_hardware() && !cpu.has_avx()) {
+      // Some versions of NSS have a bug that causes AVX instructions to be
+      // used without testing whether XSAVE is enabled by the operating system.
+      // In order to work around this, we disable AES-NI in NSS when we find
+      // that |has_avx()| is false (which includes the XSAVE test).

[Ryan Sleevi, 2013/11/20 21:46:28]

Let's include an NSS version check (or softoken version check) here so that
we're not always disabling it on Linux.

The softoken version is exposed via PKCS#11 as part of the library version
.major/.minor, which will handle the cases (such as RHEL) where softoken may be
older than NSS.

bool use_hw_aes_if_present = true;
CK_TOKEN_INFO token_info;
ScopedPK11Slot slot(PK11_GetInternalSlot());
if (!slot.get() || PK11_GetTokenInfo(slot, &token_info) != SECSuccess) {
  use_hw_aes_if_present = false;
}
if (token_info.hardwareVersion.major < XXX ||
    (token_info.hardwareVersion.major == XXX &&
     token_info.hardwareVersion.minor < YYY)) {
  use_hw_aes_if_present = cpu.has_avx_hardware() && cpu.has_avx();
}
if (!use_hw_aes_if_present) {
  setenv(...)
}

[wtc, 2013/11/21 21:03:43]

In this particular case, it is enough to check the version of the main NSS
library.

We can safely assume this bug will be fixed in NSS 3.15.4. Since Softoken 3.15.3
is not FIPS 140-2 validated, there is no reason for RHEL to use a newer version
of NSS with Softoken 3.15.3.

We can use the test
    NSS_VersionCheck("3.15") && !NSS_VersionCheck("3.15.4")
to detect the buggy version range (3.15 <= version < 3.15.4). Although the bug
was introduced in NSS 3.14.2, there were no NSS PK11_ functions to access
AES-GCM until NSS 3.15.

+      setenv("NSS_DISABLE_HW_AES", "1", 1 /* overwrite */);
+    }
+  }
+
   // If this is set to true NSS is forced to be initialized without a DB.
   static bool force_nodb_init_;