Replace NEON assembly memset16 and memset32 with intrinsic versions.

src/opts/SkUtils_opts_arm_neon.cpp

Index: src/opts/SkUtils_opts_arm_neon.cpp
diff --git a/src/opts/SkUtils_opts_arm_neon.cpp b/src/opts/SkUtils_opts_arm_neon.cpp
new file mode 100644
index 0000000000000000000000000000000000000000..b7d05046178bfb2d77157d91cfb69adab0235a8d
--- /dev/null
+++ b/src/opts/SkUtils_opts_arm_neon.cpp
@@ -0,0 +1,66 @@
+/*
+ * Copyright 2015 Google Inc.
+ *
+ * Use of this source code is governed by a BSD-style license that can be
+ * found in the LICENSE file.
+ */
+
+#include "SkTypes.h"
+#include <arm_neon.h>
+
+void sk_memset32_neon(uint32_t dst[], uint32_t value, int count) {
+    uint32x4_t   v4  = vdupq_n_u32(value);
+    uint32x4x4_t v16 = { v4, v4, v4, v4 };

[msarett, 2015/04/10 12:23:31]

Does this operation involve copying the contents of the register that contains
v4 to 3 other registers?

If so, do you think this is a lot of overhead when count is less than 16?

I'm guessing that when count is small memset is very fast regardless, so we are
probably optimizing for the case when count is very large?  In that case, this
seems like a good strategy.

[mtklein, 2015/04/10 12:45:55]

Yes.
 
Right.  We can actually assert count >= 10 today.  Lower than that and we'll do
an inline loop in C rather than going through the overhead of CPU feature
detection and function pointers.  (Just added that yesterday.)

I did try guarding the 16-at-a-time section with an if (count >= 16), and it
seemed no better.

[msarett, 2015/04/10 12:55:10]

Cool good to know!

+
+    while (count >= 16) {
+        vst4q_u32(dst, v16);  // This swizzles, but we don't care: all lanes are the same, value.

[msarett, 2015/04/10 12:23:31]

I just read a little more about ARM deinterleaving loads and interleaving
stores, since I've never really used or seen these before.

It looks like they were designed specifically for processing RGBA data.  Might
be good to be on the lookout for places that we can take advantage of these
instructions, where we might actually want the interleaving.

Also, as far as I can tell and as you probably discovered as well, there doesn't
seem to be any multi-element loads/stores that don't interleave, which could be
frustrating in a situation where we didn't want to interleave.

Have you checked that this is faster than four calls to a normal 1 vector store?
 I would bet it is, but I just wanted to ask in case there is a reason ARM
doesn't provide a non-interleaving 4 element store.

[mtklein, 2015/04/10 12:45:55]

Yep.  You'll find them used quite a bit under src/opts.  I'd love to find a way
to incorporate something like this into our SkPMFloat model without hurting too
much on Intel.
Yes, not in intrinsics.  In assembly, you can do them almost arbitrarily, all
the way up to something like vst1.u32 {d0,d1,d2,d3,d4,d5,d6,d7}, [r0] to do a
giant cache-line write to a pointer.
Yep.  As far as I could tell, using vst4q is what pushed me down into the range
where we're competitive with the old assembly.  It is faster than the best code
I could get out of 4 vst1q_u32 calls.  It's not clear to me that's faster than
the best code generated possible using vst1.32 instructions (i.e. inline
assembly), but I very much want to stick to intrinsics for maintainability and
portability across ARM versions.

[msarett, 2015/04/10 12:55:10]

That makes sense.  +1 for sticking with intrinsics.  The readability of this
code vs the assembly is a tremendous gain.

+        dst   += 16;
+        count -= 16;
+    }
+    SkASSERT(count < 16);
+    switch (count / 4) {
+        case 3: vst1q_u32(dst, v4); dst += 4; count -= 4;
+        case 2: vst1q_u32(dst, v4); dst += 4; count -= 4;
+        case 1: vst1q_u32(dst, v4); dst += 4; count -= 4;
+    }
+    SkASSERT(count < 4);
+    if (count >= 2) {
+        vst1_u32(dst, vget_low_u32(v4));

[msarett, 2015/04/10 12:23:31]

vget_low_u32 is a free call, right?

[mtklein, 2015/04/10 12:45:56]

Yep.  It just translates from whatever q-register we're in to the corresponding
d-register.  (vget_high is free too for the same reason.)

+        dst   += 2;
+        count -= 2;
+    }
+    SkASSERT(count < 2);
+    if (count > 0) {
+        *dst = value;
+    }
+}
+
+void sk_memset16_neon(uint16_t dst[], uint16_t value, int count) {
+    uint16x8_t   v8  = vdupq_n_u16(value);
+    uint16x8x4_t v32 = { v8, v8, v8, v8 };
+
+    while (count >= 32) {
+        vst4q_u16(dst, v32);  // This swizzles, but we don't care: all lanes are the same, value.
+        dst   += 32;
+        count -= 32;
+    }
+    SkASSERT(count < 32);
+    switch (count / 8) {
+        case 3: vst1q_u16(dst, v8); dst += 8; count -= 8;
+        case 2: vst1q_u16(dst, v8); dst += 8; count -= 8;
+        case 1: vst1q_u16(dst, v8); dst += 8; count -= 8;
+    }
+    SkASSERT(count < 8);
+    if (count >= 4) {
+        vst1_u16(dst, vget_low_u16(v8));
+        dst   += 4;
+        count -= 4;
+    }
+    SkASSERT(count < 4);
+    switch (count) {
+        case 3: *dst++ = value;
+        case 2: *dst++ = value;
+        case 1: *dst   = value;
+    }
+}
+