[Hacky prototype] Create a shared-memory high-performance reporting service.

[Hacky prototype] Create a shared-memory high-performance reporting service.

This is an exploration into ways to share high-frequency performance data from child services to a central aggregation service. I'd anticipate that BlamableObjects would route their data to appropriate structures in these shared memory segments and that the central service would asynchronously scrape the data on its own schedule. Round-trip IPC would only be required when new shared memory segments are required.

This is built on top of the PersistentMemoryAllocator infrastructure built for UMA. This is effectively a lock-free thread-safe allocator built with atomics and with performance in mind. It is self-describing and allows a remote process to iterate and discover data as it is written.

BUG=

[chrisha, 2017-05-18 21:58:35]

There are aborted bits of code in here that can be ignore. BlameNode was an
early attempt at starting to create what will now be a BlamableObject (altimin@
working on that). The shared memory allocator is about the only interesting bit.

[Sami, 2017-05-19 13:44:03]

skyostil@chromium.org changed reviewers:
+ skyostil@chromium.org

[Sami, 2017-05-19 13:44:04]

I've didn't look at the BlameNode stuff too deeply. Is the basic idea with the
shared memory slabs that each client gets an arena where they can push
statistics at some undefined rate and the service will poll the contents?

We might wanna couple that with
https://cs.chromium.org/chromium/src/device/base/synchronization/shared_memor...
to avoid issues with buffer tearing.

https://codereview.chromium.org/2885363004/diff/1/services/blamer/public/inte...
File services/blamer/public/interfaces/shared_memory_heap_registry.mojom
(right):

https://codereview.chromium.org/2885363004/diff/1/services/blamer/public/inte...
services/blamer/public/interfaces/shared_memory_heap_registry.mojom:40: uint32
size;
This is set by the client, i.e., renderer, right? Can we trust it?

https://codereview.chromium.org/2885363004/diff/1/services/blamer/shared_memo...
File services/blamer/shared_memory_heap.cc (right):

https://codereview.chromium.org/2885363004/diff/1/services/blamer/shared_memo...
services/blamer/shared_memory_heap.cc:84: slabs_.push_back(std::move(slab));
I wondering about re-entrancy issues here, e.g., this could end up going into
memory-infra which might call back into CreateNewSlab. Maybe we should at least
add some DCHECKs against that?

https://codereview.chromium.org/2885363004/diff/1/services/blamer/shared_memo...
services/blamer/shared_memory_heap.cc:114: }
CHECK(ref); ?

[oystein (OOO til 10th of July), 2017-05-22 21:43:33]

oysteine@chromium.org changed reviewers:
+ oysteine@chromium.org

[oystein (OOO til 10th of July), 2017-05-22 21:43:34]

Closing as I'm landing this for holte@ in
https://codereview.chromium.org/2893943004.

[oystein (OOO til 10th of July), 2017-05-22 21:44:06]

On 2017/05/22 at 21:43:34, oystein wrote:
> Closing as I'm landing this for holte@ in
https://codereview.chromium.org/2893943004.

Aaand this was the wrong CL, sorry about that.

[chrisha, 2017-05-25 17:59:52]

On 2017/05/19 13:44:04, Sami wrote:
> I've didn't look at the BlameNode stuff too deeply. Is the basic idea with the
> shared memory slabs that each client gets an arena where they can push
> statistics at some undefined rate and the service will poll the contents?

Don't worry about the BlameNode stuff. Was starting to lift "BlameContext" into
this, but altimin's plan for that is good.

> We might wanna couple that with
>
https://cs.chromium.org/chromium/src/device/base/synchronization/shared_memor...
> to avoid issues with buffer tearing.

The underlying primitive here is already multiple-writer multiple-reader
thread-safe using various lock-free algos. See:

https://cs.chromium.org/chromium/src/base/metrics/persistent_memory_allocator.h

[chrisha, 2017-05-25 18:05:08]

https://codereview.chromium.org/2885363004/diff/1/services/blamer/public/inte...
File services/blamer/public/interfaces/shared_memory_heap_registry.mojom
(right):

https://codereview.chromium.org/2885363004/diff/1/services/blamer/public/inte...
services/blamer/public/interfaces/shared_memory_heap_registry.mojom:40: uint32
size;
On 2017/05/19 13:44:04, Sami wrote:
> This is set by the client, i.e., renderer, right? Can we trust it?

Good point. We could invert the logic and have the renderer request a slab from
the browser. Slab creation should be infrequent so the round-trip isn't too
expensive.

https://codereview.chromium.org/2885363004/diff/1/services/blamer/shared_memo...
File services/blamer/shared_memory_heap.cc (right):

https://codereview.chromium.org/2885363004/diff/1/services/blamer/shared_memo...
services/blamer/shared_memory_heap.cc:84: slabs_.push_back(std::move(slab));
On 2017/05/19 13:44:04, Sami wrote:
> I wondering about re-entrancy issues here, e.g., this could end up going into
> memory-infra which might call back into CreateNewSlab. Maybe we should at
least
> add some DCHECKs against that?

Yeah, fair point. Depending on what the memory-infra code itself is doing we
might actually need to explicitly handle this case rather than just DCHECK
against it.

[Sami, 2017-05-26 09:42:11]

On 2017/05/25 17:59:52, chrisha wrote:
> The underlying primitive here is already multiple-writer multiple-reader
> thread-safe using various lock-free algos. See:
> 
>
https://cs.chromium.org/chromium/src/base/metrics/persistent_memory_allocator.h

I might have missed something but looks like that makes it safe to share the
allocations themselves but doesn't give any guarantees about the contents of the
memory:

https://cs.chromium.org/chromium/src/base/metrics/persistent_memory_allocator...

Or maybe we're never modifying the objects written to the allocation?

[chrisha, 2017-05-26 18:36:43]

On 2017/05/26 09:42:11, Sami wrote:
> On 2017/05/25 17:59:52, chrisha wrote:
> > The underlying primitive here is already multiple-writer multiple-reader
> > thread-safe using various lock-free algos. See:
> > 
> >
>
https://cs.chromium.org/chromium/src/base/metrics/persistent_memory_allocator.h
> 
> I might have missed something but looks like that makes it safe to share the
> allocations themselves but doesn't give any guarantees about the contents of
the
> memory:
> 
>
https://cs.chromium.org/chromium/src/base/metrics/persistent_memory_allocator...
> 
> Or maybe we're never modifying the objects written to the allocation?

As long as we use appropriately aligned atomic primitives for the actual content
of the allocations then we shouldn't have to worry about tearing. But yes, if
we're using things that cross cache lines all bets are off. I was primarily
imagining this for numeric data (counts, sizes, rates, what have you) using
atomic primitives.

If we need to communicate more complicated things then yes, we'll need to worry
about this problem.