Create "persistent memory allocator" for persisting and sharing objects.

base/memory/persistent_memory_allocator.h

+// Copyright (c) 2015 The Chromium Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+#ifndef BASE_MEMORY_PERSISTENT_MEMORY_ALLOCATOR_H_
+#define BASE_MEMORY_PERSISTENT_MEMORY_ALLOCATOR_H_
+
+#include <stdint.h>
+#include <atomic>
+#include <string>
+
+#include "base/atomicops.h"

[Dmitry Vyukov, 2015/12/03 20:51:37]

Do you still need this include?

+#include "base/base_export.h"
+#include "base/gtest_prod_util.h"
+#include "base/macros.h"
+
+namespace base {
+
+class HistogramBase;
+
+// Simple allocator for pieces of a memory block that may be persistent
+// to some storage or shared across multiple processes.
+//
+// This class provides for thread-secure (i.e. safe against other threads
+// or processes that may be compromised and thus have malicious intent)
+// allocation of memory within a designated block and also a mechanism by
+// which other threads can learn of these allocations.
+//
+// There is (currently) no way to release an allocated block of data because
+// doing so would risk invalidating pointers held by other processes and
+// greatly complicate the allocation algorithm.
+//
+// Construction of this object can accept new, clean (i.e. zeroed) memory
+// or previously initialized memory. In the first case, construction must
+// be allowed to complete before letting other allocators attach to the same
+// segment. In other words, don't share the segment until at least one
+// allocator has been attached to it.
+//
+// It should be noted that memory doesn't need to actually have zeros written

[Dmitry Vyukov, 2015/12/03 20:51:37]

I still don't understand this comment. All memory is like that. Malloc() can
well give you non-pinned/non-existent memory as well. OS can also push any page
to swap at any moment, so that page suddenly becomes non-pinned. Moreover, even
if you write a non-zero value to a page (so that now it is surely exists), but
then write zero, OS can observe that and vanish the page asynchronously.
The distinction you are talking about is non-important and non-observable from
user-space. This remark equally applies to any code that just needs zeros. E.g.
is a global variable actually exist in memory, or just "reads as zero"?

I think we need to remove this part. Besides being non-necessary (how could it
not support such memory?) it can be very confusing for developers who are not
familiar with virtual memory/OS internal works.

[bcwhite, 2015/12/03 21:53:41]

I guess it is somewhat redundant since the previous paragraph states the memory
must be "clean (i.e. zeroed)".  Done.

Note, however, that malloc() doesn't guarantee that the memory will be zero if
it was un-pinned but now is pinned because malloc doesn't guarantee
initialization of any kind.  It could pin the memory on read and return whatever
garbage was there previously; such might be inefficient but within spec.

+// throughout; it just needs to read as zero until something diffferent is
+// written to a location. This is an important distinction as it supports the
+// use-case of non-pinned memory, such as from a demand-allocated region by
+// the OS or a memory-mapped file that auto-grows from a starting size of zero.
+class BASE_EXPORT PersistentMemoryAllocator {
+ public:
+  typedef uint32_t Reference;
+
+  // Internal state information when iterating over memory allocations.
+  class Iterator {
+   public:
+    Iterator() : last(0) {}
+
+    bool operator==(const Iterator& rhs) const { return last == rhs.last; }
+    bool operator!=(const Iterator& rhs) const { return last != rhs.last; }
+
+    void clear() { last = 0; }
+    bool is_clear() { return last == 0; }
+
+   private:
+    friend class PersistentMemoryAllocator;
+
+    Reference last;
+    uint32_t niter;
+  };
+
+  // Returned information about the internal state of the heap.
+  struct MemoryInfo {
+    size_t total;
+    size_t free;
+  };
+
+  enum : uint32_t {
+    kTypeIdAny = 0  // Match any type-id inside GetAsObject().
+  };
+
+  // The allocator operates on any arbitrary block of memory. Creation and
+  // persisting or sharing of that block with another process is the
+  // responsibility of the caller. The allocator needs to know only the
+  // block's |base| address, the total |size| of the block, and any internal
+  // |page| size (zero if not paged) across which allocations should not span.
+  // The |name|, if provided, is used to distinguish histograms for this
+  // allocator. Only the primary owner of the segment should define this value;
+  // other processes can learn it from the shared state.
+  //
+  // PersistentMemoryAllocator does NOT take ownership of the memory block.
+  // The caller must manage it and ensure it stays available throughout the
+  // lifetime of this object.
+  //
+  // Memory segments for sharing must have had an allocator attached to them
+  // before actually being shared. If the memory segment was just created, it
+  // should be zeroed before being passed here. If it was an existing segment,
+  // the values here will be compared to copies stored in the shared segment
+  // as a guard against corruption.
+  PersistentMemoryAllocator(void* base, size_t size, size_t page_size,
+                            const std::string& name);
+  ~PersistentMemoryAllocator();
+
+  // Get an object referenced by a |ref|. For safety reasons, the |type_id|
+  // code and size-of(|T|) are compared to ensure the reference is valid
+  // and cannot return an object outside of the memory segment. A |type_id| of
+  // zero will match any though the size is still checked. NULL is returned
+  // if any problem is detected, such as corrupted storage or incorrect
+  // parameters. Callers MUST check that the returned value is not-null EVERY
+  // TIME before accessing it or risk crashing!  Once dereferenced, the pointer
+  // is safe to reuse forever.
+  //
+  // NOTE: Though this method will guarantee that an object of the specified
+  // type can be accessed without going outside the bounds of the memory
+  // segment, it makes no guarantees of the validity of the data within the
+  // object itself. If it is expected that the contents of the segment could
+  // be compromised with malicious intent, the object must be hardened as well.
+  template <typename T>
+  T* GetAsObject(Reference ref, uint32_t type_id) {
+    // Though the persistent data may be "volatile" in that it is shared with
+    // other processes, it is not necessarily the case. The internal
+    // "volatile" designation is discarded here so as to not propagate the
+    // viral nature of that keyword to the caller.
+    return const_cast<T*>(
+        reinterpret_cast<volatile T*>(GetBlockData(ref, type_id, sizeof(T))));
+  }
+
+  // Get the number of bytes allocated to a block. This is useful when storing
+  // arrays in order to validate the ending boundary. The returned value will
+  // include any padding added to achieve the required alignment and so could
+  // be larger than given in the original Allocate() request.
+  size_t GetAllocSize(Reference ref);
+
+  // Access the internal "type" of an object. This generally isn't necessary
+  // but can be used to "clear" the type and so effectively mark it as deleted
+  // even though the memory stays valid and allocated.
+  uint32_t GetType(Reference ref);
+  void SetType(Reference ref, uint32_t type_id);
+
+  // Reserve space in the memory segment of the desired |size| and |type_id|.
+  // A return value of zero indicates the allocation failed, otherwise the
+  // returned reference can be used by any process to get a real pointer via
+  // the GetAsObject() call.
+  Reference Allocate(size_t size, uint32_t type_id);
+
+  // Allocated objects can be added to an internal list that can then be
+  // iterated over by other processes. If an allocated object can be found
+  // another way, such as by having its reference within a different object
+  // that will be made iterable, then this call is not necessary. This always
+  // succeeds unless corruption is detected; check IsCorrupted() to find out.
+  void MakeIterable(Reference ref);
+
+  // Get the information about the amount of free space in the allocator. The
+  // amount of free space should be treated as approximate due to extras from
+  // alignment and metadata. Concurrent allocations from other threads will
+  // also make the true amount less than what is reported.
+  void GetMemoryInfo(MemoryInfo* meminfo);
+
+  // Iterating uses a |state| structure (initialized by CreateIterator) and
+  // returns both the reference to the object as well as the |type_id| of
+  // that object. A zero return value indicates there are currently no more
+  // objects to be found but future attempts can be made without having to
+  // reset the iterator to "first".
+  void CreateIterator(Iterator* state);
+  Reference GetNextIterable(Iterator* state, uint32_t* type_id);
+
+  // If there is some indication that the memory has become corrupted,
+  // calling this will attempt to prevent further damage by indicating to
+  // all processes that something is not as expected.
+  void SetCorrupt();
+
+  // This can be called to determine if corruption has been detected in the
+  // segment, possibly my a malicious actor. Once detected, future allocations
+  // will fail and iteration may not locate all objects.
+  bool IsCorrupt();
+
+  // Flag set if an allocation has failed because the memory segment was full.
+  bool IsFull();
+
+  // Update static-state histograms. This should be called on a periodic basis
+  // to record such things as how much of the total space is used.
+  void UpdateStaticHistograms();
+
+ protected:
+  volatile char* const mem_base_;  // Memory base. (char so sizeof guaranteed 1)
+  const uint32_t mem_size_;        // Size of entire memory segment.
+  const uint32_t mem_page_;        // Page size allocations shouldn't cross.
+
+ private:
+  struct SharedMetadata;
+  struct BlockHeader;
+  static const Reference kReferenceQueue;
+  static const Reference kReferenceNull;
+
+  // The shared metadata is always located at the top of the memory segment.
+  // This convenience function eliminates constant casting of the base pointer
+  // within the code.
+  volatile SharedMetadata* shared_meta() {
+    return reinterpret_cast<volatile SharedMetadata*>(mem_base_);
+  }
+
+  volatile BlockHeader* GetBlock(Reference ref, uint32_t type_id, uint32_t size,
+                                 bool queue_ok, bool free_ok);
+  volatile void* GetBlockData(Reference ref, uint32_t type_id, uint32_t size);
+
+  std::atomic<bool> corrupt_;        // Local version of "corrupted" flag.
+
+  HistogramBase* allocs_histogram_;  // Histogram recording allocs.
+  HistogramBase* used_histogram_;    // Histogram recording used space.
+
+  FRIEND_TEST_ALL_PREFIXES(PersistentMemoryAllocatorTest, AllocateAndIterate);
+  DISALLOW_COPY_AND_ASSIGN(PersistentMemoryAllocator);
+};
+
+
+// This allocator uses a local memory block it allocates from the general
+// heap. It is generally used when some kind of "death rattle" handler will
+// save the contents to persistent storage during process shutdown. It is
+// also useful for testing.
+class BASE_EXPORT LocalPersistentMemoryAllocator
+    : public PersistentMemoryAllocator {
+ public:
+  LocalPersistentMemoryAllocator(size_t size, const std::string& name);
+  ~LocalPersistentMemoryAllocator();
+
+ private:
+  DISALLOW_COPY_AND_ASSIGN(LocalPersistentMemoryAllocator);
+};
+
+}  // namespace base
+
+#endif  // BASE_MEMORY_PERSISTENT_MEMORY_ALLOCATOR_H_