OLD | NEW |
1 This document describes how malloc / new calls are routed in the various Chrome | 1 This document describes how malloc / new calls are routed in the various Chrome |
2 platforms. | 2 platforms. |
3 | 3 |
4 Bare in mind that the chromium codebase does not always just use `malloc()`. | 4 Bare in mind that the chromium codebase does not always just use `malloc()`. |
5 Some examples: | 5 Some examples: |
6 - Large parts of the renderer (Blink) use two home-brewed allocators, | 6 - Large parts of the renderer (Blink) use two home-brewed allocators, |
7 PartitionAlloc and BlinkGC (Oilpan). | 7 PartitionAlloc and BlinkGC (Oilpan). |
8 - Some subsystems, such as the V8 JavaScript engine, handle memory management | 8 - Some subsystems, such as the V8 JavaScript engine, handle memory management |
9 autonomously. | 9 autonomously. |
10 - Various parts of the codebase use abstractions such as `SharedMemory` or | 10 - Various parts of the codebase use abstractions such as `SharedMemory` or |
(...skipping 67 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
78 | 78 |
79 The tcmalloc library originates outside of Chromium and exists in | 79 The tcmalloc library originates outside of Chromium and exists in |
80 `../../third_party/tcmalloc` (currently, the actual location is defined in the | 80 `../../third_party/tcmalloc` (currently, the actual location is defined in the |
81 allocator.gyp file). The third party sources use a vendor-branch SCM pattern to | 81 allocator.gyp file). The third party sources use a vendor-branch SCM pattern to |
82 track Chromium-specific changes independently from upstream changes. | 82 track Chromium-specific changes independently from upstream changes. |
83 | 83 |
84 The general intent is to push local changes upstream so that over | 84 The general intent is to push local changes upstream so that over |
85 time we no longer need any forked files. | 85 time we no longer need any forked files. |
86 | 86 |
87 | 87 |
| 88 Unified allocator shim |
| 89 ---------------------- |
| 90 On most platform, Chrome overrides the malloc / operator new symbols (and |
| 91 corresponding free / delete and other variants). This is to enforce security |
| 92 checks and lately to enable the |
| 93 [memory-infra heap profiler][url-memory-infra-heap-profiler]. |
| 94 Historically each platform had its special logic for defining the allocator |
| 95 symbols in different places of the codebase. The unified allocator shim is |
| 96 a project aimed to unify the symbol definition and allocator routing logic in |
| 97 a central place. |
| 98 |
| 99 - Full documentation: [Allocator shim design doc][url-allocator-shim]. |
| 100 - Current state: Available and enabled by default on Linux and CrOS. |
| 101 - Tracking bug: [https://crbug.com/550886][crbug.com/550886]. |
| 102 - Build-time flag: `use_experimental_allocator_shim`. |
| 103 |
| 104 **Overview of the unified allocator shim** |
| 105 The allocator shim consists of three stages: |
| 106 ``` |
| 107 +-------------------------+ +-----------------------+ +----------------+ |
| 108 | malloc & friends | -> | shim layer | -> | Routing to | |
| 109 | symbols definition | | implementation | | allocator | |
| 110 +-------------------------+ +-----------------------+ +----------------+ |
| 111 | - libc symbols (malloc, | | - Security checks | | - tcmalloc | |
| 112 | calloc, free, ...) | | - Chain of dispatchers| | - glibc | |
| 113 | - C++ symbols (operator | | that can intercept | | - Android | |
| 114 | new, delete, ...) | | and override | | bionic | |
| 115 | - glibc weak symbols | | allocations | | - WinHeap | |
| 116 | (__libc_malloc, ...) | +-----------------------+ +----------------+ |
| 117 +-------------------------+ |
| 118 ``` |
| 119 |
| 120 **1. malloc symbols definition** |
| 121 This stage takes care of overriding the symbols `malloc`, `free`, |
| 122 `operator new`, `operator delete` and friends and routing those calls inside the |
| 123 allocator shim (next point). |
| 124 This is taken care of by the headers in `allocator_shim_override_*`. |
| 125 |
| 126 *On Linux/CrOS*: the allocator symbols are defined as exported global symbols |
| 127 in `allocator_shim_override_libc_symbols.h` (for `malloc`, `free` and friends) |
| 128 and in `allocator_shim_override_cpp_symbols.h` (for `operator new`, |
| 129 `operator delete` and friends). |
| 130 This enables proper interposition of malloc symbols referenced by the main |
| 131 executable and any third party libraries. Symbol resolution on Linux is a breadt
h first search that starts from the root link unit, that is the executable |
| 132 (see EXECUTABLE AND LINKABLE FORMAT (ELF) - Portable Formats Specification). |
| 133 Additionally, when tcmalloc is the default allocator, some extra glibc symbols |
| 134 are also defined in `allocator_shim_override_glibc_weak_symbols.h`, for subtle |
| 135 reasons explained in that file. |
| 136 The Linux/CrOS shim was introduced by |
| 137 [crrev.com/1675143004](https://crrev.com/1675143004). |
| 138 |
| 139 *On Android*: load-time symbol interposition (unlike the Linux/CrOS case) is not |
| 140 possible. This is because Android processes are `fork()`-ed from the Android |
| 141 zygote, which pre-loads libc.so and only later native code gets loaded via |
| 142 `dlopen()` (symbols from `dlopen()`-ed libraries get a different resolution |
| 143 scope). |
| 144 In this case, the approach instead of wrapping symbol resolution at link time |
| 145 (i.e. during the build), via the `--Wl,-wrap,malloc` linker flag. |
| 146 The use of this wrapping flag causes: |
| 147 - All references to allocator symbols in the Chrome codebase to be rewritten as |
| 148 references to `__wrap_malloc` and friends. The `__wrap_malloc` symbols are |
| 149 defined in the `allocator_shim_override_linker_wrapped_symbols.h` and |
| 150 route allocator calls inside the shim layer. |
| 151 - The reference to the original `malloc` symbols (which typically is defined by |
| 152 the system's libc.so) are accessible via the special `__real_malloc` and |
| 153 friends symbols (which will be relocated, at load time, against `malloc`). |
| 154 |
| 155 In summary, this approach is transparent to the dynamic loader, which still sees |
| 156 undefined symbol references to malloc symbols. |
| 157 These symbols will be resolved against libc.so as usual. |
| 158 More details in [crrev.com/1719433002](https://crrev.com/1719433002). |
| 159 |
| 160 **2. Shim layer implementation** |
| 161 This stage contains the actual shim implementation. This consists of: |
| 162 - A singly linked list of dispatchers (structs with function pointers to `malloc
`-like functions). Dispatchers can be dynamically inserted at runtime |
| 163 (using the `InsertAllocatorDispatch` API). They can intercept and override |
| 164 allocator calls. |
| 165 - The security checks (suicide on malloc-failure via `std::new_handler`, etc). |
| 166 This happens inside `allocator_shim.cc` |
| 167 |
| 168 **3. Final allocator routing** |
| 169 The final element of the aforementioned dispatcher chain is statically defined |
| 170 at build time and ultimately routes the allocator calls to the actual allocator |
| 171 (as described in the *Background* section above). This is taken care of by the |
| 172 headers in `allocator_shim_default_dispatch_to_*` files. |
| 173 |
| 174 |
88 Appendixes | 175 Appendixes |
89 ---------- | 176 ---------- |
90 **How does the Windows shim layer replace the malloc symbols?** | 177 **How does the Windows shim layer replace the malloc symbols?** |
91 The mechanism for hooking LIBCMT in Windows is rather tricky. The core | 178 The mechanism for hooking LIBCMT in Windows is rather tricky. The core |
92 problem is that by default, the Windows library does not declare malloc and | 179 problem is that by default, the Windows library does not declare malloc and |
93 free as weak symbols. Because of this, they cannot be overridden. To work | 180 free as weak symbols. Because of this, they cannot be overridden. To work |
94 around this, we start with the LIBCMT.LIB, and manually remove all allocator | 181 around this, we start with the LIBCMT.LIB, and manually remove all allocator |
95 related functions from it using the visual studio library tool. Once removed, | 182 related functions from it using the visual studio library tool. Once removed, |
96 we can now link against the library and provide custom versions of the | 183 we can now link against the library and provide custom versions of the |
97 allocator related functionality. | 184 allocator related functionality. |
98 See the script `preb_libc.py` in this folder. | 185 See the script `preb_libc.py` in this folder. |
99 | 186 |
100 Related links | 187 Related links |
101 ------------- | 188 ------------- |
102 - [Allocator Cleanup Doc - Jan 2016][url-allocator-cleanup] | 189 - [Unified allocator shim doc - Feb 2016][url-allocator-shim] |
| 190 - [Allocator cleanup doc - Jan 2016][url-allocator-cleanup] |
103 - [Proposal to use PartitionAlloc as default allocator](https://crbug.com/339604
) | 191 - [Proposal to use PartitionAlloc as default allocator](https://crbug.com/339604
) |
104 - [Memory-Infra: Tools to profile memory usage in Chrome](components/tracing/doc
s/memory_infra.md) | 192 - [Memory-Infra: Tools to profile memory usage in Chrome](components/tracing/doc
s/memory_infra.md) |
105 | 193 |
106 [url-allocator-cleanup]: https://docs.google.com/document/d/1V77Kgp_4tfaaWPEZVxN
evoD02wXiatnAv7Ssgr0hmjg/edit?usp=sharing | 194 [url-allocator-cleanup]: https://docs.google.com/document/d/1V77Kgp_4tfaaWPEZVxN
evoD02wXiatnAv7Ssgr0hmjg/edit?usp=sharing |
| 195 [url-memory-infra-heap-profiler]: components/tracing/docs/heap_profiler.md |
| 196 [url-allocator-shim]: https://docs.google.com/document/d/1yKlO1AO4XjpDad9rjcBOI1
5EKdAGsuGO_IeZy0g0kxo/edit?usp=sharing |
OLD | NEW |