OLD | NEW |
1 # Chrome's Blob Storage System Design | 1 # Chrome's Blob Storage System Design |
2 | 2 |
3 Elaboration of the blob storage system in Chrome. | 3 Elaboration of the blob storage system in Chrome. |
4 | 4 |
5 # What are blobs? | 5 # What are blobs? |
6 | 6 |
7 Please see the [FileAPI Spec](https://www.w3.org/TR/FileAPI/) for the full | 7 Please see the [FileAPI Spec](https://www.w3.org/TR/FileAPI/) for the full |
8 specification for Blobs, or [Mozilla's Blob documentation]( | 8 specification for Blobs, or [Mozilla's Blob documentation]( |
9 https://developer.mozilla.org/en-US/docs/Web/API/Blob) for a description of how | 9 https://developer.mozilla.org/en-US/docs/Web/API/Blob) for a description of how |
10 Blobs are used in the Web Platform in general. For the purposes of this | 10 Blobs are used in the Web Platform in general. For the purposes of this |
(...skipping 64 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
75 size. | 75 size. |
76 | 76 |
77 # Blob Storage Limits | 77 # Blob Storage Limits |
78 | 78 |
79 We calculate the storage limits [here]( | 79 We calculate the storage limits [here]( |
80 https://cs.chromium.org/chromium/src/storage/browser/blob/blob_memory_controller
.cc?q=CalculateBlobStorageLimitsImpl&sq=package:chromium). | 80 https://cs.chromium.org/chromium/src/storage/browser/blob/blob_memory_controller
.cc?q=CalculateBlobStorageLimitsImpl&sq=package:chromium). |
81 | 81 |
82 **In-Memory Storage Limit** | 82 **In-Memory Storage Limit** |
83 | 83 |
84 * If the architecture is x64 and NOT Chrome OS or Android: `2GB` | 84 * If the architecture is x64 and NOT Chrome OS or Android: `2GB` |
85 * Otherwise: `total_physical_memory / 5` | 85 * If Chrome OS: `total_physical_memory / 5` |
| 86 * If Android: `total_physical_memory / 100` |
| 87 |
86 | 88 |
87 **Disk Storage Limit** | 89 **Disk Storage Limit** |
88 | 90 |
89 * If Chrome OS: `disk_size / 2` | 91 * If Chrome OS: `disk_size / 2` |
90 * If Android: `disk_size / 20` | 92 * If Android: `6 * disk_size / 100` |
91 * Else: `disk_size / 10` | 93 * Else: `disk_size / 10` |
92 | 94 |
93 Note: Chrome OS's disk is part of the user partition, which is separate from the | 95 Note: Chrome OS's disk is part of the user partition, which is separate from the |
94 system partition. | 96 system partition. |
95 | 97 |
96 **Minimum Disk Availability** | 98 **Minimum Disk Availability** |
97 | 99 |
98 We limit our disk limit to accomidate a minimum disk availability. The equation | 100 We limit our disk limit to accomidate a minimum disk availability. The equation |
99 we use is: | 101 we use is: |
100 | 102 |
101 `min_disk_availability = in_memory_limit * 2` | 103 `min_disk_availability = in_memory_limit * 2` |
102 | 104 |
103 ## Example Limits | 105 ## Example Limits |
104 | 106 |
105 (All sizes in GB) | |
106 | |
107 | Device | Ram | In-Memory Limit | Disk | Disk Limit | Min Disk Availability | | 107 | Device | Ram | In-Memory Limit | Disk | Disk Limit | Min Disk Availability | |
108 | --- | --- | --- | --- | --- | --- | | 108 | --- | --- | --- | --- | --- | --- | |
109 | Cast | 0.5 | 0.1 | 0 | 0 | 0 | | 109 | Cast | 512 MB | 102 MB | 0 | 0 | 0 | |
110 | Android Minimal | 0.5 | 0.1 | 8 | 0.4 | 0.2 | | 110 | Android Minimal | 512 MB | 5 MB | 8 GB | 491 MB | 10 MB | |
111 | Android Fat | 2 | 0.4 | 32 | 1.5 | 0.8 | | 111 | Android Fat | 2 GB | 20 MB | 32 GB | 1.9 GB | 40 MB | |
112 | CrOS | 2 | 0.4 | 8 | 4 | 0.8 | | 112 | CrOS | 2 GB | 409 MB | 8 GB | 4 GB | 0.8 GB | |
113 | Desktop 32 | 3 | 0.6 | 500 | 50 | 1.2 | | 113 | Desktop 32 | 3 GB | 614 MB | 500 GB | 50 GB | 1.2 GB | |
114 | Desktop 64 | 4 | 2 | 500 | 50 | 4 | | 114 | Desktop 64 | 4 GB | 2 GB | 500 GB | 50 GB | 4 GB | |
115 | 115 |
116 # Common Pitfalls | 116 # Common Pitfalls |
117 | 117 |
118 ## Creating Large Blobs Too Fast | 118 ## Creating Large Blobs Too Fast |
119 | 119 |
120 Creating a lot of blobs, especially if they are very large blobs, can cause | 120 Creating a lot of blobs, especially if they are very large blobs, can cause |
121 the renderer memory to grow too fast and result in an OOM on the renderer side. | 121 the renderer memory to grow too fast and result in an OOM on the renderer side. |
122 This is because the renderer temporarily stores the blob data while it waits | 122 This is because the renderer temporarily stores the blob data while it waits |
123 for the browser to request it. Meanwhile, Javascript can continue executing. | 123 for the browser to request it. Meanwhile, Javascript can continue executing. |
124 Transfering the data can take a lot of time if the blob is large enough to save | 124 Transfering the data can take a lot of time if the blob is large enough to save |
125 it directly to a file, as this means we need to wait for disk operations before | 125 it directly to a file, as this means we need to wait for disk operations before |
126 the renderer can get rid of the data. | 126 the renderer can get rid of the data. |
127 | 127 |
128 ## Leaking Blob References | 128 ## Leaking Blob References |
129 | 129 |
130 If the blob object in Javascript is kept around, then the data will never be | 130 If the blob object in Javascript is kept around, then the data will never be |
131 cleaned up in the backend. This will unnecessarily us memory, so make sure to | 131 cleaned up in the backend. This will unnecessarily use memory, so make sure to |
132 dereference blob objects if they are no longer needed. | 132 dereference blob objects if they are no longer needed. |
133 | 133 |
134 Similarily if a URL is created for a blob, this will keep the blob data around | 134 Similarily if a URL is created for a blob, this will keep the blob data around |
135 until the URL is revoked (and the blob object is dereferenced). However, the | 135 until the URL is revoked (and the blob object is dereferenced). However, the |
136 URL is automatically revoked when the browser context is destroyed. | 136 URL is automatically revoked when the browser context is destroyed. |
137 | 137 |
138 # How to use Blobs (Browser-side) | 138 # How to use Blobs (Browser-side) |
139 | 139 |
140 ## Building | 140 ## Building |
141 All blob interaction should go through the `BlobStorageContext`. Blobs are | 141 All blob interaction should go through the `BlobStorageContext`. Blobs are |
(...skipping 197 matching lines...) Expand 10 before | Expand all | Expand 10 after Loading... |
339 | 339 |
340 The `BlobMemoryController` is responsable for: | 340 The `BlobMemoryController` is responsable for: |
341 | 341 |
342 1. Determining storage quota limits for files and memory, including restricting | 342 1. Determining storage quota limits for files and memory, including restricting |
343 file quota when disk space is low. | 343 file quota when disk space is low. |
344 2. Determining whether a blob can fit and the transportation strategy to use. | 344 2. Determining whether a blob can fit and the transportation strategy to use. |
345 3. Tracking memory quota. | 345 3. Tracking memory quota. |
346 4. Tracking file quota and creating files. | 346 4. Tracking file quota and creating files. |
347 5. Accumulating and evicting old blob data to files to disk. | 347 5. Accumulating and evicting old blob data to files to disk. |
348 | 348 |
OLD | NEW |