LRUDict: add timestamps

LRUDict: add timestamps

lru.LRUDict currently stores the order of items, but not time when an
item was used most recently.

Add timestamps to LRUDict internal representation and update persistence
format. Preserve ability to load a dict from the old format for backward
compatibility.

Also remove lru.LRUDict.batch_insert_oldest because it complicates this
CL. It wasn't used.

R=maruel@chromium.org
BUG=654562
Committed: https://github.com/luci/luci-py/commit/c40ca4d446515a4a97cdfac59a6aa3a1e3b0a781

[nodir, 2016-10-12 06:09:53]

PTAL

[M-A Ruel, 2016-10-12 19:47:10]

https://codereview.chromium.org/2409103003/diff/20001/client/utils/lru.py
File client/utils/lru.py (right):

https://codereview.chromium.org/2409103003/diff/20001/client/utils/lru.py#new...
client/utils/lru.py:16: (key, (value, timestamp)) pairs in order they are
I kind of disagree here; value just happens to be a tuple, it's still a value.

I'm actually confused as to why this class is being modified. Why does it need
to know about the fact the value is a tuple(int, float) ?

https://codereview.chromium.org/2409103003/diff/20001/client/utils/lru.py#new...
client/utils/lru.py:156: self._items[key] = (self._items.pop(key)[0],
self.time_fn())
This is only the real place where the value has to be mutated because of
ordering.

But call sites of touch() could be changed for a add(key, (size, time.time())
and that would work, and I feel it's much less complex.

wdyt?

[nodir, 2016-10-12 20:33:31]

indeed

[nodir, 2016-10-13 19:35:09]

I've writing a CL where I introduce named_cache.CacheManager classes that
manages named cache. I need an ordered dict with timestamps there that can be
saved to and loaded from a file, which is essentially this. Otherwise LRUDict is
deleted in https://codereview.chromium.org/2414543003/

now I think that we should resurrect this CL and use in both DiskCache and
named_cache.CacheManager

[nodir, 2016-10-13 19:35:17]

Description was changed from

==========
LRUDict: add timestamps

lru.LRUDict currently stores the order of items, but not time when an
item was used most recently.

Add timestamps to LRUDict internal representation and update persistence
format. Preserve ability to load a dict from the old format for backward
compatibility.

Also remove lru.LRUDict.batch_insert_oldest because it complicates this
CL. It wasn't used.

R=maruel@chromium.org
BUG=654562
==========

to

==========
LRUDict: add timestamps

lru.LRUDict currently stores the order of items, but not time when an
item was used most recently.

Add timestamps to LRUDict internal representation and update persistence
format. Preserve ability to load a dict from the old format for backward
compatibility.

Also remove lru.LRUDict.batch_insert_oldest because it complicates this
CL. It wasn't used.

R=maruel@chromium.org
BUG=654562
==========

[nodir, 2016-10-17 18:09:11]

On 2016/10/13 19:35:09, nodir wrote:
> I've writing a CL where I introduce named_cache.CacheManager classes that
> manages named cache. I need an ordered dict with timestamps there that can be
> saved to and loaded from a file, which is essentially this. Otherwise LRUDict
is
> deleted in https://codereview.chromium.org/2414543003/
> 
> now I think that we should resurrect this CL and use in both DiskCache and
> named_cache.CacheManager

ping

[M-A Ruel, 2016-10-18 00:43:51]

lgtm

[nodir, 2016-10-18 22:34:13]

The CQ bit was checked by nodir@chromium.org

[commit-bot: I haz the power, 2016-10-18 22:34:20]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-18 22:39:10]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-18 22:39:11]

Try jobs failed on following builders:
  Luci-py Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/31f2c22f63a84810)

[nodir, 2016-10-18 23:04:28]

The CQ bit was checked by nodir@chromium.org

[nodir, 2016-10-18 23:04:28]

The patchset sent to the CQ was uploaded after l-g-t-m from maruel@chromium.org
Link to the patchset: https://codereview.chromium.org/2409103003/#ps40001
(title: "fix test")

[commit-bot: I haz the power, 2016-10-18 23:04:34]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[nodir, 2016-10-18 23:26:14]

The CQ bit was unchecked by nodir@chromium.org

[nodir, 2016-10-18 23:26:28]

The CQ bit was checked by nodir@chromium.org

[commit-bot: I haz the power, 2016-10-18 23:26:35]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-18 23:37:30]

Description was changed from

==========
LRUDict: add timestamps

lru.LRUDict currently stores the order of items, but not time when an
item was used most recently.

Add timestamps to LRUDict internal representation and update persistence
format. Preserve ability to load a dict from the old format for backward
compatibility.

Also remove lru.LRUDict.batch_insert_oldest because it complicates this
CL. It wasn't used.

R=maruel@chromium.org
BUG=654562
==========

to

==========
LRUDict: add timestamps

lru.LRUDict currently stores the order of items, but not time when an
item was used most recently.

Add timestamps to LRUDict internal representation and update persistence
format. Preserve ability to load a dict from the old format for backward
compatibility.

Also remove lru.LRUDict.batch_insert_oldest because it complicates this
CL. It wasn't used.

R=maruel@chromium.org
BUG=654562

Committed:
https://github.com/luci/luci-py/commit/c40ca4d446515a4a97cdfac59a6aa3a1e3b0a781
==========

[commit-bot: I haz the power, 2016-10-18 23:37:31]

Committed patchset #3 (id:40001) as
https://github.com/luci/luci-py/commit/c40ca4d446515a4a97cdfac59a6aa3a1e3b0a781