[qcms] Fix integer overflow in lut_inverse_interp16

[qcms] Fix integer overflow in lut_inverse_interp16

Inverting a large input gamma table that is degenerate (has poles and
zeros) can cause an integer overflow in lut_inverse_interp16 (noticed
by UBSan, see bug) when computing the binary search range.

Use float to compute the search range to avoid integer overflow, then
truncate to int when setting the range variables.

BUG=711751
Review-Url: https://codereview.chromium.org/2825623002
Cr-Commit-Position: refs/heads/master@{#466059}
Committed: https://chromium.googlesource.com/chromium/src/+/e4ce3b5145a1fb2a038ccd071bcab45799a3895e

[Noel Gordon, 2017-04-18 01:37:12]

The CQ bit was checked by noel@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-18 01:37:31]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Noel Gordon, 2017-04-18 01:38:16]

noel@chromium.org changed reviewers:
+ hubbe@chromium.org

[Noel Gordon, 2017-04-18 01:38:16]

PTAL.

[commit-bot: I haz the power, 2017-04-18 02:47:47]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-18 02:47:47]

Dry run: This issue passed the CQ dry run.

[Noel Gordon, 2017-04-18 10:46:50]

Description was changed from

==========
[qcms] Fix integer overflow in lut_inverse_interp16

Inverting a large input gamma table that is degenerate (has poles and
zeros) can cause an integer overflow in lut_inverse_interp16 (noticed
by UBSan, see bug) when computing the binary search range. Use float
to compute the search range to avoid integer overflow.

BUG=711751
==========

to

==========
[qcms] Fix integer overflow in lut_inverse_interp16

Inverting a large input gamma table that is degenerate (has poles and
zeros) can cause an integer overflow in lut_inverse_interp16 (noticed
by UBSan, see bug) when computing the binary search range.

Use float to compute the search range to avoid integer overflow, then
truncate to int when setting the range variables.

BUG=711751
==========

[Noel Gordon, 2017-04-20 15:32:17]

noel@chromium.org changed reviewers:
+ robert.bradford@intel.com
- hubbe@chromium.org

[Noel Gordon, 2017-04-20 15:32:18]

Maybe Robert could do this one.

[robert.bradford, 2017-04-20 16:13:26]

lgtm. I do wonder whether there needs to be a check for sane LUT lengths too
(not here specifically, but in general.)

[Noel Gordon, 2017-04-20 16:21:02]

On 2017/04/20 16:13:26, robert.bradford wrote:
> lgtm. I do wonder whether there needs to be a check for sane LUT lengths too
> (not here specifically, but in general.)

I think our QCMS fuzzer would detect insane LUT length and advise, but a limit
on how much memory a LUT could make us allocated might also be a good idea (in
general).

[Noel Gordon, 2017-04-20 16:21:11]

The CQ bit was checked by noel@chromium.org

[commit-bot: I haz the power, 2017-04-20 16:21:49]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-20 17:49:23]

CQ is committing da patch.

Bot data: {"patchset_id": 1, "attempt_start_ts": 1492705271847580, "parent_rev":
"e6fccd69c039c3799ae8be0cade4f040f256fefd", "commit_rev":
"e4ce3b5145a1fb2a038ccd071bcab45799a3895e"}

[commit-bot: I haz the power, 2017-04-20 17:49:33]

Description was changed from

==========
[qcms] Fix integer overflow in lut_inverse_interp16

Inverting a large input gamma table that is degenerate (has poles and
zeros) can cause an integer overflow in lut_inverse_interp16 (noticed
by UBSan, see bug) when computing the binary search range.

Use float to compute the search range to avoid integer overflow, then
truncate to int when setting the range variables.

BUG=711751
==========

to

==========
[qcms] Fix integer overflow in lut_inverse_interp16

Inverting a large input gamma table that is degenerate (has poles and
zeros) can cause an integer overflow in lut_inverse_interp16 (noticed
by UBSan, see bug) when computing the binary search range.

Use float to compute the search range to avoid integer overflow, then
truncate to int when setting the range variables.

BUG=711751

Review-Url: https://codereview.chromium.org/2825623002
Cr-Commit-Position: refs/heads/master@{#466059}
Committed:
https://chromium.googlesource.com/chromium/src/+/e4ce3b5145a1fb2a038ccd071bca...
==========

[commit-bot: I haz the power, 2017-04-20 17:49:34]

Committed patchset #1 (id:1) as
https://chromium.googlesource.com/chromium/src/+/e4ce3b5145a1fb2a038ccd071bca...