color: general transformation -> affine transformation

color: general transformation -> affine transformation

Implement a way to get an affine approximation of a color transformation.

The plan is to update cc/output/gl_output.cc to use this method to
calculate the correct coefficients for converting video frames to the
screen color space. We'll test the affine transformation with a set of
sample colors to see if the affine transformation is close enough, and
if not, we will fall back on using a 3D LUT.

The implementation itself uses a least-squares method which should return
a reasonable approximation for almost any transformation.

BUG=622133

[hubbe, 2016-12-08 20:12:23]

The CQ bit was checked by hubbe@chromium.org to run a CQ dry run

[hubbe, 2016-12-08 20:12:24]

hubbe@chromium.org changed reviewers:
+ ccameron@chromium.org

[commit-bot: I haz the power, 2016-12-08 20:13:34]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[ccameron, 2016-12-08 20:40:55]

My bias here is to limit the number of conversion paths that we support, so I'm
biased against adding this.

Is this to save power? Or is it to add compatibility with overlay hardware?

[hubbe, 2016-12-08 20:59:15]

On 2016/12/08 20:40:55, ccameron wrote:
> My bias here is to limit the number of conversion paths that we support, so
I'm
> biased against adding this.
> 
> Is this to save power? Or is it to add compatibility with overlay hardware?

The primary goal is to replace the hard-coded coefficients in gl_output.cc with
computed ones which would let it handle a much larger set of inputs and outputs.

Eventually I want the logic to look something like:

if (overlays_available && color_error_caused_by_overalays(input, output) < 5%)
  use_overlays();
else if (color_error_caused_by_using_affine_approximation(intput, output) < 1%)
  use_affine();  // This a slightly updated version of what's in gl_output.cc
today
else
  use_lut();

The 5% and 1% are made up, the point is that we'll allow larger sacrifices in
accuracy for overlays since they are also a lot more profitable in terms of
power.
The goal would be for most video (bt709 on srgb screens) to hit the overlay path
when possible, and the affine path when overalys are not possible, but if you
have a different
type of screen, or an unusual video of some sort, you might have to fall back
to the LUT.

[ccameron, 2016-12-08 21:07:43]

On 2016/12/08 20:59:15, hubbe wrote:
> On 2016/12/08 20:40:55, ccameron wrote:
> > My bias here is to limit the number of conversion paths that we support, so
> I'm
> > biased against adding this.
> > 
> > Is this to save power? Or is it to add compatibility with overlay hardware?
> 
> The primary goal is to replace the hard-coded coefficients in gl_output.cc
with
> computed ones which would let it handle a much larger set of inputs and
outputs.
> 
> Eventually I want the logic to look something like:
> 
> if (overlays_available && color_error_caused_by_overalays(input, output) < 5%)
>   use_overlays();
> else if (color_error_caused_by_using_affine_approximation(intput, output) <
1%)
>   use_affine();  // This a slightly updated version of what's in gl_output.cc
> today
> else
>   use_lut();
> 
> The 5% and 1% are made up, the point is that we'll allow larger sacrifices in
> accuracy for overlays since they are also a lot more profitable in terms of
> power.
> The goal would be for most video (bt709 on srgb screens) to hit the overlay
path
> when possible, and the affine path when overalys are not possible, but if you
> have a different
> type of screen, or an unusual video of some sort, you might have to fall back
> to the LUT.

Hmm, my preferences are different here.

WRT overlays, the power difference is so large, we should change our color
behavior to match whatever overlays are capable of (no matter how wrong).

WRT affine, I don't think that the power savings are substantial enough to
warrant adding a separate path.

[commit-bot: I haz the power, 2016-12-08 21:09:08]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-12-08 21:09:09]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[hubbe, 2016-12-08 21:19:20]

On 2016/12/08 21:07:43, ccameron wrote:
> On 2016/12/08 20:59:15, hubbe wrote:
> > On 2016/12/08 20:40:55, ccameron wrote:
> > > My bias here is to limit the number of conversion paths that we support,
so
> > I'm
> > > biased against adding this.
> > > 
> > > Is this to save power? Or is it to add compatibility with overlay
hardware?
> > 
> > The primary goal is to replace the hard-coded coefficients in gl_output.cc
> with
> > computed ones which would let it handle a much larger set of inputs and
> outputs.
> > 
> > Eventually I want the logic to look something like:
> > 
> > if (overlays_available && color_error_caused_by_overalays(input, output) <
5%)
> >   use_overlays();
> > else if (color_error_caused_by_using_affine_approximation(intput, output) <
> 1%)
> >   use_affine();  // This a slightly updated version of what's in
gl_output.cc
> > today
> > else
> >   use_lut();
> > 
> > The 5% and 1% are made up, the point is that we'll allow larger sacrifices
in
> > accuracy for overlays since they are also a lot more profitable in terms of
> > power.
> > The goal would be for most video (bt709 on srgb screens) to hit the overlay
> path
> > when possible, and the affine path when overalys are not possible, but if
you
> > have a different
> > type of screen, or an unusual video of some sort, you might have to fall
back
> > to the LUT.
> 
> Hmm, my preferences are different here.
> 
> WRT overlays, the power difference is so large, we should change our color
> behavior to match whatever overlays are capable of (no matter how wrong).
> 

I think we agree in principle: Overlays save a lot of power, and we should
*almost* always use them.

However, there are some cases when we should not, like if the input or output
is HDR for example. (Unless both input AND output are HDR...) Overlays also
vary in capabilities, some may have the ability to specify a translation matrix,
some may have an enum, or some other way to describe what will happen.

Using an error function to make the decision if we should use overlays or not
seems much easier than trying to hard-code it based on input and output
properties.

> WRT affine, I don't think that the power savings are substantial enough to
> warrant adding a separate path.

My measurements says it's 1-4% difference depending on hardware. Not much,
but a lot of people are super-paranoid about power usage nowadays.

[ccameron, 2016-12-09 00:30:50]

I don't want us to continue down this direction.

In particular, I want to add support for analytic transfer functions (of the
SkColorSpaceTransferFn), and stop there. In particular
- use the analytic conversions whenever possible
- fall back to using a LUT in the remaining (hopefully vanishingly few) cases

On 2016/12/08 21:19:20, hubbe wrote:
> On 2016/12/08 21:07:43, ccameron wrote:
> > On 2016/12/08 20:59:15, hubbe wrote:
> > > On 2016/12/08 20:40:55, ccameron wrote:
> > > > My bias here is to limit the number of conversion paths that we support,
> so
> > > I'm
> > > > biased against adding this.
> > > > 
> > > > Is this to save power? Or is it to add compatibility with overlay
> hardware?
> > > 
> > > The primary goal is to replace the hard-coded coefficients in gl_output.cc
> > with
> > > computed ones which would let it handle a much larger set of inputs and
> > outputs.
> > > 
> > > Eventually I want the logic to look something like:
> > > 
> > > if (overlays_available && color_error_caused_by_overalays(input, output) <
> 5%)
> > >   use_overlays();
> > > else if (color_error_caused_by_using_affine_approximation(intput, output)
<
> > 1%)
> > >   use_affine();  // This a slightly updated version of what's in
> gl_output.cc
> > > today
> > > else
> > >   use_lut();
> > > 
> > > The 5% and 1% are made up, the point is that we'll allow larger sacrifices
> in
> > > accuracy for overlays since they are also a lot more profitable in terms
of
> > > power.
> > > The goal would be for most video (bt709 on srgb screens) to hit the
overlay
> > path
> > > when possible, and the affine path when overalys are not possible, but if
> you
> > > have a different
> > > type of screen, or an unusual video of some sort, you might have to fall
> back
> > > to the LUT.
> > 
> > Hmm, my preferences are different here.
> > 
> > WRT overlays, the power difference is so large, we should change our color
> > behavior to match whatever overlays are capable of (no matter how wrong).
> > 
> 
> I think we agree in principle: Overlays save a lot of power, and we should
> *almost* always use them.
> 
> However, there are some cases when we should not, like if the input or output
> is HDR for example. (Unless both input AND output are HDR...) Overlays also
> vary in capabilities, some may have the ability to specify a translation
matrix,
> some may have an enum, or some other way to describe what will happen.
> 
> Using an error function to make the decision if we should use overlays or not
> seems much easier than trying to hard-code it based on input and output
> properties.

Definitely agree about about not trying to overlay-ify HDR.

> > WRT affine, I don't think that the power savings are substantial enough to
> > warrant adding a separate path.
> 
> My measurements says it's 1-4% difference depending on hardware. Not much,
> but a lot of people are super-paranoid about power usage nowadays.

How are these measurements done, and where does support for analytic transfer
functions fit? According to the below investigation, they were also a below
LUTs, and suffer no loss of accuracy
https://docs.google.com/document/d/1lTyFs7_BtBAdL0ChoUtq1RVmP_1boGcnpqiNuNY5U...

[hubbe, 2016-12-09 01:19:31]

On 2016/12/09 00:30:50, ccameron wrote:
> I don't want us to continue down this direction.
> 
> In particular, I want to add support for analytic transfer functions (of the
> SkColorSpaceTransferFn), and stop there. In particular
> - use the analytic conversions whenever possible
> - fall back to using a LUT in the remaining (hopefully vanishingly few) cases
> 
> On 2016/12/08 21:19:20, hubbe wrote:
> > On 2016/12/08 21:07:43, ccameron wrote:
> > > On 2016/12/08 20:59:15, hubbe wrote:
> > > > On 2016/12/08 20:40:55, ccameron wrote:
> > > > > My bias here is to limit the number of conversion paths that we
support,
> > so
> > > > I'm
> > > > > biased against adding this.
> > > > > 
> > > > > Is this to save power? Or is it to add compatibility with overlay
> > hardware?
> > > > 
> > > > The primary goal is to replace the hard-coded coefficients in
gl_output.cc
> > > with
> > > > computed ones which would let it handle a much larger set of inputs and
> > > outputs.
> > > > 
> > > > Eventually I want the logic to look something like:
> > > > 
> > > > if (overlays_available && color_error_caused_by_overalays(input, output)
<
> > 5%)
> > > >   use_overlays();
> > > > else if (color_error_caused_by_using_affine_approximation(intput,
output)
> <
> > > 1%)
> > > >   use_affine();  // This a slightly updated version of what's in
> > gl_output.cc
> > > > today
> > > > else
> > > >   use_lut();
> > > > 
> > > > The 5% and 1% are made up, the point is that we'll allow larger
sacrifices
> > in
> > > > accuracy for overlays since they are also a lot more profitable in terms
> of
> > > > power.
> > > > The goal would be for most video (bt709 on srgb screens) to hit the
> overlay
> > > path
> > > > when possible, and the affine path when overalys are not possible, but
if
> > you
> > > > have a different
> > > > type of screen, or an unusual video of some sort, you might have to fall
> > back
> > > > to the LUT.
> > > 
> > > Hmm, my preferences are different here.
> > > 
> > > WRT overlays, the power difference is so large, we should change our color
> > > behavior to match whatever overlays are capable of (no matter how wrong).
> > > 
> > 
> > I think we agree in principle: Overlays save a lot of power, and we should
> > *almost* always use them.
> > 
> > However, there are some cases when we should not, like if the input or
output
> > is HDR for example. (Unless both input AND output are HDR...) Overlays also
> > vary in capabilities, some may have the ability to specify a translation
> matrix,
> > some may have an enum, or some other way to describe what will happen.
> > 
> > Using an error function to make the decision if we should use overlays or
not
> > seems much easier than trying to hard-code it based on input and output
> > properties.
> 
> Definitely agree about about not trying to overlay-ify HDR.
> 
> > > WRT affine, I don't think that the power savings are substantial enough to
> > > warrant adding a separate path.
> > 
> > My measurements says it's 1-4% difference depending on hardware. Not much,
> > but a lot of people are super-paranoid about power usage nowadays.
> 
> How are these measurements done, and where does support for analytic transfer
> functions fit? According to the below investigation, they were also a below
> LUTs, and suffer no loss of accuracy
>
https://docs.google.com/document/d/1lTyFs7_BtBAdL0ChoUtq1RVmP_1boGcnpqiNuNY5U...

The measurements are done by running through the av-analysis dashboard, which
measures
total power used while playing a video. I would expect analytic transfer
functions
to be somewhere between affine and LUTs in terms of power, but be a lot more
generic,
so they could handle most color transforms, except for ugly ICC profiles and
some of
the complicated HDR curves.

Implementing analytic transfer functions seems like a lot more work though.

[hubbe, 2017-03-16 17:53:11]

Description was changed from

==========
color: general transformation -> affine transformation

Implement a way to get an affine approximation of a color transformation.

The plan is to update cc/output/gl_output.cc to use this method to
calculate the correct coefficients for converting video frames to the
screen color space. We'll test the affine transformation with a set of
sample colors to see if the affine transformation is close enough, and
if not, we will fall back on using a 3D LUT.

The implementation itself uses a least-squares method which should return
a reasonable approximation for almost any transformation.

BUG=622133
==========

to

==========
color: general transformation -> affine transformation

Implement a way to get an affine approximation of a color transformation.

The plan is to update cc/output/gl_output.cc to use this method to
calculate the correct coefficients for converting video frames to the
screen color space. We'll test the affine transformation with a set of
sample colors to see if the affine transformation is close enough, and
if not, we will fall back on using a 3D LUT.

The implementation itself uses a least-squares method which should return
a reasonable approximation for almost any transformation.

BUG=622133
==========

[hubbe, 2017-03-16 17:53:11]

hubbe@chromium.org changed reviewers:
- ccameron@chromium.org