Implemented HEVC video demuxing and parsing

net/base/mime_util.cc

 // Copyright (c) 2012 The Chromium Authors. All rights reserved.

[ddorwin, 2015/02/23 20:00:49]

If you intend to support MSE, there may be other code that needs to be updated
to support these specific codecs.

[servolk, 2015/02/24 02:15:13]

There is also some changes in stream_parser_factory.cc, if that's what you mean.
In fact I've already tested playing some HEVC content through MSE with private
Chromium builds with this CL. I've been able to play videos both through
software FFmpeg decoder and on a hardware board that supports hevc.

[ddorwin, 2015/02/24 18:07:32]

Acknowledged.

 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
 #include <algorithm>
 #include <iterator>
 #include <map>
 #include <string>
 
 #include "base/containers/hash_tables.h"
 #include "base/lazy_instance.h"
@@ skipping 26 matching lines @@
     MPEG2_AAC_MAIN,
     MPEG2_AAC_SSR,
     MPEG4_AAC_LC,
     MPEG4_AAC_SBR_v1,
     MPEG4_AAC_SBR_PS_v2,
     VORBIS,
     OPUS,
     H264_BASELINE,
     H264_MAIN,
     H264_HIGH,
+    HEVC_MAIN,
     VP8,
     VP9,
     THEORA
   };
 
   bool GetMimeTypeFromExtension(const base::FilePath::StringType& ext,
                                 std::string* mime_type) const;
 
   bool GetMimeTypeFromFile(const base::FilePath& file_path,
                            std::string* mime_type) const;
@@ skipping 399 matching lines @@
     case MimeUtil::MPEG4_AAC_LC:
     case MimeUtil::MPEG4_AAC_SBR_v1:
     case MimeUtil::MPEG4_AAC_SBR_PS_v2:
     case MimeUtil::H264_BASELINE:
     case MimeUtil::H264_MAIN:
     case MimeUtil::H264_HIGH:
     case MimeUtil::VP8:
     case MimeUtil::VORBIS:
       return true;
 
+    case MimeUtil::HEVC_MAIN:
+      return false;
+
     case MimeUtil::MPEG2_AAC_LC:
     case MimeUtil::MPEG2_AAC_MAIN:
     case MimeUtil::MPEG2_AAC_SSR:
       // MPEG-2 variants of AAC are not supported on Android.
       return false;
 
     case MimeUtil::VP9:
       // VP9 is supported only in KitKat+ (API Level 19).
       return base::android::BuildInfo::GetInstance()->sdk_int() >= 19;
 
@@ skipping 39 matching lines @@
 //   mp4a.40.29  - MPEG-4 HE-AAC v2 (AAC LC + SBR + PS)
 //
 //   avc1.42E0xx - H.264 Baseline
 //   avc1.4D40xx - H.264 Main
 //   avc1.6400xx - H.264 High
 static const char kMP4AudioCodecsExpression[] =
     "mp4a.66,mp4a.67,mp4a.68,mp4a.69,mp4a.6B,mp4a.40.2,mp4a.40.02,mp4a.40.5,"
     "mp4a.40.05,mp4a.40.29";
 static const char kMP4VideoCodecsExpression[] =
     "avc1.42E00A,avc1.4D400A,avc1.64000A,"
+#if defined(ENABLE_HEVC_DEMUXING)

[ddorwin, 2015/02/21 02:09:21]

"DEMUXING" is not really accurate. This enables support in general.

[servolk, 2015/02/23 19:16:16]

No, I believe this is accurate. What exactly do you mean by "support in
general"? This change doesn't contain anything to support h265 decoding (which
would require us to deal with h265 licensing), it only enables demuxing and
parsing (which doesn't require any licenses) and this will allow us to support
hevc/h265 on platforms which have "external" h265 decoder, e.g. hardware h265
decoder, for which license costs are going to be paid by hardware manufacturer,
or OS-provided h265 decoder, for which license costs are going to be paid by OS
vendor. So, again, I believe the term "demuxing" is correct in this context, and
in fact we should be very careful about this, since changing it to something
more generic, like "enable_hevc" could get us in trouble by not being clear
enough that all we do is demuxing. Please let me know if you disagree and we can
discuss further.

[ddorwin, 2015/02/23 20:00:49]

The primary "support" it enables is advertising support via canPlayType().

Regardless of what this is called, we're clearly not enabling a software
decoder, but it seems reasonable to reuse this value if one needed to use it to
enable code around their decoder.

[servolk, 2015/02/24 02:15:13]

Yes, this does enable hevc support via canPlayType, but my understanding was
that only platforms that actually support hevc would explicitly set
enable_hevc_demuxing, so it's probably ok to gate mime type checks with this
flag as well.

[ddorwin, 2015/02/24 18:07:33]

Agreed. My point is that this flag enables HEVC, not just HEVC deumuxing. Since
it would only be anbled on platforms that support HEVC, I don't think we need to
worry about the other issues you mentioned and can just call it ENABLE_HEVC.

+    "hev1,hvc1,"

[ddorwin, 2015/02/21 02:09:21]

See below.

[ddorwin, 2015/02/23 20:00:49]

See some of the comments about how avc1 is handled in
https://codereview.chromium.org/952673002/. We should only have a single
unambiguous HEVC_MAIN string here.

[ddorwin, 2015/02/23 20:12:43]

Note that you must have an unambiguous string or you'll hit a DCHECK when
format_codec_mappings is processed.

[servolk, 2015/02/24 02:15:13]

Wait, I don't understand - even in that CL we have both avc1 and avc3 - so
there's two strings for hevc as well. Are you saying that there should be only
avc3+hvc1?

[ddorwin, 2015/02/24 18:07:33]

I didn't change anything in that CL, I just explained why things are the way
they are. (One of the things we need to do is replace these strings with
H264_BASELINE, etc.)
Look at line 529 here. We only use "avc1..." (not "avc3...). The whole point of
this is to add H264_BASELINE, etc. to a map.

In addition, these strings are unambiguous. You need an actual "hev1.xyzabc" or
you will hit a DCHECK.

+#endif
     "mp4a.66,mp4a.67,mp4a.68,mp4a.69,mp4a.6B,mp4a.40.2,mp4a.40.02,mp4a.40.5,"
     "mp4a.40.05,mp4a.40.29";
 
 static const MediaFormatStrict format_codec_mappings[] = {
   { "video/webm", "opus,vorbis,vp8,vp8.0,vp9,vp9.0" },
   { "audio/webm", "opus,vorbis" },
   { "audio/wav", "1" },
   { "audio/x-wav", "1" },
   { "video/ogg", "opus,theora,vorbis" },
   { "audio/ogg", "opus,vorbis" },
@@ skipping 13 matching lines @@
   const char* const codec_id;
   MimeUtil::Codec codec;
 };
 
 // List of codec IDs that provide enough information to determine the
 // codec and profile being requested.
 //
 // The "mp4a" strings come from RFC 6381.
 static const CodecIDMappings kUnambiguousCodecIDs[] = {
     {"1", MimeUtil::PCM},  // We only allow this for WAV so it isn't ambiguous.
+#if defined(ENABLE_HEVC_DEMUXING)
+    {"hev1", MimeUtil::HEVC_MAIN},

[ddorwin, 2015/02/21 02:09:21]

I don't think we should support these extension-less versions. It was a mistake
to support "avc1", which is not spec-compliant, and now we are stuck supporting
it forever.

Are there really two separate variants? Why? Can we pick one and lead the
industry in that direction?

[ddorwin, 2015/02/21 02:44:40]

To clarify, this may not have been so much a mistake as needing to be compatible
with other browsers that supported it. But as the first (?) UA to support HEVC,
please try to avoid setting such a precedent.

[servolk, 2015/02/23 19:16:16]

As far as I know the difference between hev1 and hvc1 is analogous to that
between avc1 and avc3. It's defined in ISO/IEC 14496-15 (for avc3/hev1 decoder
params PPS/SPS/VPS are embedded in elementary stream, whereas avc1/hvc1 contain
all params in avcC/hevC record). Chrome actually supports both avc1 and avc3, so
I think we should do the same with hev1/hvc1 - i.e. we should support both, they
are both standard but cover slightly different use cases, which gives some
flexibility to encoders. My understanding is that people do really need to have
both, one is typically used e.g. in live streaming scenarios, where it's
essential to be able to change decoder parameters on the fly to accommodate
bandwidth and content bitrate changes. Whereas another one is typically used by
offline encoders, which can afford to preprocess the whole media stream
beforehand, collect all the necessary parameter sets and write them down in the
decoder config box of the MP4 file. It doesn't really matter if we are the first
UA to support HEVC (I doubt it), we don't get to pick and choose. There is an
international standard ISO/IEC 14496-15 and we need to try and implement/support
as much of it as possible. I don't see a good reason to go against the standard
and support only one type, when the standard clearly says there's two and
clearly specifies the difference between the two.
Btw, other media stacks/libraries are also going to support both, for example
take a look at FFmpeg's libavformat/isom.c - they also support both hev1 and
hvc1.

[ddorwin, 2015/02/23 20:00:50]

Thanks for the exmplanation of "hev" vs. "hvc". I thought maybe there was no
clear decision in this area (vs. being defined to be different by a spec). I
agree that supporting both makes sense if it is like avc1 vs. avc3.

However, I still don't think we should support the extensionless strings (unless
that is also specified to be valid by some spec). (For avc1, it is not
spec-compliant.) This should be in the Ambiguous array like avc1/avc3.

[ddorwin, 2015/02/23 20:12:43]

Oops, strike that. Because we do not want to support the extensionless strings,
this should not appear in either list. ParseHEVCCodecID will handle this like we
do for avc1.

+    {"hvc1", MimeUtil::HEVC_MAIN},
+#endif
     {"mp3", MimeUtil::MP3},
     {"mp4a.66", MimeUtil::MPEG2_AAC_MAIN},
     {"mp4a.67", MimeUtil::MPEG2_AAC_LC},
     {"mp4a.68", MimeUtil::MPEG2_AAC_SSR},
     {"mp4a.69", MimeUtil::MP3},
     {"mp4a.6B", MimeUtil::MP3},
     {"mp4a.40.2", MimeUtil::MPEG4_AAC_LC},
     {"mp4a.40.02", MimeUtil::MPEG4_AAC_LC},
     {"mp4a.40.5", MimeUtil::MPEG4_AAC_SBR_v1},
     {"mp4a.40.05", MimeUtil::MPEG4_AAC_SBR_v1},
@@ skipping 432 matching lines @@
                              Codec* codec,
                              bool* is_ambiguous) const {
   StringToCodecMappings::const_iterator itr =
       string_to_codec_map_.find(codec_id);
   if (itr != string_to_codec_map_.end()) {
     *codec = itr->second.codec;
     *is_ambiguous = itr->second.is_ambiguous;
     return true;
   }
 
+#if defined(ENABLE_HEVC_DEMUXING)

[ddorwin, 2015/02/23 20:00:49]

Move this code to ParseHEVCCodecID or something like that.
We should try to keep this function as simple as possible.

+  if (StartsWithASCII(codec_id, "hev1.", true) ||

[ddorwin, 2015/02/21 02:09:21]

Once the other issues are addressed, you'll need to update
media_canplaytype_browsertest.cc. (Wait for
https://codereview.chromium.org/918463002/.)

[servolk, 2015/02/23 19:16:16]

Ok, yeah, once we have more clarity here we'll try to add more unit tests,
including mime type tests.

[ddorwin, 2015/02/23 20:00:49]

You should add at least some basic tests soon. This code is complex and
difficult to get right.

[servolk, 2015/02/24 02:15:13]

Sure, I will add some tests. In fact I've already added some unit tests for
ffmpeg and chunk demuxer. Do we have any end-to-end MSE tests for Chromium?

[ddorwin, 2015/02/24 18:07:32]

I think we have pipeline tests and content and chrome browser tests, both for
clear and encrypted content.

I was specifically referring to canPlayType() tests, though. The CL I referenced
has landed, so if you sync, you'll see examples of the detailed coverage that is
necessary for these codec strings.

+      StartsWithASCII(codec_id, "hvc1.", true)) {
+    // TODO(servolk): Implement parsing of hevc codec ids as described in

[ddorwin, 2015/02/21 02:09:21]

Our avc1 string handling is a mess. Similar to my comments above, we should be
accurate and restrictive from the beginning to avoid similar problems.

[servolk, 2015/02/23 19:16:16]

I understand your concerns, but we are in the early stages of implementation
here. We just don't know yet which profiles our hardware partners are going to
need/support. So for now we'd prefer to allow them all, to facilitate testing.

[ddorwin, 2015/02/23 20:00:49]

Can we at least check for valid values, similar to IsValidH264Level()?

[servolk, 2015/02/24 02:15:13]

Yep, will do

+    // ETSI TS 126 244 standard section A.2.2, but for now allow any
+    // HEVC tiers/profiles and let decoder decide if it can handle that.
+    *codec = MimeUtil::HEVC_MAIN;
+    *is_ambiguous = false;

[ddorwin, 2015/02/21 02:09:21]

This should be true since you do no verification.

[servolk, 2015/02/23 19:16:16]

yep, that's a good point, I'll change this.

+    return true;
+  }
+#endif
+
   // If |codec_id| is not in |string_to_codec_map_|, then we assume that it is
   // an H.264 codec ID because currently those are the only ones that can't be
   // stored in the |string_to_codec_map_| and require parsing.
   return ParseH264CodecID(codec_id, codec, is_ambiguous);
 }
 
 bool MimeUtil::IsCodecSupported(Codec codec) const {
   DCHECK_NE(codec, INVALID_CODEC);
 
 #if defined(OS_ANDROID)
@@ skipping 10 matching lines @@
     case MP3:
     case MPEG2_AAC_LC:
     case MPEG2_AAC_MAIN:
     case MPEG2_AAC_SSR:
     case MPEG4_AAC_LC:
     case MPEG4_AAC_SBR_v1:
     case MPEG4_AAC_SBR_PS_v2:
     case H264_BASELINE:
     case H264_MAIN:
     case H264_HIGH:
+    case HEVC_MAIN:
       return true;
 
     case PCM:
     case VORBIS:
     case OPUS:
     case VP8:
     case VP9:
     case THEORA:
       return false;
   }
@@ skipping 328 matching lines @@
   post_data->append("\r\n" + value + "\r\n");
 }
 
 void AddMultipartFinalDelimiterForUpload(const std::string& mime_boundary,
                                          std::string* post_data) {
   DCHECK(post_data);
   post_data->append("--" + mime_boundary + "--\r\n");
 }
 
 }  // namespace net