[PATCH v5, 0/8] Support H264 multi-core encoder on MT8195

Irui Wang posted 8 patches 2 years, 1 month ago
There is a newer version of this series
.../media/mediatek,vcodec-encoder-core.yaml   | 218 ++++++++++++++++
.../media/mediatek,vcodec-encoder.yaml        |   1 -
.../media/platform/mediatek/vcodec/Makefile   |   4 +-
.../platform/mediatek/vcodec/mtk_vcodec_drv.h |  28 +-
.../platform/mediatek/vcodec/mtk_vcodec_enc.c | 113 ++++++--
.../platform/mediatek/vcodec/mtk_vcodec_enc.h |  11 +-
.../mediatek/vcodec/mtk_vcodec_enc_drv.c      |  44 +++-
.../mediatek/vcodec/mtk_vcodec_enc_hw.c       | 156 +++++++++++
.../mediatek/vcodec/mtk_vcodec_enc_hw.h       |  34 +++
.../mediatek/vcodec/mtk_vcodec_enc_pm.c       | 178 +++++++++++--
.../mediatek/vcodec/mtk_vcodec_enc_pm.h       |  11 +-
.../mediatek/vcodec/mtk_vcodec_util.c         |  19 ++
.../mediatek/vcodec/mtk_vcodec_util.h         |   3 +
.../mediatek/vcodec/venc/venc_h264_if.c       | 243 ++++++++++++++----
.../mediatek/vcodec/venc/venc_vp8_if.c        |   3 +-
.../platform/mediatek/vcodec/venc_drv_if.c    |  75 ++++--
.../platform/mediatek/vcodec/venc_drv_if.h    |   6 +
.../platform/mediatek/vcodec/venc_vpu_if.c    |   9 +-
.../platform/mediatek/vcodec/venc_vpu_if.h    |   3 +-
19 files changed, 1028 insertions(+), 131 deletions(-)
create mode 100644 Documentation/devicetree/bindings/media/mediatek,vcodec-encoder-core.yaml
create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.c
create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.h
[PATCH v5, 0/8] Support H264 multi-core encoder on MT8195
Posted by Irui Wang 2 years, 1 month ago
MT8195 has two H264 encoder hardware, named core0 and core1, this two
cores can encode two input frames separately at the same time to achieve
higher performance.

This series of patches are used to enable the two H264 encoder cores,
the difference between encoding process before and after enable two
cores is just like as below:
As-Is: Synchronous
V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> wait encoder IRQ
-->
encoding done with result --> job_finish
V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> wait encoder IRQ
-->
encoding done with result --> job_finish
...
To-Be: Asynchronous
V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> job_finish
..V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> job_finish
(venc core0 may encode done here, done the encoding result to client)
V4L2_VIDIOC_QBUF#2 --> device_run(triger encoder) --> job_finish.

---
changes compared with v4:
- reabse to the newer linux media stage tree.
- remove "mediatek,venc-multi-core" property since sub-device can
  be probed by of_platform_populate api directly.
- some modifications for patch v4's review comments.

changes compared with v3:
- rebase to the newer linux media stage.
- add a capability to indicate scp firmware support multi-core.
- probe core0 as main device, core1 as sub-device.

changes compared with v2:
- update venc core dt-bindings, add two new properties for current
  usage.
- parse venc multi_core mode from device tree.
- rebase to the newer linux media stage.

changes compared with v1:
- of_platform_populate was used in place of the component framework.
- new yaml file for venc cores.
- some modifications for patch v1's review comments.
---

Irui Wang (8):
  dt-bindings: media: mediatek: vcodec: Adds encoder cores dt-bindings
    for mt8195
  media: mediatek: vcodec: Enable venc dual core usage
  media: mediatek: vcodec: Refactor venc power manage function
  media: mediatek: vcodec: Add more extra processing for multi-core
    encoding
  media: mediatek: vcodec: Add venc power on/off function
  media: mediatek: vcodec: Refactor encoder clock on/off function
  media: mediatek: vcodec: Add multi-core encoding process
  media: mediatek: vcodec: Return encoding result in asynchronous mode

 .../media/mediatek,vcodec-encoder-core.yaml   | 218 ++++++++++++++++
 .../media/mediatek,vcodec-encoder.yaml        |   1 -
 .../media/platform/mediatek/vcodec/Makefile   |   4 +-
 .../platform/mediatek/vcodec/mtk_vcodec_drv.h |  28 +-
 .../platform/mediatek/vcodec/mtk_vcodec_enc.c | 113 ++++++--
 .../platform/mediatek/vcodec/mtk_vcodec_enc.h |  11 +-
 .../mediatek/vcodec/mtk_vcodec_enc_drv.c      |  44 +++-
 .../mediatek/vcodec/mtk_vcodec_enc_hw.c       | 156 +++++++++++
 .../mediatek/vcodec/mtk_vcodec_enc_hw.h       |  34 +++
 .../mediatek/vcodec/mtk_vcodec_enc_pm.c       | 178 +++++++++++--
 .../mediatek/vcodec/mtk_vcodec_enc_pm.h       |  11 +-
 .../mediatek/vcodec/mtk_vcodec_util.c         |  19 ++
 .../mediatek/vcodec/mtk_vcodec_util.h         |   3 +
 .../mediatek/vcodec/venc/venc_h264_if.c       | 243 ++++++++++++++----
 .../mediatek/vcodec/venc/venc_vp8_if.c        |   3 +-
 .../platform/mediatek/vcodec/venc_drv_if.c    |  75 ++++--
 .../platform/mediatek/vcodec/venc_drv_if.h    |   6 +
 .../platform/mediatek/vcodec/venc_vpu_if.c    |   9 +-
 .../platform/mediatek/vcodec/venc_vpu_if.h    |   3 +-
 19 files changed, 1028 insertions(+), 131 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/media/mediatek,vcodec-encoder-core.yaml
 create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.c
 create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.h

-- 
2.18.0
Re: [PATCH v5, 0/8] Support H264 multi-core encoder on MT8195
Posted by Nicolas Dufresne 2 years, 1 month ago
Hi Irui,

Le vendredi 29 juillet 2022 à 11:51 +0800, Irui Wang a écrit :
> MT8195 has two H264 encoder hardware, named core0 and core1, this two
> cores can encode two input frames separately at the same time to achieve
> higher performance.

I suspect you could provide some performance numbers on that specific HW before
and after your change ? This could help integrator regarding what type of gain
they can expect of back-porting this feature. This is not strictly mandatory
though.

> 
> This series of patches are used to enable the two H264 encoder cores,
> the difference between encoding process before and after enable two
> cores is just like as below:
> As-Is: Synchronous
> V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> wait encoder IRQ
> -->
> encoding done with result --> job_finish
> V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> wait encoder IRQ
> -->
> encoding done with result --> job_finish
> ...
> To-Be: Asynchronous
> V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> job_finish
> ..V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> job_finish
> (venc core0 may encode done here, done the encoding result to client)
> V4L2_VIDIOC_QBUF#2 --> device_run(triger encoder) --> job_finish.
> 
> ---
> changes compared with v4:
> - reabse to the newer linux media stage tree.
> - remove "mediatek,venc-multi-core" property since sub-device can
>   be probed by of_platform_populate api directly.
> - some modifications for patch v4's review comments.
> 
> changes compared with v3:
> - rebase to the newer linux media stage.
> - add a capability to indicate scp firmware support multi-core.
> - probe core0 as main device, core1 as sub-device.
> 
> changes compared with v2:
> - update venc core dt-bindings, add two new properties for current
>   usage.
> - parse venc multi_core mode from device tree.
> - rebase to the newer linux media stage.
> 
> changes compared with v1:
> - of_platform_populate was used in place of the component framework.
> - new yaml file for venc cores.
> - some modifications for patch v1's review comments.
> ---
> 
> Irui Wang (8):
>   dt-bindings: media: mediatek: vcodec: Adds encoder cores dt-bindings
>     for mt8195
>   media: mediatek: vcodec: Enable venc dual core usage
>   media: mediatek: vcodec: Refactor venc power manage function
>   media: mediatek: vcodec: Add more extra processing for multi-core
>     encoding
>   media: mediatek: vcodec: Add venc power on/off function
>   media: mediatek: vcodec: Refactor encoder clock on/off function
>   media: mediatek: vcodec: Add multi-core encoding process
>   media: mediatek: vcodec: Return encoding result in asynchronous mode
> 
>  .../media/mediatek,vcodec-encoder-core.yaml   | 218 ++++++++++++++++
>  .../media/mediatek,vcodec-encoder.yaml        |   1 -
>  .../media/platform/mediatek/vcodec/Makefile   |   4 +-
>  .../platform/mediatek/vcodec/mtk_vcodec_drv.h |  28 +-
>  .../platform/mediatek/vcodec/mtk_vcodec_enc.c | 113 ++++++--
>  .../platform/mediatek/vcodec/mtk_vcodec_enc.h |  11 +-
>  .../mediatek/vcodec/mtk_vcodec_enc_drv.c      |  44 +++-
>  .../mediatek/vcodec/mtk_vcodec_enc_hw.c       | 156 +++++++++++
>  .../mediatek/vcodec/mtk_vcodec_enc_hw.h       |  34 +++
>  .../mediatek/vcodec/mtk_vcodec_enc_pm.c       | 178 +++++++++++--
>  .../mediatek/vcodec/mtk_vcodec_enc_pm.h       |  11 +-
>  .../mediatek/vcodec/mtk_vcodec_util.c         |  19 ++
>  .../mediatek/vcodec/mtk_vcodec_util.h         |   3 +
>  .../mediatek/vcodec/venc/venc_h264_if.c       | 243 ++++++++++++++----
>  .../mediatek/vcodec/venc/venc_vp8_if.c        |   3 +-
>  .../platform/mediatek/vcodec/venc_drv_if.c    |  75 ++++--
>  .../platform/mediatek/vcodec/venc_drv_if.h    |   6 +
>  .../platform/mediatek/vcodec/venc_vpu_if.c    |   9 +-
>  .../platform/mediatek/vcodec/venc_vpu_if.h    |   3 +-
>  19 files changed, 1028 insertions(+), 131 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/media/mediatek,vcodec-encoder-core.yaml
>  create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.c
>  create mode 100644 drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.h
> 
Re: [PATCH v5, 0/8] Support H264 multi-core encoder on MT8195
Posted by Irui Wang 2 years, 1 month ago
Dear Nicolas,

Sorry for late, we have tested this change by chrome tast test before:

tast run $IP video.PlatformEncoding.v4l2_h264*

here are the average fps results:
Before:
180p: 1913
180_meet: 1935

360p: 554
360_meet: 571

720p: 159
720_meet: 189

After:
180p: 2236
180_meet: 2337

360p: 820
360_meet: 830

720p: 324
720_meet: 390

Thanks
BRs
On Fri, 2022-07-29 at 11:08 -0400, Nicolas Dufresne wrote:
> Hi Irui,
> 
> Le vendredi 29 juillet 2022 à 11:51 +0800, Irui Wang a écrit :
> > MT8195 has two H264 encoder hardware, named core0 and core1, this
> > two
> > cores can encode two input frames separately at the same time to
> > achieve
> > higher performance.
> 
> I suspect you could provide some performance numbers on that specific
> HW before
> and after your change ? This could help integrator regarding what
> type of gain
> they can expect of back-porting this feature. This is not strictly
> mandatory
> though.
> 
> > 
> > This series of patches are used to enable the two H264 encoder
> > cores,
> > the difference between encoding process before and after enable two
> > cores is just like as below:
> > As-Is: Synchronous
> > V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> wait encoder
> > IRQ
> > -->
> > encoding done with result --> job_finish
> > V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> wait encoder
> > IRQ
> > -->
> > encoding done with result --> job_finish
> > ...
> > To-Be: Asynchronous
> > V4L2_VIDIOC_QBUF#0 --> device_run(triger encoder) --> job_finish
> > ..V4l2_VIDIOC_QBUF#1 --> device_run(triger encoder) --> job_finish
> > (venc core0 may encode done here, done the encoding result to
> > client)
> > V4L2_VIDIOC_QBUF#2 --> device_run(triger encoder) --> job_finish.
> > 
> > ---
> > changes compared with v4:
> > - reabse to the newer linux media stage tree.
> > - remove "mediatek,venc-multi-core" property since sub-device can
> >   be probed by of_platform_populate api directly.
> > - some modifications for patch v4's review comments.
> > 
> > changes compared with v3:
> > - rebase to the newer linux media stage.
> > - add a capability to indicate scp firmware support multi-core.
> > - probe core0 as main device, core1 as sub-device.
> > 
> > changes compared with v2:
> > - update venc core dt-bindings, add two new properties for current
> >   usage.
> > - parse venc multi_core mode from device tree.
> > - rebase to the newer linux media stage.
> > 
> > changes compared with v1:
> > - of_platform_populate was used in place of the component
> > framework.
> > - new yaml file for venc cores.
> > - some modifications for patch v1's review comments.
> > ---
> > 
> > Irui Wang (8):
> >   dt-bindings: media: mediatek: vcodec: Adds encoder cores dt-
> > bindings
> >     for mt8195
> >   media: mediatek: vcodec: Enable venc dual core usage
> >   media: mediatek: vcodec: Refactor venc power manage function
> >   media: mediatek: vcodec: Add more extra processing for multi-core
> >     encoding
> >   media: mediatek: vcodec: Add venc power on/off function
> >   media: mediatek: vcodec: Refactor encoder clock on/off function
> >   media: mediatek: vcodec: Add multi-core encoding process
> >   media: mediatek: vcodec: Return encoding result in asynchronous
> > mode
> > 
> >  .../media/mediatek,vcodec-encoder-core.yaml   | 218
> > ++++++++++++++++
> >  .../media/mediatek,vcodec-encoder.yaml        |   1 -
> >  .../media/platform/mediatek/vcodec/Makefile   |   4 +-
> >  .../platform/mediatek/vcodec/mtk_vcodec_drv.h |  28 +-
> >  .../platform/mediatek/vcodec/mtk_vcodec_enc.c | 113 ++++++--
> >  .../platform/mediatek/vcodec/mtk_vcodec_enc.h |  11 +-
> >  .../mediatek/vcodec/mtk_vcodec_enc_drv.c      |  44 +++-
> >  .../mediatek/vcodec/mtk_vcodec_enc_hw.c       | 156 +++++++++++
> >  .../mediatek/vcodec/mtk_vcodec_enc_hw.h       |  34 +++
> >  .../mediatek/vcodec/mtk_vcodec_enc_pm.c       | 178 +++++++++++--
> >  .../mediatek/vcodec/mtk_vcodec_enc_pm.h       |  11 +-
> >  .../mediatek/vcodec/mtk_vcodec_util.c         |  19 ++
> >  .../mediatek/vcodec/mtk_vcodec_util.h         |   3 +
> >  .../mediatek/vcodec/venc/venc_h264_if.c       | 243
> > ++++++++++++++----
> >  .../mediatek/vcodec/venc/venc_vp8_if.c        |   3 +-
> >  .../platform/mediatek/vcodec/venc_drv_if.c    |  75 ++++--
> >  .../platform/mediatek/vcodec/venc_drv_if.h    |   6 +
> >  .../platform/mediatek/vcodec/venc_vpu_if.c    |   9 +-
> >  .../platform/mediatek/vcodec/venc_vpu_if.h    |   3 +-
> >  19 files changed, 1028 insertions(+), 131 deletions(-)
> >  create mode 100644
> > Documentation/devicetree/bindings/media/mediatek,vcodec-encoder-
> > core.yaml
> >  create mode 100644
> > drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.c
> >  create mode 100644
> > drivers/media/platform/mediatek/vcodec/mtk_vcodec_enc_hw.h
> > 
> 
>