[PATCH v2 00/12] media: rkvdec: Add support for VDPU381 and VDPU383

Detlev Casanova posted 12 patches 1 month, 3 weeks ago
.../media/platform/rockchip/rkvdec/Kconfig    |    1 +
.../media/platform/rockchip/rkvdec/Makefile   |   13 +-
.../platform/rockchip/rkvdec/rkvdec-cabac.c   | 3944 +++++++++++++++++
.../rockchip/rkvdec/rkvdec-h264-common.c      |  253 ++
.../rockchip/rkvdec/rkvdec-h264-common.h      |   81 +
.../platform/rockchip/rkvdec/rkvdec-h264.c    |  891 +---
.../rockchip/rkvdec/rkvdec-hevc-common.c      |  511 +++
.../rockchip/rkvdec/rkvdec-hevc-common.h      |  101 +
.../platform/rockchip/rkvdec/rkvdec-rcb.c     |  174 +
.../platform/rockchip/rkvdec/rkvdec-rcb.h     |   29 +
.../platform/rockchip/rkvdec/rkvdec-regs.h    |  567 ++-
.../rockchip/rkvdec/rkvdec-vdpu381-h264.c     |  469 ++
.../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     |  596 +++
.../rockchip/rkvdec/rkvdec-vdpu381-regs.h     |  427 ++
.../rockchip/rkvdec/rkvdec-vdpu383-h264.c     |  582 +++
.../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     |  688 +++
.../rockchip/rkvdec/rkvdec-vdpu383-regs.h     |  284 ++
.../platform/rockchip/rkvdec/rkvdec-vp9.c     |  230 +-
.../media/platform/rockchip/rkvdec/rkvdec.c   |  569 ++-
.../media/platform/rockchip/rkvdec/rkvdec.h   |   39 +
20 files changed, 9260 insertions(+), 1189 deletions(-)
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-cabac.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-h264-common.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-h264-common.h
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-rcb.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-rcb.h
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-regs.h
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-h264.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-regs.h
[PATCH v2 00/12] media: rkvdec: Add support for VDPU381 and VDPU383
Posted by Detlev Casanova 1 month, 3 weeks ago
These variants are found respectively in the RK3588 and RK3576 SoCs.
This patch only adds support for H264 and H265 in both variants.

As there is a considerable part of the code that can be shared with the
already supported rkvdec decoder driver, the support for these variants
is added here rather than writing a new driver.

This patch set uses the newly introduced hevc_ext_sps_[ls]t_rps v4l2
controls for HEVC [1].
Therefore, a patched version of userpace tools is needed for HEVC
support (added for GStreamer[2] and in an early stage for FFmpeg[3]).

This patch set also depends on the preparation patch set sent earlier [4]
as well as the iommu restore fix [5] (already merged in linux-media) and
Nicolas Frattaroli's bitmap patch [6] to support setting registers that
uses upper 16 bits as masks.

[1]: https://lore.kernel.org/all/20250807194327.69900-1-detlev.casanova@collabora.com/
[2]: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9355
[3]: https://gitlab.collabora.com/detlev/ffmpeg
[4]: https://lore.kernel.org/all/20250623160722.55938-1-detlev.casanova@collabora.com/
[5]: https://lore.kernel.org/all/20250508-rkvdec-iommu-reset-v1-1-c46b6efa6e9b@collabora.com/
[6]: https://lore.kernel.org/all/20250623-byeword-update-v2-1-cf1fc08a2e1f@collabora.com/

Changes since v1:
 - Add parsing of the short and long term ref frame sets from the new v4l2
   controls
 - Add RPS cache to avoid parsing the same data again
 - Fix HEVC pixel formats selection
 - Fix multiple indentation errors

Detlev Casanova (12):
  media: rkvdec: Switch to using structs instead of writel
  media: rkvdec: Move cabac table to its own source file
  media: rkvdec: Use structs to represent the HW RPS
  media: rkvdec: Move h264 functions to common file
  media: rkvdec: Add per variant configuration
  media: rkvdec: Add RCB and SRAM support
  media: rkvdec: Support per-variant interrupt handler
  media: rkvdec: Enable all clocks without naming them
  media: rkvdec: Add H264 support for the VDPU381 variant
  media: rkvdec: Add H264 support for the VDPU383 variant
  media: rkvdec: Add HEVC support for the VDPU381 variant
  media: rkvdec: Add HEVC support for the VDPU383 variant

 .../media/platform/rockchip/rkvdec/Kconfig    |    1 +
 .../media/platform/rockchip/rkvdec/Makefile   |   13 +-
 .../platform/rockchip/rkvdec/rkvdec-cabac.c   | 3944 +++++++++++++++++
 .../rockchip/rkvdec/rkvdec-h264-common.c      |  253 ++
 .../rockchip/rkvdec/rkvdec-h264-common.h      |   81 +
 .../platform/rockchip/rkvdec/rkvdec-h264.c    |  891 +---
 .../rockchip/rkvdec/rkvdec-hevc-common.c      |  511 +++
 .../rockchip/rkvdec/rkvdec-hevc-common.h      |  101 +
 .../platform/rockchip/rkvdec/rkvdec-rcb.c     |  174 +
 .../platform/rockchip/rkvdec/rkvdec-rcb.h     |   29 +
 .../platform/rockchip/rkvdec/rkvdec-regs.h    |  567 ++-
 .../rockchip/rkvdec/rkvdec-vdpu381-h264.c     |  469 ++
 .../rockchip/rkvdec/rkvdec-vdpu381-hevc.c     |  596 +++
 .../rockchip/rkvdec/rkvdec-vdpu381-regs.h     |  427 ++
 .../rockchip/rkvdec/rkvdec-vdpu383-h264.c     |  582 +++
 .../rockchip/rkvdec/rkvdec-vdpu383-hevc.c     |  688 +++
 .../rockchip/rkvdec/rkvdec-vdpu383-regs.h     |  284 ++
 .../platform/rockchip/rkvdec/rkvdec-vp9.c     |  230 +-
 .../media/platform/rockchip/rkvdec/rkvdec.c   |  569 ++-
 .../media/platform/rockchip/rkvdec/rkvdec.h   |   39 +
 20 files changed, 9260 insertions(+), 1189 deletions(-)
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-cabac.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-h264-common.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-h264-common.h
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-hevc-common.h
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-rcb.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-rcb.h
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-h264.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-hevc.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu381-regs.h
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-h264.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-hevc.c
 create mode 100644 drivers/media/platform/rockchip/rkvdec/rkvdec-vdpu383-regs.h

-- 
2.50.1
Re: [PATCH v2 00/12] media: rkvdec: Add support for VDPU381 and VDPU383
Posted by Piotr Oniszczuk 1 month, 3 weeks ago

> Wiadomość napisana przez Detlev Casanova <detlev.casanova@collabora.com> w dniu 8 sie 2025, o godz. 22:03:
> 
> These variants are found respectively in the RK3588 and RK3576 SoCs.
> This patch only adds support for H264 and H265 in both variants.
> 
> As there is a considerable part of the code that can be shared with the
> already supported rkvdec decoder driver, the support for these variants
> is added here rather than writing a new driver.
> 
> This patch set uses the newly introduced hevc_ext_sps_[ls]t_rps v4l2
> controls for HEVC [1].
> Therefore, a patched version of userpace tools is needed for HEVC
> support (added for GStreamer[2] and in an early stage for FFmpeg[3]).
> 
> This patch set also depends on the preparation patch set sent earlier [4]
> as well as the iommu restore fix [5] (already merged in linux-media) and
> Nicolas Frattaroli's bitmap patch [6] to support setting registers that
> uses upper 16 bits as masks.
> 
> [1]: https://lore.kernel.org/all/20250807194327.69900-1-detlev.casanova@collabora.com/
> [2]: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9355
> [3]: https://gitlab.collabora.com/detlev/ffmpeg
> [4]: https://lore.kernel.org/all/20250623160722.55938-1-detlev.casanova@collabora.com/
> [5]: https://lore.kernel.org/all/20250508-rkvdec-iommu-reset-v1-1-c46b6efa6e9b@collabora.com/
> [6]: https://lore.kernel.org/all/20250623-byeword-update-v2-1-cf1fc08a2e1f@collabora.com/
> 
> Changes since v1:
> - Add parsing of the short and long term ref frame sets from the new v4l2
>   controls
> - Add RPS cache to avoid parsing the same data again
> - Fix HEVC pixel formats selection
> - Fix multiple indentation errors
> 
> Detlev Casanova (12):
>  media: rkvdec: Switch to using structs instead of writel
>  media: rkvdec: Move cabac table to its own source file
>  media: rkvdec: Use structs to represent the HW RPS
>  media: rkvdec: Move h264 functions to common file
>  media: rkvdec: Add per variant configuration
>  media: rkvdec: Add RCB and SRAM support
>  media: rkvdec: Support per-variant interrupt handler
>  media: rkvdec: Enable all clocks without naming them
>  media: rkvdec: Add H264 support for the VDPU381 variant
>  media: rkvdec: Add H264 support for the VDPU383 variant
>  media: rkvdec: Add HEVC support for the VDPU381 variant
>  media: rkvdec: Add HEVC support for the VDPU383 variant
> 
> ..

Detlev,

I give run for this series on rk3576 and rk3588 devices (various SBC boards) on mainline 6.16 kernel.
Userspace was: KODI, MythTV and mpv.
All are using ffmpeg - but without [3] applied*.
Tested video rendering pipelines was: EGL DAMBuf and DRM direct to plane

Happy to report:
-all h264 content** i tested was decoded ok.
-on some rk3576 h264 rendering manifest known "green lines" issue***  
-hevc content**** was also decoded ok except samples requiring long sps/rps hinting from userspace (from ffmpeg in my case)


* - it looks (to me) your's ffmpeg branch changes are incompatible with yours rkvdec code (i.e. ffmpeg refers to V4L2_HEVC_EXT_SPS_RPS_FLAG_INTER_REF_PIC_SET_PRED but kernel v2 driver don't have it)

**  - i'm referring to multiple h.264 movies and TV HD channels

*** - issue of thin green lines we discussed on rockchip IRC channel

**** - as my ffmpeg7.1 has not applied newly introduced hevc_ext_sps_[ls]t_rps v4l2
controls for HEVC (due *) - some content is not decoded properly.

If any extra tests can be helpful - i'' be more that happy to do so!

br 
Re: [PATCH v2 00/12] media: rkvdec: Add support for VDPU381 and VDPU383
Posted by Diederik de Haas 2 weeks, 3 days ago
Hi Detlev,

On Fri Aug 8, 2025 at 10:03 PM CEST, Detlev Casanova wrote:
> These variants are found respectively in the RK3588 and RK3576 SoCs.
> This patch only adds support for H264 and H265 in both variants.

I tested this on my Rock 5B (rk3588) with (ffmpeg and) mpv on Debian
Forky with sway and I was quite impressed with the results :-)

In my earlier testing I found that the classic BBB video made for a
really good test video and it played REALLY well. The 1080p version in
both x264 and 8-bit x265 played without frame drops, although when
pressing 'I' to show video info over the video it dropped some frames.
But that never resulted in any visible artifacts.
On the 2160p version I did see quite a number of dropped frames, but
didn't notice any visual artifacts. Maybe that it got displayed on a
1080p monitor had something to do with it?

10-bit encoded x265 files (still) resulted in a completely blue output,
but I have strong reasons to believe that's due to 'missing pieces' (ie
NV15 support) in the rest of the display pipeline.

The displayed video seemed sharper and better then in my other tests.
I thought I did see an artifact around 5:50 when a big rock almost fell
on the little animal. Looking further, it appears to be an artifact in
the original video :-O (I saw it also on my AMD GPU on my amd64 system
and also when using software decoding).
The fact I only noticed it when testing this patch set is saying a lot
... in a positive way! So I'm happy to provide my

Tested-by: Diederik de Haas <didi.debian@cknow.org>  # Rock 5B
Re: [PATCH v2 00/12] media: rkvdec: Add support for VDPU381 and VDPU383
Posted by Nicolas Dufresne 2 weeks, 2 days ago
Hi,

Le mercredi 17 septembre 2025 à 19:34 +0200, Diederik de Haas a écrit :
> Hi Detlev,
> 
> On Fri Aug 8, 2025 at 10:03 PM CEST, Detlev Casanova wrote:
> > These variants are found respectively in the RK3588 and RK3576 SoCs.
> > This patch only adds support for H264 and H265 in both variants.
> 
> I tested this on my Rock 5B (rk3588) with (ffmpeg and) mpv on Debian
> Forky with sway and I was quite impressed with the results :-)
> 
> In my earlier testing I found that the classic BBB video made for a
> really good test video and it played REALLY well. The 1080p version in
> both x264 and 8-bit x265 played without frame drops, although when
> pressing 'I' to show video info over the video it dropped some frames.
> But that never resulted in any visible artifacts.
> On the 2160p version I did see quite a number of dropped frames, but
> didn't notice any visual artifacts. Maybe that it got displayed on a
> 1080p monitor had something to do with it?

Sounds like we have "correctness" with ok performance, so that is enough for me
to consider the patches. There is few things that are not implemented that could
explain lower performance compare to the downstream stuff, such as the lack of
AFBC. It will hopefully come later, giving nice boost when doing GPU streaming.

> 
> 10-bit encoded x265 files (still) resulted in a completely blue output,
> but I have strong reasons to believe that's due to 'missing pieces' (ie
> NV15 support) in the rest of the display pipeline.

Its still quite early days for NV15 (and its derivatives) upstream of
everything, so I would not block on that. It passes the tests when we read back
the pixels from CPU I believe.

> 
> The displayed video seemed sharper and better then in my other tests.
> I thought I did see an artifact around 5:50 when a big rock almost fell
> on the little animal. Looking further, it appears to be an artifact in
> the original video :-O (I saw it also on my AMD GPU on my amd64 system
> and also when using software decoding).
> The fact I only noticed it when testing this patch set is saying a lot
> ... in a positive way! So I'm happy to provide my

That's it, your video watching experience is now ruined for the rest of your
life :-D

> 
> Tested-by: Diederik de Haas <didi.debian@cknow.org>  # Rock 5B

thanks a lot for testing,
Nicolas
Re: [PATCH v2 00/12] media: rkvdec: Add support for VDPU381 and VDPU383
Posted by Heiko Stübner 1 month, 3 weeks ago
Hi Detlev,

Am Freitag, 8. August 2025, 22:03:22 Mitteleuropäische Sommerzeit schrieb Detlev Casanova:
> These variants are found respectively in the RK3588 and RK3576 SoCs.
> This patch only adds support for H264 and H265 in both variants.
> 
> As there is a considerable part of the code that can be shared with the
> already supported rkvdec decoder driver, the support for these variants
> is added here rather than writing a new driver.
> 
> This patch set uses the newly introduced hevc_ext_sps_[ls]t_rps v4l2
> controls for HEVC [1].
> Therefore, a patched version of userpace tools is needed for HEVC
> support (added for GStreamer[2] and in an early stage for FFmpeg[3]).
> 
> This patch set also depends on the preparation patch set sent earlier [4]
> as well as the iommu restore fix [5] (already merged in linux-media) and
> Nicolas Frattaroli's bitmap patch [6] to support setting registers that
> uses upper 16 bits as masks.
> 
> [1]: https://lore.kernel.org/all/20250807194327.69900-1-detlev.casanova@collabora.com/
> [2]: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9355
> [3]: https://gitlab.collabora.com/detlev/ffmpeg
> [4]: https://lore.kernel.org/all/20250623160722.55938-1-detlev.casanova@collabora.com/
> [5]: https://lore.kernel.org/all/20250508-rkvdec-iommu-reset-v1-1-c46b6efa6e9b@collabora.com/
> [6]: https://lore.kernel.org/all/20250623-byeword-update-v2-1-cf1fc08a2e1f@collabora.com/
> 
> Changes since v1:
>  - Add parsing of the short and long term ref frame sets from the new v4l2
>    controls
>  - Add RPS cache to avoid parsing the same data again
>  - Fix HEVC pixel formats selection
>  - Fix multiple indentation errors

when applying the series, git was a bit unhappy about some whitespaces:


> Detlev Casanova (12):
>   media: rkvdec: Switch to using structs instead of writel
>   media: rkvdec: Move cabac table to its own source file

Applying: media: rkvdec: Move cabac table to its own source file
.git/rebase-apply/patch:535: new blank line at EOF.
+
Warnung: 1 Zeile fügt Whitespace-Fehler hinzu.


>   media: rkvdec: Use structs to represent the HW RPS
>   media: rkvdec: Move h264 functions to common file

Applying: media: rkvdec: Move h264 functions to common file
.git/rebase-apply/patch:278: new blank line at EOF.
+
Warnung: 1 Zeile fügt Whitespace-Fehler hinzu.


>   media: rkvdec: Add per variant configuration
>   media: rkvdec: Add RCB and SRAM support

Applying: media: rkvdec: Add RCB and SRAM support
.git/rebase-apply/patch:200: new blank line at EOF.
+
Warnung: 1 Zeile fügt Whitespace-Fehler hinzu.


>   media: rkvdec: Support per-variant interrupt handler
>   media: rkvdec: Enable all clocks without naming them
>   media: rkvdec: Add H264 support for the VDPU381 variant
>   media: rkvdec: Add H264 support for the VDPU383 variant
>   media: rkvdec: Add HEVC support for the VDPU381 variant

Applying: media: rkvdec: Add HEVC support for the VDPU381 variant
.git/rebase-apply/patch:3483: new blank line at EOF.
+
.git/rebase-apply/patch:4035: new blank line at EOF.
+
Warnung: 2 Zeilen fügen Whitespace-Fehler hinzu.


Heiko
Re: [PATCH v2 00/12] media: rkvdec: Add support for VDPU381 and VDPU383
Posted by Detlev Casanova 1 month, 3 weeks ago
Hi Heiko,

On Monday, 11 August 2025 05:56:00 EDT Heiko Stübner wrote:
> Hi Detlev,
> 
> Am Freitag, 8. August 2025, 22:03:22 Mitteleuropäische Sommerzeit schrieb 
Detlev Casanova:
> > These variants are found respectively in the RK3588 and RK3576 SoCs.
> > This patch only adds support for H264 and H265 in both variants.
> > 
> > As there is a considerable part of the code that can be shared with the
> > already supported rkvdec decoder driver, the support for these variants
> > is added here rather than writing a new driver.
> > 
> > This patch set uses the newly introduced hevc_ext_sps_[ls]t_rps v4l2
> > controls for HEVC [1].
> > Therefore, a patched version of userpace tools is needed for HEVC
> > support (added for GStreamer[2] and in an early stage for FFmpeg[3]).
> > 
> > This patch set also depends on the preparation patch set sent earlier [4]
> > as well as the iommu restore fix [5] (already merged in linux-media) and
> > Nicolas Frattaroli's bitmap patch [6] to support setting registers that
> > uses upper 16 bits as masks.
> > 
> > [1]:
> > https://lore.kernel.org/all/20250807194327.69900-1-detlev.casanova@collab
> > ora.com/ [2]:
> > https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9355
> > [3]: https://gitlab.collabora.com/detlev/ffmpeg
> > [4]:
> > https://lore.kernel.org/all/20250623160722.55938-1-detlev.casanova@collab
> > ora.com/ [5]:
> > https://lore.kernel.org/all/20250508-rkvdec-iommu-reset-v1-1-c46b6efa6e9b
> > @collabora.com/ [6]:
> > https://lore.kernel.org/all/20250623-byeword-update-v2-1-cf1fc08a2e1f@col
> > labora.com/> 
> > Changes since v1:
> >  - Add parsing of the short and long term ref frame sets from the new v4l2
> >  
> >    controls
> >  
> >  - Add RPS cache to avoid parsing the same data again
> >  - Fix HEVC pixel formats selection
> >  - Fix multiple indentation errors
> 
> when applying the series, git was a bit unhappy about some whitespaces:

Mmh, it looks like some rebase issues. Thanks, I'll fix those !

> > Detlev Casanova (12):
> >   media: rkvdec: Switch to using structs instead of writel
> >   media: rkvdec: Move cabac table to its own source file
> 
> Applying: media: rkvdec: Move cabac table to its own source file
> .git/rebase-apply/patch:535: new blank line at EOF.
> +
> Warnung: 1 Zeile fügt Whitespace-Fehler hinzu.
> 
> >   media: rkvdec: Use structs to represent the HW RPS
> >   media: rkvdec: Move h264 functions to common file
> 
> Applying: media: rkvdec: Move h264 functions to common file
> .git/rebase-apply/patch:278: new blank line at EOF.
> +
> Warnung: 1 Zeile fügt Whitespace-Fehler hinzu.
> 
> >   media: rkvdec: Add per variant configuration
> >   media: rkvdec: Add RCB and SRAM support
> 
> Applying: media: rkvdec: Add RCB and SRAM support
> .git/rebase-apply/patch:200: new blank line at EOF.
> +
> Warnung: 1 Zeile fügt Whitespace-Fehler hinzu.
> 
> >   media: rkvdec: Support per-variant interrupt handler
> >   media: rkvdec: Enable all clocks without naming them
> >   media: rkvdec: Add H264 support for the VDPU381 variant
> >   media: rkvdec: Add H264 support for the VDPU383 variant
> >   media: rkvdec: Add HEVC support for the VDPU381 variant
> 
> Applying: media: rkvdec: Add HEVC support for the VDPU381 variant
> .git/rebase-apply/patch:3483: new blank line at EOF.
> +
> .git/rebase-apply/patch:4035: new blank line at EOF.
> +
> Warnung: 2 Zeilen fügen Whitespace-Fehler hinzu.
> 
> 
> Heiko