arch/arm64/boot/dts/nvidia/tegra210.dtsi | 14 +- drivers/gpu/drm/tegra/Makefile | 1 + drivers/gpu/drm/tegra/drm.c | 2 + drivers/gpu/drm/tegra/drm.h | 1 + drivers/gpu/drm/tegra/nvjpg.c | 379 +++++++++++++++++++++++++++++++ 5 files changed, 396 insertions(+), 1 deletion(-)
Hello,
This series adds support for the NVJPG hardware accelerator found in the
Tegra210 SoC.
The kernel driver is essentially a copy of the NVDEC driver as both
engines are Falcon-based.
For the userspace part I have written a Mesa Gallium backend [1] that,
while still very much experimental, works in decoding images with VA-API.
I have been using ffmpeg to call VA-API with the following command:
ffmpeg -v verbose -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev /dev/fb0
which decodes <input.jpg> and shows the result in the framebuffer.
The firmware for the engine can be obtained from a Linux for Tegra
distribution. Due to the way the Gallium implementation works for Tegra
the GPU also needs to be enabled.
Thanks!
Diogo
To: Thierry Reding <thierry.reding@gmail.com>
To: Mikko Perttunen <mperttunen@nvidia.com>
To: David Airlie <airlied@gmail.com>
To: Simona Vetter <simona@ffwll.ch>
To: Jonathan Hunter <jonathanh@nvidia.com>
To: Philipp Zabel <p.zabel@pengutronix.de>
To: Rob Herring <robh@kernel.org>
To: Krzysztof Kozlowski <krzk+dt@kernel.org>
To: Conor Dooley <conor+dt@kernel.org>
Cc: linux-kernel@vger.kernel.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-tegra@vger.kernel.org
Cc: devicetree@vger.kernel.org
Signed-off-by: Diogo Ivo <diogo.ivo@tecnico.ulisboa.pt>
[1]: https://gitlab.freedesktop.org/d.ivo/mesa/-/tree/diogo/vaapi_gl?ref_type=heads
---
Diogo Ivo (3):
drm/tegra: Add NVJPG driver
arm64: tegra: Add NVJPG power-domain node
arm64: tegra: Add NVJPG node
arch/arm64/boot/dts/nvidia/tegra210.dtsi | 14 +-
drivers/gpu/drm/tegra/Makefile | 1 +
drivers/gpu/drm/tegra/drm.c | 2 +
drivers/gpu/drm/tegra/drm.h | 1 +
drivers/gpu/drm/tegra/nvjpg.c | 379 +++++++++++++++++++++++++++++++
5 files changed, 396 insertions(+), 1 deletion(-)
---
base-commit: 386b76a190ce68afc19d481f17cab1f5479c719b
change-id: 20250605-diogo-nvjpg-e0d4c57126c5
Best regards,
--
Diogo Ivo <diogo.ivo@tecnico.ulisboa.pt>
On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: > Hello, > > This series adds support for the NVJPG hardware accelerator found in the > Tegra210 SoC. > > The kernel driver is essentially a copy of the NVDEC driver as both > engines are Falcon-based. > > For the userspace part I have written a Mesa Gallium backend [1] that, > while still very much experimental, works in decoding images with VA-API. > > I have been using ffmpeg to call VA-API with the following command: > > ffmpeg -v verbose -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev /dev/fb0 > > which decodes <input.jpg> and shows the result in the framebuffer. > > The firmware for the engine can be obtained from a Linux for Tegra > distribution. By the way, have you tried running this on anything newer than Tegra210? Given your progress on this, we can probably start thinking about submitting the binaries to linux-firmware. > Due to the way the Gallium implementation works for Tegra > the GPU also needs to be enabled. I wonder if maybe we can get rid of this requirement. While it's certainly nice to have the GPU enabled, there may be cases where using only the other engines may be advantageous. Originally when I had worked on VIC, I was looking at how it could be used for compositing without getting the GPU involved. That's something that Android devices tend(ed) to do because of the power savings that come with it. Anyway, not a big deal, depending on the GPU for now is fine. Thierry
On 6/10/25 10:05 AM, Thierry Reding wrote: > On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >> Hello, >> >> This series adds support for the NVJPG hardware accelerator found in the >> Tegra210 SoC. >> >> The kernel driver is essentially a copy of the NVDEC driver as both >> engines are Falcon-based. >> >> For the userspace part I have written a Mesa Gallium backend [1] that, >> while still very much experimental, works in decoding images with VA-API. >> >> I have been using ffmpeg to call VA-API with the following command: >> >> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev /dev/fb0 >> >> which decodes <input.jpg> and shows the result in the framebuffer. >> >> The firmware for the engine can be obtained from a Linux for Tegra >> distribution. > > By the way, have you tried running this on anything newer than Tegra210? > Given your progress on this, we can probably start thinking about > submitting the binaries to linux-firmware. Since I don't have access to other Tegra platforms I haven't been able to test this driver there. For this I need help from someone with access to more hardware, I can send a version that just adds the extra compatibles and we could see if it works. As for the firmwares that would be great! >> Due to the way the Gallium implementation works for Tegra >> the GPU also needs to be enabled. > > I wonder if maybe we can get rid of this requirement. While it's > certainly nice to have the GPU enabled, there may be cases where using > only the other engines may be advantageous. Originally when I had worked > on VIC, I was looking at how it could be used for compositing without > getting the GPU involved. That's something that Android devices tend(ed) > to do because of the power savings that come with it. Yes I think this is possible to do, it mainly has involves properly handling the Gallium driver initialization. I'll take a look at it before submitting the MR for the Gallium driver. Diogo > Anyway, not a big deal, depending on the GPU for now is fine.
On Wed, Jun 11, 2025 at 01:04:14PM +0100, Diogo Ivo wrote: > > > On 6/10/25 10:05 AM, Thierry Reding wrote: > > On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: > > > Hello, > > > > > > This series adds support for the NVJPG hardware accelerator found in the > > > Tegra210 SoC. > > > > > > The kernel driver is essentially a copy of the NVDEC driver as both > > > engines are Falcon-based. > > > > > > For the userspace part I have written a Mesa Gallium backend [1] that, > > > while still very much experimental, works in decoding images with VA-API. > > > > > > I have been using ffmpeg to call VA-API with the following command: > > > > > > ffmpeg -v verbose -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev /dev/fb0 > > > > > > which decodes <input.jpg> and shows the result in the framebuffer. > > > > > > The firmware for the engine can be obtained from a Linux for Tegra > > > distribution. > > > > By the way, have you tried running this on anything newer than Tegra210? > > Given your progress on this, we can probably start thinking about > > submitting the binaries to linux-firmware. > > Since I don't have access to other Tegra platforms I haven't been able > to test this driver there. For this I need help from someone with access > to more hardware, I can send a version that just adds the extra compatibles > and we could see if it works. > > As for the firmwares that would be great! I think both of these are things that Mikko and I can help with. > > > Due to the way the Gallium implementation works for Tegra > > > the GPU also needs to be enabled. > > > > I wonder if maybe we can get rid of this requirement. While it's > > certainly nice to have the GPU enabled, there may be cases where using > > only the other engines may be advantageous. Originally when I had worked > > on VIC, I was looking at how it could be used for compositing without > > getting the GPU involved. That's something that Android devices tend(ed) > > to do because of the power savings that come with it. > > Yes I think this is possible to do, it mainly has involves properly > handling the Gallium driver initialization. I'll take a look at it > before submitting the MR for the Gallium driver. Okay, great. But I think it's definitely something that we can defer if it's non-trivial. Thierry
On 6/11/25 4:10 PM, Thierry Reding wrote: > On Wed, Jun 11, 2025 at 01:04:14PM +0100, Diogo Ivo wrote: >> >> >> On 6/10/25 10:05 AM, Thierry Reding wrote: >>> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>>> Hello, >>>> >>>> This series adds support for the NVJPG hardware accelerator found in the >>>> Tegra210 SoC. >>>> >>>> The kernel driver is essentially a copy of the NVDEC driver as both >>>> engines are Falcon-based. >>>> >>>> For the userspace part I have written a Mesa Gallium backend [1] that, >>>> while still very much experimental, works in decoding images with VA-API. >>>> >>>> I have been using ffmpeg to call VA-API with the following command: >>>> >>>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev /dev/fb0 >>>> >>>> which decodes <input.jpg> and shows the result in the framebuffer. >>>> >>>> The firmware for the engine can be obtained from a Linux for Tegra >>>> distribution. >>> >>> By the way, have you tried running this on anything newer than Tegra210? >>> Given your progress on this, we can probably start thinking about >>> submitting the binaries to linux-firmware. >> >> Since I don't have access to other Tegra platforms I haven't been able >> to test this driver there. For this I need help from someone with access >> to more hardware, I can send a version that just adds the extra compatibles >> and we could see if it works. >> >> As for the firmwares that would be great! > > I think both of these are things that Mikko and I can help with. That would be great (as I imagine you have access to the hardware :) ). Let me know if you have any questions! >>>> Due to the way the Gallium implementation works for Tegra >>>> the GPU also needs to be enabled. >>> >>> I wonder if maybe we can get rid of this requirement. While it's >>> certainly nice to have the GPU enabled, there may be cases where using >>> only the other engines may be advantageous. Originally when I had worked >>> on VIC, I was looking at how it could be used for compositing without >>> getting the GPU involved. That's something that Android devices tend(ed) >>> to do because of the power savings that come with it. >> >> Yes I think this is possible to do, it mainly has involves properly >> handling the Gallium driver initialization. I'll take a look at it >> before submitting the MR for the Gallium driver. > > Okay, great. But I think it's definitely something that we can defer if > it's non-trivial. Perfect then, in that case I'll skip it for now and open a Mesa MR after taking care of some final details. Thanks, Diogo
On 6/10/25 6:05 PM, Thierry Reding wrote: > On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >> Hello, >> >> This series adds support for the NVJPG hardware accelerator found in the >> Tegra210 SoC. >> >> The kernel driver is essentially a copy of the NVDEC driver as both >> engines are Falcon-based. >> >> For the userspace part I have written a Mesa Gallium backend [1] that, >> while still very much experimental, works in decoding images with VA-API. >> >> I have been using ffmpeg to call VA-API with the following command: >> >> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev /dev/fb0 >> >> which decodes <input.jpg> and shows the result in the framebuffer. >> >> The firmware for the engine can be obtained from a Linux for Tegra >> distribution. > > By the way, have you tried running this on anything newer than Tegra210? > Given your progress on this, we can probably start thinking about > submitting the binaries to linux-firmware. FWIW, the impression I have is that NVJPG is basically unchanged all the way to Tegra234. So if we add stream ID support and the firmwares, it'll probably just work. Tegra234 has the quirk that it has two instances of NVJPG -- these have to be distinguished by their different class IDs. But we should go ahead with the T210 support first. > >> Due to the way the Gallium implementation works for Tegra >> the GPU also needs to be enabled. > > I wonder if maybe we can get rid of this requirement. While it's > certainly nice to have the GPU enabled, there may be cases where using > only the other engines may be advantageous. Originally when I had worked > on VIC, I was looking at how it could be used for compositing without > getting the GPU involved. That's something that Android devices tend(ed) > to do because of the power savings that come with it. > > Anyway, not a big deal, depending on the GPU for now is fine. > > Thierry
On 6/10/25 10:52 AM, Mikko Perttunen wrote: > On 6/10/25 6:05 PM, Thierry Reding wrote: >> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>> Hello, >>> >>> This series adds support for the NVJPG hardware accelerator found in the >>> Tegra210 SoC. >>> >>> The kernel driver is essentially a copy of the NVDEC driver as both >>> engines are Falcon-based. >>> >>> For the userspace part I have written a Mesa Gallium backend [1] that, >>> while still very much experimental, works in decoding images with VA- >>> API. >>> >>> I have been using ffmpeg to call VA-API with the following command: >>> >>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 >>> -i <input.jpg> -pix_fmt bgra -f fbdev /dev/fb0 >>> >>> which decodes <input.jpg> and shows the result in the framebuffer. >>> >>> The firmware for the engine can be obtained from a Linux for Tegra >>> distribution. >> >> By the way, have you tried running this on anything newer than Tegra210? >> Given your progress on this, we can probably start thinking about >> submitting the binaries to linux-firmware. > > FWIW, the impression I have is that NVJPG is basically unchanged all the > way to Tegra234. So if we add stream ID support and the firmwares, it'll > probably just work. Tegra234 has the quirk that it has two instances of > NVJPG -- these have to be distinguished by their different class IDs. > But we should go ahead with the T210 support first. I have a question here, what exactly are the stream IDs? While working on the driver this came up and I didn't manage to figure it out. Diogo
On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: > > > On 6/10/25 10:52 AM, Mikko Perttunen wrote: > > On 6/10/25 6:05 PM, Thierry Reding wrote: > > > On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: > > > > Hello, > > > > > > > > This series adds support for the NVJPG hardware accelerator found in the > > > > Tegra210 SoC. > > > > > > > > The kernel driver is essentially a copy of the NVDEC driver as both > > > > engines are Falcon-based. > > > > > > > > For the userspace part I have written a Mesa Gallium backend [1] that, > > > > while still very much experimental, works in decoding images > > > > with VA- API. > > > > > > > > I have been using ffmpeg to call VA-API with the following command: > > > > > > > > ffmpeg -v verbose -hwaccel vaapi -hwaccel_device > > > > /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev > > > > /dev/fb0 > > > > > > > > which decodes <input.jpg> and shows the result in the framebuffer. > > > > > > > > The firmware for the engine can be obtained from a Linux for Tegra > > > > distribution. > > > > > > By the way, have you tried running this on anything newer than Tegra210? > > > Given your progress on this, we can probably start thinking about > > > submitting the binaries to linux-firmware. > > > > FWIW, the impression I have is that NVJPG is basically unchanged all the > > way to Tegra234. So if we add stream ID support and the firmwares, it'll > > probably just work. Tegra234 has the quirk that it has two instances of > > NVJPG -- these have to be distinguished by their different class IDs. > > But we should go ahead with the T210 support first. > > I have a question here, what exactly are the stream IDs? While working > on the driver this came up and I didn't manage to figure it out. Stream IDs are a way to identify memory transactions as belonging to a certain device. This comes into play when working with the IOMMU (which is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and is used to isolate DMA capable devices. Basically for every stream ID you get a separate I/O address space. NVJPG will have its own address space, and so will VIC. Each device can only access whatever has been mapped to it's I/O address space. That means NVJPG can't interfere with VIC and vice-versa. And neither can any of these engines read from or write to random system memory if badly programmed. For Tegra SMMU there's no such thing as programmable stream IDs, so the stream ID is fixed for the given device. On newer chips (Tegra186 and later, or maybe it wasn't until Tegra194), certain IP blocks have special registers that can be used to override the stream ID. There's also a way to set the stream ID via command streams, which means that you can have different I/O address spaces (I think we call them memory context) per engine, which means that you can isolate different processes using the same engine from each other. Again, for Tegra210 that's nothing we need to worry about. For newer chips it's probably just a matter of adding .get_streamid_offset() and .can_use_memory_ctx() implementations. Thierry
On 6/11/25 4:06 PM, Thierry Reding wrote: > On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >> >> >> On 6/10/25 10:52 AM, Mikko Perttunen wrote: >>> On 6/10/25 6:05 PM, Thierry Reding wrote: >>>> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>>>> Hello, >>>>> >>>>> This series adds support for the NVJPG hardware accelerator found in the >>>>> Tegra210 SoC. >>>>> >>>>> The kernel driver is essentially a copy of the NVDEC driver as both >>>>> engines are Falcon-based. >>>>> >>>>> For the userspace part I have written a Mesa Gallium backend [1] that, >>>>> while still very much experimental, works in decoding images >>>>> with VA- API. >>>>> >>>>> I have been using ffmpeg to call VA-API with the following command: >>>>> >>>>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device >>>>> /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev >>>>> /dev/fb0 >>>>> >>>>> which decodes <input.jpg> and shows the result in the framebuffer. >>>>> >>>>> The firmware for the engine can be obtained from a Linux for Tegra >>>>> distribution. >>>> >>>> By the way, have you tried running this on anything newer than Tegra210? >>>> Given your progress on this, we can probably start thinking about >>>> submitting the binaries to linux-firmware. >>> >>> FWIW, the impression I have is that NVJPG is basically unchanged all the >>> way to Tegra234. So if we add stream ID support and the firmwares, it'll >>> probably just work. Tegra234 has the quirk that it has two instances of >>> NVJPG -- these have to be distinguished by their different class IDs. >>> But we should go ahead with the T210 support first. >> >> I have a question here, what exactly are the stream IDs? While working >> on the driver this came up and I didn't manage to figure it out. > > Stream IDs are a way to identify memory transactions as belonging to a > certain device. This comes into play when working with the IOMMU (which > is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and > is used to isolate DMA capable devices. Basically for every stream ID > you get a separate I/O address space. NVJPG will have its own address > space, and so will VIC. Each device can only access whatever has been > mapped to it's I/O address space. That means NVJPG can't interfere with > VIC and vice-versa. And neither can any of these engines read from or > write to random system memory if badly programmed. So if I understand this correctly a Stream ID corresponds to an IOMMU domain right? > For Tegra SMMU there's no such thing as programmable stream IDs, so the > stream ID is fixed for the given device. > > On newer chips (Tegra186 and later, or maybe it wasn't until Tegra194), > certain IP blocks have special registers that can be used to override > the stream ID. There's also a way to set the stream ID via command > streams, which means that you can have different I/O address spaces (I > think we call them memory context) per engine, which means that you can > isolate different processes using the same engine from each other. > > Again, for Tegra210 that's nothing we need to worry about. For newer > chips it's probably just a matter of adding .get_streamid_offset() and > .can_use_memory_ctx() implementations. Ok, then in that case I'll keep the driver in its current state without these implementations if that's ok. Connected with this I wanted to know your thoughts on the best way to upstream this, is it better to wait for testing on different platforms first and then if things work merge a driver that works for all of them or go with Tegra210 first and then add more platforms later on? Thanks, Diogo
On 6/16/25 7:21 PM, Diogo Ivo wrote: > > > On 6/11/25 4:06 PM, Thierry Reding wrote: >> On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >>> >>> >>> On 6/10/25 10:52 AM, Mikko Perttunen wrote: >>>> On 6/10/25 6:05 PM, Thierry Reding wrote: >>>>> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>>>>> Hello, >>>>>> >>>>>> This series adds support for the NVJPG hardware accelerator found >>>>>> in the >>>>>> Tegra210 SoC. >>>>>> >>>>>> The kernel driver is essentially a copy of the NVDEC driver as both >>>>>> engines are Falcon-based. >>>>>> >>>>>> For the userspace part I have written a Mesa Gallium backend [1] >>>>>> that, >>>>>> while still very much experimental, works in decoding images >>>>>> with VA- API. >>>>>> >>>>>> I have been using ffmpeg to call VA-API with the following command: >>>>>> >>>>>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device >>>>>> /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev >>>>>> /dev/fb0 >>>>>> >>>>>> which decodes <input.jpg> and shows the result in the framebuffer. >>>>>> >>>>>> The firmware for the engine can be obtained from a Linux for Tegra >>>>>> distribution. >>>>> >>>>> By the way, have you tried running this on anything newer than >>>>> Tegra210? >>>>> Given your progress on this, we can probably start thinking about >>>>> submitting the binaries to linux-firmware. >>>> >>>> FWIW, the impression I have is that NVJPG is basically unchanged all >>>> the >>>> way to Tegra234. So if we add stream ID support and the firmwares, >>>> it'll >>>> probably just work. Tegra234 has the quirk that it has two instances of >>>> NVJPG -- these have to be distinguished by their different class IDs. >>>> But we should go ahead with the T210 support first. >>> >>> I have a question here, what exactly are the stream IDs? While working >>> on the driver this came up and I didn't manage to figure it out. >> >> Stream IDs are a way to identify memory transactions as belonging to a >> certain device. This comes into play when working with the IOMMU (which >> is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and >> is used to isolate DMA capable devices. Basically for every stream ID >> you get a separate I/O address space. NVJPG will have its own address >> space, and so will VIC. Each device can only access whatever has been >> mapped to it's I/O address space. That means NVJPG can't interfere with >> VIC and vice-versa. And neither can any of these engines read from or >> write to random system memory if badly programmed. > > So if I understand this correctly a Stream ID corresponds to an IOMMU > domain right? Technically not necessarily, but in practice that's the case, as the IOMMU driver creates IOMMU domains for each stream ID in the device tree. They are similar to the SWGROUPs on Tegra210. > >> For Tegra SMMU there's no such thing as programmable stream IDs, so the >> stream ID is fixed for the given device. >> >> On newer chips (Tegra186 and later, or maybe it wasn't until Tegra194), >> certain IP blocks have special registers that can be used to override >> the stream ID. There's also a way to set the stream ID via command >> streams, which means that you can have different I/O address spaces (I >> think we call them memory context) per engine, which means that you can >> isolate different processes using the same engine from each other. >> >> Again, for Tegra210 that's nothing we need to worry about. For newer >> chips it's probably just a matter of adding .get_streamid_offset() and >> .can_use_memory_ctx() implementations. > > Ok, then in that case I'll keep the driver in its current state without > these implementations if that's ok. Connected with this I wanted to know > your thoughts on the best way to upstream this, is it better to wait for > testing on different platforms first and then if things work merge a > driver that works for all of them or go with Tegra210 first and then add > more platforms later on? Personally, I'd say to go for Tegra210 first. > > Thanks, > Diogo > Cheers Mikko
On 6/17/25 5:40 AM, Mikko Perttunen wrote: > > > On 6/16/25 7:21 PM, Diogo Ivo wrote: >> >> >> On 6/11/25 4:06 PM, Thierry Reding wrote: >>> On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >>>> >>>> >>>> On 6/10/25 10:52 AM, Mikko Perttunen wrote: >>>>> On 6/10/25 6:05 PM, Thierry Reding wrote: >>>>>> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>>>>>> Hello, >>>>>>> >>>>>>> This series adds support for the NVJPG hardware accelerator found >>>>>>> in the >>>>>>> Tegra210 SoC. >>>>>>> >>>>>>> The kernel driver is essentially a copy of the NVDEC driver as both >>>>>>> engines are Falcon-based. >>>>>>> >>>>>>> For the userspace part I have written a Mesa Gallium backend [1] >>>>>>> that, >>>>>>> while still very much experimental, works in decoding images >>>>>>> with VA- API. >>>>>>> >>>>>>> I have been using ffmpeg to call VA-API with the following command: >>>>>>> >>>>>>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device >>>>>>> /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev >>>>>>> /dev/fb0 >>>>>>> >>>>>>> which decodes <input.jpg> and shows the result in the framebuffer. >>>>>>> >>>>>>> The firmware for the engine can be obtained from a Linux for Tegra >>>>>>> distribution. >>>>>> >>>>>> By the way, have you tried running this on anything newer than >>>>>> Tegra210? >>>>>> Given your progress on this, we can probably start thinking about >>>>>> submitting the binaries to linux-firmware. >>>>> >>>>> FWIW, the impression I have is that NVJPG is basically unchanged >>>>> all the >>>>> way to Tegra234. So if we add stream ID support and the firmwares, >>>>> it'll >>>>> probably just work. Tegra234 has the quirk that it has two >>>>> instances of >>>>> NVJPG -- these have to be distinguished by their different class IDs. >>>>> But we should go ahead with the T210 support first. >>>> >>>> I have a question here, what exactly are the stream IDs? While working >>>> on the driver this came up and I didn't manage to figure it out. >>> >>> Stream IDs are a way to identify memory transactions as belonging to a >>> certain device. This comes into play when working with the IOMMU (which >>> is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and >>> is used to isolate DMA capable devices. Basically for every stream ID >>> you get a separate I/O address space. NVJPG will have its own address >>> space, and so will VIC. Each device can only access whatever has been >>> mapped to it's I/O address space. That means NVJPG can't interfere with >>> VIC and vice-versa. And neither can any of these engines read from or >>> write to random system memory if badly programmed. >> >> So if I understand this correctly a Stream ID corresponds to an IOMMU >> domain right? > > Technically not necessarily, but in practice that's the case, as the > IOMMU driver creates IOMMU domains for each stream ID in the device > tree. They are similar to the SWGROUPs on Tegra210. Ok that makes sense, thank you for the clarification :) >> Ok, then in that case I'll keep the driver in its current state without >> these implementations if that's ok. Connected with this I wanted to know >> your thoughts on the best way to upstream this, is it better to wait for >> testing on different platforms first and then if things work merge a >> driver that works for all of them or go with Tegra210 first and then add >> more platforms later on? > > Personally, I'd say to go for Tegra210 first. In that case I believe that in the v2 I sent out of the driver I addressed both yours and Thierry's reviews and the driver should be in good condition for Tegra210. What are the next steps in order to merge it? Thanks, Diogo > Cheers > Mikko
On 6/17/25 6:26 PM, Diogo Ivo wrote: > > > On 6/17/25 5:40 AM, Mikko Perttunen wrote: >> >> >> On 6/16/25 7:21 PM, Diogo Ivo wrote: >>> >>> >>> On 6/11/25 4:06 PM, Thierry Reding wrote: >>>> On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >>>>> >>>>> >>>>> On 6/10/25 10:52 AM, Mikko Perttunen wrote: >>>>>> On 6/10/25 6:05 PM, Thierry Reding wrote: >>>>>>> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> This series adds support for the NVJPG hardware accelerator >>>>>>>> found in the >>>>>>>> Tegra210 SoC. >>>>>>>> >>>>>>>> The kernel driver is essentially a copy of the NVDEC driver as both >>>>>>>> engines are Falcon-based. >>>>>>>> >>>>>>>> For the userspace part I have written a Mesa Gallium backend [1] >>>>>>>> that, >>>>>>>> while still very much experimental, works in decoding images >>>>>>>> with VA- API. >>>>>>>> >>>>>>>> I have been using ffmpeg to call VA-API with the following command: >>>>>>>> >>>>>>>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device >>>>>>>> /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev >>>>>>>> /dev/fb0 >>>>>>>> >>>>>>>> which decodes <input.jpg> and shows the result in the framebuffer. >>>>>>>> >>>>>>>> The firmware for the engine can be obtained from a Linux for Tegra >>>>>>>> distribution. >>>>>>> >>>>>>> By the way, have you tried running this on anything newer than >>>>>>> Tegra210? >>>>>>> Given your progress on this, we can probably start thinking about >>>>>>> submitting the binaries to linux-firmware. >>>>>> >>>>>> FWIW, the impression I have is that NVJPG is basically unchanged >>>>>> all the >>>>>> way to Tegra234. So if we add stream ID support and the firmwares, >>>>>> it'll >>>>>> probably just work. Tegra234 has the quirk that it has two >>>>>> instances of >>>>>> NVJPG -- these have to be distinguished by their different class IDs. >>>>>> But we should go ahead with the T210 support first. >>>>> >>>>> I have a question here, what exactly are the stream IDs? While working >>>>> on the driver this came up and I didn't manage to figure it out. >>>> >>>> Stream IDs are a way to identify memory transactions as belonging to a >>>> certain device. This comes into play when working with the IOMMU (which >>>> is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and >>>> is used to isolate DMA capable devices. Basically for every stream ID >>>> you get a separate I/O address space. NVJPG will have its own address >>>> space, and so will VIC. Each device can only access whatever has been >>>> mapped to it's I/O address space. That means NVJPG can't interfere with >>>> VIC and vice-versa. And neither can any of these engines read from or >>>> write to random system memory if badly programmed. >>> >>> So if I understand this correctly a Stream ID corresponds to an IOMMU >>> domain right? >> >> Technically not necessarily, but in practice that's the case, as the >> IOMMU driver creates IOMMU domains for each stream ID in the device >> tree. They are similar to the SWGROUPs on Tegra210. > > Ok that makes sense, thank you for the clarification :) > >>> Ok, then in that case I'll keep the driver in its current state without >>> these implementations if that's ok. Connected with this I wanted to know >>> your thoughts on the best way to upstream this, is it better to wait for >>> testing on different platforms first and then if things work merge a >>> driver that works for all of them or go with Tegra210 first and then add >>> more platforms later on? >> >> Personally, I'd say to go for Tegra210 first. > > In that case I believe that in the v2 I sent out of the driver I addressed > both yours and Thierry's reviews and the driver should be in good condition > for Tegra210. What are the next steps in order to merge it? I left one more comment on the v2 patch. With that fixed, if Thierry has no further objections, hopefully he can merge. Cheers, Mikko > > Thanks, > Diogo > >> Cheers >> Mikko >
Hi Mikko, On 6/30/25 7:14 AM, Mikko Perttunen wrote: > On 6/17/25 6:26 PM, Diogo Ivo wrote: >> >> >> On 6/17/25 5:40 AM, Mikko Perttunen wrote: >>> >>> >>> On 6/16/25 7:21 PM, Diogo Ivo wrote: >>>> >>>> >>>> On 6/11/25 4:06 PM, Thierry Reding wrote: >>>>> On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >>>>>> >>>>>> >>>>>> On 6/10/25 10:52 AM, Mikko Perttunen wrote: >>>>>>> On 6/10/25 6:05 PM, Thierry Reding wrote: >>>>>>>> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> This series adds support for the NVJPG hardware accelerator >>>>>>>>> found in the >>>>>>>>> Tegra210 SoC. >>>>>>>>> >>>>>>>>> The kernel driver is essentially a copy of the NVDEC driver as >>>>>>>>> both >>>>>>>>> engines are Falcon-based. >>>>>>>>> >>>>>>>>> For the userspace part I have written a Mesa Gallium backend >>>>>>>>> [1] that, >>>>>>>>> while still very much experimental, works in decoding images >>>>>>>>> with VA- API. >>>>>>>>> >>>>>>>>> I have been using ffmpeg to call VA-API with the following >>>>>>>>> command: >>>>>>>>> >>>>>>>>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device >>>>>>>>> /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev >>>>>>>>> /dev/fb0 >>>>>>>>> >>>>>>>>> which decodes <input.jpg> and shows the result in the framebuffer. >>>>>>>>> >>>>>>>>> The firmware for the engine can be obtained from a Linux for Tegra >>>>>>>>> distribution. >>>>>>>> >>>>>>>> By the way, have you tried running this on anything newer than >>>>>>>> Tegra210? >>>>>>>> Given your progress on this, we can probably start thinking about >>>>>>>> submitting the binaries to linux-firmware. >>>>>>> >>>>>>> FWIW, the impression I have is that NVJPG is basically unchanged >>>>>>> all the >>>>>>> way to Tegra234. So if we add stream ID support and the >>>>>>> firmwares, it'll >>>>>>> probably just work. Tegra234 has the quirk that it has two >>>>>>> instances of >>>>>>> NVJPG -- these have to be distinguished by their different class >>>>>>> IDs. >>>>>>> But we should go ahead with the T210 support first. >>>>>> >>>>>> I have a question here, what exactly are the stream IDs? While >>>>>> working >>>>>> on the driver this came up and I didn't manage to figure it out. >>>>> >>>>> Stream IDs are a way to identify memory transactions as belonging to a >>>>> certain device. This comes into play when working with the IOMMU >>>>> (which >>>>> is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and >>>>> is used to isolate DMA capable devices. Basically for every stream ID >>>>> you get a separate I/O address space. NVJPG will have its own address >>>>> space, and so will VIC. Each device can only access whatever has been >>>>> mapped to it's I/O address space. That means NVJPG can't interfere >>>>> with >>>>> VIC and vice-versa. And neither can any of these engines read from or >>>>> write to random system memory if badly programmed. >>>> >>>> So if I understand this correctly a Stream ID corresponds to an IOMMU >>>> domain right? >>> >>> Technically not necessarily, but in practice that's the case, as the >>> IOMMU driver creates IOMMU domains for each stream ID in the device >>> tree. They are similar to the SWGROUPs on Tegra210. >> >> Ok that makes sense, thank you for the clarification :) >> >>>> Ok, then in that case I'll keep the driver in its current state without >>>> these implementations if that's ok. Connected with this I wanted to >>>> know >>>> your thoughts on the best way to upstream this, is it better to wait >>>> for >>>> testing on different platforms first and then if things work merge a >>>> driver that works for all of them or go with Tegra210 first and then >>>> add >>>> more platforms later on? >>> >>> Personally, I'd say to go for Tegra210 first. >> >> In that case I believe that in the v2 I sent out of the driver I >> addressed >> both yours and Thierry's reviews and the driver should be in good >> condition >> for Tegra210. What are the next steps in order to merge it? > > I left one more comment on the v2 patch. With that fixed, if Thierry has > no further objections, hopefully he can merge. I sent out v3 addressing this comment and picked up your Acked-by. Thanks for the review! Diogo > Cheers, > Mikko > >> >> Thanks, >> Diogo >> >>> Cheers >>> Mikko >> >
On 6/12/25 12:06 AM, Thierry Reding wrote: > On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >> >> >> On 6/10/25 10:52 AM, Mikko Perttunen wrote: >>> On 6/10/25 6:05 PM, Thierry Reding wrote: >>>> On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >>>>> Hello, >>>>> >>>>> This series adds support for the NVJPG hardware accelerator found in the >>>>> Tegra210 SoC. >>>>> >>>>> The kernel driver is essentially a copy of the NVDEC driver as both >>>>> engines are Falcon-based. >>>>> >>>>> For the userspace part I have written a Mesa Gallium backend [1] that, >>>>> while still very much experimental, works in decoding images >>>>> with VA- API. >>>>> >>>>> I have been using ffmpeg to call VA-API with the following command: >>>>> >>>>> ffmpeg -v verbose -hwaccel vaapi -hwaccel_device >>>>> /dev/dri/renderD129 -i <input.jpg> -pix_fmt bgra -f fbdev >>>>> /dev/fb0 >>>>> >>>>> which decodes <input.jpg> and shows the result in the framebuffer. >>>>> >>>>> The firmware for the engine can be obtained from a Linux for Tegra >>>>> distribution. >>>> >>>> By the way, have you tried running this on anything newer than Tegra210? >>>> Given your progress on this, we can probably start thinking about >>>> submitting the binaries to linux-firmware. >>> >>> FWIW, the impression I have is that NVJPG is basically unchanged all the >>> way to Tegra234. So if we add stream ID support and the firmwares, it'll >>> probably just work. Tegra234 has the quirk that it has two instances of >>> NVJPG -- these have to be distinguished by their different class IDs. >>> But we should go ahead with the T210 support first. >> >> I have a question here, what exactly are the stream IDs? While working >> on the driver this came up and I didn't manage to figure it out. > > Stream IDs are a way to identify memory transactions as belonging to a > certain device. This comes into play when working with the IOMMU (which > is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and > is used to isolate DMA capable devices. Basically for every stream ID > you get a separate I/O address space. NVJPG will have its own address > space, and so will VIC. Each device can only access whatever has been > mapped to it's I/O address space. That means NVJPG can't interfere with > VIC and vice-versa. And neither can any of these engines read from or > write to random system memory if badly programmed. > > For Tegra SMMU there's no such thing as programmable stream IDs, so the > stream ID is fixed for the given device. > > On newer chips (Tegra186 and later, or maybe it wasn't until Tegra194), Tegra186 and newer -- all chips with the ARM SMMU. To add a little bit, each engine can address two stream IDs, one for firmware and one for data. All user specified buffers are mapped into the data IOMMU domain, and these are switched between jobs / applications. As an aside, currently each engine has its own firmware stream ID, but that's a bit wasteful, since the kernel allocates a separate IOMMU domain for each. The firmwares are all read-only so they could be in a shared one. We've had to consolidate these on some platforms that ran out of IOMMU domains otherwise. Not really a concern with upstream platforms, though. > certain IP blocks have special registers that can be used to override > the stream ID. There's also a way to set the stream ID via command > streams, which means that you can have different I/O address spaces (I > think we call them memory context) per engine, which means that you can > isolate different processes using the same engine from each other. > > Again, for Tegra210 that's nothing we need to worry about. For newer > chips it's probably just a matter of adding .get_streamid_offset() and > .can_use_memory_ctx() implementations. Also need to program the THI_STREAMID / TRANSCFG registers during boot. Cheers, Mikko > > Thierry
On 6/12/25 2:55 AM, Mikko Perttunen wrote: > On 6/12/25 12:06 AM, Thierry Reding wrote: >> On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >>> I have a question here, what exactly are the stream IDs? While working >>> on the driver this came up and I didn't manage to figure it out. >> >> Stream IDs are a way to identify memory transactions as belonging to a >> certain device. This comes into play when working with the IOMMU (which >> is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and >> is used to isolate DMA capable devices. Basically for every stream ID >> you get a separate I/O address space. NVJPG will have its own address >> space, and so will VIC. Each device can only access whatever has been >> mapped to it's I/O address space. That means NVJPG can't interfere with >> VIC and vice-versa. And neither can any of these engines read from or >> write to random system memory if badly programmed. >> >> For Tegra SMMU there's no such thing as programmable stream IDs, so the >> stream ID is fixed for the given device. >> >> On newer chips (Tegra186 and later, or maybe it wasn't until Tegra194), > > Tegra186 and newer -- all chips with the ARM SMMU. To add a little bit, > each engine can address two stream IDs, one for firmware and one for > data. All user specified buffers are mapped into the data IOMMU domain, > and these are switched between jobs / applications. > > As an aside, currently each engine has its own firmware stream ID, but > that's a bit wasteful, since the kernel allocates a separate IOMMU > domain for each. The firmwares are all read-only so they could be in a > shared one. We've had to consolidate these on some platforms that ran > out of IOMMU domains otherwise. Not really a concern with upstream > platforms, though. Does this dual Stream ID also apply to Tegra210? > Also need to program the THI_STREAMID / TRANSCFG registers during boot. Thanks, Diogo
On 6/16/25 7:23 PM, Diogo Ivo wrote: > > > On 6/12/25 2:55 AM, Mikko Perttunen wrote: >> On 6/12/25 12:06 AM, Thierry Reding wrote: >>> On Wed, Jun 11, 2025 at 01:05:40PM +0100, Diogo Ivo wrote: >>>> I have a question here, what exactly are the stream IDs? While working >>>> on the driver this came up and I didn't manage to figure it out. >>> >>> Stream IDs are a way to identify memory transactions as belonging to a >>> certain device. This comes into play when working with the IOMMU (which >>> is a Tegra SMMU on Tegra210 and earlier, and an ARM SMMU on Tegra) and >>> is used to isolate DMA capable devices. Basically for every stream ID >>> you get a separate I/O address space. NVJPG will have its own address >>> space, and so will VIC. Each device can only access whatever has been >>> mapped to it's I/O address space. That means NVJPG can't interfere with >>> VIC and vice-versa. And neither can any of these engines read from or >>> write to random system memory if badly programmed. >>> >>> For Tegra SMMU there's no such thing as programmable stream IDs, so the >>> stream ID is fixed for the given device. >>> >>> On newer chips (Tegra186 and later, or maybe it wasn't until Tegra194), >> >> Tegra186 and newer -- all chips with the ARM SMMU. To add a little >> bit, each engine can address two stream IDs, one for firmware and one >> for data. All user specified buffers are mapped into the data IOMMU >> domain, and these are switched between jobs / applications. >> >> As an aside, currently each engine has its own firmware stream ID, but >> that's a bit wasteful, since the kernel allocates a separate IOMMU >> domain for each. The firmwares are all read-only so they could be in a >> shared one. We've had to consolidate these on some platforms that ran >> out of IOMMU domains otherwise. Not really a concern with upstream >> platforms, though. > > Does this dual Stream ID also apply to Tegra210? No, only Tegra186 and later (chips with ARM SMMU). > >> Also need to program the THI_STREAMID / TRANSCFG registers during boot. > > Thanks, > Diogo >
On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: > Hello, > > This series adds support for the NVJPG hardware accelerator found in the > Tegra210 SoC. > > The kernel driver is essentially a copy of the NVDEC driver as both > engines are Falcon-based. > > For the userspace part I have written a Mesa Gallium backend [1] that, > while still very much experimental, works in decoding images with VA-API. Nice. It's good to see that there's some use in this after all. I haven't taken an in-depth look yet, but from a high-level point of view this looks like what I had imagined back when I started out with this driver. This made my day, thank you! Thierry
On 6/10/25 9:58 AM, Thierry Reding wrote: > On Fri, Jun 06, 2025 at 11:45:33AM +0100, Diogo Ivo wrote: >> Hello, >> >> This series adds support for the NVJPG hardware accelerator found in the >> Tegra210 SoC. >> >> The kernel driver is essentially a copy of the NVDEC driver as both >> engines are Falcon-based. >> >> For the userspace part I have written a Mesa Gallium backend [1] that, >> while still very much experimental, works in decoding images with VA-API. > > Nice. It's good to see that there's some use in this after all. I > haven't taken an in-depth look yet, but from a high-level point of view > this looks like what I had imagined back when I started out with this > driver. Good to know that the structure makes sense to you :) I still have some kinks to iron out before submitting a merge request in Mesa but if you have the time to take a look before that and you already have some comments before that that would be great. > This made my day, thank you! Happy to know that! Diogo
© 2016 - 2025 Red Hat, Inc.