Don't change the quantization of an RGB format when converting between
RGB and YUV with the RGA3. As the RGA3 only supports doing conversions
to full range YUV with BT601, it wants to announce it through
try_fmt/s_fmt.

As it is only relevant, when converting between RGB and YUV, it's
guarded by a given condition. But the condition also causes the
RGB format quantization to be adjusted to limited range, which
is incorrect.

Therefore simplify the condition to only apply when the current format
is a YUV format. Also document the reason for checking if the other
format is an RGB format.

Fixes: 24a63d4c9d3c ("media: rockchip: rga: add rga3 support")
Signed-off-by: Sven Püschel <s.pueschel@pengutronix.de>
---
This patch is a fix to the upcoming RGA3 support. By mistake it'll also
adjust the RGB quantization instead of only adjusting the quantization
of the YUV formats. Therefore remove the incorrect conditions and
document why we also check the format of the other side.

I've just noticed this flawed condition and didn't check/reproduce it in
a practical scenario.

    v4l2-compliance 1.32.0, 64 bits, 64-bit time_t
    ....
    	Card type        : rga3
    ...
    Total for rockchip-rga device /dev/video0: 48, Succeeded: 48, Failed: 0, Warnings: 0

    v4l2-compliance 1.32.0, 64 bits, 64-bit time_t
    ...
    	Card type        : rga2
    ...
    Total for rockchip-rga device /dev/video1: 48, Succeeded: 48, Failed: 0, Warnings: 0
---
 drivers/media/platform/rockchip/rga/rga3-hw.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/rockchip/rga/rga3-hw.c b/drivers/media/platform/rockchip/rga/rga3-hw.c
index ca1c268303dd4..c4a71306278ba 100644
--- a/drivers/media/platform/rockchip/rga/rga3-hw.c
+++ b/drivers/media/platform/rockchip/rga/rga3-hw.c
@@ -450,10 +450,14 @@ static void *rga3_adjust_and_map_format(struct rga_ctx *ctx,
 	other_format = is_output ? &ctx->out.pix : &ctx->in.pix;
 	other_format_info = v4l2_format_info(other_format->pixelformat);
 
-	if ((v4l2_is_format_rgb(format_info) &&
-	     v4l2_is_format_yuv(other_format_info)) ||
-	    (v4l2_is_format_yuv(format_info) &&
-	     v4l2_is_format_rgb(other_format_info))) {
+	/*
+	 * Only apply the quantization restrictions when we need to
+	 * convert between RGB and YUV. Otherwise there is no point
+	 * to limit the quantization for operations like scaling or
+	 * rotations.
+	 */
+	if (v4l2_is_format_yuv(format_info) &&
+	    v4l2_is_format_rgb(other_format_info)) {
 		/*
 		 * The RGA3 only supports BT601, BT709 and BT2020 RGB<->YUV conversions
 		 * Additionally BT709 and BT2020 only support limited range YUV.

---
base-commit: 6a75e3d4f6428b90f398354212e3a2e0172851d6
change-id: 20260605-spu-fixrga3quantization-061c761a1d80

Best regards,
--  
Sven Püschel <s.pueschel@pengutronix.de>