[v2] media: verisilicon: Fix crash when probing encoder

[PATCH v2] media: verisilicon: Fix crash when probing encoder

Posted by Benjamin Gaignard 2 years, 8 months ago

ctx->vpu_dst_fmt is no more initialized before calling hantro_try_fmt()
so assigne it to vpu_fmt led to crash the kernel.
Like for decoder case use 'fmt' as format for encoder and clean up
the code.

Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com>
Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
Fixes: db6f68b51e5c ("media: verisilicon: Do not set context src/dst formats in reset functions")
---
version 2:
- Remove useless vpu_fmt.

 drivers/media/platform/verisilicon/hantro_v4l2.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
index 8f1414085f47..d71f79471396 100644
--- a/drivers/media/platform/verisilicon/hantro_v4l2.c
+++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
@@ -275,7 +275,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 			  struct v4l2_pix_format_mplane *pix_mp,
 			  enum v4l2_buf_type type)
 {
-	const struct hantro_fmt *fmt, *vpu_fmt;
+	const struct hantro_fmt *fmt;
 	bool capture = V4L2_TYPE_IS_CAPTURE(type);
 	bool coded;
 
@@ -295,11 +295,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 
 	if (coded) {
 		pix_mp->num_planes = 1;
-		vpu_fmt = fmt;
-	} else if (ctx->is_encoder) {
-		vpu_fmt = ctx->vpu_dst_fmt;
-	} else {
-		vpu_fmt = fmt;
+	} else if (!ctx->is_encoder) {
 		/*
 		 * Width/height on the CAPTURE end of a decoder are ignored and
 		 * replaced by the OUTPUT ones.
@@ -311,7 +307,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
 	pix_mp->field = V4L2_FIELD_NONE;
 
 	v4l2_apply_frmsize_constraints(&pix_mp->width, &pix_mp->height,
-				       &vpu_fmt->frmsize);
+				       &fmt->frmsize);
 
 	if (!coded) {
 		/* Fill remaining fields */
-- 
2.34.1

Re: [PATCH v2] media: verisilicon: Fix crash when probing encoder

Posted by Michael Tretter 2 years, 6 months ago

Hi Benjamin,

On Thu, 13 Apr 2023 12:47:56 +0200, Benjamin Gaignard wrote:
> ctx->vpu_dst_fmt is no more initialized before calling hantro_try_fmt()
> so assigne it to vpu_fmt led to crash the kernel.
> Like for decoder case use 'fmt' as format for encoder and clean up
> the code.
> 
> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Fixes: db6f68b51e5c ("media: verisilicon: Do not set context src/dst formats in reset functions")
> ---
> version 2:
> - Remove useless vpu_fmt.
> 
>  drivers/media/platform/verisilicon/hantro_v4l2.c | 10 +++-------
>  1 file changed, 3 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
> index 8f1414085f47..d71f79471396 100644
> --- a/drivers/media/platform/verisilicon/hantro_v4l2.c
> +++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
> @@ -275,7 +275,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>  			  struct v4l2_pix_format_mplane *pix_mp,
>  			  enum v4l2_buf_type type)
>  {
> -	const struct hantro_fmt *fmt, *vpu_fmt;
> +	const struct hantro_fmt *fmt;
>  	bool capture = V4L2_TYPE_IS_CAPTURE(type);
>  	bool coded;
>  
> @@ -295,11 +295,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>  
>  	if (coded) {
>  		pix_mp->num_planes = 1;
> -		vpu_fmt = fmt;
> -	} else if (ctx->is_encoder) {
> -		vpu_fmt = ctx->vpu_dst_fmt;
> -	} else {
> -		vpu_fmt = fmt;
> +	} else if (!ctx->is_encoder) {
>  		/*
>  		 * Width/height on the CAPTURE end of a decoder are ignored and
>  		 * replaced by the OUTPUT ones.
> @@ -311,7 +307,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>  	pix_mp->field = V4L2_FIELD_NONE;
>  
>  	v4l2_apply_frmsize_constraints(&pix_mp->width, &pix_mp->height,
> -				       &vpu_fmt->frmsize);
> +				       &fmt->frmsize);

This causes a regression on the OUTPUT device of the encoder. fmt->frmsize is
only valid for coded ("bitstream") formats, but fmt on the OUTPUT of an
encoder will be a raw format. This results in width and height to be clamped
to 0.

I think the correct fix would be to apply the frmsize constraints of the
currently configured coded format, but as ctx->vpu_dst_fmt is not initialized
before calling this code, I don't know how to get the coded format.

Michael

>  
>  	if (!coded) {
>  		/* Fill remaining fields */
> -- 
> 2.34.1
>

Re: [PATCH v2] media: verisilicon: Fix crash when probing encoder

Posted by Benjamin Gaignard 2 years, 6 months ago

Le 01/06/2023 à 15:27, Michael Tretter a écrit :
> Hi Benjamin,
>
> On Thu, 13 Apr 2023 12:47:56 +0200, Benjamin Gaignard wrote:
>> ctx->vpu_dst_fmt is no more initialized before calling hantro_try_fmt()
>> so assigne it to vpu_fmt led to crash the kernel.
>> Like for decoder case use 'fmt' as format for encoder and clean up
>> the code.
>>
>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com>
>> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
>> Fixes: db6f68b51e5c ("media: verisilicon: Do not set context src/dst formats in reset functions")
>> ---
>> version 2:
>> - Remove useless vpu_fmt.
>>
>>   drivers/media/platform/verisilicon/hantro_v4l2.c | 10 +++-------
>>   1 file changed, 3 insertions(+), 7 deletions(-)
>>
>> diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
>> index 8f1414085f47..d71f79471396 100644
>> --- a/drivers/media/platform/verisilicon/hantro_v4l2.c
>> +++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
>> @@ -275,7 +275,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>>   			  struct v4l2_pix_format_mplane *pix_mp,
>>   			  enum v4l2_buf_type type)
>>   {
>> -	const struct hantro_fmt *fmt, *vpu_fmt;
>> +	const struct hantro_fmt *fmt;
>>   	bool capture = V4L2_TYPE_IS_CAPTURE(type);
>>   	bool coded;
>>   
>> @@ -295,11 +295,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>>   
>>   	if (coded) {
>>   		pix_mp->num_planes = 1;
>> -		vpu_fmt = fmt;
>> -	} else if (ctx->is_encoder) {
>> -		vpu_fmt = ctx->vpu_dst_fmt;
>> -	} else {
>> -		vpu_fmt = fmt;
>> +	} else if (!ctx->is_encoder) {
>>   		/*
>>   		 * Width/height on the CAPTURE end of a decoder are ignored and
>>   		 * replaced by the OUTPUT ones.
>> @@ -311,7 +307,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>>   	pix_mp->field = V4L2_FIELD_NONE;
>>   
>>   	v4l2_apply_frmsize_constraints(&pix_mp->width, &pix_mp->height,
>> -				       &vpu_fmt->frmsize);
>> +				       &fmt->frmsize);
> This causes a regression on the OUTPUT device of the encoder. fmt->frmsize is
> only valid for coded ("bitstream") formats, but fmt on the OUTPUT of an
> encoder will be a raw format. This results in width and height to be clamped
> to 0.
>
> I think the correct fix would be to apply the frmsize constraints of the
> currently configured coded format, but as ctx->vpu_dst_fmt is not initialized
> before calling this code, I don't know how to get the coded format.

if ctx->dst_fmt is correctly set (and it should be) then doing:

pix_mp->width = ctx->dst_fmt.width;
pix_mp->height = ctx->dst_fmt.height;

should solve the issue.

Benjamin

>
> Michael
>
>>   
>>   	if (!coded) {
>>   		/* Fill remaining fields */
>> -- 
>> 2.34.1
>>

Re: [PATCH v2] media: verisilicon: Fix crash when probing encoder

Posted by Michael Tretter 2 years, 5 months ago

On Thu, 01 Jun 2023 16:08:13 +0200, Benjamin Gaignard wrote:
> Le 01/06/2023 à 15:27, Michael Tretter a écrit :
> > On Thu, 13 Apr 2023 12:47:56 +0200, Benjamin Gaignard wrote:
> > > ctx->vpu_dst_fmt is no more initialized before calling hantro_try_fmt()
> > > so assigne it to vpu_fmt led to crash the kernel.
> > > Like for decoder case use 'fmt' as format for encoder and clean up
> > > the code.
> > > 
> > > Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com>
> > > Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> > > Fixes: db6f68b51e5c ("media: verisilicon: Do not set context src/dst formats in reset functions")
> > > ---
> > > version 2:
> > > - Remove useless vpu_fmt.
> > > 
> > >   drivers/media/platform/verisilicon/hantro_v4l2.c | 10 +++-------
> > >   1 file changed, 3 insertions(+), 7 deletions(-)
> > > 
> > > diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
> > > index 8f1414085f47..d71f79471396 100644
> > > --- a/drivers/media/platform/verisilicon/hantro_v4l2.c
> > > +++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
> > > @@ -275,7 +275,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
> > >   			  struct v4l2_pix_format_mplane *pix_mp,
> > >   			  enum v4l2_buf_type type)
> > >   {
> > > -	const struct hantro_fmt *fmt, *vpu_fmt;
> > > +	const struct hantro_fmt *fmt;
> > >   	bool capture = V4L2_TYPE_IS_CAPTURE(type);
> > >   	bool coded;
> > > @@ -295,11 +295,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
> > >   	if (coded) {
> > >   		pix_mp->num_planes = 1;
> > > -		vpu_fmt = fmt;
> > > -	} else if (ctx->is_encoder) {
> > > -		vpu_fmt = ctx->vpu_dst_fmt;
> > > -	} else {
> > > -		vpu_fmt = fmt;
> > > +	} else if (!ctx->is_encoder) {
> > >   		/*
> > >   		 * Width/height on the CAPTURE end of a decoder are ignored and
> > >   		 * replaced by the OUTPUT ones.
> > > @@ -311,7 +307,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
> > >   	pix_mp->field = V4L2_FIELD_NONE;
> > >   	v4l2_apply_frmsize_constraints(&pix_mp->width, &pix_mp->height,
> > > -				       &vpu_fmt->frmsize);
> > > +				       &fmt->frmsize);
> > This causes a regression on the OUTPUT device of the encoder. fmt->frmsize is
> > only valid for coded ("bitstream") formats, but fmt on the OUTPUT of an
> > encoder will be a raw format. This results in width and height to be clamped
> > to 0.
> > 
> > I think the correct fix would be to apply the frmsize constraints of the
> > currently configured coded format, but as ctx->vpu_dst_fmt is not initialized
> > before calling this code, I don't know how to get the coded format.
> 
> if ctx->dst_fmt is correctly set (and it should be) then doing:
> 
> pix_mp->width = ctx->dst_fmt.width;
> pix_mp->height = ctx->dst_fmt.height;
> 
> should solve the issue.

Using the width and height of dst_fmt for OUTPUT is not correct, since the
v4l2 stateless encoder spec dictates that the size is set on the OUTPUT device
and will be ignored on the CAPTURE device.

I sent a patch [0] to apply the frame constraints using dst_fmt.

Michael

[0] https://lore.kernel.org/linux-media/20230706071510.1717684-1-m.tretter@pengutronix.de/

> > >   	if (!coded) {
> > >   		/* Fill remaining fields */
> > > -- 
> > > 2.34.1
> > > 
>

Re: [PATCH v2] media: verisilicon: Fix crash when probing encoder

Posted by AngeloGioacchino Del Regno 2 years, 8 months ago

Il 13/04/23 12:47, Benjamin Gaignard ha scritto:
> ctx->vpu_dst_fmt is no more initialized before calling hantro_try_fmt()
> so assigne it to vpu_fmt led to crash the kernel.
> Like for decoder case use 'fmt' as format for encoder and clean up
> the code.
> 
> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Fixes: db6f68b51e5c ("media: verisilicon: Do not set context src/dst formats in reset functions")

Reviewed-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>

Re: [PATCH v2] media: verisilicon: Fix crash when probing encoder

Posted by Nicolas Dufresne 2 years, 8 months ago

Hi,

cause I sent a Rb on old one after v2 was sent, the extra cleanup make sense to
me.

Le jeudi 13 avril 2023 à 12:47 +0200, Benjamin Gaignard a écrit :
> ctx->vpu_dst_fmt is no more initialized before calling hantro_try_fmt()
> so assigne it to vpu_fmt led to crash the kernel.
> Like for decoder case use 'fmt' as format for encoder and clean up
> the code.
> 
> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com>
> Tested-by: Marek Szyprowski <m.szyprowski@samsung.com>
> Fixes: db6f68b51e5c ("media: verisilicon: Do not set context src/dst formats in reset functions")


Reviewed-by: Nicolas Dufresne <nicolas.dufresne@collabora.com>

> ---
> version 2:
> - Remove useless vpu_fmt.
> 
>  drivers/media/platform/verisilicon/hantro_v4l2.c | 10 +++-------
>  1 file changed, 3 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/media/platform/verisilicon/hantro_v4l2.c b/drivers/media/platform/verisilicon/hantro_v4l2.c
> index 8f1414085f47..d71f79471396 100644
> --- a/drivers/media/platform/verisilicon/hantro_v4l2.c
> +++ b/drivers/media/platform/verisilicon/hantro_v4l2.c
> @@ -275,7 +275,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>  			  struct v4l2_pix_format_mplane *pix_mp,
>  			  enum v4l2_buf_type type)
>  {
> -	const struct hantro_fmt *fmt, *vpu_fmt;
> +	const struct hantro_fmt *fmt;
>  	bool capture = V4L2_TYPE_IS_CAPTURE(type);
>  	bool coded;
>  
> @@ -295,11 +295,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>  
>  	if (coded) {
>  		pix_mp->num_planes = 1;
> -		vpu_fmt = fmt;
> -	} else if (ctx->is_encoder) {
> -		vpu_fmt = ctx->vpu_dst_fmt;
> -	} else {
> -		vpu_fmt = fmt;
> +	} else if (!ctx->is_encoder) {
>  		/*
>  		 * Width/height on the CAPTURE end of a decoder are ignored and
>  		 * replaced by the OUTPUT ones.
> @@ -311,7 +307,7 @@ static int hantro_try_fmt(const struct hantro_ctx *ctx,
>  	pix_mp->field = V4L2_FIELD_NONE;
>  
>  	v4l2_apply_frmsize_constraints(&pix_mp->width, &pix_mp->height,
> -				       &vpu_fmt->frmsize);
> +				       &fmt->frmsize);
>  
>  	if (!coded) {
>  		/* Fill remaining fields */
> -- 
> 2.34.1
> 
>