rbd: avoid double free memory on error path in rbd_dev_create()

[PATCH] rbd: avoid double free memory on error path in rbd_dev_create()

Posted by Natalia Petrova 2 years, 7 months ago

If rbd_dev_create() fails after assignment 'opts' to 'rbd_dev->opts',
double free of 'rbd_options' happens:
one is in rbd_dev_free() and another one is in do_rbd_add().

Found by Linux Verification Center (linuxtesting.org) with SVACE.

Fixes: 1643dfa4c2c8 ("rbd: introduce a per-device ordered workqueue")
Signed-off-by: Natalia Petrova <n.petrova@fintech.ru>
Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
---
 drivers/block/rbd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 04453f4a319c..ab6bfc352cde 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -5357,7 +5357,6 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
 	if (!rbd_dev)
 		return NULL;
 
-	rbd_dev->opts = opts;
 
 	/* get an id and fill in device name */
 	rbd_dev->dev_id = ida_simple_get(&rbd_dev_id_ida, 0,
@@ -5372,6 +5371,7 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
 	if (!rbd_dev->task_wq)
 		goto fail_dev_id;
 
+	rbd_dev->opts = opts;
 	/* we have a ref from do_rbd_add() */
 	__module_get(THIS_MODULE);
 
-- 
2.34.1

Re: [PATCH] rbd: avoid double free memory on error path in rbd_dev_create()

Posted by Ilya Dryomov 2 years, 7 months ago

On Fri, Feb 3, 2023 at 3:15 PM Natalia Petrova <n.petrova@fintech.ru> wrote:
>
> If rbd_dev_create() fails after assignment 'opts' to 'rbd_dev->opts',
> double free of 'rbd_options' happens:
> one is in rbd_dev_free() and another one is in do_rbd_add().
>
> Found by Linux Verification Center (linuxtesting.org) with SVACE.
>
> Fixes: 1643dfa4c2c8 ("rbd: introduce a per-device ordered workqueue")
> Signed-off-by: Natalia Petrova <n.petrova@fintech.ru>
> Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
> ---
>  drivers/block/rbd.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 04453f4a319c..ab6bfc352cde 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -5357,7 +5357,6 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>         if (!rbd_dev)
>                 return NULL;
>
> -       rbd_dev->opts = opts;
>
>         /* get an id and fill in device name */
>         rbd_dev->dev_id = ida_simple_get(&rbd_dev_id_ida, 0,
> @@ -5372,6 +5371,7 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>         if (!rbd_dev->task_wq)
>                 goto fail_dev_id;
>
> +       rbd_dev->opts = opts;
>         /* we have a ref from do_rbd_add() */
>         __module_get(THIS_MODULE);
>
> --
> 2.34.1
>

Hi Natalia,

It seems like a similar issue is affecting rbd_dev->rbd_client and
rbd_dev->spec.  Unlike rbd_dev->opts, they are ref-counted and I'm
guessing that the verification tool doesn't go that deep.

I'd prefer all three to be addressed in the same change, since it's the
same error path.  Would you be willing to look into that and post a new
revision or should I treat just this patch as a bug report?

Thanks,

                Ilya

RE: [PATCH] rbd: avoid double free memory on error path in rbd_dev_create()

Posted by Петрова Наталия Михайловна 2 years, 7 months ago

Hi Ilya!
Thanks for your response! I don't quite understand your idea and suggestion. The patch is designed to avoid double free memory. I explored the code again and suppose there is another situation for rbd_dev->rbd_client and rbd_dev->spec. Free memory of these pointers is possible only once in rbd_dev_free() function. In do_rbd_add() deallocation memory is only for rbd_opts: drivers/block/rbd.c 7157.
Correct me if I'm wrong.

Thanks, 
Natalia

-----Original Message-----
From: Ilya Dryomov <idryomov@gmail.com> 
Sent: Monday, February 6, 2023 2:59 PM
To: Петрова Наталия Михайловна <n.petrova@fintech.ru>
Cc: Dongsheng Yang <dongsheng.yang@easystack.cn>; Jens Axboe <axboe@kernel.dk>; ceph-devel@vger.kernel.org; linux-block@vger.kernel.org; linux-kernel@vger.kernel.org; lvc-project@linuxtesting.org; Alexey Khoroshilov <khoroshilov@ispras.ru>
Subject: Re: [PATCH] rbd: avoid double free memory on error path in rbd_dev_create()

On Fri, Feb 3, 2023 at 3:15 PM Natalia Petrova <n.petrova@fintech.ru> wrote:
>
> If rbd_dev_create() fails after assignment 'opts' to 'rbd_dev->opts', 
> double free of 'rbd_options' happens:
> one is in rbd_dev_free() and another one is in do_rbd_add().
>
> Found by Linux Verification Center (linuxtesting.org) with SVACE.
>
> Fixes: 1643dfa4c2c8 ("rbd: introduce a per-device ordered workqueue")
> Signed-off-by: Natalia Petrova <n.petrova@fintech.ru>
> Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
> ---
>  drivers/block/rbd.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 
> 04453f4a319c..ab6bfc352cde 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -5357,7 +5357,6 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>         if (!rbd_dev)
>                 return NULL;
>
> -       rbd_dev->opts = opts;
>
>         /* get an id and fill in device name */
>         rbd_dev->dev_id = ida_simple_get(&rbd_dev_id_ida, 0, @@ 
> -5372,6 +5371,7 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>         if (!rbd_dev->task_wq)
>                 goto fail_dev_id;
>
> +       rbd_dev->opts = opts;
>         /* we have a ref from do_rbd_add() */
>         __module_get(THIS_MODULE);
>
> --
> 2.34.1
>

Hi Natalia,

It seems like a similar issue is affecting rbd_dev->rbd_client and rbd_dev->spec.  Unlike rbd_dev->opts, they are ref-counted and I'm guessing that the verification tool doesn't go that deep.

I'd prefer all three to be addressed in the same change, since it's the same error path.  Would you be willing to look into that and post a new revision or should I treat just this patch as a bug report?

Thanks,

                Ilya

Re: [PATCH] rbd: avoid double free memory on error path in rbd_dev_create()

Posted by Xiubo Li 2 years, 7 months ago

On 06/02/2023 23:15, Петрова Наталия Михайловна wrote:
> Hi Ilya!
> Thanks for your response! I don't quite understand your idea and suggestion. The patch is designed to avoid double free memory. I explored the code again and suppose there is another situation for rbd_dev->rbd_client and rbd_dev->spec. Free memory of these pointers is possible only once in rbd_dev_free() function. In do_rbd_add() deallocation memory is only for rbd_opts: drivers/block/rbd.c 7157.

Hi Петрова,

If the rbd_dev_create() fails, for spec it will be freed in 
rbd_dev_create()->rbd_spec_put() first and then in do_rbd_add() it will 
call rbd_spec_put() again.

It won't trigger double free but this should generate a warning when the 
refcount underflow, because the refcount_dec_and_test() will warn and 
then return false when underflow happens.

The same for rbd_client.

Thanks,

- Xiubo

> Correct me if I'm wrong.
>
> Thanks,
> Natalia
>
> -----Original Message-----
> From: Ilya Dryomov <idryomov@gmail.com>
> Sent: Monday, February 6, 2023 2:59 PM
> To: Петрова Наталия Михайловна <n.petrova@fintech.ru>
> Cc: Dongsheng Yang <dongsheng.yang@easystack.cn>; Jens Axboe <axboe@kernel.dk>; ceph-devel@vger.kernel.org; linux-block@vger.kernel.org; linux-kernel@vger.kernel.org; lvc-project@linuxtesting.org; Alexey Khoroshilov <khoroshilov@ispras.ru>
> Subject: Re: [PATCH] rbd: avoid double free memory on error path in rbd_dev_create()
>
> On Fri, Feb 3, 2023 at 3:15 PM Natalia Petrova <n.petrova@fintech.ru> wrote:
>> If rbd_dev_create() fails after assignment 'opts' to 'rbd_dev->opts',
>> double free of 'rbd_options' happens:
>> one is in rbd_dev_free() and another one is in do_rbd_add().
>>
>> Found by Linux Verification Center (linuxtesting.org) with SVACE.
>>
>> Fixes: 1643dfa4c2c8 ("rbd: introduce a per-device ordered workqueue")
>> Signed-off-by: Natalia Petrova <n.petrova@fintech.ru>
>> Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
>> ---
>>   drivers/block/rbd.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index
>> 04453f4a319c..ab6bfc352cde 100644
>> --- a/drivers/block/rbd.c
>> +++ b/drivers/block/rbd.c
>> @@ -5357,7 +5357,6 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>>          if (!rbd_dev)
>>                  return NULL;
>>
>> -       rbd_dev->opts = opts;
>>
>>          /* get an id and fill in device name */
>>          rbd_dev->dev_id = ida_simple_get(&rbd_dev_id_ida, 0, @@
>> -5372,6 +5371,7 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>>          if (!rbd_dev->task_wq)
>>                  goto fail_dev_id;
>>
>> +       rbd_dev->opts = opts;
>>          /* we have a ref from do_rbd_add() */
>>          __module_get(THIS_MODULE);
>>
>> --
>> 2.34.1
>>
> Hi Natalia,
>
> It seems like a similar issue is affecting rbd_dev->rbd_client and rbd_dev->spec.  Unlike rbd_dev->opts, they are ref-counted and I'm guessing that the verification tool doesn't go that deep.
>
> I'd prefer all three to be addressed in the same change, since it's the same error path.  Would you be willing to look into that and post a new revision or should I treat just this patch as a bug report?
>
> Thanks,
>
>                  Ilya

-- 
Best Regards,

Xiubo Li (李秀波)

Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li

[PATCH v2] rbd: fix freeing memory of 'rbd_dev->opts', 'rbd_dev->spec', 'rbd_dev->rbd_client'

Posted by Natalia Petrova 2 years, 7 months ago

If the rbd_dev_create() fails after assignment 'opts' to 'rbd_dev->opts',
double free of 'rbd_options' happens:
one is in rbd_dev_free() and another one is in do_rbd_add().

If the rbd_dev_create() fails, for 'spec' it will be freed in
rbd_dev_create()->rbd_spec_put() first and then in do_rbd_add()
it will call rbd_spec_put() again. The same for 'rbd_client'.
Unlike 'rbd_dev->opts', 'rbd_dev->spec' and 'rbd_dev->rbd_client'
are ref-counted, that's why the ref-count underflow warning
should be generated in rbd_spec_put() and rbd_put_client()
to handle the return values of kref_put().

Found by Linux Verification Center (linuxtesting.org) with SVACE.

Fixes: 1643dfa4c2c8 ("rbd: introduce a per-device ordered workqueue")
Signed-off-by: Natalia Petrova <n.petrova@fintech.ru>
Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
Signed-off-by: Nikita Zhandarovich <n.zhandarovich@fintech.ru>
---
v2: Remarks on the processing of 'rbd_dev->spec' and 'rbd_dev->rbd_client' 
by Ilya Dryomov <idryomov@gmail.com> and Xiubo Li <xiubli@redhat.com> 
were taken into account.
 drivers/block/rbd.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 04453f4a319c..f3f253febe0f 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -889,8 +889,10 @@ static void rbd_client_release(struct kref *kref)
  */
 static void rbd_put_client(struct rbd_client *rbdc)
 {
-	if (rbdc)
-		kref_put(&rbdc->kref, rbd_client_release);
+	if (rbdc) {
+		if (!kref_put(&rbdc->kref, rbd_client_release))
+			pr_warn("The reference count underflow\n");
+	}
 }
 
 /*
@@ -5225,8 +5227,10 @@ static struct rbd_spec *rbd_spec_get(struct rbd_spec *spec)
 static void rbd_spec_free(struct kref *kref);
 static void rbd_spec_put(struct rbd_spec *spec)
 {
-	if (spec)
-		kref_put(&spec->kref, rbd_spec_free);
+	if (spec) {
+		if (!kref_put(&spec->kref, rbd_spec_free))
+			pr_warn("The reference count underflow\n");
+	}
 }
 
 static struct rbd_spec *rbd_spec_alloc(void)
@@ -5357,7 +5361,6 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
 	if (!rbd_dev)
 		return NULL;
 
-	rbd_dev->opts = opts;
 
 	/* get an id and fill in device name */
 	rbd_dev->dev_id = ida_simple_get(&rbd_dev_id_ida, 0,
@@ -5372,6 +5375,7 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
 	if (!rbd_dev->task_wq)
 		goto fail_dev_id;
 
+	rbd_dev->opts = opts;
 	/* we have a ref from do_rbd_add() */
 	__module_get(THIS_MODULE);
 
-- 
2.34.1

Re: [PATCH v2] rbd: fix freeing memory of 'rbd_dev->opts', 'rbd_dev->spec', 'rbd_dev->rbd_client'

Posted by Ilya Dryomov 2 years, 7 months ago

On Thu, Feb 9, 2023 at 1:09 PM Natalia Petrova <n.petrova@fintech.ru> wrote:
>
> If the rbd_dev_create() fails after assignment 'opts' to 'rbd_dev->opts',
> double free of 'rbd_options' happens:
> one is in rbd_dev_free() and another one is in do_rbd_add().
>
> If the rbd_dev_create() fails, for 'spec' it will be freed in
> rbd_dev_create()->rbd_spec_put() first and then in do_rbd_add()
> it will call rbd_spec_put() again. The same for 'rbd_client'.
> Unlike 'rbd_dev->opts', 'rbd_dev->spec' and 'rbd_dev->rbd_client'
> are ref-counted, that's why the ref-count underflow warning
> should be generated in rbd_spec_put() and rbd_put_client()
> to handle the return values of kref_put().

Hi Natalia,

I think you misinterpreted Xiubo.  The underflow warning would be
printed by kref_put() (if one is lucky and the freed memory doesn't get
immediately reallocated and overwritten in which case straight memory
corruption would occur).  There is no need to attempt to print another
warning here.

The problem is potential use-after-free on struct rbd_spec and struct
rbd_client (which is what the warning is for).  This use-after-free is
very similar in nature to what the tool that you are using found for
struct rbd_options (the same bug on the same error path) except that
reference counting is involved: instead of kfree() being called
directly, it's called indirectly from rbd_spec_free() and
rbd_client_release() through rbd_spec_put() and rbd_put_client()
respectively.  Both of these structs have a refcount of 1 here which
means that the first rbd_spec_free() or rbd_client_release() call is
equivalent to kfree() and, when either of them is called again from
do_rbd_add(), use-after-free would occur.

Hope this helps,

                Ilya

>
> Found by Linux Verification Center (linuxtesting.org) with SVACE.
>
> Fixes: 1643dfa4c2c8 ("rbd: introduce a per-device ordered workqueue")
> Signed-off-by: Natalia Petrova <n.petrova@fintech.ru>
> Signed-off-by: Alexey Khoroshilov <khoroshilov@ispras.ru>
> Signed-off-by: Nikita Zhandarovich <n.zhandarovich@fintech.ru>
> ---
> v2: Remarks on the processing of 'rbd_dev->spec' and 'rbd_dev->rbd_client'
> by Ilya Dryomov <idryomov@gmail.com> and Xiubo Li <xiubli@redhat.com>
> were taken into account.
>  drivers/block/rbd.c | 14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
> index 04453f4a319c..f3f253febe0f 100644
> --- a/drivers/block/rbd.c
> +++ b/drivers/block/rbd.c
> @@ -889,8 +889,10 @@ static void rbd_client_release(struct kref *kref)
>   */
>  static void rbd_put_client(struct rbd_client *rbdc)
>  {
> -       if (rbdc)
> -               kref_put(&rbdc->kref, rbd_client_release);
> +       if (rbdc) {
> +               if (!kref_put(&rbdc->kref, rbd_client_release))
> +                       pr_warn("The reference count underflow\n");
> +       }
>  }
>
>  /*
> @@ -5225,8 +5227,10 @@ static struct rbd_spec *rbd_spec_get(struct rbd_spec *spec)
>  static void rbd_spec_free(struct kref *kref);
>  static void rbd_spec_put(struct rbd_spec *spec)
>  {
> -       if (spec)
> -               kref_put(&spec->kref, rbd_spec_free);
> +       if (spec) {
> +               if (!kref_put(&spec->kref, rbd_spec_free))
> +                       pr_warn("The reference count underflow\n");
> +       }
>  }
>
>  static struct rbd_spec *rbd_spec_alloc(void)
> @@ -5357,7 +5361,6 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>         if (!rbd_dev)
>                 return NULL;
>
> -       rbd_dev->opts = opts;
>
>         /* get an id and fill in device name */
>         rbd_dev->dev_id = ida_simple_get(&rbd_dev_id_ida, 0,
> @@ -5372,6 +5375,7 @@ static struct rbd_device *rbd_dev_create(struct rbd_client *rbdc,
>         if (!rbd_dev->task_wq)
>                 goto fail_dev_id;
>
> +       rbd_dev->opts = opts;
>         /* we have a ref from do_rbd_add() */
>         __module_get(THIS_MODULE);
>
> --
> 2.34.1
>