With kdump enabled, when kernel crashes, the system could boot into the
kdump kernel and dump the memory image i.e. /proc/vmcore to a specified
target. Currently, when dumping vmcore to a LUKS encrypted device, there
are two problems,
 - for some machines, the user may don't have a chance enter the password
   to decrypt the device after kernel crashes and kdump initrd is loaded
 - LUKS2 by default use the memory-hard Argon2 key derivation function
   which is quite memory-consuming compared to the limited memory reserved
   for kdump. Take Fedora example, by default, only 256M is reserved for
   systems having memory between 4G-64G. With LUKS enabled, ~1300M needs
   to be reserved for kdump.
 
Besides the users (at least for Fedora) usually expect kdump to work out
of the box i.e. no manual password input is needed. And it doesn't make
sense to derivate the master key again in kdump kernel which seems to be
redundant work.

Based on Milan's feedback [1] on Kairui's ideas to support kdump with
LUKS encryption, this patch set addresses the above issues by
 1) first saving the LUKS master key to kexec when opening the encrypted
    device 
 2) then saving the master key to the reserved memory for kdump when 
    loading kdump kernel image.

So the LUKS master key never leaves the kernel space and once the key has
been saved to the reserved memory for kdump, it would be wiped 
immediately. If there is no security concern with this approach or any
other concern, I will drop the following assumptions made for this RFC
version in v1,
 - only x86 is supported
 - there is only one LUKS device for the system

to extend the support to other architectures including POWER, ARM and
s390x and address the case of multiple LUKS devices. Any feedback will be 
appreciated, thanks!

For a proof of concept, I've patched cryptsetup [2] in a quick-and-dirty
way to support a new option "--kdump-kernel-master-key"
and hacked systemd [3]. It works for Fedora 34.

[1] https://yhbt.net/lore/all/e5abd089-3398-fdb4-7991-0019be434b79@gmail.com/
[2] https://gitlab.com/coxu/cryptsetup/-/commit/ee54bb15445da0bc3f9155a7227a9799da4dac20
[3] https://github.com/coiby/systemd/tree/reuse_kdump_master_key

Coiby Xu (4):
  kexec, dm-crypt: receive LUKS master key from dm-crypt and pass it to
    kdump
  kdump, x86: pass the LUKS master key to kdump kernel using a kernel
    command line parameter luksmasterkey
  crash_dump: retrieve LUKS master key in kdump kernel
  dm-crypt: reuse LUKS master key in kdump kernel

 arch/x86/include/asm/crash.h      |  1 +
 arch/x86/kernel/crash.c           | 42 ++++++++++++++++++-
 arch/x86/kernel/kexec-bzimage64.c |  7 ++++
 drivers/md/dm-crypt.c             | 26 +++++++++---
 include/linux/crash_dump.h        |  4 ++
 include/linux/kexec.h             |  7 ++++
 kernel/crash_dump.c               | 69 +++++++++++++++++++++++++++++++
 kernel/kexec_core.c               | 66 +++++++++++++++++++++++++++++
 8 files changed, 215 insertions(+), 7 deletions(-)

-- 
2.34.1

On 18/03/2022 11:34, Coiby Xu wrote:
> With kdump enabled, when kernel crashes, the system could boot into the
> kdump kernel and dump the memory image i.e. /proc/vmcore to a specified
> target. Currently, when dumping vmcore to a LUKS encrypted device, there
> are two problems,
>   - for some machines, the user may don't have a chance enter the password
>     to decrypt the device after kernel crashes and kdump initrd is loaded
>   - LUKS2 by default use the memory-hard Argon2 key derivation function
>     which is quite memory-consuming compared to the limited memory reserved
>     for kdump. Take Fedora example, by default, only 256M is reserved for
>     systems having memory between 4G-64G. With LUKS enabled, ~1300M needs
>     to be reserved for kdump.
>   
> Besides the users (at least for Fedora) usually expect kdump to work out
> of the box i.e. no manual password input is needed. And it doesn't make
> sense to derivate the master key again in kdump kernel which seems to be
> redundant work.
> 
> Based on Milan's feedback [1] on Kairui's ideas to support kdump with
> LUKS encryption, this patch set addresses the above issues by

Hi,

I think you are creating another attack vector here, storing the encryption
key to yet another place... but I already mentioned that in the referenced mail.

Why is it not done through keyring and forcing kdump to retain key there
(under the same keyring key name as dm-crypt used)?
Kernel dm-crypt supports this already; LUKS2 uses keyring by default too.
That's all you need, or not? Why do you need to add another "kdump:" thing?
IOW why kdump cannot copy the key to keyring under the name dm-crypt
has in the mapping table and let dm-crypt activate the device almost without
code changes?

Anyway, please fix the naming before this patchset can be read or reviewed!

LUKS is user-space key management only (on-disk metadata); the kernel has
no idea how the key is derived or what LUKS is - dm-crypt only knows the key
(either through keyring or directly in the mapping table).

Polluting kernel namespace with "luks" names variables is wrong - dm-crypt
is used in many other mappings (plain, bitlocker, veracrypt, ...)
Just use the dm-crypt key, do not reference LUKS at all.

Milan



>   1) first saving the LUKS master key to kexec when opening the encrypted
>      device
>   2) then saving the master key to the reserved memory for kdump when
>      loading kdump kernel image.
> 
> So the LUKS master key never leaves the kernel space and once the key has
> been saved to the reserved memory for kdump, it would be wiped
> immediately. If there is no security concern with this approach or any
> other concern, I will drop the following assumptions made for this RFC
> version in v1,
>   - only x86 is supported
>   - there is only one LUKS device for the system
> 
> to extend the support to other architectures including POWER, ARM and
> s390x and address the case of multiple LUKS devices. Any feedback will be
> appreciated, thanks!
> 
> For a proof of concept, I've patched cryptsetup [2] in a quick-and-dirty
> way to support a new option "--kdump-kernel-master-key"
> and hacked systemd [3]. It works for Fedora 34.
> 
> [1] https://yhbt.net/lore/all/e5abd089-3398-fdb4-7991-0019be434b79@gmail.com/
> [2] https://gitlab.com/coxu/cryptsetup/-/commit/ee54bb15445da0bc3f9155a7227a9799da4dac20
> [3] https://github.com/coiby/systemd/tree/reuse_kdump_master_key
> 
> Coiby Xu (4):
>    kexec, dm-crypt: receive LUKS master key from dm-crypt and pass it to
>      kdump
>    kdump, x86: pass the LUKS master key to kdump kernel using a kernel
>      command line parameter luksmasterkey
>    crash_dump: retrieve LUKS master key in kdump kernel
>    dm-crypt: reuse LUKS master key in kdump kernel
> 
>   arch/x86/include/asm/crash.h      |  1 +
>   arch/x86/kernel/crash.c           | 42 ++++++++++++++++++-
>   arch/x86/kernel/kexec-bzimage64.c |  7 ++++
>   drivers/md/dm-crypt.c             | 26 +++++++++---
>   include/linux/crash_dump.h        |  4 ++
>   include/linux/kexec.h             |  7 ++++
>   kernel/crash_dump.c               | 69 +++++++++++++++++++++++++++++++
>   kernel/kexec_core.c               | 66 +++++++++++++++++++++++++++++
>   8 files changed, 215 insertions(+), 7 deletions(-)
>

Hi Milan,

On Fri, Mar 18, 2022 at 12:29:06PM +0100, Milan Broz wrote:
>On 18/03/2022 11:34, Coiby Xu wrote:
>>With kdump enabled, when kernel crashes, the system could boot into the
>>kdump kernel and dump the memory image i.e. /proc/vmcore to a specified
>>target. Currently, when dumping vmcore to a LUKS encrypted device, there
>>are two problems,
>>  - for some machines, the user may don't have a chance enter the password
>>    to decrypt the device after kernel crashes and kdump initrd is loaded
>>  - LUKS2 by default use the memory-hard Argon2 key derivation function
>>    which is quite memory-consuming compared to the limited memory reserved
>>    for kdump. Take Fedora example, by default, only 256M is reserved for
>>    systems having memory between 4G-64G. With LUKS enabled, ~1300M needs
>>    to be reserved for kdump.
>>Besides the users (at least for Fedora) usually expect kdump to work out
>>of the box i.e. no manual password input is needed. And it doesn't make
>>sense to derivate the master key again in kdump kernel which seems to be
>>redundant work.
>>
>>Based on Milan's feedback [1] on Kairui's ideas to support kdump with
>>LUKS encryption, this patch set addresses the above issues by
>
>Hi,
>
>I think you are creating another attack vector here, storing the encryption
>key to yet another place... but I already mentioned that in the referenced mail.

Thanks for the quick response! I thought the take-home message of the
referenced is we shouldn't leave the master key in the user space. Sorry 
I seem to still misunderstand what's mentioned in the referenced mail. But
take AES as an example, my reasoning is as follows:
   the expanded key has already been stored in "struct crypto_aes_ctx".
   If a hacker can somehow access the master key stored in the memory
   reserved exclusively for kdump, it means he/she can also access the
   expanded key stored in struct crypto_aes_ctx. So  not creating
   another attack vector?

>
>Why is it not done through keyring and forcing kdump to retain key there
>(under the same keyring key name as dm-crypt used)?
>Kernel dm-crypt supports this already; LUKS2 uses keyring by default too.
>That's all you need, or not? Why do you need to add another "kdump:" thing?
>IOW why kdump cannot copy the key to keyring under the name dm-crypt
>has in the mapping table and let dm-crypt activate the device almost without
>code changes?

Sorry, I haven't explained how kdump works. Once the 1st kernel crashes and
the system boots into the kdump kernel, the kdump kernel only have direct
access to the memory exclusively reserved for it i.e. the kdump kernel
loses the direct access to the keyring constructed in the 1st kernel. In
theory, the kdump kernel could do some "hacking" to find out the key
stored in the memory directly managed by the 1st kernel but I imagine
this would be difficult task (imagine I present the memory dump of my
computer to you and ask you to rebuild all the relevant kernel data
structures and find the key). Besides, it's not reliable to read the
memory directly managed by the first kernel for example the memory could
be corrupt. So we have to pass the master key from the 1st kernel to the kdump
kernel.

Btw, if we store the key in the kernel keyring, doesn't it mean storing the
"encryption key to another place" and thus creating "another attach vector"?

>
>Anyway, please fix the naming before this patchset can be read or reviewed!
>
>LUKS is user-space key management only (on-disk metadata); the kernel has
>no idea how the key is derived or what LUKS is - dm-crypt only knows the key
>(either through keyring or directly in the mapping table).
>
>Polluting kernel namespace with "luks" names variables is wrong - dm-crypt
>is used in many other mappings (plain, bitlocker, veracrypt, ...)
>Just use the dm-crypt key, do not reference LUKS at all.

Thanks for pointing out my mistake of misusing the word LUKS! I'll fix this
error once we settle down what's the correct approach to reuse master
key for the kdump kernel.

>
>Milan
>
>
>
>>  1) first saving the LUKS master key to kexec when opening the encrypted
>>     device
>>  2) then saving the master key to the reserved memory for kdump when
>>     loading kdump kernel image.
>>
>>So the LUKS master key never leaves the kernel space and once the key has
>>been saved to the reserved memory for kdump, it would be wiped
>>immediately. If there is no security concern with this approach or any
>>other concern, I will drop the following assumptions made for this RFC
>>version in v1,
>>  - only x86 is supported
>>  - there is only one LUKS device for the system
>>
>>to extend the support to other architectures including POWER, ARM and
>>s390x and address the case of multiple LUKS devices. Any feedback will be
>>appreciated, thanks!
>>
>>For a proof of concept, I've patched cryptsetup [2] in a quick-and-dirty
>>way to support a new option "--kdump-kernel-master-key"
>>and hacked systemd [3]. It works for Fedora 34.
>>
>>[1] https://yhbt.net/lore/all/e5abd089-3398-fdb4-7991-0019be434b79@gmail.com/
>>[2] https://gitlab.com/coxu/cryptsetup/-/commit/ee54bb15445da0bc3f9155a7227a9799da4dac20
>>[3] https://github.com/coiby/systemd/tree/reuse_kdump_master_key
>>
>>Coiby Xu (4):
>>   kexec, dm-crypt: receive LUKS master key from dm-crypt and pass it to
>>     kdump
>>   kdump, x86: pass the LUKS master key to kdump kernel using a kernel
>>     command line parameter luksmasterkey
>>   crash_dump: retrieve LUKS master key in kdump kernel
>>   dm-crypt: reuse LUKS master key in kdump kernel
>>
>>  arch/x86/include/asm/crash.h      |  1 +
>>  arch/x86/kernel/crash.c           | 42 ++++++++++++++++++-
>>  arch/x86/kernel/kexec-bzimage64.c |  7 ++++
>>  drivers/md/dm-crypt.c             | 26 +++++++++---
>>  include/linux/crash_dump.h        |  4 ++
>>  include/linux/kexec.h             |  7 ++++
>>  kernel/crash_dump.c               | 69 +++++++++++++++++++++++++++++++
>>  kernel/kexec_core.c               | 66 +++++++++++++++++++++++++++++
>>  8 files changed, 215 insertions(+), 7 deletions(-)
>>
>

-- 
Best regards,
Coiby

On 18/03/2022 13:21, Coiby Xu wrote:
...
>> Why is it not done through keyring and forcing kdump to retain key there
>> (under the same keyring key name as dm-crypt used)?
>> Kernel dm-crypt supports this already; LUKS2 uses keyring by default too.
>> That's all you need, or not? Why do you need to add another "kdump:" thing?
>> IOW why kdump cannot copy the key to keyring under the name dm-crypt
>> has in the mapping table and let dm-crypt activate the device almost without
>> code changes?
> 
> Sorry, I haven't explained how kdump works. Once the 1st kernel crashes and
> the system boots into the kdump kernel, the kdump kernel only have direct
> access to the memory exclusively reserved for it i.e. the kdump kernel
> loses the direct access to the keyring constructed in the 1st kernel. In
> theory, the kdump kernel could do some "hacking" to find out the key
> stored in the memory directly managed by the 1st kernel but I imagine
> this would be difficult task (imagine I present the memory dump of my
> computer to you and ask you to rebuild all the relevant kernel data
> structures and find the key). Besides, it's not reliable to read the
> memory directly managed by the first kernel for example the memory could
> be corrupt. So we have to pass the master key from the 1st kernel to the kdump
> kernel.

OK, then why you cannot store it to the (2nd) kdump kernel keyring?
(From the kdump area copy, then you do not need to patch anything else
in dm-crypt than that one line storing the key to the kdump area.)

A clear approach would be to store the key in the 2nd kernel kdump keyring
and allow userspace to read it.
Then cryptsetup can just validate the key (LUKS key digest does not use Argon)
and activates it without asking for a passphrase.
Perhaps this will need some new cryptsetup option (or API call), but I think
it can be done.

Or, you can actually simulate it with
   cryptsetup open ... --master-key-file <file>
where this keyfile contains directly the volume key, not a passphrase.
The key digest is verified in this case only; no costly PBKDF is needed.

If you have a way to retrieve the kdump stored key to kdump userspace, this
is perhaps a much simpler solution.

All this is against all countermeasures to not expose encryption key
directly - but if kdump is debugging environment, just saying...

Milan

On Fri, Mar 18, 2022 at 02:53:37PM +0100, Milan Broz wrote:
>On 18/03/2022 13:21, Coiby Xu wrote:
>...
>>>Why is it not done through keyring and forcing kdump to retain key there
>>>(under the same keyring key name as dm-crypt used)?
>>>Kernel dm-crypt supports this already; LUKS2 uses keyring by default too.
>>>That's all you need, or not? Why do you need to add another "kdump:" thing?
>>>IOW why kdump cannot copy the key to keyring under the name dm-crypt
>>>has in the mapping table and let dm-crypt activate the device almost without
>>>code changes?
>>
>>Sorry, I haven't explained how kdump works. Once the 1st kernel crashes and
>>the system boots into the kdump kernel, the kdump kernel only have direct
>>access to the memory exclusively reserved for it i.e. the kdump kernel
>>loses the direct access to the keyring constructed in the 1st kernel. In
>>theory, the kdump kernel could do some "hacking" to find out the key
>>stored in the memory directly managed by the 1st kernel but I imagine
>>this would be difficult task (imagine I present the memory dump of my
>>computer to you and ask you to rebuild all the relevant kernel data
>>structures and find the key). Besides, it's not reliable to read the
>>memory directly managed by the first kernel for example the memory could
>>be corrupt. So we have to pass the master key from the 1st kernel to the kdump
>>kernel.
>
>OK, then why you cannot store it to the (2nd) kdump kernel keyring?
>(From the kdump area copy, then you do not need to patch anything else
>in dm-crypt than that one line storing the key to the kdump area.)

cryptsetup stores the master key in the thread keyring
(KEY_SPEC_THREAD_KEYRING). I couldn't come up with a easy way to store
the key to the (2nd) kdump kernel keyring. For example, one problem is
how can I know in adance the thread ID of cryptsetup? I imagine
drivers/md/dm-crypt.c needs to support a new type of message in
crypt_message so cryptsetup can tell dm-crypt to store the master key
in the thread keyring as a preparation for opening the device. So one
line change of code isn't sufficient. What's more important is I don't see 
any benefit by doing so I regards storing the key to the kdump kernel
keyring first as an unnecessary step.

>
>A clear approach would be to store the key in the 2nd kernel kdump keyring
>and allow userspace to read it.
>Then cryptsetup can just validate the key (LUKS key digest does not use Argon)
>and activates it without asking for a passphrase.

Thanks for bringing my attention to validating the key! Currently I
simply skip key validation in cryptsetup when implementing the
"--kdump-master-key" option because it requires pass the master key to
the user space. I have a side question, if somehow the master key is
changed and the wrong key is used to get the expanded key, will it lead
to a devastating effect say corrupting the user data on the disk?

>Perhaps this will need some new cryptsetup option (or API call), but I think
>it can be done.
>
>Or, you can actually simulate it with
>  cryptsetup open ... --master-key-file <file>
>where this keyfile contains directly the volume key, not a passphrase.
>The key digest is verified in this case only; no costly PBKDF is needed.
>
>If you have a way to retrieve the kdump stored key to kdump userspace, this
>is perhaps a much simpler solution.
>
>All this is against all countermeasures to not expose encryption key
>directly - but if kdump is debugging environment, just saying...

I respect and appreciate all the efforts done by you and other community
members to increase security margin so the last thing I want to do is to
reverse the security measures. Besides, I believe some users choose to
dump vmcore to an encrypted disk exactly out of security concern. The
only thing this patch set does about master key is to store it in the
memory exclusively reserved for kdump. To be precise, storing the master
key in the memory exclusively reserved for kdump may be different from
what you mean by "exposing encryption key directly". This reserved memory
won't be used by the 1st kernel (which means if we reserve 1GB more for
the kdump kernel, the 1st kernel would have 1GB less kernel to use. And I
would expect strong objection from the user due to a loss of 1GB memory).
Only kexec knows how to store the master key in this exclusively reserved
memory area and but kexec itself doesn't know to retrieve the master key in
the 1st kernel. So the situation is similar to how expanded key is stored.
struct crypto_aes_ctx has the expanded key, but there is no API to retrieve
it. If you still think this line of reasoning doesn't convince you, how
about encrypting the master key before storing the key to the kdump memory?
kexec could calculate the SHA256 digest of loaded kdump kernel and initrd.
We can use this SHA256 digest to get an encryption key and then encrypt
the master key.

>
>Milan
>

-- 
Best regards,
Coiby

On 18/03/2022 07:34, Coiby Xu wrote:
> [...]
> Based on Milan's feedback [1] on Kairui's ideas to support kdump with
> LUKS encryption, this patch set addresses the above issues by
>  1) first saving the LUKS master key to kexec when opening the encrypted
>     device 
>  2) then saving the master key to the reserved memory for kdump when 
>     loading kdump kernel image.
> 
> So the LUKS master key never leaves the kernel space and once the key has
> been saved to the reserved memory for kdump, it would be wiped 
> immediately. If there is no security concern with this approach or any
> other concern, I will drop the following assumptions made for this RFC
> version in v1,
>  - only x86 is supported
>  - there is only one LUKS device for the system
> 
> to extend the support to other architectures including POWER, ARM and
> s390x and address the case of multiple LUKS devices. Any feedback will be 
> appreciated, thanks!
> 

Hi Coiby, thanks for the very interesting work!
I confess I didn't review the code as I have not much experience in
dm-crypt/key management, but I have a generic question related with the
motivation of the patch set.

My understanding is that one (the main?) motivation of this series would
be to protect the saved memory (vmcore) from being read by some
"unauthorized" entity - in order to achieve this goal, it is hereby
proposed to allow kdump kernel to access a memory-saved key and with
that, mount an encrypted volume, saving the vmcore over there correct?

So, what if instead of playing with the volume key, users with this
concern address that by reserving some *unencrypted partition* for
saving the vmcore, but then *encrypt the vmcore* itself! So, instead of
requiring saving a full-volume key, mount everything, risk data
corruption if something goes bad...we just have makedumpfile encrypting
the vmcore with some preloaded key (which might be saved inside the
kdump minimal intird, for example), and saving the encrypted file into a
clear/unencrypted volume? This way we also prevent excessive memory
consumption during kdump due to the lvm/dm-userspace paraphernalia usage.

Does it make sense or am I being silly or missing something?
Cheers,


Guilherme

On Sat, Mar 19, 2022 at 05:13:21PM -0300, Guilherme G. Piccoli wrote:
>On 18/03/2022 07:34, Coiby Xu wrote:
>> [...]
>> Based on Milan's feedback [1] on Kairui's ideas to support kdump with
>> LUKS encryption, this patch set addresses the above issues by
>>  1) first saving the LUKS master key to kexec when opening the encrypted
>>     device
>>  2) then saving the master key to the reserved memory for kdump when
>>     loading kdump kernel image.
>>
>> So the LUKS master key never leaves the kernel space and once the key has
>> been saved to the reserved memory for kdump, it would be wiped
>> immediately. If there is no security concern with this approach or any
>> other concern, I will drop the following assumptions made for this RFC
>> version in v1,
>>  - only x86 is supported
>>  - there is only one LUKS device for the system
>>
>> to extend the support to other architectures including POWER, ARM and
>> s390x and address the case of multiple LUKS devices. Any feedback will be
>> appreciated, thanks!
>>
>
>Hi Coiby, thanks for the very interesting work!

Hi Guilherme,

I'm glad this work interests you and thanks for sharing your thoughts!

>I confess I didn't review the code as I have not much experience in
>dm-crypt/key management, but I have a generic question related with the
>motivation of the patch set.
>
>My understanding is that one (the main?) motivation of this series would
>be to protect the saved memory (vmcore) from being read by some
>"unauthorized" entity - in order to achieve this goal, it is hereby
>proposed to allow kdump kernel to access a memory-saved key and with
>that, mount an encrypted volume, saving the vmcore over there correct?

>
>So, what if instead of playing with the volume key, users with this
>concern address that by reserving some *unencrypted partition* for
>saving the vmcore, but then *encrypt the vmcore* itself! So, instead of
>requiring saving a full-volume key, mount everything, risk data
>corruption if something goes bad...we just have makedumpfile encrypting
>the vmcore with some preloaded key (which might be saved inside the
>kdump minimal intird, for example), and saving the encrypted file into a
>clear/unencrypted volume? This way we also prevent excessive memory
>consumption during kdump due to the lvm/dm-userspace paraphernalia usage.

I believe some users have security concern for where to save vmcore.
This use case exactly fits your description and your proposed solution
shall be good for this type of users. But I think many more users may
just choose to encrypt the hard drive when installing the system and
they would naturally expect kdump to work for the case of full disk
encryption. So your proposed solution may not address the latter case 
where there is a much large user base.

>
>Does it make sense or am I being silly or missing something?
>Cheers,
>
>
>Guilherme
>
>_______________________________________________
>kexec mailing list
>kexec@lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/kexec
>

-- 
Best regards,
Coiby

On 20/03/2022 22:41, Coiby Xu wrote:
> [...]
> 
> I believe some users have security concern for where to save vmcore.
> This use case exactly fits your description and your proposed solution
> shall be good for this type of users. But I think many more users may
> just choose to encrypt the hard drive when installing the system and
> they would naturally expect kdump to work for the case of full disk
> encryption. So your proposed solution may not address the latter case 
> where there is a much large user base.
> 

Thanks Coiby, makes sense, your idea is more generic and seems to
address all the use cases!
Cheers,


Guilherme