block/meson.build | 1 + block/monitor/bitmap-qmp-cmds.c | 61 ++++++++ block/raw-format-bitmap.c | 163 ++++++++++++++++++++ block/raw-format.c | 256 ++++++++++++++++++++++++++++++-- block/raw-format.h | 50 +++++++ qapi/block-core.json | 64 +++++++- 6 files changed, 583 insertions(+), 12 deletions(-) create mode 100644 block/raw-format-bitmap.c create mode 100644 block/raw-format.h
Currently, QEMU doesn't support persistent dirty bitmaps for raw format and also dirty bitmaps are for internal use only, and cannot be accessed using third-party applications. These facts are very limiting in case someone would like to develop their own backup tool becaouse without access to the dirty bitmap it would be possible to implement only full backups. And without persistent dirty bitmaps, it wouldn't be possible to keep track of changed data after QEMU is restarted. And this is exactly what I do as a part of my bachelor thesis. I've developed a tool that is able to create incremental backups of drives in raw format that are LVM volumes (ability to create snapshot is required). Please keep in mind that this is my first submission to such a large project and also the first time when I send patch over the email. So I hope I did it correctly. Patrik Janoušek (2): block/raw: added support of persistent dirty bitmaps qapi: implementation of the block-dirty-bitmap-dump command block/meson.build | 1 + block/monitor/bitmap-qmp-cmds.c | 61 ++++++++ block/raw-format-bitmap.c | 163 ++++++++++++++++++++ block/raw-format.c | 256 ++++++++++++++++++++++++++++++-- block/raw-format.h | 50 +++++++ qapi/block-core.json | 64 +++++++- 6 files changed, 583 insertions(+), 12 deletions(-) create mode 100644 block/raw-format-bitmap.c create mode 100644 block/raw-format.h -- 2.31.0
Hi Patrik! 20.03.2021 12:32, Patrik Janoušek wrote: > Currently, QEMU doesn't support persistent dirty bitmaps for raw format That's right, we don't have such feature now. > and also dirty bitmaps are for internal use only, and cannot be accessed > using third-party applications. And that's is not. Bitmaps are accessible through bitmap QMP API block-dirty-bitmap-{add,remove,clear,merge} And to retrieve the context of dirty bitmap you can export it through NBD protocol (see bitmaps argument in nbd specific options of block-export-add command) > These facts are very limiting > in case someone would like to develop their own backup tool becaouse > without access to the dirty bitmap it would be possible to implement > only full backups. We do have external incremental backups, based on Qemu bitmap API. But it depends of course on qcow2 persistent bitmaps feature. > And without persistent dirty bitmaps, it wouldn't > be possible to keep track of changed data after QEMU is restarted. And > this is exactly what I do as a part of my bachelor thesis. I've > developed a tool that is able to create incremental backups of drives > in raw format that are LVM volumes (ability to create snapshot is > required). > > Please keep in mind that this is my first submission to such a large > project and also the first time when I send patch over the email. > So I hope I did it correctly. > > Patrik Janoušek (2): > block/raw: added support of persistent dirty bitmaps > qapi: implementation of the block-dirty-bitmap-dump command > > block/meson.build | 1 + > block/monitor/bitmap-qmp-cmds.c | 61 ++++++++ > block/raw-format-bitmap.c | 163 ++++++++++++++++++++ > block/raw-format.c | 256 ++++++++++++++++++++++++++++++-- > block/raw-format.h | 50 +++++++ > qapi/block-core.json | 64 +++++++- > 6 files changed, 583 insertions(+), 12 deletions(-) > create mode 100644 block/raw-format-bitmap.c > create mode 100644 block/raw-format.h > -- Best regards, Vladimir
Hello, thank you for the quick reply. On 3/22/21 9:29 AM, Vladimir Sementsov-Ogievskiy wrote: > Hi Patrik! > > 20.03.2021 12:32, Patrik Janoušek wrote: >> Currently, QEMU doesn't support persistent dirty bitmaps for raw format > > That's right, we don't have such feature now. > >> and also dirty bitmaps are for internal use only, and cannot be accessed >> using third-party applications. > > And that's is not. Bitmaps are accessible through bitmap QMP API > > block-dirty-bitmap-{add,remove,clear,merge} I know about these commands, but they're useless to me without providing access to the content of bitmap. > > And to retrieve the context of dirty bitmap you can export it through > NBD protocol (see bitmaps argument in nbd specific options of > block-export-add command) Ok, I'll look at it. > >> These facts are very limiting >> in case someone would like to develop their own backup tool becaouse >> without access to the dirty bitmap it would be possible to implement >> only full backups. > > We do have external incremental backups, based on Qemu bitmap API. But > it depends of course on qcow2 persistent bitmaps feature. Yes, I know. And that's the problem. The point of my bachelor thesis is to implement a backup solution for the raw format. > >> And without persistent dirty bitmaps, it wouldn't >> be possible to keep track of changed data after QEMU is restarted. And >> this is exactly what I do as a part of my bachelor thesis. I've >> developed a tool that is able to create incremental backups of drives >> in raw format that are LVM volumes (ability to create snapshot is >> required). >> >> Please keep in mind that this is my first submission to such a large >> project and also the first time when I send patch over the email. >> So I hope I did it correctly. >> >> Patrik Janoušek (2): >> block/raw: added support of persistent dirty bitmaps >> qapi: implementation of the block-dirty-bitmap-dump command >> >> block/meson.build | 1 + >> block/monitor/bitmap-qmp-cmds.c | 61 ++++++++ >> block/raw-format-bitmap.c | 163 ++++++++++++++++++++ >> block/raw-format.c | 256 ++++++++++++++++++++++++++++++-- >> block/raw-format.h | 50 +++++++ >> qapi/block-core.json | 64 +++++++- >> 6 files changed, 583 insertions(+), 12 deletions(-) >> create mode 100644 block/raw-format-bitmap.c >> create mode 100644 block/raw-format.h >> > > Patrik Janoušek IČO: 09291849 +420 724 123 897 <tel:+420 724 123 897> pj@patrikjanousek.cz <mailto:pj@patrikjanousek.cz>
Hi Patrik, Am 22.03.2021 um 09:57 hat Patrik Janoušek geschrieben: > On 3/22/21 9:29 AM, Vladimir Sementsov-Ogievskiy wrote: > > We do have external incremental backups, based on Qemu bitmap API. But > > it depends of course on qcow2 persistent bitmaps feature. > > Yes, I know. And that's the problem. The point of my bachelor thesis is > to implement a backup solution for the raw format. the problem with this is that raw isn't really a format, it's more the absence of a format. You just have the content of the virtual disk in a file and that's it. This means not having any metadata (apart from the metadata stored in the filesystem, of course). As soon as you add metadata in some way (in your case, by referencing additional metadata files in runtime options), it's not raw any more. If you write to the raw image file without updating the metadata, the metadata becomes inconsistent with the content. In other words, both files form a single disk image together and can only be used together or you're breaking them. This in turn means that you have just invented a new image format. It's a bit unconventional in that it's spread across multiple files, and that some of the metadata that brings everything together is even in command line options instead of a file, but you have to combine these components in the same way every time you start the VM, so it really is a new image format. We have gone through such discussions a while ago because obviously "raw with dirty bitmaps" was a request that came soon after we discussed persistent dirty maps. But as we came to the conclusion that any addition to raw would create another new image format specific to QEMU, we decided that we can as well use qcow2 for this, which is already the fully featured QEMU image format. I hope this background helps a bit to explain the reactions you have received so far. Kevin
© 2016 - 2024 Red Hat, Inc.