[PULL 00/17] Block patches

Test docker-quick@centos7 failed
Test docker-mingw@fedora failed
Test checkpatch failed
Test FreeBSD failed
Failed in applying to current master (apply log)
There is a newer version of this series
block/block-gen.h                      |  49 ++++
block/coroutines.h                     |  65 +++++
include/block/block.h                  |  36 ++-
include/qemu/vfio-helpers.h            |   2 +-
block.c                                |  97 +------
block/io.c                             | 339 ++++---------------------
block/nvme.c                           |  73 +++---
tests/test-bdrv-drain.c                |   2 +-
util/vfio-helpers.c                    | 133 +++++++++-
block/meson.build                      |   8 +
docs/devel/block-coroutine-wrapper.rst |  54 ++++
docs/devel/index.rst                   |   1 +
qemu-options.hx                        |  10 +-
scripts/block-coroutine-wrapper.py     | 188 ++++++++++++++
14 files changed, 629 insertions(+), 428 deletions(-)
create mode 100644 block/block-gen.h
create mode 100644 block/coroutines.h
create mode 100644 docs/devel/block-coroutine-wrapper.rst
create mode 100644 scripts/block-coroutine-wrapper.py
[PULL 00/17] Block patches
Posted by Stefan Hajnoczi 3 years, 7 months ago
The following changes since commit b150cb8f67bf491a49a1cb1c7da151eeacbdbcc9:

  Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2020-09-29 13:18:54 +0100)

are available in the Git repository at:

  https://gitlab.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to bc47831ff28d6f5830c9c8d74220131dc54c5253:

  util/vfio-helpers: Rework the IOVA allocator to avoid IOVA reserved regions (2020-09-30 10:23:05 +0100)

----------------------------------------------------------------
Pull request

Note I have switched from GitHub to GitLab.

----------------------------------------------------------------

Eric Auger (2):
  util/vfio-helpers: Collect IOVA reserved regions
  util/vfio-helpers: Rework the IOVA allocator to avoid IOVA reserved
    regions

Philippe Mathieu-Daudé (6):
  util/vfio-helpers: Pass page protections to qemu_vfio_pci_map_bar()
  block/nvme: Map doorbells pages write-only
  block/nvme: Reduce I/O registers scope
  block/nvme: Drop NVMeRegs structure, directly use NvmeBar
  block/nvme: Use register definitions from 'block/nvme.h'
  block/nvme: Replace magic value by SCALE_MS definition

Stefano Garzarella (1):
  docs: add 'io_uring' option to 'aio' param in qemu-options.hx

Vladimir Sementsov-Ogievskiy (8):
  block: return error-code from bdrv_invalidate_cache
  block/io: refactor coroutine wrappers
  block: declare some coroutine functions in block/coroutines.h
  scripts: add block-coroutine-wrapper.py
  block: generate coroutine-wrapper code
  block: drop bdrv_prwv
  block/io: refactor save/load vmstate
  include/block/block.h: drop non-ascii quotation mark

 block/block-gen.h                      |  49 ++++
 block/coroutines.h                     |  65 +++++
 include/block/block.h                  |  36 ++-
 include/qemu/vfio-helpers.h            |   2 +-
 block.c                                |  97 +------
 block/io.c                             | 339 ++++---------------------
 block/nvme.c                           |  73 +++---
 tests/test-bdrv-drain.c                |   2 +-
 util/vfio-helpers.c                    | 133 +++++++++-
 block/meson.build                      |   8 +
 docs/devel/block-coroutine-wrapper.rst |  54 ++++
 docs/devel/index.rst                   |   1 +
 qemu-options.hx                        |  10 +-
 scripts/block-coroutine-wrapper.py     | 188 ++++++++++++++
 14 files changed, 629 insertions(+), 428 deletions(-)
 create mode 100644 block/block-gen.h
 create mode 100644 block/coroutines.h
 create mode 100644 docs/devel/block-coroutine-wrapper.rst
 create mode 100644 scripts/block-coroutine-wrapper.py

-- 
2.26.2

Re: [PULL 00/17] Block patches
Posted by Peter Maydell 3 years, 7 months ago
On Wed, 30 Sep 2020 at 11:13, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit b150cb8f67bf491a49a1cb1c7da151eeacbdbcc9:
>
>   Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2020-09-29 13:18:54 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to bc47831ff28d6f5830c9c8d74220131dc54c5253:
>
>   util/vfio-helpers: Rework the IOVA allocator to avoid IOVA reserved regions (2020-09-30 10:23:05 +0100)
>
> ----------------------------------------------------------------
> Pull request
>
> Note I have switched from GitHub to GitLab.
>
> ----------------------------------------------------------------

This produces this error message on ppc64be Linux:

make: Entering directory `/home/pm215/qemu/build/all'
make[1]: Entering directory `/home/pm215/qemu/slirp'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/pm215/qemu/slirp'
Generating qemu-version.h with a meson_exe.py custom command
Generating qemu-options.def with a meson_exe.py custom command
Generating block-gen.c with a custom command
YAML:1:83: error: unknown enumerated scalar
{"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
"BreakBeforeBraces": "Custom", "SortIncludes": false,
"MaxEmptyLinesToKeep": 2}

           ^~~~~~~~
Error parsing -style: Invalid argument, using LLVM style
YAML:1:83: error: unknown enumerated scalar
{"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
"BreakBeforeBraces": "Custom", "SortIncludes": false,
"MaxEmptyLinesToKeep": 2}

           ^~~~~~~~
Error parsing -style: Invalid argument, using LLVM style
Compiling C object libqemuutil.a.p/util_qemu-error.c.o
Compiling C object libqemuutil.a.p/util_qemu-sockets.c.o
Compiling C object libqemuutil.a.p/util_aio-posix.c.o
Compiling C object libqemuutil.a.p/util_osdep.c.o

The error does not cause the build to fail, which seems
like it's also a bug...

(My guess is this is due to some script implicitly wanting
a newer version of something or other than the PPC box
happens to have installed, rather than being an endianness
issue.)

thanks
-- PMM

Re: [PULL 00/17] Block patches
Posted by Vladimir Sementsov-Ogievskiy 3 years, 7 months ago
01.10.2020 14:23, Peter Maydell wrote:
> On Wed, 30 Sep 2020 at 11:13, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>
>> The following changes since commit b150cb8f67bf491a49a1cb1c7da151eeacbdbcc9:
>>
>>    Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2020-09-29 13:18:54 +0100)
>>
>> are available in the Git repository at:
>>
>>    https://gitlab.com/stefanha/qemu.git tags/block-pull-request
>>
>> for you to fetch changes up to bc47831ff28d6f5830c9c8d74220131dc54c5253:
>>
>>    util/vfio-helpers: Rework the IOVA allocator to avoid IOVA reserved regions (2020-09-30 10:23:05 +0100)
>>
>> ----------------------------------------------------------------
>> Pull request
>>
>> Note I have switched from GitHub to GitLab.
>>
>> ----------------------------------------------------------------
> 
> This produces this error message on ppc64be Linux:
> 
> make: Entering directory `/home/pm215/qemu/build/all'
> make[1]: Entering directory `/home/pm215/qemu/slirp'
> make[1]: Nothing to be done for `all'.
> make[1]: Leaving directory `/home/pm215/qemu/slirp'
> Generating qemu-version.h with a meson_exe.py custom command
> Generating qemu-options.def with a meson_exe.py custom command
> Generating block-gen.c with a custom command
> YAML:1:83: error: unknown enumerated scalar
> {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> "BreakBeforeBraces": "Custom", "SortIncludes": false,
> "MaxEmptyLinesToKeep": 2}
> 
>             ^~~~~~~~
> Error parsing -style: Invalid argument, using LLVM style
> YAML:1:83: error: unknown enumerated scalar
> {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> "BreakBeforeBraces": "Custom", "SortIncludes": false,
> "MaxEmptyLinesToKeep": 2}
> 
>             ^~~~~~~~
> Error parsing -style: Invalid argument, using LLVM style
> Compiling C object libqemuutil.a.p/util_qemu-error.c.o
> Compiling C object libqemuutil.a.p/util_qemu-sockets.c.o
> Compiling C object libqemuutil.a.p/util_aio-posix.c.o
> Compiling C object libqemuutil.a.p/util_osdep.c.o
> 
> The error does not cause the build to fail, which seems
> like it's also a bug...

YAML:1:83 is most probably "Custom" value for "BeakBeforeBraces" key,
which is needed to enable custom brace wrapping, set with separate
BraceWrapping key.

Not simple to find the version of clang which is first to support "Custom'
here, but at least clang 3.6.1 doesn't support it.

Hm, I can "reproduce" like this:

[root@kvm work]# clang-format -style='{"BreakBeforeBraces": "Mega"}'
int c() {}
YAML:1:23: error: unknown enumerated scalar
{"BreakBeforeBraces": "Mega"}
                       ^~~~~~
Error parsing -style: Invalid argument
[root@kvm work]# echo $?
1

.

Build doesn't fail because clang-formatting of generated code is optional,
and error is ignored. But stderr output of clang-format is still printed
and this is wrong.

To fix we need:

diff --git a/scripts/block-coroutine-wrapper.py b/scripts/block-coroutine-wrapper.py
index 505e070660..1f66eb903b 100644
--- a/scripts/block-coroutine-wrapper.py
+++ b/scripts/block-coroutine-wrapper.py
@@ -43,7 +43,7 @@ def prettify(code: str) -> str:
          })
          p = subprocess.run(['clang-format', f'-style={style}'], check=True,
                             encoding='utf-8', input=code,
-                           stdout=subprocess.PIPE)
+                           stdout=subprocess.PIPE, stderr=subprocess.PIPE)
          return p.stdout
      except FileNotFoundError:
          return code



I can send a separate patch if it is convenient.


> 
> (My guess is this is due to some script implicitly wanting
> a newer version of something or other than the PPC box
> happens to have installed, rather than being an endianness
> issue.)
> 
> thanks
> -- PMM
> 


-- 
Best regards,
Vladimir

Re: [PULL 00/17] Block patches
Posted by Stefan Hajnoczi 3 years, 7 months ago
On Thu, Oct 01, 2020 at 12:23:00PM +0100, Peter Maydell wrote:
> On Wed, 30 Sep 2020 at 11:13, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > The following changes since commit b150cb8f67bf491a49a1cb1c7da151eeacbdbcc9:
> >
> >   Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2020-09-29 13:18:54 +0100)
> >
> > are available in the Git repository at:
> >
> >   https://gitlab.com/stefanha/qemu.git tags/block-pull-request
> >
> > for you to fetch changes up to bc47831ff28d6f5830c9c8d74220131dc54c5253:
> >
> >   util/vfio-helpers: Rework the IOVA allocator to avoid IOVA reserved regions (2020-09-30 10:23:05 +0100)
> >
> > ----------------------------------------------------------------
> > Pull request
> >
> > Note I have switched from GitHub to GitLab.
> >
> > ----------------------------------------------------------------
> 
> This produces this error message on ppc64be Linux:
> 
> make: Entering directory `/home/pm215/qemu/build/all'
> make[1]: Entering directory `/home/pm215/qemu/slirp'
> make[1]: Nothing to be done for `all'.
> make[1]: Leaving directory `/home/pm215/qemu/slirp'
> Generating qemu-version.h with a meson_exe.py custom command
> Generating qemu-options.def with a meson_exe.py custom command
> Generating block-gen.c with a custom command
> YAML:1:83: error: unknown enumerated scalar
> {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> "BreakBeforeBraces": "Custom", "SortIncludes": false,
> "MaxEmptyLinesToKeep": 2}
> 
>            ^~~~~~~~
> Error parsing -style: Invalid argument, using LLVM style
> YAML:1:83: error: unknown enumerated scalar
> {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> "BreakBeforeBraces": "Custom", "SortIncludes": false,
> "MaxEmptyLinesToKeep": 2}
> 
>            ^~~~~~~~
> Error parsing -style: Invalid argument, using LLVM style
> Compiling C object libqemuutil.a.p/util_qemu-error.c.o
> Compiling C object libqemuutil.a.p/util_qemu-sockets.c.o
> Compiling C object libqemuutil.a.p/util_aio-posix.c.o
> Compiling C object libqemuutil.a.p/util_osdep.c.o
> 
> The error does not cause the build to fail, which seems
> like it's also a bug...
> 
> (My guess is this is due to some script implicitly wanting
> a newer version of something or other than the PPC box
> happens to have installed, rather than being an endianness
> issue.)

Please rerun with make -j1 V=1 so the full command is printed. I'm not
sure what is emitting these errors.

Thanks,
Stefan
Re: [PULL 00/17] Block patches
Posted by Peter Maydell 3 years, 7 months ago
On Thu, 1 Oct 2020 at 16:03, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Thu, Oct 01, 2020 at 12:23:00PM +0100, Peter Maydell wrote:
> > This produces this error message on ppc64be Linux:
> >
> > make: Entering directory `/home/pm215/qemu/build/all'
> > make[1]: Entering directory `/home/pm215/qemu/slirp'
> > make[1]: Nothing to be done for `all'.
> > make[1]: Leaving directory `/home/pm215/qemu/slirp'
> > Generating qemu-version.h with a meson_exe.py custom command
> > Generating qemu-options.def with a meson_exe.py custom command
> > Generating block-gen.c with a custom command
> > YAML:1:83: error: unknown enumerated scalar
> > {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> > "BreakBeforeBraces": "Custom", "SortIncludes": false,
> > "MaxEmptyLinesToKeep": 2}
> >
> >            ^~~~~~~~
> > Error parsing -style: Invalid argument, using LLVM style
> > YAML:1:83: error: unknown enumerated scalar
> > {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> > "BreakBeforeBraces": "Custom", "SortIncludes": false,
> > "MaxEmptyLinesToKeep": 2}
> >
> >            ^~~~~~~~
> > Error parsing -style: Invalid argument, using LLVM style
> > Compiling C object libqemuutil.a.p/util_qemu-error.c.o
> > Compiling C object libqemuutil.a.p/util_qemu-sockets.c.o
> > Compiling C object libqemuutil.a.p/util_aio-posix.c.o
> > Compiling C object libqemuutil.a.p/util_osdep.c.o
> >
> > The error does not cause the build to fail, which seems
> > like it's also a bug...
> >
> > (My guess is this is due to some script implicitly wanting
> > a newer version of something or other than the PPC box
> > happens to have installed, rather than being an endianness
> > issue.)
>
> Please rerun with make -j1 V=1 so the full command is printed. I'm not
> sure what is emitting these errors.

Build tree already overwritten to handle a different pullreq,
I'm afraid. I can come back and retry later...

thanks
-- PMM

Re: [PULL 00/17] Block patches
Posted by Stefan Hajnoczi 3 years, 7 months ago
On Thu, Oct 01, 2020 at 04:12:12PM +0100, Peter Maydell wrote:
> On Thu, 1 Oct 2020 at 16:03, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Thu, Oct 01, 2020 at 12:23:00PM +0100, Peter Maydell wrote:
> > > This produces this error message on ppc64be Linux:
> > >
> > > make: Entering directory `/home/pm215/qemu/build/all'
> > > make[1]: Entering directory `/home/pm215/qemu/slirp'
> > > make[1]: Nothing to be done for `all'.
> > > make[1]: Leaving directory `/home/pm215/qemu/slirp'
> > > Generating qemu-version.h with a meson_exe.py custom command
> > > Generating qemu-options.def with a meson_exe.py custom command
> > > Generating block-gen.c with a custom command
> > > YAML:1:83: error: unknown enumerated scalar
> > > {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> > > "BreakBeforeBraces": "Custom", "SortIncludes": false,
> > > "MaxEmptyLinesToKeep": 2}
> > >
> > >            ^~~~~~~~
> > > Error parsing -style: Invalid argument, using LLVM style
> > > YAML:1:83: error: unknown enumerated scalar
> > > {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
> > > "BreakBeforeBraces": "Custom", "SortIncludes": false,
> > > "MaxEmptyLinesToKeep": 2}
> > >
> > >            ^~~~~~~~
> > > Error parsing -style: Invalid argument, using LLVM style
> > > Compiling C object libqemuutil.a.p/util_qemu-error.c.o
> > > Compiling C object libqemuutil.a.p/util_qemu-sockets.c.o
> > > Compiling C object libqemuutil.a.p/util_aio-posix.c.o
> > > Compiling C object libqemuutil.a.p/util_osdep.c.o
> > >
> > > The error does not cause the build to fail, which seems
> > > like it's also a bug...
> > >
> > > (My guess is this is due to some script implicitly wanting
> > > a newer version of something or other than the PPC box
> > > happens to have installed, rather than being an endianness
> > > issue.)
> >
> > Please rerun with make -j1 V=1 so the full command is printed. I'm not
> > sure what is emitting these errors.
> 
> Build tree already overwritten to handle a different pullreq,
> I'm afraid. I can come back and retry later...

No problem. Thanks for pointing out the issue, Vladimir and Peter.

I will send a v2.

Stefan
Re: [PULL 00/17] Block patches
Posted by Vladimir Sementsov-Ogievskiy 3 years, 7 months ago
01.10.2020 18:02, Stefan Hajnoczi wrote:
> On Thu, Oct 01, 2020 at 12:23:00PM +0100, Peter Maydell wrote:
>> On Wed, 30 Sep 2020 at 11:13, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>>
>>> The following changes since commit b150cb8f67bf491a49a1cb1c7da151eeacbdbcc9:
>>>
>>>    Merge remote-tracking branch 'remotes/mst/tags/for_upstream' into staging (2020-09-29 13:18:54 +0100)
>>>
>>> are available in the Git repository at:
>>>
>>>    https://gitlab.com/stefanha/qemu.git tags/block-pull-request
>>>
>>> for you to fetch changes up to bc47831ff28d6f5830c9c8d74220131dc54c5253:
>>>
>>>    util/vfio-helpers: Rework the IOVA allocator to avoid IOVA reserved regions (2020-09-30 10:23:05 +0100)
>>>
>>> ----------------------------------------------------------------
>>> Pull request
>>>
>>> Note I have switched from GitHub to GitLab.
>>>
>>> ----------------------------------------------------------------
>>
>> This produces this error message on ppc64be Linux:
>>
>> make: Entering directory `/home/pm215/qemu/build/all'
>> make[1]: Entering directory `/home/pm215/qemu/slirp'
>> make[1]: Nothing to be done for `all'.
>> make[1]: Leaving directory `/home/pm215/qemu/slirp'
>> Generating qemu-version.h with a meson_exe.py custom command
>> Generating qemu-options.def with a meson_exe.py custom command
>> Generating block-gen.c with a custom command
>> YAML:1:83: error: unknown enumerated scalar
>> {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
>> "BreakBeforeBraces": "Custom", "SortIncludes": false,
>> "MaxEmptyLinesToKeep": 2}
>>
>>             ^~~~~~~~
>> Error parsing -style: Invalid argument, using LLVM style
>> YAML:1:83: error: unknown enumerated scalar
>> {"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
>> "BreakBeforeBraces": "Custom", "SortIncludes": false,
>> "MaxEmptyLinesToKeep": 2}
>>
>>             ^~~~~~~~
>> Error parsing -style: Invalid argument, using LLVM style
>> Compiling C object libqemuutil.a.p/util_qemu-error.c.o
>> Compiling C object libqemuutil.a.p/util_qemu-sockets.c.o
>> Compiling C object libqemuutil.a.p/util_aio-posix.c.o
>> Compiling C object libqemuutil.a.p/util_osdep.c.o
>>
>> The error does not cause the build to fail, which seems
>> like it's also a bug...
>>
>> (My guess is this is due to some script implicitly wanting
>> a newer version of something or other than the PPC box
>> happens to have installed, rather than being an endianness
>> issue.)
> 
> Please rerun with make -j1 V=1 so the full command is printed. I'm not
> sure what is emitting these errors.
> 

For sure it's block-coroutine-wrapper.py. I've send a letter with some investigation and solution, did you get it?


-- 
Best regards,
Vladimir

Re: [PULL 00/17] Block patches
Posted by Peter Maydell 3 years, 7 months ago
On Thu, 1 Oct 2020 at 16:03, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> Please rerun with make -j1 V=1 so the full command is printed. I'm not
> sure what is emitting these errors.

Vladimir's guess was correct:

/usr/bin/python3
/home/pm215/qemu/block/../scripts/block-coroutine-wrapper.py
block/block-gen.c ../../block/../include/block/block.h
../../block/coroutines.h && if test -e block/block-gen.c; then printf
'%s\n' block/block-gen.c > block/block-gen.c.stamp; fi
YAML:1:83: error: unknown enumerated scalar
{"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
"BreakBeforeBraces": "Custom", "SortIncludes": false,
"MaxEmptyLinesToKeep": 2}

           ^~~~~~~~
Error parsing -style: Invalid argument, using LLVM style
YAML:1:83: error: unknown enumerated scalar
{"IndentWidth": 4, "BraceWrapping": {"AfterFunction": true},
"BreakBeforeBraces": "Custom", "SortIncludes": false,
"MaxEmptyLinesToKeep": 2}

           ^~~~~~~~
Error parsing -style: Invalid argument, using LLVM style

thanks
-- PMM