This is a followup to

  v1: https://www.redhat.com/archives/libvir-list/2020-January/msg00900.html

At the end of this series we have 100% eliminated use of GNULIB
from libvirt.

The first 10 or so patches have been reviewed by Pavel already
but I include them here anyway. Rather than wait for all of
the series to be review, it is probably more productive to
push patches in batches of 10 or so.

Some things to note

 - I have build tested this on Travis platforms and manually
   via FreeBSD 11/12. This covers make, make syntax-check &
   make check

 - I've validated that virsh still works with mingw64 builds
   on Windows 2008r2.

 - I've done basic functional testing on Fedora 31, starting
   and stopping VMs & other other simple APIs

The config.h we generate is much much smaller than before as we
eliminated alot of gnulib macros.

The risk here is that we are no longer setting some HAVE_XXX
in config.h that we rely on. To mitigate this I did a diff
of config.h before & after this series to determinw which
HAVE_XXX we no longer set. I then grepped the source to see
if we actually use any of them. This identified a few mistakes
which I fixed in testing this series.

The builds times for libvirt after applying this series have
some significant gains, improving speed of all stages (autogen,
configure & make).

Overall while this was time consuming work (due to massive number
of builds for testing each step), it is surprising just how easy
it was eliminate need for GNULIB.  GLib helped a little bit in
this respect, but the biggest factor is simply that a large
number of issues GNULIB fixes only matter for ancient / obsolete
OS platforms.

With libvirt only targetting modern Linux, FreeBSD, macOS & MinGW,
the only really hard stuff where GNULIB was a big help is the
Windows sockets portability.

GNULIB was a pretty valuable approach when there were countless
flavours of UNIX to worry about with poor levels of POSIX API
compatibility. With a typical modern set of platforms, I think
it is better to just use a library like GLib and deal with any
other portability problems explicitly.

Almost certainly someone will appear after next release and
complain that libvirt no longer builds on some platform that
we don't officially support. My expectation is that when this
happens it will be reasonably easy to fix whatever problem
they report. Also at that time we can also consider whether
the platform needs to be added to CI.

Daniel P. Berrangé (56):
  tests: stop setting $SHELL env variable
  util: add a virArchFromHost() impl for Windows
  util: add API for reading password from the console
  src: remove usage of strchrnul function
  build: generate configmake.h in root directory
  util: use getgrouplist() directly instead of mgetgroups
  tools: replace wcwidth() with g_unichar_* APIs
  src: remove unused sys/utsname.h includes
  util: explicitly include windows.h
  storage: remove use of stat-time.h headers
  src: implement APIs for passing FDs over UNIX sockets
  rpc: conditionalize signal handling
  src: only import sys/uio.h when journald is built
  src: replace mkdir() with g_mkdir()
  m4: disable polkit build on Windows
  util: conditionalize more of virCommand on WIN32
  src: remove all traces of Cygwin support
  util: conditionalize virProcess APIs on Windows
  src: conditionalize use of net/if.h
  configure: add check for sys/ioctl.h
  src: conditionalize use of S_ISSOCK macro
  configure: request system specific extensions
  src: stop using O_DIRECTORY in resctrl
  src: ensure O_CLOEXEC is defined on Windows
  src: conditionalize use of F_DUPFD_CLOEXEC
  src: conditionalize use of O_DIRECT
  src: conditionalize use of O_BINARY
  src: conditionalize use of chown & stat constants
  src: convert all code to use virsocket.h
  tests: conditionalize use of SIGPIPE
  src: conditionalize EAI_ADDRFAMILY
  bootstrap: remove 18 more gnulib modules
  src: introduce helper API for creating GSource for socket
  rpc: convert RPC client to use GMainLoop instead of poll
  tests: convert eventtest to use public event APIs
  tests: remove event loop from command test
  tests: refactor event test to not run lock step
  tools: convert to use g_poll instead of poll
  util: import an event loop impl based on GMainContext
  util: switch to use the GLib event loop impl
  util: delete the poll() based event loop impl
  src: conditionalize / remove use of poll.h
  util: conditionalize FD stream to exclude WIN32
  src: remove sys/wait.h from many files
  configure: request 64-bit APIs on 32-bit platforms
  examples: remove obsolete workaround for mingw
  src: introduce a wrapper for the pipe2() system call
  src: convert code to use virPipe APIs
  tools: conditionalize use of O_SYNC flag
  m4: add check for pthread library
  src: assume sys/sysmacros.h always exists on Linux
  src: add define of ENOMSG for MinGW
  src: optionally include xlocale.h header
  src: ensure use of g_printf / g_fprintf functions
  src: remove virFilePrintf in favour of g_fprintf
  gnulib: delete all gnulib integration

 .color_coded.in                       |    2 -
 .gitignore                            |    9 +-
 .gitmodules                           |    3 -
 .gnulib                               |    1 -
 .ycm_extra_conf.py.in                 |    2 -
 Makefile.am                           |   46 +-
 README-hacking                        |    9 +-
 autogen.sh                            |  219 +----
 bootstrap                             | 1073 -------------------------
 bootstrap.conf                        |  136 ----
 build-aux/syntax-check.mk             |  132 +--
 ci/build.sh                           |    4 +-
 config-post.h                         |    5 +-
 configure.ac                          |   49 +-
 docs/compiling.html.in                |   25 -
 docs/hacking.html.in                  |    5 +-
 examples/c/domain/domtop.c            |   15 -
 examples/c/domain/suspend.c           |   14 -
 gnulib/lib/Makefile.am                |   30 -
 libvirt.spec.in                       |    2 -
 m4/virt-compile-pie.m4                |    2 +-
 m4/virt-compile-warnings.m4           |   18 +-
 m4/virt-polkit.m4                     |    4 +
 m4/virt-pthread.m4                    |   26 +-
 m4/virt-win-common.m4                 |    8 +-
 m4/virt-win-cygwin.m4                 |   32 -
 m4/virt-win-symbols.m4                |    4 +-
 m4/virt-win-windres.m4                |    4 +-
 m4/virt-xdr.m4                        |    9 +-
 po/POTFILES.in                        |    3 +-
 src/Makefile.am                       |   17 +-
 src/admin/Makefile.inc.am             |    2 -
 src/bhyve/Makefile.inc.am             |    1 -
 src/conf/domain_audit.c               |    4 +-
 src/conf/network_conf.c               |    2 -
 src/esx/esx_util.c                    |    3 +-
 src/esx/esx_util.h                    |    1 -
 src/interface/Makefile.inc.am         |    1 -
 src/internal.h                        |   30 +
 src/libvirt-domain.c                  |    2 +
 src/libvirt.c                         |   10 +-
 src/libvirt_private.syms              |   24 +-
 src/libxl/Makefile.inc.am             |    1 -
 src/libxl/libxl_conf.c                |    2 -
 src/libxl/libxl_migration.c           |    6 +-
 src/locking/Makefile.inc.am           |    9 +-
 src/locking/lock_daemon.c             |    2 +-
 src/logging/Makefile.inc.am           |    2 -
 src/logging/log_daemon.c              |    2 +-
 src/logging/log_handler.c             |    6 +-
 src/lxc/Makefile.inc.am               |    2 -
 src/lxc/lxc_container.c               |    1 -
 src/lxc/lxc_controller.c              |    8 +-
 src/lxc/lxc_driver.c                  |    8 +-
 src/lxc/lxc_process.c                 |    5 +-
 src/network/Makefile.inc.am           |    3 +-
 src/network/bridge_driver.c           |    1 -
 src/node_device/Makefile.inc.am       |    2 -
 src/nwfilter/Makefile.inc.am          |    1 -
 src/nwfilter/nwfilter_dhcpsnoop.c     |    3 -
 src/nwfilter/nwfilter_learnipaddr.c   |    4 +-
 src/openvz/openvz_conf.c              |    5 +-
 src/openvz/openvz_driver.c            |    1 -
 src/qemu/Makefile.inc.am              |    1 -
 src/qemu/qemu_agent.c                 |    4 +-
 src/qemu/qemu_capabilities.c          |    1 -
 src/qemu/qemu_conf.c                  |    4 +-
 src/qemu/qemu_domain.c                |    4 +-
 src/qemu/qemu_driver.c                |    4 +-
 src/qemu/qemu_interface.c             |    4 +-
 src/qemu/qemu_migration.c             |   28 +-
 src/qemu/qemu_monitor.c               |    3 +-
 src/qemu/qemu_monitor_json.c          |    4 +-
 src/qemu/qemu_tpm.c                   |    5 +-
 src/remote/Makefile.inc.am            |    1 -
 src/remote/qemu_protocol.x            |    1 -
 src/remote/remote_daemon.c            |    3 +-
 src/remote/remote_protocol.x          |    2 +-
 src/rpc/Makefile.inc.am               |    3 -
 src/rpc/genprotocol.pl                |    2 +-
 src/rpc/virnetclient.c                |  248 +++---
 src/rpc/virnetdaemon.c                |   44 +-
 src/rpc/virnetdaemon.h                |    4 +
 src/rpc/virnetprotocol.x              |    2 +-
 src/rpc/virnetsocket.c                |   44 +-
 src/secret/Makefile.inc.am            |    1 -
 src/security/Makefile.inc.am          |    1 -
 src/security/security_dac.c           |    4 +
 src/security/security_manager.c       |    2 +
 src/security/security_selinux.c       |   16 +-
 src/storage/Makefile.inc.am           |   16 -
 src/storage/storage_backend_iscsi.c   |    1 -
 src/storage/storage_backend_logical.c |    1 -
 src/storage/storage_util.c            |   25 +-
 src/util/Makefile.inc.am              |    6 +-
 src/util/iohelper.c                   |    4 +
 src/util/viralloc.h                   |    3 +-
 src/util/virarch.c                    |   52 +-
 src/util/virarptable.c                |    1 -
 src/util/virbitmap.c                  |    4 +-
 src/util/vircgroup.c                  |   18 +-
 src/util/vircommand.c                 |  183 +++--
 src/util/virdevmapper.c               |    4 +-
 src/util/virdnsmasq.c                 |    9 +-
 src/util/virevent.c                   |   21 +-
 src/util/vireventglib.c               |  455 +++++++++++
 src/util/vireventglib.h               |   28 +
 src/util/vireventglibwatch.c          |  248 ++++++
 src/util/vireventglibwatch.h          |   48 ++
 src/util/vireventpoll.c               |  772 ------------------
 src/util/vireventpoll.h               |  126 ---
 src/util/virfdstream.c                |  131 ++-
 src/util/virfile.c                    |  102 +--
 src/util/virfile.h                    |    3 -
 src/util/virgettext.c                 |    3 +
 src/util/virhook.c                    |    1 -
 src/util/virhostcpu.c                 |    1 -
 src/util/virhostmem.c                 |    1 -
 src/util/viriptables.c                |    1 -
 src/util/virlog.c                     |    8 +-
 src/util/virnetdev.c                  |    1 -
 src/util/virnetdev.h                  |    4 +-
 src/util/virnetdevbridge.c            |    8 +-
 src/util/virnetdevip.c                |    4 +-
 src/util/virnetdevmacvlan.c           |    3 +-
 src/util/virnetdevtap.c               |    4 +-
 src/util/virnetdevveth.c              |    2 -
 src/util/virnetdevvportprofile.c      |    3 +-
 src/util/virnetlink.c                 |    3 +-
 src/util/virpolkit.c                  |    4 +-
 src/util/virprocess.c                 |  108 ++-
 src/util/virresctrl.c                 |    2 +-
 src/util/virsocket.c                  |  139 +++-
 src/util/virsocket.h                  |   21 +-
 src/util/virsocketaddr.c              |    2 -
 src/util/virsocketaddr.h              |   20 +-
 src/util/virstring.c                  |    3 +
 src/util/virsysinfo.c                 |   15 +-
 src/util/virsystemd.c                 |    9 +-
 src/util/virutil.c                    |  146 +++-
 src/util/virutil.h                    |   36 +
 src/util/virxdrdefs.h                 |   12 +-
 src/vbox/Makefile.inc.am              |    1 -
 src/vbox/vbox_MSCOMGlue.c             |    6 +-
 src/vz/Makefile.inc.am                |    1 -
 src/vz/vz_driver.c                    |    1 -
 tests/Makefile.am                     |   26 +-
 tests/commanddata/test27.log          |    6 +-
 tests/commanddata/test3.log           |    2 +-
 tests/commandhelper.c                 |    5 +-
 tests/commandtest.c                   |  115 +--
 tests/eventtest.c                     |  219 +++--
 tests/fdstreamtest.c                  |   42 +-
 tests/libxlmock.c                     |    2 +-
 tests/nsstest.c                       |    3 +-
 tests/qemusecuritytest.c              |    2 +-
 tests/shunloadtest.c                  |    2 +-
 tests/ssh.c                           |    3 +-
 tests/testutils.c                     |    5 +-
 tests/testutils.h                     |    6 -
 tests/testutilsqemu.c                 |    4 +-
 tests/virauthconfigtest.c             |    2 +
 tests/vircgroupmock.c                 |    8 +-
 tests/virkeyfiletest.c                |    2 +
 tests/virlockspacetest.c              |    4 +-
 tests/virnetmessagetest.c             |    2 +
 tests/virnetsockettest.c              |    3 +-
 tests/virnettlscontexttest.c          |    3 +-
 tests/virnettlshelpers.c              |    1 -
 tests/virnettlssessiontest.c          |    3 +-
 tests/virportallocatormock.c          |    5 +-
 tests/virstringtest.c                 |   37 +-
 tests/virtestmock.c                   |    9 +-
 tests/virtimetest.c                   |    2 +
 tests/viruritest.c                    |    2 +
 tests/vshtabletest.c                  |    4 +
 tools/Makefile.am                     |    9 +-
 tools/nss/libvirt_nss.c               |    7 +-
 tools/virsh-domain.c                  |   72 +-
 tools/virsh.c                         |    6 -
 tools/virt-admin.c                    |    5 -
 tools/virt-login-shell.c              |    6 +-
 tools/vsh-table.c                     |    2 +-
 tools/vsh.c                           |   31 +-
 184 files changed, 2295 insertions(+), 3708 deletions(-)
 delete mode 160000 .gnulib
 delete mode 100755 bootstrap
 delete mode 100644 bootstrap.conf
 delete mode 100644 gnulib/lib/Makefile.am
 delete mode 100644 m4/virt-win-cygwin.m4
 create mode 100644 src/util/vireventglib.c
 create mode 100644 src/util/vireventglib.h
 create mode 100644 src/util/vireventglibwatch.c
 create mode 100644 src/util/vireventglibwatch.h
 delete mode 100644 src/util/vireventpoll.c
 delete mode 100644 src/util/vireventpoll.h

-- 
2.24.1

On Tue, Jan 28, 2020 at 01:10:41PM +0000, Daniel P. Berrangé wrote:
> This is a followup to
> 
>   v1: https://www.redhat.com/archives/libvir-list/2020-January/msg00900.html
> 
> At the end of this series we have 100% eliminated use of GNULIB
> from libvirt.
> 
> The first 10 or so patches have been reviewed by Pavel already
> but I include them here anyway. Rather than wait for all of
> the series to be review, it is probably more productive to
> push patches in batches of 10 or so.
> 
> Some things to note
> 
>  - I have build tested this on Travis platforms and manually
>    via FreeBSD 11/12. This covers make, make syntax-check &
>    make check
> 
>  - I've validated that virsh still works with mingw64 builds
>    on Windows 2008r2.
> 
>  - I've done basic functional testing on Fedora 31, starting
>    and stopping VMs & other other simple APIs
> 
> The config.h we generate is much much smaller than before as we
> eliminated alot of gnulib macros.
> 
> The risk here is that we are no longer setting some HAVE_XXX
> in config.h that we rely on. To mitigate this I did a diff
> of config.h before & after this series to determinw which
> HAVE_XXX we no longer set. I then grepped the source to see
> if we actually use any of them. This identified a few mistakes
> which I fixed in testing this series.
> 
> The builds times for libvirt after applying this series have
> some significant gains, improving speed of all stages (autogen,
> configure & make).
> 
> Overall while this was time consuming work (due to massive number
> of builds for testing each step), it is surprising just how easy
> it was eliminate need for GNULIB.  GLib helped a little bit in
> this respect, but the biggest factor is simply that a large
> number of issues GNULIB fixes only matter for ancient / obsolete
> OS platforms.
> 
> With libvirt only targetting modern Linux, FreeBSD, macOS & MinGW,
> the only really hard stuff where GNULIB was a big help is the
> Windows sockets portability.
> 
> GNULIB was a pretty valuable approach when there were countless
> flavours of UNIX to worry about with poor levels of POSIX API
> compatibility. With a typical modern set of platforms, I think
> it is better to just use a library like GLib and deal with any
> other portability problems explicitly.
> 
> Almost certainly someone will appear after next release and
> complain that libvirt no longer builds on some platform that
> we don't officially support. My expectation is that when this
> happens it will be reasonably easy to fix whatever problem
> they report. Also at that time we can also consider whether
> the platform needs to be added to CI.

Awesome job, I'll continue with review so that we can get this pushed
ASAP and continue with conversion to Meson which I'm working on in the
mean time to split it into patches.

Pavel