From nobody Mon Apr 29 05:43:41 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1517821858140935.4092911827397; Mon, 5 Feb 2018 01:10:58 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AAD5968563; Mon, 5 Feb 2018 09:10:56 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 801F2605FB; Mon, 5 Feb 2018 09:10:56 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 3A98318033DA; Mon, 5 Feb 2018 09:10:56 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w1598U0e000821 for ; Mon, 5 Feb 2018 04:08:30 -0500 Received: by smtp.corp.redhat.com (Postfix) id 71A1D5D6A2; Mon, 5 Feb 2018 09:08:30 +0000 (UTC) Received: from mx1.redhat.com (ext-mx08.extmail.prod.ext.phx2.redhat.com [10.5.110.32]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 66CEB5EDEA for ; Mon, 5 Feb 2018 09:08:28 +0000 (UTC) Received: from prv3-mh.provo.novell.com (prv3-mh.provo.novell.com [137.65.250.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5689FC0587D1 for ; Mon, 5 Feb 2018 09:08:25 +0000 (UTC) Received: from localhost.localdomain (prv-ext-foundry1int.gns.novell.com [137.65.251.240]) by prv3-mh.provo.novell.com with ESMTP (NOT encrypted); Mon, 05 Feb 2018 02:08:14 -0700 From: river To: libvir-list@redhat.com Date: Mon, 5 Feb 2018 17:07:54 +0800 Message-Id: <1517821676-14399-2-git-send-email-lfu@suse.com> In-Reply-To: <1517821676-14399-1-git-send-email-lfu@suse.com> References: <1517821676-14399-1-git-send-email-lfu@suse.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 207 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) for IP:'137.65.250.26' DOMAIN:'prv3-mh.provo.novell.com' HELO:'prv3-mh.provo.novell.com' FROM:'lfu@suse.com' RCPT:'' X-RedHat-Spam-Score: -2.301 (RCVD_IN_DNSWL_MED, SPF_PASS) 137.65.250.26 prv3-mh.provo.novell.com 137.65.250.26 prv3-mh.provo.novell.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.32 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 1/3] Add a plugin dlm for lock manager X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Mon, 05 Feb 2018 09:10:57 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" dlm is implemented by linux kernel, provides userspace API by 'libdlm' to lock/unlock resource, cooperated with cluster communication software, such as corosync, could satisfy the demands of libvirt lock manager. Signed-off-by: river --- configure.ac | 6 + m4/virt-cpg.m4 | 37 ++ m4/virt-dlm.m4 | 36 ++ src/Makefile.am | 15 + src/locking/lock_driver_dlm.c | 1056 +++++++++++++++++++++++++++++++++++++= ++++ src/util/virlist.h | 110 +++++ 6 files changed, 1260 insertions(+) create mode 100644 m4/virt-cpg.m4 create mode 100644 m4/virt-dlm.m4 create mode 100644 src/locking/lock_driver_dlm.c create mode 100644 src/util/virlist.h diff --git a/configure.ac b/configure.ac index 4cccf7f4d..4ad90470d 100644 --- a/configure.ac +++ b/configure.ac @@ -265,6 +265,8 @@ LIBVIRT_ARG_PM_UTILS LIBVIRT_ARG_POLKIT LIBVIRT_ARG_READLINE LIBVIRT_ARG_SANLOCK +LIBVIRT_ARG_CPG +LIBVIRT_ARG_DLM LIBVIRT_ARG_SASL LIBVIRT_ARG_SELINUX LIBVIRT_ARG_SSH2 @@ -307,6 +309,8 @@ LIBVIRT_CHECK_POLKIT LIBVIRT_CHECK_PTHREAD LIBVIRT_CHECK_READLINE LIBVIRT_CHECK_SANLOCK +LIBVIRT_CHECK_CPG +LIBVIRT_CHECK_DLM LIBVIRT_CHECK_SASL LIBVIRT_CHECK_SELINUX LIBVIRT_CHECK_SSH2 @@ -1005,6 +1009,8 @@ LIBVIRT_RESULT_POLKIT LIBVIRT_RESULT_RBD LIBVIRT_RESULT_READLINE LIBVIRT_RESULT_SANLOCK +LIBVIRT_RESULT_CPG +LIBVIRT_RESULT_DLM LIBVIRT_RESULT_SASL LIBVIRT_RESULT_SELINUX LIBVIRT_RESULT_SSH2 diff --git a/m4/virt-cpg.m4 b/m4/virt-cpg.m4 new file mode 100644 index 000000000..27acda665 --- /dev/null +++ b/m4/virt-cpg.m4 @@ -0,0 +1,37 @@ +dnl The libcpg.so library +dnl +dnl Copyright (C) 2018 SUSE LINUX Products, Beijing, China. +dnl +dnl This library is free software; you can redistribute it and/or +dnl modify it under the terms of the GNU Lesser General Public +dnl License as published by the Free Software Foundation; either +dnl version 2.1 of the License, or (at your option) any later version. +dnl +dnl This library is distributed in the hope that it will be useful, +dnl but WITHOUT ANY WARRANTY; without even the implied warranty of +dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +dnl Lesser General Public License for more details. +dnl +dnl You should have received a copy of the GNU Lesser General Public +dnl License along with this library. If not, see +dnl . +dnl + +AC_DEFUN([LIBVIRT_ARG_CPG],[ + LIBVIRT_ARG_WITH_FEATURE([CPG], [cluster engine CPG library], [check]) +]) + +AC_DEFUN([LIBVIRT_CHECK_CPG],[ + dnl in some distribution, Version is `UNKNOW` in libcpg.pc + if test "x$with_cpg" !=3D "xno"; then + PKG_CHECK_MODULES([CPG], [libcpg], [ + with_cpg=3Dyes + ],[ + with_cpg=3Dno + ]) + fi +]) + +AC_DEFUN([LIBVIRT_RESULT_CPG],[ + LIBVIRT_RESULT_LIB([CPG]) +]) diff --git a/m4/virt-dlm.m4 b/m4/virt-dlm.m4 new file mode 100644 index 000000000..dc5f5f152 --- /dev/null +++ b/m4/virt-dlm.m4 @@ -0,0 +1,36 @@ +dnl The libdlm.so library +dnl +dnl Copyright (C) 2018 SUSE LINUX Products, Beijing, China. +dnl +dnl This library is free software; you can redistribute it and/or +dnl modify it under the terms of the GNU Lesser General Public +dnl License as published by the Free Software Foundation; either +dnl version 2.1 of the License, or (at your option) any later version. +dnl +dnl This library is distributed in the hope that it will be useful, +dnl but WITHOUT ANY WARRANTY; without even the implied warranty of +dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +dnl Lesser General Public License for more details. +dnl +dnl You should have received a copy of the GNU Lesser General Public +dnl License along with this library. If not, see +dnl . +dnl + +AC_DEFUN([LIBVIRT_ARG_DLM],[ + LIBVIRT_ARG_WITH_FEATURE([DLM], [Distributed Lock Manager library], [che= ck]) +]) + +AC_DEFUN([LIBVIRT_CHECK_DLM],[ + AC_REQUIRE([LIBVIRT_CHECK_CPG]) + LIBVIRT_CHECK_PKG([DLM], [libdlm], [4.0.0]) + + if test "x$with_dlm" =3D=3D "xyes" && test "x$with_cpg" !=3D "xyes"; then + AC_MSG_ERROR([You must install libcpg to build dlm lock]) + fi +]) + +AC_DEFUN([LIBVIRT_RESULT_DLM],[ + AC_REQUIRE([LIBVIRT_RESULT_CPG]) + LIBVIRT_RESULT_LIB([DLM]) +]) diff --git a/src/Makefile.am b/src/Makefile.am index 79adc9ba5..8742921fa 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -305,6 +305,10 @@ DRIVER_SOURCES =3D \ logging/log_manager.c logging/log_manager.h \ $(NULL) =20 +LOCK_DRIVER_DLM_SOURCES =3D \ + locking/lock_driver_dlm.c \ + $(NULL) + LOCK_DRIVER_SANLOCK_SOURCES =3D \ locking/lock_driver_sanlock.c =20 @@ -2879,6 +2883,17 @@ virtlogd.socket: logging/virtlogd.socket.in $(top_bu= ilddir)/config.status < $< > $@-t && \ mv $@-t $@ =20 +if WITH_DLM +lockdriver_LTLIBRARIES +=3D dlm.la +dlm_la_SOURCES =3D $(LOCK_DRIVER_DLM_SOURCES) +dlm_la_CFLAGS =3D -I$(srcdir)/conf $(AM_CFLAGS) +dlm_la_LDFLAGS =3D -module -avoid-version $(AM_LDFLAGS) +dlm_la_LIBADD =3D ../gnulib/lib/libgnu.la \ + $(CPG_LIBS) \ + $(DLM_LIBS) +else ! WITH_DLM +EXTRA_DIST +=3D $(LOCK_DRIVER_DLM_SOURCES) +endif =20 if WITH_SANLOCK lockdriver_LTLIBRARIES +=3D sanlock.la diff --git a/src/locking/lock_driver_dlm.c b/src/locking/lock_driver_dlm.c new file mode 100644 index 000000000..e197c0bdf --- /dev/null +++ b/src/locking/lock_driver_dlm.c @@ -0,0 +1,1056 @@ +/* + * lock_driver_dlm.c: a lock driver for dlm + * + * Copyright (C) 2018 SUSE LINUX Products, Beijing, China. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * . + * + */ + +#include + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "lock_driver.h" +#include "viralloc.h" +#include "virconf.h" +#include "vircrypto.h" +#include "virerror.h" +#include "virfile.h" +#include "virlist.h" +#include "virlog.h" +#include "virstring.h" +#include "virthread.h" +#include "viruuid.h" + +#define VIR_FROM_THIS VIR_FROM_LOCKING + +#define DLM_LOCKSPACE_MODE 0600 +#define DLM_LOCKSPACE_NAME "libvirt" + +#define LOCK_RECORD_FILE_MODE 0644 +#define LOCK_RECORD_FILE_PATH "/tmp/libvirtd-dlm-file" + +#define PRMODE "PRMODE" +#define EXMODE "EXMODE" + +#define STATUS "STATUS" +#define RESOURCE_NAME "RESOURCE_NAME" +#define LOCK_ID "LOCK_ID" +#define LOCK_MODE "LOCK_MODE" +#define VM_PID "VM_PID" + +#define BUFFERLEN 128 + +/* This will be set after dlm_controld is started. */ +#define DLM_CLUSTER_NAME_PATH "/sys/kernel/config/dlm/cluster/cluster_name" + +VIR_LOG_INIT("locking.lock_driver_dlm"); + +typedef struct _virLockInformation virLockInformation; +typedef virLockInformation *virLockInformationPtr; + +typedef struct _virLockManagerDlmResource virLockManagerDlmResource; +typedef virLockManagerDlmResource *virLockManagerDlmResourcePtr; + +typedef struct _virLockManagerDlmPrivate virLockManagerDlmPrivate; +typedef virLockManagerDlmPrivate *virLockManagerDlmPrivatePtr; + +typedef struct _virLockManagerDlmDriver virLockManagerDlmDriver; +typedef virLockManagerDlmDriver *virLockManagerDlmDriverPtr; + +typedef struct _virListWait virListWait; +typedef virListWait *virListWaitPtr; + +struct _virLockInformation { + virListHead entry; + char *name; + uint32_t mode; + uint32_t lkid; + pid_t vm_pid; +}; + +struct _virLockManagerDlmResource { + char *name; + uint32_t mode; +}; + +struct _virLockManagerDlmPrivate { + unsigned char vm_uuid[VIR_UUID_BUFLEN]; + char *vm_name; + pid_t vm_pid; + int vm_id; + + size_t nresources; + virLockManagerDlmResourcePtr resources; + + bool hasRWDisks; +}; + +struct _virLockManagerDlmDriver { + bool autoDiskLease; + bool requireLeaseForDisks; + + bool purgeLockspace; + char *lockspaceName; + char *lockRecordFilePath; +}; + +struct _virListWait { + virMutex listMutex; + virMutex fileMutex; + virListHead list; +}; + +static virLockManagerDlmDriverPtr driver; +static dlm_lshandle_t lockspace; +static virListWait lockListWait; + +static int virLockManagerDlmLoadConfig(const char *configFile) +{ + virConfPtr conf =3D NULL; + int ret =3D -1; + + if (access(configFile, R_OK) =3D=3D -1) { + if (errno !=3D ENOENT) { + virReportSystemError(errno, + _("Unable to access config file %s"), + configFile); + return -1; + } + return 0; + } + + if (!(conf =3D virConfReadFile(configFile, 0))) + return -1; + + if (virConfGetValueBool(conf, "auto_disk_leases", &driver->autoDiskLea= se) < 0) + goto cleanup; + + driver->requireLeaseForDisks =3D !driver->autoDiskLease; + if (virConfGetValueBool(conf, "require_lease_for_disks", &driver->requ= ireLeaseForDisks) < 0) + goto cleanup; + + if (virConfGetValueBool(conf, "purge_lockspace", &driver->purgeLockspa= ce) < 0) + goto cleanup; + + if (virConfGetValueString(conf, "lockspace_name", &driver->lockspaceNa= me) < 0) + goto cleanup; + + if (virConfGetValueString(conf, "lock_record_file_path", &driver->lock= RecordFilePath) < 0) + goto cleanup; + + ret =3D 0; + + cleanup: + virConfFree(conf); + return ret; +} + +static int virLockManagerDlmToModeUint(const char *token) +{ + if (STREQ(token, PRMODE)) + return LKM_PRMODE; + if (STREQ(token, EXMODE)) + return LKM_EXMODE; + + return 0; +} + +static const char *virLockManagerDlmToModeText(const uint32_t mode) +{ + switch (mode) { + case LKM_PRMODE: + return PRMODE; + case LKM_EXMODE: + return EXMODE; + default: + return NULL; + } +} + +static virLockInformationPtr virLockManagerDlmRecordLock(const char *name, + const uint32_t mo= de, + const uint32_t lk= id, + const pid_t vm_pi= d) +{ + virLockInformationPtr lock =3D NULL; + + if (VIR_ALLOC(lock) < 0) + goto error; + + if (VIR_STRDUP(lock->name, name) < 0) + goto error; + + lock->mode =3D mode; + lock->lkid =3D lkid; + lock->vm_pid =3D vm_pid; + + virMutexLock(&(lockListWait.listMutex)); + virListAddTail(&lock->entry, &(lockListWait.list)); + virMutexUnlock(&(lockListWait.listMutex)); + + VIR_DEBUG("record lock sucessfully, lockName=3D%s lockMode=3D%s lockId= =3D%d", + NULLSTR(name), NULLSTR(virLockManagerDlmToModeText(mode)), l= kid); + + return lock; + + error: + if (lock) + VIR_FREE(lock->name); + VIR_FREE(lock); + return NULL; +} + +static void virLockManagerDlmWriteLock(virLockInformationPtr lock, int fd,= bool status) +{ + char buffer[BUFFERLEN] =3D {0}; + off_t offset =3D 0, rv =3D 0; + + if (!lock) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("lock is NULL")); + return; + } + + /* + * STATUS RESOURCE_NAME LOCK_MODE VM_PID\n + * 6 64 9 10 + * 93 =3D 6 + 1 + 64 + 1 + 9 + 1 + 10 + 1 + */ + offset =3D 93 * lock->lkid; + rv =3D lseek(fd, offset, SEEK_SET); + if (rv < 0) { + virReportSystemError(errno, + _("unable to lseek fd '%d'"), + fd); + return; + } + + snprintf(buffer, sizeof(buffer), "%6d %64s %9s %10jd\n", \ + status, lock->name, + NULLSTR(virLockManagerDlmToModeText(lock->mode)), + (intmax_t)lock->vm_pid); + + if (safewrite(fd, buffer, strlen(buffer)) !=3D strlen(buffer)) { + virReportSystemError(errno, + _("unable to write lock information '%s' to f= ile '%s'"), + buffer, NULLSTR(driver->lockRecordFilePath)); + return; + } + + VIR_DEBUG("write '%s' to fd=3D%d", buffer, fd); + + fdatasync(fd); + + return; +} + +static void virLockManagerDlmAdoptLock(char *raw) { + char *str =3D NULL, *subtoken =3D NULL, *saveptr =3D NULL, *endptr =3D= NULL; + int i =3D 0, status =3D 0; + char *name =3D NULL; + uint32_t mode =3D 0; + pid_t vm_pid =3D 0; + struct dlm_lksb lksb =3D {0}; + + /* every line is the following format: + * STATUS RESOURCE_NAME LOCK_MODE VM_PID + */ + for (i =3D 0, str =3D raw, status =3D 0; ; str =3D NULL, i++) { + subtoken =3D strtok_r(str, " \n", &saveptr); + if (subtoken =3D=3D NULL) + break; + + switch(i) { + case 0: + if (virStrToLong_i(subtoken, &endptr, 10, &status) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("cannot extract lock status '%s'"), subto= ken); + goto cleanup; + } + break; + case 1: + if (VIR_STRDUP(name, subtoken) !=3D 1) + goto cleanup; + break; + case 2: + mode =3D virLockManagerDlmToModeUint(subtoken); + if (!mode) + goto cleanup; + break; + case 3: + if ((virStrToLong_i(subtoken, &endptr, 10, &vm_pid) < 0) || !v= m_pid) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("cannot extract lock vm_pid '%s'"), subto= ken); + goto cleanup; + } + break; + default: + goto cleanup; + break; + } + + if (status !=3D 1) + goto cleanup; + } + + if (i !=3D 4) + goto cleanup; + + /* copy from `lm_adopt_dlm` in daemons/lvmlockd/lvmlockd-dlm.c of lvm2: + * dlm returns 0 for success, -EAGAIN if an orphan is + * found with another mode, and -ENOENT if no orphan. + * + * cast/bast/param are (void *)1 because the kernel + * returns errors if some are null. + */ + + status =3D dlm_ls_lockx(lockspace, mode, &lksb, LKF_PERSISTENT|LKF_ORP= HAN, + name, strlen(name), 0, + (void *)1, (void *)1, (void *)1, + NULL, NULL); + if (status) { + virReportSystemError(errno, + _("unable to adopt lock, rv=3D%d lockName=3D%= s lockMode=3D%s"), + status, name, NULLSTR(virLockManagerDlmToMode= Text(mode))); + goto cleanup; + } + + if (!virLockManagerDlmRecordLock(name, mode, lksb.sb_lkid, vm_pid)) { + virReportSystemError(errno, + _("unable to record lock information, " + "lockName=3D%s lockMode=3D%s lockId=3D%d = vm_pid=3D%jd"), + NULLSTR(name), NULLSTR(virLockManagerDlmToMod= eText(mode)), + lksb.sb_lkid, (intmax_t)vm_pid); + } + + + cleanup: + if (name) + VIR_FREE(name); + + return; +} + +static int virLockManagerDlmPrepareLockList(const char *lockRecordFilePath) +{ + FILE *fp =3D NULL; + int line =3D 0; + size_t n =3D 0; + ssize_t count =3D 0; + char *buffer =3D NULL; + + fp =3D fopen(lockRecordFilePath, "r"); + if (!fp) { + virReportSystemError(errno, + _("unable to open '%s'"), lockRecordFilePath); + return -1; + } + + /* lock information is from the second line */ + for (line =3D 0; !feof(fp); line++) { + count =3D getline(&buffer, &n, fp); + if (count <=3D 0) + break; + + switch (line) { + case 0: + break; + default: + virLockManagerDlmAdoptLock(buffer); + break; + } + } + + VIR_FORCE_FCLOSE(fp); + VIR_FREE(buffer); + + return 0; +} + +static int virLockManagerDlmGetLocalNodeId(uint32_t *nodeId) +{ + cpg_handle_t handle =3D 0; + int rv =3D -1; + + if (cpg_model_initialize(&handle, CPG_MODEL_V1, NULL, NULL) !=3D CS_OK= ) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to create a new connection to the CPG ser= vice")); + return -1; + } + + if( cpg_local_get(handle, nodeId) !=3D CS_OK) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to get the local node id by the CPG servi= ce")); + goto cleanup; + } + + VIR_DEBUG("the local nodeid=3D%u", *nodeId); + + rv =3D 0; + + cleanup: + if (cpg_finalize(handle) !=3D CS_OK) + VIR_WARN("unable to finalize the CPG service"); + + return rv; +} + +static int virLockManagerDlmDumpLockList(const char *lockRecordFilePath) +{ + virLockInformationPtr theLock =3D NULL; + char buffer[BUFFERLEN] =3D {0}; + int fd =3D -1, rv =3D -1; + + /* not need mutex because of only one instance would be initialized */ + fd =3D open(lockRecordFilePath, O_WRONLY|O_CREAT|O_TRUNC, LOCK_RECORD_= FILE_MODE); + if (fd < 0) { + virReportSystemError(errno, + _("unable to open '%s'"), + lockRecordFilePath); + return -1; + } + + snprintf(buffer, sizeof(buffer), "%6s %64s %9s %10s\n", \ + STATUS, RESOURCE_NAME, LOCK_MODE, VM_PID); + if (safewrite(fd, buffer, strlen(buffer)) !=3D strlen(buffer)) { + virReportSystemError(errno, + _("unable to write '%s' to '%s'"), + buffer, lockRecordFilePath); + goto cleanup; + } + + virListForEachEntry(theLock, &(lockListWait.list), entry) { + virLockManagerDlmWriteLock(theLock, fd, 1); + } + + if (VIR_CLOSE(fd) < 0) { + virReportSystemError(errno, + _("unable to close file '%s'"), + lockRecordFilePath); + goto cleanup; + } + + rv =3D 0; + + cleanup: + if (rv) + VIR_FORCE_CLOSE(fd); + return rv; +} + +static int virLockManagerDlmSetupLockRecordFile(const char *lockRecordFile= Path, + const bool newLockspace, + const bool purgeLockspace) +{ + uint32_t nodeId =3D 0; + + /* there maybe some orphan locks recorded in the lock record file which + * should be adopted if lockspace is opened instead of created, we ado= pt + * them then add them in the list. + */ + if (!newLockspace && + virLockManagerDlmPrepareLockList(lockRecordFilePath)) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unable to adopt locks from '%s'"), + NULLSTR(lockRecordFilePath)); + return -1; + } + + /* purgeLockspace flag means purging orphan locks belong to any process + * in this lockspace. + */ + if (purgeLockspace && !virLockManagerDlmGetLocalNodeId(&nodeId)) { + if (dlm_ls_purge(lockspace, nodeId, 0)) { + VIR_WARN("node=3D%u purge DLM locks failed in lockspace=3D%s", + nodeId, NULLSTR(driver->lockspaceName)); + } + else + VIR_DEBUG("node=3D%u purge DLM locks success in lockspace=3D%s= ", + nodeId, NULLSTR(driver->lockspaceName)); + } + + /* initialize the lock record file */ + if (virLockManagerDlmDumpLockList(lockRecordFilePath)) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unable to initialize the lock record file '%s'"), + lockRecordFilePath); + return -1; + } + + return 0; +} + +static int virLockManagerDlmSetup(void) +{ + bool newLockspace =3D false; + + virListHeadInit(&(lockListWait.list)); + if ((virMutexInit(&(lockListWait.listMutex)) < 0) || + (virMutexInit(&(lockListWait.fileMutex)) < 0)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to initialize mutex")); + return -1; + } + + + /* check whether dlm is running or not */ + if (access(DLM_CLUSTER_NAME_PATH, F_OK)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("check dlm_controld, ensure it has setuped")); + return -1; + } + + /* open lockspace, create it if it doesn't exist */ + lockspace =3D dlm_open_lockspace(driver->lockspaceName); + if (!lockspace) { + lockspace =3D dlm_create_lockspace(driver->lockspaceName, DLM_LOCK= SPACE_MODE); + if (!lockspace) { + virReportSystemError(errno, "%s", + _("unable to open and create DLM lockspac= e")); + return -1; + } + newLockspace =3D true; + } + + /* create thread to receive notification from kernel */ + if (dlm_ls_pthread_init(lockspace)) { + if (errno !=3D EEXIST) { + virReportSystemError(errno, "%s", + _("unable to initialize lockspace")); + return -1; + } + } + + /* we need file to record lock information used by rebooted libvirtd */ + if (virLockManagerDlmSetupLockRecordFile(driver->lockRecordFilePath, + newLockspace, + driver->purgeLockspace)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unable to initialize DLM lock file")); + return -1; + } + + return 0; +} + +static int virLockManagerDlmDeinit(void); + +static int virLockManagerDlmInit(unsigned int version, + const char *configFile, + unsigned int flags) +{ + VIR_DEBUG("version=3D%u configFile=3D%s flags=3D0x%x", version, NULLST= R(configFile), flags); + + virCheckFlags(0, -1); + + if (driver) + return 0; + + if (geteuid() !=3D 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("dlm lock requires root privileges")); + return -1; + } + + if (VIR_ALLOC(driver) < 0) + return -1; + + driver->autoDiskLease =3D true; + driver->requireLeaseForDisks =3D !driver->autoDiskLease; + driver->purgeLockspace =3D true; + + if (virAsprintf(&driver->lockspaceName, + "%s", DLM_LOCKSPACE_NAME) < 0) + goto error; + + if (virAsprintf(&driver->lockRecordFilePath, + "%s", LOCK_RECORD_FILE_PATH) < 0) + goto error; + + if (virLockManagerDlmLoadConfig(configFile) < 0) + goto error; + + if (virLockManagerDlmSetup() < 0) + goto error; + + return 0; + + error: + virLockManagerDlmDeinit(); + return -1; +} + +static int virLockManagerDlmDeinit(void) +{ + virLockInformationPtr theLock =3D NULL; + + if (!driver) + return 0; + + if(lockspace) + dlm_close_lockspace(lockspace); + + /* not care about whether adopting lock or not, + * just release those to prevent memory leak + */ + virListForEachEntry(theLock, &(lockListWait.list), entry) { + virListDelete(&(theLock->entry)); + VIR_FREE(theLock->name); + VIR_FREE(theLock); + } + + VIR_FREE(driver->lockspaceName); + VIR_FREE(driver->lockRecordFilePath); + VIR_FREE(driver); + + return 0; +} + +static int virLockManagerDlmNew(virLockManagerPtr lock, + unsigned int type, + size_t nparams, + virLockManagerParamPtr params, + unsigned int flags) +{ + virLockManagerDlmPrivatePtr priv =3D NULL; + size_t i; + + virCheckFlags(VIR_LOCK_MANAGER_NEW_STARTED, -1); + + if (!driver) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("dlm plugin is not initialized")); + return -1; + } + + if (type !=3D VIR_LOCK_MANAGER_OBJECT_TYPE_DOMAIN) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unsupported object type %d"), type); + return -1; + } + + if (VIR_ALLOC(priv) < 0) + return -1; + + for (i =3D 0; i< nparams; i++) { + if (STREQ(params[i].key, "uuid")) { + memcpy(priv->vm_uuid, params[i].value.uuid, VIR_UUID_BUFLEN); + } else if (STREQ(params[i].key, "name")) { + if (VIR_STRDUP(priv->vm_name, params[i].value.str) < 0) + return -1; + } else if (STREQ(params[i].key, "id")) { + priv->vm_id =3D params[i].value.ui; + } else if (STREQ(params[i].key, "pid")) { + priv->vm_pid =3D params[i].value.iv; + } else if (STREQ(params[i].key, "uri")) { + /* there would be a warning in some case according to the hist= ory patch, + * so ignored + */ + } else { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unexpected parameter %s for object"), + params[i].key); + } + } + + /* check the following to prevent some unexpexted state in some case */ + if (priv->vm_pid =3D=3D 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing PID parameter for domain object")); + return -1; + } + if (!priv->vm_name) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing name parameter for domain object")); + return -1; + } + + if (priv->vm_id =3D=3D 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing ID parameter for domain object")); + return -1; + } + if (!virUUIDIsValid(priv->vm_uuid)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing UUID parameter for domain object")); + return -1; + } + + lock->privateData =3D priv; + + return 0; +} + +static void virLockManagerDlmFree(virLockManagerPtr lock) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + size_t i; + + if (!priv) + return; + + for (i =3D 0; i < priv->nresources; i++) + VIR_FREE(priv->resources[i].name); + + VIR_FREE(priv->resources); + VIR_FREE(priv->vm_name); + VIR_FREE(priv); + lock->privateData =3D NULL; + + return; +} + +static int virLockManagerDlmAddResource(virLockManagerPtr lock, + unsigned int type, const char *nam= e, + size_t nparams, + virLockManagerParamPtr params, + unsigned int flags) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + char *newName =3D NULL; + + virCheckFlags(VIR_LOCK_MANAGER_RESOURCE_READONLY | + VIR_LOCK_MANAGER_RESOURCE_SHARED, -1); + + /* Treat read only resources as a no-op lock request */ + if (flags & VIR_LOCK_MANAGER_RESOURCE_READONLY) + return 0; + + switch (type) { + case VIR_LOCK_MANAGER_RESOURCE_TYPE_DISK: + if (params || nparams) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("unexpected parameters for disk resource"= )); + return -1; + } + + if (!driver->autoDiskLease) { + if (!(flags & (VIR_LOCK_MANAGER_RESOURCE_SHARED | + VIR_LOCK_MANAGER_RESOURCE_READONLY))) { + priv->hasRWDisks =3D true; + /* ignore disk resource without error */ + return 0; + } + } + + if (virCryptoHashString(VIR_CRYPTO_HASH_SHA256, name, &newName= ) < 0) + goto cleanup; + + break; + + case VIR_LOCK_MANAGER_RESOURCE_TYPE_LEASE: + /* we need format the lock information, so the lock name must be t= he constant length */ + if (virCryptoHashString(VIR_CRYPTO_HASH_SHA256, name, &newName) < = 0) + goto cleanup; + + break; + + default: + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unknown lock manager object type %d"), + type); + return -1; + } + + if (VIR_EXPAND_N(priv->resources, priv->nresources, 1) < 0) + goto cleanup; + + priv->resources[priv->nresources-1].name =3D newName; + + if (!!(flags & VIR_LOCK_MANAGER_RESOURCE_SHARED)) + priv->resources[priv->nresources-1].mode =3D LKM_PRMODE; + else + priv->resources[priv->nresources-1].mode =3D LKM_EXMODE; + + return 0; + + cleanup: + VIR_FREE(newName); + + return -1; +} + +static int virLockManagerDlmAcquire(virLockManagerPtr lock, + const char *state ATTRIBUTE_UNUSED, + unsigned int flags, + virDomainLockFailureAction action ATTR= IBUTE_UNUSED, + int *fd) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + virLockInformationPtr theLock =3D NULL; + struct dlm_lksb lksb =3D {0}; + int rv =3D -1, theFd =3D -1; + size_t i; + + virCheckFlags(VIR_LOCK_MANAGER_ACQUIRE_REGISTER_ONLY | + VIR_LOCK_MANAGER_ACQUIRE_RESTRICT, -1); + + /* allowed to start a guest which has read/write disks, but without an= y leases */ + if (priv->nresources =3D=3D 0 && + priv->hasRWDisks && + driver->requireLeaseForDisks) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("read/write, exclusive access, disks were present= , but no leases specified")); + return -1; + } + + /* accorting to git patch history, add `fd` parameter in order to + * 'ensure sanlock socket is labelled with the VM process label', + * however, fixing sanlock socket security labelling remove related + * code. Now, `fd` parameter is useless. + */ + if (fd) + *fd =3D -1; + + if(!lockspace) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("lockspace is not opened")); + return -1; + } + + if (!(flags & VIR_LOCK_MANAGER_ACQUIRE_REGISTER_ONLY)) { + VIR_DEBUG("Acquiring object %zu", priv->nresources); + + theFd =3D open(driver->lockRecordFilePath, O_RDWR); + if (theFd < 0) { + virReportSystemError(errno, + _("unable to open '%s'"), driver->lockRec= ordFilePath); + return -1; + } + + for (i =3D 0; i < priv->nresources; i++) { + VIR_DEBUG("Acquiring object %zu", priv->nresources); + + memset(&lksb, 0, sizeof(lksb)); + rv =3D dlm_ls_lock_wait(lockspace, priv->resources[i].mode, + &lksb, LKF_NOQUEUE|LKF_PERSISTENT, + priv->resources[i].name, strlen(priv->re= sources[i].name), + 0, NULL, NULL, NULL); + /* both `rv` and `lksb.sb_status` equal 0 means lock sucessful= ly */ + if (rv || lksb.sb_status) { + if (lksb.sb_status =3D=3D EAGAIN) + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("failed to acquire lock: the lock cou= ld not be granted")); + else { + virReportSystemError(errno, + _("failed to acquire lock: rv=3D%= d lockStatus=3D%d"), + rv, lksb.sb_status); + } + /* rv would be 0 although acquiring lock failed */ + rv =3D -1; + goto cleanup; + } + + theLock =3D virLockManagerDlmRecordLock(priv->resources[i].nam= e, + priv->resources[i].mode, + lksb.sb_lkid, + priv->vm_pid); + if (!theLock) { + virReportSystemError(errno, + _("unable to record lock information,= " + "lockName=3D%s lockMode=3D%s lockI= d=3D%d vm_pid=3D%jd"), + NULLSTR(priv->resources[i].name), + NULLSTR(virLockManagerDlmToModeText(p= riv->resources[i].mode)), + lksb.sb_lkid, (intmax_t)priv->vm_pid); + /* record lock failed, we can't save lock information in m= emory, so release it */ + rv =3D dlm_ls_unlock_wait(lockspace, lksb.sb_lkid, 0, &lks= b); + if (!rv) + virReportSystemError(errno, + _("failed to release lock: rv=3D%= d lockStatue=3D%d"), + rv, lksb.sb_status); + rv =3D -1; + goto cleanup; + } + + virMutexLock(&(lockListWait.fileMutex)); + virLockManagerDlmWriteLock(theLock, theFd, 1); + virMutexUnlock(&(lockListWait.fileMutex)); + } + + if(VIR_CLOSE(theFd) < 0) { + virReportSystemError(errno, + _("unable to save file '%s'"), + driver->lockRecordFilePath); + goto cleanup; + } + } + + if (flags & VIR_LOCK_MANAGER_ACQUIRE_RESTRICT) { + /* no daemon watches this fd, do nothing here, just close lockspac= e before `execv` + * `dlm_close_lockspace` always return 0, so ignore return value + */ + ignore_value(dlm_close_lockspace(lockspace)); + lockspace =3D NULL; + } + + rv =3D 0; + + cleanup: + if (rv) + VIR_FORCE_CLOSE(theFd); + return rv; +} + +static void virLockManagerDlmDeleteLock(const virLockInformationPtr lock, + const char *lockRecordFilePath) +{ + int fd =3D -1; + + if (!lock) + return; + + virMutexLock(&(lockListWait.listMutex)); + virListDelete(&(lock->entry)); + virMutexUnlock(&(lockListWait.listMutex)); + + fd =3D open(lockRecordFilePath, O_RDWR); + if (fd < 0) { + virReportSystemError(errno, + _("unable to open '%s'"), lockRecordFilePath); + goto cleanup; + } + + virMutexLock(&(lockListWait.fileMutex)); + virLockManagerDlmWriteLock(lock, fd, 0); + virMutexUnlock(&(lockListWait.fileMutex)); + + if (VIR_CLOSE(fd) < 0) { + virReportSystemError(errno, + _("unable to save file '%s'"), + lockRecordFilePath); + VIR_FORCE_CLOSE(fd); + } + + cleanup: + VIR_FREE(lock->name); + VIR_FREE(lock); +} + +static int virLockManagerDlmRelease(virLockManagerPtr lock, + char **state, + unsigned int flags) +{ + virLockManagerDlmPrivatePtr priv =3D lock->privateData; + virLockManagerDlmResourcePtr resource =3D NULL; + virLockInformationPtr theLock =3D NULL; + struct dlm_lksb lksb =3D {0}; + int rv =3D -1; + size_t i; + + virCheckFlags(0, -1); + + if(state) + *state =3D NULL; + + if(!lockspace) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("lockspace is not opened")); + return -1; + } + + for (i =3D 0; i < priv->nresources; i++) { + resource =3D priv->resources + i; + + virListForEachEntry (theLock, &(lockListWait.list), entry) { + if((theLock->vm_pid =3D=3D priv->vm_pid) && + STREQ(theLock->name, resource->name) && + (theLock->mode =3D=3D resource->mode)) { + + /* + * there are some locks from adopting, the existence of `(= void *)1` + * when adopting makes 'terminated by signal SIGSEGV (Addr= ess + * boundary error)' error appear. + * + * The following code reference to lvm2 project's implemen= t. + */ + lksb.sb_lkid =3D theLock->lkid; + rv =3D dlm_ls_lock_wait(lockspace, LKM_NLMODE, + &lksb, LKF_CONVERT, + resource->name, + strlen(resource->name), + 0, NULL, NULL, NULL); + + if (rv < 0) { + virReportSystemError(errno, + _("failed to convert lock: rv=3D%= d lockStatus=3D%d"), + rv, lksb.sb_status); + goto cleanup; + } + + /* don't care whether the lock is released or not, + * it will be automatically released after the libvirtd de= ad + */ + virLockManagerDlmDeleteLock(theLock, driver->lockRecordFil= ePath); + + rv =3D dlm_ls_unlock_wait(lockspace, lksb.sb_lkid, 0, &lks= b); + if (rv < 0) { + virReportSystemError(errno, + _("failed to release lock: rv=3D%= d lockStatus=3D%d"), + rv, lksb.sb_status); + goto cleanup; + } + + break; + } + } + } + + rv =3D 0; + + cleanup: + return rv; +} + +static int virLockManagerDlmInquire(virLockManagerPtr lock ATTRIBUTE_UNUSE= D, + char **state, + unsigned int flags) +{ + /* not support mannual lock, so this function almost does nothing */ + virCheckFlags(0, -1); + + if (state) + *state =3D NULL; + + return 0; +} + +virLockDriver virLockDriverImpl =3D +{ + .version =3D VIR_LOCK_MANAGER_VERSION, + + .flags =3D VIR_LOCK_MANAGER_USES_STATE, // currently not used + + .drvInit =3D virLockManagerDlmInit, + .drvDeinit =3D virLockManagerDlmDeinit, + + .drvNew =3D virLockManagerDlmNew, + .drvFree =3D virLockManagerDlmFree, + + .drvAddResource =3D virLockManagerDlmAddResource, + + .drvAcquire =3D virLockManagerDlmAcquire, + .drvRelease =3D virLockManagerDlmRelease, + .drvInquire =3D virLockManagerDlmInquire, +}; diff --git a/src/util/virlist.h b/src/util/virlist.h new file mode 100644 index 000000000..4ac3626c7 --- /dev/null +++ b/src/util/virlist.h @@ -0,0 +1,110 @@ +/* + * virlist.h: methods for managing list, logic is copied from Linux Kernel + * + * Copyright (C) 2018 SUSE LINUX Products, Beijing, China. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * . + * + */ + +#ifndef __VIR_LIST_H +#define __VIR_LIST_H + +#include + +typedef struct _virListHead virListHead; +typedef virListHead *virListHeadPtr; + +struct _virListHead { + struct _virListHead *next, *prev; +}; + +static inline void +virListHeadInit(virListHeadPtr name) +{ + name->next =3D name; + name->prev =3D name; +} + +static inline void +__virListAdd(virListHeadPtr entry, + virListHeadPtr prev, virListHeadPtr next) +{ + next->prev =3D entry; + entry->next =3D next; + entry->prev =3D prev; + prev->next =3D entry; +} + +static inline void +virListAdd(virListHeadPtr entry, virListHeadPtr head) +{ + __virListAdd(entry, head, head->next); +} + +static inline void +virListAddTail(virListHeadPtr entry, virListHeadPtr head) +{ + __virListAdd(entry, head->prev, head); +} + +static inline void +__virListDelete(virListHeadPtr prev, virListHeadPtr next) +{ + next->prev =3D prev; + prev->next =3D next; +} + +static inline void +virListDelete(virListHeadPtr entry) +{ + __virListDelete(entry->prev, entry->next); +} + +static inline bool +virListEmpty(virListHeadPtr head) +{ + return head->next =3D=3D head; +} + +#ifndef virContainerOf +#define virContainerOf(ptr, type, member) \ + (type *)((char *)(ptr) - (char *) &((type *)0)->member) +#endif + +#define virListEntry(ptr, type, member) \ + virContainerOf(ptr, type, member) + +#define virListFirstEntry(ptr, type, member) \ + virListEntry((ptr)->next, type, member) + +#define virListLastEntry(ptr, type, member) \ + virListEntry((ptr)->prev, type, member) + +#define __virContainerOf(ptr, sample, member) \ + (void *)virContainerOf((ptr), typeof(*(sample)), member) + +#define virListForEachEntry(pos, head, member) \ + for (pos =3D __virContainerOf((head)->next, pos, member); \ + &pos->member !=3D (head); \ + pos =3D __virContainerOf(pos->member.next, pos, member)) + +#define virListForEachEntrySafe(pos, tmp, head, member) \ + for (pos =3D __virContainerOf((head)->next, pos, member), \ + tmp =3D __virContainerOf(pos->member.next, pos, member); \ + &pos->member !=3D (head); \ + pos =3D tmp, tmp =3D __virContainerOf(pos->member.next, tmp, member)) + +#endif --=20 2.15.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Mon Apr 29 05:43:41 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1517821721554815.4628664797551; Mon, 5 Feb 2018 01:08:41 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 711307E428; Mon, 5 Feb 2018 09:08:38 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0FC2A18EE1; Mon, 5 Feb 2018 09:08:37 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id C71C318033D9; Mon, 5 Feb 2018 09:08:32 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w1598UeZ000815 for ; Mon, 5 Feb 2018 04:08:30 -0500 Received: by smtp.corp.redhat.com (Postfix) id 3604F691CC; Mon, 5 Feb 2018 09:08:30 +0000 (UTC) Received: from mx1.redhat.com (ext-mx08.extmail.prod.ext.phx2.redhat.com [10.5.110.32]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2B66460F81 for ; Mon, 5 Feb 2018 09:08:26 +0000 (UTC) Received: from prv3-mh.provo.novell.com (victor.provo.novell.com [137.65.250.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 56221C0587C1 for ; Mon, 5 Feb 2018 09:08:25 +0000 (UTC) Received: from localhost.localdomain (prv-ext-foundry1int.gns.novell.com [137.65.251.240]) by prv3-mh.provo.novell.com with ESMTP (NOT encrypted); Mon, 05 Feb 2018 02:08:17 -0700 From: river To: libvir-list@redhat.com Date: Mon, 5 Feb 2018 17:07:55 +0800 Message-Id: <1517821676-14399-3-git-send-email-lfu@suse.com> In-Reply-To: <1517821676-14399-1-git-send-email-lfu@suse.com> References: <1517821676-14399-1-git-send-email-lfu@suse.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 207 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) for IP:'137.65.250.26' DOMAIN:'victor.provo.novell.com' HELO:'prv3-mh.provo.novell.com' FROM:'lfu@suse.com' RCPT:'' X-RedHat-Spam-Score: -2.301 (RCVD_IN_DNSWL_MED, SPF_PASS) 137.65.250.26 victor.provo.novell.com 137.65.250.26 victor.provo.novell.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.32 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 2/3] Fix release of locks with dlm plugin X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Mon, 05 Feb 2018 09:08:40 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The dlm lock plugin couldn't release locks when the vm instance was destroyed, kill manually because that we can't use file descriptor which couldn't adopt after libvirtd reboot to be aware of the process dead, we need to proactively to release the lock just like unlink the .xml and .pid files. This operation could be found in libxl. Signed-off-by: river --- src/qemu/qemu_process.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 5a364730c..49c64be6f 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -6626,6 +6626,11 @@ void qemuProcessStop(virQEMUDriverPtr driver, =20 qemuProcessRemoveDomainStatus(driver, vm); =20 + VIR_FREE(priv->lockState); + if (virDomainLockProcessPause(driver->lockManager, vm, &priv->lockStat= e) < 0) + VIR_WARN("Unable to release lease on %s", vm->def->name); + VIR_DEBUG("Preserving lock state '%s'", NULLSTR(priv->lockState)); + /* Remove VNC and Spice ports from port reservation bitmap, but only if they were reserved by the driver (autoport=3Dyes) */ --=20 2.15.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list From nobody Mon Apr 29 05:43:41 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1517822142556596.3717444040748; Mon, 5 Feb 2018 01:15:42 -0800 (PST) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 64FA681DE4; Mon, 5 Feb 2018 09:15:41 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CFAAB608F0; Mon, 5 Feb 2018 09:15:40 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id DA0CD18033D9; Mon, 5 Feb 2018 09:15:39 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w1598bqE000855 for ; Mon, 5 Feb 2018 04:08:37 -0500 Received: by smtp.corp.redhat.com (Postfix) id 170D118EE3; Mon, 5 Feb 2018 09:08:37 +0000 (UTC) Received: from mx1.redhat.com (ext-mx02.extmail.prod.ext.phx2.redhat.com [10.5.110.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 05C2B18EDF for ; Mon, 5 Feb 2018 09:08:33 +0000 (UTC) Received: from prv3-mh.provo.novell.com (victor.provo.novell.com [137.65.250.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6660B76545 for ; Mon, 5 Feb 2018 09:08:25 +0000 (UTC) Received: from localhost.localdomain (prv-ext-foundry1int.gns.novell.com [137.65.251.240]) by prv3-mh.provo.novell.com with ESMTP (NOT encrypted); Mon, 05 Feb 2018 02:08:21 -0700 From: river To: libvir-list@redhat.com Date: Mon, 5 Feb 2018 17:07:56 +0800 Message-Id: <1517821676-14399-4-git-send-email-lfu@suse.com> In-Reply-To: <1517821676-14399-1-git-send-email-lfu@suse.com> References: <1517821676-14399-1-git-send-email-lfu@suse.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 207 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 05 Feb 2018 09:08:25 +0000 (UTC) for IP:'137.65.250.26' DOMAIN:'victor.provo.novell.com' HELO:'prv3-mh.provo.novell.com' FROM:'lfu@suse.com' RCPT:'' X-RedHat-Spam-Score: -2.301 (RCVD_IN_DNSWL_MED, SPF_PASS) 137.65.250.26 victor.provo.novell.com 137.65.250.26 victor.provo.novell.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.26 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH 3/3] Add configuration files for dlm plugin X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Mon, 05 Feb 2018 09:15:42 +0000 (UTC) X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" Signed-off-by: river --- .gitignore | 2 ++ src/Makefile.am | 40 +++++++++++++++++++++++++++- src/libxl/libxl.conf | 8 +++--- src/locking/dlm.conf | 52 +++++++++++++++++++++++++++++++++= ++++ src/locking/libvirt_dlm.aug | 35 +++++++++++++++++++++++++ src/locking/test_libvirt_dlm.aug.in | 9 +++++++ src/qemu/qemu.conf | 8 +++--- 7 files changed, 146 insertions(+), 8 deletions(-) create mode 100644 src/locking/dlm.conf create mode 100644 src/locking/libvirt_dlm.aug create mode 100644 src/locking/test_libvirt_dlm.aug.in diff --git a/.gitignore b/.gitignore index 189116a3d..d9b0aaec1 100644 --- a/.gitignore +++ b/.gitignore @@ -139,10 +139,12 @@ /src/libvirt_*helper /src/libvirt_*probes.h /src/libvirt_lxc +/src/locking/libxl-dlm.conf /src/locking/libxl-lockd.conf /src/locking/libxl-sanlock.conf /src/locking/lock_daemon_dispatch_stubs.h /src/locking/lock_protocol.[ch] +/src/locking/qemu-dlm.conf /src/locking/qemu-lockd.conf /src/locking/qemu-sanlock.conf /src/locking/test_libvirt_sanlock.aug diff --git a/src/Makefile.am b/src/Makefile.am index 8742921fa..95b46dc89 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -2111,6 +2111,19 @@ else ! WITH_LXC check-augeas-lxc: endif ! WITH_LXC =20 +if WITH_DLM +test_libvirt_dlm.aug: locking/test_libvirt_dlm.aug.in \ + locking/qemu-dlm.conf $(AUG_GENTEST) + $(AM_V_GEN)$(AUG_GENTEST) locking/qemu-dlm.conf $< $@ + +check-augeas-dlm: test_libvirt_dlm.aug + $(AM_V_GEN)if test -x '$(AUGPARSE)'; then \ + '$(AUGPARSE)' -I $(srcdir)/locking test_libvirt_dlm.aug; \ + fi +else ! WITH_DLM +check-augeas-dlm: +endif ! WITH_DLM + if WITH_SANLOCK test_libvirt_sanlock.aug: locking/test_libvirt_sanlock.aug.in \ locking/qemu-sanlock.conf $(AUG_GENTEST) @@ -2891,9 +2904,34 @@ dlm_la_LDFLAGS =3D -module -avoid-version $(AM_LDFLA= GS) dlm_la_LIBADD =3D ../gnulib/lib/libgnu.la \ $(CPG_LIBS) \ $(DLM_LIBS) + +augeas_DATA +=3D locking/libvirt_dlm.aug +augeastest_DATA +=3D test_libvirt_dlm.aug +CLEANFILES +=3D test_libvirt_dlm.aug + +locking/%-dlm.conf: $(srcdir)/locking/dlm.conf + $(AM_V_GEN)$(MKDIR_P) locking ; \ + cp $< $@ + +if WITH_QEMU +nodist_conf_DATA +=3D locking/qemu-dlm.conf +BUILT_SOURCES +=3D locking/qemu-dlm.conf +DISTCLEANFILES +=3D locking/qemu-dlm.conf +endif WITH_QEMU + +if WITH_LIBXL +nodist_conf_DATA +=3D locking/libxl-dlm.conf +BUILT_SOURCES +=3D locking/libxl-dlm.conf +DISTCLEANFILES +=3D locking/libxl-dlm.conf +endif WITH_LIBXL + else ! WITH_DLM EXTRA_DIST +=3D $(LOCK_DRIVER_DLM_SOURCES) -endif +endif ! WITH_DLM + +EXTRA_DIST +=3D locking/dlm.conf \ + locking/libvirt_dlm.aug \ + locking/test_libvirt_dlm.aug.in =20 if WITH_SANLOCK lockdriver_LTLIBRARIES +=3D sanlock.la diff --git a/src/libxl/libxl.conf b/src/libxl/libxl.conf index 264af7cf9..85be09b99 100644 --- a/src/libxl/libxl.conf +++ b/src/libxl/libxl.conf @@ -13,11 +13,13 @@ =20 =20 # In order to prevent accidentally starting two domains that -# share one writable disk, libvirt offers two approaches for -# locking files: sanlock and virtlockd. sanlock is an external +# share one writable disk, libvirt offers three approaches for +# locking files: sanlock, virtlockd and dlm. sanlock is an external # project which libvirt integrates with via the libvirt-lock-sanlock # package. virtlockd is a libvirt implementation that is enabled with -# "lockd". Accepted values are "sanlock" and "lockd". +# "lockd", dlm is the linux kernel implementation lock manager that +# is using cluster software by other projects. Accepted values are +# "sanlock", "lockd" and "dlm". # #lock_manager =3D "lockd" =20 diff --git a/src/locking/dlm.conf b/src/locking/dlm.conf new file mode 100644 index 000000000..7cc60f5c2 --- /dev/null +++ b/src/locking/dlm.conf @@ -0,0 +1,52 @@ +# +# The default lockd behaviour is to acquire locks directly +# against each configured disk file / block device. If the +# application wishes to instead manually manage leases in +# the guest XML, then this parameter can be disabled +# +#auto_disk_leases =3D 1 + +# +# Flag to determine whether we allow starting of guests +# which do not have any elements defined in their +# configuration. +# +# If 'auto_disk_leases' is disabled, this setting defaults +# to enabled, otherwise it defaults to disabled. +# +#require_lease_for_disks =3D 0 + +# +# The DLM allows locks to be partitioned into "lockspaces", +# The purpose of lockspaces is to provide a private namespace +# for locks that are part of a single application. Lockspaces +# are identified by name and are cluster-wide. A lockspace +# named "myLS" is the same lockspace on all nodes in the +# cluster and locks will contend for resources the same as +# if they were on the same system. Lockspace names are +# case-sensitive so "MyLS" is a distinct lockspace to "myLS". +# +# More information refers to '3.8. Lockspaces' in +# http://people.redhat.com/ccaulfie/docs/rhdlmbook.pdf +# +#lockspace_name =3D "libvirt" + +# +# One lock must belong to one lockspace, and is associated +# with one lock resource, owned by one process. The lock +# without `LKF_PERSISTENT` would disappear if owner process +# was dead. However, a lock with `LKF_PERSISTENT` flag means +# lock is alive although owner process was dead. However, it +# will become a orphan lock and could't be released. In order +# to adopt those orphan locks we need a file to record the +# lock information. +# +#lock_record_file_path =3D "/tmp/libvirtd-dlm-file" +# + +# +# Flag to determine to whether purge orphan locks which could +# not be adopted or not during the dlm lock plugin +# initialization. +# +#purge_lockspace =3D 1 diff --git a/src/locking/libvirt_dlm.aug b/src/locking/libvirt_dlm.aug new file mode 100644 index 000000000..257e4c421 --- /dev/null +++ b/src/locking/libvirt_dlm.aug @@ -0,0 +1,35 @@ +(* /etc/libvirt/qemu-lockd.conf *) + +module Libvirt_lockd =3D + autoload xfm + + let eol =3D del /[ \t]*\n/ "\n" + let value_sep =3D del /[ \t]*=3D[ \t]*/ " =3D " + let indent =3D del /[ \t]*/ "" + + let str_val =3D del /\"/ "\"" . store /[^\"]*/ . del /\"/ "\"" + let bool_val =3D store /0|1/ + let int_val =3D store /[0-9]+/ + + let str_entry (kw:string) =3D [ key kw . value_sep . str_val ] + let bool_entry (kw:string) =3D [ key kw . value_sep . bool_val ] + let int_entry (kw:string) =3D [ key kw . value_sep . int_val ] + + + (* Each enty in the config is one of the following three ... *) + let entry =3D bool_entry "auto_disk_leases" + | bool_entry "require_lease_for_disks" + | str_entry "lockspace_name" + | str_entry "lock_record_file_path" + | str_entry "purge_lockspace" + let comment =3D [ label "#comment" . del /#[ \t]*/ "# " . store /([^ \= t\n][^\n]*)?/ . del /\n/ "\n" ] + let empty =3D [ label "#empty" . eol ] + + let record =3D indent . entry . eol + + let lns =3D ( record | comment | empty ) * + + let filter =3D incl "/etc/libvirt/qemu-lockd.conf" + . Util.stdexcl + + let xfm =3D transform lns filter diff --git a/src/locking/test_libvirt_dlm.aug.in b/src/locking/test_libvirt= _dlm.aug.in new file mode 100644 index 000000000..ceb8a0324 --- /dev/null +++ b/src/locking/test_libvirt_dlm.aug.in @@ -0,0 +1,9 @@ +module Test_libvirt_dlm =3D + ::CONFIG:: + + test Libvirt_dlm.lns get conf =3D +{ "auto_disk_leases" =3D "1" } +{ "require_lease_for_disks" =3D "0" } +{ "lockspace_name" =3D "libvirt" } +{ "lock_record_file_path" =3D "/tmp/libvirtd-dlm-file" } +{ "purge_lockspace" =3D "1" } diff --git a/src/qemu/qemu.conf b/src/qemu/qemu.conf index 62c4265ea..bcb6b80bd 100644 --- a/src/qemu/qemu.conf +++ b/src/qemu/qemu.conf @@ -629,10 +629,10 @@ =20 =20 # In order to prevent accidentally starting two domains that -# share one writable disk, libvirt offers two approaches for -# locking files. The first one is sanlock, the other one, -# virtlockd, is then our own implementation. Accepted values -# are "sanlock" and "lockd". +# share one writable disk, libvirt offers three approaches for +# locking files. The first one is sanlock, the second one, +# virtlockd, is then our own implementation, the third one is +# dlm. Accepted values are "sanlock", "lockd" and "dlm". # #lock_manager =3D "lockd" =20 --=20 2.15.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list