From nobody Sat Feb 7 08:58:33 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.39 as permitted sender) client-ip=209.132.183.39; envelope-from=libvir-list-bounces@redhat.com; helo=mx6-phx2.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.39 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by mx.zohomail.com with SMTPS id 1486972138524588.8355008744384; Sun, 12 Feb 2017 23:48:58 -0800 (PST) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v1D7jMe9008486; Mon, 13 Feb 2017 02:45:22 -0500 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v1D7jJ1E011432 for ; Mon, 13 Feb 2017 02:45:19 -0500 Received: from mx1.redhat.com (ext-mx07.extmail.prod.ext.phx2.redhat.com [10.5.110.31]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v1D7jJgE032347 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 13 Feb 2017 02:45:19 -0500 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id EDCD3C04B945; Mon, 13 Feb 2017 07:45:17 +0000 (UTC) Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 12 Feb 2017 23:45:16 -0800 Received: from s2600wt.sh.intel.com ([10.239.48.158]) by orsmga005.jf.intel.com with ESMTP; 12 Feb 2017 23:45:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.35,155,1484035200"; d="scan'208";a="63990452" From: Eli Qiao To: libvir-list@redhat.com Date: Mon, 13 Feb 2017 15:46:47 +0800 Message-Id: <1486972007-15514-10-git-send-email-liyong.qiao@intel.com> In-Reply-To: <1486972007-15514-1-git-send-email-liyong.qiao@intel.com> References: <1486972007-15514-1-git-send-email-liyong.qiao@intel.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 200 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Mon, 13 Feb 2017 07:45:18 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Mon, 13 Feb 2017 07:45:18 +0000 (UTC) for IP:'192.55.52.115' DOMAIN:'mga14.intel.com' HELO:'mga14.intel.com' FROM:'liyong.qiao@intel.com' RCPT:'' X-RedHat-Spam-Score: -4.622 (BAYES_50, DCC_REPUT_00_12, RCVD_IN_DNSWL_HI, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RP_MATCHES_RCVD, SPF_PASS) 192.55.52.115 mga14.intel.com 192.55.52.115 mga14.intel.com X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Scanned-By: MIMEDefang 2.78 on 10.5.110.31 X-loop: libvir-list@redhat.com Cc: mkletzan@redhat.com, qiaowei.ren@intel.com, mtosatti@redhat.com, qiaoliyong@gmail.com Subject: [libvirt] [v5 9/9] Resctrl: concurrence support X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The internal struct list domainall is a list which are resctral domain status shared by all VMs, especiall the default domain, each VM should access it concomitantly. Ues a mutex to control it. Each bank's cache_left field is also a global shared resource we need to be care, add a mutex for each bank. We need also to add lock to access /sys/fs/resctrl, use flock. Signed-off-by: Eli Qiao --- src/util/virresctrl.c | 89 ++++++++++++++++++++++++++++++++++++++++++++---= ---- src/util/virresctrl.h | 3 ++ 2 files changed, 81 insertions(+), 11 deletions(-) diff --git a/src/util/virresctrl.c b/src/util/virresctrl.c index 3552761..26df276 100644 --- a/src/util/virresctrl.c +++ b/src/util/virresctrl.c @@ -24,6 +24,7 @@ #if defined HAVE_SYS_SYSCALL_H # include #endif +#include #include #include #include @@ -68,8 +69,8 @@ do { \ =20 #define VIR_RESCTRL_GET_SCHEMATA(count) ((1 << count) - 1) =20 -#define VIR_RESCTRL_SET_SCHEMATA(p, type, pos, val) \ - p->schematas[type]->schemata_items[pos] =3D val +#define VIR_RESCTRL_LOCK(fd, op) flock(fd, op) +#define VIR_RESCTRL_UNLOCK(fd) flock(fd, LOCK_UN) =20 /** * a virResSchemata represents a schemata object under a resource control @@ -114,6 +115,8 @@ typedef virResCtrlDomain *virResCtrlDomainPtr; struct _virResCtrlDomain { unsigned int num_domains; virResDomainPtr domains; + + virMutex lock; }; =20 =20 @@ -166,16 +169,34 @@ static int virResCtrlGetStr(const char *domain_name, = const char *item_name, char { char *path; int rc =3D 0; + int readfd; + int len; =20 CONSTRUCT_RESCTRL_PATH(domain_name, item_name); =20 - if (virFileReadAll(path, MAX_FILE_LEN, ret) < 0) { - rc =3D -1; + if (!virFileExists(path)) + goto cleanup; + + if ((readfd =3D open(path, O_RDONLY)) < 0) + goto cleanup; + + if (VIR_RESCTRL_LOCK(readfd, LOCK_SH) < 0) { + virReportSystemError(errno, _("Unable to lock '%s'"), path); + goto cleanup; + } + + len =3D virFileReadLimFD(readfd, MAX_FILE_LEN, ret); + + if (len < 0) { + virReportSystemError(errno, _("Failed to read file '%s'"), path); goto cleanup; } + rc =3D 0; =20 cleanup: VIR_FREE(path); + VIR_RESCTRL_UNLOCK(readfd); + VIR_FORCE_CLOSE(readfd); return rc; } =20 @@ -616,6 +637,9 @@ virResCtrlRefresh(void) unsigned int origin_count =3D domainall.num_domains; virResDomainPtr p, pre, del; pre =3D domainall.domains; + + virMutexLock(&domainall.lock); + p =3D del =3D NULL; if (pre) p =3D pre->next; @@ -640,10 +664,11 @@ virResCtrlRefresh(void) p =3D p->next; } VIR_FREE(tasks); - } =20 - return virResCtrlRefreshSchemata(); + virResCtrlRefreshSchemata(); + virMutexUnlock(&domainall.lock); + return 0; } =20 /* Get a domain ptr by domain's name*/ @@ -689,6 +714,11 @@ virResCtrlWrite(const char *name, const char *item, co= nst char *content) if ((writefd =3D open(path, O_WRONLY | O_APPEND, S_IRUSR | S_IWUSR)) <= 0) goto cleanup; =20 + if (VIR_RESCTRL_LOCK(writefd, LOCK_EX) < 0) { + virReportSystemError(errno, _("Unable to lock '%s'"), path); + goto cleanup; + } + if (safewrite(writefd, content, strlen(content)) < 0) goto cleanup; =20 @@ -696,6 +726,7 @@ virResCtrlWrite(const char *name, const char *item, con= st char *content) =20 cleanup: VIR_FREE(path); + VIR_RESCTRL_UNLOCK(writefd); VIR_FORCE_CLOSE(writefd); return rc; } @@ -868,10 +899,15 @@ static int virResCtrlAppendDomain(virResDomainPtr dom) { virResDomainPtr p =3D domainall.domains; + + virMutexLock(&domainall.lock); + while (p->next !=3D NULL) p =3D p->next; p->next =3D dom; dom->pre =3D p; domainall.num_domains +=3D 1; + + virMutexUnlock(&domainall.lock); return 0; } =20 @@ -894,18 +930,22 @@ virResCtrlCalculateSchemata(int type, { size_t i; int count; + int rc =3D -1; virResDomainPtr p; unsigned int tmp_schemata; unsigned int schemata_sum =3D 0; int pair_type =3D 0; =20 + virMutexLock(&resctrlall[type].cache_banks[sid].lock); + if (resctrlall[type].cache_banks[sid].cache_left < size) { VIR_ERROR(_("Not enough cache left on bank %u"), hostid); - return -1; + goto cleanup; } + if ((count =3D size / resctrlall[type].cache_banks[sid].cache_min) <= =3D 0) { VIR_ERROR(_("Error cache size %llu"), size); - return -1; + goto cleanup; } =20 tmp_schemata =3D VIR_RESCTRL_GET_SCHEMATA(count); @@ -920,7 +960,7 @@ virResCtrlCalculateSchemata(int type, if (type =3D=3D VIR_RDT_RESOURCE_L3CODE) pair_type =3D VIR_RDT_RESOURCE_L3DATA; =20 - for (i =3D 1; i < domainall.num_domains; i ++) { + for (i =3D 1; i < domainall.num_domains; i++) { schemata_sum |=3D p->schematas[type]->schemata_items[sid].schemata; if (pair_type > 0) schemata_sum |=3D p->schematas[pair_type]->schemata_items[sid]= .schemata; @@ -931,7 +971,16 @@ virResCtrlCalculateSchemata(int type, =20 while ((tmp_schemata & schemata_sum) !=3D 0) tmp_schemata =3D tmp_schemata >> 1; - return tmp_schemata; + + resctrlall[type].cache_banks[sid].cache_left -=3D size; + if (pair_type > 0) + resctrlall[pair_type].cache_banks[sid].cache_left =3D resctrlall[t= ype].cache_banks[sid].cache_left; + + rc =3D tmp_schemata; + + cleanup: + virMutexUnlock(&resctrlall[type].cache_banks[sid].lock); + return rc; } =20 int virResCtrlSetCacheBanks(virDomainCachetunePtr cachetune, @@ -1043,7 +1092,7 @@ int virResCtrlUpdate(void) int virResCtrlInit(void) { - size_t i =3D 0; + size_t i, j; char *tmp; int rc =3D 0; =20 @@ -1063,6 +1112,24 @@ virResCtrlInit(void) =20 domainall.domains =3D virResCtrlGetAllDomains(&(domainall.num_domains)= ); =20 + for (i =3D 0; i < VIR_RDT_RESOURCE_LAST; i++) { + if (VIR_RESCTRL_ENABLED(i)) { + for (j =3D 0; j < resctrlall[i].num_banks; j++) { + if (virMutexInit(&resctrlall[i].cache_banks[j].lock) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Unable to initialize mutex")); + return -1; + } + } + } + } + + if (virMutexInit(&domainall.lock) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Unable to initialize mutex")); + return -1; + } + if ((rc =3D virResCtrlRefresh()) < 0) VIR_ERROR(_("Failed to refresh resource control")); return rc; diff --git a/src/util/virresctrl.h b/src/util/virresctrl.h index ee7e115..07b298d 100644 --- a/src/util/virresctrl.h +++ b/src/util/virresctrl.h @@ -27,6 +27,7 @@ # include "virutil.h" # include "virbitmap.h" # include "domain_conf.h" +# include "virthread.h" =20 #define MAX_CPU_SOCKET_NUM 8 =20 @@ -49,6 +50,8 @@ struct _virResCacheBank { unsigned long long cache_left; unsigned long long cache_min; virBitmapPtr cpu_mask; + + virMutex lock; }; =20 /** --=20 1.9.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list