From nobody Tue Feb 10 00:57:51 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.24 as permitted sender) client-ip=209.132.183.24; envelope-from=libvir-list-bounces@redhat.com; helo=mx3-phx2.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.24 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by mx.zohomail.com with SMTPS id 1487237886744360.84336203594637; Thu, 16 Feb 2017 01:38:06 -0800 (PST) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v1G9Y53o016240; Thu, 16 Feb 2017 04:34:05 -0500 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v1G9XnJQ030723 for ; Thu, 16 Feb 2017 04:33:49 -0500 Received: from mx1.redhat.com (ext-mx10.extmail.prod.ext.phx2.redhat.com [10.5.110.39]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v1G9XnCD004681 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 16 Feb 2017 04:33:49 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0F6C561BB0; Thu, 16 Feb 2017 09:33:48 +0000 (UTC) Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Feb 2017 01:33:46 -0800 Received: from s2600wt.sh.intel.com ([10.239.48.158]) by fmsmga006.fm.intel.com with ESMTP; 16 Feb 2017 01:33:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.35,168,1484035200"; d="scan'208";a="66473651" From: Eli Qiao To: libvir-list@redhat.com Date: Thu, 16 Feb 2017 17:35:20 +0800 Message-Id: <1487237721-74466-10-git-send-email-liyong.qiao@intel.com> In-Reply-To: <1487237721-74466-1-git-send-email-liyong.qiao@intel.com> References: <1487237721-74466-1-git-send-email-liyong.qiao@intel.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 200 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 16 Feb 2017 09:33:48 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 16 Feb 2017 09:33:48 +0000 (UTC) for IP:'192.55.52.120' DOMAIN:'mga04.intel.com' HELO:'mga04.intel.com' FROM:'liyong.qiao@intel.com' RCPT:'' X-RedHat-Spam-Score: -1.922 (BAYES_50, DCC_REPUT_00_12, RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RP_MATCHES_RCVD, SPF_PASS) 192.55.52.120 mga04.intel.com 192.55.52.120 mga04.intel.com X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Scanned-By: MIMEDefang 2.78 on 10.5.110.39 X-loop: libvir-list@redhat.com Cc: mkletzan@redhat.com, qiaowei.ren@intel.com, mtosatti@redhat.com, qiaoliyong@gmail.com Subject: [libvirt] [v7 09/10] Resctrl: concurrence support X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" The internal struct list domainall is a list which are resctral domain status shared by all VMs, especiall the default domain, each VM should access it concomitantly. Ues a mutex to control it. Each bank's cache_left field is also a global shared resource we need to be care, add a mutex for each bank. We need also to add lock to access /sys/fs/resctrl, use flock. Signed-off-by: Eli Qiao --- src/util/virresctrl.c | 97 ++++++++++++++++++++++++++++++++++++++++++-----= ---- src/util/virresctrl.h | 3 ++ 2 files changed, 83 insertions(+), 17 deletions(-) diff --git a/src/util/virresctrl.c b/src/util/virresctrl.c index 17d95a8..2b1ee41 100644 --- a/src/util/virresctrl.c +++ b/src/util/virresctrl.c @@ -24,6 +24,7 @@ #if defined HAVE_SYS_SYSCALL_H # include #endif +#include #include #include #include @@ -68,8 +69,8 @@ do { \ =20 #define VIR_RESCTRL_GET_SCHEMATA(count) ((1 << count) - 1) =20 -#define VIR_RESCTRL_SET_SCHEMATA(p, type, pos, val) \ - p->schematas[type]->schemata_items[pos] =3D val +#define VIR_RESCTRL_LOCK(fd, op) flock(fd, op) +#define VIR_RESCTRL_UNLOCK(fd) flock(fd, LOCK_UN) =20 /** * a virResSchemata represents a schemata object under a resource control @@ -114,6 +115,8 @@ typedef virResCtrlDomain *virResCtrlDomainPtr; struct _virResCtrlDomain { unsigned int num_domains; virResDomainPtr domains; + + virMutex lock; }; =20 =20 @@ -169,11 +172,16 @@ static int virResCtrlGetStr(const char *domain_name, = const char *item_name, char =20 CONSTRUCT_RESCTRL_PATH(domain_name, item_name); =20 + if (!virFileExists(path)) + goto cleanup; + if (virFileReadAll(path, MAX_FILE_LEN, ret) < 0) { rc =3D -1; goto cleanup; } =20 + rc =3D 0; + cleanup: VIR_FREE(path); return rc; @@ -616,6 +624,9 @@ virResCtrlRefresh(void) unsigned int origin_count =3D domainall.num_domains; virResDomainPtr p, pre, del; pre =3D domainall.domains; + + virMutexLock(&domainall.lock); + p =3D del =3D NULL; if (pre) p =3D pre->next; @@ -640,10 +651,11 @@ virResCtrlRefresh(void) p =3D p->next; } VIR_FREE(tasks); - } =20 - return virResCtrlRefreshSchemata(); + virResCtrlRefreshSchemata(); + virMutexUnlock(&domainall.lock); + return 0; } =20 /* Get a domain ptr by domain's name*/ @@ -873,10 +885,15 @@ static int virResCtrlAppendDomain(virResDomainPtr dom) { virResDomainPtr p =3D domainall.domains; + + virMutexLock(&domainall.lock); + while (p->next !=3D NULL) p =3D p->next; p->next =3D dom; dom->pre =3D p; domainall.num_domains +=3D 1; + + virMutexUnlock(&domainall.lock); return 0; } =20 @@ -899,18 +916,22 @@ virResCtrlCalculateSchemata(int type, { size_t i; int count; + int rc =3D -1; virResDomainPtr p; unsigned int tmp_schemata; unsigned int schemata_sum =3D 0; int pair_type =3D 0; =20 + virMutexLock(&resctrlall[type].cache_banks[sid].lock); + if (resctrlall[type].cache_banks[sid].cache_left < size) { VIR_ERROR(_("Not enough cache left on bank %u"), hostid); - return -1; + goto cleanup; } + if ((count =3D size / resctrlall[type].cache_banks[sid].cache_min) <= =3D 0) { VIR_ERROR(_("Error cache size %llu"), size); - return -1; + goto cleanup; } =20 tmp_schemata =3D VIR_RESCTRL_GET_SCHEMATA(count); @@ -925,7 +946,7 @@ virResCtrlCalculateSchemata(int type, if (type =3D=3D VIR_RDT_RESOURCE_L3CODE) pair_type =3D VIR_RDT_RESOURCE_L3DATA; =20 - for (i =3D 1; i < domainall.num_domains; i ++) { + for (i =3D 1; i < domainall.num_domains; i++) { schemata_sum |=3D p->schematas[type]->schemata_items[sid].schemata; if (pair_type > 0) schemata_sum |=3D p->schematas[pair_type]->schemata_items[sid]= .schemata; @@ -936,7 +957,16 @@ virResCtrlCalculateSchemata(int type, =20 while ((tmp_schemata & schemata_sum) !=3D 0) tmp_schemata =3D tmp_schemata >> 1; - return tmp_schemata; + + resctrlall[type].cache_banks[sid].cache_left -=3D size; + if (pair_type > 0) + resctrlall[pair_type].cache_banks[sid].cache_left =3D resctrlall[t= ype].cache_banks[sid].cache_left; + + rc =3D tmp_schemata; + + cleanup: + virMutexUnlock(&resctrlall[type].cache_banks[sid].lock); + return rc; } =20 int virResCtrlSetCacheBanks(virDomainCachetunePtr cachetune, @@ -949,8 +979,8 @@ int virResCtrlSetCacheBanks(virDomainCachetunePtr cache= tune, int pair_type =3D -1; int sid; int schemata; - - virUUIDFormat(uuid, name); + int lockfd; + int rc =3D -1; =20 for (i =3D 0; i < cachetune->n_banks; i++) { VIR_DEBUG("cache_banks %u, %u, %llu, %s", @@ -963,12 +993,21 @@ int virResCtrlSetCacheBanks(virDomainCachetunePtr cac= hetune, if (cachetune->n_banks < 1) return 0; =20 + virUUIDFormat(uuid, name); + + if ((lockfd =3D open(RESCTRL_DIR, O_RDONLY)) < 0) + goto cleanup; + + if (VIR_RESCTRL_LOCK(lockfd, LOCK_EX) < 0) { + virReportSystemError(errno, _("Unable to lock '%s'"), RESCTRL_DIR); + goto cleanup; + } + p =3D virResCtrlGetDomain(name); if (p =3D=3D NULL) { VIR_DEBUG("no domain name %s found, create new one!", name); p =3D virResCtrlCreateDomain(name); } - if (p !=3D NULL) { =20 virResCtrlAppendDomain(p); @@ -1014,19 +1053,25 @@ int virResCtrlSetCacheBanks(virDomainCachetunePtr c= achetune, if (virResCtrlFlushDomainToSysfs(p) < 0) { VIR_ERROR(_("failed to flush domain %s to sysfs"), name); virResCtrlDestroyDomain(p); - return -1; + goto cleanup; } } else { VIR_ERROR(_("Failed to create a domain in sysfs")); - return -1; + goto cleanup; } =20 virResCtrlRefresh(); /* after refresh, flush header's schemata changes to sys fs */ - if (virResCtrlFlushDomainToSysfs(domainall.domains) < 0) - VIR_WARN("failed to flush domain to sysfs"); + if (virResCtrlFlushDomainToSysfs(domainall.domains) < 0) { + VIR_ERROR(_("failed to flush domain to sysfs")); + goto cleanup; + } =20 - return 0; + rc =3D 0; + + cleanup: + VIR_RESCTRL_UNLOCK(lockfd); + return rc; } =20 /* Should be called after pid disappeared, we recalculate @@ -1048,7 +1093,7 @@ int virResCtrlUpdate(void) int virResCtrlInit(void) { - size_t i =3D 0; + size_t i, j; char *tmp; int rc =3D 0; =20 @@ -1068,6 +1113,24 @@ virResCtrlInit(void) =20 domainall.domains =3D virResCtrlGetAllDomains(&(domainall.num_domains)= ); =20 + for (i =3D 0; i < VIR_RDT_RESOURCE_LAST; i++) { + if (VIR_RESCTRL_ENABLED(i)) { + for (j =3D 0; j < resctrlall[i].num_banks; j++) { + if (virMutexInit(&resctrlall[i].cache_banks[j].lock) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Unable to initialize mutex")); + return -1; + } + } + } + } + + if (virMutexInit(&domainall.lock) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Unable to initialize mutex")); + return -1; + } + if ((rc =3D virResCtrlRefresh()) < 0) VIR_ERROR(_("Failed to refresh resource control")); return rc; diff --git a/src/util/virresctrl.h b/src/util/virresctrl.h index ee7e115..07b298d 100644 --- a/src/util/virresctrl.h +++ b/src/util/virresctrl.h @@ -27,6 +27,7 @@ # include "virutil.h" # include "virbitmap.h" # include "domain_conf.h" +# include "virthread.h" =20 #define MAX_CPU_SOCKET_NUM 8 =20 @@ -49,6 +50,8 @@ struct _virResCacheBank { unsigned long long cache_left; unsigned long long cache_min; virBitmapPtr cpu_mask; + + virMutex lock; }; =20 /** --=20 1.9.1 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list