From nobody Mon Sep 16 19:43:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=quarantine dis=quarantine) header.from=huawei.com ARC-Seal: i=1; a=rsa-sha256; t=1678201685; cv=none; d=zohomail.com; s=zohoarc; b=VhcjqJEmDRwTkdylKsZSaO0A9AdaUMaOXHM9icFe8YFj5GI3jckw+xQ0s4l3IePGWASQmY2UQHJs+M/r+RNfAKw7LJ5QNAvw5VV6BjIlPoorqd0Hp3ytfuQvBrauclyTJfgzArI3zBvzDPr66xBOMTbjQgHI7gE7U7s7tvoPs2Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1678201685; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=pkuU4psuO8dt7rVd9VP6mBBFSgRe4K2ur/w+lJDxt3E=; b=R4l6e48UOliKc+jR42Au6Pk56I1LfvCEcw5R99BqkGFoUVDnEWK2215OtBLf2+182OjwbDLXSi3JttbUUU0rzaWvpXRTRXnJu7BMXT4A+rtEhiIqvff7/ArfK5HV7CYsjslMANz2qFw8IP1QDhFNwGn7K4vf/ccK6wRoBlrpG64= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail header.from= (p=quarantine dis=quarantine) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1678201685366470.94563373865924; Tue, 7 Mar 2023 07:08:05 -0800 (PST) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-345-z5Y7i0pYONSXDXI9lUBn8w-1; Tue, 07 Mar 2023 10:05:29 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 301D03815F64; Tue, 7 Mar 2023 15:03:12 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 176E140C10FA; Tue, 7 Mar 2023 15:03:12 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 84A031946594; Tue, 7 Mar 2023 15:02:53 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 4ECBB19465A3 for ; Tue, 7 Mar 2023 15:02:52 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 42179112132D; Tue, 7 Mar 2023 15:02:52 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3ACD81121314 for ; Tue, 7 Mar 2023 15:02:52 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 171D585CCEC for ; Tue, 7 Mar 2023 15:02:52 +0000 (UTC) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-498-GCXisq6MM6iA9icJRRVzWg-1; Tue, 07 Mar 2023 10:02:37 -0500 Received: from dggpeml500022.china.huawei.com (unknown [172.30.72.54]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4PWJZ06rgZzrRt8; Tue, 7 Mar 2023 23:01:44 +0800 (CST) Received: from localhost.localdomain (10.175.124.27) by dggpeml500022.china.huawei.com (7.185.36.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Tue, 7 Mar 2023 23:02:30 +0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678201684; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=pkuU4psuO8dt7rVd9VP6mBBFSgRe4K2ur/w+lJDxt3E=; b=Ytb4T35b/oRLoDAcx3+pvy58zYyt+FkzBz83f4m45UWMCSACLdZsT6eoVXBxxjoh/wogWn GcrdTDcjcNLFExTWzHdgusDn1PUbgOnwyTVulngqjseRkQEj+wZ07Ba2l+P4pY++Zibr4/ /MAu9fCaBfbvWlR0Trlh0VuO8w746Aw= X-MC-Unique: z5Y7i0pYONSXDXI9lUBn8w-1 X-Original-To: libvir-list@listman.corp.redhat.com X-MC-Unique: GCXisq6MM6iA9icJRRVzWg-1 From: Jiang Jiacheng To: Subject: [PATCH v2 4/8] migration/migration-pin: Implement qemuDomainPinMigrationThread Date: Tue, 7 Mar 2023 22:52:16 +0800 Message-ID: <20230307145220.3767207-5-jiangjiacheng@huawei.com> In-Reply-To: <20230307145220.3767207-1-jiangjiacheng@huawei.com> References: <20230307145220.3767207-1-jiangjiacheng@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.124.27] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To dggpeml500022.china.huawei.com (7.185.36.66) X-CFilter-Loop: Reflected X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yubihong@huawei.com, quintela@redhat.com, linyilu@huawei.com, jiangjiacheng@huawei.com, dgilbert@redhat.com, xiexiangyou@huawei.com, zhengchuan@huawei.com, caozhongwang1@huawei.com, wanghao232@huawei.com Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1678201687366100001 Content-Type: text/plain; charset="utf-8" Add cgroup for migation thread and implement qemuDomainPinMigrationThread to pin migraiton thread to given cpumap. Signed-off-by: zhengchuan Signed-off-by: Jiang Jiacheng --- src/conf/domain_conf.c | 9 ++++++ src/conf/domain_conf.h | 10 +++++++ src/conf/virconftypes.h | 2 ++ src/libvirt_private.syms | 1 + src/qemu/qemu_domain.c | 1 + src/qemu/qemu_domain.h | 1 + src/qemu/qemu_driver.c | 56 ++++++++++++++++++++++++++++++++++++ src/qemu/qemu_process.c | 61 ++++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_process.h | 10 +++++++ src/util/vircgroup.c | 3 ++ src/util/vircgroup.h | 1 + 11 files changed, 155 insertions(+) diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index f7f9ec3c0a..68200dbbab 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -30883,3 +30883,12 @@ virDomainWatchdogDefFind(const virDomainDef *def, =20 return -1; } + +void +virDomainMigrationIDDefFree(virDomainMigrationIDDef *def) +{ + if (!def) + return; + virBitmapFree(def->cpumask); + VIR_FREE(def); +} diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index 21ea4a48b3..fff846f26f 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -4469,3 +4469,13 @@ virDomainObjGetMessages(virDomainObj *vm, =20 bool virDomainDefHasSpiceGraphics(const virDomainDef *def); + +struct _virDomainMigrationIDDef { + bool autofill; + int thread_id; + virBitmap *cpumask; + virDomainThreadSchedParam sched; +}; + +void +virDomainMigrationIDDefFree(virDomainMigrationIDDef *def); diff --git a/src/conf/virconftypes.h b/src/conf/virconftypes.h index e07f967814..84a28a08b8 100644 --- a/src/conf/virconftypes.h +++ b/src/conf/virconftypes.h @@ -261,3 +261,5 @@ typedef struct _virDomainXMLOption virDomainXMLOption; typedef struct _virDomainXMLPrivateDataCallbacks virDomainXMLPrivateDataCa= llbacks; =20 typedef struct _virDomainXenbusControllerOpts virDomainXenbusControllerOpt= s; + +typedef struct _virDomainMigrationIDDef virDomainMigrationIDDef; diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index d5b1b9cb72..b5d9aa62ae 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -525,6 +525,7 @@ virDomainMemoryModelTypeToString; virDomainMemoryRemove; virDomainMemorySourceTypeFromString; virDomainMemorySourceTypeToString; +virDomainMigrationIDDefFree; virDomainMouseModeTypeFromString; virDomainMouseModeTypeToString; virDomainNetAllocateActualDevice; diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index ba8d1055ae..3d497dd5d0 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -1853,6 +1853,7 @@ qemuDomainObjPrivateDataClear(qemuDomainObjPrivate *p= riv) /* remove automatic pinning data */ g_clear_pointer(&priv->autoNodeset, virBitmapFree); g_clear_pointer(&priv->autoCpuset, virBitmapFree); + g_clear_pointer(&priv->pcpumap, virBitmapFree); g_clear_pointer(&priv->pciaddrs, virDomainPCIAddressSetFree); g_clear_pointer(&priv->usbaddrs, virDomainUSBAddressSetFree); g_clear_pointer(&priv->origCPU, virCPUDefFree); diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 42115a20ef..af986b5423 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -171,6 +171,7 @@ struct _qemuDomainObjPrivate { /* Bitmaps below hold data from the auto NUMA feature */ virBitmap *autoNodeset; virBitmap *autoCpuset; + virBitmap *pcpumap; =20 bool signalIOError; /* true if the domain condition should be signalle= d on I/O error */ diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index c546e35953..2b9a952c85 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -20581,6 +20581,61 @@ qemuDomainFDAssociate(virDomainPtr domain, return ret; } =20 +static int +qemuDomainPinMigrationThread(virDomainPtr dom, + unsigned char *cpumap, + int maplen) +{ + int ret =3D -1; + virQEMUDriver *driver =3D dom->conn->privateData; + g_autoptr(virQEMUDriverConfig) cfg =3D NULL; + virDomainObj *vm =3D NULL; + g_autoptr(virBitmap) pcpumap =3D NULL; + qemuDomainObjPrivate *priv =3D NULL; + qemuDomainMigThreadInfo **migthreads =3D NULL; + size_t i; + + cfg =3D virQEMUDriverGetConfig(driver); + + if (!(vm =3D qemuDomainObjFromDomain(dom))) + goto cleanup; + + priv =3D vm->privateData; + + if (virDomainPinMigrationThreadEnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (virDomainObjBeginJob(vm, VIR_JOB_QUERY) < 0) + goto cleanup; + + if (!(pcpumap =3D virBitmapNewData(cpumap, maplen))) + goto endjob; + + if (virBitmapIsAllClear(pcpumap)) { + virReportError(VIR_ERR_INVALID_ARG, "%s", + _("Empty migration thread cpumap list for pinning")= ); + goto endjob; + } + + virBitmapFree(priv->pcpumap); + priv->pcpumap =3D virBitmapNewCopy(pcpumap); + migthreads =3D priv->migThreadsInfo; + + if (migthreads && pcpumap) { + for (i =3D 0; i < priv->migThreadCount; i++) { + qemuProcessSetMigThreadAffinity(priv, vm, migthreads[i]->threa= d_id, pcpumap); + } + } + + ret =3D 0; + + endjob: + virDomainObjEndJob(vm); + + cleanup: + virDomainObjEndAPI(&vm); + return ret; +} =20 static virHypervisorDriver qemuHypervisorDriver =3D { .name =3D QEMU_DRIVER_NAME, @@ -20831,6 +20886,7 @@ static virHypervisorDriver qemuHypervisorDriver =3D= { .domainStartDirtyRateCalc =3D qemuDomainStartDirtyRateCalc, /* 7.2.0 */ .domainSetLaunchSecurityState =3D qemuDomainSetLaunchSecurityState, /*= 8.0.0 */ .domainFDAssociate =3D qemuDomainFDAssociate, /* 9.0.0 */ + .domainPinMigrationThread =3D qemuDomainPinMigrationThread, /* 9.1.0 */ }; =20 =20 diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 9a612ca443..946aeb81b1 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -9547,3 +9547,64 @@ qemuProcessQMPStart(qemuProcessQMP *proc) =20 return 0; } + +int +qemuProcessSetupMigration(virDomainObj *vm, + virDomainMigrationIDDef *migration) +{ + return qemuProcessSetupPid(vm, migration->thread_id, + VIR_CGROUP_THREAD_MIGRATION_THREAD, + 0, + vm->def->cputune.emulatorpin, + vm->def->cputune.emulator_period, + vm->def->cputune.emulator_quota, + &migration->sched); +} + +/* + * In order to set migration thread affinity when vm is migrating, + * we should create the cgroup for migration thread. + */ +void +qemuProcessSetMigThreadAffinity(qemuDomainObjPrivate *priv, + virDomainObj *vm, + int mpid, + virBitmap *pcpumap) +{ + virDomainMigrationIDDef *migration =3D NULL; + int migration_id =3D 0; + virCgroup *cgroup_migthread =3D NULL; + + if (!pcpumap) + return; + + migration =3D g_new0(virDomainMigrationIDDef, 1); + migration->thread_id =3D mpid; + if (qemuProcessSetupMigration(vm, migration) < 0) { + VIR_ERROR(_("fail to setup migration cgroup")); + goto cleanup; + } + + if (virCgroupHasController(priv->cgroup, + VIR_CGROUP_CONTROLLER_CPUSET)) { + if (virCgroupNewThread(priv->cgroup, VIR_CGROUP_THREAD_MIGRATION_T= HREAD, + migration_id, false, &cgroup_migthread) < 0) + goto cleanup; + + if (virDomainCgroupSetupCpusetCpus(cgroup_migthread, pcpumap) < 0)= { + virReportError(VIR_ERR_OPERATION_INVALID, + _("failed to set cpuset.cpus in cgroup" + " for migration%d thread"), migration_id); + goto cleanup; + } + } + + if (virProcessSetAffinity(mpid, pcpumap, false) < 0) + VIR_WARN("failed to set affinity in migration"); + + cleanup: + if (cgroup_migthread) + virCgroupFree(cgroup_migthread); + virDomainMigrationIDDefFree(migration); + return; +} diff --git a/src/qemu/qemu_process.h b/src/qemu/qemu_process.h index b171f0464c..2af1e2ee87 100644 --- a/src/qemu/qemu_process.h +++ b/src/qemu/qemu_process.h @@ -235,3 +235,13 @@ bool qemuProcessRebootAllowed(const virDomainDef *def); =20 void qemuProcessCleanupMigrationJob(virQEMUDriver *driver, virDomainObj *vm); + +int +qemuProcessSetupMigration(virDomainObj *vm, + virDomainMigrationIDDef *migration); + +void +qemuProcessSetMigThreadAffinity(qemuDomainObjPrivate *priv, + virDomainObj *vm, + int mpid, + virBitmap *pcpumap); diff --git a/src/util/vircgroup.c b/src/util/vircgroup.c index 73675b4478..d4307c0d72 100644 --- a/src/util/vircgroup.c +++ b/src/util/vircgroup.c @@ -1123,6 +1123,9 @@ virCgroupNewThread(virCgroup *domain, case VIR_CGROUP_THREAD_IOTHREAD: name =3D g_strdup_printf("iothread%d", id); break; + case VIR_CGROUP_THREAD_MIGRATION_THREAD: + name =3D g_strdup_printf("migthread%d", id); + break; case VIR_CGROUP_THREAD_LAST: virReportError(VIR_ERR_INTERNAL_ERROR, _("unexpected name value %d"), nameval); diff --git a/src/util/vircgroup.h b/src/util/vircgroup.h index adf3850b22..c5f6ddd7a6 100644 --- a/src/util/vircgroup.h +++ b/src/util/vircgroup.h @@ -53,6 +53,7 @@ typedef enum { VIR_CGROUP_THREAD_VCPU =3D 0, VIR_CGROUP_THREAD_EMULATOR, VIR_CGROUP_THREAD_IOTHREAD, + VIR_CGROUP_THREAD_MIGRATION_THREAD, =20 VIR_CGROUP_THREAD_LAST } virCgroupThreadName; --=20 2.33.0