From nobody Sun May 12 09:25:32 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 162695755125887.59979177172329; Thu, 22 Jul 2021 05:39:11 -0700 (PDT) Received: from localhost ([::1]:43944 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6Xyo-000623-0m for importer@patchew.org; Thu, 22 Jul 2021 08:39:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwg-000398-UC for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:00 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22964) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwd-0004Th-6j for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:36:57 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-537-JL6u_c7PNtKTjsZsRGQHqA-1; Thu, 22 Jul 2021 08:36:53 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5CE941074667; Thu, 22 Jul 2021 12:36:52 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-116.ams2.redhat.com [10.36.112.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 64F9817A70; Thu, 22 Jul 2021 12:36:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626957414; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VaBCFUm5u8ChbQFJ9QE9XCxxc/MfcJcOIinlHfTKUUg=; b=Oqk2LBCgLRW1H9O9Nz2XBTO+dytwaod64FErcTKy8gN2zsoBF5NqURksSskpJYnlmEALjg GUKSfeP92PUr+KEzM9BlPkvEeGYiGmknmBIkDigZKSS9BY/etnQczG0tdSKiCmCu43Uewi f7Jbp/4Uadh8ZOhkuLsqchEwxwY39Gw= X-MC-Unique: JL6u_c7PNtKTjsZsRGQHqA-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 1/6] util/oslib-posix: Support MADV_POPULATE_WRITE for os_mem_prealloc() Date: Thu, 22 Jul 2021 14:36:30 +0200 Message-Id: <20210722123635.60608-2-david@redhat.com> In-Reply-To: <20210722123635.60608-1-david@redhat.com> References: <20210722123635.60608-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pankaj Gupta , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eduardo Habkost , "Michael S. Tsirkin" , David Hildenbrand , "Dr . David Alan Gilbert" , Pankaj Gupta , Igor Mammedov , Paolo Bonzini , Marek Kedzierski Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626957552983100001 Content-Type: text/plain; charset="utf-8" Let's sense support and use it for preallocation. MADV_POPULATE_WRITE does not require a SIGBUS handler, doesn't actually touch page content, and avoids context switches; it is, therefore, faster and easier to handle than our current approach. While MADV_POPULATE_WRITE is, in general, faster than manual prefaulting, and especially faster with 4k pages, there is still value in prefaulting using multiple threads to speed up preallocation. More details on MADV_POPULATE_WRITE can be found in the Linux commit 4ca9b3859dac ("mm/madvise: introduce MADV_POPULATE_(READ|WRITE) to prefault page tables") and in the man page proposal [1]. [1] https://lkml.kernel.org/r/20210712083917.16361-1-david@redhat.com This resolves the TODO in do_touch_pages(). In the future, we might want to look into using fallocate(), eventually combined with MADV_POPULATE_READ, when dealing with shared file mappings. Reviewed-by: Pankaj Gupta Signed-off-by: David Hildenbrand --- include/qemu/osdep.h | 7 ++++ util/oslib-posix.c | 88 +++++++++++++++++++++++++++++++++----------- 2 files changed, 74 insertions(+), 21 deletions(-) diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h index 60718fc342..d1660d67fa 100644 --- a/include/qemu/osdep.h +++ b/include/qemu/osdep.h @@ -471,6 +471,11 @@ static inline void qemu_cleanup_generic_vfree(void *p) #else #define QEMU_MADV_REMOVE QEMU_MADV_DONTNEED #endif +#ifdef MADV_POPULATE_WRITE +#define QEMU_MADV_POPULATE_WRITE MADV_POPULATE_WRITE +#else +#define QEMU_MADV_POPULATE_WRITE QEMU_MADV_INVALID +#endif =20 #elif defined(CONFIG_POSIX_MADVISE) =20 @@ -484,6 +489,7 @@ static inline void qemu_cleanup_generic_vfree(void *p) #define QEMU_MADV_HUGEPAGE QEMU_MADV_INVALID #define QEMU_MADV_NOHUGEPAGE QEMU_MADV_INVALID #define QEMU_MADV_REMOVE QEMU_MADV_DONTNEED +#define QEMU_MADV_POPULATE_WRITE QEMU_MADV_INVALID =20 #else /* no-op */ =20 @@ -497,6 +503,7 @@ static inline void qemu_cleanup_generic_vfree(void *p) #define QEMU_MADV_HUGEPAGE QEMU_MADV_INVALID #define QEMU_MADV_NOHUGEPAGE QEMU_MADV_INVALID #define QEMU_MADV_REMOVE QEMU_MADV_INVALID +#define QEMU_MADV_POPULATE_WRITE QEMU_MADV_INVALID =20 #endif =20 diff --git a/util/oslib-posix.c b/util/oslib-posix.c index e8bdb02e1d..1cb80bf94c 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -484,10 +484,6 @@ static void *do_touch_pages(void *arg) * * 'volatile' to stop compiler optimizing this away * to a no-op - * - * TODO: get a better solution from kernel so we - * don't need to write at all so we don't cause - * wear on the storage backing the region... */ *(volatile char *)addr =3D *addr; addr +=3D hpagesize; @@ -497,6 +493,31 @@ static void *do_touch_pages(void *arg) return NULL; } =20 +static void *do_madv_populate_write_pages(void *arg) +{ + MemsetThread *memset_args =3D (MemsetThread *)arg; + const size_t size =3D memset_args->numpages * memset_args->hpagesize; + char * const addr =3D memset_args->addr; + int ret; + + if (!size) { + return NULL; + } + + /* See do_touch_pages(). */ + qemu_mutex_lock(&page_mutex); + while (!threads_created_flag) { + qemu_cond_wait(&page_cond, &page_mutex); + } + qemu_mutex_unlock(&page_mutex); + + ret =3D qemu_madvise(addr, size, QEMU_MADV_POPULATE_WRITE); + if (ret) { + memset_thread_failed =3D true; + } + return NULL; +} + static inline int get_memset_num_threads(int smp_cpus) { long host_procs =3D sysconf(_SC_NPROCESSORS_ONLN); @@ -510,10 +531,11 @@ static inline int get_memset_num_threads(int smp_cpus) } =20 static bool touch_all_pages(char *area, size_t hpagesize, size_t numpages, - int smp_cpus) + int smp_cpus, bool use_madv_populate_write) { static gsize initialized =3D 0; size_t numpages_per_thread, leftover; + void *(*touch_fn)(void *); char *addr =3D area; int i =3D 0; =20 @@ -523,6 +545,12 @@ static bool touch_all_pages(char *area, size_t hpagesi= ze, size_t numpages, g_once_init_leave(&initialized, 1); } =20 + if (use_madv_populate_write) { + touch_fn =3D do_madv_populate_write_pages; + } else { + touch_fn =3D do_touch_pages; + } + memset_thread_failed =3D false; threads_created_flag =3D false; memset_num_threads =3D get_memset_num_threads(smp_cpus); @@ -534,7 +562,7 @@ static bool touch_all_pages(char *area, size_t hpagesiz= e, size_t numpages, memset_thread[i].numpages =3D numpages_per_thread + (i < leftover); memset_thread[i].hpagesize =3D hpagesize; qemu_thread_create(&memset_thread[i].pgthread, "touch_pages", - do_touch_pages, &memset_thread[i], + touch_fn, &memset_thread[i], QEMU_THREAD_JOINABLE); addr +=3D memset_thread[i].numpages * hpagesize; } @@ -553,6 +581,12 @@ static bool touch_all_pages(char *area, size_t hpagesi= ze, size_t numpages, return memset_thread_failed; } =20 +static bool madv_populate_write_possible(char *area, size_t pagesize) +{ + return !qemu_madvise(area, pagesize, QEMU_MADV_POPULATE_WRITE) || + errno !=3D EINVAL; +} + void os_mem_prealloc(int fd, char *area, size_t memory, int smp_cpus, Error **errp) { @@ -560,29 +594,41 @@ void os_mem_prealloc(int fd, char *area, size_t memor= y, int smp_cpus, struct sigaction act, oldact; size_t hpagesize =3D qemu_fd_getpagesize(fd); size_t numpages =3D DIV_ROUND_UP(memory, hpagesize); + bool use_madv_populate_write; =20 - memset(&act, 0, sizeof(act)); - act.sa_handler =3D &sigbus_handler; - act.sa_flags =3D 0; - - ret =3D sigaction(SIGBUS, &act, &oldact); - if (ret) { - error_setg_errno(errp, errno, - "os_mem_prealloc: failed to install signal handler"); - return; + /* + * Sense on every invocation, as MADV_POPULATE_WRITE cannot be used for + * some special mappings, such as mapping /dev/mem. + */ + use_madv_populate_write =3D madv_populate_write_possible(area, hpagesi= ze); + + if (!use_madv_populate_write) { + memset(&act, 0, sizeof(act)); + act.sa_handler =3D &sigbus_handler; + act.sa_flags =3D 0; + + ret =3D sigaction(SIGBUS, &act, &oldact); + if (ret) { + error_setg_errno(errp, errno, + "os_mem_prealloc: failed to install signal handler"); + return; + } } =20 /* touch pages simultaneously */ - if (touch_all_pages(area, hpagesize, numpages, smp_cpus)) { + if (touch_all_pages(area, hpagesize, numpages, smp_cpus, + use_madv_populate_write)) { error_setg(errp, "os_mem_prealloc: Insufficient free host memory " "pages available to allocate guest RAM"); } =20 - ret =3D sigaction(SIGBUS, &oldact, NULL); - if (ret) { - /* Terminate QEMU since it can't recover from error */ - perror("os_mem_prealloc: failed to reinstall signal handler"); - exit(1); + if (!use_madv_populate_write) { + ret =3D sigaction(SIGBUS, &oldact, NULL); + if (ret) { + /* Terminate QEMU since it can't recover from error */ + perror("os_mem_prealloc: failed to reinstall signal handler"); + exit(1); + } } } =20 --=20 2.31.1 From nobody Sun May 12 09:25:32 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1626957560804535.6365933800686; Thu, 22 Jul 2021 05:39:20 -0700 (PDT) Received: from localhost ([::1]:44848 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6Xyx-0006cm-Da for importer@patchew.org; Thu, 22 Jul 2021 08:39:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51030) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwj-0003BC-35 for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:21172) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwg-0004VK-GR for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:00 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-460-jDWcma8nOxOZ3UxGsdu_wA-1; Thu, 22 Jul 2021 08:36:55 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E8953802C80; Thu, 22 Jul 2021 12:36:54 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-116.ams2.redhat.com [10.36.112.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA1D517A63; Thu, 22 Jul 2021 12:36:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626957417; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+qAU83THdgeQguJ8G6XvnPUlZLqvsJzMQQl3IMu6I6I=; b=IBATPEK4btSiAOdEP0eLPqdB4u+wmTU6YIbwhYO8qsWaufMUJdn8zouqu4FQQsdIRUVK+m 4KTVaIsEdckhcsQjLOwG7c/J7i/avS52TiJQYu3E1FnQy2BCgpfDIXsXFv1C+DlATv/KVb UTV2O90tUAz/+5ohvTja4gld8maCR90= X-MC-Unique: jDWcma8nOxOZ3UxGsdu_wA-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 2/6] util/oslib-posix: Introduce and use MemsetContext for touch_all_pages() Date: Thu, 22 Jul 2021 14:36:31 +0200 Message-Id: <20210722123635.60608-3-david@redhat.com> In-Reply-To: <20210722123635.60608-1-david@redhat.com> References: <20210722123635.60608-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pankaj Gupta , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eduardo Habkost , "Michael S. Tsirkin" , David Hildenbrand , "Dr . David Alan Gilbert" , Igor Mammedov , Paolo Bonzini , Marek Kedzierski Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626957562049100001 Let's minimize the number of global variables to prepare for os_mem_prealloc() getting called concurrently and make the code a bit easier to read. The only consumer that really needs a global variable is the sigbus handler, which will require protection via a mutex in the future either way as we cannot concurrently mess with the SIGBUS handler. Reviewed-by: Daniel P. Berrang=C3=A9 Signed-off-by: David Hildenbrand --- util/oslib-posix.c | 81 ++++++++++++++++++++++++++++------------------ 1 file changed, 50 insertions(+), 31 deletions(-) diff --git a/util/oslib-posix.c b/util/oslib-posix.c index 1cb80bf94c..2ae6c3aaab 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -73,22 +73,30 @@ =20 #define MAX_MEM_PREALLOC_THREAD_COUNT 16 =20 +struct MemsetThread; + +typedef struct MemsetContext { + bool all_threads_created; + bool any_thread_failed; + struct MemsetThread *threads; + int num_threads; +} MemsetContext; + struct MemsetThread { char *addr; size_t numpages; size_t hpagesize; QemuThread pgthread; sigjmp_buf env; + MemsetContext *context; }; typedef struct MemsetThread MemsetThread; =20 -static MemsetThread *memset_thread; -static int memset_num_threads; -static bool memset_thread_failed; +/* used by sigbus_handler() */ +static MemsetContext *sigbus_memset_context; =20 static QemuMutex page_mutex; static QemuCond page_cond; -static bool threads_created_flag; =20 int qemu_get_thread_id(void) { @@ -439,10 +447,13 @@ const char *qemu_get_exec_dir(void) static void sigbus_handler(int signal) { int i; - if (memset_thread) { - for (i =3D 0; i < memset_num_threads; i++) { - if (qemu_thread_is_self(&memset_thread[i].pgthread)) { - siglongjmp(memset_thread[i].env, 1); + + if (sigbus_memset_context) { + for (i =3D 0; i < sigbus_memset_context->num_threads; i++) { + MemsetThread *thread =3D &sigbus_memset_context->threads[i]; + + if (qemu_thread_is_self(&thread->pgthread)) { + siglongjmp(thread->env, 1); } } } @@ -459,7 +470,7 @@ static void *do_touch_pages(void *arg) * clearing until all threads have been created. */ qemu_mutex_lock(&page_mutex); - while(!threads_created_flag){ + while (!memset_args->context->all_threads_created) { qemu_cond_wait(&page_cond, &page_mutex); } qemu_mutex_unlock(&page_mutex); @@ -470,7 +481,7 @@ static void *do_touch_pages(void *arg) pthread_sigmask(SIG_UNBLOCK, &set, &oldset); =20 if (sigsetjmp(memset_args->env, 1)) { - memset_thread_failed =3D true; + memset_args->context->any_thread_failed =3D true; } else { char *addr =3D memset_args->addr; size_t numpages =3D memset_args->numpages; @@ -506,14 +517,14 @@ static void *do_madv_populate_write_pages(void *arg) =20 /* See do_touch_pages(). */ qemu_mutex_lock(&page_mutex); - while (!threads_created_flag) { + while (!memset_args->context->all_threads_created) { qemu_cond_wait(&page_cond, &page_mutex); } qemu_mutex_unlock(&page_mutex); =20 ret =3D qemu_madvise(addr, size, QEMU_MADV_POPULATE_WRITE); if (ret) { - memset_thread_failed =3D true; + memset_args->context->any_thread_failed =3D true; } return NULL; } @@ -534,6 +545,9 @@ static bool touch_all_pages(char *area, size_t hpagesiz= e, size_t numpages, int smp_cpus, bool use_madv_populate_write) { static gsize initialized =3D 0; + MemsetContext context =3D { + .num_threads =3D get_memset_num_threads(smp_cpus), + }; size_t numpages_per_thread, leftover; void *(*touch_fn)(void *); char *addr =3D area; @@ -551,34 +565,39 @@ static bool touch_all_pages(char *area, size_t hpages= ize, size_t numpages, touch_fn =3D do_touch_pages; } =20 - memset_thread_failed =3D false; - threads_created_flag =3D false; - memset_num_threads =3D get_memset_num_threads(smp_cpus); - memset_thread =3D g_new0(MemsetThread, memset_num_threads); - numpages_per_thread =3D numpages / memset_num_threads; - leftover =3D numpages % memset_num_threads; - for (i =3D 0; i < memset_num_threads; i++) { - memset_thread[i].addr =3D addr; - memset_thread[i].numpages =3D numpages_per_thread + (i < leftover); - memset_thread[i].hpagesize =3D hpagesize; - qemu_thread_create(&memset_thread[i].pgthread, "touch_pages", - touch_fn, &memset_thread[i], + context.threads =3D g_new0(MemsetThread, context.num_threads); + numpages_per_thread =3D numpages / context.num_threads; + leftover =3D numpages % context.num_threads; + for (i =3D 0; i < context.num_threads; i++) { + context.threads[i].addr =3D addr; + context.threads[i].numpages =3D numpages_per_thread + (i < leftove= r); + context.threads[i].hpagesize =3D hpagesize; + context.threads[i].context =3D &context; + qemu_thread_create(&context.threads[i].pgthread, "touch_pages", + touch_fn, &context.threads[i], QEMU_THREAD_JOINABLE); - addr +=3D memset_thread[i].numpages * hpagesize; + addr +=3D context.threads[i].numpages * hpagesize; + } + + if (!use_madv_populate_write) { + sigbus_memset_context =3D &context; } =20 qemu_mutex_lock(&page_mutex); - threads_created_flag =3D true; + context.all_threads_created =3D true; qemu_cond_broadcast(&page_cond); qemu_mutex_unlock(&page_mutex); =20 - for (i =3D 0; i < memset_num_threads; i++) { - qemu_thread_join(&memset_thread[i].pgthread); + for (i =3D 0; i < context.num_threads; i++) { + qemu_thread_join(&context.threads[i].pgthread); + } + + if (!use_madv_populate_write) { + sigbus_memset_context =3D NULL; } - g_free(memset_thread); - memset_thread =3D NULL; + g_free(context.threads); =20 - return memset_thread_failed; + return context.any_thread_failed; } =20 static bool madv_populate_write_possible(char *area, size_t pagesize) --=20 2.31.1 From nobody Sun May 12 09:25:32 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 162695776086091.2297820057953; Thu, 22 Jul 2021 05:42:40 -0700 (PDT) Received: from localhost ([::1]:52014 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6Y2B-0003Ek-NA for importer@patchew.org; Thu, 22 Jul 2021 08:42:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51056) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwl-0003Jn-P4 for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:47175) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwj-0004Xx-64 for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:03 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-494-18RXM16hM1esfxJufUOkXg-1; Thu, 22 Jul 2021 08:36:58 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C725B8B0604; Thu, 22 Jul 2021 12:36:57 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-116.ams2.redhat.com [10.36.112.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4EEFF17A63; Thu, 22 Jul 2021 12:36:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626957419; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/FitQKoUeWPrW3Oa0Z7olkyu6wdTP4ncaP1qRsOB7Kw=; b=J9QZYBjdAgY0Emu4v/6VhHfrkuBG6NtXsLUfkRtmZ93KWg7cQrCo+MrUoiWJLseaVxlKAo VmdWoucSRRyGa2YsfjG9ljz/M8JositxidEW1+yH5bS9d6OLRCZhQRatdtjZIvrMsdFdK/ VuNhDs7fY+Ifu2Z4LZp4J3aEQVTOwcc= X-MC-Unique: 18RXM16hM1esfxJufUOkXg-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 3/6] util/oslib-posix: Don't create too many threads with small memory or little pages Date: Thu, 22 Jul 2021 14:36:32 +0200 Message-Id: <20210722123635.60608-4-david@redhat.com> In-Reply-To: <20210722123635.60608-1-david@redhat.com> References: <20210722123635.60608-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pankaj Gupta , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eduardo Habkost , "Michael S. Tsirkin" , David Hildenbrand , "Dr . David Alan Gilbert" , Igor Mammedov , Paolo Bonzini , Marek Kedzierski Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626957761827100001 Content-Type: text/plain; charset="utf-8" Let's limit the number of threads to something sane, especially that - We don't have more threads than the number of pages we have - We don't have threads that initialize small (< 64 MiB) memory Signed-off-by: David Hildenbrand Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Pankaj Gupta --- util/oslib-posix.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/util/oslib-posix.c b/util/oslib-posix.c index 2ae6c3aaab..a1d309d495 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -40,6 +40,7 @@ #include #include "qemu/cutils.h" #include "qemu/compiler.h" +#include "qemu/units.h" =20 #ifdef CONFIG_LINUX #include @@ -529,7 +530,8 @@ static void *do_madv_populate_write_pages(void *arg) return NULL; } =20 -static inline int get_memset_num_threads(int smp_cpus) +static inline int get_memset_num_threads(size_t hpagesize, size_t numpages, + int smp_cpus) { long host_procs =3D sysconf(_SC_NPROCESSORS_ONLN); int ret =3D 1; @@ -537,6 +539,12 @@ static inline int get_memset_num_threads(int smp_cpus) if (host_procs > 0) { ret =3D MIN(MIN(host_procs, MAX_MEM_PREALLOC_THREAD_COUNT), smp_cp= us); } + + /* Especially with gigantic pages, don't create more threads than page= s. */ + ret =3D MIN(ret, numpages); + /* Don't start threads to prealloc comparatively little memory. */ + ret =3D MIN(ret, MAX(1, hpagesize * numpages / (64 * MiB))); + /* In case sysconf() fails, we fall back to single threaded */ return ret; } @@ -546,7 +554,7 @@ static bool touch_all_pages(char *area, size_t hpagesiz= e, size_t numpages, { static gsize initialized =3D 0; MemsetContext context =3D { - .num_threads =3D get_memset_num_threads(smp_cpus), + .num_threads =3D get_memset_num_threads(hpagesize, numpages, smp_c= pus), }; size_t numpages_per_thread, leftover; void *(*touch_fn)(void *); --=20 2.31.1 From nobody Sun May 12 09:25:32 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1626958008755377.9774753552832; Thu, 22 Jul 2021 05:46:48 -0700 (PDT) Received: from localhost ([::1]:59858 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6Y6A-0000On-Po for importer@patchew.org; Thu, 22 Jul 2021 08:46:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51090) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwu-0003Yz-EX for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:59294) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xws-0004cJ-Ew for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:12 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-172-UOY-q3kBNpyF389Z1BS-cg-1; Thu, 22 Jul 2021 08:37:08 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D1979107ACF5; Thu, 22 Jul 2021 12:37:07 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-116.ams2.redhat.com [10.36.112.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2ECB817A63; Thu, 22 Jul 2021 12:36:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626957429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/q54+MYPayn2IfoKvbOy90CyQUuFc7XRv2g4VY2wxac=; b=HfMzNYN583UHuujhAzYKuM88QK08YQPyF2wV37Gwfb7lKC+G+9NPLDLWRYKbdKtvXqQTd6 M0LzstINYgYu6GonSR85WUBKsd6dG3ck3LPmJIJWxDPtK/D/ZLa0yfdycmu3ppkAyCHgiI BVfv8aag0nWAEDOuf/LY2BaRpnnv7Jk= X-MC-Unique: UOY-q3kBNpyF389Z1BS-cg-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 4/6] util/oslib-posix: Avoid creating a single thread with MADV_POPULATE_WRITE Date: Thu, 22 Jul 2021 14:36:33 +0200 Message-Id: <20210722123635.60608-5-david@redhat.com> In-Reply-To: <20210722123635.60608-1-david@redhat.com> References: <20210722123635.60608-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pankaj Gupta , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eduardo Habkost , "Michael S. Tsirkin" , David Hildenbrand , "Dr . David Alan Gilbert" , Igor Mammedov , Paolo Bonzini , Marek Kedzierski Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626958009844100001 Content-Type: text/plain; charset="utf-8" Let's simplify the case when we only want a single thread and don't have to mess with signal handlers. Signed-off-by: David Hildenbrand Reviewed-by: Pankaj Gupta --- util/oslib-posix.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/util/oslib-posix.c b/util/oslib-posix.c index a1d309d495..1483e985c6 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -568,6 +568,14 @@ static bool touch_all_pages(char *area, size_t hpagesi= ze, size_t numpages, } =20 if (use_madv_populate_write) { + /* Avoid creating a single thread for MADV_POPULATE_WRITE */ + if (context.num_threads =3D=3D 1) { + if (qemu_madvise(area, hpagesize * numpages, + QEMU_MADV_POPULATE_WRITE)) { + return true; + } + return false; + } touch_fn =3D do_madv_populate_write_pages; } else { touch_fn =3D do_touch_pages; --=20 2.31.1 From nobody Sun May 12 09:25:32 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1626957739091425.01687723650184; Thu, 22 Jul 2021 05:42:19 -0700 (PDT) Received: from localhost ([::1]:51122 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6Y1p-0002d1-SE for importer@patchew.org; Thu, 22 Jul 2021 08:42:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51120) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwy-0003hb-Jm for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:24658) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwv-0004dt-N8 for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:16 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-144-OilhnhvpNeCPz3Eu2RJAIQ-1; Thu, 22 Jul 2021 08:37:11 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DF61E107BEF5; Thu, 22 Jul 2021 12:37:10 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-116.ams2.redhat.com [10.36.112.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 39D5517A63; Thu, 22 Jul 2021 12:37:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626957433; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YX7hNU89BW6PbK95O7NPslg9GylgQ2hW2BI3KSopRQA=; b=jCu5Dx1YDmIoT17nYyhnJAkUsYkj5yv0pNnm0FfUGoF1zl9sxs2mmk4io4L3b7ud9BmnbM gDg1xI4Xmkl3dMiJGDbTPe3mgUD0+JN1d0v5DmXfMiyNmJ8+eDBLymLBeNc7vr2Vm+yMVv fW4E2ePaM/C+DbbkP+gu8PeWbhHX56I= X-MC-Unique: OilhnhvpNeCPz3Eu2RJAIQ-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 5/6] util/oslib-posix: Support concurrent os_mem_prealloc() invocation Date: Thu, 22 Jul 2021 14:36:34 +0200 Message-Id: <20210722123635.60608-6-david@redhat.com> In-Reply-To: <20210722123635.60608-1-david@redhat.com> References: <20210722123635.60608-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pankaj Gupta , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eduardo Habkost , "Michael S. Tsirkin" , David Hildenbrand , "Dr . David Alan Gilbert" , Pankaj Gupta , Igor Mammedov , Paolo Bonzini , Marek Kedzierski Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626957741548100001 Content-Type: text/plain; charset="utf-8" Add a mutex to protect the SIGBUS case, as we cannot mess concurrently with the sigbus handler and we have to manage the global variable sigbus_memset_context. The MADV_POPULATE_WRITE path can run concurrently. Note that page_mutex and page_cond are shared between concurrent invocations, which shouldn't be a problem. This is a preparation for future virtio-mem prealloc code, which will call os_mem_prealloc() asynchronously from an iothread when handling guest requests. Add a comment that messing with the SIGBUS handler is frowned upon and can result in problems we fortunately haven't seen so far. Note that forwarding signals to the already installed SIGBUS handler isn't clean either, as that one might modify the SIGBUS handler again. Reviewed-by: Pankaj Gupta Signed-off-by: David Hildenbrand --- util/oslib-posix.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/util/oslib-posix.c b/util/oslib-posix.c index 1483e985c6..7c75848a67 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -95,6 +95,7 @@ typedef struct MemsetThread MemsetThread; =20 /* used by sigbus_handler() */ static MemsetContext *sigbus_memset_context; +static QemuMutex sigbus_mutex; =20 static QemuMutex page_mutex; static QemuCond page_cond; @@ -625,6 +626,7 @@ static bool madv_populate_write_possible(char *area, si= ze_t pagesize) void os_mem_prealloc(int fd, char *area, size_t memory, int smp_cpus, Error **errp) { + static gsize initialized; int ret; struct sigaction act, oldact; size_t hpagesize =3D qemu_fd_getpagesize(fd); @@ -638,6 +640,12 @@ void os_mem_prealloc(int fd, char *area, size_t memory= , int smp_cpus, use_madv_populate_write =3D madv_populate_write_possible(area, hpagesi= ze); =20 if (!use_madv_populate_write) { + if (g_once_init_enter(&initialized)) { + qemu_mutex_init(&sigbus_mutex); + g_once_init_leave(&initialized, 1); + } + + qemu_mutex_lock(&sigbus_mutex); memset(&act, 0, sizeof(act)); act.sa_handler =3D &sigbus_handler; act.sa_flags =3D 0; @@ -664,6 +672,7 @@ void os_mem_prealloc(int fd, char *area, size_t memory,= int smp_cpus, perror("os_mem_prealloc: failed to reinstall signal handler"); exit(1); } + qemu_mutex_unlock(&sigbus_mutex); } } =20 --=20 2.31.1 From nobody Sun May 12 09:25:32 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1626958170089154.03415025632899; Thu, 22 Jul 2021 05:49:30 -0700 (PDT) Received: from localhost ([::1]:34686 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6Y8m-0002bS-7k for importer@patchew.org; Thu, 22 Jul 2021 08:49:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51144) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xx2-0003no-HR for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:37761) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6Xwz-0004ex-DC for qemu-devel@nongnu.org; Thu, 22 Jul 2021 08:37:20 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-253-kY8OPyyONp2_Npt34xI9KQ-1; Thu, 22 Jul 2021 08:37:14 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B3D271812FC4; Thu, 22 Jul 2021 12:37:13 +0000 (UTC) Received: from t480s.redhat.com (ovpn-112-116.ams2.redhat.com [10.36.112.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 495EC17A63; Thu, 22 Jul 2021 12:37:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626957435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xQiZ38bKR7rys56efAQ/EowX8/jrobhq8WUOFUUEszI=; b=RfrSlZgYlqfyaiahAJQbua/qWwMdd7pcOlh6VQOmF5jJY3dM0gSiO60jHop/93UeKpnOnd t5rMEFwP2szCJ8Lhaowb2wsXNddwDf4w3o1Oy3Wombt315Lp7uslUZe5HkVS8/oQK26DcA 8rxWWexR4ueBGBoqink+MNoV8UzwH38= X-MC-Unique: kY8OPyyONp2_Npt34xI9KQ-1 From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v2 6/6] util/oslib-posix: Forward SIGBUS to MCE handler under Linux Date: Thu, 22 Jul 2021 14:36:35 +0200 Message-Id: <20210722123635.60608-7-david@redhat.com> In-Reply-To: <20210722123635.60608-1-david@redhat.com> References: <20210722123635.60608-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.472, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pankaj Gupta , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Eduardo Habkost , "Michael S. Tsirkin" , David Hildenbrand , "Dr . David Alan Gilbert" , Igor Mammedov , Paolo Bonzini , Marek Kedzierski Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626958171224100001 Content-Type: text/plain; charset="utf-8" Temporarily modifying the SIGBUS handler is really nasty, as we might be unlucky and receive an MCE SIGBUS while having our handler registered. Unfortunately, there is no way around messing with SIGBUS when MADV_POPULATE_WRITE is not applicable or not around. Let's forward SIGBUS that don't belong to us to the already registered handler and document the situation. Signed-off-by: David Hildenbrand --- softmmu/cpus.c | 4 ++++ util/oslib-posix.c | 36 +++++++++++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/softmmu/cpus.c b/softmmu/cpus.c index 071085f840..23bca46b07 100644 --- a/softmmu/cpus.c +++ b/softmmu/cpus.c @@ -352,6 +352,10 @@ static void qemu_init_sigbus(void) { struct sigaction action; =20 + /* + * ALERT: when modifying this, take care that SIGBUS forwarding in + * os_mem_prealloc() will continue working as expected. + */ memset(&action, 0, sizeof(action)); action.sa_flags =3D SA_SIGINFO; action.sa_sigaction =3D sigbus_handler; diff --git a/util/oslib-posix.c b/util/oslib-posix.c index 7c75848a67..4f10108600 100644 --- a/util/oslib-posix.c +++ b/util/oslib-posix.c @@ -95,6 +95,7 @@ typedef struct MemsetThread MemsetThread; =20 /* used by sigbus_handler() */ static MemsetContext *sigbus_memset_context; +struct sigaction sigbus_oldact; static QemuMutex sigbus_mutex; =20 static QemuMutex page_mutex; @@ -446,7 +447,11 @@ const char *qemu_get_exec_dir(void) return exec_dir; } =20 +#ifdef CONFIG_LINUX +static void sigbus_handler(int signal, siginfo_t *siginfo, void *ctx) +#else /* CONFIG_LINUX */ static void sigbus_handler(int signal) +#endif /* CONFIG_LINUX */ { int i; =20 @@ -459,6 +464,26 @@ static void sigbus_handler(int signal) } } } + +#ifdef CONFIG_LINUX + /* + * We assume that the MCE SIGBUS handler could have been registered. We + * should never receive BUS_MCEERR_AO on any of our threads, but only = on + * the main thread registered for PR_MCE_KILL_EARLY. Further, we shoul= d not + * receive BUS_MCEERR_AR triggered by action of other threads on one of + * our threads. So, no need to check for unrelated SIGBUS when seeing = one + * for our threads. + * + * We will forward to the MCE handler, which will either handle the SI= GBUS + * or reinstall the default SIGBUS handler and reraise the SIGBUS. The + * default SIGBUS handler will crash the process, so we don't care. + */ + if (sigbus_oldact.sa_flags & SA_SIGINFO) { + sigbus_oldact.sa_sigaction(signal, siginfo, ctx); + return; + } +#endif /* CONFIG_LINUX */ + warn_report("os_mem_prealloc: unrelated SIGBUS detected and ignored"); } =20 static void *do_touch_pages(void *arg) @@ -628,10 +653,10 @@ void os_mem_prealloc(int fd, char *area, size_t memor= y, int smp_cpus, { static gsize initialized; int ret; - struct sigaction act, oldact; size_t hpagesize =3D qemu_fd_getpagesize(fd); size_t numpages =3D DIV_ROUND_UP(memory, hpagesize); bool use_madv_populate_write; + struct sigaction act; =20 /* * Sense on every invocation, as MADV_POPULATE_WRITE cannot be used for @@ -647,10 +672,15 @@ void os_mem_prealloc(int fd, char *area, size_t memor= y, int smp_cpus, =20 qemu_mutex_lock(&sigbus_mutex); memset(&act, 0, sizeof(act)); +#ifdef CONFIG_LINUX + act.sa_sigaction =3D &sigbus_handler; + act.sa_flags =3D SA_SIGINFO; +#else /* CONFIG_LINUX */ act.sa_handler =3D &sigbus_handler; act.sa_flags =3D 0; +#endif /* CONFIG_LINUX */ =20 - ret =3D sigaction(SIGBUS, &act, &oldact); + ret =3D sigaction(SIGBUS, &act, &sigbus_oldact); if (ret) { error_setg_errno(errp, errno, "os_mem_prealloc: failed to install signal handler"); @@ -666,7 +696,7 @@ void os_mem_prealloc(int fd, char *area, size_t memory,= int smp_cpus, } =20 if (!use_madv_populate_write) { - ret =3D sigaction(SIGBUS, &oldact, NULL); + ret =3D sigaction(SIGBUS, &sigbus_oldact, NULL); if (ret) { /* Terminate QEMU since it can't recover from error */ perror("os_mem_prealloc: failed to reinstall signal handler"); --=20 2.31.1