From nobody Sat Feb 7 17:09:23 2026 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 839C52D94AE; Wed, 2 Jul 2025 15:21:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751469673; cv=none; b=RQuiqySlevkWxdNM0NAnfA3TU/jeD/X3xyUUh0+RymL/eOLgQGIckpdvX6UZ6ThcN8gv/1Rn5GkV4fdMTfY9LOo6x3L2DBLtiNsxtY6p7Ichipw3hZrrrWI7GxM09z3y+Bb2Cltup2huX3EAcQs0JMF5MhkNvnnbJyjdhiJq5PQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1751469673; c=relaxed/simple; bh=GMtVapX1eB8J9sNu8WUrobQHVGXMfT9ddIn4Eusv5b8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=u0NVzRzzyeYOB6jtX6xLZ6GImFZf/Ib0lENc13zSDmwE1Y8bg/weAelQt9C2v9A93zpHncD6301lSUmNfxsYPmx7gbPdfAG4hn0ifn7SykSLnhN02PSupTOcnAYi2RIdKaif39cSy6u8Krtkp2pMBXVEc5ZpMU4WXSWJqArhJNQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LN4XWHLh; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LN4XWHLh" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-b31c6c9959cso5381833a12.1; Wed, 02 Jul 2025 08:21:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1751469668; x=1752074468; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VJYBtk2G1C72cD2Dv9g/QOoRaPRgmYYTGO42JCkNFJU=; b=LN4XWHLhElQSicYY+PVX79XfwZGWXbrDduIZmoXou/gpPm6nkrlp3P/QiM9aJ7pjc4 7lrWIzxS4S8EumXk4Nw88/wzR3s+7/2OmScBTXe0FMnY2bw5mSPzT8wflrineIa89rfv njfC41IkHcZmOPZQLetX8Vhk2k+iNLfgi4rA0DCjCuro3rbWF7K/aL6rNDKzQGFxa1ff bS/Z8Zu2gzN8INvC+EUZYc1EdhBWcjHPSlmcKL3ffsoC/38DgPb9FFUbG2/Dn9UqM5EC q913xJ56wAAtoDHjzdPeichwBy+GRf3YzjeeO1sDKNsAd+FLbXAqQfYSzww8A6RDZzBZ QhiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751469668; x=1752074468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VJYBtk2G1C72cD2Dv9g/QOoRaPRgmYYTGO42JCkNFJU=; b=PHLYbRazT9sV2+pp7oeAX7btj1dVWxlLnuigbzK4AiKIIUHtTu/aiJ0fJQvBBV69tC 3K+sf78cbebZkHn2VMwkSFkbmnRicUnqXc59xvcQsxk7W3VG93Xt4oLCIOLcfAcZNQkj TJuU9Z7PrkX4gxxok/ap2n/+Kxxow38cmdWVeeMUyQ/xkKrhL1TLH1rIJ4kRhr6KPyiV IYARdg0ZQ8z+lz6H0GCbKtzNc/+3pCgyJenuTQg9ltKhHYicmm57TcZqKxQX22Jts8VE WepDEHLpUPHcE2+JeqmBXILOCHK6saZ8Ogr1xJyeiiUKZiY7sm5EUapKgxKbtssZ1gSp 30Pg== X-Forwarded-Encrypted: i=1; AJvYcCVEO3544VCHGMF03dJoijp7gsHmu/5hKIvaVj5q/nbjoxcPhNtV9hijObxsY5HlynhdLvxSIbhdFCp0zGI=@vger.kernel.org, AJvYcCXfhI0wDVX8cOGkKYQuF7wytaaItl0hb742tDSnCiS32O/YYqMOttJCHDuorKY/UTsR8AAUx1/4LwSh/i8WTrIZ@vger.kernel.org X-Gm-Message-State: AOJu0YwLCuIftJl2ro3jehYLAbxqY8LcgxdN6Ltn+AG1HLvVOG1BnJcp aylaZ+Cbe4cW11E78Ya7Wa2oSFpUka9Kpo0KM9eZXBeQQFETxEyja4E= X-Gm-Gg: ASbGncsb5ZyOpcN/9jhLtKDRH+nZbPFDd8FBtHbL8x9M3mb+6ow7TEjhE4gMiwJppjM dqNadIn5gx6UoZSKM6HP1+ptKj9WKJoAGRJluqaw0DC9JmcNHN/4AwdOstXv3nWgvc9k56Plwww VJxvdg2USgxSouN2Jxqnp3S59MmrtUnVetreDdXTpNf+CP/q+NT2qMGTRQ3FXcab2PrkKlAP6Ky XVxtYLpEZXrfXAhAf9165cSb6eHFqeT7b1wrA4PT6nL31bOVf7cxEc7dW5hjdddaH2LR9N1xX5x UXvmJwCz1ImlXPbjyYkOC65F6ojLuHRdscSnJCXOQgnmI8tMGsrHT3kKxIVMmChPibutaXz9 X-Google-Smtp-Source: AGHT+IGwB5iImNII7TrsJzQiAIcRmKZ0nlQgiTx5a7UjRQiogg97YIvnke2d5GC6fouWDYDcSUHAVw== X-Received: by 2002:a05:6a21:3e10:b0:215:fac3:2ce2 with SMTP id adf61e73a8af0-222d7e835b2mr4000038637.23.1751469666975; Wed, 02 Jul 2025 08:21:06 -0700 (PDT) Received: from debian.ujwal.com ([223.185.132.62]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-74af56cb5d9sm14398153b3a.123.2025.07.02.08.21.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Jul 2025 08:21:06 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, peterx@redhat.com, shuah@kernel.org, jackmanb@google.com Cc: linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Ujwal Kundur Subject: [PATCH v6 1/1] selftests/mm/uffd: Refactor non-composite global vars into struct Date: Wed, 2 Jul 2025 20:50:57 +0530 Message-Id: <20250702152057.4067-1-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor macros and non-composite global variable definitions into a struct that is defined at the start of a test and is passed around instead of relying on global vars. Signed-off-by: Ujwal Kundur Acked-by: Peter Xu Tested-by: Brendan Jackman --- Changes since v5: - ensure uffd_global_test_opts_t instances are initialized - verified output remains unchanged using virtme-ng Changes since v4: - define gopts as global within uffd-stress.c to retain existing sigalrm handler logic Changes since v3: - more formatting fixes Changes since v2: - redo patch on mm-new branch Changes since v1: - indentation fixes - squash into single patch to assist bisections tools/testing/selftests/mm/uffd-common.c | 272 ++++----- tools/testing/selftests/mm/uffd-common.h | 78 +-- tools/testing/selftests/mm/uffd-stress.c | 228 ++++---- tools/testing/selftests/mm/uffd-unit-tests.c | 559 ++++++++++--------- tools/testing/selftests/mm/uffd-wp-mremap.c | 23 +- 5 files changed, 619 insertions(+), 541 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selft= ests/mm/uffd-common.c index a37088a23ffe..b5c33e5e5b3c 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -7,18 +7,30 @@ =20 #include "uffd-common.h" =20 -#define BASE_PMD_ADDR ((void *)(1UL << 30)) - -volatile bool test_uffdio_copy_eexist =3D true; -unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; -int uffd =3D -1, uffd_flags, finished, *pipefd, test_type; -bool map_shared; -bool test_uffdio_wp =3D true; -unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; uffd_test_case_ops_t *uffd_test_case_ops; -atomic_bool ready_for_fork; + +#define BASE_PMD_ADDR ((void *)(1UL << 30)) + +/* pthread_mutex_t starts at page offset 0 */ +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts) +{ + return (pthread_mutex_t *) (area + nr * gopts->page_size); +} + +/* + * count is placed in the page after pthread_mutex_t naturally aligned + * to avoid non alignment faults on non-x86 archs. + */ +volatile unsigned long long *area_count( + char *area, unsigned long nr, + uffd_global_test_opts_t *gopts) +{ + return (volatile unsigned long long *) + ((unsigned long)(area + nr * gopts->page_size + + sizeof(pthread_mutex_t) + sizeof(unsigned long long) - 1) & + ~(unsigned long)(sizeof(unsigned long long) - 1)); +} =20 static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { @@ -40,15 +52,15 @@ static int uffd_mem_fd_create(off_t mem_size, bool huge= tlb) return mem_fd; } =20 -static void anon_release_pages(char *rel_area) +static void anon_release_pages(uffd_global_test_opts_t *gopts, char *rel_a= rea) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } =20 -static int anon_allocate_area(void **alloc_area, bool is_src) +static int anon_allocate_area(uffd_global_test_opts_t *gopts, void **alloc= _area, bool is_src) { - *alloc_area =3D mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, + *alloc_area =3D mmap(NULL, gopts->nr_pages * gopts->page_size, PROT_READ = | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); if (*alloc_area =3D=3D MAP_FAILED) { *alloc_area =3D NULL; @@ -57,31 +69,32 @@ static int anon_allocate_area(void **alloc_area, bool i= s_src) return 0; } =20 -static void noop_alias_mapping(__u64 *start, size_t len, unsigned long off= set) +static void noop_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *star= t, + size_t len, unsigned long offset) { } =20 -static void hugetlb_release_pages(char *rel_area) +static void hugetlb_release_pages(uffd_global_test_opts_t *gopts, char *re= l_area) { - if (!map_shared) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (!gopts->map_shared) { + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } else { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } } =20 -static int hugetlb_allocate_area(void **alloc_area, bool is_src) +static int hugetlb_allocate_area(uffd_global_test_opts_t *gopts, void **al= loc_area, bool is_src) { - off_t size =3D nr_pages * page_size; + off_t size =3D gopts->nr_pages * gopts->page_size; off_t offset =3D is_src ? 0 : size; void *area_alias =3D NULL; char **alloc_area_alias; int mem_fd =3D uffd_mem_fd_create(size * 2, true); =20 *alloc_area =3D mmap(NULL, size, PROT_READ | PROT_WRITE, - (map_shared ? MAP_SHARED : MAP_PRIVATE) | + (gopts->map_shared ? MAP_SHARED : MAP_PRIVATE) | (is_src ? 0 : MAP_NORESERVE), mem_fd, offset); if (*alloc_area =3D=3D MAP_FAILED) { @@ -89,7 +102,7 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) return -errno; } =20 - if (map_shared) { + if (gopts->map_shared) { area_alias =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, mem_fd, offset); if (area_alias =3D=3D MAP_FAILED) @@ -97,9 +110,9 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) } =20 if (is_src) { - alloc_area_alias =3D &area_src_alias; + alloc_area_alias =3D &gopts->area_src_alias; } else { - alloc_area_alias =3D &area_dst_alias; + alloc_area_alias =3D &gopts->area_dst_alias; } if (area_alias) *alloc_area_alias =3D area_alias; @@ -108,24 +121,25 @@ static int hugetlb_allocate_area(void **alloc_area, b= ool is_src) return 0; } =20 -static void hugetlb_alias_mapping(__u64 *start, size_t len, unsigned long = offset) +static void hugetlb_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *s= tart, + size_t len, unsigned long offset) { - if (!map_shared) + if (!gopts->map_shared) return; =20 - *start =3D (unsigned long) area_dst_alias + offset; + *start =3D (unsigned long) gopts->area_dst_alias + offset; } =20 -static void shmem_release_pages(char *rel_area) +static void shmem_release_pages(uffd_global_test_opts_t *gopts, char *rel_= area) { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } =20 -static int shmem_allocate_area(void **alloc_area, bool is_src) +static int shmem_allocate_area(uffd_global_test_opts_t *gopts, void **allo= c_area, bool is_src) { void *area_alias =3D NULL; - size_t bytes =3D nr_pages * page_size, hpage_size =3D read_pmd_pagesize(); + size_t bytes =3D gopts->nr_pages * gopts->page_size, hpage_size =3D read_= pmd_pagesize(); unsigned long offset =3D is_src ? 0 : bytes; char *p =3D NULL, *p_alias =3D NULL; int mem_fd =3D uffd_mem_fd_create(bytes * 2, false); @@ -159,22 +173,23 @@ static int shmem_allocate_area(void **alloc_area, boo= l is_src) err("mmap of anonymous memory failed at %p", p_alias); =20 if (is_src) - area_src_alias =3D area_alias; + gopts->area_src_alias =3D area_alias; else - area_dst_alias =3D area_alias; + gopts->area_dst_alias =3D area_alias; =20 close(mem_fd); return 0; } =20 -static void shmem_alias_mapping(__u64 *start, size_t len, unsigned long of= fset) +static void shmem_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *sta= rt, + size_t len, unsigned long offset) { - *start =3D (unsigned long)area_dst_alias + offset; + *start =3D (unsigned long)gopts->area_dst_alias + offset; } =20 -static void shmem_check_pmd_mapping(void *p, int expect_nr_hpages) +static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *= p, int expect_nr_hpages) { - if (!check_huge_shmem(area_dst_alias, expect_nr_hpages, + if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages, read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); @@ -234,18 +249,18 @@ void uffd_stats_report(struct uffd_args *args, int n_= cpus) printf("\n"); } =20 -int userfaultfd_open(uint64_t *features) +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features) { struct uffdio_api uffdio_api; =20 - uffd =3D uffd_open(UFFD_FLAGS); - if (uffd < 0) + gopts->uffd =3D uffd_open(UFFD_FLAGS); + if (gopts->uffd < 0) return -1; - uffd_flags =3D fcntl(uffd, F_GETFD, NULL); + gopts->uffd_flags =3D fcntl(gopts->uffd, F_GETFD, NULL); =20 uffdio_api.api =3D UFFD_API; uffdio_api.features =3D *features; - if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + if (ioctl(gopts->uffd, UFFDIO_API, &uffdio_api)) /* Probably lack of CAP_PTRACE? */ return -1; if (uffdio_api.api !=3D UFFD_API) @@ -255,59 +270,63 @@ int userfaultfd_open(uint64_t *features) return 0; } =20 -static inline void munmap_area(void **area) +static inline void munmap_area(uffd_global_test_opts_t *gopts, void **area) { if (*area) - if (munmap(*area, nr_pages * page_size)) + if (munmap(*area, gopts->nr_pages * gopts->page_size)) err("munmap"); =20 *area =3D NULL; } =20 -void uffd_test_ctx_clear(void) +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts) { size_t i; =20 - if (pipefd) { - for (i =3D 0; i < nr_parallel * 2; ++i) { - if (close(pipefd[i])) + if (gopts->pipefd) { + for (i =3D 0; i < gopts->nr_parallel * 2; ++i) { + if (close(gopts->pipefd[i])) err("close pipefd"); } - free(pipefd); - pipefd =3D NULL; + free(gopts->pipefd); + gopts->pipefd =3D NULL; } =20 - if (count_verify) { - free(count_verify); - count_verify =3D NULL; + if (gopts->count_verify) { + free(gopts->count_verify); + gopts->count_verify =3D NULL; } =20 - if (uffd !=3D -1) { - if (close(uffd)) + if (gopts->uffd !=3D -1) { + if (close(gopts->uffd)) err("close uffd"); - uffd =3D -1; + gopts->uffd =3D -1; } =20 - munmap_area((void **)&area_src); - munmap_area((void **)&area_src_alias); - munmap_area((void **)&area_dst); - munmap_area((void **)&area_dst_alias); - munmap_area((void **)&area_remap); + munmap_area(gopts, (void **)&gopts->area_src); + munmap_area(gopts, (void **)&gopts->area_src_alias); + munmap_area(gopts, (void **)&gopts->area_dst); + munmap_area(gopts, (void **)&gopts->area_dst_alias); + munmap_area(gopts, (void **)&gopts->area_remap); } =20 -int uffd_test_ctx_init(uint64_t features, const char **errmsg) +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg) { unsigned long nr, cpu; int ret; =20 + gopts->area_src_alias =3D NULL; + gopts->area_dst_alias =3D NULL; + gopts->area_remap =3D NULL; + if (uffd_test_case_ops && uffd_test_case_ops->pre_alloc) { - ret =3D uffd_test_case_ops->pre_alloc(errmsg); + ret =3D uffd_test_case_ops->pre_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D uffd_test_ops->allocate_area((void **)&area_src, true); - ret |=3D uffd_test_ops->allocate_area((void **)&area_dst, false); + ret =3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_src, t= rue); + ret |=3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_dst, = false); if (ret) { if (errmsg) *errmsg =3D "memory allocation failed"; @@ -315,26 +334,26 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) } =20 if (uffd_test_case_ops && uffd_test_case_ops->post_alloc) { - ret =3D uffd_test_case_ops->post_alloc(errmsg); + ret =3D uffd_test_case_ops->post_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D userfaultfd_open(&features); + ret =3D userfaultfd_open(gopts, &features); if (ret) { if (errmsg) *errmsg =3D "possible lack of privilege"; return ret; } =20 - count_verify =3D malloc(nr_pages * sizeof(unsigned long long)); - if (!count_verify) + gopts->count_verify =3D malloc(gopts->nr_pages * sizeof(unsigned long lon= g)); + if (!gopts->count_verify) err("count_verify"); =20 - for (nr =3D 0; nr < nr_pages; nr++) { - *area_mutex(area_src, nr) =3D + for (nr =3D 0; nr < gopts->nr_pages; nr++) { + *area_mutex(gopts->area_src, nr, gopts) =3D (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER; - count_verify[nr] =3D *area_count(area_src, nr) =3D 1; + gopts->count_verify[nr] =3D *area_count(gopts->area_src, nr, gopts) =3D = 1; /* * In the transition between 255 to 256, powerpc will * read out of order in my_bcmp and see both bytes as @@ -342,7 +361,7 @@ int uffd_test_ctx_init(uint64_t features, const char **= errmsg) * after the count, to avoid my_bcmp to trigger false * positives. */ - *(area_count(area_src, nr) + 1) =3D 1; + *(area_count(gopts->area_src, nr, gopts) + 1) =3D 1; } =20 /* @@ -363,13 +382,13 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) * proactively split the thp and drop any accidentally initialized * pages within area_dst. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - pipefd =3D malloc(sizeof(int) * nr_parallel * 2); - if (!pipefd) + gopts->pipefd =3D malloc(sizeof(int) * gopts->nr_parallel * 2); + if (!gopts->pipefd) err("pipefd"); - for (cpu =3D 0; cpu < nr_parallel; cpu++) - if (pipe2(&pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) + if (pipe2(&gopts->pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) err("pipe"); =20 return 0; @@ -416,9 +435,9 @@ static void continue_range(int ufd, __u64 start, __u64 = len, bool wp) ret, (int64_t) req.mapped); } =20 -int uffd_read_msg(int ufd, struct uffd_msg *msg) +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg) { - int ret =3D read(uffd, msg, sizeof(*msg)); + int ret =3D read(gopts->uffd, msg, sizeof(*msg)); =20 if (ret !=3D sizeof(*msg)) { if (ret < 0) { @@ -433,7 +452,8 @@ int uffd_read_msg(int ufd, struct uffd_msg *msg) return 0; } =20 -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args) +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, struct uffd_ms= g *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -442,7 +462,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) =20 if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP) { /* Write protect page faults */ - wp_range(uffd, msg->arg.pagefault.address, page_size, false); + wp_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, fals= e); args->wp_faults++; } else if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR) { uint8_t *area; @@ -460,12 +480,12 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) * (UFFD-registered). */ =20 - area =3D (uint8_t *)(area_dst + - ((char *)msg->arg.pagefault.address - - area_dst_alias)); - for (b =3D 0; b < page_size; ++b) + area =3D (uint8_t *)(gopts->area_dst + + ((char *)msg->arg.pagefault.address - + gopts->area_dst_alias)); + for (b =3D 0; b < gopts->page_size; ++b) area[b] =3D ~area[b]; - continue_range(uffd, msg->arg.pagefault.address, page_size, + continue_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, args->apply_wp); args->minor_faults++; } else { @@ -493,10 +513,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WRITE) err("unexpected write fault"); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->ar= ea_dst; + offset &=3D ~(gopts->page_size-1); =20 - if (copy_page(uffd, offset, args->apply_wp)) + if (copy_page(gopts, offset, args->apply_wp)) args->missing_faults++; } } @@ -504,6 +524,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) void *uffd_poll_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; unsigned long cpu =3D args->cpu; struct pollfd pollfd[2]; struct uffd_msg msg; @@ -514,12 +535,12 @@ void *uffd_poll_thread(void *arg) if (!args->handle_fault) args->handle_fault =3D uffd_handle_page_fault; =20 - pollfd[0].fd =3D uffd; + pollfd[0].fd =3D gopts->uffd; pollfd[0].events =3D POLLIN; - pollfd[1].fd =3D pipefd[cpu*2]; + pollfd[1].fd =3D gopts->pipefd[cpu*2]; pollfd[1].events =3D POLLIN; =20 - ready_for_fork =3D true; + gopts->ready_for_fork =3D true; =20 for (;;) { ret =3D poll(pollfd, 2, -1); @@ -537,30 +558,30 @@ void *uffd_poll_thread(void *arg) } if (!(pollfd[0].revents & POLLIN)) err("pollfd[0].revents %d", pollfd[0].revents); - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; switch (msg.event) { default: err("unexpected msg event %u\n", msg.event); break; case UFFD_EVENT_PAGEFAULT: - args->handle_fault(&msg, args); + args->handle_fault(gopts, &msg, args); break; case UFFD_EVENT_FORK: - close(uffd); - uffd =3D msg.arg.fork.ufd; - pollfd[0].fd =3D uffd; + close(gopts->uffd); + gopts->uffd =3D msg.arg.fork.ufd; + pollfd[0].fd =3D gopts->uffd; break; case UFFD_EVENT_REMOVE: uffd_reg.range.start =3D msg.arg.remove.start; uffd_reg.range.len =3D msg.arg.remove.end - msg.arg.remove.start; - if (ioctl(uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) + if (ioctl(gopts->uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) err("remove failure"); break; case UFFD_EVENT_REMAP: - area_remap =3D area_dst; /* save for later unmap */ - area_dst =3D (char *)(unsigned long)msg.arg.remap.to; + gopts->area_remap =3D gopts->area_dst; /* save for later unmap */ + gopts->area_dst =3D (char *)(unsigned long)msg.arg.remap.to; break; } } @@ -568,17 +589,18 @@ void *uffd_poll_thread(void *arg) return NULL; } =20 -static void retry_copy_page(int ufd, struct uffdio_copy *uffdio_copy, +static void retry_copy_page(uffd_global_test_opts_t *gopts, struct uffdio_= copy *uffdio_copy, unsigned long offset) { - uffd_test_ops->alias_mapping(&uffdio_copy->dst, + uffd_test_ops->alias_mapping(gopts, + &uffdio_copy->dst, uffdio_copy->len, offset); - if (ioctl(ufd, UFFDIO_COPY, uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy->copy !=3D -EEXIST) err("UFFDIO_COPY retry error: %"PRId64, - (int64_t)uffdio_copy->copy); + (int64_t)uffdio_copy->copy); } else { err("UFFDIO_COPY retry unexpected: %"PRId64, (int64_t)uffdio_copy->copy); @@ -597,60 +619,60 @@ static void wake_range(int ufd, unsigned long addr, u= nsigned long len) addr), exit(1); } =20 -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp) +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp) { struct uffdio_copy uffdio_copy; =20 - if (offset >=3D nr_pages * page_size) + if (offset >=3D gopts->nr_pages * gopts->page_size) err("unexpected offset %lu\n", offset); - uffdio_copy.dst =3D (unsigned long) area_dst + offset; - uffdio_copy.src =3D (unsigned long) area_src + offset; - uffdio_copy.len =3D page_size; + uffdio_copy.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_copy.src =3D (unsigned long) gopts->area_src + offset; + uffdio_copy.len =3D gopts->page_size; if (wp) uffdio_copy.mode =3D UFFDIO_COPY_MODE_WP; else uffdio_copy.mode =3D 0; uffdio_copy.copy =3D 0; - if (ioctl(ufd, UFFDIO_COPY, &uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, &uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy.copy !=3D -EEXIST) err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); - wake_range(ufd, uffdio_copy.dst, page_size); - } else if (uffdio_copy.copy !=3D page_size) { + wake_range(gopts->uffd, uffdio_copy.dst, gopts->page_size); + } else if (uffdio_copy.copy !=3D gopts->page_size) { err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); } else { - if (test_uffdio_copy_eexist && retry) { - test_uffdio_copy_eexist =3D false; - retry_copy_page(ufd, &uffdio_copy, offset); + if (gopts->test_uffdio_copy_eexist && retry) { + gopts->test_uffdio_copy_eexist =3D false; + retry_copy_page(gopts, &uffdio_copy, offset); } return 1; } return 0; } =20 -int copy_page(int ufd, unsigned long offset, bool wp) +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p) { - return __copy_page(ufd, offset, false, wp); + return __copy_page(gopts, offset, false, wp); } =20 -int move_page(int ufd, unsigned long offset, unsigned long len) +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len) { struct uffdio_move uffdio_move; =20 - if (offset + len > nr_pages * page_size) + if (offset + len > gopts->nr_pages * gopts->page_size) err("unexpected offset %lu and length %lu\n", offset, len); - uffdio_move.dst =3D (unsigned long) area_dst + offset; - uffdio_move.src =3D (unsigned long) area_src + offset; + uffdio_move.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_move.src =3D (unsigned long) gopts->area_src + offset; uffdio_move.len =3D len; uffdio_move.mode =3D UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES; uffdio_move.move =3D 0; - if (ioctl(ufd, UFFDIO_MOVE, &uffdio_move)) { + if (ioctl(gopts->uffd, UFFDIO_MOVE, &uffdio_move)) { /* real retval in uffdio_move.move */ if (uffdio_move.move !=3D -EEXIST) err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); - wake_range(ufd, uffdio_move.dst, len); + wake_range(gopts->uffd, uffdio_move.dst, len); } else if (uffdio_move.move !=3D len) { err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); } else diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selft= ests/mm/uffd-common.h index 7700cbfa3975..37d3ca55905f 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -56,20 +56,17 @@ =20 #define err(fmt, ...) errexit(1, fmt, ##__VA_ARGS__) =20 -/* pthread_mutex_t starts at page offset 0 */ -#define area_mutex(___area, ___nr) \ - ((pthread_mutex_t *) ((___area) + (___nr)*page_size)) -/* - * count is placed in the page after pthread_mutex_t naturally aligned - * to avoid non alignment faults on non-x86 archs. - */ -#define area_count(___area, ___nr) \ - ((volatile unsigned long long *) ((unsigned long) \ - ((___area) + (___nr)*page_size + \ - sizeof(pthread_mutex_t) + \ - sizeof(unsigned long long) - 1) & \ - ~(unsigned long)(sizeof(unsigned long long) \ - - 1))) +struct uffd_global_test_opts { + unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; + char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; + int uffd, uffd_flags, finished, *pipefd, test_type; + bool map_shared; + bool test_uffdio_wp; + unsigned long long *count_verify; + volatile bool test_uffdio_copy_eexist; + atomic_bool ready_for_fork; +}; +typedef struct uffd_global_test_opts uffd_global_test_opts_t; =20 /* Userfaultfd test statistics */ struct uffd_args { @@ -79,50 +76,55 @@ struct uffd_args { unsigned long missing_faults; unsigned long wp_faults; unsigned long minor_faults; + struct uffd_global_test_opts *gopts; =20 /* A custom fault handler; defaults to uffd_handle_page_fault. */ - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args); + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, + struct uffd_args *args); }; =20 struct uffd_test_ops { - int (*allocate_area)(void **alloc_area, bool is_src); - void (*release_pages)(char *rel_area); - void (*alias_mapping)(__u64 *start, size_t len, unsigned long offset); - void (*check_pmd_mapping)(void *p, int expect_nr_hpages); + int (*allocate_area)(uffd_global_test_opts_t *gopts, void **alloc_area, b= ool is_src); + void (*release_pages)(uffd_global_test_opts_t *gopts, char *rel_area); + void (*alias_mapping)(uffd_global_test_opts_t *gopts, + __u64 *start, + size_t len, + unsigned long offset); + void (*check_pmd_mapping)(uffd_global_test_opts_t *gopts, void *p, int ex= pect_nr_hpages); }; typedef struct uffd_test_ops uffd_test_ops_t; =20 struct uffd_test_case_ops { - int (*pre_alloc)(const char **errmsg); - int (*post_alloc)(const char **errmsg); + int (*pre_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); + int (*post_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); }; typedef struct uffd_test_case_ops uffd_test_case_ops_t; =20 -extern unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -extern char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_= remap; -extern int uffd, uffd_flags, finished, *pipefd, test_type; -extern bool map_shared; -extern bool test_uffdio_wp; -extern unsigned long long *count_verify; -extern volatile bool test_uffdio_copy_eexist; -extern atomic_bool ready_for_fork; - +extern uffd_global_test_opts_t *uffd_gtest_opts; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; extern uffd_test_ops_t hugetlb_uffd_test_ops; extern uffd_test_ops_t *uffd_test_ops; extern uffd_test_case_ops_t *uffd_test_case_ops; =20 +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts); +volatile unsigned long long *area_count(char *area, + unsigned long nr, + uffd_global_test_opts_t *gopts); + void uffd_stats_report(struct uffd_args *args, int n_cpus); -int uffd_test_ctx_init(uint64_t features, const char **errmsg); -void uffd_test_ctx_clear(void); -int userfaultfd_open(uint64_t *features); -int uffd_read_msg(int ufd, struct uffd_msg *msg); +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg); +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts); +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features); +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg); void wp_range(int ufd, __u64 start, __u64 len, bool wp); -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); -int copy_page(int ufd, unsigned long offset, bool wp); -int move_page(int ufd, unsigned long offset, unsigned long len); +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args); +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp); +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p); +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len); void *uffd_poll_thread(void *arg); =20 int uffd_open_dev(unsigned int flags); diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selft= ests/mm/uffd-stress.c index 40af7f67c407..1b69e10dc284 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -44,6 +44,12 @@ uint64_t features; #define BOUNCE_VERIFY (1<<2) #define BOUNCE_POLL (1<<3) static int bounces; +/* defined globally for this particular test as the sigalrm handler + * depends on test_uffdio_*_eexist. + * XXX: define gopts in main() when we figure out a way to deal with + * test_uffdio_*_eexist. + */ +static uffd_global_test_opts_t *gopts; =20 /* exercise the test_uffdio_*_eexist every ALARM_INTERVAL_SECS */ #define ALARM_INTERVAL_SECS 10 @@ -76,54 +82,58 @@ static void usage(void) exit(1); } =20 -static void uffd_stats_reset(struct uffd_args *args, unsigned long n_cpus) +static void uffd_stats_reset(uffd_global_test_opts_t *gopts, struct uffd_a= rgs *args, + unsigned long n_cpus) { int i; =20 for (i =3D 0; i < n_cpus; i++) { args[i].cpu =3D i; - args[i].apply_wp =3D test_uffdio_wp; + args[i].apply_wp =3D gopts->test_uffdio_wp; args[i].missing_faults =3D 0; args[i].wp_faults =3D 0; args[i].minor_faults =3D 0; + args[i].gopts =3D gopts; } } =20 static void *locking_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr; unsigned long long count; =20 if (!(bounces & BOUNCE_RANDOM)) { page_nr =3D -bounces; if (!(bounces & BOUNCE_RACINGFAULTS)) - page_nr +=3D cpu * nr_pages_per_cpu; + page_nr +=3D cpu * gopts->nr_pages_per_cpu; } =20 - while (!finished) { + while (!gopts->finished) { if (bounces & BOUNCE_RANDOM) { if (getrandom(&page_nr, sizeof(page_nr), 0) !=3D sizeof(page_nr)) err("getrandom failed"); } else page_nr +=3D 1; - page_nr %=3D nr_pages; - pthread_mutex_lock(area_mutex(area_dst, page_nr)); - count =3D *area_count(area_dst, page_nr); - if (count !=3D count_verify[page_nr]) + page_nr %=3D gopts->nr_pages; + pthread_mutex_lock(area_mutex(gopts->area_dst, page_nr, gopts)); + count =3D *area_count(gopts->area_dst, page_nr, gopts); + if (count !=3D gopts->count_verify[page_nr]) err("page_nr %lu memory corruption %llu %llu", - page_nr, count, count_verify[page_nr]); + page_nr, count, gopts->count_verify[page_nr]); count++; - *area_count(area_dst, page_nr) =3D count_verify[page_nr] =3D count; - pthread_mutex_unlock(area_mutex(area_dst, page_nr)); + *area_count(gopts->area_dst, page_nr, gopts) =3D gopts->count_verify[pag= e_nr] =3D count; + pthread_mutex_unlock(area_mutex(gopts->area_dst, page_nr, gopts)); } =20 return NULL; } =20 -static int copy_page_retry(int ufd, unsigned long offset) +static int copy_page_retry(uffd_global_test_opts_t *gopts, unsigned long o= ffset) { - return __copy_page(ufd, offset, true, test_uffdio_wp); + return __copy_page(gopts, offset, true, gopts->test_uffdio_wp); } =20 pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INITIALIZER; @@ -131,15 +141,16 @@ pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INI= TIALIZER; static void *uffd_read_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; struct uffd_msg msg; =20 pthread_mutex_unlock(&uffd_read_mutex); /* from here cancellation is ok */ =20 for (;;) { - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; - uffd_handle_page_fault(&msg, args); + uffd_handle_page_fault(gopts, &msg, args); } =20 return NULL; @@ -147,32 +158,34 @@ static void *uffd_read_thread(void *arg) =20 static void *background_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr, start_nr, mid_nr, end_nr; =20 - start_nr =3D cpu * nr_pages_per_cpu; - end_nr =3D (cpu+1) * nr_pages_per_cpu; + start_nr =3D cpu * gopts->nr_pages_per_cpu; + end_nr =3D (cpu+1) * gopts->nr_pages_per_cpu; mid_nr =3D (start_nr + end_nr) / 2; =20 /* Copy the first half of the pages */ for (page_nr =3D start_nr; page_nr < mid_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 /* * If we need to test uffd-wp, set it up now. Then we'll have * at least the first half of the pages mapped already which * can be write-protected for testing */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst + start_nr * page_size, - nr_pages_per_cpu * page_size, true); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst + start_nr * gopts-= >page_size, + gopts->nr_pages_per_cpu * gopts->page_size, true); =20 /* * Continue the 2nd half of the page copying, handling write * protection faults if any */ for (page_nr =3D mid_nr; page_nr < end_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 return NULL; } @@ -180,17 +193,21 @@ static void *background_thread(void *arg) static int stress(struct uffd_args *args) { unsigned long cpu; - pthread_t locking_threads[nr_parallel]; - pthread_t uffd_threads[nr_parallel]; - pthread_t background_threads[nr_parallel]; + uffd_global_test_opts_t *gopts =3D args->gopts; + pthread_t locking_threads[gopts->nr_parallel]; + pthread_t uffd_threads[gopts->nr_parallel]; + pthread_t background_threads[gopts->nr_parallel]; =20 - finished =3D 0; - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + gopts->finished =3D 0; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { if (pthread_create(&locking_threads[cpu], &attr, - locking_thread, (void *)cpu)) + locking_thread, (void *)&args[cpu])) return 1; if (bounces & BOUNCE_POLL) { - if (pthread_create(&uffd_threads[cpu], &attr, uffd_poll_thread, &args[c= pu])) + if (pthread_create(&uffd_threads[cpu], + &attr, + uffd_poll_thread, + (void *) &args[cpu])) err("uffd_poll_thread create"); } else { if (pthread_create(&uffd_threads[cpu], &attr, @@ -200,10 +217,10 @@ static int stress(struct uffd_args *args) pthread_mutex_lock(&uffd_read_mutex); } if (pthread_create(&background_threads[cpu], &attr, - background_thread, (void *)cpu)) + background_thread, (void *)&args[cpu])) return 1; } - for (cpu =3D 0; cpu < nr_parallel; cpu++) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(background_threads[cpu], NULL)) return 1; =20 @@ -216,17 +233,17 @@ static int stress(struct uffd_args *args) * UFFDIO_COPY without writing zero pages into area_dst * because the background threads already completed). */ - uffd_test_ops->release_pages(area_src); + uffd_test_ops->release_pages(gopts, gopts->area_src); =20 - finished =3D 1; - for (cpu =3D 0; cpu < nr_parallel; cpu++) + gopts->finished =3D 1; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(locking_threads[cpu], NULL)) return 1; =20 - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { char c; if (bounces & BOUNCE_POLL) { - if (write(pipefd[cpu*2+1], &c, 1) !=3D 1) + if (write(gopts->pipefd[cpu*2+1], &c, 1) !=3D 1) err("pipefd write error"); if (pthread_join(uffd_threads[cpu], (void *)&args[cpu])) @@ -242,26 +259,26 @@ static int stress(struct uffd_args *args) return 0; } =20 -static int userfaultfd_stress(void) +static int userfaultfd_stress(uffd_global_test_opts_t *gopts) { void *area; unsigned long nr; - struct uffd_args args[nr_parallel]; - uint64_t mem_size =3D nr_pages * page_size; + struct uffd_args args[gopts->nr_parallel]; + uint64_t mem_size =3D gopts->nr_pages * gopts->page_size; int flags =3D 0; =20 - memset(args, 0, sizeof(struct uffd_args) * nr_parallel); + memset(args, 0, sizeof(struct uffd_args) * gopts->nr_parallel); =20 - if (features & UFFD_FEATURE_WP_UNPOPULATED && test_type =3D=3D TEST_ANON) + if (features & UFFD_FEATURE_WP_UNPOPULATED && gopts->test_type =3D=3D TES= T_ANON) flags =3D UFFD_FEATURE_WP_UNPOPULATED; =20 - if (uffd_test_ctx_init(flags, NULL)) + if (uffd_test_ctx_init(gopts, flags, NULL)) err("context init failed"); =20 - if (posix_memalign(&area, page_size, page_size)) + if (posix_memalign(&area, gopts->page_size, gopts->page_size)) err("out of memory"); zeropage =3D area; - bzero(zeropage, page_size); + bzero(zeropage, gopts->page_size); =20 pthread_mutex_lock(&uffd_read_mutex); =20 @@ -284,18 +301,18 @@ static int userfaultfd_stress(void) fflush(stdout); =20 if (bounces & BOUNCE_POLL) - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); else - fcntl(uffd, F_SETFL, uffd_flags & ~O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags & ~O_NONBLOCK); =20 /* register */ - if (uffd_register(uffd, area_dst, mem_size, - true, test_uffdio_wp, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure"); =20 - if (area_dst_alias) { - if (uffd_register(uffd, area_dst_alias, mem_size, - true, test_uffdio_wp, false)) + if (gopts->area_dst_alias) { + if (uffd_register(gopts->uffd, gopts->area_dst_alias, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure alias"); } =20 @@ -323,87 +340,88 @@ static int userfaultfd_stress(void) * MADV_DONTNEED only after the UFFDIO_REGISTER, so it's * required to MADV_DONTNEED here. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - uffd_stats_reset(args, nr_parallel); + uffd_stats_reset(gopts, args, gopts->nr_parallel); =20 /* bounce pass */ if (stress(args)) { - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); return 1; } =20 /* Clear all the write protections if there is any */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst, - nr_pages * page_size, false); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst, + gopts->nr_pages * gopts->page_size, false); =20 /* unregister */ - if (uffd_unregister(uffd, area_dst, mem_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, mem_size)) err("unregister failure"); - if (area_dst_alias) { - if (uffd_unregister(uffd, area_dst_alias, mem_size)) + if (gopts->area_dst_alias) { + if (uffd_unregister(gopts->uffd, gopts->area_dst_alias, mem_size)) err("unregister failure alias"); } =20 /* verification */ if (bounces & BOUNCE_VERIFY) - for (nr =3D 0; nr < nr_pages; nr++) - if (*area_count(area_dst, nr) !=3D count_verify[nr]) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + if (*area_count(gopts->area_dst, nr, gopts) !=3D + gopts->count_verify[nr]) err("error area_count %llu %llu %lu\n", - *area_count(area_src, nr), - count_verify[nr], nr); + *area_count(gopts->area_src, nr, gopts), + gopts->count_verify[nr], nr); =20 /* prepare next bounce */ - swap(area_src, area_dst); + swap(gopts->area_src, gopts->area_dst); =20 - swap(area_src_alias, area_dst_alias); + swap(gopts->area_src_alias, gopts->area_dst_alias); =20 - uffd_stats_report(args, nr_parallel); + uffd_stats_report(args, gopts->nr_parallel); } - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); =20 return 0; } =20 -static void set_test_type(const char *type) +static void set_test_type(uffd_global_test_opts_t *gopts, const char *type) { if (!strcmp(type, "anon")) { - test_type =3D TEST_ANON; + gopts->test_type =3D TEST_ANON; uffd_test_ops =3D &anon_uffd_test_ops; } else if (!strcmp(type, "hugetlb")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; - map_shared =3D true; + gopts->map_shared =3D true; } else if (!strcmp(type, "hugetlb-private")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; } else if (!strcmp(type, "shmem")) { - map_shared =3D true; - test_type =3D TEST_SHMEM; + gopts->map_shared =3D true; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } else if (!strcmp(type, "shmem-private")) { - test_type =3D TEST_SHMEM; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } } =20 -static void parse_test_type_arg(const char *raw_type) +static void parse_test_type_arg(uffd_global_test_opts_t *gopts, const char= *raw_type) { - set_test_type(raw_type); + set_test_type(gopts, raw_type); =20 - if (!test_type) + if (!gopts->test_type) err("failed to parse test type argument: '%s'", raw_type); =20 - if (test_type =3D=3D TEST_HUGETLB) - page_size =3D default_huge_page_size(); + if (gopts->test_type =3D=3D TEST_HUGETLB) + gopts->page_size =3D default_huge_page_size(); else - page_size =3D sysconf(_SC_PAGE_SIZE); + gopts->page_size =3D sysconf(_SC_PAGE_SIZE); =20 - if (!page_size) + if (!gopts->page_size) err("Unable to determine page size"); - if ((unsigned long) area_count(NULL, 0) + sizeof(unsigned long long) * 2 - > page_size) + if ((unsigned long) area_count(NULL, 0, gopts) + sizeof(unsigned long lon= g) * 2 + > gopts->page_size) err("Impossible to run this test"); =20 /* @@ -415,21 +433,21 @@ static void parse_test_type_arg(const char *raw_type) if (uffd_get_features(&features) && errno =3D=3D ENOENT) ksft_exit_skip("failed to get available features (%d)\n", errno); =20 - test_uffdio_wp =3D test_uffdio_wp && + gopts->test_uffdio_wp =3D gopts->test_uffdio_wp && (features & UFFD_FEATURE_PAGEFAULT_FLAG_WP); =20 - if (test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGETLBFS_SH= MEM)) - test_uffdio_wp =3D false; + if (gopts->test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGET= LBFS_SHMEM)) + gopts->test_uffdio_wp =3D false; =20 - close(uffd); - uffd =3D -1; + close(gopts->uffd); + gopts->uffd =3D -1; } =20 static void sigalrm(int sig) { if (sig !=3D SIGALRM) abort(); - test_uffdio_copy_eexist =3D true; + gopts->test_uffdio_copy_eexist =3D true; alarm(ALARM_INTERVAL_SECS); } =20 @@ -438,6 +456,8 @@ int main(int argc, char **argv) unsigned long nr_cpus; size_t bytes; =20 + gopts =3D (uffd_global_test_opts_t *) malloc(sizeof(uffd_global_test_opts= _t)); + if (argc < 4) usage(); =20 @@ -445,11 +465,11 @@ int main(int argc, char **argv) err("failed to arm SIGALRM"); alarm(ALARM_INTERVAL_SECS); =20 - parse_test_type_arg(argv[1]); + parse_test_type_arg(gopts, argv[1]); bytes =3D atol(argv[2]) * 1024 * 1024; =20 - if (test_type =3D=3D TEST_HUGETLB && - get_free_hugepages() < bytes / page_size) { + if (gopts->test_type =3D=3D TEST_HUGETLB && + get_free_hugepages() < bytes / gopts->page_size) { printf("skip: Skipping userfaultfd... not enough hugepages\n"); return KSFT_SKIP; } @@ -459,15 +479,15 @@ int main(int argc, char **argv) /* Don't let calculation below go to zero. */ ksft_print_msg("_SC_NPROCESSORS_ONLN (%lu) too large, capping nr_threads= to 32\n", nr_cpus); - nr_parallel =3D 32; + gopts->nr_parallel =3D 32; } else { - nr_parallel =3D nr_cpus; + gopts->nr_parallel =3D nr_cpus; } =20 - nr_pages_per_cpu =3D bytes / page_size / nr_parallel; - if (!nr_pages_per_cpu) { + gopts->nr_pages_per_cpu =3D bytes / gopts->page_size / gopts->nr_parallel; + if (!gopts->nr_pages_per_cpu) { _err("pages_per_cpu =3D 0, cannot test (%lu / %lu / %lu)", - bytes, page_size, nr_parallel); + bytes, gopts->page_size, gopts->nr_parallel); usage(); } =20 @@ -476,11 +496,11 @@ int main(int argc, char **argv) _err("invalid bounces"); usage(); } - nr_pages =3D nr_pages_per_cpu * nr_parallel; + gopts->nr_pages =3D gopts->nr_pages_per_cpu * gopts->nr_parallel; =20 printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n", - nr_pages, nr_pages_per_cpu); - return userfaultfd_stress(); + gopts->nr_pages, gopts->nr_pages_per_cpu); + return userfaultfd_stress(gopts); } =20 #else /* __NR_userfaultfd */ diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/s= elftests/mm/uffd-unit-tests.c index 50501b38e34e..9e3be2ee7f1b 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -76,7 +76,7 @@ struct uffd_test_args { typedef struct uffd_test_args uffd_test_args_t; =20 /* Returns: UFFD_TEST_* */ -typedef void (*uffd_test_fn)(uffd_test_args_t *); +typedef void (*uffd_test_fn)(uffd_global_test_opts_t *, uffd_test_args_t *= ); =20 typedef struct { const char *name; @@ -181,33 +181,6 @@ static int test_uffd_api(bool use_dev) return 1; } =20 -/* - * This function initializes the global variables. TODO: remove global - * vars and then remove this. - */ -static int -uffd_setup_environment(uffd_test_args_t *args, uffd_test_case_t *test, - mem_type_t *mem_type, const char **errmsg) -{ - map_shared =3D mem_type->shared; - uffd_test_ops =3D mem_type->mem_ops; - uffd_test_case_ops =3D test->test_case_ops; - - if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) - page_size =3D default_huge_page_size(); - else - page_size =3D psize(); - - /* Ensure we have at least 2 pages */ - nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, page_size * 2) / page_size; - /* TODO: remove this global var.. it's so ugly */ - nr_parallel =3D 1; - - /* Initialize test arguments */ - args->mem_type =3D mem_type; - - return uffd_test_ctx_init(test->uffd_feature_required, errmsg); -} =20 static bool uffd_feature_supported(uffd_test_case_t *test) { @@ -237,7 +210,8 @@ static int pagemap_open(void) } while (0) =20 typedef struct { - int parent_uffd, child_uffd; + uffd_global_test_opts_t *gopts; + int child_uffd; } fork_event_args; =20 static void *fork_event_consumer(void *data) @@ -245,10 +219,10 @@ static void *fork_event_consumer(void *data) fork_event_args *args =3D data; struct uffd_msg msg =3D { 0 }; =20 - ready_for_fork =3D true; + args->gopts->ready_for_fork =3D true; =20 /* Read until a full msg received */ - while (uffd_read_msg(args->parent_uffd, &msg)); + while (uffd_read_msg(args->gopts, &msg)); =20 if (msg.event !=3D UFFD_EVENT_FORK) err("wrong message: %u\n", msg.event); @@ -304,9 +278,9 @@ static void unpin_pages(pin_args *args) args->pinned =3D false; } =20 -static int pagemap_test_fork(int uffd, bool with_event, bool test_pin) +static int pagemap_test_fork(uffd_global_test_opts_t *gopts, bool with_eve= nt, bool test_pin) { - fork_event_args args =3D { .parent_uffd =3D uffd, .child_uffd =3D -1 }; + fork_event_args args =3D { .gopts =3D gopts, .child_uffd =3D -1 }; pthread_t thread; pid_t child; uint64_t value; @@ -314,10 +288,10 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 /* Prepare a thread to resolve EVENT_FORK */ if (with_event) { - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; if (pthread_create(&thread, NULL, fork_event_consumer, &args)) err("pthread_create()"); - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ } =20 @@ -328,14 +302,14 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 fd =3D pagemap_open(); =20 - if (test_pin && pin_pages(&args, area_dst, page_size)) + if (test_pin && pin_pages(&args, gopts->area_dst, gopts->page_size)) /* * Normally when reach here we have pinned in * previous tests, so shouldn't fail anymore */ err("pin page failed in child"); =20 - value =3D pagemap_get_entry(fd, area_dst); + value =3D pagemap_get_entry(fd, gopts->area_dst); /* * After fork(), we should handle uffd-wp bit differently: * @@ -361,70 +335,70 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) return result; } =20 -static void uffd_wp_unpopulated_test(uffd_test_args_t *args) +static void uffd_wp_unpopulated_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args) { uint64_t value; int pagemap_fd; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Test applying pte marker to anon unpopulated */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); =20 /* Test unprotect on anon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test zap on anon marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - if (madvise(area_dst, page_size, MADV_DONTNEED)) + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test fault in after marker removed */ - *area_dst =3D 1; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 /* Test read-zero-page upon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - *(volatile char *)area_dst; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + *(volatile char *)gopts->area_dst; /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 uffd_test_pass(); } =20 -static void uffd_wp_fork_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_test_common(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args, bool with_event) { int pagemap_fd; uint64_t value; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in present pte", with_event ? "missing" : "stall"); goto out; @@ -442,79 +416,80 @@ static void uffd_wp_fork_test_common(uffd_test_args_t= *args, * to expose pte markers. */ if (args->mem_type->shared) { - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("MADV_DONTNEED"); } else { /* * NOTE: ignore retval because private-hugetlb doesn't yet * support swapping, so it could fail. */ - madvise(area_dst, page_size, MADV_PAGEOUT); + madvise(gopts->area_dst, gopts->page_size, MADV_PAGEOUT); } =20 /* Uffd-wp should persist even swapped out */ - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in zapped pte", with_event ? "missing" : "stall"); goto out; } =20 /* Unprotect; this tests swap pte modifications */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Fault in the page from disk */ - *area_dst =3D 2; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 2; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, nr_pages * page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts= ->page_size)) err("unregister failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_test(uffd_test_args_t *args) +static void uffd_wp_fork_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_wp_fork_test_common(args, false); + uffd_wp_fork_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_with_event_test(uffd_global_test_opts_t *gopts, u= ffd_test_args_t *args) { - uffd_wp_fork_test_common(args, true); + uffd_wp_fork_test_common(gopts, args, true); } =20 -static void uffd_wp_fork_pin_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_pin_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, bool with_event) { int pagemap_fd; pin_args pin_args =3D {}; =20 - if (uffd_register(uffd, area_dst, page_size, false, true, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->page_size, false, = true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); =20 /* * 1. First pin, then fork(). This tests fork() special path when * doing early CoW if the page is private. */ - if (pin_pages(&pin_args, area_dst, page_size)) { + if (pin_pages(&pin_args, gopts->area_dst, gopts->page_size)) { uffd_test_skip("Possibly CONFIG_GUP_TEST missing " "or unprivileged"); close(pagemap_fd); - uffd_unregister(uffd, area_dst, page_size); + uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size); return; } =20 - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in early CoW of fork()", with_event ? "missing" : "stall"); unpin_pages(&pin_args); @@ -527,49 +502,50 @@ static void uffd_wp_fork_pin_test_common(uffd_test_ar= gs_t *args, * 2. First fork(), then pin (in the child, where test_pin=3D=3Dtrue). * This tests COR, aka, page unsharing on private memories. */ - if (pagemap_test_fork(uffd, with_event, true)) { + if (pagemap_test_fork(gopts, with_event, true)) { uffd_test_fail("Detected %s uffd-wp bit when RO pin", with_event ? "missing" : "stall"); goto out; } uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("register failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_pin_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_test(uffd_global_test_opts_t *gopts, uffd_tes= t_args_t *args) { - uffd_wp_fork_pin_test_common(args, false); + uffd_wp_fork_pin_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_pin_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_with_event_test(uffd_global_test_opts_t *gopt= s, uffd_test_args_t *args) { - uffd_wp_fork_pin_test_common(args, true); + uffd_wp_fork_pin_test_common(gopts, args, true); } =20 -static void check_memory_contents(char *p) +static void check_memory_contents(uffd_global_test_opts_t *gopts, char *p) { unsigned long i, j; uint8_t expected_byte; =20 - for (i =3D 0; i < nr_pages; ++i) { + for (i =3D 0; i < gopts->nr_pages; ++i) { expected_byte =3D ~((uint8_t)(i % ((uint8_t)-1))); - for (j =3D 0; j < page_size; j++) { - uint8_t v =3D *(uint8_t *)(p + (i * page_size) + j); + for (j =3D 0; j < gopts->page_size; j++) { + uint8_t v =3D *(uint8_t *)(p + (i * gopts->page_size) + j); if (v !=3D expected_byte) err("unexpected page contents"); } } } =20 -static void uffd_minor_test_common(bool test_collapse, bool test_wp) +static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool te= st_collapse, bool test_wp) { unsigned long p; pthread_t uffd_mon; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 /* * NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing @@ -577,7 +553,7 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) */ assert(!(test_collapse && test_wp)); =20 - if (uffd_register(uffd, area_dst_alias, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst_alias, gopts->nr_pages * g= opts->page_size, /* NOTE! MADV_COLLAPSE may not work with uffd-wp */ false, test_wp, true)) err("register failure"); @@ -586,9 +562,9 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) * After registering with UFFD, populate the non-UFFD-registered side of * the shared mapping. This should *not* trigger any UFFD minor faults. */ - for (p =3D 0; p < nr_pages; ++p) - memset(area_dst + (p * page_size), p % ((uint8_t)-1), - page_size); + for (p =3D 0; p < gopts->nr_pages; ++p) + memset(gopts->area_dst + (p * gopts->page_size), p % ((uint8_t)-1), + gopts->page_size); =20 args.apply_wp =3D test_wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -600,50 +576,51 @@ static void uffd_minor_test_common(bool test_collapse= , bool test_wp) * fault. uffd_poll_thread will resolve the fault by bit-flipping the * page's contents, and then issuing a CONTINUE ioctl. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); =20 if (test_collapse) { - if (madvise(area_dst_alias, nr_pages * page_size, + if (madvise(gopts->area_dst_alias, gopts->nr_pages * gopts->page_size, MADV_COLLAPSE)) { /* It's fine to fail for this one... */ uffd_test_skip("MADV_COLLAPSE failed"); return; } =20 - uffd_test_ops->check_pmd_mapping(area_dst, - nr_pages * page_size / + uffd_test_ops->check_pmd_mapping(gopts, + gopts->area_dst, + gopts->nr_pages * gopts->page_size / read_pmd_pagesize()); /* * This won't cause uffd-fault - it purely just makes sure there * was no corruption. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); } =20 - if (args.missing_faults !=3D 0 || args.minor_faults !=3D nr_pages) + if (args.missing_faults !=3D 0 || args.minor_faults !=3D gopts->nr_pages) uffd_test_fail("stats check error"); else uffd_test_pass(); } =20 -void uffd_minor_test(uffd_test_args_t *args) +void uffd_minor_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *arg= s) { - uffd_minor_test_common(false, false); + uffd_minor_test_common(gopts, false, false); } =20 -void uffd_minor_wp_test(uffd_test_args_t *args) +void uffd_minor_wp_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *= args) { - uffd_minor_test_common(false, true); + uffd_minor_test_common(gopts, false, true); } =20 -void uffd_minor_collapse_test(uffd_test_args_t *args) +void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_minor_test_common(true, false); + uffd_minor_test_common(gopts, true, false); } =20 static sigjmp_buf jbuf, *sigbuf; @@ -678,7 +655,7 @@ static void sighndl(int sig, siginfo_t *siginfo, void *= ptr) * This also tests UFFD_FEATURE_EVENT_FORK event along with the signal * feature. Using monitor thread, verify no userfault events are generated. */ -static int faulting_process(int signal_test, bool wp) +static int faulting_process(uffd_global_test_opts_t *gopts, int signal_tes= t, bool wp) { unsigned long nr, i; unsigned long long count; @@ -687,7 +664,7 @@ static int faulting_process(int signal_test, bool wp) struct sigaction act; volatile unsigned long signalled =3D 0; =20 - split_nr_pages =3D (nr_pages + 1) / 2; + split_nr_pages =3D (gopts->nr_pages + 1) / 2; =20 if (signal_test) { sigbuf =3D &jbuf; @@ -701,7 +678,7 @@ static int faulting_process(int signal_test, bool wp) =20 for (nr =3D 0; nr < split_nr_pages; nr++) { volatile int steps =3D 1; - unsigned long offset =3D nr * page_size; + unsigned long offset =3D nr * gopts->page_size; =20 if (signal_test) { if (sigsetjmp(*sigbuf, 1) !=3D 0) { @@ -713,15 +690,15 @@ static int faulting_process(int signal_test, bool wp) if (steps =3D=3D 1) { /* This is a MISSING request */ steps++; - if (copy_page(uffd, offset, wp)) + if (copy_page(gopts, offset, wp)) signalled++; } else { /* This is a WP request */ assert(steps =3D=3D 2); - wp_range(uffd, - (__u64)area_dst + + wp_range(gopts->uffd, + (__u64)gopts->area_dst + offset, - page_size, false); + gopts->page_size, false); } } else { signalled++; @@ -730,51 +707,53 @@ static int faulting_process(int signal_test, bool wp) } } =20 - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 if (signal_test) return signalled !=3D split_nr_pages; =20 - area_dst =3D mremap(area_dst, nr_pages * page_size, nr_pages * page_size, - MREMAP_MAYMOVE | MREMAP_FIXED, area_src); - if (area_dst =3D=3D MAP_FAILED) + gopts->area_dst =3D mremap(gopts->area_dst, gopts->nr_pages * gopts->page= _size, + gopts->nr_pages * gopts->page_size, + MREMAP_MAYMOVE | MREMAP_FIXED, + gopts->area_src); + if (gopts->area_dst =3D=3D MAP_FAILED) err("mremap"); /* Reset area_src since we just clobbered it */ - area_src =3D NULL; + gopts->area_src =3D NULL; =20 - for (; nr < nr_pages; nr++) { - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) { + for (; nr < gopts->nr_pages; nr++) { + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) { err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); } /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - for (nr =3D 0; nr < nr_pages; nr++) - for (i =3D 0; i < page_size; i++) - if (*(area_dst + nr * page_size + i) !=3D 0) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + for (i =3D 0; i < gopts->page_size; i++) + if (*(gopts->area_dst + nr * gopts->page_size + i) !=3D 0) err("page %lu offset %lu is not zero", nr, i); =20 return 0; } =20 -static void uffd_sigbus_test_common(bool wp) +static void uffd_sigbus_test_common(uffd_global_test_opts_t *gopts, bool w= p) { unsigned long userfaults; pthread_t uffd_mon; @@ -782,25 +761,26 @@ static void uffd_sigbus_test_common(bool wp) int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 - if (faulting_process(1, wp)) + if (faulting_process(gopts, 1, wp)) err("faulting process failed"); =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 args.apply_wp =3D wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -808,12 +788,12 @@ static void uffd_sigbus_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(2, wp)); + exit(faulting_process(gopts, 2, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, (void **)&userfaults)) err("pthread_join()"); @@ -824,28 +804,29 @@ static void uffd_sigbus_test_common(bool wp) uffd_test_pass(); } =20 -static void uffd_sigbus_test(uffd_test_args_t *args) +static void uffd_sigbus_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_sigbus_test_common(false); + uffd_sigbus_test_common(gopts, false); } =20 -static void uffd_sigbus_wp_test(uffd_test_args_t *args) +static void uffd_sigbus_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_sigbus_test_common(true); + uffd_sigbus_test_common(gopts, true); } =20 -static void uffd_events_test_common(bool wp) +static void uffd_events_test_common(uffd_global_test_opts_t *gopts, bool w= p) { pthread_t uffd_mon; pid_t pid; int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); - if (uffd_register(uffd, area_dst, nr_pages * page_size, + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 @@ -853,7 +834,7 @@ static void uffd_events_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -861,39 +842,39 @@ static void uffd_events_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(0, wp)); + exit(faulting_process(gopts, 0, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (args.missing_faults !=3D nr_pages) + if (args.missing_faults !=3D gopts->nr_pages) uffd_test_fail("Fault counts wrong"); else uffd_test_pass(); } =20 -static void uffd_events_test(uffd_test_args_t *args) +static void uffd_events_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_events_test_common(false); + uffd_events_test_common(gopts, false); } =20 -static void uffd_events_wp_test(uffd_test_args_t *args) +static void uffd_events_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_events_test_common(true); + uffd_events_test_common(gopts, true); } =20 -static void retry_uffdio_zeropage(int ufd, +static void retry_uffdio_zeropage(uffd_global_test_opts_t *gopts, struct uffdio_zeropage *uffdio_zeropage) { - uffd_test_ops->alias_mapping(&uffdio_zeropage->range.start, + uffd_test_ops->alias_mapping(gopts, &uffdio_zeropage->range.start, uffdio_zeropage->range.len, 0); - if (ioctl(ufd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { + if (ioctl(gopts->uffd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { if (uffdio_zeropage->zeropage !=3D -EEXIST) err("UFFDIO_ZEROPAGE error: %"PRId64, (int64_t)uffdio_zeropage->zeropage); @@ -903,16 +884,16 @@ static void retry_uffdio_zeropage(int ufd, } } =20 -static bool do_uffdio_zeropage(int ufd, bool has_zeropage) +static bool do_uffdio_zeropage(uffd_global_test_opts_t *gopts, bool has_ze= ropage) { struct uffdio_zeropage uffdio_zeropage =3D { 0 }; int ret; __s64 res; =20 - uffdio_zeropage.range.start =3D (unsigned long) area_dst; - uffdio_zeropage.range.len =3D page_size; + uffdio_zeropage.range.start =3D (unsigned long) gopts->area_dst; + uffdio_zeropage.range.len =3D gopts->page_size; uffdio_zeropage.mode =3D 0; - ret =3D ioctl(ufd, UFFDIO_ZEROPAGE, &uffdio_zeropage); + ret =3D ioctl(gopts->uffd, UFFDIO_ZEROPAGE, &uffdio_zeropage); res =3D uffdio_zeropage.zeropage; if (ret) { /* real retval in ufdio_zeropage.zeropage */ @@ -921,10 +902,10 @@ static bool do_uffdio_zeropage(int ufd, bool has_zero= page) else if (res !=3D -EINVAL) err("UFFDIO_ZEROPAGE not -EINVAL"); } else if (has_zeropage) { - if (res !=3D page_size) + if (res !=3D gopts->page_size) err("UFFDIO_ZEROPAGE unexpected size"); else - retry_uffdio_zeropage(ufd, &uffdio_zeropage); + retry_uffdio_zeropage(gopts, &uffdio_zeropage); return true; } else err("UFFDIO_ZEROPAGE succeeded"); @@ -950,25 +931,29 @@ uffd_register_detect_zeropage(int uffd, void *addr, u= int64_t len) } =20 /* exercise UFFDIO_ZEROPAGE */ -static void uffd_zeropage_test(uffd_test_args_t *args) +static void uffd_zeropage_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *args) { bool has_zeropage; int i; =20 - has_zeropage =3D uffd_register_detect_zeropage(uffd, area_dst, page_size); - if (area_dst_alias) + has_zeropage =3D uffd_register_detect_zeropage(gopts->uffd, + gopts->area_dst, + gopts->page_size); + if (gopts->area_dst_alias) /* Ignore the retval; we already have it */ - uffd_register_detect_zeropage(uffd, area_dst_alias, page_size); + uffd_register_detect_zeropage(gopts->uffd, gopts->area_dst_alias, gopts-= >page_size); =20 - if (do_uffdio_zeropage(uffd, has_zeropage)) - for (i =3D 0; i < page_size; i++) - if (area_dst[i] !=3D 0) + if (do_uffdio_zeropage(gopts, has_zeropage)) + for (i =3D 0; i < gopts->page_size; i++) + if (gopts->area_dst[i] !=3D 0) err("data non-zero at offset %d\n", i); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); =20 - if (area_dst_alias && uffd_unregister(uffd, area_dst_alias, page_size)) + if (gopts->area_dst_alias && uffd_unregister(gopts->uffd, + gopts->area_dst_alias, + gopts->page_size)) err("unregister"); =20 uffd_test_pass(); @@ -987,26 +972,27 @@ static void uffd_register_poison(int uffd, void *addr= , uint64_t len) err("registered area doesn't support COPY and POISON ioctls"); } =20 -static void do_uffdio_poison(int uffd, unsigned long offset) +static void do_uffdio_poison(uffd_global_test_opts_t *gopts, unsigned long= offset) { struct uffdio_poison uffdio_poison =3D { 0 }; int ret; __s64 res; =20 - uffdio_poison.range.start =3D (unsigned long) area_dst + offset; - uffdio_poison.range.len =3D page_size; + uffdio_poison.range.start =3D (unsigned long) gopts->area_dst + offset; + uffdio_poison.range.len =3D gopts->page_size; uffdio_poison.mode =3D 0; - ret =3D ioctl(uffd, UFFDIO_POISON, &uffdio_poison); + ret =3D ioctl(gopts->uffd, UFFDIO_POISON, &uffdio_poison); res =3D uffdio_poison.updated; =20 if (ret) err("UFFDIO_POISON error: %"PRId64, (int64_t)res); - else if (res !=3D page_size) + else if (res !=3D gopts->page_size) err("UFFDIO_POISON unexpected size: %"PRId64, (int64_t)res); } =20 -static void uffd_poison_handle_fault( - struct uffd_msg *msg, struct uffd_args *args) +static void uffd_poison_handle_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -1017,20 +1003,20 @@ static void uffd_poison_handle_fault( (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; + offset &=3D ~(gopts->page_size-1); =20 /* Odd pages -> copy zeroed page; even pages -> poison. */ - if (offset & page_size) - copy_page(uffd, offset, false); + if (offset & gopts->page_size) + copy_page(gopts, offset, false); else - do_uffdio_poison(uffd, offset); + do_uffdio_poison(gopts, offset); } =20 /* Make sure to cover odd/even, and minimum duplications */ #define UFFD_POISON_TEST_NPAGES 4 =20 -static void uffd_poison_test(uffd_test_args_t *targs) +static void uffd_poison_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *targs) { pthread_t uffd_mon; char c; @@ -1039,15 +1025,17 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) unsigned long nr_sigbus =3D 0; unsigned long nr, poison_pages =3D UFFD_POISON_TEST_NPAGES; =20 - if (nr_pages < poison_pages) { - uffd_test_skip("Too few pages for POISON test"); + if (gopts->nr_pages < poison_pages) { + uffd_test_skip("Too less pages for POISON test"); return; } =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + args.gopts =3D gopts; + + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - uffd_register_poison(uffd, area_dst, poison_pages * page_size); - memset(area_src, 0, poison_pages * page_size); + uffd_register_poison(gopts->uffd, gopts->area_dst, poison_pages * gopts->= page_size); + memset(gopts->area_src, 0, poison_pages * gopts->page_size); =20 args.handle_fault =3D uffd_poison_handle_fault; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -1060,8 +1048,8 @@ static void uffd_poison_test(uffd_test_args_t *targs) err("sigaction"); =20 for (nr =3D 0; nr < poison_pages; ++nr) { - unsigned long offset =3D nr * page_size; - const char *bytes =3D (const char *) area_dst + offset; + unsigned long offset =3D nr * gopts->page_size; + const char *bytes =3D (const char *) gopts->area_dst + offset; const char *i; =20 if (sigsetjmp(*sigbuf, 1)) { @@ -1074,14 +1062,14 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) continue; } =20 - for (i =3D bytes; i < bytes + page_size; ++i) { + for (i =3D bytes; i < bytes + gopts->page_size; ++i) { if (*i) err("nonzero byte in area_dst (%p) at %p: %u", - area_dst, i, *i); + gopts->area_dst, i, *i); } } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); @@ -1094,7 +1082,9 @@ static void uffd_poison_test(uffd_test_args_t *targs) } =20 static void -uffd_move_handle_fault_common(struct uffd_msg *msg, struct uffd_args *args, +uffd_move_handle_fault_common(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args, unsigned long len) { unsigned long offset; @@ -1106,28 +1096,32 @@ uffd_move_handle_fault_common(struct uffd_msg *msg,= struct uffd_args *args, (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR | UFFD_PAGEFAULT_= FLAG_WRITE)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; offset &=3D ~(len-1); =20 - if (move_page(uffd, offset, len)) + if (move_page(gopts, offset, len)) args->missing_faults++; } =20 -static void uffd_move_handle_fault(struct uffd_msg *msg, +static void uffd_move_handle_fault(uffd_global_test_opts_t *gopts, struct = uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, page_size); + uffd_move_handle_fault_common(gopts, msg, args, gopts->page_size); } =20 -static void uffd_move_pmd_handle_fault(struct uffd_msg *msg, +static void uffd_move_pmd_handle_fault(uffd_global_test_opts_t *gopts, str= uct uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, read_pmd_pagesize()); + uffd_move_handle_fault_common(gopts, msg, args, read_pmd_pagesize()); } =20 static void -uffd_move_test_common(uffd_test_args_t *targs, unsigned long chunk_size, - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args)) +uffd_move_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *targs, + unsigned long chunk_size, + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, struct uffd_args *args) +) { unsigned long nr; pthread_t uffd_mon; @@ -1139,11 +1133,13 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, unsigned long src_offs =3D 0; unsigned long dst_offs =3D 0; =20 + args.gopts =3D gopts; + /* Prevent source pages from being mapped more than once */ - if (madvise(area_src, nr_pages * page_size, MADV_DONTFORK)) + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DON= TFORK)) err("madvise(MADV_DONTFORK) failure"); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("register failure"); =20 @@ -1151,22 +1147,22 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - step_size =3D chunk_size / page_size; - step_count =3D nr_pages / step_size; + step_size =3D chunk_size / gopts->page_size; + step_count =3D gopts->nr_pages / step_size; =20 - if (chunk_size > page_size) { - char *aligned_src =3D ALIGN_UP(area_src, chunk_size); - char *aligned_dst =3D ALIGN_UP(area_dst, chunk_size); + if (chunk_size > gopts->page_size) { + char *aligned_src =3D ALIGN_UP(gopts->area_src, chunk_size); + char *aligned_dst =3D ALIGN_UP(gopts->area_dst, chunk_size); =20 - if (aligned_src !=3D area_src || aligned_dst !=3D area_dst) { - src_offs =3D (aligned_src - area_src) / page_size; - dst_offs =3D (aligned_dst - area_dst) / page_size; + if (aligned_src !=3D gopts->area_src || aligned_dst !=3D gopts->area_dst= ) { + src_offs =3D (aligned_src - gopts->area_src) / gopts->page_size; + dst_offs =3D (aligned_dst - gopts->area_dst) / gopts->page_size; step_count--; } - orig_area_src =3D area_src; - orig_area_dst =3D area_dst; - area_src =3D aligned_src; - area_dst =3D aligned_dst; + orig_area_src =3D gopts->area_src; + orig_area_dst =3D gopts->area_dst; + gopts->area_src =3D aligned_src; + gopts->area_dst =3D aligned_dst; } =20 /* @@ -1180,34 +1176,34 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, =20 /* Check area_src content */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); - if (count !=3D count_verify[src_offs + nr + i]) + count =3D *area_count(gopts->area_src, nr + i, gopts); + if (count !=3D gopts->count_verify[src_offs + nr + i]) err("nr %lu source memory invalid %llu %llu\n", - nr + i, count, count_verify[src_offs + nr + i]); + nr + i, count, gopts->count_verify[src_offs + nr + i]); } =20 /* Faulting into area_dst should move the page or the huge page */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_dst, nr + i); - if (count !=3D count_verify[dst_offs + nr + i]) + count =3D *area_count(gopts->area_dst, nr + i, gopts); + if (count !=3D gopts->count_verify[dst_offs + nr + i]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[dst_offs + nr + i]); + nr, count, gopts->count_verify[dst_offs + nr + i]); } =20 /* Re-check area_src content which should be empty */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); + count =3D *area_count(gopts->area_src, nr + i, gopts); if (count !=3D 0) err("nr %lu move failed %llu %llu\n", - nr, count, count_verify[src_offs + nr + i]); + nr, count, gopts->count_verify[src_offs + nr + i]); } } - if (chunk_size > page_size) { - area_src =3D orig_area_src; - area_dst =3D orig_area_dst; + if (chunk_size > gopts->page_size) { + gopts->area_src =3D orig_area_src; + gopts->area_dst =3D orig_area_dst; } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); @@ -1218,24 +1214,24 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, uffd_test_pass(); } =20 -static void uffd_move_test(uffd_test_args_t *targs) +static void uffd_move_test(uffd_global_test_opts_t *gopts, uffd_test_args_= t *targs) { - uffd_move_test_common(targs, page_size, uffd_move_handle_fault); + uffd_move_test_common(gopts, targs, gopts->page_size, uffd_move_handle_fa= ult); } =20 -static void uffd_move_pmd_test(uffd_test_args_t *targs) +static void uffd_move_pmd_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) err("madvise(MADV_HUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static void uffd_move_pmd_split_test(uffd_test_args_t *targs) +static void uffd_move_pmd_split_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) err("madvise(MADV_NOHUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 @@ -1295,6 +1291,11 @@ typedef enum { THR_STATE_UNINTERRUPTIBLE, } thread_state; =20 +typedef struct { + uffd_global_test_opts_t *gopts; + volatile pid_t *pid; +} mmap_changing_thread_args; + static void sleep_short(void) { usleep(1000); @@ -1337,7 +1338,9 @@ static void thread_state_until(pid_t tid, thread_stat= e state) =20 static void *uffd_mmap_changing_thread(void *opaque) { - volatile pid_t *pid =3D opaque; + mmap_changing_thread_args *args =3D opaque; + uffd_global_test_opts_t *gopts =3D args->gopts; + volatile pid_t *pid =3D args->pid; int ret; =20 /* Unfortunately, it's only fetch-able from the thread itself.. */ @@ -1345,21 +1348,21 @@ static void *uffd_mmap_changing_thread(void *opaque) *pid =3D syscall(SYS_gettid); =20 /* Inject an event, this will hang solid until the event read */ - ret =3D madvise(area_dst, page_size, MADV_REMOVE); + ret =3D madvise(gopts->area_dst, gopts->page_size, MADV_REMOVE); if (ret) err("madvise(MADV_REMOVE) failed"); =20 return NULL; } =20 -static void uffd_consume_message(int fd) +static void uffd_consume_message(uffd_global_test_opts_t *gopts) { struct uffd_msg msg =3D { 0 }; =20 - while (uffd_read_msg(fd, &msg)); + while (uffd_read_msg(gopts, &msg)); } =20 -static void uffd_mmap_changing_test(uffd_test_args_t *targs) +static void uffd_mmap_changing_test(uffd_global_test_opts_t *gopts, uffd_t= est_args_t *targs) { /* * This stores the real PID (which can be different from how tid is @@ -1368,13 +1371,14 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) pid_t pid =3D 0; pthread_t tid; int ret; + mmap_changing_thread_args args =3D { gopts, &pid }; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("uffd_register() failed"); =20 /* Create a thread to generate the racy event */ - ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &pid); + ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &args); if (ret) err("pthread_create() failed"); =20 @@ -1388,26 +1392,26 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) /* Wait until the thread hangs at REMOVE event */ thread_state_until(pid, THR_STATE_UNINTERRUPTIBLE); =20 - if (!uffdio_mmap_changing_test_copy(uffd)) + if (!uffdio_mmap_changing_test_copy(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_zeropage(uffd)) + if (!uffdio_mmap_changing_test_zeropage(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_move(uffd)) + if (!uffdio_mmap_changing_test_move(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_poison(uffd)) + if (!uffdio_mmap_changing_test_poison(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_continue(uffd)) + if (!uffdio_mmap_changing_test_continue(gopts->uffd)) return; =20 /* * All succeeded above! Recycle everything. Start by reading the * event so as to kick the thread roll again.. */ - uffd_consume_message(uffd); + uffd_consume_message(gopts); =20 ret =3D pthread_join(tid, NULL); assert(ret =3D=3D 0); @@ -1415,10 +1419,10 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) uffd_test_pass(); } =20 -static int prevent_hugepages(const char **errmsg) +static int prevent_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_NOHUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) { /* Ignore only if CONFIG_TRANSPARENT_HUGEPAGE=3Dn */ if (errno !=3D EINVAL) { if (errmsg) @@ -1429,10 +1433,10 @@ static int prevent_hugepages(const char **errmsg) return 0; } =20 -static int request_hugepages(const char **errmsg) +static int request_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_HUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) { if (errmsg) { *errmsg =3D (errno =3D=3D EINVAL) ? "CONFIG_TRANSPARENT_HUGEPAGE is not set" : @@ -1456,13 +1460,17 @@ struct uffd_test_case_ops uffd_move_test_pmd_case_o= ps =3D { * Note that _UFFDIO_ZEROPAGE is tested separately in the zeropage test. */ static void -do_register_ioctls_test(uffd_test_args_t *args, bool miss, bool wp, bool m= inor) +do_register_ioctls_test(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, + bool miss, + bool wp, + bool minor) { uint64_t ioctls =3D 0, expected =3D BIT_ULL(_UFFDIO_WAKE); mem_type_t *mem_type =3D args->mem_type; int ret; =20 - ret =3D uffd_register_with_ioctls(uffd, area_dst, page_size, + ret =3D uffd_register_with_ioctls(gopts->uffd, gopts->area_dst, gopts->pa= ge_size, miss, wp, minor, &ioctls); =20 /* @@ -1493,18 +1501,18 @@ do_register_ioctls_test(uffd_test_args_t *args, boo= l miss, bool wp, bool minor) "(miss=3D%d, wp=3D%d, minor=3D%d): expected=3D0x%"PRIx64", " "returned=3D0x%"PRIx64, miss, wp, minor, expected, ioctls); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); } =20 -static void uffd_register_ioctls_test(uffd_test_args_t *args) +static void uffd_register_ioctls_test(uffd_global_test_opts_t *gopts, uffd= _test_args_t *args) { int miss, wp, minor; =20 for (miss =3D 0; miss <=3D 1; miss++) for (wp =3D 0; wp <=3D 1; wp++) for (minor =3D 0; minor <=3D 1; minor++) - do_register_ioctls_test(args, miss, wp, minor); + do_register_ioctls_test(gopts, args, miss, wp, minor); =20 uffd_test_pass(); } @@ -1742,6 +1750,28 @@ int main(int argc, char *argv[]) } for (j =3D 0; j < n_mems; j++) { mem_type =3D &mem_types[j]; + + /* Initialize global test options */ + uffd_global_test_opts_t gopts =3D { 0 }; + + gopts.map_shared =3D mem_type->shared; + uffd_test_ops =3D mem_type->mem_ops; + uffd_test_case_ops =3D test->test_case_ops; + + if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) + gopts.page_size =3D default_huge_page_size(); + else + gopts.page_size =3D psize(); + + /* Ensure we have at least 2 pages */ + gopts.nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, gopts.page_size * 2) + / gopts.page_size; + + gopts.nr_parallel =3D 1; + + /* Initialize test arguments */ + args.mem_type =3D mem_type; + if (!(test->mem_targets & mem_type->mem_flag)) continue; =20 @@ -1756,13 +1786,12 @@ int main(int argc, char *argv[]) uffd_test_skip("feature missing"); continue; } - if (uffd_setup_environment(&args, test, mem_type, - &errmsg)) { + if (uffd_test_ctx_init(&gopts, test->uffd_feature_required, &errmsg)) { uffd_test_skip(errmsg); continue; } - test->uffd_fn(&args); - uffd_test_ctx_clear(); + test->uffd_fn(&gopts, &args); + uffd_test_ctx_clear(&gopts); } } =20 diff --git a/tools/testing/selftests/mm/uffd-wp-mremap.c b/tools/testing/se= lftests/mm/uffd-wp-mremap.c index c2ba7d46c7b4..c9da7be91b64 100644 --- a/tools/testing/selftests/mm/uffd-wp-mremap.c +++ b/tools/testing/selftests/mm/uffd-wp-mremap.c @@ -157,7 +157,11 @@ static bool range_is_swapped(void *addr, size_t size) return true; } =20 -static void test_one_folio(size_t size, bool private, bool swapout, bool h= ugetlb) +static void test_one_folio(uffd_global_test_opts_t *gopts, + size_t size, + bool private, + bool swapout, + bool hugetlb) { struct uffdio_writeprotect wp_prms; uint64_t features =3D 0; @@ -181,21 +185,21 @@ static void test_one_folio(size_t size, bool private,= bool swapout, bool hugetlb } =20 /* Register range for uffd-wp. */ - if (userfaultfd_open(&features)) { + if (userfaultfd_open(gopts, &features)) { if (errno =3D=3D ENOENT) ksft_test_result_skip("userfaultfd not available\n"); else ksft_test_result_fail("userfaultfd_open() failed\n"); goto out; } - if (uffd_register(uffd, mem, size, false, true, false)) { + if (uffd_register(gopts->uffd, mem, size, false, true, false)) { ksft_test_result_fail("uffd_register() failed\n"); goto out; } wp_prms.mode =3D UFFDIO_WRITEPROTECT_MODE_WP; wp_prms.range.start =3D (uintptr_t)mem; wp_prms.range.len =3D size; - if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { + if (ioctl(gopts->uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { ksft_test_result_fail("ioctl(UFFDIO_WRITEPROTECT) failed\n"); goto out; } @@ -242,9 +246,9 @@ static void test_one_folio(size_t size, bool private, b= ool swapout, bool hugetlb out: if (mem) munmap(mem, size); - if (uffd >=3D 0) { - close(uffd); - uffd =3D -1; + if (gopts->uffd >=3D 0) { + close(gopts->uffd); + gopts->uffd =3D -1; } } =20 @@ -336,6 +340,7 @@ static const struct testcase testcases[] =3D { =20 int main(int argc, char **argv) { + uffd_global_test_opts_t gopts =3D { 0 }; struct thp_settings settings; int i, j, plan =3D 0; =20 @@ -367,8 +372,8 @@ int main(int argc, char **argv) const struct testcase *tc =3D &testcases[i]; =20 for (j =3D 0; j < *tc->nr_sizes; j++) - test_one_folio(tc->sizes[j], tc->private, tc->swapout, - tc->hugetlb); + test_one_folio(&gopts, tc->sizes[j], tc->private, + tc->swapout, tc->hugetlb); } =20 /* If THP is supported, restore original THP settings. */ --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DF1327F000; Mon, 16 Jun 2025 10:12:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750068778; cv=none; b=VJLryPM5UUAYDTP60b0O66A4q3W6sDTKh9cPlkdCPEKNBM/QjOr9T9vLff8ddYwEAHek4Bf+iv7DmTaIZ3nbTRupnYtiH3+JR0/BvTkSiUAoPPfeaUao1UM5Iv0MB6WN8PZhul9vp1ZNjIkAGcgs3KWhGMmEPZBqKmzSYtxFezk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750068778; c=relaxed/simple; bh=MW6lHhdy6SAZG/vzrNbDbqUcoafoNOheUnF8+4z3e9E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iiY8NT7341MRTtY2eS3NpKEI/9zcIrf864XMmI2+NmVvab3gc0fDfcJACaYxtH/U4lLY8QnMuXOpqqClX9sVcGsRGgOew5NtEMdv7wiIROLSQaXuhG7j77WX1EImfnIXYPX67hwl/T4f4ZhQKvfgeuZGsJEPjUjI0KPWBaGCgqE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lXRH3uYS; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lXRH3uYS" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-235d6de331fso50618315ad.3; Mon, 16 Jun 2025 03:12:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750068773; x=1750673573; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8CpoHasizp+Vyaehmd2LRubbocj3sLYHIdiu6CVUrFo=; b=lXRH3uYSSECxAPVLT+yhenTyIB02Kbw5dBtRBXc1Ct6XaVvsVUaPeUUahC/eialwx0 hqEqImrCiBAfYVmOUg5y/M5X7GaYZZy3XRuDI5cP6xlollhPrPBLsBEx+B23qbKZ/ycB pCURd05plZbGVFJM5DRIjhdUZuHoNABGSrdQBIlxxvbJuU0edpAE6QnQGrpr34BQ7iSC 63ksQ1BuelRwd08c4JdnrtyPmC6s46ob3OYwG0hyhzjyYVqmeMZ7hYN7eYJcsllDKj4V ot5fqhAEw9KcQIqScdnMslgzIEOEOioDQ/kZUeWAk1b8ifrex1mOub8jduzw8gA3MP2+ Zhyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750068773; x=1750673573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8CpoHasizp+Vyaehmd2LRubbocj3sLYHIdiu6CVUrFo=; b=bY6lbOnNrHWoLhfT0c32jK44EiFcUexzZXaOGKWYsa1Ouao/bJnltlea64bIdaLy5C iH5tNr4adkt48RlhzQDZmPhusAwykP4nqqQpuFtFyCkOheWDrHC8LfNo9zQDBkjCtJrV /kWKUFPzePZSmwidMl4cCQ4MDsCyvBGgpnNOSAybeBLwzYrTqXCkiU+FG7Bzu1zNYTuT YtsOzbQOdLWp+rwPmT6PzBLJzyTPepwxYdvofBVldnQsAAFlTPYZ9oDbgyLNN3foewrC Fu0xksswHMSNps6McdlvRVLOJtCvUVIIXiQNfFwxvh+jxpq1FwT4jbDJfdnoJWIMLRFR HvOw== X-Forwarded-Encrypted: i=1; AJvYcCUStjqSCdpcOWpni287RgA3HkrQDx0IY/LFuFjMhs6ZYXd3PS4jquYnAHE8ODHIDv4xyTNZoP6qvnNzven7vrU=@vger.kernel.org X-Gm-Message-State: AOJu0YxFGUHEtWXLyBVHGc4DzJUghcME06ClqlJJqvLLu+PqgB2RUlwg h/SdwPypvEMQfHkyEsMvXGT3ED2XctyKILHHucYpd5hMXwDRcBaUPBQ= X-Gm-Gg: ASbGncs4jbey/9HI4JZlSngvrTge5Gq/RHDqqaHo43NOw6PjBH2sVKUEqo1GE50ZdHx eb6OfDbXfjaIWY3qeZjmlrHlT1/uTENMoP3KgvzH91GlvZftOyky0+uryBD4JQ7tbimdVb8aE3q ZKMza0HowLbwT7HmM2axCWMLiV0F7hNPqlrImLVvU8duMsJvTQzJfKC7vQUY3IS37afcygIE8nu u1/qyC6eKfhm258ojC6YGcH0++4dIsPWGtiyXJW/hWR+Tbjm+N8a+WNyjtTjRTW6yAxijhRynhw SsJm/8p7qkWxmscBJagqWjri/ctR3D1iXwGTzoQTuctdz0yTjXGOrPsSxriY8/l1Rmc4wlRfsQ= = X-Google-Smtp-Source: AGHT+IF2bXZhhb3vdhidFX9wvsUFOJUKNsxILveA4wQ+DcNApc7xk0X0+T/Ggkc6dMDXswlSI2R2kw== X-Received: by 2002:a17:903:18f:b0:235:eb8b:9968 with SMTP id d9443c01a7336-2366b149d13mr132772815ad.36.1750068772939; Mon, 16 Jun 2025 03:12:52 -0700 (PDT) Received: from debian.ujwal.com ([223.185.134.131]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2365d8a19b0sm57709635ad.61.2025.06.16.03.12.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Jun 2025 03:12:52 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, shuah@kernel.org, peterx@redhat.com, jackmanb@google.com Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Ujwal Kundur Subject: [PATCH v5 1/1] selftests/mm/uffd: Refactor non-composite global vars into struct Date: Mon, 16 Jun 2025 15:34:06 +0530 Message-Id: <20250616100406.2853-1-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor macros and non-composite global variable definitions into a struct that is defined at the start of a test and is passed around instead of relying on global vars. Signed-off-by: Ujwal Kundur Tested-by: Brendan Jackman --- Changes since v4: - define gopts as global within uffd-stress.c to retain existing sigalrm handler logic Changes since v3: - more formatting fixes Changes since v2: - redo patch on mm-new branch Changes since v1: - indentation fixes - squash into single patch to assist bisections tools/testing/selftests/mm/uffd-common.c | 268 ++++----- tools/testing/selftests/mm/uffd-common.h | 78 +-- tools/testing/selftests/mm/uffd-stress.c | 228 ++++---- tools/testing/selftests/mm/uffd-unit-tests.c | 561 ++++++++++--------- tools/testing/selftests/mm/uffd-wp-mremap.c | 23 +- 5 files changed, 616 insertions(+), 542 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selft= ests/mm/uffd-common.c index a37088a23ffe..6067cf4bdae6 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -7,18 +7,30 @@ =20 #include "uffd-common.h" =20 -#define BASE_PMD_ADDR ((void *)(1UL << 30)) - -volatile bool test_uffdio_copy_eexist =3D true; -unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; -int uffd =3D -1, uffd_flags, finished, *pipefd, test_type; -bool map_shared; -bool test_uffdio_wp =3D true; -unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; uffd_test_case_ops_t *uffd_test_case_ops; -atomic_bool ready_for_fork; + +#define BASE_PMD_ADDR ((void *)(1UL << 30)) + +/* pthread_mutex_t starts at page offset 0 */ +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts) +{ + return (pthread_mutex_t *) (area + nr * gopts->page_size); +} + +/* + * count is placed in the page after pthread_mutex_t naturally aligned + * to avoid non alignment faults on non-x86 archs. + */ +volatile unsigned long long *area_count( + char *area, unsigned long nr, + uffd_global_test_opts_t *gopts) +{ + return (volatile unsigned long long *) + ((unsigned long)(area + nr * gopts->page_size + + sizeof(pthread_mutex_t) + sizeof(unsigned long long) - 1) & + ~(unsigned long)(sizeof(unsigned long long) - 1)); +} =20 static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { @@ -40,15 +52,15 @@ static int uffd_mem_fd_create(off_t mem_size, bool huge= tlb) return mem_fd; } =20 -static void anon_release_pages(char *rel_area) +static void anon_release_pages(uffd_global_test_opts_t *gopts, char *rel_a= rea) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } =20 -static int anon_allocate_area(void **alloc_area, bool is_src) +static int anon_allocate_area(uffd_global_test_opts_t *gopts, void **alloc= _area, bool is_src) { - *alloc_area =3D mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, + *alloc_area =3D mmap(NULL, gopts->nr_pages * gopts->page_size, PROT_READ = | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); if (*alloc_area =3D=3D MAP_FAILED) { *alloc_area =3D NULL; @@ -57,31 +69,32 @@ static int anon_allocate_area(void **alloc_area, bool i= s_src) return 0; } =20 -static void noop_alias_mapping(__u64 *start, size_t len, unsigned long off= set) +static void noop_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *star= t, + size_t len, unsigned long offset) { } =20 -static void hugetlb_release_pages(char *rel_area) +static void hugetlb_release_pages(uffd_global_test_opts_t *gopts, char *re= l_area) { - if (!map_shared) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (!gopts->map_shared) { + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } else { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } } =20 -static int hugetlb_allocate_area(void **alloc_area, bool is_src) +static int hugetlb_allocate_area(uffd_global_test_opts_t *gopts, void **al= loc_area, bool is_src) { - off_t size =3D nr_pages * page_size; + off_t size =3D gopts->nr_pages * gopts->page_size; off_t offset =3D is_src ? 0 : size; void *area_alias =3D NULL; char **alloc_area_alias; int mem_fd =3D uffd_mem_fd_create(size * 2, true); =20 *alloc_area =3D mmap(NULL, size, PROT_READ | PROT_WRITE, - (map_shared ? MAP_SHARED : MAP_PRIVATE) | + (gopts->map_shared ? MAP_SHARED : MAP_PRIVATE) | (is_src ? 0 : MAP_NORESERVE), mem_fd, offset); if (*alloc_area =3D=3D MAP_FAILED) { @@ -89,7 +102,7 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) return -errno; } =20 - if (map_shared) { + if (gopts->map_shared) { area_alias =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, mem_fd, offset); if (area_alias =3D=3D MAP_FAILED) @@ -97,9 +110,9 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) } =20 if (is_src) { - alloc_area_alias =3D &area_src_alias; + alloc_area_alias =3D &gopts->area_src_alias; } else { - alloc_area_alias =3D &area_dst_alias; + alloc_area_alias =3D &gopts->area_dst_alias; } if (area_alias) *alloc_area_alias =3D area_alias; @@ -108,24 +121,25 @@ static int hugetlb_allocate_area(void **alloc_area, b= ool is_src) return 0; } =20 -static void hugetlb_alias_mapping(__u64 *start, size_t len, unsigned long = offset) +static void hugetlb_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *s= tart, + size_t len, unsigned long offset) { - if (!map_shared) + if (!gopts->map_shared) return; =20 - *start =3D (unsigned long) area_dst_alias + offset; + *start =3D (unsigned long) gopts->area_dst_alias + offset; } =20 -static void shmem_release_pages(char *rel_area) +static void shmem_release_pages(uffd_global_test_opts_t *gopts, char *rel_= area) { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } =20 -static int shmem_allocate_area(void **alloc_area, bool is_src) +static int shmem_allocate_area(uffd_global_test_opts_t *gopts, void **allo= c_area, bool is_src) { void *area_alias =3D NULL; - size_t bytes =3D nr_pages * page_size, hpage_size =3D read_pmd_pagesize(); + size_t bytes =3D gopts->nr_pages * gopts->page_size, hpage_size =3D read_= pmd_pagesize(); unsigned long offset =3D is_src ? 0 : bytes; char *p =3D NULL, *p_alias =3D NULL; int mem_fd =3D uffd_mem_fd_create(bytes * 2, false); @@ -159,22 +173,23 @@ static int shmem_allocate_area(void **alloc_area, boo= l is_src) err("mmap of anonymous memory failed at %p", p_alias); =20 if (is_src) - area_src_alias =3D area_alias; + gopts->area_src_alias =3D area_alias; else - area_dst_alias =3D area_alias; + gopts->area_dst_alias =3D area_alias; =20 close(mem_fd); return 0; } =20 -static void shmem_alias_mapping(__u64 *start, size_t len, unsigned long of= fset) +static void shmem_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *sta= rt, + size_t len, unsigned long offset) { - *start =3D (unsigned long)area_dst_alias + offset; + *start =3D (unsigned long)gopts->area_dst_alias + offset; } =20 -static void shmem_check_pmd_mapping(void *p, int expect_nr_hpages) +static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *= p, int expect_nr_hpages) { - if (!check_huge_shmem(area_dst_alias, expect_nr_hpages, + if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages, read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); @@ -234,18 +249,18 @@ void uffd_stats_report(struct uffd_args *args, int n_= cpus) printf("\n"); } =20 -int userfaultfd_open(uint64_t *features) +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features) { struct uffdio_api uffdio_api; =20 - uffd =3D uffd_open(UFFD_FLAGS); - if (uffd < 0) + gopts->uffd =3D uffd_open(UFFD_FLAGS); + if (gopts->uffd < 0) return -1; - uffd_flags =3D fcntl(uffd, F_GETFD, NULL); + gopts->uffd_flags =3D fcntl(gopts->uffd, F_GETFD, NULL); =20 uffdio_api.api =3D UFFD_API; uffdio_api.features =3D *features; - if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + if (ioctl(gopts->uffd, UFFDIO_API, &uffdio_api)) /* Probably lack of CAP_PTRACE? */ return -1; if (uffdio_api.api !=3D UFFD_API) @@ -255,59 +270,59 @@ int userfaultfd_open(uint64_t *features) return 0; } =20 -static inline void munmap_area(void **area) +static inline void munmap_area(uffd_global_test_opts_t *gopts, void **area) { if (*area) - if (munmap(*area, nr_pages * page_size)) + if (munmap(*area, gopts->nr_pages * gopts->page_size)) err("munmap"); =20 *area =3D NULL; } =20 -void uffd_test_ctx_clear(void) +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts) { size_t i; =20 - if (pipefd) { - for (i =3D 0; i < nr_parallel * 2; ++i) { - if (close(pipefd[i])) + if (gopts->pipefd) { + for (i =3D 0; i < gopts->nr_parallel * 2; ++i) { + if (close(gopts->pipefd[i])) err("close pipefd"); } - free(pipefd); - pipefd =3D NULL; + free(gopts->pipefd); + gopts->pipefd =3D NULL; } =20 - if (count_verify) { - free(count_verify); - count_verify =3D NULL; + if (gopts->count_verify) { + free(gopts->count_verify); + gopts->count_verify =3D NULL; } =20 - if (uffd !=3D -1) { - if (close(uffd)) + if (gopts->uffd !=3D -1) { + if (close(gopts->uffd)) err("close uffd"); - uffd =3D -1; + gopts->uffd =3D -1; } =20 - munmap_area((void **)&area_src); - munmap_area((void **)&area_src_alias); - munmap_area((void **)&area_dst); - munmap_area((void **)&area_dst_alias); - munmap_area((void **)&area_remap); + munmap_area(gopts, (void **)&gopts->area_src); + munmap_area(gopts, (void **)&gopts->area_src_alias); + munmap_area(gopts, (void **)&gopts->area_dst); + munmap_area(gopts, (void **)&gopts->area_dst_alias); + munmap_area(gopts, (void **)&gopts->area_remap); } =20 -int uffd_test_ctx_init(uint64_t features, const char **errmsg) +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg) { unsigned long nr, cpu; int ret; =20 if (uffd_test_case_ops && uffd_test_case_ops->pre_alloc) { - ret =3D uffd_test_case_ops->pre_alloc(errmsg); + ret =3D uffd_test_case_ops->pre_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D uffd_test_ops->allocate_area((void **)&area_src, true); - ret |=3D uffd_test_ops->allocate_area((void **)&area_dst, false); + ret =3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_src, t= rue); + ret |=3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_dst, = false); if (ret) { if (errmsg) *errmsg =3D "memory allocation failed"; @@ -315,26 +330,26 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) } =20 if (uffd_test_case_ops && uffd_test_case_ops->post_alloc) { - ret =3D uffd_test_case_ops->post_alloc(errmsg); + ret =3D uffd_test_case_ops->post_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D userfaultfd_open(&features); + ret =3D userfaultfd_open(gopts, &features); if (ret) { if (errmsg) *errmsg =3D "possible lack of privilege"; return ret; } =20 - count_verify =3D malloc(nr_pages * sizeof(unsigned long long)); - if (!count_verify) + gopts->count_verify =3D malloc(gopts->nr_pages * sizeof(unsigned long lon= g)); + if (!gopts->count_verify) err("count_verify"); =20 - for (nr =3D 0; nr < nr_pages; nr++) { - *area_mutex(area_src, nr) =3D + for (nr =3D 0; nr < gopts->nr_pages; nr++) { + *area_mutex(gopts->area_src, nr, gopts) =3D (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER; - count_verify[nr] =3D *area_count(area_src, nr) =3D 1; + gopts->count_verify[nr] =3D *area_count(gopts->area_src, nr, gopts) =3D = 1; /* * In the transition between 255 to 256, powerpc will * read out of order in my_bcmp and see both bytes as @@ -342,7 +357,7 @@ int uffd_test_ctx_init(uint64_t features, const char **= errmsg) * after the count, to avoid my_bcmp to trigger false * positives. */ - *(area_count(area_src, nr) + 1) =3D 1; + *(area_count(gopts->area_src, nr, gopts) + 1) =3D 1; } =20 /* @@ -363,13 +378,13 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) * proactively split the thp and drop any accidentally initialized * pages within area_dst. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - pipefd =3D malloc(sizeof(int) * nr_parallel * 2); - if (!pipefd) + gopts->pipefd =3D malloc(sizeof(int) * gopts->nr_parallel * 2); + if (!gopts->pipefd) err("pipefd"); - for (cpu =3D 0; cpu < nr_parallel; cpu++) - if (pipe2(&pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) + if (pipe2(&gopts->pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) err("pipe"); =20 return 0; @@ -416,9 +431,9 @@ static void continue_range(int ufd, __u64 start, __u64 = len, bool wp) ret, (int64_t) req.mapped); } =20 -int uffd_read_msg(int ufd, struct uffd_msg *msg) +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg) { - int ret =3D read(uffd, msg, sizeof(*msg)); + int ret =3D read(gopts->uffd, msg, sizeof(*msg)); =20 if (ret !=3D sizeof(*msg)) { if (ret < 0) { @@ -433,7 +448,8 @@ int uffd_read_msg(int ufd, struct uffd_msg *msg) return 0; } =20 -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args) +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, struct uffd_ms= g *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -442,7 +458,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) =20 if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP) { /* Write protect page faults */ - wp_range(uffd, msg->arg.pagefault.address, page_size, false); + wp_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, fals= e); args->wp_faults++; } else if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR) { uint8_t *area; @@ -460,12 +476,12 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) * (UFFD-registered). */ =20 - area =3D (uint8_t *)(area_dst + - ((char *)msg->arg.pagefault.address - - area_dst_alias)); - for (b =3D 0; b < page_size; ++b) + area =3D (uint8_t *)(gopts->area_dst + + ((char *)msg->arg.pagefault.address - + gopts->area_dst_alias)); + for (b =3D 0; b < gopts->page_size; ++b) area[b] =3D ~area[b]; - continue_range(uffd, msg->arg.pagefault.address, page_size, + continue_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, args->apply_wp); args->minor_faults++; } else { @@ -493,10 +509,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WRITE) err("unexpected write fault"); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->ar= ea_dst; + offset &=3D ~(gopts->page_size-1); =20 - if (copy_page(uffd, offset, args->apply_wp)) + if (copy_page(gopts, offset, args->apply_wp)) args->missing_faults++; } } @@ -504,6 +520,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) void *uffd_poll_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; unsigned long cpu =3D args->cpu; struct pollfd pollfd[2]; struct uffd_msg msg; @@ -514,12 +531,12 @@ void *uffd_poll_thread(void *arg) if (!args->handle_fault) args->handle_fault =3D uffd_handle_page_fault; =20 - pollfd[0].fd =3D uffd; + pollfd[0].fd =3D gopts->uffd; pollfd[0].events =3D POLLIN; - pollfd[1].fd =3D pipefd[cpu*2]; + pollfd[1].fd =3D gopts->pipefd[cpu*2]; pollfd[1].events =3D POLLIN; =20 - ready_for_fork =3D true; + gopts->ready_for_fork =3D true; =20 for (;;) { ret =3D poll(pollfd, 2, -1); @@ -537,30 +554,30 @@ void *uffd_poll_thread(void *arg) } if (!(pollfd[0].revents & POLLIN)) err("pollfd[0].revents %d", pollfd[0].revents); - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; switch (msg.event) { default: err("unexpected msg event %u\n", msg.event); break; case UFFD_EVENT_PAGEFAULT: - args->handle_fault(&msg, args); + args->handle_fault(gopts, &msg, args); break; case UFFD_EVENT_FORK: - close(uffd); - uffd =3D msg.arg.fork.ufd; - pollfd[0].fd =3D uffd; + close(gopts->uffd); + gopts->uffd =3D msg.arg.fork.ufd; + pollfd[0].fd =3D gopts->uffd; break; case UFFD_EVENT_REMOVE: uffd_reg.range.start =3D msg.arg.remove.start; uffd_reg.range.len =3D msg.arg.remove.end - msg.arg.remove.start; - if (ioctl(uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) + if (ioctl(gopts->uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) err("remove failure"); break; case UFFD_EVENT_REMAP: - area_remap =3D area_dst; /* save for later unmap */ - area_dst =3D (char *)(unsigned long)msg.arg.remap.to; + gopts->area_remap =3D gopts->area_dst; /* save for later unmap */ + gopts->area_dst =3D (char *)(unsigned long)msg.arg.remap.to; break; } } @@ -568,17 +585,18 @@ void *uffd_poll_thread(void *arg) return NULL; } =20 -static void retry_copy_page(int ufd, struct uffdio_copy *uffdio_copy, +static void retry_copy_page(uffd_global_test_opts_t *gopts, struct uffdio_= copy *uffdio_copy, unsigned long offset) { - uffd_test_ops->alias_mapping(&uffdio_copy->dst, + uffd_test_ops->alias_mapping(gopts, + &uffdio_copy->dst, uffdio_copy->len, offset); - if (ioctl(ufd, UFFDIO_COPY, uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy->copy !=3D -EEXIST) err("UFFDIO_COPY retry error: %"PRId64, - (int64_t)uffdio_copy->copy); + (int64_t)uffdio_copy->copy); } else { err("UFFDIO_COPY retry unexpected: %"PRId64, (int64_t)uffdio_copy->copy); @@ -597,60 +615,60 @@ static void wake_range(int ufd, unsigned long addr, u= nsigned long len) addr), exit(1); } =20 -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp) +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp) { struct uffdio_copy uffdio_copy; =20 - if (offset >=3D nr_pages * page_size) + if (offset >=3D gopts->nr_pages * gopts->page_size) err("unexpected offset %lu\n", offset); - uffdio_copy.dst =3D (unsigned long) area_dst + offset; - uffdio_copy.src =3D (unsigned long) area_src + offset; - uffdio_copy.len =3D page_size; + uffdio_copy.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_copy.src =3D (unsigned long) gopts->area_src + offset; + uffdio_copy.len =3D gopts->page_size; if (wp) uffdio_copy.mode =3D UFFDIO_COPY_MODE_WP; else uffdio_copy.mode =3D 0; uffdio_copy.copy =3D 0; - if (ioctl(ufd, UFFDIO_COPY, &uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, &uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy.copy !=3D -EEXIST) err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); - wake_range(ufd, uffdio_copy.dst, page_size); - } else if (uffdio_copy.copy !=3D page_size) { + wake_range(gopts->uffd, uffdio_copy.dst, gopts->page_size); + } else if (uffdio_copy.copy !=3D gopts->page_size) { err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); } else { - if (test_uffdio_copy_eexist && retry) { - test_uffdio_copy_eexist =3D false; - retry_copy_page(ufd, &uffdio_copy, offset); + if (gopts->test_uffdio_copy_eexist && retry) { + gopts->test_uffdio_copy_eexist =3D false; + retry_copy_page(gopts, &uffdio_copy, offset); } return 1; } return 0; } =20 -int copy_page(int ufd, unsigned long offset, bool wp) +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p) { - return __copy_page(ufd, offset, false, wp); + return __copy_page(gopts, offset, false, wp); } =20 -int move_page(int ufd, unsigned long offset, unsigned long len) +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len) { struct uffdio_move uffdio_move; =20 - if (offset + len > nr_pages * page_size) + if (offset + len > gopts->nr_pages * gopts->page_size) err("unexpected offset %lu and length %lu\n", offset, len); - uffdio_move.dst =3D (unsigned long) area_dst + offset; - uffdio_move.src =3D (unsigned long) area_src + offset; + uffdio_move.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_move.src =3D (unsigned long) gopts->area_src + offset; uffdio_move.len =3D len; uffdio_move.mode =3D UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES; uffdio_move.move =3D 0; - if (ioctl(ufd, UFFDIO_MOVE, &uffdio_move)) { + if (ioctl(gopts->uffd, UFFDIO_MOVE, &uffdio_move)) { /* real retval in uffdio_move.move */ if (uffdio_move.move !=3D -EEXIST) err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); - wake_range(ufd, uffdio_move.dst, len); + wake_range(gopts->uffd, uffdio_move.dst, len); } else if (uffdio_move.move !=3D len) { err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); } else diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selft= ests/mm/uffd-common.h index 7700cbfa3975..37d3ca55905f 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -56,20 +56,17 @@ =20 #define err(fmt, ...) errexit(1, fmt, ##__VA_ARGS__) =20 -/* pthread_mutex_t starts at page offset 0 */ -#define area_mutex(___area, ___nr) \ - ((pthread_mutex_t *) ((___area) + (___nr)*page_size)) -/* - * count is placed in the page after pthread_mutex_t naturally aligned - * to avoid non alignment faults on non-x86 archs. - */ -#define area_count(___area, ___nr) \ - ((volatile unsigned long long *) ((unsigned long) \ - ((___area) + (___nr)*page_size + \ - sizeof(pthread_mutex_t) + \ - sizeof(unsigned long long) - 1) & \ - ~(unsigned long)(sizeof(unsigned long long) \ - - 1))) +struct uffd_global_test_opts { + unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; + char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; + int uffd, uffd_flags, finished, *pipefd, test_type; + bool map_shared; + bool test_uffdio_wp; + unsigned long long *count_verify; + volatile bool test_uffdio_copy_eexist; + atomic_bool ready_for_fork; +}; +typedef struct uffd_global_test_opts uffd_global_test_opts_t; =20 /* Userfaultfd test statistics */ struct uffd_args { @@ -79,50 +76,55 @@ struct uffd_args { unsigned long missing_faults; unsigned long wp_faults; unsigned long minor_faults; + struct uffd_global_test_opts *gopts; =20 /* A custom fault handler; defaults to uffd_handle_page_fault. */ - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args); + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, + struct uffd_args *args); }; =20 struct uffd_test_ops { - int (*allocate_area)(void **alloc_area, bool is_src); - void (*release_pages)(char *rel_area); - void (*alias_mapping)(__u64 *start, size_t len, unsigned long offset); - void (*check_pmd_mapping)(void *p, int expect_nr_hpages); + int (*allocate_area)(uffd_global_test_opts_t *gopts, void **alloc_area, b= ool is_src); + void (*release_pages)(uffd_global_test_opts_t *gopts, char *rel_area); + void (*alias_mapping)(uffd_global_test_opts_t *gopts, + __u64 *start, + size_t len, + unsigned long offset); + void (*check_pmd_mapping)(uffd_global_test_opts_t *gopts, void *p, int ex= pect_nr_hpages); }; typedef struct uffd_test_ops uffd_test_ops_t; =20 struct uffd_test_case_ops { - int (*pre_alloc)(const char **errmsg); - int (*post_alloc)(const char **errmsg); + int (*pre_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); + int (*post_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); }; typedef struct uffd_test_case_ops uffd_test_case_ops_t; =20 -extern unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -extern char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_= remap; -extern int uffd, uffd_flags, finished, *pipefd, test_type; -extern bool map_shared; -extern bool test_uffdio_wp; -extern unsigned long long *count_verify; -extern volatile bool test_uffdio_copy_eexist; -extern atomic_bool ready_for_fork; - +extern uffd_global_test_opts_t *uffd_gtest_opts; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; extern uffd_test_ops_t hugetlb_uffd_test_ops; extern uffd_test_ops_t *uffd_test_ops; extern uffd_test_case_ops_t *uffd_test_case_ops; =20 +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts); +volatile unsigned long long *area_count(char *area, + unsigned long nr, + uffd_global_test_opts_t *gopts); + void uffd_stats_report(struct uffd_args *args, int n_cpus); -int uffd_test_ctx_init(uint64_t features, const char **errmsg); -void uffd_test_ctx_clear(void); -int userfaultfd_open(uint64_t *features); -int uffd_read_msg(int ufd, struct uffd_msg *msg); +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg); +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts); +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features); +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg); void wp_range(int ufd, __u64 start, __u64 len, bool wp); -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); -int copy_page(int ufd, unsigned long offset, bool wp); -int move_page(int ufd, unsigned long offset, unsigned long len); +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args); +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp); +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p); +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len); void *uffd_poll_thread(void *arg); =20 int uffd_open_dev(unsigned int flags); diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selft= ests/mm/uffd-stress.c index 40af7f67c407..1b69e10dc284 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -44,6 +44,12 @@ uint64_t features; #define BOUNCE_VERIFY (1<<2) #define BOUNCE_POLL (1<<3) static int bounces; +/* defined globally for this particular test as the sigalrm handler + * depends on test_uffdio_*_eexist. + * XXX: define gopts in main() when we figure out a way to deal with + * test_uffdio_*_eexist. + */ +static uffd_global_test_opts_t *gopts; =20 /* exercise the test_uffdio_*_eexist every ALARM_INTERVAL_SECS */ #define ALARM_INTERVAL_SECS 10 @@ -76,54 +82,58 @@ static void usage(void) exit(1); } =20 -static void uffd_stats_reset(struct uffd_args *args, unsigned long n_cpus) +static void uffd_stats_reset(uffd_global_test_opts_t *gopts, struct uffd_a= rgs *args, + unsigned long n_cpus) { int i; =20 for (i =3D 0; i < n_cpus; i++) { args[i].cpu =3D i; - args[i].apply_wp =3D test_uffdio_wp; + args[i].apply_wp =3D gopts->test_uffdio_wp; args[i].missing_faults =3D 0; args[i].wp_faults =3D 0; args[i].minor_faults =3D 0; + args[i].gopts =3D gopts; } } =20 static void *locking_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr; unsigned long long count; =20 if (!(bounces & BOUNCE_RANDOM)) { page_nr =3D -bounces; if (!(bounces & BOUNCE_RACINGFAULTS)) - page_nr +=3D cpu * nr_pages_per_cpu; + page_nr +=3D cpu * gopts->nr_pages_per_cpu; } =20 - while (!finished) { + while (!gopts->finished) { if (bounces & BOUNCE_RANDOM) { if (getrandom(&page_nr, sizeof(page_nr), 0) !=3D sizeof(page_nr)) err("getrandom failed"); } else page_nr +=3D 1; - page_nr %=3D nr_pages; - pthread_mutex_lock(area_mutex(area_dst, page_nr)); - count =3D *area_count(area_dst, page_nr); - if (count !=3D count_verify[page_nr]) + page_nr %=3D gopts->nr_pages; + pthread_mutex_lock(area_mutex(gopts->area_dst, page_nr, gopts)); + count =3D *area_count(gopts->area_dst, page_nr, gopts); + if (count !=3D gopts->count_verify[page_nr]) err("page_nr %lu memory corruption %llu %llu", - page_nr, count, count_verify[page_nr]); + page_nr, count, gopts->count_verify[page_nr]); count++; - *area_count(area_dst, page_nr) =3D count_verify[page_nr] =3D count; - pthread_mutex_unlock(area_mutex(area_dst, page_nr)); + *area_count(gopts->area_dst, page_nr, gopts) =3D gopts->count_verify[pag= e_nr] =3D count; + pthread_mutex_unlock(area_mutex(gopts->area_dst, page_nr, gopts)); } =20 return NULL; } =20 -static int copy_page_retry(int ufd, unsigned long offset) +static int copy_page_retry(uffd_global_test_opts_t *gopts, unsigned long o= ffset) { - return __copy_page(ufd, offset, true, test_uffdio_wp); + return __copy_page(gopts, offset, true, gopts->test_uffdio_wp); } =20 pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INITIALIZER; @@ -131,15 +141,16 @@ pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INI= TIALIZER; static void *uffd_read_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; struct uffd_msg msg; =20 pthread_mutex_unlock(&uffd_read_mutex); /* from here cancellation is ok */ =20 for (;;) { - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; - uffd_handle_page_fault(&msg, args); + uffd_handle_page_fault(gopts, &msg, args); } =20 return NULL; @@ -147,32 +158,34 @@ static void *uffd_read_thread(void *arg) =20 static void *background_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr, start_nr, mid_nr, end_nr; =20 - start_nr =3D cpu * nr_pages_per_cpu; - end_nr =3D (cpu+1) * nr_pages_per_cpu; + start_nr =3D cpu * gopts->nr_pages_per_cpu; + end_nr =3D (cpu+1) * gopts->nr_pages_per_cpu; mid_nr =3D (start_nr + end_nr) / 2; =20 /* Copy the first half of the pages */ for (page_nr =3D start_nr; page_nr < mid_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 /* * If we need to test uffd-wp, set it up now. Then we'll have * at least the first half of the pages mapped already which * can be write-protected for testing */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst + start_nr * page_size, - nr_pages_per_cpu * page_size, true); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst + start_nr * gopts-= >page_size, + gopts->nr_pages_per_cpu * gopts->page_size, true); =20 /* * Continue the 2nd half of the page copying, handling write * protection faults if any */ for (page_nr =3D mid_nr; page_nr < end_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 return NULL; } @@ -180,17 +193,21 @@ static void *background_thread(void *arg) static int stress(struct uffd_args *args) { unsigned long cpu; - pthread_t locking_threads[nr_parallel]; - pthread_t uffd_threads[nr_parallel]; - pthread_t background_threads[nr_parallel]; + uffd_global_test_opts_t *gopts =3D args->gopts; + pthread_t locking_threads[gopts->nr_parallel]; + pthread_t uffd_threads[gopts->nr_parallel]; + pthread_t background_threads[gopts->nr_parallel]; =20 - finished =3D 0; - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + gopts->finished =3D 0; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { if (pthread_create(&locking_threads[cpu], &attr, - locking_thread, (void *)cpu)) + locking_thread, (void *)&args[cpu])) return 1; if (bounces & BOUNCE_POLL) { - if (pthread_create(&uffd_threads[cpu], &attr, uffd_poll_thread, &args[c= pu])) + if (pthread_create(&uffd_threads[cpu], + &attr, + uffd_poll_thread, + (void *) &args[cpu])) err("uffd_poll_thread create"); } else { if (pthread_create(&uffd_threads[cpu], &attr, @@ -200,10 +217,10 @@ static int stress(struct uffd_args *args) pthread_mutex_lock(&uffd_read_mutex); } if (pthread_create(&background_threads[cpu], &attr, - background_thread, (void *)cpu)) + background_thread, (void *)&args[cpu])) return 1; } - for (cpu =3D 0; cpu < nr_parallel; cpu++) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(background_threads[cpu], NULL)) return 1; =20 @@ -216,17 +233,17 @@ static int stress(struct uffd_args *args) * UFFDIO_COPY without writing zero pages into area_dst * because the background threads already completed). */ - uffd_test_ops->release_pages(area_src); + uffd_test_ops->release_pages(gopts, gopts->area_src); =20 - finished =3D 1; - for (cpu =3D 0; cpu < nr_parallel; cpu++) + gopts->finished =3D 1; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(locking_threads[cpu], NULL)) return 1; =20 - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { char c; if (bounces & BOUNCE_POLL) { - if (write(pipefd[cpu*2+1], &c, 1) !=3D 1) + if (write(gopts->pipefd[cpu*2+1], &c, 1) !=3D 1) err("pipefd write error"); if (pthread_join(uffd_threads[cpu], (void *)&args[cpu])) @@ -242,26 +259,26 @@ static int stress(struct uffd_args *args) return 0; } =20 -static int userfaultfd_stress(void) +static int userfaultfd_stress(uffd_global_test_opts_t *gopts) { void *area; unsigned long nr; - struct uffd_args args[nr_parallel]; - uint64_t mem_size =3D nr_pages * page_size; + struct uffd_args args[gopts->nr_parallel]; + uint64_t mem_size =3D gopts->nr_pages * gopts->page_size; int flags =3D 0; =20 - memset(args, 0, sizeof(struct uffd_args) * nr_parallel); + memset(args, 0, sizeof(struct uffd_args) * gopts->nr_parallel); =20 - if (features & UFFD_FEATURE_WP_UNPOPULATED && test_type =3D=3D TEST_ANON) + if (features & UFFD_FEATURE_WP_UNPOPULATED && gopts->test_type =3D=3D TES= T_ANON) flags =3D UFFD_FEATURE_WP_UNPOPULATED; =20 - if (uffd_test_ctx_init(flags, NULL)) + if (uffd_test_ctx_init(gopts, flags, NULL)) err("context init failed"); =20 - if (posix_memalign(&area, page_size, page_size)) + if (posix_memalign(&area, gopts->page_size, gopts->page_size)) err("out of memory"); zeropage =3D area; - bzero(zeropage, page_size); + bzero(zeropage, gopts->page_size); =20 pthread_mutex_lock(&uffd_read_mutex); =20 @@ -284,18 +301,18 @@ static int userfaultfd_stress(void) fflush(stdout); =20 if (bounces & BOUNCE_POLL) - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); else - fcntl(uffd, F_SETFL, uffd_flags & ~O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags & ~O_NONBLOCK); =20 /* register */ - if (uffd_register(uffd, area_dst, mem_size, - true, test_uffdio_wp, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure"); =20 - if (area_dst_alias) { - if (uffd_register(uffd, area_dst_alias, mem_size, - true, test_uffdio_wp, false)) + if (gopts->area_dst_alias) { + if (uffd_register(gopts->uffd, gopts->area_dst_alias, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure alias"); } =20 @@ -323,87 +340,88 @@ static int userfaultfd_stress(void) * MADV_DONTNEED only after the UFFDIO_REGISTER, so it's * required to MADV_DONTNEED here. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - uffd_stats_reset(args, nr_parallel); + uffd_stats_reset(gopts, args, gopts->nr_parallel); =20 /* bounce pass */ if (stress(args)) { - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); return 1; } =20 /* Clear all the write protections if there is any */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst, - nr_pages * page_size, false); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst, + gopts->nr_pages * gopts->page_size, false); =20 /* unregister */ - if (uffd_unregister(uffd, area_dst, mem_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, mem_size)) err("unregister failure"); - if (area_dst_alias) { - if (uffd_unregister(uffd, area_dst_alias, mem_size)) + if (gopts->area_dst_alias) { + if (uffd_unregister(gopts->uffd, gopts->area_dst_alias, mem_size)) err("unregister failure alias"); } =20 /* verification */ if (bounces & BOUNCE_VERIFY) - for (nr =3D 0; nr < nr_pages; nr++) - if (*area_count(area_dst, nr) !=3D count_verify[nr]) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + if (*area_count(gopts->area_dst, nr, gopts) !=3D + gopts->count_verify[nr]) err("error area_count %llu %llu %lu\n", - *area_count(area_src, nr), - count_verify[nr], nr); + *area_count(gopts->area_src, nr, gopts), + gopts->count_verify[nr], nr); =20 /* prepare next bounce */ - swap(area_src, area_dst); + swap(gopts->area_src, gopts->area_dst); =20 - swap(area_src_alias, area_dst_alias); + swap(gopts->area_src_alias, gopts->area_dst_alias); =20 - uffd_stats_report(args, nr_parallel); + uffd_stats_report(args, gopts->nr_parallel); } - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); =20 return 0; } =20 -static void set_test_type(const char *type) +static void set_test_type(uffd_global_test_opts_t *gopts, const char *type) { if (!strcmp(type, "anon")) { - test_type =3D TEST_ANON; + gopts->test_type =3D TEST_ANON; uffd_test_ops =3D &anon_uffd_test_ops; } else if (!strcmp(type, "hugetlb")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; - map_shared =3D true; + gopts->map_shared =3D true; } else if (!strcmp(type, "hugetlb-private")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; } else if (!strcmp(type, "shmem")) { - map_shared =3D true; - test_type =3D TEST_SHMEM; + gopts->map_shared =3D true; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } else if (!strcmp(type, "shmem-private")) { - test_type =3D TEST_SHMEM; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } } =20 -static void parse_test_type_arg(const char *raw_type) +static void parse_test_type_arg(uffd_global_test_opts_t *gopts, const char= *raw_type) { - set_test_type(raw_type); + set_test_type(gopts, raw_type); =20 - if (!test_type) + if (!gopts->test_type) err("failed to parse test type argument: '%s'", raw_type); =20 - if (test_type =3D=3D TEST_HUGETLB) - page_size =3D default_huge_page_size(); + if (gopts->test_type =3D=3D TEST_HUGETLB) + gopts->page_size =3D default_huge_page_size(); else - page_size =3D sysconf(_SC_PAGE_SIZE); + gopts->page_size =3D sysconf(_SC_PAGE_SIZE); =20 - if (!page_size) + if (!gopts->page_size) err("Unable to determine page size"); - if ((unsigned long) area_count(NULL, 0) + sizeof(unsigned long long) * 2 - > page_size) + if ((unsigned long) area_count(NULL, 0, gopts) + sizeof(unsigned long lon= g) * 2 + > gopts->page_size) err("Impossible to run this test"); =20 /* @@ -415,21 +433,21 @@ static void parse_test_type_arg(const char *raw_type) if (uffd_get_features(&features) && errno =3D=3D ENOENT) ksft_exit_skip("failed to get available features (%d)\n", errno); =20 - test_uffdio_wp =3D test_uffdio_wp && + gopts->test_uffdio_wp =3D gopts->test_uffdio_wp && (features & UFFD_FEATURE_PAGEFAULT_FLAG_WP); =20 - if (test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGETLBFS_SH= MEM)) - test_uffdio_wp =3D false; + if (gopts->test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGET= LBFS_SHMEM)) + gopts->test_uffdio_wp =3D false; =20 - close(uffd); - uffd =3D -1; + close(gopts->uffd); + gopts->uffd =3D -1; } =20 static void sigalrm(int sig) { if (sig !=3D SIGALRM) abort(); - test_uffdio_copy_eexist =3D true; + gopts->test_uffdio_copy_eexist =3D true; alarm(ALARM_INTERVAL_SECS); } =20 @@ -438,6 +456,8 @@ int main(int argc, char **argv) unsigned long nr_cpus; size_t bytes; =20 + gopts =3D (uffd_global_test_opts_t *) malloc(sizeof(uffd_global_test_opts= _t)); + if (argc < 4) usage(); =20 @@ -445,11 +465,11 @@ int main(int argc, char **argv) err("failed to arm SIGALRM"); alarm(ALARM_INTERVAL_SECS); =20 - parse_test_type_arg(argv[1]); + parse_test_type_arg(gopts, argv[1]); bytes =3D atol(argv[2]) * 1024 * 1024; =20 - if (test_type =3D=3D TEST_HUGETLB && - get_free_hugepages() < bytes / page_size) { + if (gopts->test_type =3D=3D TEST_HUGETLB && + get_free_hugepages() < bytes / gopts->page_size) { printf("skip: Skipping userfaultfd... not enough hugepages\n"); return KSFT_SKIP; } @@ -459,15 +479,15 @@ int main(int argc, char **argv) /* Don't let calculation below go to zero. */ ksft_print_msg("_SC_NPROCESSORS_ONLN (%lu) too large, capping nr_threads= to 32\n", nr_cpus); - nr_parallel =3D 32; + gopts->nr_parallel =3D 32; } else { - nr_parallel =3D nr_cpus; + gopts->nr_parallel =3D nr_cpus; } =20 - nr_pages_per_cpu =3D bytes / page_size / nr_parallel; - if (!nr_pages_per_cpu) { + gopts->nr_pages_per_cpu =3D bytes / gopts->page_size / gopts->nr_parallel; + if (!gopts->nr_pages_per_cpu) { _err("pages_per_cpu =3D 0, cannot test (%lu / %lu / %lu)", - bytes, page_size, nr_parallel); + bytes, gopts->page_size, gopts->nr_parallel); usage(); } =20 @@ -476,11 +496,11 @@ int main(int argc, char **argv) _err("invalid bounces"); usage(); } - nr_pages =3D nr_pages_per_cpu * nr_parallel; + gopts->nr_pages =3D gopts->nr_pages_per_cpu * gopts->nr_parallel; =20 printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n", - nr_pages, nr_pages_per_cpu); - return userfaultfd_stress(); + gopts->nr_pages, gopts->nr_pages_per_cpu); + return userfaultfd_stress(gopts); } =20 #else /* __NR_userfaultfd */ diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/s= elftests/mm/uffd-unit-tests.c index c73fd5d455c8..bed96f41c578 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -76,7 +76,7 @@ struct uffd_test_args { typedef struct uffd_test_args uffd_test_args_t; =20 /* Returns: UFFD_TEST_* */ -typedef void (*uffd_test_fn)(uffd_test_args_t *); +typedef void (*uffd_test_fn)(uffd_global_test_opts_t *, uffd_test_args_t *= ); =20 typedef struct { const char *name; @@ -181,33 +181,6 @@ static int test_uffd_api(bool use_dev) return 1; } =20 -/* - * This function initializes the global variables. TODO: remove global - * vars and then remove this. - */ -static int -uffd_setup_environment(uffd_test_args_t *args, uffd_test_case_t *test, - mem_type_t *mem_type, const char **errmsg) -{ - map_shared =3D mem_type->shared; - uffd_test_ops =3D mem_type->mem_ops; - uffd_test_case_ops =3D test->test_case_ops; - - if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) - page_size =3D default_huge_page_size(); - else - page_size =3D psize(); - - /* Ensure we have at least 2 pages */ - nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, page_size * 2) / page_size; - /* TODO: remove this global var.. it's so ugly */ - nr_parallel =3D 1; - - /* Initialize test arguments */ - args->mem_type =3D mem_type; - - return uffd_test_ctx_init(test->uffd_feature_required, errmsg); -} =20 static bool uffd_feature_supported(uffd_test_case_t *test) { @@ -237,7 +210,8 @@ static int pagemap_open(void) } while (0) =20 typedef struct { - int parent_uffd, child_uffd; + uffd_global_test_opts_t *gopts; + int child_uffd; } fork_event_args; =20 static void *fork_event_consumer(void *data) @@ -245,10 +219,10 @@ static void *fork_event_consumer(void *data) fork_event_args *args =3D data; struct uffd_msg msg =3D { 0 }; =20 - ready_for_fork =3D true; + args->gopts->ready_for_fork =3D true; =20 /* Read until a full msg received */ - while (uffd_read_msg(args->parent_uffd, &msg)); + while (uffd_read_msg(args->gopts, &msg)); =20 if (msg.event !=3D UFFD_EVENT_FORK) err("wrong message: %u\n", msg.event); @@ -304,9 +278,9 @@ static void unpin_pages(pin_args *args) args->pinned =3D false; } =20 -static int pagemap_test_fork(int uffd, bool with_event, bool test_pin) +static int pagemap_test_fork(uffd_global_test_opts_t *gopts, bool with_eve= nt, bool test_pin) { - fork_event_args args =3D { .parent_uffd =3D uffd, .child_uffd =3D -1 }; + fork_event_args args =3D { .gopts =3D gopts, .child_uffd =3D -1 }; pthread_t thread; pid_t child; uint64_t value; @@ -314,10 +288,10 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 /* Prepare a thread to resolve EVENT_FORK */ if (with_event) { - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; if (pthread_create(&thread, NULL, fork_event_consumer, &args)) err("pthread_create()"); - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ } =20 @@ -328,14 +302,14 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 fd =3D pagemap_open(); =20 - if (test_pin && pin_pages(&args, area_dst, page_size)) + if (test_pin && pin_pages(&args, gopts->area_dst, gopts->page_size)) /* * Normally when reach here we have pinned in * previous tests, so shouldn't fail anymore */ err("pin page failed in child"); =20 - value =3D pagemap_get_entry(fd, area_dst); + value =3D pagemap_get_entry(fd, gopts->area_dst); /* * After fork(), we should handle uffd-wp bit differently: * @@ -361,70 +335,70 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) return result; } =20 -static void uffd_wp_unpopulated_test(uffd_test_args_t *args) +static void uffd_wp_unpopulated_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args) { uint64_t value; int pagemap_fd; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Test applying pte marker to anon unpopulated */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); =20 /* Test unprotect on anon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test zap on anon marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - if (madvise(area_dst, page_size, MADV_DONTNEED)) + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test fault in after marker removed */ - *area_dst =3D 1; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 /* Test read-zero-page upon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - *(volatile char *)area_dst; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + *(volatile char *)gopts->area_dst; /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 uffd_test_pass(); } =20 -static void uffd_wp_fork_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_test_common(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args, bool with_event) { int pagemap_fd; uint64_t value; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in present pte", with_event ? "missing" : "stall"); goto out; @@ -442,79 +416,80 @@ static void uffd_wp_fork_test_common(uffd_test_args_t= *args, * to expose pte markers. */ if (args->mem_type->shared) { - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("MADV_DONTNEED"); } else { /* * NOTE: ignore retval because private-hugetlb doesn't yet * support swapping, so it could fail. */ - madvise(area_dst, page_size, MADV_PAGEOUT); + madvise(gopts->area_dst, gopts->page_size, MADV_PAGEOUT); } =20 /* Uffd-wp should persist even swapped out */ - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in zapped pte", with_event ? "missing" : "stall"); goto out; } =20 /* Unprotect; this tests swap pte modifications */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Fault in the page from disk */ - *area_dst =3D 2; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 2; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, nr_pages * page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts= ->page_size)) err("unregister failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_test(uffd_test_args_t *args) +static void uffd_wp_fork_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_wp_fork_test_common(args, false); + uffd_wp_fork_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_with_event_test(uffd_global_test_opts_t *gopts, u= ffd_test_args_t *args) { - uffd_wp_fork_test_common(args, true); + uffd_wp_fork_test_common(gopts, args, true); } =20 -static void uffd_wp_fork_pin_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_pin_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, bool with_event) { int pagemap_fd; pin_args pin_args =3D {}; =20 - if (uffd_register(uffd, area_dst, page_size, false, true, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->page_size, false, = true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); =20 /* * 1. First pin, then fork(). This tests fork() special path when * doing early CoW if the page is private. */ - if (pin_pages(&pin_args, area_dst, page_size)) { + if (pin_pages(&pin_args, gopts->area_dst, gopts->page_size)) { uffd_test_skip("Possibly CONFIG_GUP_TEST missing " "or unprivileged"); close(pagemap_fd); - uffd_unregister(uffd, area_dst, page_size); + uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size); return; } =20 - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in early CoW of fork()", with_event ? "missing" : "stall"); unpin_pages(&pin_args); @@ -527,49 +502,50 @@ static void uffd_wp_fork_pin_test_common(uffd_test_ar= gs_t *args, * 2. First fork(), then pin (in the child, where test_pin=3D=3Dtrue). * This tests COR, aka, page unsharing on private memories. */ - if (pagemap_test_fork(uffd, with_event, true)) { + if (pagemap_test_fork(gopts, with_event, true)) { uffd_test_fail("Detected %s uffd-wp bit when RO pin", with_event ? "missing" : "stall"); goto out; } uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("register failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_pin_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_test(uffd_global_test_opts_t *gopts, uffd_tes= t_args_t *args) { - uffd_wp_fork_pin_test_common(args, false); + uffd_wp_fork_pin_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_pin_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_with_event_test(uffd_global_test_opts_t *gopt= s, uffd_test_args_t *args) { - uffd_wp_fork_pin_test_common(args, true); + uffd_wp_fork_pin_test_common(gopts, args, true); } =20 -static void check_memory_contents(char *p) +static void check_memory_contents(uffd_global_test_opts_t *gopts, char *p) { unsigned long i, j; uint8_t expected_byte; =20 - for (i =3D 0; i < nr_pages; ++i) { + for (i =3D 0; i < gopts->nr_pages; ++i) { expected_byte =3D ~((uint8_t)(i % ((uint8_t)-1))); - for (j =3D 0; j < page_size; j++) { - uint8_t v =3D *(uint8_t *)(p + (i * page_size) + j); + for (j =3D 0; j < gopts->page_size; j++) { + uint8_t v =3D *(uint8_t *)(p + (i * gopts->page_size) + j); if (v !=3D expected_byte) err("unexpected page contents"); } } } =20 -static void uffd_minor_test_common(bool test_collapse, bool test_wp) +static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool te= st_collapse, bool test_wp) { unsigned long p; pthread_t uffd_mon; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 /* * NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing @@ -577,7 +553,7 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) */ assert(!(test_collapse && test_wp)); =20 - if (uffd_register(uffd, area_dst_alias, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst_alias, gopts->nr_pages * g= opts->page_size, /* NOTE! MADV_COLLAPSE may not work with uffd-wp */ false, test_wp, true)) err("register failure"); @@ -586,9 +562,9 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) * After registering with UFFD, populate the non-UFFD-registered side of * the shared mapping. This should *not* trigger any UFFD minor faults. */ - for (p =3D 0; p < nr_pages; ++p) - memset(area_dst + (p * page_size), p % ((uint8_t)-1), - page_size); + for (p =3D 0; p < gopts->nr_pages; ++p) + memset(gopts->area_dst + (p * gopts->page_size), p % ((uint8_t)-1), + gopts->page_size); =20 args.apply_wp =3D test_wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -600,50 +576,51 @@ static void uffd_minor_test_common(bool test_collapse= , bool test_wp) * fault. uffd_poll_thread will resolve the fault by bit-flipping the * page's contents, and then issuing a CONTINUE ioctl. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); =20 if (test_collapse) { - if (madvise(area_dst_alias, nr_pages * page_size, + if (madvise(gopts->area_dst_alias, gopts->nr_pages * gopts->page_size, MADV_COLLAPSE)) { /* It's fine to fail for this one... */ uffd_test_skip("MADV_COLLAPSE failed"); return; } =20 - uffd_test_ops->check_pmd_mapping(area_dst, - nr_pages * page_size / + uffd_test_ops->check_pmd_mapping(gopts, + gopts->area_dst, + gopts->nr_pages * gopts->page_size / read_pmd_pagesize()); /* * This won't cause uffd-fault - it purely just makes sure there * was no corruption. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); } =20 - if (args.missing_faults !=3D 0 || args.minor_faults !=3D nr_pages) + if (args.missing_faults !=3D 0 || args.minor_faults !=3D gopts->nr_pages) uffd_test_fail("stats check error"); else uffd_test_pass(); } =20 -void uffd_minor_test(uffd_test_args_t *args) +void uffd_minor_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *arg= s) { - uffd_minor_test_common(false, false); + uffd_minor_test_common(gopts, false, false); } =20 -void uffd_minor_wp_test(uffd_test_args_t *args) +void uffd_minor_wp_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *= args) { - uffd_minor_test_common(false, true); + uffd_minor_test_common(gopts, false, true); } =20 -void uffd_minor_collapse_test(uffd_test_args_t *args) +void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_minor_test_common(true, false); + uffd_minor_test_common(gopts, true, false); } =20 static sigjmp_buf jbuf, *sigbuf; @@ -678,7 +655,7 @@ static void sighndl(int sig, siginfo_t *siginfo, void *= ptr) * This also tests UFFD_FEATURE_EVENT_FORK event along with the signal * feature. Using monitor thread, verify no userfault events are generated. */ -static int faulting_process(int signal_test, bool wp) +static int faulting_process(uffd_global_test_opts_t *gopts, int signal_tes= t, bool wp) { unsigned long nr, i; unsigned long long count; @@ -687,7 +664,7 @@ static int faulting_process(int signal_test, bool wp) struct sigaction act; volatile unsigned long signalled =3D 0; =20 - split_nr_pages =3D (nr_pages + 1) / 2; + split_nr_pages =3D (gopts->nr_pages + 1) / 2; =20 if (signal_test) { sigbuf =3D &jbuf; @@ -701,7 +678,7 @@ static int faulting_process(int signal_test, bool wp) =20 for (nr =3D 0; nr < split_nr_pages; nr++) { volatile int steps =3D 1; - unsigned long offset =3D nr * page_size; + unsigned long offset =3D nr * gopts->page_size; =20 if (signal_test) { if (sigsetjmp(*sigbuf, 1) !=3D 0) { @@ -713,15 +690,15 @@ static int faulting_process(int signal_test, bool wp) if (steps =3D=3D 1) { /* This is a MISSING request */ steps++; - if (copy_page(uffd, offset, wp)) + if (copy_page(gopts, offset, wp)) signalled++; } else { /* This is a WP request */ assert(steps =3D=3D 2); - wp_range(uffd, - (__u64)area_dst + + wp_range(gopts->uffd, + (__u64)gopts->area_dst + offset, - page_size, false); + gopts->page_size, false); } } else { signalled++; @@ -730,51 +707,53 @@ static int faulting_process(int signal_test, bool wp) } } =20 - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 if (signal_test) return signalled !=3D split_nr_pages; =20 - area_dst =3D mremap(area_dst, nr_pages * page_size, nr_pages * page_size, - MREMAP_MAYMOVE | MREMAP_FIXED, area_src); - if (area_dst =3D=3D MAP_FAILED) + gopts->area_dst =3D mremap(gopts->area_dst, gopts->nr_pages * gopts->page= _size, + gopts->nr_pages * gopts->page_size, + MREMAP_MAYMOVE | MREMAP_FIXED, + gopts->area_src); + if (gopts->area_dst =3D=3D MAP_FAILED) err("mremap"); /* Reset area_src since we just clobbered it */ - area_src =3D NULL; + gopts->area_src =3D NULL; =20 - for (; nr < nr_pages; nr++) { - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) { + for (; nr < gopts->nr_pages; nr++) { + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) { err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); } /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - for (nr =3D 0; nr < nr_pages; nr++) - for (i =3D 0; i < page_size; i++) - if (*(area_dst + nr * page_size + i) !=3D 0) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + for (i =3D 0; i < gopts->page_size; i++) + if (*(gopts->area_dst + nr * gopts->page_size + i) !=3D 0) err("page %lu offset %lu is not zero", nr, i); =20 return 0; } =20 -static void uffd_sigbus_test_common(bool wp) +static void uffd_sigbus_test_common(uffd_global_test_opts_t *gopts, bool w= p) { unsigned long userfaults; pthread_t uffd_mon; @@ -782,25 +761,26 @@ static void uffd_sigbus_test_common(bool wp) int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 - if (faulting_process(1, wp)) + if (faulting_process(gopts, 1, wp)) err("faulting process failed"); =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 args.apply_wp =3D wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -808,12 +788,12 @@ static void uffd_sigbus_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(2, wp)); + exit(faulting_process(gopts, 2, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, (void **)&userfaults)) err("pthread_join()"); @@ -824,28 +804,29 @@ static void uffd_sigbus_test_common(bool wp) uffd_test_pass(); } =20 -static void uffd_sigbus_test(uffd_test_args_t *args) +static void uffd_sigbus_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_sigbus_test_common(false); + uffd_sigbus_test_common(gopts, false); } =20 -static void uffd_sigbus_wp_test(uffd_test_args_t *args) +static void uffd_sigbus_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_sigbus_test_common(true); + uffd_sigbus_test_common(gopts, true); } =20 -static void uffd_events_test_common(bool wp) +static void uffd_events_test_common(uffd_global_test_opts_t *gopts, bool w= p) { pthread_t uffd_mon; pid_t pid; int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); - if (uffd_register(uffd, area_dst, nr_pages * page_size, + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 @@ -853,7 +834,7 @@ static void uffd_events_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -861,39 +842,39 @@ static void uffd_events_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(0, wp)); + exit(faulting_process(gopts, 0, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (args.missing_faults !=3D nr_pages) + if (args.missing_faults !=3D gopts->nr_pages) uffd_test_fail("Fault counts wrong"); else uffd_test_pass(); } =20 -static void uffd_events_test(uffd_test_args_t *args) +static void uffd_events_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_events_test_common(false); + uffd_events_test_common(gopts, false); } =20 -static void uffd_events_wp_test(uffd_test_args_t *args) +static void uffd_events_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_events_test_common(true); + uffd_events_test_common(gopts, true); } =20 -static void retry_uffdio_zeropage(int ufd, +static void retry_uffdio_zeropage(uffd_global_test_opts_t *gopts, struct uffdio_zeropage *uffdio_zeropage) { - uffd_test_ops->alias_mapping(&uffdio_zeropage->range.start, + uffd_test_ops->alias_mapping(gopts, &uffdio_zeropage->range.start, uffdio_zeropage->range.len, 0); - if (ioctl(ufd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { + if (ioctl(gopts->uffd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { if (uffdio_zeropage->zeropage !=3D -EEXIST) err("UFFDIO_ZEROPAGE error: %"PRId64, (int64_t)uffdio_zeropage->zeropage); @@ -903,16 +884,16 @@ static void retry_uffdio_zeropage(int ufd, } } =20 -static bool do_uffdio_zeropage(int ufd, bool has_zeropage) +static bool do_uffdio_zeropage(uffd_global_test_opts_t *gopts, bool has_ze= ropage) { struct uffdio_zeropage uffdio_zeropage =3D { 0 }; int ret; __s64 res; =20 - uffdio_zeropage.range.start =3D (unsigned long) area_dst; - uffdio_zeropage.range.len =3D page_size; + uffdio_zeropage.range.start =3D (unsigned long) gopts->area_dst; + uffdio_zeropage.range.len =3D gopts->page_size; uffdio_zeropage.mode =3D 0; - ret =3D ioctl(ufd, UFFDIO_ZEROPAGE, &uffdio_zeropage); + ret =3D ioctl(gopts->uffd, UFFDIO_ZEROPAGE, &uffdio_zeropage); res =3D uffdio_zeropage.zeropage; if (ret) { /* real retval in ufdio_zeropage.zeropage */ @@ -921,10 +902,10 @@ static bool do_uffdio_zeropage(int ufd, bool has_zero= page) else if (res !=3D -EINVAL) err("UFFDIO_ZEROPAGE not -EINVAL"); } else if (has_zeropage) { - if (res !=3D page_size) + if (res !=3D gopts->page_size) err("UFFDIO_ZEROPAGE unexpected size"); else - retry_uffdio_zeropage(ufd, &uffdio_zeropage); + retry_uffdio_zeropage(gopts, &uffdio_zeropage); return true; } else err("UFFDIO_ZEROPAGE succeeded"); @@ -950,25 +931,29 @@ uffd_register_detect_zeropage(int uffd, void *addr, u= int64_t len) } =20 /* exercise UFFDIO_ZEROPAGE */ -static void uffd_zeropage_test(uffd_test_args_t *args) +static void uffd_zeropage_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *args) { bool has_zeropage; int i; =20 - has_zeropage =3D uffd_register_detect_zeropage(uffd, area_dst, page_size); - if (area_dst_alias) + has_zeropage =3D uffd_register_detect_zeropage(gopts->uffd, + gopts->area_dst, + gopts->page_size); + if (gopts->area_dst_alias) /* Ignore the retval; we already have it */ - uffd_register_detect_zeropage(uffd, area_dst_alias, page_size); + uffd_register_detect_zeropage(gopts->uffd, gopts->area_dst_alias, gopts-= >page_size); =20 - if (do_uffdio_zeropage(uffd, has_zeropage)) - for (i =3D 0; i < page_size; i++) - if (area_dst[i] !=3D 0) + if (do_uffdio_zeropage(gopts, has_zeropage)) + for (i =3D 0; i < gopts->page_size; i++) + if (gopts->area_dst[i] !=3D 0) err("data non-zero at offset %d\n", i); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); =20 - if (area_dst_alias && uffd_unregister(uffd, area_dst_alias, page_size)) + if (gopts->area_dst_alias && uffd_unregister(gopts->uffd, + gopts->area_dst_alias, + gopts->page_size)) err("unregister"); =20 uffd_test_pass(); @@ -987,26 +972,27 @@ static void uffd_register_poison(int uffd, void *addr= , uint64_t len) err("registered area doesn't support COPY and POISON ioctls"); } =20 -static void do_uffdio_poison(int uffd, unsigned long offset) +static void do_uffdio_poison(uffd_global_test_opts_t *gopts, unsigned long= offset) { struct uffdio_poison uffdio_poison =3D { 0 }; int ret; __s64 res; =20 - uffdio_poison.range.start =3D (unsigned long) area_dst + offset; - uffdio_poison.range.len =3D page_size; + uffdio_poison.range.start =3D (unsigned long) gopts->area_dst + offset; + uffdio_poison.range.len =3D gopts->page_size; uffdio_poison.mode =3D 0; - ret =3D ioctl(uffd, UFFDIO_POISON, &uffdio_poison); + ret =3D ioctl(gopts->uffd, UFFDIO_POISON, &uffdio_poison); res =3D uffdio_poison.updated; =20 if (ret) err("UFFDIO_POISON error: %"PRId64, (int64_t)res); - else if (res !=3D page_size) + else if (res !=3D gopts->page_size) err("UFFDIO_POISON unexpected size: %"PRId64, (int64_t)res); } =20 -static void uffd_poison_handle_fault( - struct uffd_msg *msg, struct uffd_args *args) +static void uffd_poison_handle_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -1017,17 +1003,17 @@ static void uffd_poison_handle_fault( (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; + offset &=3D ~(gopts->page_size-1); =20 /* Odd pages -> copy zeroed page; even pages -> poison. */ - if (offset & page_size) - copy_page(uffd, offset, false); + if (offset & gopts->page_size) + copy_page(gopts, offset, false); else - do_uffdio_poison(uffd, offset); + do_uffdio_poison(gopts, offset); } =20 -static void uffd_poison_test(uffd_test_args_t *targs) +static void uffd_poison_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *targs) { pthread_t uffd_mon; char c; @@ -1036,10 +1022,12 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) unsigned long nr_sigbus =3D 0; unsigned long nr; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + args.gopts =3D gopts; + + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - uffd_register_poison(uffd, area_dst, nr_pages * page_size); - memset(area_src, 0, nr_pages * page_size); + uffd_register_poison(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopt= s->page_size); + memset(gopts->area_src, 0, gopts->nr_pages * gopts->page_size); =20 args.handle_fault =3D uffd_poison_handle_fault; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -1051,9 +1039,9 @@ static void uffd_poison_test(uffd_test_args_t *targs) if (sigaction(SIGBUS, &act, 0)) err("sigaction"); =20 - for (nr =3D 0; nr < nr_pages; ++nr) { - unsigned long offset =3D nr * page_size; - const char *bytes =3D (const char *) area_dst + offset; + for (nr =3D 0; nr < gopts->nr_pages; ++nr) { + unsigned long offset =3D nr * gopts->page_size; + const char *bytes =3D (const char *) gopts->area_dst + offset; const char *i; =20 if (sigsetjmp(*sigbuf, 1)) { @@ -1066,27 +1054,29 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) continue; } =20 - for (i =3D bytes; i < bytes + page_size; ++i) { + for (i =3D bytes; i < bytes + gopts->page_size; ++i) { if (*i) err("nonzero byte in area_dst (%p) at %p: %u", - area_dst, i, *i); + gopts->area_dst, i, *i); } } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (nr_sigbus !=3D nr_pages / 2) + if (nr_sigbus !=3D gopts->nr_pages / 2) err("expected to receive %lu SIGBUS, actually received %lu", - nr_pages / 2, nr_sigbus); + gopts->nr_pages / 2, nr_sigbus); =20 uffd_test_pass(); } =20 static void -uffd_move_handle_fault_common(struct uffd_msg *msg, struct uffd_args *args, +uffd_move_handle_fault_common(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args, unsigned long len) { unsigned long offset; @@ -1098,28 +1088,32 @@ uffd_move_handle_fault_common(struct uffd_msg *msg,= struct uffd_args *args, (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR | UFFD_PAGEFAULT_= FLAG_WRITE)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; offset &=3D ~(len-1); =20 - if (move_page(uffd, offset, len)) + if (move_page(gopts, offset, len)) args->missing_faults++; } =20 -static void uffd_move_handle_fault(struct uffd_msg *msg, +static void uffd_move_handle_fault(uffd_global_test_opts_t *gopts, struct = uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, page_size); + uffd_move_handle_fault_common(gopts, msg, args, gopts->page_size); } =20 -static void uffd_move_pmd_handle_fault(struct uffd_msg *msg, +static void uffd_move_pmd_handle_fault(uffd_global_test_opts_t *gopts, str= uct uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, read_pmd_pagesize()); + uffd_move_handle_fault_common(gopts, msg, args, read_pmd_pagesize()); } =20 static void -uffd_move_test_common(uffd_test_args_t *targs, unsigned long chunk_size, - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args)) +uffd_move_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *targs, + unsigned long chunk_size, + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, struct uffd_args *args) +) { unsigned long nr; pthread_t uffd_mon; @@ -1131,11 +1125,13 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, unsigned long src_offs =3D 0; unsigned long dst_offs =3D 0; =20 + args.gopts =3D gopts; + /* Prevent source pages from being mapped more than once */ - if (madvise(area_src, nr_pages * page_size, MADV_DONTFORK)) + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DON= TFORK)) err("madvise(MADV_DONTFORK) failure"); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("register failure"); =20 @@ -1143,22 +1139,22 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - step_size =3D chunk_size / page_size; - step_count =3D nr_pages / step_size; + step_size =3D chunk_size / gopts->page_size; + step_count =3D gopts->nr_pages / step_size; =20 - if (chunk_size > page_size) { - char *aligned_src =3D ALIGN_UP(area_src, chunk_size); - char *aligned_dst =3D ALIGN_UP(area_dst, chunk_size); + if (chunk_size > gopts->page_size) { + char *aligned_src =3D ALIGN_UP(gopts->area_src, chunk_size); + char *aligned_dst =3D ALIGN_UP(gopts->area_dst, chunk_size); =20 - if (aligned_src !=3D area_src || aligned_dst !=3D area_dst) { - src_offs =3D (aligned_src - area_src) / page_size; - dst_offs =3D (aligned_dst - area_dst) / page_size; + if (aligned_src !=3D gopts->area_src || aligned_dst !=3D gopts->area_dst= ) { + src_offs =3D (aligned_src - gopts->area_src) / gopts->page_size; + dst_offs =3D (aligned_dst - gopts->area_dst) / gopts->page_size; step_count--; } - orig_area_src =3D area_src; - orig_area_dst =3D area_dst; - area_src =3D aligned_src; - area_dst =3D aligned_dst; + orig_area_src =3D gopts->area_src; + orig_area_dst =3D gopts->area_dst; + gopts->area_src =3D aligned_src; + gopts->area_dst =3D aligned_dst; } =20 /* @@ -1172,34 +1168,34 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, =20 /* Check area_src content */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); - if (count !=3D count_verify[src_offs + nr + i]) + count =3D *area_count(gopts->area_src, nr + i, gopts); + if (count !=3D gopts->count_verify[src_offs + nr + i]) err("nr %lu source memory invalid %llu %llu\n", - nr + i, count, count_verify[src_offs + nr + i]); + nr + i, count, gopts->count_verify[src_offs + nr + i]); } =20 /* Faulting into area_dst should move the page or the huge page */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_dst, nr + i); - if (count !=3D count_verify[dst_offs + nr + i]) + count =3D *area_count(gopts->area_dst, nr + i, gopts); + if (count !=3D gopts->count_verify[dst_offs + nr + i]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[dst_offs + nr + i]); + nr, count, gopts->count_verify[dst_offs + nr + i]); } =20 /* Re-check area_src content which should be empty */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); + count =3D *area_count(gopts->area_src, nr + i, gopts); if (count !=3D 0) err("nr %lu move failed %llu %llu\n", - nr, count, count_verify[src_offs + nr + i]); + nr, count, gopts->count_verify[src_offs + nr + i]); } } - if (chunk_size > page_size) { - area_src =3D orig_area_src; - area_dst =3D orig_area_dst; + if (chunk_size > gopts->page_size) { + gopts->area_src =3D orig_area_src; + gopts->area_dst =3D orig_area_dst; } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); @@ -1210,24 +1206,24 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, uffd_test_pass(); } =20 -static void uffd_move_test(uffd_test_args_t *targs) +static void uffd_move_test(uffd_global_test_opts_t *gopts, uffd_test_args_= t *targs) { - uffd_move_test_common(targs, page_size, uffd_move_handle_fault); + uffd_move_test_common(gopts, targs, gopts->page_size, uffd_move_handle_fa= ult); } =20 -static void uffd_move_pmd_test(uffd_test_args_t *targs) +static void uffd_move_pmd_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) err("madvise(MADV_HUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static void uffd_move_pmd_split_test(uffd_test_args_t *targs) +static void uffd_move_pmd_split_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) err("madvise(MADV_NOHUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 @@ -1287,6 +1283,11 @@ typedef enum { THR_STATE_UNINTERRUPTIBLE, } thread_state; =20 +typedef struct { + uffd_global_test_opts_t *gopts; + volatile pid_t *pid; +} mmap_changing_thread_args; + static void sleep_short(void) { usleep(1000); @@ -1329,7 +1330,9 @@ static void thread_state_until(pid_t tid, thread_stat= e state) =20 static void *uffd_mmap_changing_thread(void *opaque) { - volatile pid_t *pid =3D opaque; + mmap_changing_thread_args *args =3D opaque; + uffd_global_test_opts_t *gopts =3D args->gopts; + volatile pid_t *pid =3D args->pid; int ret; =20 /* Unfortunately, it's only fetch-able from the thread itself.. */ @@ -1337,21 +1340,21 @@ static void *uffd_mmap_changing_thread(void *opaque) *pid =3D syscall(SYS_gettid); =20 /* Inject an event, this will hang solid until the event read */ - ret =3D madvise(area_dst, page_size, MADV_REMOVE); + ret =3D madvise(gopts->area_dst, gopts->page_size, MADV_REMOVE); if (ret) err("madvise(MADV_REMOVE) failed"); =20 return NULL; } =20 -static void uffd_consume_message(int fd) +static void uffd_consume_message(uffd_global_test_opts_t *gopts) { struct uffd_msg msg =3D { 0 }; =20 - while (uffd_read_msg(fd, &msg)); + while (uffd_read_msg(gopts, &msg)); } =20 -static void uffd_mmap_changing_test(uffd_test_args_t *targs) +static void uffd_mmap_changing_test(uffd_global_test_opts_t *gopts, uffd_t= est_args_t *targs) { /* * This stores the real PID (which can be different from how tid is @@ -1360,13 +1363,14 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) pid_t pid =3D 0; pthread_t tid; int ret; + mmap_changing_thread_args args =3D { gopts, &pid }; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("uffd_register() failed"); =20 /* Create a thread to generate the racy event */ - ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &pid); + ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &args); if (ret) err("pthread_create() failed"); =20 @@ -1380,26 +1384,26 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) /* Wait until the thread hangs at REMOVE event */ thread_state_until(pid, THR_STATE_UNINTERRUPTIBLE); =20 - if (!uffdio_mmap_changing_test_copy(uffd)) + if (!uffdio_mmap_changing_test_copy(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_zeropage(uffd)) + if (!uffdio_mmap_changing_test_zeropage(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_move(uffd)) + if (!uffdio_mmap_changing_test_move(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_poison(uffd)) + if (!uffdio_mmap_changing_test_poison(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_continue(uffd)) + if (!uffdio_mmap_changing_test_continue(gopts->uffd)) return; =20 /* * All succeeded above! Recycle everything. Start by reading the * event so as to kick the thread roll again.. */ - uffd_consume_message(uffd); + uffd_consume_message(gopts); =20 ret =3D pthread_join(tid, NULL); assert(ret =3D=3D 0); @@ -1407,10 +1411,10 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) uffd_test_pass(); } =20 -static int prevent_hugepages(const char **errmsg) +static int prevent_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_NOHUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) { /* Ignore only if CONFIG_TRANSPARENT_HUGEPAGE=3Dn */ if (errno !=3D EINVAL) { if (errmsg) @@ -1421,10 +1425,10 @@ static int prevent_hugepages(const char **errmsg) return 0; } =20 -static int request_hugepages(const char **errmsg) +static int request_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_HUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) { if (errmsg) { *errmsg =3D (errno =3D=3D EINVAL) ? "CONFIG_TRANSPARENT_HUGEPAGE is not set" : @@ -1448,13 +1452,17 @@ struct uffd_test_case_ops uffd_move_test_pmd_case_o= ps =3D { * Note that _UFFDIO_ZEROPAGE is tested separately in the zeropage test. */ static void -do_register_ioctls_test(uffd_test_args_t *args, bool miss, bool wp, bool m= inor) +do_register_ioctls_test(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, + bool miss, + bool wp, + bool minor) { uint64_t ioctls =3D 0, expected =3D BIT_ULL(_UFFDIO_WAKE); mem_type_t *mem_type =3D args->mem_type; int ret; =20 - ret =3D uffd_register_with_ioctls(uffd, area_dst, page_size, + ret =3D uffd_register_with_ioctls(gopts->uffd, gopts->area_dst, gopts->pa= ge_size, miss, wp, minor, &ioctls); =20 /* @@ -1485,18 +1493,18 @@ do_register_ioctls_test(uffd_test_args_t *args, boo= l miss, bool wp, bool minor) "(miss=3D%d, wp=3D%d, minor=3D%d): expected=3D0x%"PRIx64", " "returned=3D0x%"PRIx64, miss, wp, minor, expected, ioctls); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); } =20 -static void uffd_register_ioctls_test(uffd_test_args_t *args) +static void uffd_register_ioctls_test(uffd_global_test_opts_t *gopts, uffd= _test_args_t *args) { int miss, wp, minor; =20 for (miss =3D 0; miss <=3D 1; miss++) for (wp =3D 0; wp <=3D 1; wp++) for (minor =3D 0; minor <=3D 1; minor++) - do_register_ioctls_test(args, miss, wp, minor); + do_register_ioctls_test(gopts, args, miss, wp, minor); =20 uffd_test_pass(); } @@ -1734,6 +1742,28 @@ int main(int argc, char *argv[]) } for (j =3D 0; j < n_mems; j++) { mem_type =3D &mem_types[j]; + + /* Initialize global test options */ + uffd_global_test_opts_t gopts; + + gopts.map_shared =3D mem_type->shared; + uffd_test_ops =3D mem_type->mem_ops; + uffd_test_case_ops =3D test->test_case_ops; + + if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) + gopts.page_size =3D default_huge_page_size(); + else + gopts.page_size =3D psize(); + + /* Ensure we have at least 2 pages */ + gopts.nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, gopts.page_size * 2) + / gopts.page_size; + + gopts.nr_parallel =3D 1; + + /* Initialize test arguments */ + args.mem_type =3D mem_type; + if (!(test->mem_targets & mem_type->mem_flag)) continue; =20 @@ -1748,13 +1778,12 @@ int main(int argc, char *argv[]) uffd_test_skip("feature missing"); continue; } - if (uffd_setup_environment(&args, test, mem_type, - &errmsg)) { + if (uffd_test_ctx_init(&gopts, test->uffd_feature_required, &errmsg)) { uffd_test_skip(errmsg); continue; } - test->uffd_fn(&args); - uffd_test_ctx_clear(); + test->uffd_fn(&gopts, &args); + uffd_test_ctx_clear(&gopts); } } =20 diff --git a/tools/testing/selftests/mm/uffd-wp-mremap.c b/tools/testing/se= lftests/mm/uffd-wp-mremap.c index c2ba7d46c7b4..c286b1015f32 100644 --- a/tools/testing/selftests/mm/uffd-wp-mremap.c +++ b/tools/testing/selftests/mm/uffd-wp-mremap.c @@ -157,7 +157,11 @@ static bool range_is_swapped(void *addr, size_t size) return true; } =20 -static void test_one_folio(size_t size, bool private, bool swapout, bool h= ugetlb) +static void test_one_folio(uffd_global_test_opts_t *gopts, + size_t size, + bool private, + bool swapout, + bool hugetlb) { struct uffdio_writeprotect wp_prms; uint64_t features =3D 0; @@ -181,21 +185,21 @@ static void test_one_folio(size_t size, bool private,= bool swapout, bool hugetlb } =20 /* Register range for uffd-wp. */ - if (userfaultfd_open(&features)) { + if (userfaultfd_open(gopts, &features)) { if (errno =3D=3D ENOENT) ksft_test_result_skip("userfaultfd not available\n"); else ksft_test_result_fail("userfaultfd_open() failed\n"); goto out; } - if (uffd_register(uffd, mem, size, false, true, false)) { + if (uffd_register(gopts->uffd, mem, size, false, true, false)) { ksft_test_result_fail("uffd_register() failed\n"); goto out; } wp_prms.mode =3D UFFDIO_WRITEPROTECT_MODE_WP; wp_prms.range.start =3D (uintptr_t)mem; wp_prms.range.len =3D size; - if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { + if (ioctl(gopts->uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { ksft_test_result_fail("ioctl(UFFDIO_WRITEPROTECT) failed\n"); goto out; } @@ -242,9 +246,9 @@ static void test_one_folio(size_t size, bool private, b= ool swapout, bool hugetlb out: if (mem) munmap(mem, size); - if (uffd >=3D 0) { - close(uffd); - uffd =3D -1; + if (gopts->uffd >=3D 0) { + close(gopts->uffd); + gopts->uffd =3D -1; } } =20 @@ -336,6 +340,7 @@ static const struct testcase testcases[] =3D { =20 int main(int argc, char **argv) { + uffd_global_test_opts_t gopts; struct thp_settings settings; int i, j, plan =3D 0; =20 @@ -367,8 +372,8 @@ int main(int argc, char **argv) const struct testcase *tc =3D &testcases[i]; =20 for (j =3D 0; j < *tc->nr_sizes; j++) - test_one_folio(tc->sizes[j], tc->private, tc->swapout, - tc->hugetlb); + test_one_folio(&gopts, tc->sizes[j], tc->private, + tc->swapout, tc->hugetlb); } =20 /* If THP is supported, restore original THP settings. */ --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23B671A2C0B; Sat, 31 May 2025 07:47:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748677667; cv=none; b=I5BKYNidHv+xtMIuV6gDZWqPjMBOyJZrD3xnKpO5ekFi7Cb89GFfsvpbfEGSqYaLXh6jZXQdzoHGnHzM3u5QluhVL2TucbOnVwHgU0M/E1qV1xZ6tUjyE07rsSabmLzK3R89Dxx8t8IPiNcwSWNPA0vkEvapoc+0tjUm5uEPfq4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748677667; c=relaxed/simple; bh=yjPzyI7yBqAL56dE8kfsvym+2Bdn2WO7Ivdq8Oc6Qwo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mHdl7TTDvFM1XjO7N9NrfMlk8amSN8y2HTdwEMz2Ac8KJUt+nPdqy2WW0dAmXOsR4MSWpz6EUAcC22pQ/auARsPHMNLW2KG1ZfIENAaKRPK7bavyWb8GMf/uRSoLjvJxqIgynTDBlRnsB9hXSvO4UUlEffwNJrbohO50e6DTNps= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fD0Vs2jy; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fD0Vs2jy" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-22c336fcdaaso25640475ad.3; Sat, 31 May 2025 00:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1748677662; x=1749282462; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UAmG2goL53Cw5ew974oJqfpgUSyostqjHqftpfpF/AA=; b=fD0Vs2jyrlhD/fH8osIz6IL60idiKRLuChdZLdx21D5UIn6e9tflxmL4ieXy5PyQVU gWKqT0yD8xya64Epe1F0uO3/RdErgrvppczm36RpEA4L+Lo9ftMVcX08I3q8scj2AVlo ERvjKTtlK+DpeYR+7YKEUaPO80sQy/HkoWnthG3wPyt9Nn0jAiUBYJznACQbjL+iktQ1 hNNHTGrQdZtWvqEoiClfD+2n2sisObb1hMR95N2ZcmCS3ua/PcH10LDsaZK+ieHz+1cj ZWPdmsCRRqeruMueozKPayG+u4ytik36LpYvG4JJYm+n9zAwV09YKA/YkecNWJbqc0SC ugBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748677662; x=1749282462; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UAmG2goL53Cw5ew974oJqfpgUSyostqjHqftpfpF/AA=; b=GdjsQWNd3i4esZu8RaPvPXRqUMo8xP/YaQ+7PA1wtxgDMnmGMPlbRQRSU3LLsvnQ/W fdqjG94cNp4ltFKM4YETI9CCqau+TYQ7v5YZmsBdntCe5P1BssFhVWCgTpRLhsZOROp0 m4DC8b7jpHWHbnwQwyB1AhDkZbuE4DkcCx7jsBMrObV87sMlprTtYZwDGAP6iQNKupvf tVRm2GkrpFD8EwrZ/PvdCWDyZ3f/sYKNpMr9r63qG+wLn05/Xeq9l4exjNb2yf27eOXx Eh5S0Aib6QIILM4oLj60lA4+ufz3ZoooNuyfDp61MOF/vUXxcwRVXMAy15862rR/xoLA xKaQ== X-Forwarded-Encrypted: i=1; AJvYcCU29Dr9/z1gAYuA8QWAVOTQD6H3DJ4Gl3AJ8pMIWyalqOlZQczPBDXWTBZBkr39fvNDwqMAmwDuUxzKLHOMbUE=@vger.kernel.org X-Gm-Message-State: AOJu0YyWUYNBM6s8JeiEnlf0jqSE8opYd1qpvGNasynKlVTczzm2qKtN fdt/Y2P/bX9XfW5H8eWetCMYcKY9UJUvLEMykOdjWCgrjaWTxWlw9XY= X-Gm-Gg: ASbGncscTe0USBglwZOJEQQfp7zzhICTeqUW3Wy/HvDiAI1TfPjR5IvrvLnP2T19Mru +mu1fxbg6IEv/t7b8ISuzDfB101vkDtBD2YDFYo1kDrTXNYZGaxAmWE0HHJgtWNEf5RJCEIk38S 6WnC7ogux3wZka47q//gyTh7KLQzfBQ3wx5QuFFm/HJ0XIFnXuHKetdoGR8822X2pRjkCw0UWkA DAOifSg4f8kOfVban8S9Ra1lP+cmJOCfhj/fkt71BYNhmMTn1QOT3T498Ud2woh3K/ub+erSDkA 7jOmTjGQQ9lEZ32OqFsQL7TE0btEifrQkiBFRyOTcOWPthZ83Vw2ruaMyDx5H3mUCxOW/buQdw= = X-Google-Smtp-Source: AGHT+IF1SmerG9I/ZzOxc023TRJRttRP2/QrlTz9GNkrP+iawmh958ziPbbk+Ce7cxfrbR9cZnE+eg== X-Received: by 2002:a17:902:f682:b0:234:d399:f948 with SMTP id d9443c01a7336-2355f74fc98mr15789065ad.33.1748677661789; Sat, 31 May 2025 00:47:41 -0700 (PDT) Received: from debian.ujwal.com ([2409:40f2:3152:41fa:f908:b623:3840:6a0c]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23506bc8a28sm38505125ad.51.2025.05.31.00.47.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 May 2025 00:47:41 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, peterx@redhat.com, shuah@kernel.org, jackmanb@google.com Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, Ujwal Kundur Subject: [PATCH v4 1/1] selftests/mm/uffd: Refactor non-composite global vars into struct Date: Sat, 31 May 2025 13:16:25 +0530 Message-Id: <20250531074625.478-1-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor macros and non-composite global variable definitions into a struct that is defined at the start of a test and is passed around instead of relying on global vars. Renders the implementation / use of test_uffdio_copy_eexist vestigial and will be addressed in a later patch. Signed-off-by: Ujwal Kundur Tested-by: Brendan Jackman --- Changes since v3: - more formatting fixes Changes since v2: - redo patch on mm-new branch Changes since v1: - indentation fixes - squash into single patch to assist bisections tools/testing/selftests/mm/uffd-common.c | 268 ++++----- tools/testing/selftests/mm/uffd-common.h | 78 +-- tools/testing/selftests/mm/uffd-stress.c | 224 ++++---- tools/testing/selftests/mm/uffd-unit-tests.c | 561 ++++++++++--------- tools/testing/selftests/mm/uffd-wp-mremap.c | 23 +- 5 files changed, 612 insertions(+), 542 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selft= ests/mm/uffd-common.c index a37088a23ffe..6067cf4bdae6 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -7,18 +7,30 @@ =20 #include "uffd-common.h" =20 -#define BASE_PMD_ADDR ((void *)(1UL << 30)) - -volatile bool test_uffdio_copy_eexist =3D true; -unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; -int uffd =3D -1, uffd_flags, finished, *pipefd, test_type; -bool map_shared; -bool test_uffdio_wp =3D true; -unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; uffd_test_case_ops_t *uffd_test_case_ops; -atomic_bool ready_for_fork; + +#define BASE_PMD_ADDR ((void *)(1UL << 30)) + +/* pthread_mutex_t starts at page offset 0 */ +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts) +{ + return (pthread_mutex_t *) (area + nr * gopts->page_size); +} + +/* + * count is placed in the page after pthread_mutex_t naturally aligned + * to avoid non alignment faults on non-x86 archs. + */ +volatile unsigned long long *area_count( + char *area, unsigned long nr, + uffd_global_test_opts_t *gopts) +{ + return (volatile unsigned long long *) + ((unsigned long)(area + nr * gopts->page_size + + sizeof(pthread_mutex_t) + sizeof(unsigned long long) - 1) & + ~(unsigned long)(sizeof(unsigned long long) - 1)); +} =20 static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { @@ -40,15 +52,15 @@ static int uffd_mem_fd_create(off_t mem_size, bool huge= tlb) return mem_fd; } =20 -static void anon_release_pages(char *rel_area) +static void anon_release_pages(uffd_global_test_opts_t *gopts, char *rel_a= rea) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } =20 -static int anon_allocate_area(void **alloc_area, bool is_src) +static int anon_allocate_area(uffd_global_test_opts_t *gopts, void **alloc= _area, bool is_src) { - *alloc_area =3D mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, + *alloc_area =3D mmap(NULL, gopts->nr_pages * gopts->page_size, PROT_READ = | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); if (*alloc_area =3D=3D MAP_FAILED) { *alloc_area =3D NULL; @@ -57,31 +69,32 @@ static int anon_allocate_area(void **alloc_area, bool i= s_src) return 0; } =20 -static void noop_alias_mapping(__u64 *start, size_t len, unsigned long off= set) +static void noop_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *star= t, + size_t len, unsigned long offset) { } =20 -static void hugetlb_release_pages(char *rel_area) +static void hugetlb_release_pages(uffd_global_test_opts_t *gopts, char *re= l_area) { - if (!map_shared) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (!gopts->map_shared) { + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } else { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } } =20 -static int hugetlb_allocate_area(void **alloc_area, bool is_src) +static int hugetlb_allocate_area(uffd_global_test_opts_t *gopts, void **al= loc_area, bool is_src) { - off_t size =3D nr_pages * page_size; + off_t size =3D gopts->nr_pages * gopts->page_size; off_t offset =3D is_src ? 0 : size; void *area_alias =3D NULL; char **alloc_area_alias; int mem_fd =3D uffd_mem_fd_create(size * 2, true); =20 *alloc_area =3D mmap(NULL, size, PROT_READ | PROT_WRITE, - (map_shared ? MAP_SHARED : MAP_PRIVATE) | + (gopts->map_shared ? MAP_SHARED : MAP_PRIVATE) | (is_src ? 0 : MAP_NORESERVE), mem_fd, offset); if (*alloc_area =3D=3D MAP_FAILED) { @@ -89,7 +102,7 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) return -errno; } =20 - if (map_shared) { + if (gopts->map_shared) { area_alias =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, mem_fd, offset); if (area_alias =3D=3D MAP_FAILED) @@ -97,9 +110,9 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) } =20 if (is_src) { - alloc_area_alias =3D &area_src_alias; + alloc_area_alias =3D &gopts->area_src_alias; } else { - alloc_area_alias =3D &area_dst_alias; + alloc_area_alias =3D &gopts->area_dst_alias; } if (area_alias) *alloc_area_alias =3D area_alias; @@ -108,24 +121,25 @@ static int hugetlb_allocate_area(void **alloc_area, b= ool is_src) return 0; } =20 -static void hugetlb_alias_mapping(__u64 *start, size_t len, unsigned long = offset) +static void hugetlb_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *s= tart, + size_t len, unsigned long offset) { - if (!map_shared) + if (!gopts->map_shared) return; =20 - *start =3D (unsigned long) area_dst_alias + offset; + *start =3D (unsigned long) gopts->area_dst_alias + offset; } =20 -static void shmem_release_pages(char *rel_area) +static void shmem_release_pages(uffd_global_test_opts_t *gopts, char *rel_= area) { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } =20 -static int shmem_allocate_area(void **alloc_area, bool is_src) +static int shmem_allocate_area(uffd_global_test_opts_t *gopts, void **allo= c_area, bool is_src) { void *area_alias =3D NULL; - size_t bytes =3D nr_pages * page_size, hpage_size =3D read_pmd_pagesize(); + size_t bytes =3D gopts->nr_pages * gopts->page_size, hpage_size =3D read_= pmd_pagesize(); unsigned long offset =3D is_src ? 0 : bytes; char *p =3D NULL, *p_alias =3D NULL; int mem_fd =3D uffd_mem_fd_create(bytes * 2, false); @@ -159,22 +173,23 @@ static int shmem_allocate_area(void **alloc_area, boo= l is_src) err("mmap of anonymous memory failed at %p", p_alias); =20 if (is_src) - area_src_alias =3D area_alias; + gopts->area_src_alias =3D area_alias; else - area_dst_alias =3D area_alias; + gopts->area_dst_alias =3D area_alias; =20 close(mem_fd); return 0; } =20 -static void shmem_alias_mapping(__u64 *start, size_t len, unsigned long of= fset) +static void shmem_alias_mapping(uffd_global_test_opts_t *gopts, __u64 *sta= rt, + size_t len, unsigned long offset) { - *start =3D (unsigned long)area_dst_alias + offset; + *start =3D (unsigned long)gopts->area_dst_alias + offset; } =20 -static void shmem_check_pmd_mapping(void *p, int expect_nr_hpages) +static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *= p, int expect_nr_hpages) { - if (!check_huge_shmem(area_dst_alias, expect_nr_hpages, + if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages, read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); @@ -234,18 +249,18 @@ void uffd_stats_report(struct uffd_args *args, int n_= cpus) printf("\n"); } =20 -int userfaultfd_open(uint64_t *features) +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features) { struct uffdio_api uffdio_api; =20 - uffd =3D uffd_open(UFFD_FLAGS); - if (uffd < 0) + gopts->uffd =3D uffd_open(UFFD_FLAGS); + if (gopts->uffd < 0) return -1; - uffd_flags =3D fcntl(uffd, F_GETFD, NULL); + gopts->uffd_flags =3D fcntl(gopts->uffd, F_GETFD, NULL); =20 uffdio_api.api =3D UFFD_API; uffdio_api.features =3D *features; - if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + if (ioctl(gopts->uffd, UFFDIO_API, &uffdio_api)) /* Probably lack of CAP_PTRACE? */ return -1; if (uffdio_api.api !=3D UFFD_API) @@ -255,59 +270,59 @@ int userfaultfd_open(uint64_t *features) return 0; } =20 -static inline void munmap_area(void **area) +static inline void munmap_area(uffd_global_test_opts_t *gopts, void **area) { if (*area) - if (munmap(*area, nr_pages * page_size)) + if (munmap(*area, gopts->nr_pages * gopts->page_size)) err("munmap"); =20 *area =3D NULL; } =20 -void uffd_test_ctx_clear(void) +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts) { size_t i; =20 - if (pipefd) { - for (i =3D 0; i < nr_parallel * 2; ++i) { - if (close(pipefd[i])) + if (gopts->pipefd) { + for (i =3D 0; i < gopts->nr_parallel * 2; ++i) { + if (close(gopts->pipefd[i])) err("close pipefd"); } - free(pipefd); - pipefd =3D NULL; + free(gopts->pipefd); + gopts->pipefd =3D NULL; } =20 - if (count_verify) { - free(count_verify); - count_verify =3D NULL; + if (gopts->count_verify) { + free(gopts->count_verify); + gopts->count_verify =3D NULL; } =20 - if (uffd !=3D -1) { - if (close(uffd)) + if (gopts->uffd !=3D -1) { + if (close(gopts->uffd)) err("close uffd"); - uffd =3D -1; + gopts->uffd =3D -1; } =20 - munmap_area((void **)&area_src); - munmap_area((void **)&area_src_alias); - munmap_area((void **)&area_dst); - munmap_area((void **)&area_dst_alias); - munmap_area((void **)&area_remap); + munmap_area(gopts, (void **)&gopts->area_src); + munmap_area(gopts, (void **)&gopts->area_src_alias); + munmap_area(gopts, (void **)&gopts->area_dst); + munmap_area(gopts, (void **)&gopts->area_dst_alias); + munmap_area(gopts, (void **)&gopts->area_remap); } =20 -int uffd_test_ctx_init(uint64_t features, const char **errmsg) +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg) { unsigned long nr, cpu; int ret; =20 if (uffd_test_case_ops && uffd_test_case_ops->pre_alloc) { - ret =3D uffd_test_case_ops->pre_alloc(errmsg); + ret =3D uffd_test_case_ops->pre_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D uffd_test_ops->allocate_area((void **)&area_src, true); - ret |=3D uffd_test_ops->allocate_area((void **)&area_dst, false); + ret =3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_src, t= rue); + ret |=3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_dst, = false); if (ret) { if (errmsg) *errmsg =3D "memory allocation failed"; @@ -315,26 +330,26 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) } =20 if (uffd_test_case_ops && uffd_test_case_ops->post_alloc) { - ret =3D uffd_test_case_ops->post_alloc(errmsg); + ret =3D uffd_test_case_ops->post_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D userfaultfd_open(&features); + ret =3D userfaultfd_open(gopts, &features); if (ret) { if (errmsg) *errmsg =3D "possible lack of privilege"; return ret; } =20 - count_verify =3D malloc(nr_pages * sizeof(unsigned long long)); - if (!count_verify) + gopts->count_verify =3D malloc(gopts->nr_pages * sizeof(unsigned long lon= g)); + if (!gopts->count_verify) err("count_verify"); =20 - for (nr =3D 0; nr < nr_pages; nr++) { - *area_mutex(area_src, nr) =3D + for (nr =3D 0; nr < gopts->nr_pages; nr++) { + *area_mutex(gopts->area_src, nr, gopts) =3D (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER; - count_verify[nr] =3D *area_count(area_src, nr) =3D 1; + gopts->count_verify[nr] =3D *area_count(gopts->area_src, nr, gopts) =3D = 1; /* * In the transition between 255 to 256, powerpc will * read out of order in my_bcmp and see both bytes as @@ -342,7 +357,7 @@ int uffd_test_ctx_init(uint64_t features, const char **= errmsg) * after the count, to avoid my_bcmp to trigger false * positives. */ - *(area_count(area_src, nr) + 1) =3D 1; + *(area_count(gopts->area_src, nr, gopts) + 1) =3D 1; } =20 /* @@ -363,13 +378,13 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) * proactively split the thp and drop any accidentally initialized * pages within area_dst. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - pipefd =3D malloc(sizeof(int) * nr_parallel * 2); - if (!pipefd) + gopts->pipefd =3D malloc(sizeof(int) * gopts->nr_parallel * 2); + if (!gopts->pipefd) err("pipefd"); - for (cpu =3D 0; cpu < nr_parallel; cpu++) - if (pipe2(&pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) + if (pipe2(&gopts->pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) err("pipe"); =20 return 0; @@ -416,9 +431,9 @@ static void continue_range(int ufd, __u64 start, __u64 = len, bool wp) ret, (int64_t) req.mapped); } =20 -int uffd_read_msg(int ufd, struct uffd_msg *msg) +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg) { - int ret =3D read(uffd, msg, sizeof(*msg)); + int ret =3D read(gopts->uffd, msg, sizeof(*msg)); =20 if (ret !=3D sizeof(*msg)) { if (ret < 0) { @@ -433,7 +448,8 @@ int uffd_read_msg(int ufd, struct uffd_msg *msg) return 0; } =20 -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args) +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, struct uffd_ms= g *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -442,7 +458,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) =20 if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP) { /* Write protect page faults */ - wp_range(uffd, msg->arg.pagefault.address, page_size, false); + wp_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, fals= e); args->wp_faults++; } else if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR) { uint8_t *area; @@ -460,12 +476,12 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) * (UFFD-registered). */ =20 - area =3D (uint8_t *)(area_dst + - ((char *)msg->arg.pagefault.address - - area_dst_alias)); - for (b =3D 0; b < page_size; ++b) + area =3D (uint8_t *)(gopts->area_dst + + ((char *)msg->arg.pagefault.address - + gopts->area_dst_alias)); + for (b =3D 0; b < gopts->page_size; ++b) area[b] =3D ~area[b]; - continue_range(uffd, msg->arg.pagefault.address, page_size, + continue_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, args->apply_wp); args->minor_faults++; } else { @@ -493,10 +509,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WRITE) err("unexpected write fault"); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->ar= ea_dst; + offset &=3D ~(gopts->page_size-1); =20 - if (copy_page(uffd, offset, args->apply_wp)) + if (copy_page(gopts, offset, args->apply_wp)) args->missing_faults++; } } @@ -504,6 +520,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) void *uffd_poll_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; unsigned long cpu =3D args->cpu; struct pollfd pollfd[2]; struct uffd_msg msg; @@ -514,12 +531,12 @@ void *uffd_poll_thread(void *arg) if (!args->handle_fault) args->handle_fault =3D uffd_handle_page_fault; =20 - pollfd[0].fd =3D uffd; + pollfd[0].fd =3D gopts->uffd; pollfd[0].events =3D POLLIN; - pollfd[1].fd =3D pipefd[cpu*2]; + pollfd[1].fd =3D gopts->pipefd[cpu*2]; pollfd[1].events =3D POLLIN; =20 - ready_for_fork =3D true; + gopts->ready_for_fork =3D true; =20 for (;;) { ret =3D poll(pollfd, 2, -1); @@ -537,30 +554,30 @@ void *uffd_poll_thread(void *arg) } if (!(pollfd[0].revents & POLLIN)) err("pollfd[0].revents %d", pollfd[0].revents); - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; switch (msg.event) { default: err("unexpected msg event %u\n", msg.event); break; case UFFD_EVENT_PAGEFAULT: - args->handle_fault(&msg, args); + args->handle_fault(gopts, &msg, args); break; case UFFD_EVENT_FORK: - close(uffd); - uffd =3D msg.arg.fork.ufd; - pollfd[0].fd =3D uffd; + close(gopts->uffd); + gopts->uffd =3D msg.arg.fork.ufd; + pollfd[0].fd =3D gopts->uffd; break; case UFFD_EVENT_REMOVE: uffd_reg.range.start =3D msg.arg.remove.start; uffd_reg.range.len =3D msg.arg.remove.end - msg.arg.remove.start; - if (ioctl(uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) + if (ioctl(gopts->uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) err("remove failure"); break; case UFFD_EVENT_REMAP: - area_remap =3D area_dst; /* save for later unmap */ - area_dst =3D (char *)(unsigned long)msg.arg.remap.to; + gopts->area_remap =3D gopts->area_dst; /* save for later unmap */ + gopts->area_dst =3D (char *)(unsigned long)msg.arg.remap.to; break; } } @@ -568,17 +585,18 @@ void *uffd_poll_thread(void *arg) return NULL; } =20 -static void retry_copy_page(int ufd, struct uffdio_copy *uffdio_copy, +static void retry_copy_page(uffd_global_test_opts_t *gopts, struct uffdio_= copy *uffdio_copy, unsigned long offset) { - uffd_test_ops->alias_mapping(&uffdio_copy->dst, + uffd_test_ops->alias_mapping(gopts, + &uffdio_copy->dst, uffdio_copy->len, offset); - if (ioctl(ufd, UFFDIO_COPY, uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy->copy !=3D -EEXIST) err("UFFDIO_COPY retry error: %"PRId64, - (int64_t)uffdio_copy->copy); + (int64_t)uffdio_copy->copy); } else { err("UFFDIO_COPY retry unexpected: %"PRId64, (int64_t)uffdio_copy->copy); @@ -597,60 +615,60 @@ static void wake_range(int ufd, unsigned long addr, u= nsigned long len) addr), exit(1); } =20 -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp) +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp) { struct uffdio_copy uffdio_copy; =20 - if (offset >=3D nr_pages * page_size) + if (offset >=3D gopts->nr_pages * gopts->page_size) err("unexpected offset %lu\n", offset); - uffdio_copy.dst =3D (unsigned long) area_dst + offset; - uffdio_copy.src =3D (unsigned long) area_src + offset; - uffdio_copy.len =3D page_size; + uffdio_copy.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_copy.src =3D (unsigned long) gopts->area_src + offset; + uffdio_copy.len =3D gopts->page_size; if (wp) uffdio_copy.mode =3D UFFDIO_COPY_MODE_WP; else uffdio_copy.mode =3D 0; uffdio_copy.copy =3D 0; - if (ioctl(ufd, UFFDIO_COPY, &uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, &uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy.copy !=3D -EEXIST) err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); - wake_range(ufd, uffdio_copy.dst, page_size); - } else if (uffdio_copy.copy !=3D page_size) { + wake_range(gopts->uffd, uffdio_copy.dst, gopts->page_size); + } else if (uffdio_copy.copy !=3D gopts->page_size) { err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); } else { - if (test_uffdio_copy_eexist && retry) { - test_uffdio_copy_eexist =3D false; - retry_copy_page(ufd, &uffdio_copy, offset); + if (gopts->test_uffdio_copy_eexist && retry) { + gopts->test_uffdio_copy_eexist =3D false; + retry_copy_page(gopts, &uffdio_copy, offset); } return 1; } return 0; } =20 -int copy_page(int ufd, unsigned long offset, bool wp) +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p) { - return __copy_page(ufd, offset, false, wp); + return __copy_page(gopts, offset, false, wp); } =20 -int move_page(int ufd, unsigned long offset, unsigned long len) +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len) { struct uffdio_move uffdio_move; =20 - if (offset + len > nr_pages * page_size) + if (offset + len > gopts->nr_pages * gopts->page_size) err("unexpected offset %lu and length %lu\n", offset, len); - uffdio_move.dst =3D (unsigned long) area_dst + offset; - uffdio_move.src =3D (unsigned long) area_src + offset; + uffdio_move.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_move.src =3D (unsigned long) gopts->area_src + offset; uffdio_move.len =3D len; uffdio_move.mode =3D UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES; uffdio_move.move =3D 0; - if (ioctl(ufd, UFFDIO_MOVE, &uffdio_move)) { + if (ioctl(gopts->uffd, UFFDIO_MOVE, &uffdio_move)) { /* real retval in uffdio_move.move */ if (uffdio_move.move !=3D -EEXIST) err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); - wake_range(ufd, uffdio_move.dst, len); + wake_range(gopts->uffd, uffdio_move.dst, len); } else if (uffdio_move.move !=3D len) { err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); } else diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selft= ests/mm/uffd-common.h index 7700cbfa3975..37d3ca55905f 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -56,20 +56,17 @@ =20 #define err(fmt, ...) errexit(1, fmt, ##__VA_ARGS__) =20 -/* pthread_mutex_t starts at page offset 0 */ -#define area_mutex(___area, ___nr) \ - ((pthread_mutex_t *) ((___area) + (___nr)*page_size)) -/* - * count is placed in the page after pthread_mutex_t naturally aligned - * to avoid non alignment faults on non-x86 archs. - */ -#define area_count(___area, ___nr) \ - ((volatile unsigned long long *) ((unsigned long) \ - ((___area) + (___nr)*page_size + \ - sizeof(pthread_mutex_t) + \ - sizeof(unsigned long long) - 1) & \ - ~(unsigned long)(sizeof(unsigned long long) \ - - 1))) +struct uffd_global_test_opts { + unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; + char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; + int uffd, uffd_flags, finished, *pipefd, test_type; + bool map_shared; + bool test_uffdio_wp; + unsigned long long *count_verify; + volatile bool test_uffdio_copy_eexist; + atomic_bool ready_for_fork; +}; +typedef struct uffd_global_test_opts uffd_global_test_opts_t; =20 /* Userfaultfd test statistics */ struct uffd_args { @@ -79,50 +76,55 @@ struct uffd_args { unsigned long missing_faults; unsigned long wp_faults; unsigned long minor_faults; + struct uffd_global_test_opts *gopts; =20 /* A custom fault handler; defaults to uffd_handle_page_fault. */ - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args); + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, + struct uffd_args *args); }; =20 struct uffd_test_ops { - int (*allocate_area)(void **alloc_area, bool is_src); - void (*release_pages)(char *rel_area); - void (*alias_mapping)(__u64 *start, size_t len, unsigned long offset); - void (*check_pmd_mapping)(void *p, int expect_nr_hpages); + int (*allocate_area)(uffd_global_test_opts_t *gopts, void **alloc_area, b= ool is_src); + void (*release_pages)(uffd_global_test_opts_t *gopts, char *rel_area); + void (*alias_mapping)(uffd_global_test_opts_t *gopts, + __u64 *start, + size_t len, + unsigned long offset); + void (*check_pmd_mapping)(uffd_global_test_opts_t *gopts, void *p, int ex= pect_nr_hpages); }; typedef struct uffd_test_ops uffd_test_ops_t; =20 struct uffd_test_case_ops { - int (*pre_alloc)(const char **errmsg); - int (*post_alloc)(const char **errmsg); + int (*pre_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); + int (*post_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); }; typedef struct uffd_test_case_ops uffd_test_case_ops_t; =20 -extern unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -extern char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_= remap; -extern int uffd, uffd_flags, finished, *pipefd, test_type; -extern bool map_shared; -extern bool test_uffdio_wp; -extern unsigned long long *count_verify; -extern volatile bool test_uffdio_copy_eexist; -extern atomic_bool ready_for_fork; - +extern uffd_global_test_opts_t *uffd_gtest_opts; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; extern uffd_test_ops_t hugetlb_uffd_test_ops; extern uffd_test_ops_t *uffd_test_ops; extern uffd_test_case_ops_t *uffd_test_case_ops; =20 +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts); +volatile unsigned long long *area_count(char *area, + unsigned long nr, + uffd_global_test_opts_t *gopts); + void uffd_stats_report(struct uffd_args *args, int n_cpus); -int uffd_test_ctx_init(uint64_t features, const char **errmsg); -void uffd_test_ctx_clear(void); -int userfaultfd_open(uint64_t *features); -int uffd_read_msg(int ufd, struct uffd_msg *msg); +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg); +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts); +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features); +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg); void wp_range(int ufd, __u64 start, __u64 len, bool wp); -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); -int copy_page(int ufd, unsigned long offset, bool wp); -int move_page(int ufd, unsigned long offset, unsigned long len); +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args); +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp); +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p); +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len); void *uffd_poll_thread(void *arg); =20 int uffd_open_dev(unsigned int flags); diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selft= ests/mm/uffd-stress.c index 40af7f67c407..bf3cd0abe203 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -76,54 +76,58 @@ static void usage(void) exit(1); } =20 -static void uffd_stats_reset(struct uffd_args *args, unsigned long n_cpus) +static void uffd_stats_reset(uffd_global_test_opts_t *gopts, struct uffd_a= rgs *args, + unsigned long n_cpus) { int i; =20 for (i =3D 0; i < n_cpus; i++) { args[i].cpu =3D i; - args[i].apply_wp =3D test_uffdio_wp; + args[i].apply_wp =3D gopts->test_uffdio_wp; args[i].missing_faults =3D 0; args[i].wp_faults =3D 0; args[i].minor_faults =3D 0; + args[i].gopts =3D gopts; } } =20 static void *locking_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr; unsigned long long count; =20 if (!(bounces & BOUNCE_RANDOM)) { page_nr =3D -bounces; if (!(bounces & BOUNCE_RACINGFAULTS)) - page_nr +=3D cpu * nr_pages_per_cpu; + page_nr +=3D cpu * gopts->nr_pages_per_cpu; } =20 - while (!finished) { + while (!gopts->finished) { if (bounces & BOUNCE_RANDOM) { if (getrandom(&page_nr, sizeof(page_nr), 0) !=3D sizeof(page_nr)) err("getrandom failed"); } else page_nr +=3D 1; - page_nr %=3D nr_pages; - pthread_mutex_lock(area_mutex(area_dst, page_nr)); - count =3D *area_count(area_dst, page_nr); - if (count !=3D count_verify[page_nr]) + page_nr %=3D gopts->nr_pages; + pthread_mutex_lock(area_mutex(gopts->area_dst, page_nr, gopts)); + count =3D *area_count(gopts->area_dst, page_nr, gopts); + if (count !=3D gopts->count_verify[page_nr]) err("page_nr %lu memory corruption %llu %llu", - page_nr, count, count_verify[page_nr]); + page_nr, count, gopts->count_verify[page_nr]); count++; - *area_count(area_dst, page_nr) =3D count_verify[page_nr] =3D count; - pthread_mutex_unlock(area_mutex(area_dst, page_nr)); + *area_count(gopts->area_dst, page_nr, gopts) =3D gopts->count_verify[pag= e_nr] =3D count; + pthread_mutex_unlock(area_mutex(gopts->area_dst, page_nr, gopts)); } =20 return NULL; } =20 -static int copy_page_retry(int ufd, unsigned long offset) +static int copy_page_retry(uffd_global_test_opts_t *gopts, unsigned long o= ffset) { - return __copy_page(ufd, offset, true, test_uffdio_wp); + return __copy_page(gopts, offset, true, gopts->test_uffdio_wp); } =20 pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INITIALIZER; @@ -131,15 +135,16 @@ pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INI= TIALIZER; static void *uffd_read_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; struct uffd_msg msg; =20 pthread_mutex_unlock(&uffd_read_mutex); /* from here cancellation is ok */ =20 for (;;) { - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; - uffd_handle_page_fault(&msg, args); + uffd_handle_page_fault(gopts, &msg, args); } =20 return NULL; @@ -147,32 +152,34 @@ static void *uffd_read_thread(void *arg) =20 static void *background_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr, start_nr, mid_nr, end_nr; =20 - start_nr =3D cpu * nr_pages_per_cpu; - end_nr =3D (cpu+1) * nr_pages_per_cpu; + start_nr =3D cpu * gopts->nr_pages_per_cpu; + end_nr =3D (cpu+1) * gopts->nr_pages_per_cpu; mid_nr =3D (start_nr + end_nr) / 2; =20 /* Copy the first half of the pages */ for (page_nr =3D start_nr; page_nr < mid_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 /* * If we need to test uffd-wp, set it up now. Then we'll have * at least the first half of the pages mapped already which * can be write-protected for testing */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst + start_nr * page_size, - nr_pages_per_cpu * page_size, true); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst + start_nr * gopts-= >page_size, + gopts->nr_pages_per_cpu * gopts->page_size, true); =20 /* * Continue the 2nd half of the page copying, handling write * protection faults if any */ for (page_nr =3D mid_nr; page_nr < end_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 return NULL; } @@ -180,17 +187,21 @@ static void *background_thread(void *arg) static int stress(struct uffd_args *args) { unsigned long cpu; - pthread_t locking_threads[nr_parallel]; - pthread_t uffd_threads[nr_parallel]; - pthread_t background_threads[nr_parallel]; + uffd_global_test_opts_t *gopts =3D args->gopts; + pthread_t locking_threads[gopts->nr_parallel]; + pthread_t uffd_threads[gopts->nr_parallel]; + pthread_t background_threads[gopts->nr_parallel]; =20 - finished =3D 0; - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + gopts->finished =3D 0; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { if (pthread_create(&locking_threads[cpu], &attr, - locking_thread, (void *)cpu)) + locking_thread, (void *)&args[cpu])) return 1; if (bounces & BOUNCE_POLL) { - if (pthread_create(&uffd_threads[cpu], &attr, uffd_poll_thread, &args[c= pu])) + if (pthread_create(&uffd_threads[cpu], + &attr, + uffd_poll_thread, + (void *) &args[cpu])) err("uffd_poll_thread create"); } else { if (pthread_create(&uffd_threads[cpu], &attr, @@ -200,10 +211,10 @@ static int stress(struct uffd_args *args) pthread_mutex_lock(&uffd_read_mutex); } if (pthread_create(&background_threads[cpu], &attr, - background_thread, (void *)cpu)) + background_thread, (void *)&args[cpu])) return 1; } - for (cpu =3D 0; cpu < nr_parallel; cpu++) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(background_threads[cpu], NULL)) return 1; =20 @@ -216,17 +227,17 @@ static int stress(struct uffd_args *args) * UFFDIO_COPY without writing zero pages into area_dst * because the background threads already completed). */ - uffd_test_ops->release_pages(area_src); + uffd_test_ops->release_pages(gopts, gopts->area_src); =20 - finished =3D 1; - for (cpu =3D 0; cpu < nr_parallel; cpu++) + gopts->finished =3D 1; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(locking_threads[cpu], NULL)) return 1; =20 - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { char c; if (bounces & BOUNCE_POLL) { - if (write(pipefd[cpu*2+1], &c, 1) !=3D 1) + if (write(gopts->pipefd[cpu*2+1], &c, 1) !=3D 1) err("pipefd write error"); if (pthread_join(uffd_threads[cpu], (void *)&args[cpu])) @@ -242,26 +253,26 @@ static int stress(struct uffd_args *args) return 0; } =20 -static int userfaultfd_stress(void) +static int userfaultfd_stress(uffd_global_test_opts_t *gopts) { void *area; unsigned long nr; - struct uffd_args args[nr_parallel]; - uint64_t mem_size =3D nr_pages * page_size; + struct uffd_args args[gopts->nr_parallel]; + uint64_t mem_size =3D gopts->nr_pages * gopts->page_size; int flags =3D 0; =20 - memset(args, 0, sizeof(struct uffd_args) * nr_parallel); + memset(args, 0, sizeof(struct uffd_args) * gopts->nr_parallel); =20 - if (features & UFFD_FEATURE_WP_UNPOPULATED && test_type =3D=3D TEST_ANON) + if (features & UFFD_FEATURE_WP_UNPOPULATED && gopts->test_type =3D=3D TES= T_ANON) flags =3D UFFD_FEATURE_WP_UNPOPULATED; =20 - if (uffd_test_ctx_init(flags, NULL)) + if (uffd_test_ctx_init(gopts, flags, NULL)) err("context init failed"); =20 - if (posix_memalign(&area, page_size, page_size)) + if (posix_memalign(&area, gopts->page_size, gopts->page_size)) err("out of memory"); zeropage =3D area; - bzero(zeropage, page_size); + bzero(zeropage, gopts->page_size); =20 pthread_mutex_lock(&uffd_read_mutex); =20 @@ -284,18 +295,18 @@ static int userfaultfd_stress(void) fflush(stdout); =20 if (bounces & BOUNCE_POLL) - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); else - fcntl(uffd, F_SETFL, uffd_flags & ~O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags & ~O_NONBLOCK); =20 /* register */ - if (uffd_register(uffd, area_dst, mem_size, - true, test_uffdio_wp, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure"); =20 - if (area_dst_alias) { - if (uffd_register(uffd, area_dst_alias, mem_size, - true, test_uffdio_wp, false)) + if (gopts->area_dst_alias) { + if (uffd_register(gopts->uffd, gopts->area_dst_alias, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure alias"); } =20 @@ -323,87 +334,88 @@ static int userfaultfd_stress(void) * MADV_DONTNEED only after the UFFDIO_REGISTER, so it's * required to MADV_DONTNEED here. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - uffd_stats_reset(args, nr_parallel); + uffd_stats_reset(gopts, args, gopts->nr_parallel); =20 /* bounce pass */ if (stress(args)) { - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); return 1; } =20 /* Clear all the write protections if there is any */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst, - nr_pages * page_size, false); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst, + gopts->nr_pages * gopts->page_size, false); =20 /* unregister */ - if (uffd_unregister(uffd, area_dst, mem_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, mem_size)) err("unregister failure"); - if (area_dst_alias) { - if (uffd_unregister(uffd, area_dst_alias, mem_size)) + if (gopts->area_dst_alias) { + if (uffd_unregister(gopts->uffd, gopts->area_dst_alias, mem_size)) err("unregister failure alias"); } =20 /* verification */ if (bounces & BOUNCE_VERIFY) - for (nr =3D 0; nr < nr_pages; nr++) - if (*area_count(area_dst, nr) !=3D count_verify[nr]) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + if (*area_count(gopts->area_dst, nr, gopts) !=3D + gopts->count_verify[nr]) err("error area_count %llu %llu %lu\n", - *area_count(area_src, nr), - count_verify[nr], nr); + *area_count(gopts->area_src, nr, gopts), + gopts->count_verify[nr], nr); =20 /* prepare next bounce */ - swap(area_src, area_dst); + swap(gopts->area_src, gopts->area_dst); =20 - swap(area_src_alias, area_dst_alias); + swap(gopts->area_src_alias, gopts->area_dst_alias); =20 - uffd_stats_report(args, nr_parallel); + uffd_stats_report(args, gopts->nr_parallel); } - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); =20 return 0; } =20 -static void set_test_type(const char *type) +static void set_test_type(uffd_global_test_opts_t *gopts, const char *type) { if (!strcmp(type, "anon")) { - test_type =3D TEST_ANON; + gopts->test_type =3D TEST_ANON; uffd_test_ops =3D &anon_uffd_test_ops; } else if (!strcmp(type, "hugetlb")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; - map_shared =3D true; + gopts->map_shared =3D true; } else if (!strcmp(type, "hugetlb-private")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; } else if (!strcmp(type, "shmem")) { - map_shared =3D true; - test_type =3D TEST_SHMEM; + gopts->map_shared =3D true; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } else if (!strcmp(type, "shmem-private")) { - test_type =3D TEST_SHMEM; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } } =20 -static void parse_test_type_arg(const char *raw_type) +static void parse_test_type_arg(uffd_global_test_opts_t *gopts, const char= *raw_type) { - set_test_type(raw_type); + set_test_type(gopts, raw_type); =20 - if (!test_type) + if (!gopts->test_type) err("failed to parse test type argument: '%s'", raw_type); =20 - if (test_type =3D=3D TEST_HUGETLB) - page_size =3D default_huge_page_size(); + if (gopts->test_type =3D=3D TEST_HUGETLB) + gopts->page_size =3D default_huge_page_size(); else - page_size =3D sysconf(_SC_PAGE_SIZE); + gopts->page_size =3D sysconf(_SC_PAGE_SIZE); =20 - if (!page_size) + if (!gopts->page_size) err("Unable to determine page size"); - if ((unsigned long) area_count(NULL, 0) + sizeof(unsigned long long) * 2 - > page_size) + if ((unsigned long) area_count(NULL, 0, gopts) + sizeof(unsigned long lon= g) * 2 + > gopts->page_size) err("Impossible to run this test"); =20 /* @@ -415,21 +427,22 @@ static void parse_test_type_arg(const char *raw_type) if (uffd_get_features(&features) && errno =3D=3D ENOENT) ksft_exit_skip("failed to get available features (%d)\n", errno); =20 - test_uffdio_wp =3D test_uffdio_wp && + gopts->test_uffdio_wp =3D gopts->test_uffdio_wp && (features & UFFD_FEATURE_PAGEFAULT_FLAG_WP); =20 - if (test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGETLBFS_SH= MEM)) - test_uffdio_wp =3D false; + if (gopts->test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGET= LBFS_SHMEM)) + gopts->test_uffdio_wp =3D false; =20 - close(uffd); - uffd =3D -1; + close(gopts->uffd); + gopts->uffd =3D -1; } =20 static void sigalrm(int sig) { if (sig !=3D SIGALRM) abort(); - test_uffdio_copy_eexist =3D true; + /* TODO: Remove or fix this test */ + // gopts->test_uffdio_copy_eexist =3D true; alarm(ALARM_INTERVAL_SECS); } =20 @@ -438,6 +451,9 @@ int main(int argc, char **argv) unsigned long nr_cpus; size_t bytes; =20 + uffd_global_test_opts_t *gopts =3D + (uffd_global_test_opts_t *) malloc(sizeof(uffd_global_test_opts_t)); + if (argc < 4) usage(); =20 @@ -445,11 +461,11 @@ int main(int argc, char **argv) err("failed to arm SIGALRM"); alarm(ALARM_INTERVAL_SECS); =20 - parse_test_type_arg(argv[1]); + parse_test_type_arg(gopts, argv[1]); bytes =3D atol(argv[2]) * 1024 * 1024; =20 - if (test_type =3D=3D TEST_HUGETLB && - get_free_hugepages() < bytes / page_size) { + if (gopts->test_type =3D=3D TEST_HUGETLB && + get_free_hugepages() < bytes / gopts->page_size) { printf("skip: Skipping userfaultfd... not enough hugepages\n"); return KSFT_SKIP; } @@ -459,15 +475,15 @@ int main(int argc, char **argv) /* Don't let calculation below go to zero. */ ksft_print_msg("_SC_NPROCESSORS_ONLN (%lu) too large, capping nr_threads= to 32\n", nr_cpus); - nr_parallel =3D 32; + gopts->nr_parallel =3D 32; } else { - nr_parallel =3D nr_cpus; + gopts->nr_parallel =3D nr_cpus; } =20 - nr_pages_per_cpu =3D bytes / page_size / nr_parallel; - if (!nr_pages_per_cpu) { + gopts->nr_pages_per_cpu =3D bytes / gopts->page_size / gopts->nr_parallel; + if (!gopts->nr_pages_per_cpu) { _err("pages_per_cpu =3D 0, cannot test (%lu / %lu / %lu)", - bytes, page_size, nr_parallel); + bytes, gopts->page_size, gopts->nr_parallel); usage(); } =20 @@ -476,11 +492,11 @@ int main(int argc, char **argv) _err("invalid bounces"); usage(); } - nr_pages =3D nr_pages_per_cpu * nr_parallel; + gopts->nr_pages =3D gopts->nr_pages_per_cpu * gopts->nr_parallel; =20 printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n", - nr_pages, nr_pages_per_cpu); - return userfaultfd_stress(); + gopts->nr_pages, gopts->nr_pages_per_cpu); + return userfaultfd_stress(gopts); } =20 #else /* __NR_userfaultfd */ diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/s= elftests/mm/uffd-unit-tests.c index c73fd5d455c8..bed96f41c578 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -76,7 +76,7 @@ struct uffd_test_args { typedef struct uffd_test_args uffd_test_args_t; =20 /* Returns: UFFD_TEST_* */ -typedef void (*uffd_test_fn)(uffd_test_args_t *); +typedef void (*uffd_test_fn)(uffd_global_test_opts_t *, uffd_test_args_t *= ); =20 typedef struct { const char *name; @@ -181,33 +181,6 @@ static int test_uffd_api(bool use_dev) return 1; } =20 -/* - * This function initializes the global variables. TODO: remove global - * vars and then remove this. - */ -static int -uffd_setup_environment(uffd_test_args_t *args, uffd_test_case_t *test, - mem_type_t *mem_type, const char **errmsg) -{ - map_shared =3D mem_type->shared; - uffd_test_ops =3D mem_type->mem_ops; - uffd_test_case_ops =3D test->test_case_ops; - - if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) - page_size =3D default_huge_page_size(); - else - page_size =3D psize(); - - /* Ensure we have at least 2 pages */ - nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, page_size * 2) / page_size; - /* TODO: remove this global var.. it's so ugly */ - nr_parallel =3D 1; - - /* Initialize test arguments */ - args->mem_type =3D mem_type; - - return uffd_test_ctx_init(test->uffd_feature_required, errmsg); -} =20 static bool uffd_feature_supported(uffd_test_case_t *test) { @@ -237,7 +210,8 @@ static int pagemap_open(void) } while (0) =20 typedef struct { - int parent_uffd, child_uffd; + uffd_global_test_opts_t *gopts; + int child_uffd; } fork_event_args; =20 static void *fork_event_consumer(void *data) @@ -245,10 +219,10 @@ static void *fork_event_consumer(void *data) fork_event_args *args =3D data; struct uffd_msg msg =3D { 0 }; =20 - ready_for_fork =3D true; + args->gopts->ready_for_fork =3D true; =20 /* Read until a full msg received */ - while (uffd_read_msg(args->parent_uffd, &msg)); + while (uffd_read_msg(args->gopts, &msg)); =20 if (msg.event !=3D UFFD_EVENT_FORK) err("wrong message: %u\n", msg.event); @@ -304,9 +278,9 @@ static void unpin_pages(pin_args *args) args->pinned =3D false; } =20 -static int pagemap_test_fork(int uffd, bool with_event, bool test_pin) +static int pagemap_test_fork(uffd_global_test_opts_t *gopts, bool with_eve= nt, bool test_pin) { - fork_event_args args =3D { .parent_uffd =3D uffd, .child_uffd =3D -1 }; + fork_event_args args =3D { .gopts =3D gopts, .child_uffd =3D -1 }; pthread_t thread; pid_t child; uint64_t value; @@ -314,10 +288,10 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 /* Prepare a thread to resolve EVENT_FORK */ if (with_event) { - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; if (pthread_create(&thread, NULL, fork_event_consumer, &args)) err("pthread_create()"); - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ } =20 @@ -328,14 +302,14 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 fd =3D pagemap_open(); =20 - if (test_pin && pin_pages(&args, area_dst, page_size)) + if (test_pin && pin_pages(&args, gopts->area_dst, gopts->page_size)) /* * Normally when reach here we have pinned in * previous tests, so shouldn't fail anymore */ err("pin page failed in child"); =20 - value =3D pagemap_get_entry(fd, area_dst); + value =3D pagemap_get_entry(fd, gopts->area_dst); /* * After fork(), we should handle uffd-wp bit differently: * @@ -361,70 +335,70 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) return result; } =20 -static void uffd_wp_unpopulated_test(uffd_test_args_t *args) +static void uffd_wp_unpopulated_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args) { uint64_t value; int pagemap_fd; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Test applying pte marker to anon unpopulated */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); =20 /* Test unprotect on anon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test zap on anon marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - if (madvise(area_dst, page_size, MADV_DONTNEED)) + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test fault in after marker removed */ - *area_dst =3D 1; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 /* Test read-zero-page upon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - *(volatile char *)area_dst; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + *(volatile char *)gopts->area_dst; /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 uffd_test_pass(); } =20 -static void uffd_wp_fork_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_test_common(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args, bool with_event) { int pagemap_fd; uint64_t value; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in present pte", with_event ? "missing" : "stall"); goto out; @@ -442,79 +416,80 @@ static void uffd_wp_fork_test_common(uffd_test_args_t= *args, * to expose pte markers. */ if (args->mem_type->shared) { - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("MADV_DONTNEED"); } else { /* * NOTE: ignore retval because private-hugetlb doesn't yet * support swapping, so it could fail. */ - madvise(area_dst, page_size, MADV_PAGEOUT); + madvise(gopts->area_dst, gopts->page_size, MADV_PAGEOUT); } =20 /* Uffd-wp should persist even swapped out */ - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in zapped pte", with_event ? "missing" : "stall"); goto out; } =20 /* Unprotect; this tests swap pte modifications */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Fault in the page from disk */ - *area_dst =3D 2; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 2; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, nr_pages * page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts= ->page_size)) err("unregister failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_test(uffd_test_args_t *args) +static void uffd_wp_fork_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_wp_fork_test_common(args, false); + uffd_wp_fork_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_with_event_test(uffd_global_test_opts_t *gopts, u= ffd_test_args_t *args) { - uffd_wp_fork_test_common(args, true); + uffd_wp_fork_test_common(gopts, args, true); } =20 -static void uffd_wp_fork_pin_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_pin_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, bool with_event) { int pagemap_fd; pin_args pin_args =3D {}; =20 - if (uffd_register(uffd, area_dst, page_size, false, true, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->page_size, false, = true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); =20 /* * 1. First pin, then fork(). This tests fork() special path when * doing early CoW if the page is private. */ - if (pin_pages(&pin_args, area_dst, page_size)) { + if (pin_pages(&pin_args, gopts->area_dst, gopts->page_size)) { uffd_test_skip("Possibly CONFIG_GUP_TEST missing " "or unprivileged"); close(pagemap_fd); - uffd_unregister(uffd, area_dst, page_size); + uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size); return; } =20 - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in early CoW of fork()", with_event ? "missing" : "stall"); unpin_pages(&pin_args); @@ -527,49 +502,50 @@ static void uffd_wp_fork_pin_test_common(uffd_test_ar= gs_t *args, * 2. First fork(), then pin (in the child, where test_pin=3D=3Dtrue). * This tests COR, aka, page unsharing on private memories. */ - if (pagemap_test_fork(uffd, with_event, true)) { + if (pagemap_test_fork(gopts, with_event, true)) { uffd_test_fail("Detected %s uffd-wp bit when RO pin", with_event ? "missing" : "stall"); goto out; } uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("register failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_pin_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_test(uffd_global_test_opts_t *gopts, uffd_tes= t_args_t *args) { - uffd_wp_fork_pin_test_common(args, false); + uffd_wp_fork_pin_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_pin_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_with_event_test(uffd_global_test_opts_t *gopt= s, uffd_test_args_t *args) { - uffd_wp_fork_pin_test_common(args, true); + uffd_wp_fork_pin_test_common(gopts, args, true); } =20 -static void check_memory_contents(char *p) +static void check_memory_contents(uffd_global_test_opts_t *gopts, char *p) { unsigned long i, j; uint8_t expected_byte; =20 - for (i =3D 0; i < nr_pages; ++i) { + for (i =3D 0; i < gopts->nr_pages; ++i) { expected_byte =3D ~((uint8_t)(i % ((uint8_t)-1))); - for (j =3D 0; j < page_size; j++) { - uint8_t v =3D *(uint8_t *)(p + (i * page_size) + j); + for (j =3D 0; j < gopts->page_size; j++) { + uint8_t v =3D *(uint8_t *)(p + (i * gopts->page_size) + j); if (v !=3D expected_byte) err("unexpected page contents"); } } } =20 -static void uffd_minor_test_common(bool test_collapse, bool test_wp) +static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool te= st_collapse, bool test_wp) { unsigned long p; pthread_t uffd_mon; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 /* * NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing @@ -577,7 +553,7 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) */ assert(!(test_collapse && test_wp)); =20 - if (uffd_register(uffd, area_dst_alias, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst_alias, gopts->nr_pages * g= opts->page_size, /* NOTE! MADV_COLLAPSE may not work with uffd-wp */ false, test_wp, true)) err("register failure"); @@ -586,9 +562,9 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) * After registering with UFFD, populate the non-UFFD-registered side of * the shared mapping. This should *not* trigger any UFFD minor faults. */ - for (p =3D 0; p < nr_pages; ++p) - memset(area_dst + (p * page_size), p % ((uint8_t)-1), - page_size); + for (p =3D 0; p < gopts->nr_pages; ++p) + memset(gopts->area_dst + (p * gopts->page_size), p % ((uint8_t)-1), + gopts->page_size); =20 args.apply_wp =3D test_wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -600,50 +576,51 @@ static void uffd_minor_test_common(bool test_collapse= , bool test_wp) * fault. uffd_poll_thread will resolve the fault by bit-flipping the * page's contents, and then issuing a CONTINUE ioctl. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); =20 if (test_collapse) { - if (madvise(area_dst_alias, nr_pages * page_size, + if (madvise(gopts->area_dst_alias, gopts->nr_pages * gopts->page_size, MADV_COLLAPSE)) { /* It's fine to fail for this one... */ uffd_test_skip("MADV_COLLAPSE failed"); return; } =20 - uffd_test_ops->check_pmd_mapping(area_dst, - nr_pages * page_size / + uffd_test_ops->check_pmd_mapping(gopts, + gopts->area_dst, + gopts->nr_pages * gopts->page_size / read_pmd_pagesize()); /* * This won't cause uffd-fault - it purely just makes sure there * was no corruption. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); } =20 - if (args.missing_faults !=3D 0 || args.minor_faults !=3D nr_pages) + if (args.missing_faults !=3D 0 || args.minor_faults !=3D gopts->nr_pages) uffd_test_fail("stats check error"); else uffd_test_pass(); } =20 -void uffd_minor_test(uffd_test_args_t *args) +void uffd_minor_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *arg= s) { - uffd_minor_test_common(false, false); + uffd_minor_test_common(gopts, false, false); } =20 -void uffd_minor_wp_test(uffd_test_args_t *args) +void uffd_minor_wp_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *= args) { - uffd_minor_test_common(false, true); + uffd_minor_test_common(gopts, false, true); } =20 -void uffd_minor_collapse_test(uffd_test_args_t *args) +void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_minor_test_common(true, false); + uffd_minor_test_common(gopts, true, false); } =20 static sigjmp_buf jbuf, *sigbuf; @@ -678,7 +655,7 @@ static void sighndl(int sig, siginfo_t *siginfo, void *= ptr) * This also tests UFFD_FEATURE_EVENT_FORK event along with the signal * feature. Using monitor thread, verify no userfault events are generated. */ -static int faulting_process(int signal_test, bool wp) +static int faulting_process(uffd_global_test_opts_t *gopts, int signal_tes= t, bool wp) { unsigned long nr, i; unsigned long long count; @@ -687,7 +664,7 @@ static int faulting_process(int signal_test, bool wp) struct sigaction act; volatile unsigned long signalled =3D 0; =20 - split_nr_pages =3D (nr_pages + 1) / 2; + split_nr_pages =3D (gopts->nr_pages + 1) / 2; =20 if (signal_test) { sigbuf =3D &jbuf; @@ -701,7 +678,7 @@ static int faulting_process(int signal_test, bool wp) =20 for (nr =3D 0; nr < split_nr_pages; nr++) { volatile int steps =3D 1; - unsigned long offset =3D nr * page_size; + unsigned long offset =3D nr * gopts->page_size; =20 if (signal_test) { if (sigsetjmp(*sigbuf, 1) !=3D 0) { @@ -713,15 +690,15 @@ static int faulting_process(int signal_test, bool wp) if (steps =3D=3D 1) { /* This is a MISSING request */ steps++; - if (copy_page(uffd, offset, wp)) + if (copy_page(gopts, offset, wp)) signalled++; } else { /* This is a WP request */ assert(steps =3D=3D 2); - wp_range(uffd, - (__u64)area_dst + + wp_range(gopts->uffd, + (__u64)gopts->area_dst + offset, - page_size, false); + gopts->page_size, false); } } else { signalled++; @@ -730,51 +707,53 @@ static int faulting_process(int signal_test, bool wp) } } =20 - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 if (signal_test) return signalled !=3D split_nr_pages; =20 - area_dst =3D mremap(area_dst, nr_pages * page_size, nr_pages * page_size, - MREMAP_MAYMOVE | MREMAP_FIXED, area_src); - if (area_dst =3D=3D MAP_FAILED) + gopts->area_dst =3D mremap(gopts->area_dst, gopts->nr_pages * gopts->page= _size, + gopts->nr_pages * gopts->page_size, + MREMAP_MAYMOVE | MREMAP_FIXED, + gopts->area_src); + if (gopts->area_dst =3D=3D MAP_FAILED) err("mremap"); /* Reset area_src since we just clobbered it */ - area_src =3D NULL; + gopts->area_src =3D NULL; =20 - for (; nr < nr_pages; nr++) { - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) { + for (; nr < gopts->nr_pages; nr++) { + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) { err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); } /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - for (nr =3D 0; nr < nr_pages; nr++) - for (i =3D 0; i < page_size; i++) - if (*(area_dst + nr * page_size + i) !=3D 0) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + for (i =3D 0; i < gopts->page_size; i++) + if (*(gopts->area_dst + nr * gopts->page_size + i) !=3D 0) err("page %lu offset %lu is not zero", nr, i); =20 return 0; } =20 -static void uffd_sigbus_test_common(bool wp) +static void uffd_sigbus_test_common(uffd_global_test_opts_t *gopts, bool w= p) { unsigned long userfaults; pthread_t uffd_mon; @@ -782,25 +761,26 @@ static void uffd_sigbus_test_common(bool wp) int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 - if (faulting_process(1, wp)) + if (faulting_process(gopts, 1, wp)) err("faulting process failed"); =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 args.apply_wp =3D wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -808,12 +788,12 @@ static void uffd_sigbus_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(2, wp)); + exit(faulting_process(gopts, 2, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, (void **)&userfaults)) err("pthread_join()"); @@ -824,28 +804,29 @@ static void uffd_sigbus_test_common(bool wp) uffd_test_pass(); } =20 -static void uffd_sigbus_test(uffd_test_args_t *args) +static void uffd_sigbus_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_sigbus_test_common(false); + uffd_sigbus_test_common(gopts, false); } =20 -static void uffd_sigbus_wp_test(uffd_test_args_t *args) +static void uffd_sigbus_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_sigbus_test_common(true); + uffd_sigbus_test_common(gopts, true); } =20 -static void uffd_events_test_common(bool wp) +static void uffd_events_test_common(uffd_global_test_opts_t *gopts, bool w= p) { pthread_t uffd_mon; pid_t pid; int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); - if (uffd_register(uffd, area_dst, nr_pages * page_size, + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 @@ -853,7 +834,7 @@ static void uffd_events_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -861,39 +842,39 @@ static void uffd_events_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(0, wp)); + exit(faulting_process(gopts, 0, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (args.missing_faults !=3D nr_pages) + if (args.missing_faults !=3D gopts->nr_pages) uffd_test_fail("Fault counts wrong"); else uffd_test_pass(); } =20 -static void uffd_events_test(uffd_test_args_t *args) +static void uffd_events_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_events_test_common(false); + uffd_events_test_common(gopts, false); } =20 -static void uffd_events_wp_test(uffd_test_args_t *args) +static void uffd_events_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_events_test_common(true); + uffd_events_test_common(gopts, true); } =20 -static void retry_uffdio_zeropage(int ufd, +static void retry_uffdio_zeropage(uffd_global_test_opts_t *gopts, struct uffdio_zeropage *uffdio_zeropage) { - uffd_test_ops->alias_mapping(&uffdio_zeropage->range.start, + uffd_test_ops->alias_mapping(gopts, &uffdio_zeropage->range.start, uffdio_zeropage->range.len, 0); - if (ioctl(ufd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { + if (ioctl(gopts->uffd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { if (uffdio_zeropage->zeropage !=3D -EEXIST) err("UFFDIO_ZEROPAGE error: %"PRId64, (int64_t)uffdio_zeropage->zeropage); @@ -903,16 +884,16 @@ static void retry_uffdio_zeropage(int ufd, } } =20 -static bool do_uffdio_zeropage(int ufd, bool has_zeropage) +static bool do_uffdio_zeropage(uffd_global_test_opts_t *gopts, bool has_ze= ropage) { struct uffdio_zeropage uffdio_zeropage =3D { 0 }; int ret; __s64 res; =20 - uffdio_zeropage.range.start =3D (unsigned long) area_dst; - uffdio_zeropage.range.len =3D page_size; + uffdio_zeropage.range.start =3D (unsigned long) gopts->area_dst; + uffdio_zeropage.range.len =3D gopts->page_size; uffdio_zeropage.mode =3D 0; - ret =3D ioctl(ufd, UFFDIO_ZEROPAGE, &uffdio_zeropage); + ret =3D ioctl(gopts->uffd, UFFDIO_ZEROPAGE, &uffdio_zeropage); res =3D uffdio_zeropage.zeropage; if (ret) { /* real retval in ufdio_zeropage.zeropage */ @@ -921,10 +902,10 @@ static bool do_uffdio_zeropage(int ufd, bool has_zero= page) else if (res !=3D -EINVAL) err("UFFDIO_ZEROPAGE not -EINVAL"); } else if (has_zeropage) { - if (res !=3D page_size) + if (res !=3D gopts->page_size) err("UFFDIO_ZEROPAGE unexpected size"); else - retry_uffdio_zeropage(ufd, &uffdio_zeropage); + retry_uffdio_zeropage(gopts, &uffdio_zeropage); return true; } else err("UFFDIO_ZEROPAGE succeeded"); @@ -950,25 +931,29 @@ uffd_register_detect_zeropage(int uffd, void *addr, u= int64_t len) } =20 /* exercise UFFDIO_ZEROPAGE */ -static void uffd_zeropage_test(uffd_test_args_t *args) +static void uffd_zeropage_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *args) { bool has_zeropage; int i; =20 - has_zeropage =3D uffd_register_detect_zeropage(uffd, area_dst, page_size); - if (area_dst_alias) + has_zeropage =3D uffd_register_detect_zeropage(gopts->uffd, + gopts->area_dst, + gopts->page_size); + if (gopts->area_dst_alias) /* Ignore the retval; we already have it */ - uffd_register_detect_zeropage(uffd, area_dst_alias, page_size); + uffd_register_detect_zeropage(gopts->uffd, gopts->area_dst_alias, gopts-= >page_size); =20 - if (do_uffdio_zeropage(uffd, has_zeropage)) - for (i =3D 0; i < page_size; i++) - if (area_dst[i] !=3D 0) + if (do_uffdio_zeropage(gopts, has_zeropage)) + for (i =3D 0; i < gopts->page_size; i++) + if (gopts->area_dst[i] !=3D 0) err("data non-zero at offset %d\n", i); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); =20 - if (area_dst_alias && uffd_unregister(uffd, area_dst_alias, page_size)) + if (gopts->area_dst_alias && uffd_unregister(gopts->uffd, + gopts->area_dst_alias, + gopts->page_size)) err("unregister"); =20 uffd_test_pass(); @@ -987,26 +972,27 @@ static void uffd_register_poison(int uffd, void *addr= , uint64_t len) err("registered area doesn't support COPY and POISON ioctls"); } =20 -static void do_uffdio_poison(int uffd, unsigned long offset) +static void do_uffdio_poison(uffd_global_test_opts_t *gopts, unsigned long= offset) { struct uffdio_poison uffdio_poison =3D { 0 }; int ret; __s64 res; =20 - uffdio_poison.range.start =3D (unsigned long) area_dst + offset; - uffdio_poison.range.len =3D page_size; + uffdio_poison.range.start =3D (unsigned long) gopts->area_dst + offset; + uffdio_poison.range.len =3D gopts->page_size; uffdio_poison.mode =3D 0; - ret =3D ioctl(uffd, UFFDIO_POISON, &uffdio_poison); + ret =3D ioctl(gopts->uffd, UFFDIO_POISON, &uffdio_poison); res =3D uffdio_poison.updated; =20 if (ret) err("UFFDIO_POISON error: %"PRId64, (int64_t)res); - else if (res !=3D page_size) + else if (res !=3D gopts->page_size) err("UFFDIO_POISON unexpected size: %"PRId64, (int64_t)res); } =20 -static void uffd_poison_handle_fault( - struct uffd_msg *msg, struct uffd_args *args) +static void uffd_poison_handle_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -1017,17 +1003,17 @@ static void uffd_poison_handle_fault( (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; + offset &=3D ~(gopts->page_size-1); =20 /* Odd pages -> copy zeroed page; even pages -> poison. */ - if (offset & page_size) - copy_page(uffd, offset, false); + if (offset & gopts->page_size) + copy_page(gopts, offset, false); else - do_uffdio_poison(uffd, offset); + do_uffdio_poison(gopts, offset); } =20 -static void uffd_poison_test(uffd_test_args_t *targs) +static void uffd_poison_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *targs) { pthread_t uffd_mon; char c; @@ -1036,10 +1022,12 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) unsigned long nr_sigbus =3D 0; unsigned long nr; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + args.gopts =3D gopts; + + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - uffd_register_poison(uffd, area_dst, nr_pages * page_size); - memset(area_src, 0, nr_pages * page_size); + uffd_register_poison(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopt= s->page_size); + memset(gopts->area_src, 0, gopts->nr_pages * gopts->page_size); =20 args.handle_fault =3D uffd_poison_handle_fault; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -1051,9 +1039,9 @@ static void uffd_poison_test(uffd_test_args_t *targs) if (sigaction(SIGBUS, &act, 0)) err("sigaction"); =20 - for (nr =3D 0; nr < nr_pages; ++nr) { - unsigned long offset =3D nr * page_size; - const char *bytes =3D (const char *) area_dst + offset; + for (nr =3D 0; nr < gopts->nr_pages; ++nr) { + unsigned long offset =3D nr * gopts->page_size; + const char *bytes =3D (const char *) gopts->area_dst + offset; const char *i; =20 if (sigsetjmp(*sigbuf, 1)) { @@ -1066,27 +1054,29 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) continue; } =20 - for (i =3D bytes; i < bytes + page_size; ++i) { + for (i =3D bytes; i < bytes + gopts->page_size; ++i) { if (*i) err("nonzero byte in area_dst (%p) at %p: %u", - area_dst, i, *i); + gopts->area_dst, i, *i); } } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (nr_sigbus !=3D nr_pages / 2) + if (nr_sigbus !=3D gopts->nr_pages / 2) err("expected to receive %lu SIGBUS, actually received %lu", - nr_pages / 2, nr_sigbus); + gopts->nr_pages / 2, nr_sigbus); =20 uffd_test_pass(); } =20 static void -uffd_move_handle_fault_common(struct uffd_msg *msg, struct uffd_args *args, +uffd_move_handle_fault_common(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args, unsigned long len) { unsigned long offset; @@ -1098,28 +1088,32 @@ uffd_move_handle_fault_common(struct uffd_msg *msg,= struct uffd_args *args, (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR | UFFD_PAGEFAULT_= FLAG_WRITE)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; offset &=3D ~(len-1); =20 - if (move_page(uffd, offset, len)) + if (move_page(gopts, offset, len)) args->missing_faults++; } =20 -static void uffd_move_handle_fault(struct uffd_msg *msg, +static void uffd_move_handle_fault(uffd_global_test_opts_t *gopts, struct = uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, page_size); + uffd_move_handle_fault_common(gopts, msg, args, gopts->page_size); } =20 -static void uffd_move_pmd_handle_fault(struct uffd_msg *msg, +static void uffd_move_pmd_handle_fault(uffd_global_test_opts_t *gopts, str= uct uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, read_pmd_pagesize()); + uffd_move_handle_fault_common(gopts, msg, args, read_pmd_pagesize()); } =20 static void -uffd_move_test_common(uffd_test_args_t *targs, unsigned long chunk_size, - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args)) +uffd_move_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *targs, + unsigned long chunk_size, + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, struct uffd_args *args) +) { unsigned long nr; pthread_t uffd_mon; @@ -1131,11 +1125,13 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, unsigned long src_offs =3D 0; unsigned long dst_offs =3D 0; =20 + args.gopts =3D gopts; + /* Prevent source pages from being mapped more than once */ - if (madvise(area_src, nr_pages * page_size, MADV_DONTFORK)) + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DON= TFORK)) err("madvise(MADV_DONTFORK) failure"); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("register failure"); =20 @@ -1143,22 +1139,22 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - step_size =3D chunk_size / page_size; - step_count =3D nr_pages / step_size; + step_size =3D chunk_size / gopts->page_size; + step_count =3D gopts->nr_pages / step_size; =20 - if (chunk_size > page_size) { - char *aligned_src =3D ALIGN_UP(area_src, chunk_size); - char *aligned_dst =3D ALIGN_UP(area_dst, chunk_size); + if (chunk_size > gopts->page_size) { + char *aligned_src =3D ALIGN_UP(gopts->area_src, chunk_size); + char *aligned_dst =3D ALIGN_UP(gopts->area_dst, chunk_size); =20 - if (aligned_src !=3D area_src || aligned_dst !=3D area_dst) { - src_offs =3D (aligned_src - area_src) / page_size; - dst_offs =3D (aligned_dst - area_dst) / page_size; + if (aligned_src !=3D gopts->area_src || aligned_dst !=3D gopts->area_dst= ) { + src_offs =3D (aligned_src - gopts->area_src) / gopts->page_size; + dst_offs =3D (aligned_dst - gopts->area_dst) / gopts->page_size; step_count--; } - orig_area_src =3D area_src; - orig_area_dst =3D area_dst; - area_src =3D aligned_src; - area_dst =3D aligned_dst; + orig_area_src =3D gopts->area_src; + orig_area_dst =3D gopts->area_dst; + gopts->area_src =3D aligned_src; + gopts->area_dst =3D aligned_dst; } =20 /* @@ -1172,34 +1168,34 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, =20 /* Check area_src content */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); - if (count !=3D count_verify[src_offs + nr + i]) + count =3D *area_count(gopts->area_src, nr + i, gopts); + if (count !=3D gopts->count_verify[src_offs + nr + i]) err("nr %lu source memory invalid %llu %llu\n", - nr + i, count, count_verify[src_offs + nr + i]); + nr + i, count, gopts->count_verify[src_offs + nr + i]); } =20 /* Faulting into area_dst should move the page or the huge page */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_dst, nr + i); - if (count !=3D count_verify[dst_offs + nr + i]) + count =3D *area_count(gopts->area_dst, nr + i, gopts); + if (count !=3D gopts->count_verify[dst_offs + nr + i]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[dst_offs + nr + i]); + nr, count, gopts->count_verify[dst_offs + nr + i]); } =20 /* Re-check area_src content which should be empty */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); + count =3D *area_count(gopts->area_src, nr + i, gopts); if (count !=3D 0) err("nr %lu move failed %llu %llu\n", - nr, count, count_verify[src_offs + nr + i]); + nr, count, gopts->count_verify[src_offs + nr + i]); } } - if (chunk_size > page_size) { - area_src =3D orig_area_src; - area_dst =3D orig_area_dst; + if (chunk_size > gopts->page_size) { + gopts->area_src =3D orig_area_src; + gopts->area_dst =3D orig_area_dst; } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); @@ -1210,24 +1206,24 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, uffd_test_pass(); } =20 -static void uffd_move_test(uffd_test_args_t *targs) +static void uffd_move_test(uffd_global_test_opts_t *gopts, uffd_test_args_= t *targs) { - uffd_move_test_common(targs, page_size, uffd_move_handle_fault); + uffd_move_test_common(gopts, targs, gopts->page_size, uffd_move_handle_fa= ult); } =20 -static void uffd_move_pmd_test(uffd_test_args_t *targs) +static void uffd_move_pmd_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) err("madvise(MADV_HUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static void uffd_move_pmd_split_test(uffd_test_args_t *targs) +static void uffd_move_pmd_split_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) err("madvise(MADV_NOHUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 @@ -1287,6 +1283,11 @@ typedef enum { THR_STATE_UNINTERRUPTIBLE, } thread_state; =20 +typedef struct { + uffd_global_test_opts_t *gopts; + volatile pid_t *pid; +} mmap_changing_thread_args; + static void sleep_short(void) { usleep(1000); @@ -1329,7 +1330,9 @@ static void thread_state_until(pid_t tid, thread_stat= e state) =20 static void *uffd_mmap_changing_thread(void *opaque) { - volatile pid_t *pid =3D opaque; + mmap_changing_thread_args *args =3D opaque; + uffd_global_test_opts_t *gopts =3D args->gopts; + volatile pid_t *pid =3D args->pid; int ret; =20 /* Unfortunately, it's only fetch-able from the thread itself.. */ @@ -1337,21 +1340,21 @@ static void *uffd_mmap_changing_thread(void *opaque) *pid =3D syscall(SYS_gettid); =20 /* Inject an event, this will hang solid until the event read */ - ret =3D madvise(area_dst, page_size, MADV_REMOVE); + ret =3D madvise(gopts->area_dst, gopts->page_size, MADV_REMOVE); if (ret) err("madvise(MADV_REMOVE) failed"); =20 return NULL; } =20 -static void uffd_consume_message(int fd) +static void uffd_consume_message(uffd_global_test_opts_t *gopts) { struct uffd_msg msg =3D { 0 }; =20 - while (uffd_read_msg(fd, &msg)); + while (uffd_read_msg(gopts, &msg)); } =20 -static void uffd_mmap_changing_test(uffd_test_args_t *targs) +static void uffd_mmap_changing_test(uffd_global_test_opts_t *gopts, uffd_t= est_args_t *targs) { /* * This stores the real PID (which can be different from how tid is @@ -1360,13 +1363,14 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) pid_t pid =3D 0; pthread_t tid; int ret; + mmap_changing_thread_args args =3D { gopts, &pid }; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("uffd_register() failed"); =20 /* Create a thread to generate the racy event */ - ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &pid); + ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &args); if (ret) err("pthread_create() failed"); =20 @@ -1380,26 +1384,26 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) /* Wait until the thread hangs at REMOVE event */ thread_state_until(pid, THR_STATE_UNINTERRUPTIBLE); =20 - if (!uffdio_mmap_changing_test_copy(uffd)) + if (!uffdio_mmap_changing_test_copy(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_zeropage(uffd)) + if (!uffdio_mmap_changing_test_zeropage(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_move(uffd)) + if (!uffdio_mmap_changing_test_move(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_poison(uffd)) + if (!uffdio_mmap_changing_test_poison(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_continue(uffd)) + if (!uffdio_mmap_changing_test_continue(gopts->uffd)) return; =20 /* * All succeeded above! Recycle everything. Start by reading the * event so as to kick the thread roll again.. */ - uffd_consume_message(uffd); + uffd_consume_message(gopts); =20 ret =3D pthread_join(tid, NULL); assert(ret =3D=3D 0); @@ -1407,10 +1411,10 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) uffd_test_pass(); } =20 -static int prevent_hugepages(const char **errmsg) +static int prevent_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_NOHUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) { /* Ignore only if CONFIG_TRANSPARENT_HUGEPAGE=3Dn */ if (errno !=3D EINVAL) { if (errmsg) @@ -1421,10 +1425,10 @@ static int prevent_hugepages(const char **errmsg) return 0; } =20 -static int request_hugepages(const char **errmsg) +static int request_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_HUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) { if (errmsg) { *errmsg =3D (errno =3D=3D EINVAL) ? "CONFIG_TRANSPARENT_HUGEPAGE is not set" : @@ -1448,13 +1452,17 @@ struct uffd_test_case_ops uffd_move_test_pmd_case_o= ps =3D { * Note that _UFFDIO_ZEROPAGE is tested separately in the zeropage test. */ static void -do_register_ioctls_test(uffd_test_args_t *args, bool miss, bool wp, bool m= inor) +do_register_ioctls_test(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, + bool miss, + bool wp, + bool minor) { uint64_t ioctls =3D 0, expected =3D BIT_ULL(_UFFDIO_WAKE); mem_type_t *mem_type =3D args->mem_type; int ret; =20 - ret =3D uffd_register_with_ioctls(uffd, area_dst, page_size, + ret =3D uffd_register_with_ioctls(gopts->uffd, gopts->area_dst, gopts->pa= ge_size, miss, wp, minor, &ioctls); =20 /* @@ -1485,18 +1493,18 @@ do_register_ioctls_test(uffd_test_args_t *args, boo= l miss, bool wp, bool minor) "(miss=3D%d, wp=3D%d, minor=3D%d): expected=3D0x%"PRIx64", " "returned=3D0x%"PRIx64, miss, wp, minor, expected, ioctls); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); } =20 -static void uffd_register_ioctls_test(uffd_test_args_t *args) +static void uffd_register_ioctls_test(uffd_global_test_opts_t *gopts, uffd= _test_args_t *args) { int miss, wp, minor; =20 for (miss =3D 0; miss <=3D 1; miss++) for (wp =3D 0; wp <=3D 1; wp++) for (minor =3D 0; minor <=3D 1; minor++) - do_register_ioctls_test(args, miss, wp, minor); + do_register_ioctls_test(gopts, args, miss, wp, minor); =20 uffd_test_pass(); } @@ -1734,6 +1742,28 @@ int main(int argc, char *argv[]) } for (j =3D 0; j < n_mems; j++) { mem_type =3D &mem_types[j]; + + /* Initialize global test options */ + uffd_global_test_opts_t gopts; + + gopts.map_shared =3D mem_type->shared; + uffd_test_ops =3D mem_type->mem_ops; + uffd_test_case_ops =3D test->test_case_ops; + + if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) + gopts.page_size =3D default_huge_page_size(); + else + gopts.page_size =3D psize(); + + /* Ensure we have at least 2 pages */ + gopts.nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, gopts.page_size * 2) + / gopts.page_size; + + gopts.nr_parallel =3D 1; + + /* Initialize test arguments */ + args.mem_type =3D mem_type; + if (!(test->mem_targets & mem_type->mem_flag)) continue; =20 @@ -1748,13 +1778,12 @@ int main(int argc, char *argv[]) uffd_test_skip("feature missing"); continue; } - if (uffd_setup_environment(&args, test, mem_type, - &errmsg)) { + if (uffd_test_ctx_init(&gopts, test->uffd_feature_required, &errmsg)) { uffd_test_skip(errmsg); continue; } - test->uffd_fn(&args); - uffd_test_ctx_clear(); + test->uffd_fn(&gopts, &args); + uffd_test_ctx_clear(&gopts); } } =20 diff --git a/tools/testing/selftests/mm/uffd-wp-mremap.c b/tools/testing/se= lftests/mm/uffd-wp-mremap.c index c2ba7d46c7b4..c286b1015f32 100644 --- a/tools/testing/selftests/mm/uffd-wp-mremap.c +++ b/tools/testing/selftests/mm/uffd-wp-mremap.c @@ -157,7 +157,11 @@ static bool range_is_swapped(void *addr, size_t size) return true; } =20 -static void test_one_folio(size_t size, bool private, bool swapout, bool h= ugetlb) +static void test_one_folio(uffd_global_test_opts_t *gopts, + size_t size, + bool private, + bool swapout, + bool hugetlb) { struct uffdio_writeprotect wp_prms; uint64_t features =3D 0; @@ -181,21 +185,21 @@ static void test_one_folio(size_t size, bool private,= bool swapout, bool hugetlb } =20 /* Register range for uffd-wp. */ - if (userfaultfd_open(&features)) { + if (userfaultfd_open(gopts, &features)) { if (errno =3D=3D ENOENT) ksft_test_result_skip("userfaultfd not available\n"); else ksft_test_result_fail("userfaultfd_open() failed\n"); goto out; } - if (uffd_register(uffd, mem, size, false, true, false)) { + if (uffd_register(gopts->uffd, mem, size, false, true, false)) { ksft_test_result_fail("uffd_register() failed\n"); goto out; } wp_prms.mode =3D UFFDIO_WRITEPROTECT_MODE_WP; wp_prms.range.start =3D (uintptr_t)mem; wp_prms.range.len =3D size; - if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { + if (ioctl(gopts->uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { ksft_test_result_fail("ioctl(UFFDIO_WRITEPROTECT) failed\n"); goto out; } @@ -242,9 +246,9 @@ static void test_one_folio(size_t size, bool private, b= ool swapout, bool hugetlb out: if (mem) munmap(mem, size); - if (uffd >=3D 0) { - close(uffd); - uffd =3D -1; + if (gopts->uffd >=3D 0) { + close(gopts->uffd); + gopts->uffd =3D -1; } } =20 @@ -336,6 +340,7 @@ static const struct testcase testcases[] =3D { =20 int main(int argc, char **argv) { + uffd_global_test_opts_t gopts; struct thp_settings settings; int i, j, plan =3D 0; =20 @@ -367,8 +372,8 @@ int main(int argc, char **argv) const struct testcase *tc =3D &testcases[i]; =20 for (j =3D 0; j < *tc->nr_sizes; j++) - test_one_folio(tc->sizes[j], tc->private, tc->swapout, - tc->hugetlb); + test_one_folio(&gopts, tc->sizes[j], tc->private, + tc->swapout, tc->hugetlb); } =20 /* If THP is supported, restore original THP settings. */ --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A94B072626; Sat, 10 May 2025 16:05:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746893114; cv=none; b=PkrGQ3u4Dnd2UtSxT8KXrPuLxIDi6ZaJex0c1zHT20SFH8o++86GAYNmbZomZP2IZhGj7+kYuwNSqTdzvZVtDHjMnwiwnaXlkfB5DQbBm50N871Z2mCjEqgOlt6gwbatpGzA6fvDS7D0MS8IjcwE4NUKDyA9oVDcKInTfZs0d7I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746893114; c=relaxed/simple; bh=4jxCdwvh3yxoCniZThJ+yQ2b2OcugxyLdpnZLhDnzts=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BqDr9S8D0uAKt+Fjy4oc3P4KYjHYlAVquy/T49z4BYfZUS23hCyoagrL1A+/8D9EWG8Oc3IWF8Yo0P4TNRes/R8pfKtAAFRdu2eDnf+M7dRHaBAAu4m2Whrujhkvj9q1uf8x+JdKXe1qXF4QtpYpiQBdZKupcJ6AQ1Y9oKiIiYc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jvpDwCon; arc=none smtp.client-ip=209.85.215.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jvpDwCon" Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-b200178c011so2390317a12.3; Sat, 10 May 2025 09:05:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746893110; x=1747497910; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dMhcL4VSxmAtsrehb7caGmrn3eGp2i0wzMty1gpO6uA=; b=jvpDwCont71JOjpTyridqA2Yg33sVVMX+40zOlSLdcd0btq8NF87uPLv1CzBFpMZGT dOp0ttWmDHezVYUkF6wlRJ0UdpoNlgFt68iibGMilezdSt+NNe0iempTnjpBFepR+Yh7 nI5uwwdugsWaSl+AaT2HezGNOgPGtPC+S17El2Ky4byAdAXOtDhbiFI9ZGurP59D9llm 7Ou4Omu7pdUC9OQD3PyrOLiutyzMmZsj78MJz2nLTZsh8GkD+eKyJZY7KVkqjAhBLJ9f TvekwJ5+2I+ebBKpQ+MV+D+RTLHXQowrmQPg6Ba9zX2mXWWSUgqLDGDzBF+4/YkhdfGv Qg9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746893110; x=1747497910; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dMhcL4VSxmAtsrehb7caGmrn3eGp2i0wzMty1gpO6uA=; b=tesGw56tfyf7yIl6aQL184qg47m1Bx/tiFioOX8JJRsHzZkfl9jBiQLS68s/omRVyJ LZBCMp5Vf0SpTUsbWtMdtns/GMnPl23su7e1xK0FsMUna6joBfoBkYBOKWoLr8vcDEB8 YMIujV1eLiuqJXbMpDKNh8baUO2saMU0reCwzvvoa5ZVpReKINGDAkW2GECu3FlwC6I1 F+GVP42AVl39+SRrHNrboMxeVgM8VJyyiIQLH433YvC2/nc1mec587pQanMl4+1AE3bw OfNHtu35w7DUKBk4LY0euwPmP2xEKGqkDbbVIndP7x4vQjUCH6gSEBpHrpXVBVzIuWmZ pxpg== X-Forwarded-Encrypted: i=1; AJvYcCVz6xpMRSPyiUBEi0E8aWO1dbuGZCP0iKxoXPRhjGSyF7s4k3R8W6zo7hAhgE9JIkv8iExer7UI7oOwGgE=@vger.kernel.org, AJvYcCXMQsOaWVFiIEQAD+PR5BIqJlCa5uzRMxsT+cqscH1KSUs+M3BU+EYKPgiqNo+SGA/3yVYcFrrZ7TYuNXpoHgD8@vger.kernel.org X-Gm-Message-State: AOJu0YyIqM9/Ak3yH1FXzvbfo4X8j3Z1WNYz53Quj82RGEIgTJOq5eAd aRkIVn2kXuUtLpD+dJcmvjWYSZ+ym8JEEUPITWmG4i2CrmbdZsM= X-Gm-Gg: ASbGncv7jlPkgKLliI0hZuTGFYP9ZYGljHYVXYZMZGWKwXO8a9qZ3zU/b0VLdugXvDZ sh/nwiazV+C4F3ERKcBNiHgQS5/RmDhb7q+uR6Nf2O4BxmKzoAZ8nCqnz5lThJpIv8el/acmNBP KYOGXyWlh3kJ8LwNQX6+KW1TONkFDY2Dzml6vrCEFkGxq5pGGEBbezhXDFJbUBXRXmZplh8SSoW DvzEoNnKhq63iEIxTABMatndd6zQ30TVFzuXWhOrLZmkX82U+fYwuH2Ue/kfeHmalmH1GbSB/Nb vENAtcTFJQyDujyLzCDi+/5WTAoeXVB8GsYHCNsL4R62UbQIq2/6sceiwcwIAw== X-Google-Smtp-Source: AGHT+IFvEp+sKb/mTYPsnuocFSmD6NBKJTLxsrjS/AYU06AJtZFXEVxQnsIVYPugp6s/9e4F842IvA== X-Received: by 2002:a17:902:c950:b0:22f:c83d:d726 with SMTP id d9443c01a7336-22fc8e99ec8mr103135795ad.33.1746893109249; Sat, 10 May 2025 09:05:09 -0700 (PDT) Received: from debian.ujwal.com ([223.185.131.123]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22fc828a463sm34624145ad.167.2025.05.10.09.05.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 May 2025 09:05:08 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, peterx@redhat.com, shuah@kernel.org Cc: jackmanb@google.com, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Ujwal Kundur Subject: [PATCH v3 1/1] selftests/mm/uffd: Refactor non-composite global vars into struct Date: Sat, 10 May 2025 21:33:35 +0530 Message-Id: <20250510160335.1898-1-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor macros and non-composite global variable definitions into a struct that is defined at the start of a test and is passed around instead of relying on global vars. Signed-off-by: Ujwal Kundur Reviewed-by: Brendan Jackman Tested-by: Brendan Jackman --- Changes since v2: - redo patch on mm-new branch Changes since v1: - indentation fixes - squash into single patch to assist bisections tools/testing/selftests/mm/uffd-common.c | 275 ++++----- tools/testing/selftests/mm/uffd-common.h | 78 +-- tools/testing/selftests/mm/uffd-stress.c | 226 ++++---- tools/testing/selftests/mm/uffd-unit-tests.c | 557 ++++++++++--------- tools/testing/selftests/mm/uffd-wp-mremap.c | 23 +- 5 files changed, 615 insertions(+), 544 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selft= ests/mm/uffd-common.c index a37088a23ffe..1b13107ef3c3 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -7,18 +7,31 @@ =20 #include "uffd-common.h" =20 -#define BASE_PMD_ADDR ((void *)(1UL << 30)) - -volatile bool test_uffdio_copy_eexist =3D true; -unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; -int uffd =3D -1, uffd_flags, finished, *pipefd, test_type; -bool map_shared; -bool test_uffdio_wp =3D true; -unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; uffd_test_case_ops_t *uffd_test_case_ops; -atomic_bool ready_for_fork; + +#define BASE_PMD_ADDR ((void *)(1UL << 30)) + +/* pthread_mutex_t starts at page offset 0 */ +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts) +{ + return (pthread_mutex_t *) (area + nr * gopts->page_size); +} + +/* + * count is placed in the page after pthread_mutex_t naturally aligned + * to avoid non alignment faults on non-x86 archs. + */ +volatile unsigned long long *area_count( + char *area, unsigned long nr, + uffd_global_test_opts_t *gopts) +{ + return (volatile unsigned long long *) + ((unsigned long) (area + nr * gopts->page_size + + sizeof(pthread_mutex_t) + + sizeof(unsigned long long) - 1) & + ~(unsigned long)(sizeof(unsigned long long) - 1)); +} =20 static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { @@ -40,15 +53,15 @@ static int uffd_mem_fd_create(off_t mem_size, bool huge= tlb) return mem_fd; } =20 -static void anon_release_pages(char *rel_area) +static void anon_release_pages(uffd_global_test_opts_t *gopts, char *rel_a= rea) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } =20 -static int anon_allocate_area(void **alloc_area, bool is_src) +static int anon_allocate_area(uffd_global_test_opts_t *gopts, void **alloc= _area, bool is_src) { - *alloc_area =3D mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, + *alloc_area =3D mmap(NULL, gopts->nr_pages * gopts->page_size, PROT_READ = | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); if (*alloc_area =3D=3D MAP_FAILED) { *alloc_area =3D NULL; @@ -57,31 +70,33 @@ static int anon_allocate_area(void **alloc_area, bool i= s_src) return 0; } =20 -static void noop_alias_mapping(__u64 *start, size_t len, unsigned long off= set) +static void noop_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, size_t len, + unsigned long offset) { } =20 -static void hugetlb_release_pages(char *rel_area) +static void hugetlb_release_pages(uffd_global_test_opts_t *gopts, char *re= l_area) { - if (!map_shared) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (!gopts->map_shared) { + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } else { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } } =20 -static int hugetlb_allocate_area(void **alloc_area, bool is_src) +static int hugetlb_allocate_area(uffd_global_test_opts_t *gopts, void **al= loc_area, bool is_src) { - off_t size =3D nr_pages * page_size; + off_t size =3D gopts->nr_pages * gopts->page_size; off_t offset =3D is_src ? 0 : size; void *area_alias =3D NULL; char **alloc_area_alias; int mem_fd =3D uffd_mem_fd_create(size * 2, true); =20 *alloc_area =3D mmap(NULL, size, PROT_READ | PROT_WRITE, - (map_shared ? MAP_SHARED : MAP_PRIVATE) | + (gopts->map_shared ? MAP_SHARED : MAP_PRIVATE) | (is_src ? 0 : MAP_NORESERVE), mem_fd, offset); if (*alloc_area =3D=3D MAP_FAILED) { @@ -89,7 +104,7 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) return -errno; } =20 - if (map_shared) { + if (gopts->map_shared) { area_alias =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, mem_fd, offset); if (area_alias =3D=3D MAP_FAILED) @@ -97,9 +112,9 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) } =20 if (is_src) { - alloc_area_alias =3D &area_src_alias; + alloc_area_alias =3D &gopts->area_src_alias; } else { - alloc_area_alias =3D &area_dst_alias; + alloc_area_alias =3D &gopts->area_dst_alias; } if (area_alias) *alloc_area_alias =3D area_alias; @@ -108,24 +123,26 @@ static int hugetlb_allocate_area(void **alloc_area, b= ool is_src) return 0; } =20 -static void hugetlb_alias_mapping(__u64 *start, size_t len, unsigned long = offset) +static void hugetlb_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, size_t len, + unsigned long offset) { - if (!map_shared) + if (!gopts->map_shared) return; =20 - *start =3D (unsigned long) area_dst_alias + offset; + *start =3D (unsigned long) gopts->area_dst_alias + offset; } =20 -static void shmem_release_pages(char *rel_area) +static void shmem_release_pages(uffd_global_test_opts_t *gopts, char *rel_= area) { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } =20 -static int shmem_allocate_area(void **alloc_area, bool is_src) +static int shmem_allocate_area(uffd_global_test_opts_t *gopts, void **allo= c_area, bool is_src) { void *area_alias =3D NULL; - size_t bytes =3D nr_pages * page_size, hpage_size =3D read_pmd_pagesize(); + size_t bytes =3D gopts->nr_pages * gopts->page_size, hpage_size =3D read_= pmd_pagesize(); unsigned long offset =3D is_src ? 0 : bytes; char *p =3D NULL, *p_alias =3D NULL; int mem_fd =3D uffd_mem_fd_create(bytes * 2, false); @@ -159,22 +176,24 @@ static int shmem_allocate_area(void **alloc_area, boo= l is_src) err("mmap of anonymous memory failed at %p", p_alias); =20 if (is_src) - area_src_alias =3D area_alias; + gopts->area_src_alias =3D area_alias; else - area_dst_alias =3D area_alias; + gopts->area_dst_alias =3D area_alias; =20 close(mem_fd); return 0; } =20 -static void shmem_alias_mapping(__u64 *start, size_t len, unsigned long of= fset) +static void shmem_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, + size_t len, unsigned long offset) { - *start =3D (unsigned long)area_dst_alias + offset; + *start =3D (unsigned long)gopts->area_dst_alias + offset; } =20 -static void shmem_check_pmd_mapping(void *p, int expect_nr_hpages) +static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *= p, int expect_nr_hpages) { - if (!check_huge_shmem(area_dst_alias, expect_nr_hpages, + if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages, read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); @@ -234,18 +253,18 @@ void uffd_stats_report(struct uffd_args *args, int n_= cpus) printf("\n"); } =20 -int userfaultfd_open(uint64_t *features) +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features) { struct uffdio_api uffdio_api; =20 - uffd =3D uffd_open(UFFD_FLAGS); - if (uffd < 0) + gopts->uffd =3D uffd_open(UFFD_FLAGS); + if (gopts->uffd < 0) return -1; - uffd_flags =3D fcntl(uffd, F_GETFD, NULL); + gopts->uffd_flags =3D fcntl(gopts->uffd, F_GETFD, NULL); =20 uffdio_api.api =3D UFFD_API; uffdio_api.features =3D *features; - if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + if (ioctl(gopts->uffd, UFFDIO_API, &uffdio_api)) /* Probably lack of CAP_PTRACE? */ return -1; if (uffdio_api.api !=3D UFFD_API) @@ -255,59 +274,59 @@ int userfaultfd_open(uint64_t *features) return 0; } =20 -static inline void munmap_area(void **area) +static inline void munmap_area(uffd_global_test_opts_t *gopts, void **area) { if (*area) - if (munmap(*area, nr_pages * page_size)) + if (munmap(*area, gopts->nr_pages * gopts->page_size)) err("munmap"); =20 *area =3D NULL; } =20 -void uffd_test_ctx_clear(void) +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts) { size_t i; =20 - if (pipefd) { - for (i =3D 0; i < nr_parallel * 2; ++i) { - if (close(pipefd[i])) + if (gopts->pipefd) { + for (i =3D 0; i < gopts->nr_parallel * 2; ++i) { + if (close(gopts->pipefd[i])) err("close pipefd"); } - free(pipefd); - pipefd =3D NULL; + free(gopts->pipefd); + gopts->pipefd =3D NULL; } =20 - if (count_verify) { - free(count_verify); - count_verify =3D NULL; + if (gopts->count_verify) { + free(gopts->count_verify); + gopts->count_verify =3D NULL; } =20 - if (uffd !=3D -1) { - if (close(uffd)) + if (gopts->uffd !=3D -1) { + if (close(gopts->uffd)) err("close uffd"); - uffd =3D -1; + gopts->uffd =3D -1; } =20 - munmap_area((void **)&area_src); - munmap_area((void **)&area_src_alias); - munmap_area((void **)&area_dst); - munmap_area((void **)&area_dst_alias); - munmap_area((void **)&area_remap); + munmap_area(gopts, (void **)&gopts->area_src); + munmap_area(gopts, (void **)&gopts->area_src_alias); + munmap_area(gopts, (void **)&gopts->area_dst); + munmap_area(gopts, (void **)&gopts->area_dst_alias); + munmap_area(gopts, (void **)&gopts->area_remap); } =20 -int uffd_test_ctx_init(uint64_t features, const char **errmsg) +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg) { unsigned long nr, cpu; int ret; =20 if (uffd_test_case_ops && uffd_test_case_ops->pre_alloc) { - ret =3D uffd_test_case_ops->pre_alloc(errmsg); + ret =3D uffd_test_case_ops->pre_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D uffd_test_ops->allocate_area((void **)&area_src, true); - ret |=3D uffd_test_ops->allocate_area((void **)&area_dst, false); + ret =3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_src, t= rue); + ret |=3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_dst, = false); if (ret) { if (errmsg) *errmsg =3D "memory allocation failed"; @@ -315,26 +334,26 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) } =20 if (uffd_test_case_ops && uffd_test_case_ops->post_alloc) { - ret =3D uffd_test_case_ops->post_alloc(errmsg); + ret =3D uffd_test_case_ops->post_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D userfaultfd_open(&features); + ret =3D userfaultfd_open(gopts, &features); if (ret) { if (errmsg) *errmsg =3D "possible lack of privilege"; return ret; } =20 - count_verify =3D malloc(nr_pages * sizeof(unsigned long long)); - if (!count_verify) + gopts->count_verify =3D malloc(gopts->nr_pages * sizeof(unsigned long lon= g)); + if (!gopts->count_verify) err("count_verify"); =20 - for (nr =3D 0; nr < nr_pages; nr++) { - *area_mutex(area_src, nr) =3D + for (nr =3D 0; nr < gopts->nr_pages; nr++) { + *area_mutex(gopts->area_src, nr, gopts) =3D (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER; - count_verify[nr] =3D *area_count(area_src, nr) =3D 1; + gopts->count_verify[nr] =3D *area_count(gopts->area_src, nr, gopts) =3D = 1; /* * In the transition between 255 to 256, powerpc will * read out of order in my_bcmp and see both bytes as @@ -342,7 +361,7 @@ int uffd_test_ctx_init(uint64_t features, const char **= errmsg) * after the count, to avoid my_bcmp to trigger false * positives. */ - *(area_count(area_src, nr) + 1) =3D 1; + *(area_count(gopts->area_src, nr, gopts) + 1) =3D 1; } =20 /* @@ -363,13 +382,13 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) * proactively split the thp and drop any accidentally initialized * pages within area_dst. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - pipefd =3D malloc(sizeof(int) * nr_parallel * 2); - if (!pipefd) + gopts->pipefd =3D malloc(sizeof(int) * gopts->nr_parallel * 2); + if (!gopts->pipefd) err("pipefd"); - for (cpu =3D 0; cpu < nr_parallel; cpu++) - if (pipe2(&pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) + if (pipe2(&gopts->pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) err("pipe"); =20 return 0; @@ -416,9 +435,9 @@ static void continue_range(int ufd, __u64 start, __u64 = len, bool wp) ret, (int64_t) req.mapped); } =20 -int uffd_read_msg(int ufd, struct uffd_msg *msg) +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg) { - int ret =3D read(uffd, msg, sizeof(*msg)); + int ret =3D read(gopts->uffd, msg, sizeof(*msg)); =20 if (ret !=3D sizeof(*msg)) { if (ret < 0) { @@ -433,7 +452,9 @@ int uffd_read_msg(int ufd, struct uffd_msg *msg) return 0; } =20 -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args) +void uffd_handle_page_fault( + uffd_global_test_opts_t *gopts, struct uffd_msg *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -442,7 +463,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) =20 if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP) { /* Write protect page faults */ - wp_range(uffd, msg->arg.pagefault.address, page_size, false); + wp_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, fals= e); args->wp_faults++; } else if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR) { uint8_t *area; @@ -460,12 +481,12 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) * (UFFD-registered). */ =20 - area =3D (uint8_t *)(area_dst + + area =3D (uint8_t *)(gopts->area_dst + ((char *)msg->arg.pagefault.address - - area_dst_alias)); - for (b =3D 0; b < page_size; ++b) + gopts->area_dst_alias)); + for (b =3D 0; b < gopts->page_size; ++b) area[b] =3D ~area[b]; - continue_range(uffd, msg->arg.pagefault.address, page_size, + continue_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, args->apply_wp); args->minor_faults++; } else { @@ -493,10 +514,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WRITE) err("unexpected write fault"); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->ar= ea_dst; + offset &=3D ~(gopts->page_size-1); =20 - if (copy_page(uffd, offset, args->apply_wp)) + if (copy_page(gopts, offset, args->apply_wp)) args->missing_faults++; } } @@ -504,6 +525,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) void *uffd_poll_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; unsigned long cpu =3D args->cpu; struct pollfd pollfd[2]; struct uffd_msg msg; @@ -514,12 +536,12 @@ void *uffd_poll_thread(void *arg) if (!args->handle_fault) args->handle_fault =3D uffd_handle_page_fault; =20 - pollfd[0].fd =3D uffd; + pollfd[0].fd =3D gopts->uffd; pollfd[0].events =3D POLLIN; - pollfd[1].fd =3D pipefd[cpu*2]; + pollfd[1].fd =3D gopts->pipefd[cpu*2]; pollfd[1].events =3D POLLIN; =20 - ready_for_fork =3D true; + gopts->ready_for_fork =3D true; =20 for (;;) { ret =3D poll(pollfd, 2, -1); @@ -537,30 +559,30 @@ void *uffd_poll_thread(void *arg) } if (!(pollfd[0].revents & POLLIN)) err("pollfd[0].revents %d", pollfd[0].revents); - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; switch (msg.event) { default: err("unexpected msg event %u\n", msg.event); break; case UFFD_EVENT_PAGEFAULT: - args->handle_fault(&msg, args); + args->handle_fault(gopts, &msg, args); break; case UFFD_EVENT_FORK: - close(uffd); - uffd =3D msg.arg.fork.ufd; - pollfd[0].fd =3D uffd; + close(gopts->uffd); + gopts->uffd =3D msg.arg.fork.ufd; + pollfd[0].fd =3D gopts->uffd; break; case UFFD_EVENT_REMOVE: uffd_reg.range.start =3D msg.arg.remove.start; uffd_reg.range.len =3D msg.arg.remove.end - msg.arg.remove.start; - if (ioctl(uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) + if (ioctl(gopts->uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) err("remove failure"); break; case UFFD_EVENT_REMAP: - area_remap =3D area_dst; /* save for later unmap */ - area_dst =3D (char *)(unsigned long)msg.arg.remap.to; + gopts->area_remap =3D gopts->area_dst; /* save for later unmap */ + gopts->area_dst =3D (char *)(unsigned long)msg.arg.remap.to; break; } } @@ -568,13 +590,14 @@ void *uffd_poll_thread(void *arg) return NULL; } =20 -static void retry_copy_page(int ufd, struct uffdio_copy *uffdio_copy, - unsigned long offset) +static void retry_copy_page(uffd_global_test_opts_t *gopts, struct uffdio_= copy *uffdio_copy, + unsigned long offset) { - uffd_test_ops->alias_mapping(&uffdio_copy->dst, - uffdio_copy->len, - offset); - if (ioctl(ufd, UFFDIO_COPY, uffdio_copy)) { + uffd_test_ops->alias_mapping(gopts, + &uffdio_copy->dst, + uffdio_copy->len, + offset); + if (ioctl(gopts->uffd, UFFDIO_COPY, uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy->copy !=3D -EEXIST) err("UFFDIO_COPY retry error: %"PRId64, @@ -597,60 +620,60 @@ static void wake_range(int ufd, unsigned long addr, u= nsigned long len) addr), exit(1); } =20 -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp) +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp) { struct uffdio_copy uffdio_copy; =20 - if (offset >=3D nr_pages * page_size) + if (offset >=3D gopts->nr_pages * gopts->page_size) err("unexpected offset %lu\n", offset); - uffdio_copy.dst =3D (unsigned long) area_dst + offset; - uffdio_copy.src =3D (unsigned long) area_src + offset; - uffdio_copy.len =3D page_size; + uffdio_copy.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_copy.src =3D (unsigned long) gopts->area_src + offset; + uffdio_copy.len =3D gopts->page_size; if (wp) uffdio_copy.mode =3D UFFDIO_COPY_MODE_WP; else uffdio_copy.mode =3D 0; uffdio_copy.copy =3D 0; - if (ioctl(ufd, UFFDIO_COPY, &uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, &uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy.copy !=3D -EEXIST) err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); - wake_range(ufd, uffdio_copy.dst, page_size); - } else if (uffdio_copy.copy !=3D page_size) { + wake_range(gopts->uffd, uffdio_copy.dst, gopts->page_size); + } else if (uffdio_copy.copy !=3D gopts->page_size) { err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); } else { - if (test_uffdio_copy_eexist && retry) { - test_uffdio_copy_eexist =3D false; - retry_copy_page(ufd, &uffdio_copy, offset); + if (gopts->test_uffdio_copy_eexist && retry) { + gopts->test_uffdio_copy_eexist =3D false; + retry_copy_page(gopts, &uffdio_copy, offset); } return 1; } return 0; } =20 -int copy_page(int ufd, unsigned long offset, bool wp) +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p) { - return __copy_page(ufd, offset, false, wp); + return __copy_page(gopts, offset, false, wp); } =20 -int move_page(int ufd, unsigned long offset, unsigned long len) +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len) { struct uffdio_move uffdio_move; =20 - if (offset + len > nr_pages * page_size) + if (offset + len > gopts->nr_pages * gopts->page_size) err("unexpected offset %lu and length %lu\n", offset, len); - uffdio_move.dst =3D (unsigned long) area_dst + offset; - uffdio_move.src =3D (unsigned long) area_src + offset; + uffdio_move.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_move.src =3D (unsigned long) gopts->area_src + offset; uffdio_move.len =3D len; uffdio_move.mode =3D UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES; uffdio_move.move =3D 0; - if (ioctl(ufd, UFFDIO_MOVE, &uffdio_move)) { + if (ioctl(gopts->uffd, UFFDIO_MOVE, &uffdio_move)) { /* real retval in uffdio_move.move */ if (uffdio_move.move !=3D -EEXIST) err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); - wake_range(ufd, uffdio_move.dst, len); + wake_range(gopts->uffd, uffdio_move.dst, len); } else if (uffdio_move.move !=3D len) { err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); } else diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selft= ests/mm/uffd-common.h index 7700cbfa3975..a3debf116d58 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -56,20 +56,17 @@ =20 #define err(fmt, ...) errexit(1, fmt, ##__VA_ARGS__) =20 -/* pthread_mutex_t starts at page offset 0 */ -#define area_mutex(___area, ___nr) \ - ((pthread_mutex_t *) ((___area) + (___nr)*page_size)) -/* - * count is placed in the page after pthread_mutex_t naturally aligned - * to avoid non alignment faults on non-x86 archs. - */ -#define area_count(___area, ___nr) \ - ((volatile unsigned long long *) ((unsigned long) \ - ((___area) + (___nr)*page_size + \ - sizeof(pthread_mutex_t) + \ - sizeof(unsigned long long) - 1) & \ - ~(unsigned long)(sizeof(unsigned long long) \ - - 1))) +struct uffd_global_test_opts { + unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; + char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; + int uffd, uffd_flags, finished, *pipefd, test_type; + bool map_shared; + bool test_uffdio_wp; + unsigned long long *count_verify; + volatile bool test_uffdio_copy_eexist; + atomic_bool ready_for_fork; +}; +typedef struct uffd_global_test_opts uffd_global_test_opts_t; =20 /* Userfaultfd test statistics */ struct uffd_args { @@ -79,50 +76,55 @@ struct uffd_args { unsigned long missing_faults; unsigned long wp_faults; unsigned long minor_faults; + struct uffd_global_test_opts *gopts; =20 /* A custom fault handler; defaults to uffd_handle_page_fault. */ - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args); + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, + struct uffd_args *args); }; =20 struct uffd_test_ops { - int (*allocate_area)(void **alloc_area, bool is_src); - void (*release_pages)(char *rel_area); - void (*alias_mapping)(__u64 *start, size_t len, unsigned long offset); - void (*check_pmd_mapping)(void *p, int expect_nr_hpages); + int (*allocate_area)(uffd_global_test_opts_t *gopts, void **alloc_area, b= ool is_src); + void (*release_pages)(uffd_global_test_opts_t *gopts, char *rel_area); + void (*alias_mapping)(uffd_global_test_opts_t *gopts, + __u64 *start, + size_t len, + unsigned long offset); + void (*check_pmd_mapping)(uffd_global_test_opts_t *gopts, void *p, int ex= pect_nr_hpages); }; typedef struct uffd_test_ops uffd_test_ops_t; =20 struct uffd_test_case_ops { - int (*pre_alloc)(const char **errmsg); - int (*post_alloc)(const char **errmsg); + int (*pre_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); + int (*post_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); }; typedef struct uffd_test_case_ops uffd_test_case_ops_t; =20 -extern unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -extern char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_= remap; -extern int uffd, uffd_flags, finished, *pipefd, test_type; -extern bool map_shared; -extern bool test_uffdio_wp; -extern unsigned long long *count_verify; -extern volatile bool test_uffdio_copy_eexist; -extern atomic_bool ready_for_fork; - +extern uffd_global_test_opts_t *uffd_gtest_opts; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; extern uffd_test_ops_t hugetlb_uffd_test_ops; extern uffd_test_ops_t *uffd_test_ops; extern uffd_test_case_ops_t *uffd_test_case_ops; =20 +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts); +volatile unsigned long long *area_count(char *area, + unsigned long nr, + uffd_global_test_opts_t *gopts); + void uffd_stats_report(struct uffd_args *args, int n_cpus); -int uffd_test_ctx_init(uint64_t features, const char **errmsg); -void uffd_test_ctx_clear(void); -int userfaultfd_open(uint64_t *features); -int uffd_read_msg(int ufd, struct uffd_msg *msg); +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg); +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts); +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features); +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg); void wp_range(int ufd, __u64 start, __u64 len, bool wp); -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); -int copy_page(int ufd, unsigned long offset, bool wp); -int move_page(int ufd, unsigned long offset, unsigned long len); +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args); +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp); +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p); +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len); void *uffd_poll_thread(void *arg); =20 int uffd_open_dev(unsigned int flags); diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selft= ests/mm/uffd-stress.c index 40af7f67c407..e7aac958d57d 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -76,54 +76,58 @@ static void usage(void) exit(1); } =20 -static void uffd_stats_reset(struct uffd_args *args, unsigned long n_cpus) +static void uffd_stats_reset(uffd_global_test_opts_t *gopts, struct uffd_a= rgs *args, + unsigned long n_cpus) { int i; =20 for (i =3D 0; i < n_cpus; i++) { args[i].cpu =3D i; - args[i].apply_wp =3D test_uffdio_wp; + args[i].apply_wp =3D gopts->test_uffdio_wp; args[i].missing_faults =3D 0; args[i].wp_faults =3D 0; args[i].minor_faults =3D 0; + args[i].gopts =3D gopts; } } =20 static void *locking_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr; unsigned long long count; =20 if (!(bounces & BOUNCE_RANDOM)) { page_nr =3D -bounces; if (!(bounces & BOUNCE_RACINGFAULTS)) - page_nr +=3D cpu * nr_pages_per_cpu; + page_nr +=3D cpu * gopts->nr_pages_per_cpu; } =20 - while (!finished) { + while (!gopts->finished) { if (bounces & BOUNCE_RANDOM) { if (getrandom(&page_nr, sizeof(page_nr), 0) !=3D sizeof(page_nr)) err("getrandom failed"); } else page_nr +=3D 1; - page_nr %=3D nr_pages; - pthread_mutex_lock(area_mutex(area_dst, page_nr)); - count =3D *area_count(area_dst, page_nr); - if (count !=3D count_verify[page_nr]) + page_nr %=3D gopts->nr_pages; + pthread_mutex_lock(area_mutex(gopts->area_dst, page_nr, gopts)); + count =3D *area_count(gopts->area_dst, page_nr, gopts); + if (count !=3D gopts->count_verify[page_nr]) err("page_nr %lu memory corruption %llu %llu", - page_nr, count, count_verify[page_nr]); + page_nr, count, gopts->count_verify[page_nr]); count++; - *area_count(area_dst, page_nr) =3D count_verify[page_nr] =3D count; - pthread_mutex_unlock(area_mutex(area_dst, page_nr)); + *area_count(gopts->area_dst, page_nr, gopts) =3D gopts->count_verify[pag= e_nr] =3D count; + pthread_mutex_unlock(area_mutex(gopts->area_dst, page_nr, gopts)); } =20 return NULL; } =20 -static int copy_page_retry(int ufd, unsigned long offset) +static int copy_page_retry(uffd_global_test_opts_t *gopts, unsigned long o= ffset) { - return __copy_page(ufd, offset, true, test_uffdio_wp); + return __copy_page(gopts, offset, true, gopts->test_uffdio_wp); } =20 pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INITIALIZER; @@ -131,15 +135,16 @@ pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INI= TIALIZER; static void *uffd_read_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; struct uffd_msg msg; =20 pthread_mutex_unlock(&uffd_read_mutex); /* from here cancellation is ok */ =20 for (;;) { - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; - uffd_handle_page_fault(&msg, args); + uffd_handle_page_fault(gopts, &msg, args); } =20 return NULL; @@ -147,32 +152,34 @@ static void *uffd_read_thread(void *arg) =20 static void *background_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr, start_nr, mid_nr, end_nr; =20 - start_nr =3D cpu * nr_pages_per_cpu; - end_nr =3D (cpu+1) * nr_pages_per_cpu; + start_nr =3D cpu * gopts->nr_pages_per_cpu; + end_nr =3D (cpu+1) * gopts->nr_pages_per_cpu; mid_nr =3D (start_nr + end_nr) / 2; =20 /* Copy the first half of the pages */ for (page_nr =3D start_nr; page_nr < mid_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 /* * If we need to test uffd-wp, set it up now. Then we'll have * at least the first half of the pages mapped already which * can be write-protected for testing */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst + start_nr * page_size, - nr_pages_per_cpu * page_size, true); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst + start_nr * gopts-= >page_size, + gopts->nr_pages_per_cpu * gopts->page_size, true); =20 /* * Continue the 2nd half of the page copying, handling write * protection faults if any */ for (page_nr =3D mid_nr; page_nr < end_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 return NULL; } @@ -180,17 +187,23 @@ static void *background_thread(void *arg) static int stress(struct uffd_args *args) { unsigned long cpu; - pthread_t locking_threads[nr_parallel]; - pthread_t uffd_threads[nr_parallel]; - pthread_t background_threads[nr_parallel]; + uffd_global_test_opts_t *gopts =3D args->gopts; + pthread_t locking_threads[gopts->nr_parallel]; + pthread_t uffd_threads[gopts->nr_parallel]; + pthread_t background_threads[gopts->nr_parallel]; =20 - finished =3D 0; - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + gopts->finished =3D 0; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { if (pthread_create(&locking_threads[cpu], &attr, - locking_thread, (void *)cpu)) + locking_thread, (void *)&args[cpu])) return 1; if (bounces & BOUNCE_POLL) { - if (pthread_create(&uffd_threads[cpu], &attr, uffd_poll_thread, &args[c= pu])) + if (pthread_create( + &uffd_threads[cpu], + &attr, + uffd_poll_thread, + (void *) &args[cpu]) + ) err("uffd_poll_thread create"); } else { if (pthread_create(&uffd_threads[cpu], &attr, @@ -200,10 +213,10 @@ static int stress(struct uffd_args *args) pthread_mutex_lock(&uffd_read_mutex); } if (pthread_create(&background_threads[cpu], &attr, - background_thread, (void *)cpu)) + background_thread, (void *)&args[cpu])) return 1; } - for (cpu =3D 0; cpu < nr_parallel; cpu++) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(background_threads[cpu], NULL)) return 1; =20 @@ -216,17 +229,17 @@ static int stress(struct uffd_args *args) * UFFDIO_COPY without writing zero pages into area_dst * because the background threads already completed). */ - uffd_test_ops->release_pages(area_src); + uffd_test_ops->release_pages(gopts, gopts->area_src); =20 - finished =3D 1; - for (cpu =3D 0; cpu < nr_parallel; cpu++) + gopts->finished =3D 1; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(locking_threads[cpu], NULL)) return 1; =20 - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { char c; if (bounces & BOUNCE_POLL) { - if (write(pipefd[cpu*2+1], &c, 1) !=3D 1) + if (write(gopts->pipefd[cpu*2+1], &c, 1) !=3D 1) err("pipefd write error"); if (pthread_join(uffd_threads[cpu], (void *)&args[cpu])) @@ -242,26 +255,26 @@ static int stress(struct uffd_args *args) return 0; } =20 -static int userfaultfd_stress(void) +static int userfaultfd_stress(uffd_global_test_opts_t *gopts) { void *area; unsigned long nr; - struct uffd_args args[nr_parallel]; - uint64_t mem_size =3D nr_pages * page_size; + struct uffd_args args[gopts->nr_parallel]; + uint64_t mem_size =3D gopts->nr_pages * gopts->page_size; int flags =3D 0; =20 - memset(args, 0, sizeof(struct uffd_args) * nr_parallel); + memset(args, 0, sizeof(struct uffd_args) * gopts->nr_parallel); =20 - if (features & UFFD_FEATURE_WP_UNPOPULATED && test_type =3D=3D TEST_ANON) + if (features & UFFD_FEATURE_WP_UNPOPULATED && gopts->test_type =3D=3D TES= T_ANON) flags =3D UFFD_FEATURE_WP_UNPOPULATED; =20 - if (uffd_test_ctx_init(flags, NULL)) + if (uffd_test_ctx_init(gopts, flags, NULL)) err("context init failed"); =20 - if (posix_memalign(&area, page_size, page_size)) + if (posix_memalign(&area, gopts->page_size, gopts->page_size)) err("out of memory"); zeropage =3D area; - bzero(zeropage, page_size); + bzero(zeropage, gopts->page_size); =20 pthread_mutex_lock(&uffd_read_mutex); =20 @@ -284,18 +297,18 @@ static int userfaultfd_stress(void) fflush(stdout); =20 if (bounces & BOUNCE_POLL) - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); else - fcntl(uffd, F_SETFL, uffd_flags & ~O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags & ~O_NONBLOCK); =20 /* register */ - if (uffd_register(uffd, area_dst, mem_size, - true, test_uffdio_wp, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure"); =20 - if (area_dst_alias) { - if (uffd_register(uffd, area_dst_alias, mem_size, - true, test_uffdio_wp, false)) + if (gopts->area_dst_alias) { + if (uffd_register(gopts->uffd, gopts->area_dst_alias, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure alias"); } =20 @@ -323,87 +336,88 @@ static int userfaultfd_stress(void) * MADV_DONTNEED only after the UFFDIO_REGISTER, so it's * required to MADV_DONTNEED here. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - uffd_stats_reset(args, nr_parallel); + uffd_stats_reset(gopts, args, gopts->nr_parallel); =20 /* bounce pass */ if (stress(args)) { - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); return 1; } =20 /* Clear all the write protections if there is any */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst, - nr_pages * page_size, false); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst, + gopts->nr_pages * gopts->page_size, false); =20 /* unregister */ - if (uffd_unregister(uffd, area_dst, mem_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, mem_size)) err("unregister failure"); - if (area_dst_alias) { - if (uffd_unregister(uffd, area_dst_alias, mem_size)) + if (gopts->area_dst_alias) { + if (uffd_unregister(gopts->uffd, gopts->area_dst_alias, mem_size)) err("unregister failure alias"); } =20 /* verification */ if (bounces & BOUNCE_VERIFY) - for (nr =3D 0; nr < nr_pages; nr++) - if (*area_count(area_dst, nr) !=3D count_verify[nr]) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + if (*area_count(gopts->area_dst, nr, gopts) !=3D + gopts->count_verify[nr]) err("error area_count %llu %llu %lu\n", - *area_count(area_src, nr), - count_verify[nr], nr); + *area_count(gopts->area_src, nr, gopts), + gopts->count_verify[nr], nr); =20 /* prepare next bounce */ - swap(area_src, area_dst); + swap(gopts->area_src, gopts->area_dst); =20 - swap(area_src_alias, area_dst_alias); + swap(gopts->area_src_alias, gopts->area_dst_alias); =20 - uffd_stats_report(args, nr_parallel); + uffd_stats_report(args, gopts->nr_parallel); } - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); =20 return 0; } =20 -static void set_test_type(const char *type) +static void set_test_type(uffd_global_test_opts_t *gopts, const char *type) { if (!strcmp(type, "anon")) { - test_type =3D TEST_ANON; + gopts->test_type =3D TEST_ANON; uffd_test_ops =3D &anon_uffd_test_ops; } else if (!strcmp(type, "hugetlb")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; - map_shared =3D true; + gopts->map_shared =3D true; } else if (!strcmp(type, "hugetlb-private")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; } else if (!strcmp(type, "shmem")) { - map_shared =3D true; - test_type =3D TEST_SHMEM; + gopts->map_shared =3D true; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } else if (!strcmp(type, "shmem-private")) { - test_type =3D TEST_SHMEM; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } } =20 -static void parse_test_type_arg(const char *raw_type) +static void parse_test_type_arg(uffd_global_test_opts_t *gopts, const char= *raw_type) { - set_test_type(raw_type); + set_test_type(gopts, raw_type); =20 - if (!test_type) + if (!gopts->test_type) err("failed to parse test type argument: '%s'", raw_type); =20 - if (test_type =3D=3D TEST_HUGETLB) - page_size =3D default_huge_page_size(); + if (gopts->test_type =3D=3D TEST_HUGETLB) + gopts->page_size =3D default_huge_page_size(); else - page_size =3D sysconf(_SC_PAGE_SIZE); + gopts->page_size =3D sysconf(_SC_PAGE_SIZE); =20 - if (!page_size) + if (!gopts->page_size) err("Unable to determine page size"); - if ((unsigned long) area_count(NULL, 0) + sizeof(unsigned long long) * 2 - > page_size) + if ((unsigned long) area_count(NULL, 0, gopts) + sizeof(unsigned long lon= g) * 2 + > gopts->page_size) err("Impossible to run this test"); =20 /* @@ -415,21 +429,22 @@ static void parse_test_type_arg(const char *raw_type) if (uffd_get_features(&features) && errno =3D=3D ENOENT) ksft_exit_skip("failed to get available features (%d)\n", errno); =20 - test_uffdio_wp =3D test_uffdio_wp && + gopts->test_uffdio_wp =3D gopts->test_uffdio_wp && (features & UFFD_FEATURE_PAGEFAULT_FLAG_WP); =20 - if (test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGETLBFS_SH= MEM)) - test_uffdio_wp =3D false; + if (gopts->test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGET= LBFS_SHMEM)) + gopts->test_uffdio_wp =3D false; =20 - close(uffd); - uffd =3D -1; + close(gopts->uffd); + gopts->uffd =3D -1; } =20 static void sigalrm(int sig) { if (sig !=3D SIGALRM) abort(); - test_uffdio_copy_eexist =3D true; + // TODO: Set this without access to global vars + // gopts->test_uffdio_copy_eexist =3D true; alarm(ALARM_INTERVAL_SECS); } =20 @@ -438,6 +453,9 @@ int main(int argc, char **argv) unsigned long nr_cpus; size_t bytes; =20 + uffd_global_test_opts_t *gopts =3D + (uffd_global_test_opts_t *) malloc(sizeof(uffd_global_test_opts_t)); + if (argc < 4) usage(); =20 @@ -445,11 +463,11 @@ int main(int argc, char **argv) err("failed to arm SIGALRM"); alarm(ALARM_INTERVAL_SECS); =20 - parse_test_type_arg(argv[1]); + parse_test_type_arg(gopts, argv[1]); bytes =3D atol(argv[2]) * 1024 * 1024; =20 - if (test_type =3D=3D TEST_HUGETLB && - get_free_hugepages() < bytes / page_size) { + if (gopts->test_type =3D=3D TEST_HUGETLB && + get_free_hugepages() < bytes / gopts->page_size) { printf("skip: Skipping userfaultfd... not enough hugepages\n"); return KSFT_SKIP; } @@ -459,15 +477,15 @@ int main(int argc, char **argv) /* Don't let calculation below go to zero. */ ksft_print_msg("_SC_NPROCESSORS_ONLN (%lu) too large, capping nr_threads= to 32\n", nr_cpus); - nr_parallel =3D 32; + gopts->nr_parallel =3D 32; } else { - nr_parallel =3D nr_cpus; + gopts->nr_parallel =3D nr_cpus; } =20 - nr_pages_per_cpu =3D bytes / page_size / nr_parallel; - if (!nr_pages_per_cpu) { + gopts->nr_pages_per_cpu =3D bytes / gopts->page_size / gopts->nr_parallel; + if (!gopts->nr_pages_per_cpu) { _err("pages_per_cpu =3D 0, cannot test (%lu / %lu / %lu)", - bytes, page_size, nr_parallel); + bytes, gopts->page_size, gopts->nr_parallel); usage(); } =20 @@ -476,11 +494,11 @@ int main(int argc, char **argv) _err("invalid bounces"); usage(); } - nr_pages =3D nr_pages_per_cpu * nr_parallel; + gopts->nr_pages =3D gopts->nr_pages_per_cpu * gopts->nr_parallel; =20 printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n", - nr_pages, nr_pages_per_cpu); - return userfaultfd_stress(); + gopts->nr_pages, gopts->nr_pages_per_cpu); + return userfaultfd_stress(gopts); } =20 #else /* __NR_userfaultfd */ diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/s= elftests/mm/uffd-unit-tests.c index c73fd5d455c8..d0d5c9bc9e0e 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -76,7 +76,7 @@ struct uffd_test_args { typedef struct uffd_test_args uffd_test_args_t; =20 /* Returns: UFFD_TEST_* */ -typedef void (*uffd_test_fn)(uffd_test_args_t *); +typedef void (*uffd_test_fn)(uffd_global_test_opts_t *, uffd_test_args_t *= ); =20 typedef struct { const char *name; @@ -181,33 +181,6 @@ static int test_uffd_api(bool use_dev) return 1; } =20 -/* - * This function initializes the global variables. TODO: remove global - * vars and then remove this. - */ -static int -uffd_setup_environment(uffd_test_args_t *args, uffd_test_case_t *test, - mem_type_t *mem_type, const char **errmsg) -{ - map_shared =3D mem_type->shared; - uffd_test_ops =3D mem_type->mem_ops; - uffd_test_case_ops =3D test->test_case_ops; - - if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) - page_size =3D default_huge_page_size(); - else - page_size =3D psize(); - - /* Ensure we have at least 2 pages */ - nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, page_size * 2) / page_size; - /* TODO: remove this global var.. it's so ugly */ - nr_parallel =3D 1; - - /* Initialize test arguments */ - args->mem_type =3D mem_type; - - return uffd_test_ctx_init(test->uffd_feature_required, errmsg); -} =20 static bool uffd_feature_supported(uffd_test_case_t *test) { @@ -237,7 +210,8 @@ static int pagemap_open(void) } while (0) =20 typedef struct { - int parent_uffd, child_uffd; + uffd_global_test_opts_t *gopts; + int child_uffd; } fork_event_args; =20 static void *fork_event_consumer(void *data) @@ -245,10 +219,10 @@ static void *fork_event_consumer(void *data) fork_event_args *args =3D data; struct uffd_msg msg =3D { 0 }; =20 - ready_for_fork =3D true; + args->gopts->ready_for_fork =3D true; =20 /* Read until a full msg received */ - while (uffd_read_msg(args->parent_uffd, &msg)); + while (uffd_read_msg(args->gopts, &msg)); =20 if (msg.event !=3D UFFD_EVENT_FORK) err("wrong message: %u\n", msg.event); @@ -304,9 +278,9 @@ static void unpin_pages(pin_args *args) args->pinned =3D false; } =20 -static int pagemap_test_fork(int uffd, bool with_event, bool test_pin) +static int pagemap_test_fork(uffd_global_test_opts_t *gopts, bool with_eve= nt, bool test_pin) { - fork_event_args args =3D { .parent_uffd =3D uffd, .child_uffd =3D -1 }; + fork_event_args args =3D { .gopts =3D gopts, .child_uffd =3D -1 }; pthread_t thread; pid_t child; uint64_t value; @@ -314,10 +288,10 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 /* Prepare a thread to resolve EVENT_FORK */ if (with_event) { - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; if (pthread_create(&thread, NULL, fork_event_consumer, &args)) err("pthread_create()"); - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ } =20 @@ -328,14 +302,14 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 fd =3D pagemap_open(); =20 - if (test_pin && pin_pages(&args, area_dst, page_size)) + if (test_pin && pin_pages(&args, gopts->area_dst, gopts->page_size)) /* * Normally when reach here we have pinned in * previous tests, so shouldn't fail anymore */ err("pin page failed in child"); =20 - value =3D pagemap_get_entry(fd, area_dst); + value =3D pagemap_get_entry(fd, gopts->area_dst); /* * After fork(), we should handle uffd-wp bit differently: * @@ -361,70 +335,70 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) return result; } =20 -static void uffd_wp_unpopulated_test(uffd_test_args_t *args) +static void uffd_wp_unpopulated_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args) { uint64_t value; int pagemap_fd; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Test applying pte marker to anon unpopulated */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); =20 /* Test unprotect on anon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test zap on anon marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - if (madvise(area_dst, page_size, MADV_DONTNEED)) + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test fault in after marker removed */ - *area_dst =3D 1; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 /* Test read-zero-page upon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - *(volatile char *)area_dst; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + *(volatile char *)gopts->area_dst; /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 uffd_test_pass(); } =20 -static void uffd_wp_fork_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_test_common(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args, bool with_event) { int pagemap_fd; uint64_t value; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in present pte", with_event ? "missing" : "stall"); goto out; @@ -442,79 +416,79 @@ static void uffd_wp_fork_test_common(uffd_test_args_t= *args, * to expose pte markers. */ if (args->mem_type->shared) { - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("MADV_DONTNEED"); } else { /* * NOTE: ignore retval because private-hugetlb doesn't yet * support swapping, so it could fail. */ - madvise(area_dst, page_size, MADV_PAGEOUT); + madvise(gopts->area_dst, gopts->page_size, MADV_PAGEOUT); } =20 /* Uffd-wp should persist even swapped out */ - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in zapped pte", with_event ? "missing" : "stall"); goto out; } =20 /* Unprotect; this tests swap pte modifications */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Fault in the page from disk */ - *area_dst =3D 2; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 2; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, nr_pages * page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts= ->page_size)) err("unregister failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_test(uffd_test_args_t *args) +static void uffd_wp_fork_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_wp_fork_test_common(args, false); + uffd_wp_fork_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_with_event_test(uffd_global_test_opts_t *gopts, u= ffd_test_args_t *args) { - uffd_wp_fork_test_common(args, true); + uffd_wp_fork_test_common(gopts, args, true); } =20 -static void uffd_wp_fork_pin_test_common(uffd_test_args_t *args, - bool with_event) +static void uffd_wp_fork_pin_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, bool with_event) { int pagemap_fd; pin_args pin_args =3D {}; =20 - if (uffd_register(uffd, area_dst, page_size, false, true, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->page_size, false, = true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); =20 /* * 1. First pin, then fork(). This tests fork() special path when * doing early CoW if the page is private. */ - if (pin_pages(&pin_args, area_dst, page_size)) { + if (pin_pages(&pin_args, gopts->area_dst, gopts->page_size)) { uffd_test_skip("Possibly CONFIG_GUP_TEST missing " "or unprivileged"); close(pagemap_fd); - uffd_unregister(uffd, area_dst, page_size); + uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size); return; } =20 - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in early CoW of fork()", with_event ? "missing" : "stall"); unpin_pages(&pin_args); @@ -527,49 +501,50 @@ static void uffd_wp_fork_pin_test_common(uffd_test_ar= gs_t *args, * 2. First fork(), then pin (in the child, where test_pin=3D=3Dtrue). * This tests COR, aka, page unsharing on private memories. */ - if (pagemap_test_fork(uffd, with_event, true)) { + if (pagemap_test_fork(gopts, with_event, true)) { uffd_test_fail("Detected %s uffd-wp bit when RO pin", with_event ? "missing" : "stall"); goto out; } uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("register failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_pin_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_test(uffd_global_test_opts_t *gopts, uffd_tes= t_args_t *args) { - uffd_wp_fork_pin_test_common(args, false); + uffd_wp_fork_pin_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_pin_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_with_event_test(uffd_global_test_opts_t *gopt= s, uffd_test_args_t *args) { - uffd_wp_fork_pin_test_common(args, true); + uffd_wp_fork_pin_test_common(gopts, args, true); } =20 -static void check_memory_contents(char *p) +static void check_memory_contents(uffd_global_test_opts_t *gopts, char *p) { unsigned long i, j; uint8_t expected_byte; =20 - for (i =3D 0; i < nr_pages; ++i) { + for (i =3D 0; i < gopts->nr_pages; ++i) { expected_byte =3D ~((uint8_t)(i % ((uint8_t)-1))); - for (j =3D 0; j < page_size; j++) { - uint8_t v =3D *(uint8_t *)(p + (i * page_size) + j); + for (j =3D 0; j < gopts->page_size; j++) { + uint8_t v =3D *(uint8_t *)(p + (i * gopts->page_size) + j); if (v !=3D expected_byte) err("unexpected page contents"); } } } =20 -static void uffd_minor_test_common(bool test_collapse, bool test_wp) +static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool te= st_collapse, bool test_wp) { unsigned long p; pthread_t uffd_mon; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 /* * NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing @@ -577,7 +552,7 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) */ assert(!(test_collapse && test_wp)); =20 - if (uffd_register(uffd, area_dst_alias, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst_alias, gopts->nr_pages * g= opts->page_size, /* NOTE! MADV_COLLAPSE may not work with uffd-wp */ false, test_wp, true)) err("register failure"); @@ -586,9 +561,9 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) * After registering with UFFD, populate the non-UFFD-registered side of * the shared mapping. This should *not* trigger any UFFD minor faults. */ - for (p =3D 0; p < nr_pages; ++p) - memset(area_dst + (p * page_size), p % ((uint8_t)-1), - page_size); + for (p =3D 0; p < gopts->nr_pages; ++p) + memset(gopts->area_dst + (p * gopts->page_size), p % ((uint8_t)-1), + gopts->page_size); =20 args.apply_wp =3D test_wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -600,50 +575,51 @@ static void uffd_minor_test_common(bool test_collapse= , bool test_wp) * fault. uffd_poll_thread will resolve the fault by bit-flipping the * page's contents, and then issuing a CONTINUE ioctl. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); =20 if (test_collapse) { - if (madvise(area_dst_alias, nr_pages * page_size, + if (madvise(gopts->area_dst_alias, gopts->nr_pages * gopts->page_size, MADV_COLLAPSE)) { /* It's fine to fail for this one... */ uffd_test_skip("MADV_COLLAPSE failed"); return; } =20 - uffd_test_ops->check_pmd_mapping(area_dst, - nr_pages * page_size / + uffd_test_ops->check_pmd_mapping(gopts, + gopts->area_dst, + gopts->nr_pages * gopts->page_size / read_pmd_pagesize()); /* * This won't cause uffd-fault - it purely just makes sure there * was no corruption. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); } =20 - if (args.missing_faults !=3D 0 || args.minor_faults !=3D nr_pages) + if (args.missing_faults !=3D 0 || args.minor_faults !=3D gopts->nr_pages) uffd_test_fail("stats check error"); else uffd_test_pass(); } =20 -void uffd_minor_test(uffd_test_args_t *args) +void uffd_minor_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *arg= s) { - uffd_minor_test_common(false, false); + uffd_minor_test_common(gopts, false, false); } =20 -void uffd_minor_wp_test(uffd_test_args_t *args) +void uffd_minor_wp_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *= args) { - uffd_minor_test_common(false, true); + uffd_minor_test_common(gopts, false, true); } =20 -void uffd_minor_collapse_test(uffd_test_args_t *args) +void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_minor_test_common(true, false); + uffd_minor_test_common(gopts, true, false); } =20 static sigjmp_buf jbuf, *sigbuf; @@ -678,7 +654,7 @@ static void sighndl(int sig, siginfo_t *siginfo, void *= ptr) * This also tests UFFD_FEATURE_EVENT_FORK event along with the signal * feature. Using monitor thread, verify no userfault events are generated. */ -static int faulting_process(int signal_test, bool wp) +static int faulting_process(uffd_global_test_opts_t *gopts, int signal_tes= t, bool wp) { unsigned long nr, i; unsigned long long count; @@ -687,7 +663,7 @@ static int faulting_process(int signal_test, bool wp) struct sigaction act; volatile unsigned long signalled =3D 0; =20 - split_nr_pages =3D (nr_pages + 1) / 2; + split_nr_pages =3D (gopts->nr_pages + 1) / 2; =20 if (signal_test) { sigbuf =3D &jbuf; @@ -701,7 +677,7 @@ static int faulting_process(int signal_test, bool wp) =20 for (nr =3D 0; nr < split_nr_pages; nr++) { volatile int steps =3D 1; - unsigned long offset =3D nr * page_size; + unsigned long offset =3D nr * gopts->page_size; =20 if (signal_test) { if (sigsetjmp(*sigbuf, 1) !=3D 0) { @@ -713,15 +689,15 @@ static int faulting_process(int signal_test, bool wp) if (steps =3D=3D 1) { /* This is a MISSING request */ steps++; - if (copy_page(uffd, offset, wp)) + if (copy_page(gopts, offset, wp)) signalled++; } else { /* This is a WP request */ assert(steps =3D=3D 2); - wp_range(uffd, - (__u64)area_dst + + wp_range(gopts->uffd, + (__u64)gopts->area_dst + offset, - page_size, false); + gopts->page_size, false); } } else { signalled++; @@ -730,51 +706,53 @@ static int faulting_process(int signal_test, bool wp) } } =20 - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 if (signal_test) return signalled !=3D split_nr_pages; =20 - area_dst =3D mremap(area_dst, nr_pages * page_size, nr_pages * page_size, - MREMAP_MAYMOVE | MREMAP_FIXED, area_src); - if (area_dst =3D=3D MAP_FAILED) + gopts->area_dst =3D mremap(gopts->area_dst, gopts->nr_pages * gopts->page= _size, + gopts->nr_pages * gopts->page_size, + MREMAP_MAYMOVE | MREMAP_FIXED, + gopts->area_src); + if (gopts->area_dst =3D=3D MAP_FAILED) err("mremap"); /* Reset area_src since we just clobbered it */ - area_src =3D NULL; + gopts->area_src =3D NULL; =20 - for (; nr < nr_pages; nr++) { - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) { + for (; nr < gopts->nr_pages; nr++) { + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) { err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); } /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - for (nr =3D 0; nr < nr_pages; nr++) - for (i =3D 0; i < page_size; i++) - if (*(area_dst + nr * page_size + i) !=3D 0) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + for (i =3D 0; i < gopts->page_size; i++) + if (*(gopts->area_dst + nr * gopts->page_size + i) !=3D 0) err("page %lu offset %lu is not zero", nr, i); =20 return 0; } =20 -static void uffd_sigbus_test_common(bool wp) +static void uffd_sigbus_test_common(uffd_global_test_opts_t *gopts, bool w= p) { unsigned long userfaults; pthread_t uffd_mon; @@ -782,25 +760,26 @@ static void uffd_sigbus_test_common(bool wp) int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 - if (faulting_process(1, wp)) + if (faulting_process(gopts, 1, wp)) err("faulting process failed"); =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 args.apply_wp =3D wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -808,12 +787,12 @@ static void uffd_sigbus_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(2, wp)); + exit(faulting_process(gopts, 2, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, (void **)&userfaults)) err("pthread_join()"); @@ -824,28 +803,29 @@ static void uffd_sigbus_test_common(bool wp) uffd_test_pass(); } =20 -static void uffd_sigbus_test(uffd_test_args_t *args) +static void uffd_sigbus_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_sigbus_test_common(false); + uffd_sigbus_test_common(gopts, false); } =20 -static void uffd_sigbus_wp_test(uffd_test_args_t *args) +static void uffd_sigbus_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_sigbus_test_common(true); + uffd_sigbus_test_common(gopts, true); } =20 -static void uffd_events_test_common(bool wp) +static void uffd_events_test_common(uffd_global_test_opts_t *gopts, bool w= p) { pthread_t uffd_mon; pid_t pid; int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); - if (uffd_register(uffd, area_dst, nr_pages * page_size, + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 @@ -853,7 +833,7 @@ static void uffd_events_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -861,39 +841,39 @@ static void uffd_events_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(0, wp)); + exit(faulting_process(gopts, 0, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (args.missing_faults !=3D nr_pages) + if (args.missing_faults !=3D gopts->nr_pages) uffd_test_fail("Fault counts wrong"); else uffd_test_pass(); } =20 -static void uffd_events_test(uffd_test_args_t *args) +static void uffd_events_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_events_test_common(false); + uffd_events_test_common(gopts, false); } =20 -static void uffd_events_wp_test(uffd_test_args_t *args) +static void uffd_events_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_events_test_common(true); + uffd_events_test_common(gopts, true); } =20 -static void retry_uffdio_zeropage(int ufd, +static void retry_uffdio_zeropage(uffd_global_test_opts_t *gopts, struct uffdio_zeropage *uffdio_zeropage) { - uffd_test_ops->alias_mapping(&uffdio_zeropage->range.start, + uffd_test_ops->alias_mapping(gopts, &uffdio_zeropage->range.start, uffdio_zeropage->range.len, 0); - if (ioctl(ufd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { + if (ioctl(gopts->uffd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { if (uffdio_zeropage->zeropage !=3D -EEXIST) err("UFFDIO_ZEROPAGE error: %"PRId64, (int64_t)uffdio_zeropage->zeropage); @@ -903,16 +883,16 @@ static void retry_uffdio_zeropage(int ufd, } } =20 -static bool do_uffdio_zeropage(int ufd, bool has_zeropage) +static bool do_uffdio_zeropage(uffd_global_test_opts_t *gopts, bool has_ze= ropage) { struct uffdio_zeropage uffdio_zeropage =3D { 0 }; int ret; __s64 res; =20 - uffdio_zeropage.range.start =3D (unsigned long) area_dst; - uffdio_zeropage.range.len =3D page_size; + uffdio_zeropage.range.start =3D (unsigned long) gopts->area_dst; + uffdio_zeropage.range.len =3D gopts->page_size; uffdio_zeropage.mode =3D 0; - ret =3D ioctl(ufd, UFFDIO_ZEROPAGE, &uffdio_zeropage); + ret =3D ioctl(gopts->uffd, UFFDIO_ZEROPAGE, &uffdio_zeropage); res =3D uffdio_zeropage.zeropage; if (ret) { /* real retval in ufdio_zeropage.zeropage */ @@ -921,10 +901,10 @@ static bool do_uffdio_zeropage(int ufd, bool has_zero= page) else if (res !=3D -EINVAL) err("UFFDIO_ZEROPAGE not -EINVAL"); } else if (has_zeropage) { - if (res !=3D page_size) + if (res !=3D gopts->page_size) err("UFFDIO_ZEROPAGE unexpected size"); else - retry_uffdio_zeropage(ufd, &uffdio_zeropage); + retry_uffdio_zeropage(gopts, &uffdio_zeropage); return true; } else err("UFFDIO_ZEROPAGE succeeded"); @@ -950,25 +930,27 @@ uffd_register_detect_zeropage(int uffd, void *addr, u= int64_t len) } =20 /* exercise UFFDIO_ZEROPAGE */ -static void uffd_zeropage_test(uffd_test_args_t *args) +static void uffd_zeropage_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *args) { bool has_zeropage; int i; =20 - has_zeropage =3D uffd_register_detect_zeropage(uffd, area_dst, page_size); - if (area_dst_alias) + has_zeropage =3D uffd_register_detect_zeropage(gopts->uffd, gopts->area_d= st, gopts->page_size); + if (gopts->area_dst_alias) /* Ignore the retval; we already have it */ - uffd_register_detect_zeropage(uffd, area_dst_alias, page_size); + uffd_register_detect_zeropage(gopts->uffd, gopts->area_dst_alias, gopts-= >page_size); =20 - if (do_uffdio_zeropage(uffd, has_zeropage)) - for (i =3D 0; i < page_size; i++) - if (area_dst[i] !=3D 0) + if (do_uffdio_zeropage(gopts, has_zeropage)) + for (i =3D 0; i < gopts->page_size; i++) + if (gopts->area_dst[i] !=3D 0) err("data non-zero at offset %d\n", i); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); =20 - if (area_dst_alias && uffd_unregister(uffd, area_dst_alias, page_size)) + if (gopts->area_dst_alias + && + uffd_unregister(gopts->uffd, gopts->area_dst_alias, gopts->page_size)) err("unregister"); =20 uffd_test_pass(); @@ -987,26 +969,26 @@ static void uffd_register_poison(int uffd, void *addr= , uint64_t len) err("registered area doesn't support COPY and POISON ioctls"); } =20 -static void do_uffdio_poison(int uffd, unsigned long offset) +static void do_uffdio_poison(uffd_global_test_opts_t *gopts, unsigned long= offset) { struct uffdio_poison uffdio_poison =3D { 0 }; int ret; __s64 res; =20 - uffdio_poison.range.start =3D (unsigned long) area_dst + offset; - uffdio_poison.range.len =3D page_size; + uffdio_poison.range.start =3D (unsigned long) gopts->area_dst + offset; + uffdio_poison.range.len =3D gopts->page_size; uffdio_poison.mode =3D 0; - ret =3D ioctl(uffd, UFFDIO_POISON, &uffdio_poison); + ret =3D ioctl(gopts->uffd, UFFDIO_POISON, &uffdio_poison); res =3D uffdio_poison.updated; =20 if (ret) err("UFFDIO_POISON error: %"PRId64, (int64_t)res); - else if (res !=3D page_size) + else if (res !=3D gopts->page_size) err("UFFDIO_POISON unexpected size: %"PRId64, (int64_t)res); } =20 static void uffd_poison_handle_fault( - struct uffd_msg *msg, struct uffd_args *args) + uffd_global_test_opts_t *gopts, struct uffd_msg *msg, struct uffd_args *a= rgs) { unsigned long offset; =20 @@ -1017,17 +999,17 @@ static void uffd_poison_handle_fault( (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; + offset &=3D ~(gopts->page_size-1); =20 /* Odd pages -> copy zeroed page; even pages -> poison. */ - if (offset & page_size) - copy_page(uffd, offset, false); + if (offset & gopts->page_size) + copy_page(gopts, offset, false); else - do_uffdio_poison(uffd, offset); + do_uffdio_poison(gopts, offset); } =20 -static void uffd_poison_test(uffd_test_args_t *targs) +static void uffd_poison_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *targs) { pthread_t uffd_mon; char c; @@ -1036,10 +1018,12 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) unsigned long nr_sigbus =3D 0; unsigned long nr; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + args.gopts =3D gopts; + + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - uffd_register_poison(uffd, area_dst, nr_pages * page_size); - memset(area_src, 0, nr_pages * page_size); + uffd_register_poison(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopt= s->page_size); + memset(gopts->area_src, 0, gopts->nr_pages * gopts->page_size); =20 args.handle_fault =3D uffd_poison_handle_fault; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -1051,9 +1035,9 @@ static void uffd_poison_test(uffd_test_args_t *targs) if (sigaction(SIGBUS, &act, 0)) err("sigaction"); =20 - for (nr =3D 0; nr < nr_pages; ++nr) { - unsigned long offset =3D nr * page_size; - const char *bytes =3D (const char *) area_dst + offset; + for (nr =3D 0; nr < gopts->nr_pages; ++nr) { + unsigned long offset =3D nr * gopts->page_size; + const char *bytes =3D (const char *) gopts->area_dst + offset; const char *i; =20 if (sigsetjmp(*sigbuf, 1)) { @@ -1066,28 +1050,30 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) continue; } =20 - for (i =3D bytes; i < bytes + page_size; ++i) { + for (i =3D bytes; i < bytes + gopts->page_size; ++i) { if (*i) err("nonzero byte in area_dst (%p) at %p: %u", - area_dst, i, *i); + gopts->area_dst, i, *i); } } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (nr_sigbus !=3D nr_pages / 2) + if (nr_sigbus !=3D gopts->nr_pages / 2) err("expected to receive %lu SIGBUS, actually received %lu", - nr_pages / 2, nr_sigbus); + gopts->nr_pages / 2, nr_sigbus); =20 uffd_test_pass(); } =20 static void -uffd_move_handle_fault_common(struct uffd_msg *msg, struct uffd_args *args, - unsigned long len) +uffd_move_handle_fault_common(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args, + unsigned long len) { unsigned long offset; =20 @@ -1098,28 +1084,31 @@ uffd_move_handle_fault_common(struct uffd_msg *msg,= struct uffd_args *args, (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR | UFFD_PAGEFAULT_= FLAG_WRITE)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; offset &=3D ~(len-1); =20 - if (move_page(uffd, offset, len)) + if (move_page(gopts, offset, len)) args->missing_faults++; } =20 -static void uffd_move_handle_fault(struct uffd_msg *msg, +static void uffd_move_handle_fault(uffd_global_test_opts_t *gopts, struct = uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, page_size); + uffd_move_handle_fault_common(gopts, msg, args, gopts->page_size); } =20 -static void uffd_move_pmd_handle_fault(struct uffd_msg *msg, +static void uffd_move_pmd_handle_fault(uffd_global_test_opts_t *gopts, str= uct uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, read_pmd_pagesize()); + uffd_move_handle_fault_common(gopts, msg, args, read_pmd_pagesize()); } =20 static void -uffd_move_test_common(uffd_test_args_t *targs, unsigned long chunk_size, - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args)) +uffd_move_test_common( + uffd_global_test_opts_t *gopts, uffd_test_args_t *targs, + unsigned long chunk_size, void (*handle_fault)(struct uffd_global_test_o= pts *gopts, + struct uffd_msg *msg, struct uffd_args *args) +) { unsigned long nr; pthread_t uffd_mon; @@ -1131,11 +1120,13 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, unsigned long src_offs =3D 0; unsigned long dst_offs =3D 0; =20 + args.gopts =3D gopts; + /* Prevent source pages from being mapped more than once */ - if (madvise(area_src, nr_pages * page_size, MADV_DONTFORK)) + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DON= TFORK)) err("madvise(MADV_DONTFORK) failure"); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("register failure"); =20 @@ -1143,22 +1134,22 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - step_size =3D chunk_size / page_size; - step_count =3D nr_pages / step_size; + step_size =3D chunk_size / gopts->page_size; + step_count =3D gopts->nr_pages / step_size; =20 - if (chunk_size > page_size) { - char *aligned_src =3D ALIGN_UP(area_src, chunk_size); - char *aligned_dst =3D ALIGN_UP(area_dst, chunk_size); + if (chunk_size > gopts->page_size) { + char *aligned_src =3D ALIGN_UP(gopts->area_src, chunk_size); + char *aligned_dst =3D ALIGN_UP(gopts->area_dst, chunk_size); =20 - if (aligned_src !=3D area_src || aligned_dst !=3D area_dst) { - src_offs =3D (aligned_src - area_src) / page_size; - dst_offs =3D (aligned_dst - area_dst) / page_size; + if (aligned_src !=3D gopts->area_src || aligned_dst !=3D gopts->area_dst= ) { + src_offs =3D (aligned_src - gopts->area_src) / gopts->page_size; + dst_offs =3D (aligned_dst - gopts->area_dst) / gopts->page_size; step_count--; } - orig_area_src =3D area_src; - orig_area_dst =3D area_dst; - area_src =3D aligned_src; - area_dst =3D aligned_dst; + orig_area_src =3D gopts->area_src; + orig_area_dst =3D gopts->area_dst; + gopts->area_src =3D aligned_src; + gopts->area_dst =3D aligned_dst; } =20 /* @@ -1172,34 +1163,34 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, =20 /* Check area_src content */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); - if (count !=3D count_verify[src_offs + nr + i]) + count =3D *area_count(gopts->area_src, nr + i, gopts); + if (count !=3D gopts->count_verify[src_offs + nr + i]) err("nr %lu source memory invalid %llu %llu\n", - nr + i, count, count_verify[src_offs + nr + i]); + nr + i, count, gopts->count_verify[src_offs + nr + i]); } =20 /* Faulting into area_dst should move the page or the huge page */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_dst, nr + i); - if (count !=3D count_verify[dst_offs + nr + i]) + count =3D *area_count(gopts->area_dst, nr + i, gopts); + if (count !=3D gopts->count_verify[dst_offs + nr + i]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[dst_offs + nr + i]); + nr, count, gopts->count_verify[dst_offs + nr + i]); } =20 /* Re-check area_src content which should be empty */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); + count =3D *area_count(gopts->area_src, nr + i, gopts); if (count !=3D 0) err("nr %lu move failed %llu %llu\n", - nr, count, count_verify[src_offs + nr + i]); + nr, count, gopts->count_verify[src_offs + nr + i]); } } - if (chunk_size > page_size) { - area_src =3D orig_area_src; - area_dst =3D orig_area_dst; + if (chunk_size > gopts->page_size) { + gopts->area_src =3D orig_area_src; + gopts->area_dst =3D orig_area_dst; } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); @@ -1210,24 +1201,24 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, uffd_test_pass(); } =20 -static void uffd_move_test(uffd_test_args_t *targs) +static void uffd_move_test(uffd_global_test_opts_t *gopts, uffd_test_args_= t *targs) { - uffd_move_test_common(targs, page_size, uffd_move_handle_fault); + uffd_move_test_common(gopts, targs, gopts->page_size, uffd_move_handle_fa= ult); } =20 -static void uffd_move_pmd_test(uffd_test_args_t *targs) +static void uffd_move_pmd_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) err("madvise(MADV_HUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static void uffd_move_pmd_split_test(uffd_test_args_t *targs) +static void uffd_move_pmd_split_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) err("madvise(MADV_NOHUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 @@ -1287,6 +1278,11 @@ typedef enum { THR_STATE_UNINTERRUPTIBLE, } thread_state; =20 +typedef struct { + uffd_global_test_opts_t *gopts; + volatile pid_t *pid; +} mmap_changing_thread_args; + static void sleep_short(void) { usleep(1000); @@ -1329,7 +1325,9 @@ static void thread_state_until(pid_t tid, thread_stat= e state) =20 static void *uffd_mmap_changing_thread(void *opaque) { - volatile pid_t *pid =3D opaque; + mmap_changing_thread_args *args =3D opaque; + uffd_global_test_opts_t *gopts =3D args->gopts; + volatile pid_t *pid =3D args->pid; int ret; =20 /* Unfortunately, it's only fetch-able from the thread itself.. */ @@ -1337,21 +1335,21 @@ static void *uffd_mmap_changing_thread(void *opaque) *pid =3D syscall(SYS_gettid); =20 /* Inject an event, this will hang solid until the event read */ - ret =3D madvise(area_dst, page_size, MADV_REMOVE); + ret =3D madvise(gopts->area_dst, gopts->page_size, MADV_REMOVE); if (ret) err("madvise(MADV_REMOVE) failed"); =20 return NULL; } =20 -static void uffd_consume_message(int fd) +static void uffd_consume_message(uffd_global_test_opts_t *gopts) { struct uffd_msg msg =3D { 0 }; =20 - while (uffd_read_msg(fd, &msg)); + while (uffd_read_msg(gopts, &msg)); } =20 -static void uffd_mmap_changing_test(uffd_test_args_t *targs) +static void uffd_mmap_changing_test(uffd_global_test_opts_t *gopts, uffd_t= est_args_t *targs) { /* * This stores the real PID (which can be different from how tid is @@ -1360,13 +1358,14 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) pid_t pid =3D 0; pthread_t tid; int ret; + mmap_changing_thread_args args =3D { gopts, &pid }; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("uffd_register() failed"); =20 /* Create a thread to generate the racy event */ - ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &pid); + ret =3D pthread_create(&tid, NULL, uffd_mmap_changing_thread, &args); if (ret) err("pthread_create() failed"); =20 @@ -1380,26 +1379,26 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) /* Wait until the thread hangs at REMOVE event */ thread_state_until(pid, THR_STATE_UNINTERRUPTIBLE); =20 - if (!uffdio_mmap_changing_test_copy(uffd)) + if (!uffdio_mmap_changing_test_copy(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_zeropage(uffd)) + if (!uffdio_mmap_changing_test_zeropage(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_move(uffd)) + if (!uffdio_mmap_changing_test_move(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_poison(uffd)) + if (!uffdio_mmap_changing_test_poison(gopts->uffd)) return; =20 - if (!uffdio_mmap_changing_test_continue(uffd)) + if (!uffdio_mmap_changing_test_continue(gopts->uffd)) return; =20 /* * All succeeded above! Recycle everything. Start by reading the * event so as to kick the thread roll again.. */ - uffd_consume_message(uffd); + uffd_consume_message(gopts); =20 ret =3D pthread_join(tid, NULL); assert(ret =3D=3D 0); @@ -1407,10 +1406,10 @@ static void uffd_mmap_changing_test(uffd_test_args_= t *targs) uffd_test_pass(); } =20 -static int prevent_hugepages(const char **errmsg) +static int prevent_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_NOHUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) { /* Ignore only if CONFIG_TRANSPARENT_HUGEPAGE=3Dn */ if (errno !=3D EINVAL) { if (errmsg) @@ -1421,10 +1420,10 @@ static int prevent_hugepages(const char **errmsg) return 0; } =20 -static int request_hugepages(const char **errmsg) +static int request_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_HUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) { if (errmsg) { *errmsg =3D (errno =3D=3D EINVAL) ? "CONFIG_TRANSPARENT_HUGEPAGE is not set" : @@ -1448,13 +1447,17 @@ struct uffd_test_case_ops uffd_move_test_pmd_case_o= ps =3D { * Note that _UFFDIO_ZEROPAGE is tested separately in the zeropage test. */ static void -do_register_ioctls_test(uffd_test_args_t *args, bool miss, bool wp, bool m= inor) +do_register_ioctls_test(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, + bool miss, + bool wp, + bool minor) { uint64_t ioctls =3D 0, expected =3D BIT_ULL(_UFFDIO_WAKE); mem_type_t *mem_type =3D args->mem_type; int ret; =20 - ret =3D uffd_register_with_ioctls(uffd, area_dst, page_size, + ret =3D uffd_register_with_ioctls(gopts->uffd, gopts->area_dst, gopts->pa= ge_size, miss, wp, minor, &ioctls); =20 /* @@ -1485,18 +1488,18 @@ do_register_ioctls_test(uffd_test_args_t *args, boo= l miss, bool wp, bool minor) "(miss=3D%d, wp=3D%d, minor=3D%d): expected=3D0x%"PRIx64", " "returned=3D0x%"PRIx64, miss, wp, minor, expected, ioctls); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); } =20 -static void uffd_register_ioctls_test(uffd_test_args_t *args) +static void uffd_register_ioctls_test(uffd_global_test_opts_t *gopts, uffd= _test_args_t *args) { int miss, wp, minor; =20 for (miss =3D 0; miss <=3D 1; miss++) for (wp =3D 0; wp <=3D 1; wp++) for (minor =3D 0; minor <=3D 1; minor++) - do_register_ioctls_test(args, miss, wp, minor); + do_register_ioctls_test(gopts, args, miss, wp, minor); =20 uffd_test_pass(); } @@ -1734,6 +1737,27 @@ int main(int argc, char *argv[]) } for (j =3D 0; j < n_mems; j++) { mem_type =3D &mem_types[j]; + + // Initialize global test options + uffd_global_test_opts_t gopts; + + gopts.map_shared =3D mem_type->shared; + uffd_test_ops =3D mem_type->mem_ops; + uffd_test_case_ops =3D test->test_case_ops; + + if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) + gopts.page_size =3D default_huge_page_size(); + else + gopts.page_size =3D psize(); + + /* Ensure we have at least 2 pages */ + gopts.nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, gopts.page_size * 2) / gopts= .page_size; + /* TODO: remove this global var.. it's so ugly */ + gopts.nr_parallel =3D 1; + + /* Initialize test arguments */ + args.mem_type =3D mem_type; + if (!(test->mem_targets & mem_type->mem_flag)) continue; =20 @@ -1748,13 +1772,12 @@ int main(int argc, char *argv[]) uffd_test_skip("feature missing"); continue; } - if (uffd_setup_environment(&args, test, mem_type, - &errmsg)) { + if (uffd_test_ctx_init(&gopts, test->uffd_feature_required, &errmsg)) { uffd_test_skip(errmsg); continue; } - test->uffd_fn(&args); - uffd_test_ctx_clear(); + test->uffd_fn(&gopts, &args); + uffd_test_ctx_clear(&gopts); } } =20 diff --git a/tools/testing/selftests/mm/uffd-wp-mremap.c b/tools/testing/se= lftests/mm/uffd-wp-mremap.c index c2ba7d46c7b4..a24f35728ac3 100644 --- a/tools/testing/selftests/mm/uffd-wp-mremap.c +++ b/tools/testing/selftests/mm/uffd-wp-mremap.c @@ -157,7 +157,11 @@ static bool range_is_swapped(void *addr, size_t size) return true; } =20 -static void test_one_folio(size_t size, bool private, bool swapout, bool h= ugetlb) +static void test_one_folio(uffd_global_test_opts_t *gopts, + size_t size, + bool private, + bool swapout, + bool hugetlb) { struct uffdio_writeprotect wp_prms; uint64_t features =3D 0; @@ -181,21 +185,21 @@ static void test_one_folio(size_t size, bool private,= bool swapout, bool hugetlb } =20 /* Register range for uffd-wp. */ - if (userfaultfd_open(&features)) { + if (userfaultfd_open(gopts, &features)) { if (errno =3D=3D ENOENT) ksft_test_result_skip("userfaultfd not available\n"); else ksft_test_result_fail("userfaultfd_open() failed\n"); goto out; } - if (uffd_register(uffd, mem, size, false, true, false)) { + if (uffd_register(gopts->uffd, mem, size, false, true, false)) { ksft_test_result_fail("uffd_register() failed\n"); goto out; } wp_prms.mode =3D UFFDIO_WRITEPROTECT_MODE_WP; wp_prms.range.start =3D (uintptr_t)mem; wp_prms.range.len =3D size; - if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { + if (ioctl(gopts->uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { ksft_test_result_fail("ioctl(UFFDIO_WRITEPROTECT) failed\n"); goto out; } @@ -242,9 +246,9 @@ static void test_one_folio(size_t size, bool private, b= ool swapout, bool hugetlb out: if (mem) munmap(mem, size); - if (uffd >=3D 0) { - close(uffd); - uffd =3D -1; + if (gopts->uffd >=3D 0) { + close(gopts->uffd); + gopts->uffd =3D -1; } } =20 @@ -336,6 +340,7 @@ static const struct testcase testcases[] =3D { =20 int main(int argc, char **argv) { + uffd_global_test_opts_t gopts; struct thp_settings settings; int i, j, plan =3D 0; =20 @@ -367,8 +372,8 @@ int main(int argc, char **argv) const struct testcase *tc =3D &testcases[i]; =20 for (j =3D 0; j < *tc->nr_sizes; j++) - test_one_folio(tc->sizes[j], tc->private, tc->swapout, - tc->hugetlb); + test_one_folio(&gopts, tc->sizes[j], tc->private, + tc->swapout, tc->hugetlb); } =20 /* If THP is supported, restore original THP settings. */ --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE75B9460; Sun, 4 May 2025 09:56:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352581; cv=none; b=E8yhdi+rhbNnF1RivTBJoPSKAno/2DrLL7Su1q9gZM/P/xcR2UTrWJHu2yu6GldaYs1CFumb1in4OVu1aK3Jfq3OMe+roVg0KTDKOdUIEB6ubqTx7RfUKX8AQPUuVIMHbgKijU/7TlLOsh+jaJPsnoSju9+gQDrAvt+yyFFFgSk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746352581; c=relaxed/simple; bh=03XZfHaFBAsOdZzbCOt5QNBzpE8egVEhUaR6+czfD1w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=C4MoXNqNiWV7ALJumHa1vkX21EgrnlFKdw+ITrQ/+5KPW/gD0dW+324e6V1476fYSCbxw9XoXI/0jHVfgxC40jJTuSvm/tLwD1pWkGwlXhe/aC1kpgGi2QA+P2A6MKNXZTckmXBm9QnRNpIA+9ESHtavvBDHYEm8KlFx1jXAQ3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ymo/AW0D; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ymo/AW0D" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-739be717eddso2816598b3a.2; Sun, 04 May 2025 02:56:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746352576; x=1746957376; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rAadBoxoxEZmDdeo6/8fN6GvNo6WrOG1xfj+G++BC+Y=; b=Ymo/AW0DVPvQ5t5l5BfG63/Fox73KL7cVRvaA5Xb7DXDZtz+WsA1lsRIBqubBfPMas gh55QrdWuhWsUdGkIkRxO1hdfxnvFDEVLYQ8MbeBxmEBVojSG0UiBWIkQXXpYCjMTr1u NDzxFoggvE3GFmxCHeIQdtux0OCTO67xPq168UHbLbliA4yj6OFe60Mtt4whidUGSfZ0 mc9lyZWIzIZxH1gBMFEBcHmYBRreADpkI8RyXzvKI9+45sViP/gfolykovROja7Dg6HY 4ezQwVzSGD2831pNrzOWuCsylZdkEm4YYc+TNziIbUXj/euWqfFpdx4uAZSDSH+zU9Hv zefw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746352576; x=1746957376; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rAadBoxoxEZmDdeo6/8fN6GvNo6WrOG1xfj+G++BC+Y=; b=L0igsh3lTTZf4GnvrfCJdItkWV4STF42/wpAruKkokA+6NbWi41AUNHtmuqgITjVQn DPfffNfchwEC4FQk76B9fAK6iYZpi8cLND2SEsLxebZt8XlXnx9q3ztKWv/3Av+Hcm0f 6qTBhkPrSBgAlGTYxSKzIItPqk8jnl/XvZfOq+1gCB8LmtEaMG+qDpDx4ucNmEVUcB5q 9k7SvfZJPBlPxyZZPGUj5y0JtPhBd607L12u3Zw5WFkBQV/LjLeeFZOPSGB9h7CYoh7L a6xVn2foujnZR3D11eOpWjyxtjbpjmevQvkaNV5v3GGIyQTFf4dG4srQxpdfz/Nghj7v K7Jg== X-Forwarded-Encrypted: i=1; AJvYcCUn8/zAtODQ/wTRHxFDLq/5FzO3Ktg3ayTMSUkwQPxgGpPDjJf8EOsmcPnkA6erBEYBmUYY2Z99EbssOBU=@vger.kernel.org, AJvYcCVzESl+ClOB9f0g3Rkv+mom2XBaftBZYquoNWhwzA9JP/2wGKbrrQEhjxIlJv3XOKHL9kMVwcm27FUWTy+MZwnY@vger.kernel.org X-Gm-Message-State: AOJu0YzQFGyO04kUIRZ16dcM7VQRoYB1OgRUZESRYTASY4Y/BUpJOyqJ tjuXZBP8le58IhVH6NvqsStFReyayG7olSMO5pDfJTKjLTb5Awc= X-Gm-Gg: ASbGncvCU4TJvEDYs5z9E2tB2sg1ta8nxHrtk9MlsgFG3XWAK2CYfv64It37SGXcLcy qfK6wqDT2wybVMvi4bRG1sH55OhGqcxXPoSOLFPI3MDg86gXppkL+6rR2/5yj3SH+YzA+pogox/ QbdTSzZFbBt+Zos2xCRcj9kOxgE6VKAeWQUcAbIwy4US8cTMqjxw6ME50P4w5uvr+3cX4G8W5vX PLQVEwvUVENUEHHUmA9AAfSgCKE0XqRJ6DjTh5841UdsER7ho9svvmpPF8p586RXXr1TvjjG74n +F++DfZmlk8F2RkT1cZ1NQBE8EkfQl74CRch9jfcN3BCmOlcfQ== X-Google-Smtp-Source: AGHT+IE/sMB22bIekpNg8DggkmfH6fsviI5OYmP2wbBUa766fyHaE3+jIFbIg3sFiSKnAGGGcNFmsw== X-Received: by 2002:a05:6a20:1595:b0:1f5:769a:a4c0 with SMTP id adf61e73a8af0-20e06833872mr8355600637.22.1746352575308; Sun, 04 May 2025 02:56:15 -0700 (PDT) Received: from debian.ujwal.com ([223.185.132.49]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-74058dbbaa2sm4735438b3a.57.2025.05.04.02.56.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 04 May 2025 02:56:14 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, shuah@kernel.org, peterx@redhat.com, jackmanb@google.com Cc: linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Ujwal Kundur Subject: [PATCH v2 1/1] selftests/mm/uffd: Refactor non-composite global vars into struct Date: Sun, 4 May 2025 15:18:25 +0530 Message-Id: <20250504094825.491-1-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch refactors macros and non-composite global variable definitions into a struct that is defined at the start of a test and is passed around instead of relying on global vars. Signed-off-by: Ujwal Kundur Tested-by: Brendan Jackman --- Changes since v1: - indentation fixes - squash into single patch to assist bisections tools/testing/selftests/mm/uffd-common.c | 275 +++++----- tools/testing/selftests/mm/uffd-common.h | 78 +-- tools/testing/selftests/mm/uffd-stress.c | 226 ++++---- tools/testing/selftests/mm/uffd-unit-tests.c | 523 ++++++++++--------- tools/testing/selftests/mm/uffd-wp-mremap.c | 23 +- 5 files changed, 594 insertions(+), 531 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selft= ests/mm/uffd-common.c index a37088a23ffe..1b13107ef3c3 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -7,18 +7,31 @@ =20 #include "uffd-common.h" =20 -#define BASE_PMD_ADDR ((void *)(1UL << 30)) - -volatile bool test_uffdio_copy_eexist =3D true; -unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; -int uffd =3D -1, uffd_flags, finished, *pipefd, test_type; -bool map_shared; -bool test_uffdio_wp =3D true; -unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; uffd_test_case_ops_t *uffd_test_case_ops; -atomic_bool ready_for_fork; + +#define BASE_PMD_ADDR ((void *)(1UL << 30)) + +/* pthread_mutex_t starts at page offset 0 */ +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts) +{ + return (pthread_mutex_t *) (area + nr * gopts->page_size); +} + +/* + * count is placed in the page after pthread_mutex_t naturally aligned + * to avoid non alignment faults on non-x86 archs. + */ +volatile unsigned long long *area_count( + char *area, unsigned long nr, + uffd_global_test_opts_t *gopts) +{ + return (volatile unsigned long long *) + ((unsigned long) (area + nr * gopts->page_size + + sizeof(pthread_mutex_t) + + sizeof(unsigned long long) - 1) & + ~(unsigned long)(sizeof(unsigned long long) - 1)); +} =20 static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { @@ -40,15 +53,15 @@ static int uffd_mem_fd_create(off_t mem_size, bool huge= tlb) return mem_fd; } =20 -static void anon_release_pages(char *rel_area) +static void anon_release_pages(uffd_global_test_opts_t *gopts, char *rel_a= rea) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } =20 -static int anon_allocate_area(void **alloc_area, bool is_src) +static int anon_allocate_area(uffd_global_test_opts_t *gopts, void **alloc= _area, bool is_src) { - *alloc_area =3D mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, + *alloc_area =3D mmap(NULL, gopts->nr_pages * gopts->page_size, PROT_READ = | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); if (*alloc_area =3D=3D MAP_FAILED) { *alloc_area =3D NULL; @@ -57,31 +70,33 @@ static int anon_allocate_area(void **alloc_area, bool i= s_src) return 0; } =20 -static void noop_alias_mapping(__u64 *start, size_t len, unsigned long off= set) +static void noop_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, size_t len, + unsigned long offset) { } =20 -static void hugetlb_release_pages(char *rel_area) +static void hugetlb_release_pages(uffd_global_test_opts_t *gopts, char *re= l_area) { - if (!map_shared) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (!gopts->map_shared) { + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } else { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } } =20 -static int hugetlb_allocate_area(void **alloc_area, bool is_src) +static int hugetlb_allocate_area(uffd_global_test_opts_t *gopts, void **al= loc_area, bool is_src) { - off_t size =3D nr_pages * page_size; + off_t size =3D gopts->nr_pages * gopts->page_size; off_t offset =3D is_src ? 0 : size; void *area_alias =3D NULL; char **alloc_area_alias; int mem_fd =3D uffd_mem_fd_create(size * 2, true); =20 *alloc_area =3D mmap(NULL, size, PROT_READ | PROT_WRITE, - (map_shared ? MAP_SHARED : MAP_PRIVATE) | + (gopts->map_shared ? MAP_SHARED : MAP_PRIVATE) | (is_src ? 0 : MAP_NORESERVE), mem_fd, offset); if (*alloc_area =3D=3D MAP_FAILED) { @@ -89,7 +104,7 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) return -errno; } =20 - if (map_shared) { + if (gopts->map_shared) { area_alias =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, mem_fd, offset); if (area_alias =3D=3D MAP_FAILED) @@ -97,9 +112,9 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) } =20 if (is_src) { - alloc_area_alias =3D &area_src_alias; + alloc_area_alias =3D &gopts->area_src_alias; } else { - alloc_area_alias =3D &area_dst_alias; + alloc_area_alias =3D &gopts->area_dst_alias; } if (area_alias) *alloc_area_alias =3D area_alias; @@ -108,24 +123,26 @@ static int hugetlb_allocate_area(void **alloc_area, b= ool is_src) return 0; } =20 -static void hugetlb_alias_mapping(__u64 *start, size_t len, unsigned long = offset) +static void hugetlb_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, size_t len, + unsigned long offset) { - if (!map_shared) + if (!gopts->map_shared) return; =20 - *start =3D (unsigned long) area_dst_alias + offset; + *start =3D (unsigned long) gopts->area_dst_alias + offset; } =20 -static void shmem_release_pages(char *rel_area) +static void shmem_release_pages(uffd_global_test_opts_t *gopts, char *rel_= area) { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } =20 -static int shmem_allocate_area(void **alloc_area, bool is_src) +static int shmem_allocate_area(uffd_global_test_opts_t *gopts, void **allo= c_area, bool is_src) { void *area_alias =3D NULL; - size_t bytes =3D nr_pages * page_size, hpage_size =3D read_pmd_pagesize(); + size_t bytes =3D gopts->nr_pages * gopts->page_size, hpage_size =3D read_= pmd_pagesize(); unsigned long offset =3D is_src ? 0 : bytes; char *p =3D NULL, *p_alias =3D NULL; int mem_fd =3D uffd_mem_fd_create(bytes * 2, false); @@ -159,22 +176,24 @@ static int shmem_allocate_area(void **alloc_area, boo= l is_src) err("mmap of anonymous memory failed at %p", p_alias); =20 if (is_src) - area_src_alias =3D area_alias; + gopts->area_src_alias =3D area_alias; else - area_dst_alias =3D area_alias; + gopts->area_dst_alias =3D area_alias; =20 close(mem_fd); return 0; } =20 -static void shmem_alias_mapping(__u64 *start, size_t len, unsigned long of= fset) +static void shmem_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, + size_t len, unsigned long offset) { - *start =3D (unsigned long)area_dst_alias + offset; + *start =3D (unsigned long)gopts->area_dst_alias + offset; } =20 -static void shmem_check_pmd_mapping(void *p, int expect_nr_hpages) +static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *= p, int expect_nr_hpages) { - if (!check_huge_shmem(area_dst_alias, expect_nr_hpages, + if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages, read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); @@ -234,18 +253,18 @@ void uffd_stats_report(struct uffd_args *args, int n_= cpus) printf("\n"); } =20 -int userfaultfd_open(uint64_t *features) +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features) { struct uffdio_api uffdio_api; =20 - uffd =3D uffd_open(UFFD_FLAGS); - if (uffd < 0) + gopts->uffd =3D uffd_open(UFFD_FLAGS); + if (gopts->uffd < 0) return -1; - uffd_flags =3D fcntl(uffd, F_GETFD, NULL); + gopts->uffd_flags =3D fcntl(gopts->uffd, F_GETFD, NULL); =20 uffdio_api.api =3D UFFD_API; uffdio_api.features =3D *features; - if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + if (ioctl(gopts->uffd, UFFDIO_API, &uffdio_api)) /* Probably lack of CAP_PTRACE? */ return -1; if (uffdio_api.api !=3D UFFD_API) @@ -255,59 +274,59 @@ int userfaultfd_open(uint64_t *features) return 0; } =20 -static inline void munmap_area(void **area) +static inline void munmap_area(uffd_global_test_opts_t *gopts, void **area) { if (*area) - if (munmap(*area, nr_pages * page_size)) + if (munmap(*area, gopts->nr_pages * gopts->page_size)) err("munmap"); =20 *area =3D NULL; } =20 -void uffd_test_ctx_clear(void) +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts) { size_t i; =20 - if (pipefd) { - for (i =3D 0; i < nr_parallel * 2; ++i) { - if (close(pipefd[i])) + if (gopts->pipefd) { + for (i =3D 0; i < gopts->nr_parallel * 2; ++i) { + if (close(gopts->pipefd[i])) err("close pipefd"); } - free(pipefd); - pipefd =3D NULL; + free(gopts->pipefd); + gopts->pipefd =3D NULL; } =20 - if (count_verify) { - free(count_verify); - count_verify =3D NULL; + if (gopts->count_verify) { + free(gopts->count_verify); + gopts->count_verify =3D NULL; } =20 - if (uffd !=3D -1) { - if (close(uffd)) + if (gopts->uffd !=3D -1) { + if (close(gopts->uffd)) err("close uffd"); - uffd =3D -1; + gopts->uffd =3D -1; } =20 - munmap_area((void **)&area_src); - munmap_area((void **)&area_src_alias); - munmap_area((void **)&area_dst); - munmap_area((void **)&area_dst_alias); - munmap_area((void **)&area_remap); + munmap_area(gopts, (void **)&gopts->area_src); + munmap_area(gopts, (void **)&gopts->area_src_alias); + munmap_area(gopts, (void **)&gopts->area_dst); + munmap_area(gopts, (void **)&gopts->area_dst_alias); + munmap_area(gopts, (void **)&gopts->area_remap); } =20 -int uffd_test_ctx_init(uint64_t features, const char **errmsg) +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg) { unsigned long nr, cpu; int ret; =20 if (uffd_test_case_ops && uffd_test_case_ops->pre_alloc) { - ret =3D uffd_test_case_ops->pre_alloc(errmsg); + ret =3D uffd_test_case_ops->pre_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D uffd_test_ops->allocate_area((void **)&area_src, true); - ret |=3D uffd_test_ops->allocate_area((void **)&area_dst, false); + ret =3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_src, t= rue); + ret |=3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_dst, = false); if (ret) { if (errmsg) *errmsg =3D "memory allocation failed"; @@ -315,26 +334,26 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) } =20 if (uffd_test_case_ops && uffd_test_case_ops->post_alloc) { - ret =3D uffd_test_case_ops->post_alloc(errmsg); + ret =3D uffd_test_case_ops->post_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D userfaultfd_open(&features); + ret =3D userfaultfd_open(gopts, &features); if (ret) { if (errmsg) *errmsg =3D "possible lack of privilege"; return ret; } =20 - count_verify =3D malloc(nr_pages * sizeof(unsigned long long)); - if (!count_verify) + gopts->count_verify =3D malloc(gopts->nr_pages * sizeof(unsigned long lon= g)); + if (!gopts->count_verify) err("count_verify"); =20 - for (nr =3D 0; nr < nr_pages; nr++) { - *area_mutex(area_src, nr) =3D + for (nr =3D 0; nr < gopts->nr_pages; nr++) { + *area_mutex(gopts->area_src, nr, gopts) =3D (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER; - count_verify[nr] =3D *area_count(area_src, nr) =3D 1; + gopts->count_verify[nr] =3D *area_count(gopts->area_src, nr, gopts) =3D = 1; /* * In the transition between 255 to 256, powerpc will * read out of order in my_bcmp and see both bytes as @@ -342,7 +361,7 @@ int uffd_test_ctx_init(uint64_t features, const char **= errmsg) * after the count, to avoid my_bcmp to trigger false * positives. */ - *(area_count(area_src, nr) + 1) =3D 1; + *(area_count(gopts->area_src, nr, gopts) + 1) =3D 1; } =20 /* @@ -363,13 +382,13 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) * proactively split the thp and drop any accidentally initialized * pages within area_dst. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - pipefd =3D malloc(sizeof(int) * nr_parallel * 2); - if (!pipefd) + gopts->pipefd =3D malloc(sizeof(int) * gopts->nr_parallel * 2); + if (!gopts->pipefd) err("pipefd"); - for (cpu =3D 0; cpu < nr_parallel; cpu++) - if (pipe2(&pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) + if (pipe2(&gopts->pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) err("pipe"); =20 return 0; @@ -416,9 +435,9 @@ static void continue_range(int ufd, __u64 start, __u64 = len, bool wp) ret, (int64_t) req.mapped); } =20 -int uffd_read_msg(int ufd, struct uffd_msg *msg) +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg) { - int ret =3D read(uffd, msg, sizeof(*msg)); + int ret =3D read(gopts->uffd, msg, sizeof(*msg)); =20 if (ret !=3D sizeof(*msg)) { if (ret < 0) { @@ -433,7 +452,9 @@ int uffd_read_msg(int ufd, struct uffd_msg *msg) return 0; } =20 -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args) +void uffd_handle_page_fault( + uffd_global_test_opts_t *gopts, struct uffd_msg *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -442,7 +463,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) =20 if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP) { /* Write protect page faults */ - wp_range(uffd, msg->arg.pagefault.address, page_size, false); + wp_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, fals= e); args->wp_faults++; } else if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR) { uint8_t *area; @@ -460,12 +481,12 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) * (UFFD-registered). */ =20 - area =3D (uint8_t *)(area_dst + + area =3D (uint8_t *)(gopts->area_dst + ((char *)msg->arg.pagefault.address - - area_dst_alias)); - for (b =3D 0; b < page_size; ++b) + gopts->area_dst_alias)); + for (b =3D 0; b < gopts->page_size; ++b) area[b] =3D ~area[b]; - continue_range(uffd, msg->arg.pagefault.address, page_size, + continue_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, args->apply_wp); args->minor_faults++; } else { @@ -493,10 +514,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WRITE) err("unexpected write fault"); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->ar= ea_dst; + offset &=3D ~(gopts->page_size-1); =20 - if (copy_page(uffd, offset, args->apply_wp)) + if (copy_page(gopts, offset, args->apply_wp)) args->missing_faults++; } } @@ -504,6 +525,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) void *uffd_poll_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; unsigned long cpu =3D args->cpu; struct pollfd pollfd[2]; struct uffd_msg msg; @@ -514,12 +536,12 @@ void *uffd_poll_thread(void *arg) if (!args->handle_fault) args->handle_fault =3D uffd_handle_page_fault; =20 - pollfd[0].fd =3D uffd; + pollfd[0].fd =3D gopts->uffd; pollfd[0].events =3D POLLIN; - pollfd[1].fd =3D pipefd[cpu*2]; + pollfd[1].fd =3D gopts->pipefd[cpu*2]; pollfd[1].events =3D POLLIN; =20 - ready_for_fork =3D true; + gopts->ready_for_fork =3D true; =20 for (;;) { ret =3D poll(pollfd, 2, -1); @@ -537,30 +559,30 @@ void *uffd_poll_thread(void *arg) } if (!(pollfd[0].revents & POLLIN)) err("pollfd[0].revents %d", pollfd[0].revents); - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; switch (msg.event) { default: err("unexpected msg event %u\n", msg.event); break; case UFFD_EVENT_PAGEFAULT: - args->handle_fault(&msg, args); + args->handle_fault(gopts, &msg, args); break; case UFFD_EVENT_FORK: - close(uffd); - uffd =3D msg.arg.fork.ufd; - pollfd[0].fd =3D uffd; + close(gopts->uffd); + gopts->uffd =3D msg.arg.fork.ufd; + pollfd[0].fd =3D gopts->uffd; break; case UFFD_EVENT_REMOVE: uffd_reg.range.start =3D msg.arg.remove.start; uffd_reg.range.len =3D msg.arg.remove.end - msg.arg.remove.start; - if (ioctl(uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) + if (ioctl(gopts->uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) err("remove failure"); break; case UFFD_EVENT_REMAP: - area_remap =3D area_dst; /* save for later unmap */ - area_dst =3D (char *)(unsigned long)msg.arg.remap.to; + gopts->area_remap =3D gopts->area_dst; /* save for later unmap */ + gopts->area_dst =3D (char *)(unsigned long)msg.arg.remap.to; break; } } @@ -568,13 +590,14 @@ void *uffd_poll_thread(void *arg) return NULL; } =20 -static void retry_copy_page(int ufd, struct uffdio_copy *uffdio_copy, - unsigned long offset) +static void retry_copy_page(uffd_global_test_opts_t *gopts, struct uffdio_= copy *uffdio_copy, + unsigned long offset) { - uffd_test_ops->alias_mapping(&uffdio_copy->dst, - uffdio_copy->len, - offset); - if (ioctl(ufd, UFFDIO_COPY, uffdio_copy)) { + uffd_test_ops->alias_mapping(gopts, + &uffdio_copy->dst, + uffdio_copy->len, + offset); + if (ioctl(gopts->uffd, UFFDIO_COPY, uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy->copy !=3D -EEXIST) err("UFFDIO_COPY retry error: %"PRId64, @@ -597,60 +620,60 @@ static void wake_range(int ufd, unsigned long addr, u= nsigned long len) addr), exit(1); } =20 -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp) +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp) { struct uffdio_copy uffdio_copy; =20 - if (offset >=3D nr_pages * page_size) + if (offset >=3D gopts->nr_pages * gopts->page_size) err("unexpected offset %lu\n", offset); - uffdio_copy.dst =3D (unsigned long) area_dst + offset; - uffdio_copy.src =3D (unsigned long) area_src + offset; - uffdio_copy.len =3D page_size; + uffdio_copy.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_copy.src =3D (unsigned long) gopts->area_src + offset; + uffdio_copy.len =3D gopts->page_size; if (wp) uffdio_copy.mode =3D UFFDIO_COPY_MODE_WP; else uffdio_copy.mode =3D 0; uffdio_copy.copy =3D 0; - if (ioctl(ufd, UFFDIO_COPY, &uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, &uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy.copy !=3D -EEXIST) err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); - wake_range(ufd, uffdio_copy.dst, page_size); - } else if (uffdio_copy.copy !=3D page_size) { + wake_range(gopts->uffd, uffdio_copy.dst, gopts->page_size); + } else if (uffdio_copy.copy !=3D gopts->page_size) { err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); } else { - if (test_uffdio_copy_eexist && retry) { - test_uffdio_copy_eexist =3D false; - retry_copy_page(ufd, &uffdio_copy, offset); + if (gopts->test_uffdio_copy_eexist && retry) { + gopts->test_uffdio_copy_eexist =3D false; + retry_copy_page(gopts, &uffdio_copy, offset); } return 1; } return 0; } =20 -int copy_page(int ufd, unsigned long offset, bool wp) +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p) { - return __copy_page(ufd, offset, false, wp); + return __copy_page(gopts, offset, false, wp); } =20 -int move_page(int ufd, unsigned long offset, unsigned long len) +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len) { struct uffdio_move uffdio_move; =20 - if (offset + len > nr_pages * page_size) + if (offset + len > gopts->nr_pages * gopts->page_size) err("unexpected offset %lu and length %lu\n", offset, len); - uffdio_move.dst =3D (unsigned long) area_dst + offset; - uffdio_move.src =3D (unsigned long) area_src + offset; + uffdio_move.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_move.src =3D (unsigned long) gopts->area_src + offset; uffdio_move.len =3D len; uffdio_move.mode =3D UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES; uffdio_move.move =3D 0; - if (ioctl(ufd, UFFDIO_MOVE, &uffdio_move)) { + if (ioctl(gopts->uffd, UFFDIO_MOVE, &uffdio_move)) { /* real retval in uffdio_move.move */ if (uffdio_move.move !=3D -EEXIST) err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); - wake_range(ufd, uffdio_move.dst, len); + wake_range(gopts->uffd, uffdio_move.dst, len); } else if (uffdio_move.move !=3D len) { err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); } else diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selft= ests/mm/uffd-common.h index 7700cbfa3975..a3debf116d58 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -56,20 +56,17 @@ =20 #define err(fmt, ...) errexit(1, fmt, ##__VA_ARGS__) =20 -/* pthread_mutex_t starts at page offset 0 */ -#define area_mutex(___area, ___nr) \ - ((pthread_mutex_t *) ((___area) + (___nr)*page_size)) -/* - * count is placed in the page after pthread_mutex_t naturally aligned - * to avoid non alignment faults on non-x86 archs. - */ -#define area_count(___area, ___nr) \ - ((volatile unsigned long long *) ((unsigned long) \ - ((___area) + (___nr)*page_size + \ - sizeof(pthread_mutex_t) + \ - sizeof(unsigned long long) - 1) & \ - ~(unsigned long)(sizeof(unsigned long long) \ - - 1))) +struct uffd_global_test_opts { + unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; + char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; + int uffd, uffd_flags, finished, *pipefd, test_type; + bool map_shared; + bool test_uffdio_wp; + unsigned long long *count_verify; + volatile bool test_uffdio_copy_eexist; + atomic_bool ready_for_fork; +}; +typedef struct uffd_global_test_opts uffd_global_test_opts_t; =20 /* Userfaultfd test statistics */ struct uffd_args { @@ -79,50 +76,55 @@ struct uffd_args { unsigned long missing_faults; unsigned long wp_faults; unsigned long minor_faults; + struct uffd_global_test_opts *gopts; =20 /* A custom fault handler; defaults to uffd_handle_page_fault. */ - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args); + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, + struct uffd_args *args); }; =20 struct uffd_test_ops { - int (*allocate_area)(void **alloc_area, bool is_src); - void (*release_pages)(char *rel_area); - void (*alias_mapping)(__u64 *start, size_t len, unsigned long offset); - void (*check_pmd_mapping)(void *p, int expect_nr_hpages); + int (*allocate_area)(uffd_global_test_opts_t *gopts, void **alloc_area, b= ool is_src); + void (*release_pages)(uffd_global_test_opts_t *gopts, char *rel_area); + void (*alias_mapping)(uffd_global_test_opts_t *gopts, + __u64 *start, + size_t len, + unsigned long offset); + void (*check_pmd_mapping)(uffd_global_test_opts_t *gopts, void *p, int ex= pect_nr_hpages); }; typedef struct uffd_test_ops uffd_test_ops_t; =20 struct uffd_test_case_ops { - int (*pre_alloc)(const char **errmsg); - int (*post_alloc)(const char **errmsg); + int (*pre_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); + int (*post_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); }; typedef struct uffd_test_case_ops uffd_test_case_ops_t; =20 -extern unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -extern char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_= remap; -extern int uffd, uffd_flags, finished, *pipefd, test_type; -extern bool map_shared; -extern bool test_uffdio_wp; -extern unsigned long long *count_verify; -extern volatile bool test_uffdio_copy_eexist; -extern atomic_bool ready_for_fork; - +extern uffd_global_test_opts_t *uffd_gtest_opts; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; extern uffd_test_ops_t hugetlb_uffd_test_ops; extern uffd_test_ops_t *uffd_test_ops; extern uffd_test_case_ops_t *uffd_test_case_ops; =20 +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts); +volatile unsigned long long *area_count(char *area, + unsigned long nr, + uffd_global_test_opts_t *gopts); + void uffd_stats_report(struct uffd_args *args, int n_cpus); -int uffd_test_ctx_init(uint64_t features, const char **errmsg); -void uffd_test_ctx_clear(void); -int userfaultfd_open(uint64_t *features); -int uffd_read_msg(int ufd, struct uffd_msg *msg); +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg); +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts); +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features); +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg); void wp_range(int ufd, __u64 start, __u64 len, bool wp); -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); -int copy_page(int ufd, unsigned long offset, bool wp); -int move_page(int ufd, unsigned long offset, unsigned long len); +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args); +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp); +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p); +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len); void *uffd_poll_thread(void *arg); =20 int uffd_open_dev(unsigned int flags); diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selft= ests/mm/uffd-stress.c index 40af7f67c407..e7aac958d57d 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -76,54 +76,58 @@ static void usage(void) exit(1); } =20 -static void uffd_stats_reset(struct uffd_args *args, unsigned long n_cpus) +static void uffd_stats_reset(uffd_global_test_opts_t *gopts, struct uffd_a= rgs *args, + unsigned long n_cpus) { int i; =20 for (i =3D 0; i < n_cpus; i++) { args[i].cpu =3D i; - args[i].apply_wp =3D test_uffdio_wp; + args[i].apply_wp =3D gopts->test_uffdio_wp; args[i].missing_faults =3D 0; args[i].wp_faults =3D 0; args[i].minor_faults =3D 0; + args[i].gopts =3D gopts; } } =20 static void *locking_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr; unsigned long long count; =20 if (!(bounces & BOUNCE_RANDOM)) { page_nr =3D -bounces; if (!(bounces & BOUNCE_RACINGFAULTS)) - page_nr +=3D cpu * nr_pages_per_cpu; + page_nr +=3D cpu * gopts->nr_pages_per_cpu; } =20 - while (!finished) { + while (!gopts->finished) { if (bounces & BOUNCE_RANDOM) { if (getrandom(&page_nr, sizeof(page_nr), 0) !=3D sizeof(page_nr)) err("getrandom failed"); } else page_nr +=3D 1; - page_nr %=3D nr_pages; - pthread_mutex_lock(area_mutex(area_dst, page_nr)); - count =3D *area_count(area_dst, page_nr); - if (count !=3D count_verify[page_nr]) + page_nr %=3D gopts->nr_pages; + pthread_mutex_lock(area_mutex(gopts->area_dst, page_nr, gopts)); + count =3D *area_count(gopts->area_dst, page_nr, gopts); + if (count !=3D gopts->count_verify[page_nr]) err("page_nr %lu memory corruption %llu %llu", - page_nr, count, count_verify[page_nr]); + page_nr, count, gopts->count_verify[page_nr]); count++; - *area_count(area_dst, page_nr) =3D count_verify[page_nr] =3D count; - pthread_mutex_unlock(area_mutex(area_dst, page_nr)); + *area_count(gopts->area_dst, page_nr, gopts) =3D gopts->count_verify[pag= e_nr] =3D count; + pthread_mutex_unlock(area_mutex(gopts->area_dst, page_nr, gopts)); } =20 return NULL; } =20 -static int copy_page_retry(int ufd, unsigned long offset) +static int copy_page_retry(uffd_global_test_opts_t *gopts, unsigned long o= ffset) { - return __copy_page(ufd, offset, true, test_uffdio_wp); + return __copy_page(gopts, offset, true, gopts->test_uffdio_wp); } =20 pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INITIALIZER; @@ -131,15 +135,16 @@ pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INI= TIALIZER; static void *uffd_read_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; struct uffd_msg msg; =20 pthread_mutex_unlock(&uffd_read_mutex); /* from here cancellation is ok */ =20 for (;;) { - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; - uffd_handle_page_fault(&msg, args); + uffd_handle_page_fault(gopts, &msg, args); } =20 return NULL; @@ -147,32 +152,34 @@ static void *uffd_read_thread(void *arg) =20 static void *background_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr, start_nr, mid_nr, end_nr; =20 - start_nr =3D cpu * nr_pages_per_cpu; - end_nr =3D (cpu+1) * nr_pages_per_cpu; + start_nr =3D cpu * gopts->nr_pages_per_cpu; + end_nr =3D (cpu+1) * gopts->nr_pages_per_cpu; mid_nr =3D (start_nr + end_nr) / 2; =20 /* Copy the first half of the pages */ for (page_nr =3D start_nr; page_nr < mid_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 /* * If we need to test uffd-wp, set it up now. Then we'll have * at least the first half of the pages mapped already which * can be write-protected for testing */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst + start_nr * page_size, - nr_pages_per_cpu * page_size, true); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst + start_nr * gopts-= >page_size, + gopts->nr_pages_per_cpu * gopts->page_size, true); =20 /* * Continue the 2nd half of the page copying, handling write * protection faults if any */ for (page_nr =3D mid_nr; page_nr < end_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 return NULL; } @@ -180,17 +187,23 @@ static void *background_thread(void *arg) static int stress(struct uffd_args *args) { unsigned long cpu; - pthread_t locking_threads[nr_parallel]; - pthread_t uffd_threads[nr_parallel]; - pthread_t background_threads[nr_parallel]; + uffd_global_test_opts_t *gopts =3D args->gopts; + pthread_t locking_threads[gopts->nr_parallel]; + pthread_t uffd_threads[gopts->nr_parallel]; + pthread_t background_threads[gopts->nr_parallel]; =20 - finished =3D 0; - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + gopts->finished =3D 0; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { if (pthread_create(&locking_threads[cpu], &attr, - locking_thread, (void *)cpu)) + locking_thread, (void *)&args[cpu])) return 1; if (bounces & BOUNCE_POLL) { - if (pthread_create(&uffd_threads[cpu], &attr, uffd_poll_thread, &args[c= pu])) + if (pthread_create( + &uffd_threads[cpu], + &attr, + uffd_poll_thread, + (void *) &args[cpu]) + ) err("uffd_poll_thread create"); } else { if (pthread_create(&uffd_threads[cpu], &attr, @@ -200,10 +213,10 @@ static int stress(struct uffd_args *args) pthread_mutex_lock(&uffd_read_mutex); } if (pthread_create(&background_threads[cpu], &attr, - background_thread, (void *)cpu)) + background_thread, (void *)&args[cpu])) return 1; } - for (cpu =3D 0; cpu < nr_parallel; cpu++) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(background_threads[cpu], NULL)) return 1; =20 @@ -216,17 +229,17 @@ static int stress(struct uffd_args *args) * UFFDIO_COPY without writing zero pages into area_dst * because the background threads already completed). */ - uffd_test_ops->release_pages(area_src); + uffd_test_ops->release_pages(gopts, gopts->area_src); =20 - finished =3D 1; - for (cpu =3D 0; cpu < nr_parallel; cpu++) + gopts->finished =3D 1; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(locking_threads[cpu], NULL)) return 1; =20 - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { char c; if (bounces & BOUNCE_POLL) { - if (write(pipefd[cpu*2+1], &c, 1) !=3D 1) + if (write(gopts->pipefd[cpu*2+1], &c, 1) !=3D 1) err("pipefd write error"); if (pthread_join(uffd_threads[cpu], (void *)&args[cpu])) @@ -242,26 +255,26 @@ static int stress(struct uffd_args *args) return 0; } =20 -static int userfaultfd_stress(void) +static int userfaultfd_stress(uffd_global_test_opts_t *gopts) { void *area; unsigned long nr; - struct uffd_args args[nr_parallel]; - uint64_t mem_size =3D nr_pages * page_size; + struct uffd_args args[gopts->nr_parallel]; + uint64_t mem_size =3D gopts->nr_pages * gopts->page_size; int flags =3D 0; =20 - memset(args, 0, sizeof(struct uffd_args) * nr_parallel); + memset(args, 0, sizeof(struct uffd_args) * gopts->nr_parallel); =20 - if (features & UFFD_FEATURE_WP_UNPOPULATED && test_type =3D=3D TEST_ANON) + if (features & UFFD_FEATURE_WP_UNPOPULATED && gopts->test_type =3D=3D TES= T_ANON) flags =3D UFFD_FEATURE_WP_UNPOPULATED; =20 - if (uffd_test_ctx_init(flags, NULL)) + if (uffd_test_ctx_init(gopts, flags, NULL)) err("context init failed"); =20 - if (posix_memalign(&area, page_size, page_size)) + if (posix_memalign(&area, gopts->page_size, gopts->page_size)) err("out of memory"); zeropage =3D area; - bzero(zeropage, page_size); + bzero(zeropage, gopts->page_size); =20 pthread_mutex_lock(&uffd_read_mutex); =20 @@ -284,18 +297,18 @@ static int userfaultfd_stress(void) fflush(stdout); =20 if (bounces & BOUNCE_POLL) - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); else - fcntl(uffd, F_SETFL, uffd_flags & ~O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags & ~O_NONBLOCK); =20 /* register */ - if (uffd_register(uffd, area_dst, mem_size, - true, test_uffdio_wp, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure"); =20 - if (area_dst_alias) { - if (uffd_register(uffd, area_dst_alias, mem_size, - true, test_uffdio_wp, false)) + if (gopts->area_dst_alias) { + if (uffd_register(gopts->uffd, gopts->area_dst_alias, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure alias"); } =20 @@ -323,87 +336,88 @@ static int userfaultfd_stress(void) * MADV_DONTNEED only after the UFFDIO_REGISTER, so it's * required to MADV_DONTNEED here. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - uffd_stats_reset(args, nr_parallel); + uffd_stats_reset(gopts, args, gopts->nr_parallel); =20 /* bounce pass */ if (stress(args)) { - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); return 1; } =20 /* Clear all the write protections if there is any */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst, - nr_pages * page_size, false); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst, + gopts->nr_pages * gopts->page_size, false); =20 /* unregister */ - if (uffd_unregister(uffd, area_dst, mem_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, mem_size)) err("unregister failure"); - if (area_dst_alias) { - if (uffd_unregister(uffd, area_dst_alias, mem_size)) + if (gopts->area_dst_alias) { + if (uffd_unregister(gopts->uffd, gopts->area_dst_alias, mem_size)) err("unregister failure alias"); } =20 /* verification */ if (bounces & BOUNCE_VERIFY) - for (nr =3D 0; nr < nr_pages; nr++) - if (*area_count(area_dst, nr) !=3D count_verify[nr]) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + if (*area_count(gopts->area_dst, nr, gopts) !=3D + gopts->count_verify[nr]) err("error area_count %llu %llu %lu\n", - *area_count(area_src, nr), - count_verify[nr], nr); + *area_count(gopts->area_src, nr, gopts), + gopts->count_verify[nr], nr); =20 /* prepare next bounce */ - swap(area_src, area_dst); + swap(gopts->area_src, gopts->area_dst); =20 - swap(area_src_alias, area_dst_alias); + swap(gopts->area_src_alias, gopts->area_dst_alias); =20 - uffd_stats_report(args, nr_parallel); + uffd_stats_report(args, gopts->nr_parallel); } - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); =20 return 0; } =20 -static void set_test_type(const char *type) +static void set_test_type(uffd_global_test_opts_t *gopts, const char *type) { if (!strcmp(type, "anon")) { - test_type =3D TEST_ANON; + gopts->test_type =3D TEST_ANON; uffd_test_ops =3D &anon_uffd_test_ops; } else if (!strcmp(type, "hugetlb")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; - map_shared =3D true; + gopts->map_shared =3D true; } else if (!strcmp(type, "hugetlb-private")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; } else if (!strcmp(type, "shmem")) { - map_shared =3D true; - test_type =3D TEST_SHMEM; + gopts->map_shared =3D true; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } else if (!strcmp(type, "shmem-private")) { - test_type =3D TEST_SHMEM; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } } =20 -static void parse_test_type_arg(const char *raw_type) +static void parse_test_type_arg(uffd_global_test_opts_t *gopts, const char= *raw_type) { - set_test_type(raw_type); + set_test_type(gopts, raw_type); =20 - if (!test_type) + if (!gopts->test_type) err("failed to parse test type argument: '%s'", raw_type); =20 - if (test_type =3D=3D TEST_HUGETLB) - page_size =3D default_huge_page_size(); + if (gopts->test_type =3D=3D TEST_HUGETLB) + gopts->page_size =3D default_huge_page_size(); else - page_size =3D sysconf(_SC_PAGE_SIZE); + gopts->page_size =3D sysconf(_SC_PAGE_SIZE); =20 - if (!page_size) + if (!gopts->page_size) err("Unable to determine page size"); - if ((unsigned long) area_count(NULL, 0) + sizeof(unsigned long long) * 2 - > page_size) + if ((unsigned long) area_count(NULL, 0, gopts) + sizeof(unsigned long lon= g) * 2 + > gopts->page_size) err("Impossible to run this test"); =20 /* @@ -415,21 +429,22 @@ static void parse_test_type_arg(const char *raw_type) if (uffd_get_features(&features) && errno =3D=3D ENOENT) ksft_exit_skip("failed to get available features (%d)\n", errno); =20 - test_uffdio_wp =3D test_uffdio_wp && + gopts->test_uffdio_wp =3D gopts->test_uffdio_wp && (features & UFFD_FEATURE_PAGEFAULT_FLAG_WP); =20 - if (test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGETLBFS_SH= MEM)) - test_uffdio_wp =3D false; + if (gopts->test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGET= LBFS_SHMEM)) + gopts->test_uffdio_wp =3D false; =20 - close(uffd); - uffd =3D -1; + close(gopts->uffd); + gopts->uffd =3D -1; } =20 static void sigalrm(int sig) { if (sig !=3D SIGALRM) abort(); - test_uffdio_copy_eexist =3D true; + // TODO: Set this without access to global vars + // gopts->test_uffdio_copy_eexist =3D true; alarm(ALARM_INTERVAL_SECS); } =20 @@ -438,6 +453,9 @@ int main(int argc, char **argv) unsigned long nr_cpus; size_t bytes; =20 + uffd_global_test_opts_t *gopts =3D + (uffd_global_test_opts_t *) malloc(sizeof(uffd_global_test_opts_t)); + if (argc < 4) usage(); =20 @@ -445,11 +463,11 @@ int main(int argc, char **argv) err("failed to arm SIGALRM"); alarm(ALARM_INTERVAL_SECS); =20 - parse_test_type_arg(argv[1]); + parse_test_type_arg(gopts, argv[1]); bytes =3D atol(argv[2]) * 1024 * 1024; =20 - if (test_type =3D=3D TEST_HUGETLB && - get_free_hugepages() < bytes / page_size) { + if (gopts->test_type =3D=3D TEST_HUGETLB && + get_free_hugepages() < bytes / gopts->page_size) { printf("skip: Skipping userfaultfd... not enough hugepages\n"); return KSFT_SKIP; } @@ -459,15 +477,15 @@ int main(int argc, char **argv) /* Don't let calculation below go to zero. */ ksft_print_msg("_SC_NPROCESSORS_ONLN (%lu) too large, capping nr_threads= to 32\n", nr_cpus); - nr_parallel =3D 32; + gopts->nr_parallel =3D 32; } else { - nr_parallel =3D nr_cpus; + gopts->nr_parallel =3D nr_cpus; } =20 - nr_pages_per_cpu =3D bytes / page_size / nr_parallel; - if (!nr_pages_per_cpu) { + gopts->nr_pages_per_cpu =3D bytes / gopts->page_size / gopts->nr_parallel; + if (!gopts->nr_pages_per_cpu) { _err("pages_per_cpu =3D 0, cannot test (%lu / %lu / %lu)", - bytes, page_size, nr_parallel); + bytes, gopts->page_size, gopts->nr_parallel); usage(); } =20 @@ -476,11 +494,11 @@ int main(int argc, char **argv) _err("invalid bounces"); usage(); } - nr_pages =3D nr_pages_per_cpu * nr_parallel; + gopts->nr_pages =3D gopts->nr_pages_per_cpu * gopts->nr_parallel; =20 printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n", - nr_pages, nr_pages_per_cpu); - return userfaultfd_stress(); + gopts->nr_pages, gopts->nr_pages_per_cpu); + return userfaultfd_stress(gopts); } =20 #else /* __NR_userfaultfd */ diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/s= elftests/mm/uffd-unit-tests.c index e8fd9011c2a3..86ecf21e0722 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -76,7 +76,7 @@ struct uffd_test_args { typedef struct uffd_test_args uffd_test_args_t; =20 /* Returns: UFFD_TEST_* */ -typedef void (*uffd_test_fn)(uffd_test_args_t *); +typedef void (*uffd_test_fn)(uffd_global_test_opts_t *, uffd_test_args_t *= ); =20 typedef struct { const char *name; @@ -181,33 +181,6 @@ static int test_uffd_api(bool use_dev) return 1; } =20 -/* - * This function initializes the global variables. TODO: remove global - * vars and then remove this. - */ -static int -uffd_setup_environment(uffd_test_args_t *args, uffd_test_case_t *test, - mem_type_t *mem_type, const char **errmsg) -{ - map_shared =3D mem_type->shared; - uffd_test_ops =3D mem_type->mem_ops; - uffd_test_case_ops =3D test->test_case_ops; - - if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) - page_size =3D default_huge_page_size(); - else - page_size =3D psize(); - - /* Ensure we have at least 2 pages */ - nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, page_size * 2) / page_size; - /* TODO: remove this global var.. it's so ugly */ - nr_parallel =3D 1; - - /* Initialize test arguments */ - args->mem_type =3D mem_type; - - return uffd_test_ctx_init(test->uffd_feature_required, errmsg); -} =20 static bool uffd_feature_supported(uffd_test_case_t *test) { @@ -237,7 +210,8 @@ static int pagemap_open(void) } while (0) =20 typedef struct { - int parent_uffd, child_uffd; + uffd_global_test_opts_t *gopts; + int child_uffd; } fork_event_args; =20 static void *fork_event_consumer(void *data) @@ -245,10 +219,10 @@ static void *fork_event_consumer(void *data) fork_event_args *args =3D data; struct uffd_msg msg =3D { 0 }; =20 - ready_for_fork =3D true; + args->gopts->ready_for_fork =3D true; =20 /* Read until a full msg received */ - while (uffd_read_msg(args->parent_uffd, &msg)); + while (uffd_read_msg(args->gopts, &msg)); =20 if (msg.event !=3D UFFD_EVENT_FORK) err("wrong message: %u\n", msg.event); @@ -304,9 +278,9 @@ static void unpin_pages(pin_args *args) args->pinned =3D false; } =20 -static int pagemap_test_fork(int uffd, bool with_event, bool test_pin) +static int pagemap_test_fork(uffd_global_test_opts_t *gopts, bool with_eve= nt, bool test_pin) { - fork_event_args args =3D { .parent_uffd =3D uffd, .child_uffd =3D -1 }; + fork_event_args args =3D { .gopts =3D gopts, .child_uffd =3D -1 }; pthread_t thread; pid_t child; uint64_t value; @@ -314,10 +288,10 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 /* Prepare a thread to resolve EVENT_FORK */ if (with_event) { - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; if (pthread_create(&thread, NULL, fork_event_consumer, &args)) err("pthread_create()"); - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ } =20 @@ -328,14 +302,14 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 fd =3D pagemap_open(); =20 - if (test_pin && pin_pages(&args, area_dst, page_size)) + if (test_pin && pin_pages(&args, gopts->area_dst, gopts->page_size)) /* * Normally when reach here we have pinned in * previous tests, so shouldn't fail anymore */ err("pin page failed in child"); =20 - value =3D pagemap_get_entry(fd, area_dst); + value =3D pagemap_get_entry(fd, gopts->area_dst); /* * After fork(), we should handle uffd-wp bit differently: * @@ -361,70 +335,70 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) return result; } =20 -static void uffd_wp_unpopulated_test(uffd_test_args_t *args) +static void uffd_wp_unpopulated_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args) { uint64_t value; int pagemap_fd; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Test applying pte marker to anon unpopulated */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); =20 /* Test unprotect on anon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test zap on anon marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - if (madvise(area_dst, page_size, MADV_DONTNEED)) + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test fault in after marker removed */ - *area_dst =3D 1; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 /* Test read-zero-page upon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - *(volatile char *)area_dst; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + *(volatile char *)gopts->area_dst; /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 uffd_test_pass(); } =20 -static void uffd_wp_fork_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_test_common(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args, bool with_event) { int pagemap_fd; uint64_t value; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in present pte", with_event ? "missing" : "stall"); goto out; @@ -442,79 +416,79 @@ static void uffd_wp_fork_test_common(uffd_test_args_t= *args, * to expose pte markers. */ if (args->mem_type->shared) { - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("MADV_DONTNEED"); } else { /* * NOTE: ignore retval because private-hugetlb doesn't yet * support swapping, so it could fail. */ - madvise(area_dst, page_size, MADV_PAGEOUT); + madvise(gopts->area_dst, gopts->page_size, MADV_PAGEOUT); } =20 /* Uffd-wp should persist even swapped out */ - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in zapped pte", with_event ? "missing" : "stall"); goto out; } =20 /* Unprotect; this tests swap pte modifications */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Fault in the page from disk */ - *area_dst =3D 2; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 2; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, nr_pages * page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts= ->page_size)) err("unregister failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_test(uffd_test_args_t *args) +static void uffd_wp_fork_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_wp_fork_test_common(args, false); + uffd_wp_fork_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_with_event_test(uffd_global_test_opts_t *gopts, u= ffd_test_args_t *args) { - uffd_wp_fork_test_common(args, true); + uffd_wp_fork_test_common(gopts, args, true); } =20 -static void uffd_wp_fork_pin_test_common(uffd_test_args_t *args, - bool with_event) +static void uffd_wp_fork_pin_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, bool with_event) { int pagemap_fd; pin_args pin_args =3D {}; =20 - if (uffd_register(uffd, area_dst, page_size, false, true, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->page_size, false, = true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); =20 /* * 1. First pin, then fork(). This tests fork() special path when * doing early CoW if the page is private. */ - if (pin_pages(&pin_args, area_dst, page_size)) { + if (pin_pages(&pin_args, gopts->area_dst, gopts->page_size)) { uffd_test_skip("Possibly CONFIG_GUP_TEST missing " "or unprivileged"); close(pagemap_fd); - uffd_unregister(uffd, area_dst, page_size); + uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size); return; } =20 - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in early CoW of fork()", with_event ? "missing" : "stall"); unpin_pages(&pin_args); @@ -527,49 +501,50 @@ static void uffd_wp_fork_pin_test_common(uffd_test_ar= gs_t *args, * 2. First fork(), then pin (in the child, where test_pin=3D=3Dtrue). * This tests COR, aka, page unsharing on private memories. */ - if (pagemap_test_fork(uffd, with_event, true)) { + if (pagemap_test_fork(gopts, with_event, true)) { uffd_test_fail("Detected %s uffd-wp bit when RO pin", with_event ? "missing" : "stall"); goto out; } uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("register failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_pin_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_test(uffd_global_test_opts_t *gopts, uffd_tes= t_args_t *args) { - uffd_wp_fork_pin_test_common(args, false); + uffd_wp_fork_pin_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_pin_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_with_event_test(uffd_global_test_opts_t *gopt= s, uffd_test_args_t *args) { - uffd_wp_fork_pin_test_common(args, true); + uffd_wp_fork_pin_test_common(gopts, args, true); } =20 -static void check_memory_contents(char *p) +static void check_memory_contents(uffd_global_test_opts_t *gopts, char *p) { unsigned long i, j; uint8_t expected_byte; =20 - for (i =3D 0; i < nr_pages; ++i) { + for (i =3D 0; i < gopts->nr_pages; ++i) { expected_byte =3D ~((uint8_t)(i % ((uint8_t)-1))); - for (j =3D 0; j < page_size; j++) { - uint8_t v =3D *(uint8_t *)(p + (i * page_size) + j); + for (j =3D 0; j < gopts->page_size; j++) { + uint8_t v =3D *(uint8_t *)(p + (i * gopts->page_size) + j); if (v !=3D expected_byte) err("unexpected page contents"); } } } =20 -static void uffd_minor_test_common(bool test_collapse, bool test_wp) +static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool te= st_collapse, bool test_wp) { unsigned long p; pthread_t uffd_mon; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 /* * NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing @@ -577,7 +552,7 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) */ assert(!(test_collapse && test_wp)); =20 - if (uffd_register(uffd, area_dst_alias, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst_alias, gopts->nr_pages * g= opts->page_size, /* NOTE! MADV_COLLAPSE may not work with uffd-wp */ false, test_wp, true)) err("register failure"); @@ -586,9 +561,9 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) * After registering with UFFD, populate the non-UFFD-registered side of * the shared mapping. This should *not* trigger any UFFD minor faults. */ - for (p =3D 0; p < nr_pages; ++p) - memset(area_dst + (p * page_size), p % ((uint8_t)-1), - page_size); + for (p =3D 0; p < gopts->nr_pages; ++p) + memset(gopts->area_dst + (p * gopts->page_size), p % ((uint8_t)-1), + gopts->page_size); =20 args.apply_wp =3D test_wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -600,50 +575,51 @@ static void uffd_minor_test_common(bool test_collapse= , bool test_wp) * fault. uffd_poll_thread will resolve the fault by bit-flipping the * page's contents, and then issuing a CONTINUE ioctl. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); =20 if (test_collapse) { - if (madvise(area_dst_alias, nr_pages * page_size, + if (madvise(gopts->area_dst_alias, gopts->nr_pages * gopts->page_size, MADV_COLLAPSE)) { /* It's fine to fail for this one... */ uffd_test_skip("MADV_COLLAPSE failed"); return; } =20 - uffd_test_ops->check_pmd_mapping(area_dst, - nr_pages * page_size / + uffd_test_ops->check_pmd_mapping(gopts, + gopts->area_dst, + gopts->nr_pages * gopts->page_size / read_pmd_pagesize()); /* * This won't cause uffd-fault - it purely just makes sure there * was no corruption. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); } =20 - if (args.missing_faults !=3D 0 || args.minor_faults !=3D nr_pages) + if (args.missing_faults !=3D 0 || args.minor_faults !=3D gopts->nr_pages) uffd_test_fail("stats check error"); else uffd_test_pass(); } =20 -void uffd_minor_test(uffd_test_args_t *args) +void uffd_minor_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *arg= s) { - uffd_minor_test_common(false, false); + uffd_minor_test_common(gopts, false, false); } =20 -void uffd_minor_wp_test(uffd_test_args_t *args) +void uffd_minor_wp_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *= args) { - uffd_minor_test_common(false, true); + uffd_minor_test_common(gopts, false, true); } =20 -void uffd_minor_collapse_test(uffd_test_args_t *args) +void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_minor_test_common(true, false); + uffd_minor_test_common(gopts, true, false); } =20 static sigjmp_buf jbuf, *sigbuf; @@ -678,7 +654,7 @@ static void sighndl(int sig, siginfo_t *siginfo, void *= ptr) * This also tests UFFD_FEATURE_EVENT_FORK event along with the signal * feature. Using monitor thread, verify no userfault events are generated. */ -static int faulting_process(int signal_test, bool wp) +static int faulting_process(uffd_global_test_opts_t *gopts, int signal_tes= t, bool wp) { unsigned long nr, i; unsigned long long count; @@ -687,7 +663,7 @@ static int faulting_process(int signal_test, bool wp) struct sigaction act; volatile unsigned long signalled =3D 0; =20 - split_nr_pages =3D (nr_pages + 1) / 2; + split_nr_pages =3D (gopts->nr_pages + 1) / 2; =20 if (signal_test) { sigbuf =3D &jbuf; @@ -701,7 +677,7 @@ static int faulting_process(int signal_test, bool wp) =20 for (nr =3D 0; nr < split_nr_pages; nr++) { volatile int steps =3D 1; - unsigned long offset =3D nr * page_size; + unsigned long offset =3D nr * gopts->page_size; =20 if (signal_test) { if (sigsetjmp(*sigbuf, 1) !=3D 0) { @@ -713,15 +689,15 @@ static int faulting_process(int signal_test, bool wp) if (steps =3D=3D 1) { /* This is a MISSING request */ steps++; - if (copy_page(uffd, offset, wp)) + if (copy_page(gopts, offset, wp)) signalled++; } else { /* This is a WP request */ assert(steps =3D=3D 2); - wp_range(uffd, - (__u64)area_dst + + wp_range(gopts->uffd, + (__u64)gopts->area_dst + offset, - page_size, false); + gopts->page_size, false); } } else { signalled++; @@ -730,51 +706,53 @@ static int faulting_process(int signal_test, bool wp) } } =20 - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 if (signal_test) return signalled !=3D split_nr_pages; =20 - area_dst =3D mremap(area_dst, nr_pages * page_size, nr_pages * page_size, - MREMAP_MAYMOVE | MREMAP_FIXED, area_src); - if (area_dst =3D=3D MAP_FAILED) + gopts->area_dst =3D mremap(gopts->area_dst, gopts->nr_pages * gopts->page= _size, + gopts->nr_pages * gopts->page_size, + MREMAP_MAYMOVE | MREMAP_FIXED, + gopts->area_src); + if (gopts->area_dst =3D=3D MAP_FAILED) err("mremap"); /* Reset area_src since we just clobbered it */ - area_src =3D NULL; + gopts->area_src =3D NULL; =20 - for (; nr < nr_pages; nr++) { - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) { + for (; nr < gopts->nr_pages; nr++) { + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) { err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); } /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - for (nr =3D 0; nr < nr_pages; nr++) - for (i =3D 0; i < page_size; i++) - if (*(area_dst + nr * page_size + i) !=3D 0) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + for (i =3D 0; i < gopts->page_size; i++) + if (*(gopts->area_dst + nr * gopts->page_size + i) !=3D 0) err("page %lu offset %lu is not zero", nr, i); =20 return 0; } =20 -static void uffd_sigbus_test_common(bool wp) +static void uffd_sigbus_test_common(uffd_global_test_opts_t *gopts, bool w= p) { unsigned long userfaults; pthread_t uffd_mon; @@ -782,25 +760,26 @@ static void uffd_sigbus_test_common(bool wp) int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 - if (faulting_process(1, wp)) + if (faulting_process(gopts, 1, wp)) err("faulting process failed"); =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 args.apply_wp =3D wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -808,12 +787,12 @@ static void uffd_sigbus_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(2, wp)); + exit(faulting_process(gopts, 2, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, (void **)&userfaults)) err("pthread_join()"); @@ -824,28 +803,29 @@ static void uffd_sigbus_test_common(bool wp) uffd_test_pass(); } =20 -static void uffd_sigbus_test(uffd_test_args_t *args) +static void uffd_sigbus_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_sigbus_test_common(false); + uffd_sigbus_test_common(gopts, false); } =20 -static void uffd_sigbus_wp_test(uffd_test_args_t *args) +static void uffd_sigbus_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_sigbus_test_common(true); + uffd_sigbus_test_common(gopts, true); } =20 -static void uffd_events_test_common(bool wp) +static void uffd_events_test_common(uffd_global_test_opts_t *gopts, bool w= p) { pthread_t uffd_mon; pid_t pid; int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); - if (uffd_register(uffd, area_dst, nr_pages * page_size, + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 @@ -853,7 +833,7 @@ static void uffd_events_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -861,39 +841,39 @@ static void uffd_events_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(0, wp)); + exit(faulting_process(gopts, 0, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (args.missing_faults !=3D nr_pages) + if (args.missing_faults !=3D gopts->nr_pages) uffd_test_fail("Fault counts wrong"); else uffd_test_pass(); } =20 -static void uffd_events_test(uffd_test_args_t *args) +static void uffd_events_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_events_test_common(false); + uffd_events_test_common(gopts, false); } =20 -static void uffd_events_wp_test(uffd_test_args_t *args) +static void uffd_events_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_events_test_common(true); + uffd_events_test_common(gopts, true); } =20 -static void retry_uffdio_zeropage(int ufd, +static void retry_uffdio_zeropage(uffd_global_test_opts_t *gopts, struct uffdio_zeropage *uffdio_zeropage) { - uffd_test_ops->alias_mapping(&uffdio_zeropage->range.start, + uffd_test_ops->alias_mapping(gopts, &uffdio_zeropage->range.start, uffdio_zeropage->range.len, 0); - if (ioctl(ufd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { + if (ioctl(gopts->uffd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { if (uffdio_zeropage->zeropage !=3D -EEXIST) err("UFFDIO_ZEROPAGE error: %"PRId64, (int64_t)uffdio_zeropage->zeropage); @@ -903,16 +883,16 @@ static void retry_uffdio_zeropage(int ufd, } } =20 -static bool do_uffdio_zeropage(int ufd, bool has_zeropage) +static bool do_uffdio_zeropage(uffd_global_test_opts_t *gopts, bool has_ze= ropage) { struct uffdio_zeropage uffdio_zeropage =3D { 0 }; int ret; __s64 res; =20 - uffdio_zeropage.range.start =3D (unsigned long) area_dst; - uffdio_zeropage.range.len =3D page_size; + uffdio_zeropage.range.start =3D (unsigned long) gopts->area_dst; + uffdio_zeropage.range.len =3D gopts->page_size; uffdio_zeropage.mode =3D 0; - ret =3D ioctl(ufd, UFFDIO_ZEROPAGE, &uffdio_zeropage); + ret =3D ioctl(gopts->uffd, UFFDIO_ZEROPAGE, &uffdio_zeropage); res =3D uffdio_zeropage.zeropage; if (ret) { /* real retval in ufdio_zeropage.zeropage */ @@ -921,10 +901,10 @@ static bool do_uffdio_zeropage(int ufd, bool has_zero= page) else if (res !=3D -EINVAL) err("UFFDIO_ZEROPAGE not -EINVAL"); } else if (has_zeropage) { - if (res !=3D page_size) + if (res !=3D gopts->page_size) err("UFFDIO_ZEROPAGE unexpected size"); else - retry_uffdio_zeropage(ufd, &uffdio_zeropage); + retry_uffdio_zeropage(gopts, &uffdio_zeropage); return true; } else err("UFFDIO_ZEROPAGE succeeded"); @@ -950,25 +930,27 @@ uffd_register_detect_zeropage(int uffd, void *addr, u= int64_t len) } =20 /* exercise UFFDIO_ZEROPAGE */ -static void uffd_zeropage_test(uffd_test_args_t *args) +static void uffd_zeropage_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *args) { bool has_zeropage; int i; =20 - has_zeropage =3D uffd_register_detect_zeropage(uffd, area_dst, page_size); - if (area_dst_alias) + has_zeropage =3D uffd_register_detect_zeropage(gopts->uffd, gopts->area_d= st, gopts->page_size); + if (gopts->area_dst_alias) /* Ignore the retval; we already have it */ - uffd_register_detect_zeropage(uffd, area_dst_alias, page_size); + uffd_register_detect_zeropage(gopts->uffd, gopts->area_dst_alias, gopts-= >page_size); =20 - if (do_uffdio_zeropage(uffd, has_zeropage)) - for (i =3D 0; i < page_size; i++) - if (area_dst[i] !=3D 0) + if (do_uffdio_zeropage(gopts, has_zeropage)) + for (i =3D 0; i < gopts->page_size; i++) + if (gopts->area_dst[i] !=3D 0) err("data non-zero at offset %d\n", i); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); =20 - if (area_dst_alias && uffd_unregister(uffd, area_dst_alias, page_size)) + if (gopts->area_dst_alias + && + uffd_unregister(gopts->uffd, gopts->area_dst_alias, gopts->page_size)) err("unregister"); =20 uffd_test_pass(); @@ -987,26 +969,26 @@ static void uffd_register_poison(int uffd, void *addr= , uint64_t len) err("registered area doesn't support COPY and POISON ioctls"); } =20 -static void do_uffdio_poison(int uffd, unsigned long offset) +static void do_uffdio_poison(uffd_global_test_opts_t *gopts, unsigned long= offset) { struct uffdio_poison uffdio_poison =3D { 0 }; int ret; __s64 res; =20 - uffdio_poison.range.start =3D (unsigned long) area_dst + offset; - uffdio_poison.range.len =3D page_size; + uffdio_poison.range.start =3D (unsigned long) gopts->area_dst + offset; + uffdio_poison.range.len =3D gopts->page_size; uffdio_poison.mode =3D 0; - ret =3D ioctl(uffd, UFFDIO_POISON, &uffdio_poison); + ret =3D ioctl(gopts->uffd, UFFDIO_POISON, &uffdio_poison); res =3D uffdio_poison.updated; =20 if (ret) err("UFFDIO_POISON error: %"PRId64, (int64_t)res); - else if (res !=3D page_size) + else if (res !=3D gopts->page_size) err("UFFDIO_POISON unexpected size: %"PRId64, (int64_t)res); } =20 static void uffd_poison_handle_fault( - struct uffd_msg *msg, struct uffd_args *args) + uffd_global_test_opts_t *gopts, struct uffd_msg *msg, struct uffd_args *a= rgs) { unsigned long offset; =20 @@ -1017,17 +999,17 @@ static void uffd_poison_handle_fault( (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; + offset &=3D ~(gopts->page_size-1); =20 /* Odd pages -> copy zeroed page; even pages -> poison. */ - if (offset & page_size) - copy_page(uffd, offset, false); + if (offset & gopts->page_size) + copy_page(gopts, offset, false); else - do_uffdio_poison(uffd, offset); + do_uffdio_poison(gopts, offset); } =20 -static void uffd_poison_test(uffd_test_args_t *targs) +static void uffd_poison_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *targs) { pthread_t uffd_mon; char c; @@ -1036,10 +1018,12 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) unsigned long nr_sigbus =3D 0; unsigned long nr; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + args.gopts =3D gopts; =20 - uffd_register_poison(uffd, area_dst, nr_pages * page_size); - memset(area_src, 0, nr_pages * page_size); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + + uffd_register_poison(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopt= s->page_size); + memset(gopts->area_src, 0, gopts->nr_pages * gopts->page_size); =20 args.handle_fault =3D uffd_poison_handle_fault; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -1051,9 +1035,9 @@ static void uffd_poison_test(uffd_test_args_t *targs) if (sigaction(SIGBUS, &act, 0)) err("sigaction"); =20 - for (nr =3D 0; nr < nr_pages; ++nr) { - unsigned long offset =3D nr * page_size; - const char *bytes =3D (const char *) area_dst + offset; + for (nr =3D 0; nr < gopts->nr_pages; ++nr) { + unsigned long offset =3D nr * gopts->page_size; + const char *bytes =3D (const char *) gopts->area_dst + offset; const char *i; =20 if (sigsetjmp(*sigbuf, 1)) { @@ -1066,28 +1050,30 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) continue; } =20 - for (i =3D bytes; i < bytes + page_size; ++i) { + for (i =3D bytes; i < bytes + gopts->page_size; ++i) { if (*i) err("nonzero byte in area_dst (%p) at %p: %u", - area_dst, i, *i); + gopts->area_dst, i, *i); } } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (nr_sigbus !=3D nr_pages / 2) + if (nr_sigbus !=3D gopts->nr_pages / 2) err("expected to receive %lu SIGBUS, actually received %lu", - nr_pages / 2, nr_sigbus); + gopts->nr_pages / 2, nr_sigbus); =20 uffd_test_pass(); } =20 static void -uffd_move_handle_fault_common(struct uffd_msg *msg, struct uffd_args *args, - unsigned long len) +uffd_move_handle_fault_common(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args, + unsigned long len) { unsigned long offset; =20 @@ -1098,28 +1084,31 @@ uffd_move_handle_fault_common(struct uffd_msg *msg,= struct uffd_args *args, (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR | UFFD_PAGEFAULT_= FLAG_WRITE)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; offset &=3D ~(len-1); =20 - if (move_page(uffd, offset, len)) + if (move_page(gopts, offset, len)) args->missing_faults++; } =20 -static void uffd_move_handle_fault(struct uffd_msg *msg, +static void uffd_move_handle_fault(uffd_global_test_opts_t *gopts, struct = uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, page_size); + uffd_move_handle_fault_common(gopts, msg, args, gopts->page_size); } =20 -static void uffd_move_pmd_handle_fault(struct uffd_msg *msg, +static void uffd_move_pmd_handle_fault(uffd_global_test_opts_t *gopts, str= uct uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, read_pmd_pagesize()); + uffd_move_handle_fault_common(gopts, msg, args, read_pmd_pagesize()); } =20 static void -uffd_move_test_common(uffd_test_args_t *targs, unsigned long chunk_size, - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args)) +uffd_move_test_common( + uffd_global_test_opts_t *gopts, uffd_test_args_t *targs, + unsigned long chunk_size, void (*handle_fault)(struct uffd_global_test_o= pts *gopts, + struct uffd_msg *msg, struct uffd_args *args) +) { unsigned long nr; pthread_t uffd_mon; @@ -1131,11 +1120,13 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, unsigned long src_offs =3D 0; unsigned long dst_offs =3D 0; =20 + args.gopts =3D gopts; + /* Prevent source pages from being mapped more than once */ - if (madvise(area_src, nr_pages * page_size, MADV_DONTFORK)) + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DON= TFORK)) err("madvise(MADV_DONTFORK) failure"); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("register failure"); =20 @@ -1143,22 +1134,22 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - step_size =3D chunk_size / page_size; - step_count =3D nr_pages / step_size; + step_size =3D chunk_size / gopts->page_size; + step_count =3D gopts->nr_pages / step_size; =20 - if (chunk_size > page_size) { - char *aligned_src =3D ALIGN_UP(area_src, chunk_size); - char *aligned_dst =3D ALIGN_UP(area_dst, chunk_size); + if (chunk_size > gopts->page_size) { + char *aligned_src =3D ALIGN_UP(gopts->area_src, chunk_size); + char *aligned_dst =3D ALIGN_UP(gopts->area_dst, chunk_size); =20 - if (aligned_src !=3D area_src || aligned_dst !=3D area_dst) { - src_offs =3D (aligned_src - area_src) / page_size; - dst_offs =3D (aligned_dst - area_dst) / page_size; + if (aligned_src !=3D gopts->area_src || aligned_dst !=3D gopts->area_dst= ) { + src_offs =3D (aligned_src - gopts->area_src) / gopts->page_size; + dst_offs =3D (aligned_dst - gopts->area_dst) / gopts->page_size; step_count--; } - orig_area_src =3D area_src; - orig_area_dst =3D area_dst; - area_src =3D aligned_src; - area_dst =3D aligned_dst; + orig_area_src =3D gopts->area_src; + orig_area_dst =3D gopts->area_dst; + gopts->area_src =3D aligned_src; + gopts->area_dst =3D aligned_dst; } =20 /* @@ -1172,34 +1163,34 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, =20 /* Check area_src content */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); - if (count !=3D count_verify[src_offs + nr + i]) + count =3D *area_count(gopts->area_src, nr + i, gopts); + if (count !=3D gopts->count_verify[src_offs + nr + i]) err("nr %lu source memory invalid %llu %llu\n", - nr + i, count, count_verify[src_offs + nr + i]); + nr + i, count, gopts->count_verify[src_offs + nr + i]); } =20 /* Faulting into area_dst should move the page or the huge page */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_dst, nr + i); - if (count !=3D count_verify[dst_offs + nr + i]) + count =3D *area_count(gopts->area_dst, nr + i, gopts); + if (count !=3D gopts->count_verify[dst_offs + nr + i]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[dst_offs + nr + i]); + nr, count, gopts->count_verify[dst_offs + nr + i]); } =20 /* Re-check area_src content which should be empty */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); + count =3D *area_count(gopts->area_src, nr + i, gopts); if (count !=3D 0) err("nr %lu move failed %llu %llu\n", - nr, count, count_verify[src_offs + nr + i]); + nr, count, gopts->count_verify[src_offs + nr + i]); } } - if (chunk_size > page_size) { - area_src =3D orig_area_src; - area_dst =3D orig_area_dst; + if (chunk_size > gopts->page_size) { + gopts->area_src =3D orig_area_src; + gopts->area_dst =3D orig_area_dst; } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); @@ -1210,31 +1201,31 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, uffd_test_pass(); } =20 -static void uffd_move_test(uffd_test_args_t *targs) +static void uffd_move_test(uffd_global_test_opts_t *gopts, uffd_test_args_= t *targs) { - uffd_move_test_common(targs, page_size, uffd_move_handle_fault); + uffd_move_test_common(gopts, targs, gopts->page_size, uffd_move_handle_fa= ult); } =20 -static void uffd_move_pmd_test(uffd_test_args_t *targs) +static void uffd_move_pmd_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) err("madvise(MADV_HUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static void uffd_move_pmd_split_test(uffd_test_args_t *targs) +static void uffd_move_pmd_split_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) err("madvise(MADV_NOHUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static int prevent_hugepages(const char **errmsg) +static int prevent_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_NOHUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) { /* Ignore only if CONFIG_TRANSPARENT_HUGEPAGE=3Dn */ if (errno !=3D EINVAL) { if (errmsg) @@ -1245,10 +1236,10 @@ static int prevent_hugepages(const char **errmsg) return 0; } =20 -static int request_hugepages(const char **errmsg) +static int request_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_HUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) { if (errmsg) { *errmsg =3D (errno =3D=3D EINVAL) ? "CONFIG_TRANSPARENT_HUGEPAGE is not set" : @@ -1272,13 +1263,17 @@ struct uffd_test_case_ops uffd_move_test_pmd_case_o= ps =3D { * Note that _UFFDIO_ZEROPAGE is tested separately in the zeropage test. */ static void -do_register_ioctls_test(uffd_test_args_t *args, bool miss, bool wp, bool m= inor) +do_register_ioctls_test(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, + bool miss, + bool wp, + bool minor) { uint64_t ioctls =3D 0, expected =3D BIT_ULL(_UFFDIO_WAKE); mem_type_t *mem_type =3D args->mem_type; int ret; =20 - ret =3D uffd_register_with_ioctls(uffd, area_dst, page_size, + ret =3D uffd_register_with_ioctls(gopts->uffd, gopts->area_dst, gopts->pa= ge_size, miss, wp, minor, &ioctls); =20 /* @@ -1309,18 +1304,18 @@ do_register_ioctls_test(uffd_test_args_t *args, boo= l miss, bool wp, bool minor) "(miss=3D%d, wp=3D%d, minor=3D%d): expected=3D0x%"PRIx64", " "returned=3D0x%"PRIx64, miss, wp, minor, expected, ioctls); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); } =20 -static void uffd_register_ioctls_test(uffd_test_args_t *args) +static void uffd_register_ioctls_test(uffd_global_test_opts_t *gopts, uffd= _test_args_t *args) { int miss, wp, minor; =20 for (miss =3D 0; miss <=3D 1; miss++) for (wp =3D 0; wp <=3D 1; wp++) for (minor =3D 0; minor <=3D 1; minor++) - do_register_ioctls_test(args, miss, wp, minor); + do_register_ioctls_test(gopts, args, miss, wp, minor); =20 uffd_test_pass(); } @@ -1532,6 +1527,27 @@ int main(int argc, char *argv[]) } for (j =3D 0; j < n_mems; j++) { mem_type =3D &mem_types[j]; + + // Initialize global test options + uffd_global_test_opts_t gopts; + + gopts.map_shared =3D mem_type->shared; + uffd_test_ops =3D mem_type->mem_ops; + uffd_test_case_ops =3D test->test_case_ops; + + if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) + gopts.page_size =3D default_huge_page_size(); + else + gopts.page_size =3D psize(); + + /* Ensure we have at least 2 pages */ + gopts.nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, gopts.page_size * 2) / gopts= .page_size; + /* TODO: remove this global var.. it's so ugly */ + gopts.nr_parallel =3D 1; + + /* Initialize test arguments */ + args.mem_type =3D mem_type; + if (!(test->mem_targets & mem_type->mem_flag)) continue; =20 @@ -1546,13 +1562,12 @@ int main(int argc, char *argv[]) uffd_test_skip("feature missing"); continue; } - if (uffd_setup_environment(&args, test, mem_type, - &errmsg)) { + if (uffd_test_ctx_init(&gopts, test->uffd_feature_required, &errmsg)) { uffd_test_skip(errmsg); continue; } - test->uffd_fn(&args); - uffd_test_ctx_clear(); + test->uffd_fn(&gopts, &args); + uffd_test_ctx_clear(&gopts); } } =20 diff --git a/tools/testing/selftests/mm/uffd-wp-mremap.c b/tools/testing/se= lftests/mm/uffd-wp-mremap.c index c2ba7d46c7b4..a24f35728ac3 100644 --- a/tools/testing/selftests/mm/uffd-wp-mremap.c +++ b/tools/testing/selftests/mm/uffd-wp-mremap.c @@ -157,7 +157,11 @@ static bool range_is_swapped(void *addr, size_t size) return true; } =20 -static void test_one_folio(size_t size, bool private, bool swapout, bool h= ugetlb) +static void test_one_folio(uffd_global_test_opts_t *gopts, + size_t size, + bool private, + bool swapout, + bool hugetlb) { struct uffdio_writeprotect wp_prms; uint64_t features =3D 0; @@ -181,21 +185,21 @@ static void test_one_folio(size_t size, bool private,= bool swapout, bool hugetlb } =20 /* Register range for uffd-wp. */ - if (userfaultfd_open(&features)) { + if (userfaultfd_open(gopts, &features)) { if (errno =3D=3D ENOENT) ksft_test_result_skip("userfaultfd not available\n"); else ksft_test_result_fail("userfaultfd_open() failed\n"); goto out; } - if (uffd_register(uffd, mem, size, false, true, false)) { + if (uffd_register(gopts->uffd, mem, size, false, true, false)) { ksft_test_result_fail("uffd_register() failed\n"); goto out; } wp_prms.mode =3D UFFDIO_WRITEPROTECT_MODE_WP; wp_prms.range.start =3D (uintptr_t)mem; wp_prms.range.len =3D size; - if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { + if (ioctl(gopts->uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { ksft_test_result_fail("ioctl(UFFDIO_WRITEPROTECT) failed\n"); goto out; } @@ -242,9 +246,9 @@ static void test_one_folio(size_t size, bool private, b= ool swapout, bool hugetlb out: if (mem) munmap(mem, size); - if (uffd >=3D 0) { - close(uffd); - uffd =3D -1; + if (gopts->uffd >=3D 0) { + close(gopts->uffd); + gopts->uffd =3D -1; } } =20 @@ -336,6 +340,7 @@ static const struct testcase testcases[] =3D { =20 int main(int argc, char **argv) { + uffd_global_test_opts_t gopts; struct thp_settings settings; int i, j, plan =3D 0; =20 @@ -367,8 +372,8 @@ int main(int argc, char **argv) const struct testcase *tc =3D &testcases[i]; =20 for (j =3D 0; j < *tc->nr_sizes; j++) - test_one_folio(tc->sizes[j], tc->private, tc->swapout, - tc->hugetlb); + test_one_folio(&gopts, tc->sizes[j], tc->private, + tc->swapout, tc->hugetlb); } =20 /* If THP is supported, restore original THP settings. */ --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06AAC1C5F18; Thu, 1 May 2025 16:40:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117626; cv=none; b=mjFaGXLTlukvLAthnGMO6litNSIgbNkWM+rj6eqEfRvdLjQeP0/mqztav5uqrTy6248YcTSjHv6gSf8+ed8rVBGt96a0pO1R1wzW1aQOAtwKWGhn0/KevFMX5lucGnnAcu//68CsgjUAo3TxTFZCR91+9M7pKmrVHmoyYeziHmU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117626; c=relaxed/simple; bh=upBb57xsILROHN0M91N/F4C8WOak3ply/B1BlbEfj+Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=p8QoIY6Qgcu1gbM14s+VNG7j9iOnz6zHH01VmkkYdgG5Xz/StTlHGnC8X3P3IG0vBMoXxn/z24E5BY9s4sBbj2p9kFSswBYQze22yDIGfbDdMXsKIjXKgG6oq3lg18gRjTAtIBfLNJQsOmZT9r2pldli0WNGlHEIL3eMzw57Xq4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LCEab/fa; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LCEab/fa" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2243803b776so20199385ad.0; Thu, 01 May 2025 09:40:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746117623; x=1746722423; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2B29N3ql8Km5YaO47sgwTzjZzJxM9wgtF00Tfw1QDRE=; b=LCEab/faHBNQARt4OWAM4uIMisCe+kGLOi0CXckQauVlgNv44m6pi6YkoikWcdBVYR jSHwhLlLXtPbCVcDMsAt2s7CCejSK5z2Z9r8xYtlKVGSiIIGcq0Se/l+nrIw3lK7SKt0 0JIjxs22/5kguZexEaWNeK7O5sJhFN6e3ZatJOVq+Hcry8gpqKpDivNxhhLcAbd2bZgh 0fryyzLEHG78e/V8TrL37c2T/B1yNPl9ur5X+0Boi32SquKx5tjiKnWBJr3Uhp4L4LqS P0e7K401xkwbdmxOmi7hacxhFDvDR/MC43VVC13BwQRE/iHe+5Yk9mBWktRRbMNH7U/3 l6gQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746117623; x=1746722423; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2B29N3ql8Km5YaO47sgwTzjZzJxM9wgtF00Tfw1QDRE=; b=dfcEheotwPsyLabnIwSr34JKJlnrLDmhNYMpIe4eHmsMmkS6GCsJ6b84CtBJkziY0V tMqv2JFdZRZ0l6fMDiC4CHNx5llI9lUKkZmlCHCaFALZQK0MXS4Y9ci5PuZsTalZVt76 Qych69IcHDt9/7/y7xBiCYDCs7B0YWIWz5uYsTsF1/XyggOq0c/B/UrJ25HXnrwR/eay cYdPXVYDCzRBJrWqm57mJaebXH5fjdocSkNmZYUnJcbAx7ll+Nl0fIuCO5LiPsFq/f7Q 5d0U8hIVRp4iDUNBtsS2BR5kiaI+2A2DD+BzLxZFQ0auqsVj/dVbN+VbMB5CgrZJTmEQ 2UkA== X-Forwarded-Encrypted: i=1; AJvYcCV0O5J+e0GtFJGxHiBc57VGGVqhWt2ZbLHGRLnSVZ5rl9Gs3//JXC67aGW0QLwsLnkn+1zcDhWjHGJvCO4Y8Vz/@vger.kernel.org, AJvYcCXpHOAMwwg60HsKtFmTLCol/Bn9fv0us/zP97yrmnL616XE5DLw1PEvYkehs0RaMpTu9kKBwZgzQejXdhE=@vger.kernel.org X-Gm-Message-State: AOJu0YyUXUh8zG2h/GfVZC+7eaw8iv1ZJGtJ1XW/Bgcm2hY4p8NMJzme tAQilWfy1Kewjzut3lq2TddP1zsLxIb6Yb9dhsTx+ZhDNEm+mR8= X-Gm-Gg: ASbGncvJb1v59ysoTxk3fZmPZ5x/1j6aO1f8V51Amr/ygJ68zPWM2rRQ7P7Oj8alFCx N8YZi+lz3mbMGioX2277rL0g1aBibzbc+CC8SV5nq5ueif5N+LAUMrQIIuVDkPcEgfvSnnO0fGP OfyIw1NHd1NNPRLrggOx0P6JzXqnrGmQN+iUKcEwwa/Z3CzAvh+YQiGviksCtSweU6V8fHYIvX7 M4nUgYnTDq5+4jsYTwHO+CuaRjN29RdGpIsua/dQI1D7Pyvg74Kjq10Qz48dvvT/a7HcrmNRBfo EztjIy306rX4irS38Ylwua+OPpjwGZW4aKGz4cTMvDmrSszXyDhElAujY7ZZ X-Google-Smtp-Source: AGHT+IHSslwowYhrXL1lkoPZdSSfH0o+GOL3w4k/egfH0gBxQRgNQeDx+vyOgVorJChvtRUMjaup5Q== X-Received: by 2002:a17:903:1986:b0:21f:1549:a55a with SMTP id d9443c01a7336-22e040a34ccmr54632085ad.1.1746117622936; Thu, 01 May 2025 09:40:22 -0700 (PDT) Received: from debian.ujwal.com ([223.185.132.49]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22e0bc6e678sm8797525ad.132.2025.05.01.09.40.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 May 2025 09:40:22 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, peterx@redhat.com, shuah@kernel.org Cc: linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Ujwal Kundur Subject: [PATCH 1/4] selftests/mm/uffd: Refactor non-composite global vars into struct Date: Thu, 1 May 2025 22:08:24 +0530 Message-Id: <20250501163827.2598-2-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch deals with the common uffd routines, refactoring macros and non-composite global variable definitions into a struct that is defined at the start of a test and is passed around instead of relying on global vars. Signed-off-by: Ujwal Kundur Reviewed-by: Brendan Jackman Tested-by: Brendan Jackman --- tools/testing/selftests/mm/uffd-common.c | 269 ++++++++++++----------- tools/testing/selftests/mm/uffd-common.h | 78 +++---- 2 files changed, 186 insertions(+), 161 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-common.c b/tools/testing/selft= ests/mm/uffd-common.c index a37088a23ffe..cc8e2079e207 100644 --- a/tools/testing/selftests/mm/uffd-common.c +++ b/tools/testing/selftests/mm/uffd-common.c @@ -7,18 +7,31 @@ =20 #include "uffd-common.h" =20 -#define BASE_PMD_ADDR ((void *)(1UL << 30)) - -volatile bool test_uffdio_copy_eexist =3D true; -unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; -int uffd =3D -1, uffd_flags, finished, *pipefd, test_type; -bool map_shared; -bool test_uffdio_wp =3D true; -unsigned long long *count_verify; uffd_test_ops_t *uffd_test_ops; uffd_test_case_ops_t *uffd_test_case_ops; -atomic_bool ready_for_fork; + +#define BASE_PMD_ADDR ((void *)(1UL << 30)) + +/* pthread_mutex_t starts at page offset 0 */ +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts) +{ + return (pthread_mutex_t *) (area + nr * gopts->page_size); +} + +/* + * count is placed in the page after pthread_mutex_t naturally aligned + * to avoid non alignment faults on non-x86 archs. + */ +volatile unsigned long long *area_count( + char *area, unsigned long nr, + uffd_global_test_opts_t *gopts) +{ + return (volatile unsigned long long *) + ((unsigned long) (area + nr * gopts->page_size + + sizeof(pthread_mutex_t) + + sizeof(unsigned long long) - 1) & + ~(unsigned long)(sizeof(unsigned long long) - 1)); +} =20 static int uffd_mem_fd_create(off_t mem_size, bool hugetlb) { @@ -40,15 +53,15 @@ static int uffd_mem_fd_create(off_t mem_size, bool huge= tlb) return mem_fd; } =20 -static void anon_release_pages(char *rel_area) +static void anon_release_pages(uffd_global_test_opts_t *gopts, char *rel_a= rea) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } =20 -static int anon_allocate_area(void **alloc_area, bool is_src) +static int anon_allocate_area(uffd_global_test_opts_t *gopts, void **alloc= _area, bool is_src) { - *alloc_area =3D mmap(NULL, nr_pages * page_size, PROT_READ | PROT_WRITE, + *alloc_area =3D mmap(NULL, gopts->nr_pages * gopts->page_size, PROT_READ = | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); if (*alloc_area =3D=3D MAP_FAILED) { *alloc_area =3D NULL; @@ -57,31 +70,33 @@ static int anon_allocate_area(void **alloc_area, bool i= s_src) return 0; } =20 -static void noop_alias_mapping(__u64 *start, size_t len, unsigned long off= set) +static void noop_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, size_t len, + unsigned long offset) { } =20 -static void hugetlb_release_pages(char *rel_area) +static void hugetlb_release_pages(uffd_global_test_opts_t *gopts, char *re= l_area) { - if (!map_shared) { - if (madvise(rel_area, nr_pages * page_size, MADV_DONTNEED)) + if (!gopts->map_shared) { + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); } else { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } } =20 -static int hugetlb_allocate_area(void **alloc_area, bool is_src) +static int hugetlb_allocate_area(uffd_global_test_opts_t *gopts, void **al= loc_area, bool is_src) { - off_t size =3D nr_pages * page_size; + off_t size =3D gopts->nr_pages * gopts->page_size; off_t offset =3D is_src ? 0 : size; void *area_alias =3D NULL; char **alloc_area_alias; int mem_fd =3D uffd_mem_fd_create(size * 2, true); =20 *alloc_area =3D mmap(NULL, size, PROT_READ | PROT_WRITE, - (map_shared ? MAP_SHARED : MAP_PRIVATE) | + (gopts->map_shared ? MAP_SHARED : MAP_PRIVATE) | (is_src ? 0 : MAP_NORESERVE), mem_fd, offset); if (*alloc_area =3D=3D MAP_FAILED) { @@ -89,7 +104,7 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) return -errno; } =20 - if (map_shared) { + if (gopts->map_shared) { area_alias =3D mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, mem_fd, offset); if (area_alias =3D=3D MAP_FAILED) @@ -97,9 +112,9 @@ static int hugetlb_allocate_area(void **alloc_area, bool= is_src) } =20 if (is_src) { - alloc_area_alias =3D &area_src_alias; + alloc_area_alias =3D &gopts->area_src_alias; } else { - alloc_area_alias =3D &area_dst_alias; + alloc_area_alias =3D &gopts->area_dst_alias; } if (area_alias) *alloc_area_alias =3D area_alias; @@ -108,24 +123,26 @@ static int hugetlb_allocate_area(void **alloc_area, b= ool is_src) return 0; } =20 -static void hugetlb_alias_mapping(__u64 *start, size_t len, unsigned long = offset) +static void hugetlb_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, size_t len, + unsigned long offset) { - if (!map_shared) + if (!gopts->map_shared) return; =20 - *start =3D (unsigned long) area_dst_alias + offset; + *start =3D (unsigned long) gopts->area_dst_alias + offset; } =20 -static void shmem_release_pages(char *rel_area) +static void shmem_release_pages(uffd_global_test_opts_t *gopts, char *rel_= area) { - if (madvise(rel_area, nr_pages * page_size, MADV_REMOVE)) + if (madvise(rel_area, gopts->nr_pages * gopts->page_size, MADV_REMOVE)) err("madvise(MADV_REMOVE) failed"); } =20 -static int shmem_allocate_area(void **alloc_area, bool is_src) +static int shmem_allocate_area(uffd_global_test_opts_t *gopts, void **allo= c_area, bool is_src) { void *area_alias =3D NULL; - size_t bytes =3D nr_pages * page_size, hpage_size =3D read_pmd_pagesize(); + size_t bytes =3D gopts->nr_pages * gopts->page_size, hpage_size =3D read_= pmd_pagesize(); unsigned long offset =3D is_src ? 0 : bytes; char *p =3D NULL, *p_alias =3D NULL; int mem_fd =3D uffd_mem_fd_create(bytes * 2, false); @@ -159,22 +176,24 @@ static int shmem_allocate_area(void **alloc_area, boo= l is_src) err("mmap of anonymous memory failed at %p", p_alias); =20 if (is_src) - area_src_alias =3D area_alias; + gopts->area_src_alias =3D area_alias; else - area_dst_alias =3D area_alias; + gopts->area_dst_alias =3D area_alias; =20 close(mem_fd); return 0; } =20 -static void shmem_alias_mapping(__u64 *start, size_t len, unsigned long of= fset) +static void shmem_alias_mapping( + uffd_global_test_opts_t *gopts, __u64 *start, + size_t len, unsigned long offset) { - *start =3D (unsigned long)area_dst_alias + offset; + *start =3D (unsigned long)gopts->area_dst_alias + offset; } =20 -static void shmem_check_pmd_mapping(void *p, int expect_nr_hpages) +static void shmem_check_pmd_mapping(uffd_global_test_opts_t *gopts, void *= p, int expect_nr_hpages) { - if (!check_huge_shmem(area_dst_alias, expect_nr_hpages, + if (!check_huge_shmem(gopts->area_dst_alias, expect_nr_hpages, read_pmd_pagesize())) err("Did not find expected %d number of hugepages", expect_nr_hpages); @@ -234,18 +253,18 @@ void uffd_stats_report(struct uffd_args *args, int n_= cpus) printf("\n"); } =20 -int userfaultfd_open(uint64_t *features) +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features) { struct uffdio_api uffdio_api; =20 - uffd =3D uffd_open(UFFD_FLAGS); - if (uffd < 0) + gopts->uffd =3D uffd_open(UFFD_FLAGS); + if (gopts->uffd < 0) return -1; - uffd_flags =3D fcntl(uffd, F_GETFD, NULL); + gopts->uffd_flags =3D fcntl(gopts->uffd, F_GETFD, NULL); =20 uffdio_api.api =3D UFFD_API; uffdio_api.features =3D *features; - if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + if (ioctl(gopts->uffd, UFFDIO_API, &uffdio_api)) /* Probably lack of CAP_PTRACE? */ return -1; if (uffdio_api.api !=3D UFFD_API) @@ -255,59 +274,59 @@ int userfaultfd_open(uint64_t *features) return 0; } =20 -static inline void munmap_area(void **area) +static inline void munmap_area(uffd_global_test_opts_t *gopts, void **area) { if (*area) - if (munmap(*area, nr_pages * page_size)) + if (munmap(*area, gopts->nr_pages * gopts->page_size)) err("munmap"); =20 *area =3D NULL; } =20 -void uffd_test_ctx_clear(void) +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts) { size_t i; =20 - if (pipefd) { - for (i =3D 0; i < nr_parallel * 2; ++i) { - if (close(pipefd[i])) + if (gopts->pipefd) { + for (i =3D 0; i < gopts->nr_parallel * 2; ++i) { + if (close(gopts->pipefd[i])) err("close pipefd"); } - free(pipefd); - pipefd =3D NULL; + free(gopts->pipefd); + gopts->pipefd =3D NULL; } =20 - if (count_verify) { - free(count_verify); - count_verify =3D NULL; + if (gopts->count_verify) { + free(gopts->count_verify); + gopts->count_verify =3D NULL; } =20 - if (uffd !=3D -1) { - if (close(uffd)) + if (gopts->uffd !=3D -1) { + if (close(gopts->uffd)) err("close uffd"); - uffd =3D -1; + gopts->uffd =3D -1; } =20 - munmap_area((void **)&area_src); - munmap_area((void **)&area_src_alias); - munmap_area((void **)&area_dst); - munmap_area((void **)&area_dst_alias); - munmap_area((void **)&area_remap); + munmap_area(gopts, (void **)&gopts->area_src); + munmap_area(gopts, (void **)&gopts->area_src_alias); + munmap_area(gopts, (void **)&gopts->area_dst); + munmap_area(gopts, (void **)&gopts->area_dst_alias); + munmap_area(gopts, (void **)&gopts->area_remap); } =20 -int uffd_test_ctx_init(uint64_t features, const char **errmsg) +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg) { unsigned long nr, cpu; int ret; =20 if (uffd_test_case_ops && uffd_test_case_ops->pre_alloc) { - ret =3D uffd_test_case_ops->pre_alloc(errmsg); + ret =3D uffd_test_case_ops->pre_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D uffd_test_ops->allocate_area((void **)&area_src, true); - ret |=3D uffd_test_ops->allocate_area((void **)&area_dst, false); + ret =3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_src, t= rue); + ret |=3D uffd_test_ops->allocate_area(gopts, (void **) &gopts->area_dst, = false); if (ret) { if (errmsg) *errmsg =3D "memory allocation failed"; @@ -315,26 +334,26 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) } =20 if (uffd_test_case_ops && uffd_test_case_ops->post_alloc) { - ret =3D uffd_test_case_ops->post_alloc(errmsg); + ret =3D uffd_test_case_ops->post_alloc(gopts, errmsg); if (ret) return ret; } =20 - ret =3D userfaultfd_open(&features); + ret =3D userfaultfd_open(gopts, &features); if (ret) { if (errmsg) *errmsg =3D "possible lack of privilege"; return ret; } =20 - count_verify =3D malloc(nr_pages * sizeof(unsigned long long)); - if (!count_verify) + gopts->count_verify =3D malloc(gopts->nr_pages * sizeof(unsigned long lon= g)); + if (!gopts->count_verify) err("count_verify"); =20 - for (nr =3D 0; nr < nr_pages; nr++) { - *area_mutex(area_src, nr) =3D + for (nr =3D 0; nr < gopts->nr_pages; nr++) { + *area_mutex(gopts->area_src, nr, gopts) =3D (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER; - count_verify[nr] =3D *area_count(area_src, nr) =3D 1; + gopts->count_verify[nr] =3D *area_count(gopts->area_src, nr, gopts) =3D = 1; /* * In the transition between 255 to 256, powerpc will * read out of order in my_bcmp and see both bytes as @@ -342,7 +361,7 @@ int uffd_test_ctx_init(uint64_t features, const char **= errmsg) * after the count, to avoid my_bcmp to trigger false * positives. */ - *(area_count(area_src, nr) + 1) =3D 1; + *(area_count(gopts->area_src, nr, gopts) + 1) =3D 1; } =20 /* @@ -363,13 +382,13 @@ int uffd_test_ctx_init(uint64_t features, const char = **errmsg) * proactively split the thp and drop any accidentally initialized * pages within area_dst. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - pipefd =3D malloc(sizeof(int) * nr_parallel * 2); - if (!pipefd) + gopts->pipefd =3D malloc(sizeof(int) * gopts->nr_parallel * 2); + if (!gopts->pipefd) err("pipefd"); - for (cpu =3D 0; cpu < nr_parallel; cpu++) - if (pipe2(&pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) + if (pipe2(&gopts->pipefd[cpu * 2], O_CLOEXEC | O_NONBLOCK)) err("pipe"); =20 return 0; @@ -416,9 +435,9 @@ static void continue_range(int ufd, __u64 start, __u64 = len, bool wp) ret, (int64_t) req.mapped); } =20 -int uffd_read_msg(int ufd, struct uffd_msg *msg) +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg) { - int ret =3D read(uffd, msg, sizeof(*msg)); + int ret =3D read(gopts->uffd, msg, sizeof(*msg)); =20 if (ret !=3D sizeof(*msg)) { if (ret < 0) { @@ -433,7 +452,9 @@ int uffd_read_msg(int ufd, struct uffd_msg *msg) return 0; } =20 -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args) +void uffd_handle_page_fault( + uffd_global_test_opts_t *gopts, struct uffd_msg *msg, + struct uffd_args *args) { unsigned long offset; =20 @@ -442,7 +463,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) =20 if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WP) { /* Write protect page faults */ - wp_range(uffd, msg->arg.pagefault.address, page_size, false); + wp_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, fals= e); args->wp_faults++; } else if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR) { uint8_t *area; @@ -460,12 +481,12 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) * (UFFD-registered). */ =20 - area =3D (uint8_t *)(area_dst + + area =3D (uint8_t *)(gopts->area_dst + ((char *)msg->arg.pagefault.address - - area_dst_alias)); - for (b =3D 0; b < page_size; ++b) + gopts->area_dst_alias)); + for (b =3D 0; b < gopts->page_size; ++b) area[b] =3D ~area[b]; - continue_range(uffd, msg->arg.pagefault.address, page_size, + continue_range(gopts->uffd, msg->arg.pagefault.address, gopts->page_size, args->apply_wp); args->minor_faults++; } else { @@ -493,10 +514,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, str= uct uffd_args *args) if (msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_WRITE) err("unexpected write fault"); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->ar= ea_dst; + offset &=3D ~(gopts->page_size-1); =20 - if (copy_page(uffd, offset, args->apply_wp)) + if (copy_page(gopts, offset, args->apply_wp)) args->missing_faults++; } } @@ -504,6 +525,7 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struc= t uffd_args *args) void *uffd_poll_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; unsigned long cpu =3D args->cpu; struct pollfd pollfd[2]; struct uffd_msg msg; @@ -514,12 +536,12 @@ void *uffd_poll_thread(void *arg) if (!args->handle_fault) args->handle_fault =3D uffd_handle_page_fault; =20 - pollfd[0].fd =3D uffd; + pollfd[0].fd =3D gopts->uffd; pollfd[0].events =3D POLLIN; - pollfd[1].fd =3D pipefd[cpu*2]; + pollfd[1].fd =3D gopts->pipefd[cpu*2]; pollfd[1].events =3D POLLIN; =20 - ready_for_fork =3D true; + gopts->ready_for_fork =3D true; =20 for (;;) { ret =3D poll(pollfd, 2, -1); @@ -537,30 +559,30 @@ void *uffd_poll_thread(void *arg) } if (!(pollfd[0].revents & POLLIN)) err("pollfd[0].revents %d", pollfd[0].revents); - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; switch (msg.event) { default: err("unexpected msg event %u\n", msg.event); break; case UFFD_EVENT_PAGEFAULT: - args->handle_fault(&msg, args); + args->handle_fault(gopts, &msg, args); break; case UFFD_EVENT_FORK: - close(uffd); - uffd =3D msg.arg.fork.ufd; - pollfd[0].fd =3D uffd; + close(gopts->uffd); + gopts->uffd =3D msg.arg.fork.ufd; + pollfd[0].fd =3D gopts->uffd; break; case UFFD_EVENT_REMOVE: uffd_reg.range.start =3D msg.arg.remove.start; uffd_reg.range.len =3D msg.arg.remove.end - msg.arg.remove.start; - if (ioctl(uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) + if (ioctl(gopts->uffd, UFFDIO_UNREGISTER, &uffd_reg.range)) err("remove failure"); break; case UFFD_EVENT_REMAP: - area_remap =3D area_dst; /* save for later unmap */ - area_dst =3D (char *)(unsigned long)msg.arg.remap.to; + gopts->area_remap =3D gopts->area_dst; /* save for later unmap */ + gopts->area_dst =3D (char *)(unsigned long)msg.arg.remap.to; break; } } @@ -568,13 +590,14 @@ void *uffd_poll_thread(void *arg) return NULL; } =20 -static void retry_copy_page(int ufd, struct uffdio_copy *uffdio_copy, +static void retry_copy_page(uffd_global_test_opts_t *gopts, struct uffdio_= copy *uffdio_copy, unsigned long offset) { - uffd_test_ops->alias_mapping(&uffdio_copy->dst, + uffd_test_ops->alias_mapping(gopts, + &uffdio_copy->dst, uffdio_copy->len, offset); - if (ioctl(ufd, UFFDIO_COPY, uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy->copy !=3D -EEXIST) err("UFFDIO_COPY retry error: %"PRId64, @@ -597,60 +620,60 @@ static void wake_range(int ufd, unsigned long addr, u= nsigned long len) addr), exit(1); } =20 -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp) +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp) { struct uffdio_copy uffdio_copy; =20 - if (offset >=3D nr_pages * page_size) + if (offset >=3D gopts->nr_pages * gopts->page_size) err("unexpected offset %lu\n", offset); - uffdio_copy.dst =3D (unsigned long) area_dst + offset; - uffdio_copy.src =3D (unsigned long) area_src + offset; - uffdio_copy.len =3D page_size; + uffdio_copy.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_copy.src =3D (unsigned long) gopts->area_src + offset; + uffdio_copy.len =3D gopts->page_size; if (wp) uffdio_copy.mode =3D UFFDIO_COPY_MODE_WP; else uffdio_copy.mode =3D 0; uffdio_copy.copy =3D 0; - if (ioctl(ufd, UFFDIO_COPY, &uffdio_copy)) { + if (ioctl(gopts->uffd, UFFDIO_COPY, &uffdio_copy)) { /* real retval in ufdio_copy.copy */ if (uffdio_copy.copy !=3D -EEXIST) err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); - wake_range(ufd, uffdio_copy.dst, page_size); - } else if (uffdio_copy.copy !=3D page_size) { + wake_range(gopts->uffd, uffdio_copy.dst, gopts->page_size); + } else if (uffdio_copy.copy !=3D gopts->page_size) { err("UFFDIO_COPY error: %"PRId64, (int64_t)uffdio_copy.copy); } else { - if (test_uffdio_copy_eexist && retry) { - test_uffdio_copy_eexist =3D false; - retry_copy_page(ufd, &uffdio_copy, offset); + if (gopts->test_uffdio_copy_eexist && retry) { + gopts->test_uffdio_copy_eexist =3D false; + retry_copy_page(gopts, &uffdio_copy, offset); } return 1; } return 0; } =20 -int copy_page(int ufd, unsigned long offset, bool wp) +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p) { - return __copy_page(ufd, offset, false, wp); + return __copy_page(gopts, offset, false, wp); } =20 -int move_page(int ufd, unsigned long offset, unsigned long len) +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len) { struct uffdio_move uffdio_move; =20 - if (offset + len > nr_pages * page_size) + if (offset + len > gopts->nr_pages * gopts->page_size) err("unexpected offset %lu and length %lu\n", offset, len); - uffdio_move.dst =3D (unsigned long) area_dst + offset; - uffdio_move.src =3D (unsigned long) area_src + offset; + uffdio_move.dst =3D (unsigned long) gopts->area_dst + offset; + uffdio_move.src =3D (unsigned long) gopts->area_src + offset; uffdio_move.len =3D len; uffdio_move.mode =3D UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES; uffdio_move.move =3D 0; - if (ioctl(ufd, UFFDIO_MOVE, &uffdio_move)) { + if (ioctl(gopts->uffd, UFFDIO_MOVE, &uffdio_move)) { /* real retval in uffdio_move.move */ if (uffdio_move.move !=3D -EEXIST) err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); - wake_range(ufd, uffdio_move.dst, len); + wake_range(gopts->uffd, uffdio_move.dst, len); } else if (uffdio_move.move !=3D len) { err("UFFDIO_MOVE error: %"PRId64, (int64_t)uffdio_move.move); } else diff --git a/tools/testing/selftests/mm/uffd-common.h b/tools/testing/selft= ests/mm/uffd-common.h index 7700cbfa3975..a3debf116d58 100644 --- a/tools/testing/selftests/mm/uffd-common.h +++ b/tools/testing/selftests/mm/uffd-common.h @@ -56,20 +56,17 @@ =20 #define err(fmt, ...) errexit(1, fmt, ##__VA_ARGS__) =20 -/* pthread_mutex_t starts at page offset 0 */ -#define area_mutex(___area, ___nr) \ - ((pthread_mutex_t *) ((___area) + (___nr)*page_size)) -/* - * count is placed in the page after pthread_mutex_t naturally aligned - * to avoid non alignment faults on non-x86 archs. - */ -#define area_count(___area, ___nr) \ - ((volatile unsigned long long *) ((unsigned long) \ - ((___area) + (___nr)*page_size + \ - sizeof(pthread_mutex_t) + \ - sizeof(unsigned long long) - 1) & \ - ~(unsigned long)(sizeof(unsigned long long) \ - - 1))) +struct uffd_global_test_opts { + unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; + char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_remap; + int uffd, uffd_flags, finished, *pipefd, test_type; + bool map_shared; + bool test_uffdio_wp; + unsigned long long *count_verify; + volatile bool test_uffdio_copy_eexist; + atomic_bool ready_for_fork; +}; +typedef struct uffd_global_test_opts uffd_global_test_opts_t; =20 /* Userfaultfd test statistics */ struct uffd_args { @@ -79,50 +76,55 @@ struct uffd_args { unsigned long missing_faults; unsigned long wp_faults; unsigned long minor_faults; + struct uffd_global_test_opts *gopts; =20 /* A custom fault handler; defaults to uffd_handle_page_fault. */ - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args); + void (*handle_fault)(struct uffd_global_test_opts *gopts, + struct uffd_msg *msg, + struct uffd_args *args); }; =20 struct uffd_test_ops { - int (*allocate_area)(void **alloc_area, bool is_src); - void (*release_pages)(char *rel_area); - void (*alias_mapping)(__u64 *start, size_t len, unsigned long offset); - void (*check_pmd_mapping)(void *p, int expect_nr_hpages); + int (*allocate_area)(uffd_global_test_opts_t *gopts, void **alloc_area, b= ool is_src); + void (*release_pages)(uffd_global_test_opts_t *gopts, char *rel_area); + void (*alias_mapping)(uffd_global_test_opts_t *gopts, + __u64 *start, + size_t len, + unsigned long offset); + void (*check_pmd_mapping)(uffd_global_test_opts_t *gopts, void *p, int ex= pect_nr_hpages); }; typedef struct uffd_test_ops uffd_test_ops_t; =20 struct uffd_test_case_ops { - int (*pre_alloc)(const char **errmsg); - int (*post_alloc)(const char **errmsg); + int (*pre_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); + int (*post_alloc)(uffd_global_test_opts_t *gopts, const char **errmsg); }; typedef struct uffd_test_case_ops uffd_test_case_ops_t; =20 -extern unsigned long nr_parallel, nr_pages, nr_pages_per_cpu, page_size; -extern char *area_src, *area_src_alias, *area_dst, *area_dst_alias, *area_= remap; -extern int uffd, uffd_flags, finished, *pipefd, test_type; -extern bool map_shared; -extern bool test_uffdio_wp; -extern unsigned long long *count_verify; -extern volatile bool test_uffdio_copy_eexist; -extern atomic_bool ready_for_fork; - +extern uffd_global_test_opts_t *uffd_gtest_opts; extern uffd_test_ops_t anon_uffd_test_ops; extern uffd_test_ops_t shmem_uffd_test_ops; extern uffd_test_ops_t hugetlb_uffd_test_ops; extern uffd_test_ops_t *uffd_test_ops; extern uffd_test_case_ops_t *uffd_test_case_ops; =20 +pthread_mutex_t *area_mutex(char *area, unsigned long nr, uffd_global_test= _opts_t *gopts); +volatile unsigned long long *area_count(char *area, + unsigned long nr, + uffd_global_test_opts_t *gopts); + void uffd_stats_report(struct uffd_args *args, int n_cpus); -int uffd_test_ctx_init(uint64_t features, const char **errmsg); -void uffd_test_ctx_clear(void); -int userfaultfd_open(uint64_t *features); -int uffd_read_msg(int ufd, struct uffd_msg *msg); +int uffd_test_ctx_init(uffd_global_test_opts_t *gopts, uint64_t features, = const char **errmsg); +void uffd_test_ctx_clear(uffd_global_test_opts_t *gopts); +int userfaultfd_open(uffd_global_test_opts_t *gopts, uint64_t *features); +int uffd_read_msg(uffd_global_test_opts_t *gopts, struct uffd_msg *msg); void wp_range(int ufd, __u64 start, __u64 len, bool wp); -void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args); -int __copy_page(int ufd, unsigned long offset, bool retry, bool wp); -int copy_page(int ufd, unsigned long offset, bool wp); -int move_page(int ufd, unsigned long offset, unsigned long len); +void uffd_handle_page_fault(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args); +int __copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool= retry, bool wp); +int copy_page(uffd_global_test_opts_t *gopts, unsigned long offset, bool w= p); +int move_page(uffd_global_test_opts_t *gopts, unsigned long offset, unsign= ed long len); void *uffd_poll_thread(void *arg); =20 int uffd_open_dev(unsigned int flags); --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 653C520D516; Thu, 1 May 2025 16:40:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117628; cv=none; b=MSD8soTJUU9iiUYB1Q2V2ol1vhvjIAuERLyYJ1J9UZWfEBrwzsyOVIg6qXR0RuNNlPL+x363cdmYcHX/dd+2PTNX3dD0gah5YOBLCCuAeN+bYYeejUST/VgISoMe23AxaF8ruPeEMk94c6CKdVZTUDnZu0ULbZQchkXxm2Io2X8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117628; c=relaxed/simple; bh=gJG4WpKMeRT8LmCgKCwBrhcvPflBKv0xPyIKj8UB5+0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nRqTeVTIrHZtbLY7GthneilQ4An/fGITVtE/DoUnQL7ZgihXALT1sz/MV+yqenb2IR+tD3FlbP618vX381VQVfp0EHzrZ8q/ojI3q7cIbOMvEoJUqOWi2tqW1QaAm/0YtiVg1zakoCs4KbV+FQZ5tEQyY+ThbJ3lkqN8oWTp5HM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jLKa8+DX; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jLKa8+DX" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2243803b776so20200185ad.0; Thu, 01 May 2025 09:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746117625; x=1746722425; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AF9bCGSiDG6aAnBM3zsgH3EAIDOOJCJJIDxIMwiMydU=; b=jLKa8+DXz5X9bO9hGVqU4pbucxbExzPju7rnSkIUA7b02aVv+99bNr2mT74JOB3ySi lcXnyfQ/ydHxF08b7awB+0/PU32RJNsGarzGsBSJrK9azHspSB4gd7NrkE/TyWgNBKiy t/XEtUL03VBQOB1Oj8uEjvLeALFGD0QoNJP+uqjEWpVW9OPUbuwD+BQdtwAxf+ObzYOt YKYhQUoZ11+ak/mG2ne89JHsQcgQDAbH6TknEUfuOJRMsLnj6xhaeo07Hzfxc9Fo5iuu EB1rh18of/8u5oiy3+orot34PpemFrIpuExhwccvi/KaWLMq7X0wqlRNcFCBhSYSB+10 36Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746117625; x=1746722425; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AF9bCGSiDG6aAnBM3zsgH3EAIDOOJCJJIDxIMwiMydU=; b=rIKnbUeAjeQYGqAnzGoRfXLJVve2yPyoEtxXjqodK51HAj0FC0GI2GB96yWloP+8fj xMhmz2PHbXSDWjL2ekSjFuc/bEovxpZm1XD6Nn31I4LpaKN3im5Gr8Sf8U/AmrME1TnI KMiw1x2NC+hFxiBtFFYw9LtHLdeb8KcfYYoCnWX825k84f11bUaczrbyKJ7syUS3+Jv6 SxLZ0Irf+4mvj4g1DD+SUIKOxnxAmmYWEfT//eE83Do1U5xtTvJLpVZI7g03nQ+pc3m9 Xlfm6pfkojL+rIyL0tep3nLkjMSu+kr3Im7J6SPp2iHhsD6zEmya3pp40q6URldmXCos HwTw== X-Forwarded-Encrypted: i=1; AJvYcCVEa888uLpjiKa6W2cwQ5AhnGKCnlqzv5c1qLcvhMOT1gHaJLXiupX4UZm8KUPylNCn18R4skaKiPXfzZXzdvDI@vger.kernel.org, AJvYcCVlhdytXu70ovKY5BgIvAII/ER9FQufqg/sZ5JVLofe7GJnIiQDbKJqX8nXmlpDxVrnkYqb8siBkAasH7Y=@vger.kernel.org X-Gm-Message-State: AOJu0YySqcWROPzj95pz4iE4bbx1k0MqOTfPulWpjQW5qMiVuHTYthuW VFo9eq3MLkjpRQOZY1QFLBvSoU4EMWkeoyJV+WDUhaIET2e35W0= X-Gm-Gg: ASbGnctGJ1njvbJ7g/3bHmGUEiN6PRxaD6cvOOeiynf89uEVfDp3wgZig8ZocINE0jY Ge4wIs07ADPuM2Lk9U6vWCkFG1oYrUSrkGBvhejN52B2VRjrRXLxfL92LjobhaeM/YBKzHGPgZ2 1NERfaewtbD2uTALHL8HXHokD1JpvYLxqDoa1dklEqcrCMNdyUjaaOQgFD1Qdib+fumKbMKeHQ/ UcCnAVE5C5sYeSw2Co0MWPsex7K4BFrFQBbaJ0d9IPO1cgonr2xREsRRjtKnWNLk3brB4kU1Jbu Mb1Rs0QzvuO8pVp8DvEed/g77XvHAS/ko+ZaL1u1XFTWHT5v6g== X-Google-Smtp-Source: AGHT+IGF2JIgoeWIE0b6mWruCPVwb/wE1gTNq+7B89OLJ6JmGJIc1XBbfBpmp+9eRRyxdzcTDUYPog== X-Received: by 2002:a17:902:db01:b0:22c:336f:cb54 with SMTP id d9443c01a7336-22e040c8179mr56051845ad.29.1746117625603; Thu, 01 May 2025 09:40:25 -0700 (PDT) Received: from debian.ujwal.com ([223.185.132.49]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22e0bc6e678sm8797525ad.132.2025.05.01.09.40.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 May 2025 09:40:25 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, peterx@redhat.com, shuah@kernel.org Cc: linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Ujwal Kundur Subject: [PATCH 2/4] selftests/mm/uffd: Swap global vars with global test options Date: Thu, 1 May 2025 22:08:25 +0530 Message-Id: <20250501163827.2598-3-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We use the refactored uffd_global_test_opts struct instead of global variables for uffd-stress tests. Signed-off-by: Ujwal Kundur Tested-by: Brendan Jackman --- tools/testing/selftests/mm/uffd-stress.c | 226 ++++++++++++----------- 1 file changed, 122 insertions(+), 104 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-stress.c b/tools/testing/selft= ests/mm/uffd-stress.c index 40af7f67c407..e7aac958d57d 100644 --- a/tools/testing/selftests/mm/uffd-stress.c +++ b/tools/testing/selftests/mm/uffd-stress.c @@ -76,54 +76,58 @@ static void usage(void) exit(1); } =20 -static void uffd_stats_reset(struct uffd_args *args, unsigned long n_cpus) +static void uffd_stats_reset(uffd_global_test_opts_t *gopts, struct uffd_a= rgs *args, + unsigned long n_cpus) { int i; =20 for (i =3D 0; i < n_cpus; i++) { args[i].cpu =3D i; - args[i].apply_wp =3D test_uffdio_wp; + args[i].apply_wp =3D gopts->test_uffdio_wp; args[i].missing_faults =3D 0; args[i].wp_faults =3D 0; args[i].minor_faults =3D 0; + args[i].gopts =3D gopts; } } =20 static void *locking_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr; unsigned long long count; =20 if (!(bounces & BOUNCE_RANDOM)) { page_nr =3D -bounces; if (!(bounces & BOUNCE_RACINGFAULTS)) - page_nr +=3D cpu * nr_pages_per_cpu; + page_nr +=3D cpu * gopts->nr_pages_per_cpu; } =20 - while (!finished) { + while (!gopts->finished) { if (bounces & BOUNCE_RANDOM) { if (getrandom(&page_nr, sizeof(page_nr), 0) !=3D sizeof(page_nr)) err("getrandom failed"); } else page_nr +=3D 1; - page_nr %=3D nr_pages; - pthread_mutex_lock(area_mutex(area_dst, page_nr)); - count =3D *area_count(area_dst, page_nr); - if (count !=3D count_verify[page_nr]) + page_nr %=3D gopts->nr_pages; + pthread_mutex_lock(area_mutex(gopts->area_dst, page_nr, gopts)); + count =3D *area_count(gopts->area_dst, page_nr, gopts); + if (count !=3D gopts->count_verify[page_nr]) err("page_nr %lu memory corruption %llu %llu", - page_nr, count, count_verify[page_nr]); + page_nr, count, gopts->count_verify[page_nr]); count++; - *area_count(area_dst, page_nr) =3D count_verify[page_nr] =3D count; - pthread_mutex_unlock(area_mutex(area_dst, page_nr)); + *area_count(gopts->area_dst, page_nr, gopts) =3D gopts->count_verify[pag= e_nr] =3D count; + pthread_mutex_unlock(area_mutex(gopts->area_dst, page_nr, gopts)); } =20 return NULL; } =20 -static int copy_page_retry(int ufd, unsigned long offset) +static int copy_page_retry(uffd_global_test_opts_t *gopts, unsigned long o= ffset) { - return __copy_page(ufd, offset, true, test_uffdio_wp); + return __copy_page(gopts, offset, true, gopts->test_uffdio_wp); } =20 pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INITIALIZER; @@ -131,15 +135,16 @@ pthread_mutex_t uffd_read_mutex =3D PTHREAD_MUTEX_INI= TIALIZER; static void *uffd_read_thread(void *arg) { struct uffd_args *args =3D (struct uffd_args *)arg; + uffd_global_test_opts_t *gopts =3D args->gopts; struct uffd_msg msg; =20 pthread_mutex_unlock(&uffd_read_mutex); /* from here cancellation is ok */ =20 for (;;) { - if (uffd_read_msg(uffd, &msg)) + if (uffd_read_msg(gopts, &msg)) continue; - uffd_handle_page_fault(&msg, args); + uffd_handle_page_fault(gopts, &msg, args); } =20 return NULL; @@ -147,32 +152,34 @@ static void *uffd_read_thread(void *arg) =20 static void *background_thread(void *arg) { - unsigned long cpu =3D (unsigned long) arg; + struct uffd_args *args =3D (struct uffd_args *) arg; + uffd_global_test_opts_t *gopts =3D args->gopts; + unsigned long cpu =3D (unsigned long) args->cpu; unsigned long page_nr, start_nr, mid_nr, end_nr; =20 - start_nr =3D cpu * nr_pages_per_cpu; - end_nr =3D (cpu+1) * nr_pages_per_cpu; + start_nr =3D cpu * gopts->nr_pages_per_cpu; + end_nr =3D (cpu+1) * gopts->nr_pages_per_cpu; mid_nr =3D (start_nr + end_nr) / 2; =20 /* Copy the first half of the pages */ for (page_nr =3D start_nr; page_nr < mid_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 /* * If we need to test uffd-wp, set it up now. Then we'll have * at least the first half of the pages mapped already which * can be write-protected for testing */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst + start_nr * page_size, - nr_pages_per_cpu * page_size, true); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst + start_nr * gopts-= >page_size, + gopts->nr_pages_per_cpu * gopts->page_size, true); =20 /* * Continue the 2nd half of the page copying, handling write * protection faults if any */ for (page_nr =3D mid_nr; page_nr < end_nr; page_nr++) - copy_page_retry(uffd, page_nr * page_size); + copy_page_retry(gopts, page_nr * gopts->page_size); =20 return NULL; } @@ -180,17 +187,23 @@ static void *background_thread(void *arg) static int stress(struct uffd_args *args) { unsigned long cpu; - pthread_t locking_threads[nr_parallel]; - pthread_t uffd_threads[nr_parallel]; - pthread_t background_threads[nr_parallel]; + uffd_global_test_opts_t *gopts =3D args->gopts; + pthread_t locking_threads[gopts->nr_parallel]; + pthread_t uffd_threads[gopts->nr_parallel]; + pthread_t background_threads[gopts->nr_parallel]; =20 - finished =3D 0; - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + gopts->finished =3D 0; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { if (pthread_create(&locking_threads[cpu], &attr, - locking_thread, (void *)cpu)) + locking_thread, (void *)&args[cpu])) return 1; if (bounces & BOUNCE_POLL) { - if (pthread_create(&uffd_threads[cpu], &attr, uffd_poll_thread, &args[c= pu])) + if (pthread_create( + &uffd_threads[cpu], + &attr, + uffd_poll_thread, + (void *) &args[cpu]) + ) err("uffd_poll_thread create"); } else { if (pthread_create(&uffd_threads[cpu], &attr, @@ -200,10 +213,10 @@ static int stress(struct uffd_args *args) pthread_mutex_lock(&uffd_read_mutex); } if (pthread_create(&background_threads[cpu], &attr, - background_thread, (void *)cpu)) + background_thread, (void *)&args[cpu])) return 1; } - for (cpu =3D 0; cpu < nr_parallel; cpu++) + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(background_threads[cpu], NULL)) return 1; =20 @@ -216,17 +229,17 @@ static int stress(struct uffd_args *args) * UFFDIO_COPY without writing zero pages into area_dst * because the background threads already completed). */ - uffd_test_ops->release_pages(area_src); + uffd_test_ops->release_pages(gopts, gopts->area_src); =20 - finished =3D 1; - for (cpu =3D 0; cpu < nr_parallel; cpu++) + gopts->finished =3D 1; + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) if (pthread_join(locking_threads[cpu], NULL)) return 1; =20 - for (cpu =3D 0; cpu < nr_parallel; cpu++) { + for (cpu =3D 0; cpu < gopts->nr_parallel; cpu++) { char c; if (bounces & BOUNCE_POLL) { - if (write(pipefd[cpu*2+1], &c, 1) !=3D 1) + if (write(gopts->pipefd[cpu*2+1], &c, 1) !=3D 1) err("pipefd write error"); if (pthread_join(uffd_threads[cpu], (void *)&args[cpu])) @@ -242,26 +255,26 @@ static int stress(struct uffd_args *args) return 0; } =20 -static int userfaultfd_stress(void) +static int userfaultfd_stress(uffd_global_test_opts_t *gopts) { void *area; unsigned long nr; - struct uffd_args args[nr_parallel]; - uint64_t mem_size =3D nr_pages * page_size; + struct uffd_args args[gopts->nr_parallel]; + uint64_t mem_size =3D gopts->nr_pages * gopts->page_size; int flags =3D 0; =20 - memset(args, 0, sizeof(struct uffd_args) * nr_parallel); + memset(args, 0, sizeof(struct uffd_args) * gopts->nr_parallel); =20 - if (features & UFFD_FEATURE_WP_UNPOPULATED && test_type =3D=3D TEST_ANON) + if (features & UFFD_FEATURE_WP_UNPOPULATED && gopts->test_type =3D=3D TES= T_ANON) flags =3D UFFD_FEATURE_WP_UNPOPULATED; =20 - if (uffd_test_ctx_init(flags, NULL)) + if (uffd_test_ctx_init(gopts, flags, NULL)) err("context init failed"); =20 - if (posix_memalign(&area, page_size, page_size)) + if (posix_memalign(&area, gopts->page_size, gopts->page_size)) err("out of memory"); zeropage =3D area; - bzero(zeropage, page_size); + bzero(zeropage, gopts->page_size); =20 pthread_mutex_lock(&uffd_read_mutex); =20 @@ -284,18 +297,18 @@ static int userfaultfd_stress(void) fflush(stdout); =20 if (bounces & BOUNCE_POLL) - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); else - fcntl(uffd, F_SETFL, uffd_flags & ~O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags & ~O_NONBLOCK); =20 /* register */ - if (uffd_register(uffd, area_dst, mem_size, - true, test_uffdio_wp, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure"); =20 - if (area_dst_alias) { - if (uffd_register(uffd, area_dst_alias, mem_size, - true, test_uffdio_wp, false)) + if (gopts->area_dst_alias) { + if (uffd_register(gopts->uffd, gopts->area_dst_alias, mem_size, + true, gopts->test_uffdio_wp, false)) err("register failure alias"); } =20 @@ -323,87 +336,88 @@ static int userfaultfd_stress(void) * MADV_DONTNEED only after the UFFDIO_REGISTER, so it's * required to MADV_DONTNEED here. */ - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - uffd_stats_reset(args, nr_parallel); + uffd_stats_reset(gopts, args, gopts->nr_parallel); =20 /* bounce pass */ if (stress(args)) { - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); return 1; } =20 /* Clear all the write protections if there is any */ - if (test_uffdio_wp) - wp_range(uffd, (unsigned long)area_dst, - nr_pages * page_size, false); + if (gopts->test_uffdio_wp) + wp_range(gopts->uffd, (unsigned long)gopts->area_dst, + gopts->nr_pages * gopts->page_size, false); =20 /* unregister */ - if (uffd_unregister(uffd, area_dst, mem_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, mem_size)) err("unregister failure"); - if (area_dst_alias) { - if (uffd_unregister(uffd, area_dst_alias, mem_size)) + if (gopts->area_dst_alias) { + if (uffd_unregister(gopts->uffd, gopts->area_dst_alias, mem_size)) err("unregister failure alias"); } =20 /* verification */ if (bounces & BOUNCE_VERIFY) - for (nr =3D 0; nr < nr_pages; nr++) - if (*area_count(area_dst, nr) !=3D count_verify[nr]) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + if (*area_count(gopts->area_dst, nr, gopts) !=3D + gopts->count_verify[nr]) err("error area_count %llu %llu %lu\n", - *area_count(area_src, nr), - count_verify[nr], nr); + *area_count(gopts->area_src, nr, gopts), + gopts->count_verify[nr], nr); =20 /* prepare next bounce */ - swap(area_src, area_dst); + swap(gopts->area_src, gopts->area_dst); =20 - swap(area_src_alias, area_dst_alias); + swap(gopts->area_src_alias, gopts->area_dst_alias); =20 - uffd_stats_report(args, nr_parallel); + uffd_stats_report(args, gopts->nr_parallel); } - uffd_test_ctx_clear(); + uffd_test_ctx_clear(gopts); =20 return 0; } =20 -static void set_test_type(const char *type) +static void set_test_type(uffd_global_test_opts_t *gopts, const char *type) { if (!strcmp(type, "anon")) { - test_type =3D TEST_ANON; + gopts->test_type =3D TEST_ANON; uffd_test_ops =3D &anon_uffd_test_ops; } else if (!strcmp(type, "hugetlb")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; - map_shared =3D true; + gopts->map_shared =3D true; } else if (!strcmp(type, "hugetlb-private")) { - test_type =3D TEST_HUGETLB; + gopts->test_type =3D TEST_HUGETLB; uffd_test_ops =3D &hugetlb_uffd_test_ops; } else if (!strcmp(type, "shmem")) { - map_shared =3D true; - test_type =3D TEST_SHMEM; + gopts->map_shared =3D true; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } else if (!strcmp(type, "shmem-private")) { - test_type =3D TEST_SHMEM; + gopts->test_type =3D TEST_SHMEM; uffd_test_ops =3D &shmem_uffd_test_ops; } } =20 -static void parse_test_type_arg(const char *raw_type) +static void parse_test_type_arg(uffd_global_test_opts_t *gopts, const char= *raw_type) { - set_test_type(raw_type); + set_test_type(gopts, raw_type); =20 - if (!test_type) + if (!gopts->test_type) err("failed to parse test type argument: '%s'", raw_type); =20 - if (test_type =3D=3D TEST_HUGETLB) - page_size =3D default_huge_page_size(); + if (gopts->test_type =3D=3D TEST_HUGETLB) + gopts->page_size =3D default_huge_page_size(); else - page_size =3D sysconf(_SC_PAGE_SIZE); + gopts->page_size =3D sysconf(_SC_PAGE_SIZE); =20 - if (!page_size) + if (!gopts->page_size) err("Unable to determine page size"); - if ((unsigned long) area_count(NULL, 0) + sizeof(unsigned long long) * 2 - > page_size) + if ((unsigned long) area_count(NULL, 0, gopts) + sizeof(unsigned long lon= g) * 2 + > gopts->page_size) err("Impossible to run this test"); =20 /* @@ -415,21 +429,22 @@ static void parse_test_type_arg(const char *raw_type) if (uffd_get_features(&features) && errno =3D=3D ENOENT) ksft_exit_skip("failed to get available features (%d)\n", errno); =20 - test_uffdio_wp =3D test_uffdio_wp && + gopts->test_uffdio_wp =3D gopts->test_uffdio_wp && (features & UFFD_FEATURE_PAGEFAULT_FLAG_WP); =20 - if (test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGETLBFS_SH= MEM)) - test_uffdio_wp =3D false; + if (gopts->test_type !=3D TEST_ANON && !(features & UFFD_FEATURE_WP_HUGET= LBFS_SHMEM)) + gopts->test_uffdio_wp =3D false; =20 - close(uffd); - uffd =3D -1; + close(gopts->uffd); + gopts->uffd =3D -1; } =20 static void sigalrm(int sig) { if (sig !=3D SIGALRM) abort(); - test_uffdio_copy_eexist =3D true; + // TODO: Set this without access to global vars + // gopts->test_uffdio_copy_eexist =3D true; alarm(ALARM_INTERVAL_SECS); } =20 @@ -438,6 +453,9 @@ int main(int argc, char **argv) unsigned long nr_cpus; size_t bytes; =20 + uffd_global_test_opts_t *gopts =3D + (uffd_global_test_opts_t *) malloc(sizeof(uffd_global_test_opts_t)); + if (argc < 4) usage(); =20 @@ -445,11 +463,11 @@ int main(int argc, char **argv) err("failed to arm SIGALRM"); alarm(ALARM_INTERVAL_SECS); =20 - parse_test_type_arg(argv[1]); + parse_test_type_arg(gopts, argv[1]); bytes =3D atol(argv[2]) * 1024 * 1024; =20 - if (test_type =3D=3D TEST_HUGETLB && - get_free_hugepages() < bytes / page_size) { + if (gopts->test_type =3D=3D TEST_HUGETLB && + get_free_hugepages() < bytes / gopts->page_size) { printf("skip: Skipping userfaultfd... not enough hugepages\n"); return KSFT_SKIP; } @@ -459,15 +477,15 @@ int main(int argc, char **argv) /* Don't let calculation below go to zero. */ ksft_print_msg("_SC_NPROCESSORS_ONLN (%lu) too large, capping nr_threads= to 32\n", nr_cpus); - nr_parallel =3D 32; + gopts->nr_parallel =3D 32; } else { - nr_parallel =3D nr_cpus; + gopts->nr_parallel =3D nr_cpus; } =20 - nr_pages_per_cpu =3D bytes / page_size / nr_parallel; - if (!nr_pages_per_cpu) { + gopts->nr_pages_per_cpu =3D bytes / gopts->page_size / gopts->nr_parallel; + if (!gopts->nr_pages_per_cpu) { _err("pages_per_cpu =3D 0, cannot test (%lu / %lu / %lu)", - bytes, page_size, nr_parallel); + bytes, gopts->page_size, gopts->nr_parallel); usage(); } =20 @@ -476,11 +494,11 @@ int main(int argc, char **argv) _err("invalid bounces"); usage(); } - nr_pages =3D nr_pages_per_cpu * nr_parallel; + gopts->nr_pages =3D gopts->nr_pages_per_cpu * gopts->nr_parallel; =20 printf("nr_pages: %lu, nr_pages_per_cpu: %lu\n", - nr_pages, nr_pages_per_cpu); - return userfaultfd_stress(); + gopts->nr_pages, gopts->nr_pages_per_cpu); + return userfaultfd_stress(gopts); } =20 #else /* __NR_userfaultfd */ --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48FDB213E6B; Thu, 1 May 2025 16:40:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117632; cv=none; b=hmqKrUW7EHeO0S1Dkb8RFVjvV8og5qkl1KT5MhFmlia37ApZObS5P0HrcIzR3+Zl3XvTbpGxfD3eAi4HoT8vMd2u1ta5bAagAhVrQTmO7EJJ2pSoYtMfdn1YTGGfz1G04nTh875zDa73d5AnksglUUB4sNDL/I4/T1y1k70d3yk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117632; c=relaxed/simple; bh=10xi8XnFF2pBTeJnAxDi+RXeoSO0HvTFBqqyIUQtz5s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lDmvQMOkOiQ5L0Xir8wqHGsSjTR9zph34kBDdL78UNFIqWi4DhJIwL78sWPsoUwCriVS6AwBClRF4wKYhLkjKSbxLppWYBhx+wepB0woYbjsq8UVqxG/AKxsyYVcqb3YUxUz7bvjrcJJyeKy5NC1MxdEOQyoFqc2lc1SmVEKBg4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QwcIAC0g; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QwcIAC0g" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-223fd89d036so14454195ad.1; Thu, 01 May 2025 09:40:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746117628; x=1746722428; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BB5CY2RE6jxRDaA+m+PYgNXEE166rwLgA/ve5XtJU18=; b=QwcIAC0gMBppbcIRilOyT0rFjyq8cjaTNvZTo08CZz+56uIyJ0yIkh4hnDB1ytCL1R gNDt8qnUvuefBDmA424H5LmrL+NDGSNh43Pnp42ZZB8pEje9QtDaY6EsuVEJ7eNQIcO9 LQhra2IWjShZ33tfyYfKOxIl9SQ4c66OIIXUMDGHeMKnZwzmIGxTYcIzu8TMlMOFWNJA /WSTmnlBMcVQCtOrju3/Dp0DurRZSd+6VkRDedzFD8HGNodbMfaLwTaIaOh5S3jFipPZ zyvvYH973O5PVnUvvYqqOYRL1yrh0v1I8T2IOLDSNpe0ueszty/VLeXIw4ITeox7Jn6I 8vOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746117628; x=1746722428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BB5CY2RE6jxRDaA+m+PYgNXEE166rwLgA/ve5XtJU18=; b=MCpr9g7TXdB46cLHGBbr7SJhdg26Zo4sNlzsQrFUK5Tc7JcTuo5vk/s2nOJezadGPd tDaQ9vQ1oRx85CujlFXBVD+B5vMf2VLpMpTOVkYVwktvfUG72rgQN8GfRMjpaWN+pv9X ioZnxa1oj8QQy8HHUuRcNP6xUP2wFMtHOtVZck4z9QtdRnDMEW+j9ww4KjEpLEDa0Kfv dTOmq2T5bY1TMASxQnGH46BYSn8d3m0MRT9ZMOOao/PyiI4jKronSuZIRZHsqulNf/Rm Ma25pZczNuJhT+KZebg3kdzqnoXMW7xAmKLiIUCMDXYkaCskUSIKE2ruwh2jPVNvwcXv hfcw== X-Forwarded-Encrypted: i=1; AJvYcCXezFugxGz8y56CYQjSgSKRNgIEiYv+gW0OhqDUQHFD//kguO1rWisFZszVC60qzGyA9U3IS0iMhOdQQYCjYQCT@vger.kernel.org, AJvYcCXhZnBzsIsQIuf7Er0IS3i0WIb2A+ObI9C0uPFQ5dwcEtsqqcxjyZy+NUS0854+0VdTk5pvICZslgr+G8U=@vger.kernel.org X-Gm-Message-State: AOJu0YyLQav+dL0GJcYWEd6MZH7AwdCF1zRW6Nitp/wLZZ4kTRAr4IGk H6wEbUWjGJku1j4Tw6v+1XSoNHQE/wVXZW1nuCZBUuyQc+2leec= X-Gm-Gg: ASbGncvut/AT0PYwRHzosGHRE0bjUNsjkKDKqqZ7SKM9CGSUtwRvkcVuadd2W/uP2tL SpPsYyWgLll6znWDnb/Mx6Qwh/CXtwpg7Cmf/iSs7neMwefAhZ9xYvdUxrKdBBVL/xWLs4CoEqF Yf8beZ4vIMqdLQpNcKqNl5XjR4BQgyrPg2dv6ZS/pBq0kd9xGOHyQMKtU3Nz3eBCGBbhaKdwZZF 0VL8NaUjOO2MANldWi2yD9LDvgyjXylrck6a9vGTQONhgKvpmT/w5yTX+czlUw8Jwpnw7z4bQ1o 65e/CNrrYizzO3sQR0tp6ovUhiZmt0CkbdTlCDFC0RUeTIZ5cw== X-Google-Smtp-Source: AGHT+IFF2XwkPeUfk8rGjs5PW1mV2oDabtGlr3Dx/0Dju9uVUlBdz3MpeuCAI4X/YLR7g6bIrgOsvA== X-Received: by 2002:a17:903:190:b0:216:6283:5a8c with SMTP id d9443c01a7336-22df58279b8mr106952975ad.39.1746117628251; Thu, 01 May 2025 09:40:28 -0700 (PDT) Received: from debian.ujwal.com ([223.185.132.49]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22e0bc6e678sm8797525ad.132.2025.05.01.09.40.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 May 2025 09:40:27 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, peterx@redhat.com, shuah@kernel.org Cc: linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Ujwal Kundur Subject: [PATCH 3/4] selftests/mm/uffd: Swap global variables with global test opts Date: Thu, 1 May 2025 22:08:26 +0530 Message-Id: <20250501163827.2598-4-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We use the refactored uffd_global_test_opts instead of global variables for uffd-unit-tests. Signed-off-by: Ujwal Kundur Tested-by: Brendan Jackman --- tools/testing/selftests/mm/uffd-unit-tests.c | 523 ++++++++++--------- 1 file changed, 269 insertions(+), 254 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/s= elftests/mm/uffd-unit-tests.c index e8fd9011c2a3..86ecf21e0722 100644 --- a/tools/testing/selftests/mm/uffd-unit-tests.c +++ b/tools/testing/selftests/mm/uffd-unit-tests.c @@ -76,7 +76,7 @@ struct uffd_test_args { typedef struct uffd_test_args uffd_test_args_t; =20 /* Returns: UFFD_TEST_* */ -typedef void (*uffd_test_fn)(uffd_test_args_t *); +typedef void (*uffd_test_fn)(uffd_global_test_opts_t *, uffd_test_args_t *= ); =20 typedef struct { const char *name; @@ -181,33 +181,6 @@ static int test_uffd_api(bool use_dev) return 1; } =20 -/* - * This function initializes the global variables. TODO: remove global - * vars and then remove this. - */ -static int -uffd_setup_environment(uffd_test_args_t *args, uffd_test_case_t *test, - mem_type_t *mem_type, const char **errmsg) -{ - map_shared =3D mem_type->shared; - uffd_test_ops =3D mem_type->mem_ops; - uffd_test_case_ops =3D test->test_case_ops; - - if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) - page_size =3D default_huge_page_size(); - else - page_size =3D psize(); - - /* Ensure we have at least 2 pages */ - nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, page_size * 2) / page_size; - /* TODO: remove this global var.. it's so ugly */ - nr_parallel =3D 1; - - /* Initialize test arguments */ - args->mem_type =3D mem_type; - - return uffd_test_ctx_init(test->uffd_feature_required, errmsg); -} =20 static bool uffd_feature_supported(uffd_test_case_t *test) { @@ -237,7 +210,8 @@ static int pagemap_open(void) } while (0) =20 typedef struct { - int parent_uffd, child_uffd; + uffd_global_test_opts_t *gopts; + int child_uffd; } fork_event_args; =20 static void *fork_event_consumer(void *data) @@ -245,10 +219,10 @@ static void *fork_event_consumer(void *data) fork_event_args *args =3D data; struct uffd_msg msg =3D { 0 }; =20 - ready_for_fork =3D true; + args->gopts->ready_for_fork =3D true; =20 /* Read until a full msg received */ - while (uffd_read_msg(args->parent_uffd, &msg)); + while (uffd_read_msg(args->gopts, &msg)); =20 if (msg.event !=3D UFFD_EVENT_FORK) err("wrong message: %u\n", msg.event); @@ -304,9 +278,9 @@ static void unpin_pages(pin_args *args) args->pinned =3D false; } =20 -static int pagemap_test_fork(int uffd, bool with_event, bool test_pin) +static int pagemap_test_fork(uffd_global_test_opts_t *gopts, bool with_eve= nt, bool test_pin) { - fork_event_args args =3D { .parent_uffd =3D uffd, .child_uffd =3D -1 }; + fork_event_args args =3D { .gopts =3D gopts, .child_uffd =3D -1 }; pthread_t thread; pid_t child; uint64_t value; @@ -314,10 +288,10 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 /* Prepare a thread to resolve EVENT_FORK */ if (with_event) { - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; if (pthread_create(&thread, NULL, fork_event_consumer, &args)) err("pthread_create()"); - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ } =20 @@ -328,14 +302,14 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) =20 fd =3D pagemap_open(); =20 - if (test_pin && pin_pages(&args, area_dst, page_size)) + if (test_pin && pin_pages(&args, gopts->area_dst, gopts->page_size)) /* * Normally when reach here we have pinned in * previous tests, so shouldn't fail anymore */ err("pin page failed in child"); =20 - value =3D pagemap_get_entry(fd, area_dst); + value =3D pagemap_get_entry(fd, gopts->area_dst); /* * After fork(), we should handle uffd-wp bit differently: * @@ -361,70 +335,70 @@ static int pagemap_test_fork(int uffd, bool with_even= t, bool test_pin) return result; } =20 -static void uffd_wp_unpopulated_test(uffd_test_args_t *args) +static void uffd_wp_unpopulated_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args) { uint64_t value; int pagemap_fd; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Test applying pte marker to anon unpopulated */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); =20 /* Test unprotect on anon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test zap on anon marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - if (madvise(area_dst, page_size, MADV_DONTNEED)) + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Test fault in after marker removed */ - *area_dst =3D 1; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 /* Test read-zero-page upon pte marker */ - wp_range(uffd, (uint64_t)area_dst, page_size, true); - *(volatile char *)area_dst; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + *(volatile char *)gopts->area_dst; /* Drop it to make pte none again */ - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("madvise(MADV_DONTNEED) failed"); =20 uffd_test_pass(); } =20 -static void uffd_wp_fork_test_common(uffd_test_args_t *args, +static void uffd_wp_fork_test_common(uffd_global_test_opts_t *gopts, uffd_= test_args_t *args, bool with_event) { int pagemap_fd; uint64_t value; =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, false, true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in present pte", with_event ? "missing" : "stall"); goto out; @@ -442,79 +416,79 @@ static void uffd_wp_fork_test_common(uffd_test_args_t= *args, * to expose pte markers. */ if (args->mem_type->shared) { - if (madvise(area_dst, page_size, MADV_DONTNEED)) + if (madvise(gopts->area_dst, gopts->page_size, MADV_DONTNEED)) err("MADV_DONTNEED"); } else { /* * NOTE: ignore retval because private-hugetlb doesn't yet * support swapping, so it could fail. */ - madvise(area_dst, page_size, MADV_PAGEOUT); + madvise(gopts->area_dst, gopts->page_size, MADV_PAGEOUT); } =20 /* Uffd-wp should persist even swapped out */ - value =3D pagemap_get_entry(pagemap_fd, area_dst); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, true); - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in child in zapped pte", with_event ? "missing" : "stall"); goto out; } =20 /* Unprotect; this tests swap pte modifications */ - wp_range(uffd, (uint64_t)area_dst, page_size, false); - value =3D pagemap_get_entry(pagemap_fd, area_dst); + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, false); + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); =20 /* Fault in the page from disk */ - *area_dst =3D 2; - value =3D pagemap_get_entry(pagemap_fd, area_dst); + *gopts->area_dst =3D 2; + value =3D pagemap_get_entry(pagemap_fd, gopts->area_dst); pagemap_check_wp(value, false); uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, nr_pages * page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts= ->page_size)) err("unregister failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_test(uffd_test_args_t *args) +static void uffd_wp_fork_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_wp_fork_test_common(args, false); + uffd_wp_fork_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_with_event_test(uffd_global_test_opts_t *gopts, u= ffd_test_args_t *args) { - uffd_wp_fork_test_common(args, true); + uffd_wp_fork_test_common(gopts, args, true); } =20 -static void uffd_wp_fork_pin_test_common(uffd_test_args_t *args, - bool with_event) +static void uffd_wp_fork_pin_test_common(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, bool with_event) { int pagemap_fd; pin_args pin_args =3D {}; =20 - if (uffd_register(uffd, area_dst, page_size, false, true, false)) + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->page_size, false, = true, false)) err("register failed"); =20 pagemap_fd =3D pagemap_open(); =20 /* Touch the page */ - *area_dst =3D 1; - wp_range(uffd, (uint64_t)area_dst, page_size, true); + *gopts->area_dst =3D 1; + wp_range(gopts->uffd, (uint64_t)gopts->area_dst, gopts->page_size, true); =20 /* * 1. First pin, then fork(). This tests fork() special path when * doing early CoW if the page is private. */ - if (pin_pages(&pin_args, area_dst, page_size)) { + if (pin_pages(&pin_args, gopts->area_dst, gopts->page_size)) { uffd_test_skip("Possibly CONFIG_GUP_TEST missing " "or unprivileged"); close(pagemap_fd); - uffd_unregister(uffd, area_dst, page_size); + uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size); return; } =20 - if (pagemap_test_fork(uffd, with_event, false)) { + if (pagemap_test_fork(gopts, with_event, false)) { uffd_test_fail("Detected %s uffd-wp bit in early CoW of fork()", with_event ? "missing" : "stall"); unpin_pages(&pin_args); @@ -527,49 +501,50 @@ static void uffd_wp_fork_pin_test_common(uffd_test_ar= gs_t *args, * 2. First fork(), then pin (in the child, where test_pin=3D=3Dtrue). * This tests COR, aka, page unsharing on private memories. */ - if (pagemap_test_fork(uffd, with_event, true)) { + if (pagemap_test_fork(gopts, with_event, true)) { uffd_test_fail("Detected %s uffd-wp bit when RO pin", with_event ? "missing" : "stall"); goto out; } uffd_test_pass(); out: - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("register failed"); close(pagemap_fd); } =20 -static void uffd_wp_fork_pin_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_test(uffd_global_test_opts_t *gopts, uffd_tes= t_args_t *args) { - uffd_wp_fork_pin_test_common(args, false); + uffd_wp_fork_pin_test_common(gopts, args, false); } =20 -static void uffd_wp_fork_pin_with_event_test(uffd_test_args_t *args) +static void uffd_wp_fork_pin_with_event_test(uffd_global_test_opts_t *gopt= s, uffd_test_args_t *args) { - uffd_wp_fork_pin_test_common(args, true); + uffd_wp_fork_pin_test_common(gopts, args, true); } =20 -static void check_memory_contents(char *p) +static void check_memory_contents(uffd_global_test_opts_t *gopts, char *p) { unsigned long i, j; uint8_t expected_byte; =20 - for (i =3D 0; i < nr_pages; ++i) { + for (i =3D 0; i < gopts->nr_pages; ++i) { expected_byte =3D ~((uint8_t)(i % ((uint8_t)-1))); - for (j =3D 0; j < page_size; j++) { - uint8_t v =3D *(uint8_t *)(p + (i * page_size) + j); + for (j =3D 0; j < gopts->page_size; j++) { + uint8_t v =3D *(uint8_t *)(p + (i * gopts->page_size) + j); if (v !=3D expected_byte) err("unexpected page contents"); } } } =20 -static void uffd_minor_test_common(bool test_collapse, bool test_wp) +static void uffd_minor_test_common(uffd_global_test_opts_t *gopts, bool te= st_collapse, bool test_wp) { unsigned long p; pthread_t uffd_mon; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 /* * NOTE: MADV_COLLAPSE is not yet compatible with WP, so testing @@ -577,7 +552,7 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) */ assert(!(test_collapse && test_wp)); =20 - if (uffd_register(uffd, area_dst_alias, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst_alias, gopts->nr_pages * g= opts->page_size, /* NOTE! MADV_COLLAPSE may not work with uffd-wp */ false, test_wp, true)) err("register failure"); @@ -586,9 +561,9 @@ static void uffd_minor_test_common(bool test_collapse, = bool test_wp) * After registering with UFFD, populate the non-UFFD-registered side of * the shared mapping. This should *not* trigger any UFFD minor faults. */ - for (p =3D 0; p < nr_pages; ++p) - memset(area_dst + (p * page_size), p % ((uint8_t)-1), - page_size); + for (p =3D 0; p < gopts->nr_pages; ++p) + memset(gopts->area_dst + (p * gopts->page_size), p % ((uint8_t)-1), + gopts->page_size); =20 args.apply_wp =3D test_wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -600,50 +575,51 @@ static void uffd_minor_test_common(bool test_collapse= , bool test_wp) * fault. uffd_poll_thread will resolve the fault by bit-flipping the * page's contents, and then issuing a CONTINUE ioctl. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); =20 if (test_collapse) { - if (madvise(area_dst_alias, nr_pages * page_size, + if (madvise(gopts->area_dst_alias, gopts->nr_pages * gopts->page_size, MADV_COLLAPSE)) { /* It's fine to fail for this one... */ uffd_test_skip("MADV_COLLAPSE failed"); return; } =20 - uffd_test_ops->check_pmd_mapping(area_dst, - nr_pages * page_size / + uffd_test_ops->check_pmd_mapping(gopts, + gopts->area_dst, + gopts->nr_pages * gopts->page_size / read_pmd_pagesize()); /* * This won't cause uffd-fault - it purely just makes sure there * was no corruption. */ - check_memory_contents(area_dst_alias); + check_memory_contents(gopts, gopts->area_dst_alias); } =20 - if (args.missing_faults !=3D 0 || args.minor_faults !=3D nr_pages) + if (args.missing_faults !=3D 0 || args.minor_faults !=3D gopts->nr_pages) uffd_test_fail("stats check error"); else uffd_test_pass(); } =20 -void uffd_minor_test(uffd_test_args_t *args) +void uffd_minor_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *arg= s) { - uffd_minor_test_common(false, false); + uffd_minor_test_common(gopts, false, false); } =20 -void uffd_minor_wp_test(uffd_test_args_t *args) +void uffd_minor_wp_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *= args) { - uffd_minor_test_common(false, true); + uffd_minor_test_common(gopts, false, true); } =20 -void uffd_minor_collapse_test(uffd_test_args_t *args) +void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_ar= gs_t *args) { - uffd_minor_test_common(true, false); + uffd_minor_test_common(gopts, true, false); } =20 static sigjmp_buf jbuf, *sigbuf; @@ -678,7 +654,7 @@ static void sighndl(int sig, siginfo_t *siginfo, void *= ptr) * This also tests UFFD_FEATURE_EVENT_FORK event along with the signal * feature. Using monitor thread, verify no userfault events are generated. */ -static int faulting_process(int signal_test, bool wp) +static int faulting_process(uffd_global_test_opts_t *gopts, int signal_tes= t, bool wp) { unsigned long nr, i; unsigned long long count; @@ -687,7 +663,7 @@ static int faulting_process(int signal_test, bool wp) struct sigaction act; volatile unsigned long signalled =3D 0; =20 - split_nr_pages =3D (nr_pages + 1) / 2; + split_nr_pages =3D (gopts->nr_pages + 1) / 2; =20 if (signal_test) { sigbuf =3D &jbuf; @@ -701,7 +677,7 @@ static int faulting_process(int signal_test, bool wp) =20 for (nr =3D 0; nr < split_nr_pages; nr++) { volatile int steps =3D 1; - unsigned long offset =3D nr * page_size; + unsigned long offset =3D nr * gopts->page_size; =20 if (signal_test) { if (sigsetjmp(*sigbuf, 1) !=3D 0) { @@ -713,15 +689,15 @@ static int faulting_process(int signal_test, bool wp) if (steps =3D=3D 1) { /* This is a MISSING request */ steps++; - if (copy_page(uffd, offset, wp)) + if (copy_page(gopts, offset, wp)) signalled++; } else { /* This is a WP request */ assert(steps =3D=3D 2); - wp_range(uffd, - (__u64)area_dst + + wp_range(gopts->uffd, + (__u64)gopts->area_dst + offset, - page_size, false); + gopts->page_size, false); } } else { signalled++; @@ -730,51 +706,53 @@ static int faulting_process(int signal_test, bool wp) } } =20 - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 if (signal_test) return signalled !=3D split_nr_pages; =20 - area_dst =3D mremap(area_dst, nr_pages * page_size, nr_pages * page_size, - MREMAP_MAYMOVE | MREMAP_FIXED, area_src); - if (area_dst =3D=3D MAP_FAILED) + gopts->area_dst =3D mremap(gopts->area_dst, gopts->nr_pages * gopts->page= _size, + gopts->nr_pages * gopts->page_size, + MREMAP_MAYMOVE | MREMAP_FIXED, + gopts->area_src); + if (gopts->area_dst =3D=3D MAP_FAILED) err("mremap"); /* Reset area_src since we just clobbered it */ - area_src =3D NULL; + gopts->area_src =3D NULL; =20 - for (; nr < nr_pages; nr++) { - count =3D *area_count(area_dst, nr); - if (count !=3D count_verify[nr]) { + for (; nr < gopts->nr_pages; nr++) { + count =3D *area_count(gopts->area_dst, nr, gopts); + if (count !=3D gopts->count_verify[nr]) { err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[nr]); + nr, count, gopts->count_verify[nr]); } /* * Trigger write protection if there is by writing * the same value back. */ - *area_count(area_dst, nr) =3D count; + *area_count(gopts->area_dst, nr, gopts) =3D count; } =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 - for (nr =3D 0; nr < nr_pages; nr++) - for (i =3D 0; i < page_size; i++) - if (*(area_dst + nr * page_size + i) !=3D 0) + for (nr =3D 0; nr < gopts->nr_pages; nr++) + for (i =3D 0; i < gopts->page_size; i++) + if (*(gopts->area_dst + nr * gopts->page_size + i) !=3D 0) err("page %lu offset %lu is not zero", nr, i); =20 return 0; } =20 -static void uffd_sigbus_test_common(bool wp) +static void uffd_sigbus_test_common(uffd_global_test_opts_t *gopts, bool w= p) { unsigned long userfaults; pthread_t uffd_mon; @@ -782,25 +760,26 @@ static void uffd_sigbus_test_common(bool wp) int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 - if (faulting_process(1, wp)) + if (faulting_process(gopts, 1, wp)) err("faulting process failed"); =20 - uffd_test_ops->release_pages(area_dst); + uffd_test_ops->release_pages(gopts, gopts->area_dst); =20 args.apply_wp =3D wp; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -808,12 +787,12 @@ static void uffd_sigbus_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(2, wp)); + exit(faulting_process(gopts, 2, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, (void **)&userfaults)) err("pthread_join()"); @@ -824,28 +803,29 @@ static void uffd_sigbus_test_common(bool wp) uffd_test_pass(); } =20 -static void uffd_sigbus_test(uffd_test_args_t *args) +static void uffd_sigbus_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_sigbus_test_common(false); + uffd_sigbus_test_common(gopts, false); } =20 -static void uffd_sigbus_wp_test(uffd_test_args_t *args) +static void uffd_sigbus_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_sigbus_test_common(true); + uffd_sigbus_test_common(gopts, true); } =20 -static void uffd_events_test_common(bool wp) +static void uffd_events_test_common(uffd_global_test_opts_t *gopts, bool w= p) { pthread_t uffd_mon; pid_t pid; int err; char c; struct uffd_args args =3D { 0 }; + args.gopts =3D gopts; =20 - ready_for_fork =3D false; + gopts->ready_for_fork =3D false; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); - if (uffd_register(uffd, area_dst, nr_pages * page_size, + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, wp, false)) err("register failure"); =20 @@ -853,7 +833,7 @@ static void uffd_events_test_common(bool wp) if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - while (!ready_for_fork) + while (!gopts->ready_for_fork) ; /* Wait for the poll_thread to start executing before forking */ =20 pid =3D fork(); @@ -861,39 +841,39 @@ static void uffd_events_test_common(bool wp) err("fork"); =20 if (!pid) - exit(faulting_process(0, wp)); + exit(faulting_process(gopts, 0, wp)); =20 waitpid(pid, &err, 0); if (err) err("faulting process failed"); - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (args.missing_faults !=3D nr_pages) + if (args.missing_faults !=3D gopts->nr_pages) uffd_test_fail("Fault counts wrong"); else uffd_test_pass(); } =20 -static void uffd_events_test(uffd_test_args_t *args) +static void uffd_events_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *args) { - uffd_events_test_common(false); + uffd_events_test_common(gopts, false); } =20 -static void uffd_events_wp_test(uffd_test_args_t *args) +static void uffd_events_wp_test(uffd_global_test_opts_t *gopts, uffd_test_= args_t *args) { - uffd_events_test_common(true); + uffd_events_test_common(gopts, true); } =20 -static void retry_uffdio_zeropage(int ufd, +static void retry_uffdio_zeropage(uffd_global_test_opts_t *gopts, struct uffdio_zeropage *uffdio_zeropage) { - uffd_test_ops->alias_mapping(&uffdio_zeropage->range.start, + uffd_test_ops->alias_mapping(gopts, &uffdio_zeropage->range.start, uffdio_zeropage->range.len, 0); - if (ioctl(ufd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { + if (ioctl(gopts->uffd, UFFDIO_ZEROPAGE, uffdio_zeropage)) { if (uffdio_zeropage->zeropage !=3D -EEXIST) err("UFFDIO_ZEROPAGE error: %"PRId64, (int64_t)uffdio_zeropage->zeropage); @@ -903,16 +883,16 @@ static void retry_uffdio_zeropage(int ufd, } } =20 -static bool do_uffdio_zeropage(int ufd, bool has_zeropage) +static bool do_uffdio_zeropage(uffd_global_test_opts_t *gopts, bool has_ze= ropage) { struct uffdio_zeropage uffdio_zeropage =3D { 0 }; int ret; __s64 res; =20 - uffdio_zeropage.range.start =3D (unsigned long) area_dst; - uffdio_zeropage.range.len =3D page_size; + uffdio_zeropage.range.start =3D (unsigned long) gopts->area_dst; + uffdio_zeropage.range.len =3D gopts->page_size; uffdio_zeropage.mode =3D 0; - ret =3D ioctl(ufd, UFFDIO_ZEROPAGE, &uffdio_zeropage); + ret =3D ioctl(gopts->uffd, UFFDIO_ZEROPAGE, &uffdio_zeropage); res =3D uffdio_zeropage.zeropage; if (ret) { /* real retval in ufdio_zeropage.zeropage */ @@ -921,10 +901,10 @@ static bool do_uffdio_zeropage(int ufd, bool has_zero= page) else if (res !=3D -EINVAL) err("UFFDIO_ZEROPAGE not -EINVAL"); } else if (has_zeropage) { - if (res !=3D page_size) + if (res !=3D gopts->page_size) err("UFFDIO_ZEROPAGE unexpected size"); else - retry_uffdio_zeropage(ufd, &uffdio_zeropage); + retry_uffdio_zeropage(gopts, &uffdio_zeropage); return true; } else err("UFFDIO_ZEROPAGE succeeded"); @@ -950,25 +930,27 @@ uffd_register_detect_zeropage(int uffd, void *addr, u= int64_t len) } =20 /* exercise UFFDIO_ZEROPAGE */ -static void uffd_zeropage_test(uffd_test_args_t *args) +static void uffd_zeropage_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *args) { bool has_zeropage; int i; =20 - has_zeropage =3D uffd_register_detect_zeropage(uffd, area_dst, page_size); - if (area_dst_alias) + has_zeropage =3D uffd_register_detect_zeropage(gopts->uffd, gopts->area_d= st, gopts->page_size); + if (gopts->area_dst_alias) /* Ignore the retval; we already have it */ - uffd_register_detect_zeropage(uffd, area_dst_alias, page_size); + uffd_register_detect_zeropage(gopts->uffd, gopts->area_dst_alias, gopts-= >page_size); =20 - if (do_uffdio_zeropage(uffd, has_zeropage)) - for (i =3D 0; i < page_size; i++) - if (area_dst[i] !=3D 0) + if (do_uffdio_zeropage(gopts, has_zeropage)) + for (i =3D 0; i < gopts->page_size; i++) + if (gopts->area_dst[i] !=3D 0) err("data non-zero at offset %d\n", i); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); =20 - if (area_dst_alias && uffd_unregister(uffd, area_dst_alias, page_size)) + if (gopts->area_dst_alias + && + uffd_unregister(gopts->uffd, gopts->area_dst_alias, gopts->page_size)) err("unregister"); =20 uffd_test_pass(); @@ -987,26 +969,26 @@ static void uffd_register_poison(int uffd, void *addr= , uint64_t len) err("registered area doesn't support COPY and POISON ioctls"); } =20 -static void do_uffdio_poison(int uffd, unsigned long offset) +static void do_uffdio_poison(uffd_global_test_opts_t *gopts, unsigned long= offset) { struct uffdio_poison uffdio_poison =3D { 0 }; int ret; __s64 res; =20 - uffdio_poison.range.start =3D (unsigned long) area_dst + offset; - uffdio_poison.range.len =3D page_size; + uffdio_poison.range.start =3D (unsigned long) gopts->area_dst + offset; + uffdio_poison.range.len =3D gopts->page_size; uffdio_poison.mode =3D 0; - ret =3D ioctl(uffd, UFFDIO_POISON, &uffdio_poison); + ret =3D ioctl(gopts->uffd, UFFDIO_POISON, &uffdio_poison); res =3D uffdio_poison.updated; =20 if (ret) err("UFFDIO_POISON error: %"PRId64, (int64_t)res); - else if (res !=3D page_size) + else if (res !=3D gopts->page_size) err("UFFDIO_POISON unexpected size: %"PRId64, (int64_t)res); } =20 static void uffd_poison_handle_fault( - struct uffd_msg *msg, struct uffd_args *args) + uffd_global_test_opts_t *gopts, struct uffd_msg *msg, struct uffd_args *a= rgs) { unsigned long offset; =20 @@ -1017,17 +999,17 @@ static void uffd_poison_handle_fault( (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; - offset &=3D ~(page_size-1); + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; + offset &=3D ~(gopts->page_size-1); =20 /* Odd pages -> copy zeroed page; even pages -> poison. */ - if (offset & page_size) - copy_page(uffd, offset, false); + if (offset & gopts->page_size) + copy_page(gopts, offset, false); else - do_uffdio_poison(uffd, offset); + do_uffdio_poison(gopts, offset); } =20 -static void uffd_poison_test(uffd_test_args_t *targs) +static void uffd_poison_test(uffd_global_test_opts_t *gopts, uffd_test_arg= s_t *targs) { pthread_t uffd_mon; char c; @@ -1036,10 +1018,12 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) unsigned long nr_sigbus =3D 0; unsigned long nr; =20 - fcntl(uffd, F_SETFL, uffd_flags | O_NONBLOCK); + args.gopts =3D gopts; =20 - uffd_register_poison(uffd, area_dst, nr_pages * page_size); - memset(area_src, 0, nr_pages * page_size); + fcntl(gopts->uffd, F_SETFL, gopts->uffd_flags | O_NONBLOCK); + + uffd_register_poison(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopt= s->page_size); + memset(gopts->area_src, 0, gopts->nr_pages * gopts->page_size); =20 args.handle_fault =3D uffd_poison_handle_fault; if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) @@ -1051,9 +1035,9 @@ static void uffd_poison_test(uffd_test_args_t *targs) if (sigaction(SIGBUS, &act, 0)) err("sigaction"); =20 - for (nr =3D 0; nr < nr_pages; ++nr) { - unsigned long offset =3D nr * page_size; - const char *bytes =3D (const char *) area_dst + offset; + for (nr =3D 0; nr < gopts->nr_pages; ++nr) { + unsigned long offset =3D nr * gopts->page_size; + const char *bytes =3D (const char *) gopts->area_dst + offset; const char *i; =20 if (sigsetjmp(*sigbuf, 1)) { @@ -1066,28 +1050,30 @@ static void uffd_poison_test(uffd_test_args_t *targ= s) continue; } =20 - for (i =3D bytes; i < bytes + page_size; ++i) { + for (i =3D bytes; i < bytes + gopts->page_size; ++i) { if (*i) err("nonzero byte in area_dst (%p) at %p: %u", - area_dst, i, *i); + gopts->area_dst, i, *i); } } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("pthread_join()"); =20 - if (nr_sigbus !=3D nr_pages / 2) + if (nr_sigbus !=3D gopts->nr_pages / 2) err("expected to receive %lu SIGBUS, actually received %lu", - nr_pages / 2, nr_sigbus); + gopts->nr_pages / 2, nr_sigbus); =20 uffd_test_pass(); } =20 static void -uffd_move_handle_fault_common(struct uffd_msg *msg, struct uffd_args *args, - unsigned long len) +uffd_move_handle_fault_common(uffd_global_test_opts_t *gopts, + struct uffd_msg *msg, + struct uffd_args *args, + unsigned long len) { unsigned long offset; =20 @@ -1098,28 +1084,31 @@ uffd_move_handle_fault_common(struct uffd_msg *msg,= struct uffd_args *args, (UFFD_PAGEFAULT_FLAG_WP | UFFD_PAGEFAULT_FLAG_MINOR | UFFD_PAGEFAULT_= FLAG_WRITE)) err("unexpected fault type %llu", msg->arg.pagefault.flags); =20 - offset =3D (char *)(unsigned long)msg->arg.pagefault.address - area_dst; + offset =3D (char *)(unsigned long)msg->arg.pagefault.address - gopts->are= a_dst; offset &=3D ~(len-1); =20 - if (move_page(uffd, offset, len)) + if (move_page(gopts, offset, len)) args->missing_faults++; } =20 -static void uffd_move_handle_fault(struct uffd_msg *msg, +static void uffd_move_handle_fault(uffd_global_test_opts_t *gopts, struct = uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, page_size); + uffd_move_handle_fault_common(gopts, msg, args, gopts->page_size); } =20 -static void uffd_move_pmd_handle_fault(struct uffd_msg *msg, +static void uffd_move_pmd_handle_fault(uffd_global_test_opts_t *gopts, str= uct uffd_msg *msg, struct uffd_args *args) { - uffd_move_handle_fault_common(msg, args, read_pmd_pagesize()); + uffd_move_handle_fault_common(gopts, msg, args, read_pmd_pagesize()); } =20 static void -uffd_move_test_common(uffd_test_args_t *targs, unsigned long chunk_size, - void (*handle_fault)(struct uffd_msg *msg, struct uffd_args *args)) +uffd_move_test_common( + uffd_global_test_opts_t *gopts, uffd_test_args_t *targs, + unsigned long chunk_size, void (*handle_fault)(struct uffd_global_test_o= pts *gopts, + struct uffd_msg *msg, struct uffd_args *args) +) { unsigned long nr; pthread_t uffd_mon; @@ -1131,11 +1120,13 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, unsigned long src_offs =3D 0; unsigned long dst_offs =3D 0; =20 + args.gopts =3D gopts; + /* Prevent source pages from being mapped more than once */ - if (madvise(area_src, nr_pages * page_size, MADV_DONTFORK)) + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_DON= TFORK)) err("madvise(MADV_DONTFORK) failure"); =20 - if (uffd_register(uffd, area_dst, nr_pages * page_size, + if (uffd_register(gopts->uffd, gopts->area_dst, gopts->nr_pages * gopts->= page_size, true, false, false)) err("register failure"); =20 @@ -1143,22 +1134,22 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &args)) err("uffd_poll_thread create"); =20 - step_size =3D chunk_size / page_size; - step_count =3D nr_pages / step_size; + step_size =3D chunk_size / gopts->page_size; + step_count =3D gopts->nr_pages / step_size; =20 - if (chunk_size > page_size) { - char *aligned_src =3D ALIGN_UP(area_src, chunk_size); - char *aligned_dst =3D ALIGN_UP(area_dst, chunk_size); + if (chunk_size > gopts->page_size) { + char *aligned_src =3D ALIGN_UP(gopts->area_src, chunk_size); + char *aligned_dst =3D ALIGN_UP(gopts->area_dst, chunk_size); =20 - if (aligned_src !=3D area_src || aligned_dst !=3D area_dst) { - src_offs =3D (aligned_src - area_src) / page_size; - dst_offs =3D (aligned_dst - area_dst) / page_size; + if (aligned_src !=3D gopts->area_src || aligned_dst !=3D gopts->area_dst= ) { + src_offs =3D (aligned_src - gopts->area_src) / gopts->page_size; + dst_offs =3D (aligned_dst - gopts->area_dst) / gopts->page_size; step_count--; } - orig_area_src =3D area_src; - orig_area_dst =3D area_dst; - area_src =3D aligned_src; - area_dst =3D aligned_dst; + orig_area_src =3D gopts->area_src; + orig_area_dst =3D gopts->area_dst; + gopts->area_src =3D aligned_src; + gopts->area_dst =3D aligned_dst; } =20 /* @@ -1172,34 +1163,34 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, =20 /* Check area_src content */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); - if (count !=3D count_verify[src_offs + nr + i]) + count =3D *area_count(gopts->area_src, nr + i, gopts); + if (count !=3D gopts->count_verify[src_offs + nr + i]) err("nr %lu source memory invalid %llu %llu\n", - nr + i, count, count_verify[src_offs + nr + i]); + nr + i, count, gopts->count_verify[src_offs + nr + i]); } =20 /* Faulting into area_dst should move the page or the huge page */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_dst, nr + i); - if (count !=3D count_verify[dst_offs + nr + i]) + count =3D *area_count(gopts->area_dst, nr + i, gopts); + if (count !=3D gopts->count_verify[dst_offs + nr + i]) err("nr %lu memory corruption %llu %llu\n", - nr, count, count_verify[dst_offs + nr + i]); + nr, count, gopts->count_verify[dst_offs + nr + i]); } =20 /* Re-check area_src content which should be empty */ for (i =3D 0; i < step_size; i++) { - count =3D *area_count(area_src, nr + i); + count =3D *area_count(gopts->area_src, nr + i, gopts); if (count !=3D 0) err("nr %lu move failed %llu %llu\n", - nr, count, count_verify[src_offs + nr + i]); + nr, count, gopts->count_verify[src_offs + nr + i]); } } - if (chunk_size > page_size) { - area_src =3D orig_area_src; - area_dst =3D orig_area_dst; + if (chunk_size > gopts->page_size) { + gopts->area_src =3D orig_area_src; + gopts->area_dst =3D orig_area_dst; } =20 - if (write(pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) + if (write(gopts->pipefd[1], &c, sizeof(c)) !=3D sizeof(c)) err("pipe write"); if (pthread_join(uffd_mon, NULL)) err("join() failed"); @@ -1210,31 +1201,31 @@ uffd_move_test_common(uffd_test_args_t *targs, unsi= gned long chunk_size, uffd_test_pass(); } =20 -static void uffd_move_test(uffd_test_args_t *targs) +static void uffd_move_test(uffd_global_test_opts_t *gopts, uffd_test_args_= t *targs) { - uffd_move_test_common(targs, page_size, uffd_move_handle_fault); + uffd_move_test_common(gopts, targs, gopts->page_size, uffd_move_handle_fa= ult); } =20 -static void uffd_move_pmd_test(uffd_test_args_t *targs) +static void uffd_move_pmd_test(uffd_global_test_opts_t *gopts, uffd_test_a= rgs_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_HUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) err("madvise(MADV_HUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static void uffd_move_pmd_split_test(uffd_test_args_t *targs) +static void uffd_move_pmd_split_test(uffd_global_test_opts_t *gopts, uffd_= test_args_t *targs) { - if (madvise(area_dst, nr_pages * page_size, MADV_NOHUGEPAGE)) + if (madvise(gopts->area_dst, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) err("madvise(MADV_NOHUGEPAGE) failure"); - uffd_move_test_common(targs, read_pmd_pagesize(), + uffd_move_test_common(gopts, targs, read_pmd_pagesize(), uffd_move_pmd_handle_fault); } =20 -static int prevent_hugepages(const char **errmsg) +static int prevent_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_NOHUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_NOH= UGEPAGE)) { /* Ignore only if CONFIG_TRANSPARENT_HUGEPAGE=3Dn */ if (errno !=3D EINVAL) { if (errmsg) @@ -1245,10 +1236,10 @@ static int prevent_hugepages(const char **errmsg) return 0; } =20 -static int request_hugepages(const char **errmsg) +static int request_hugepages(uffd_global_test_opts_t *gopts, const char **= errmsg) { /* This should be done before source area is populated */ - if (madvise(area_src, nr_pages * page_size, MADV_HUGEPAGE)) { + if (madvise(gopts->area_src, gopts->nr_pages * gopts->page_size, MADV_HUG= EPAGE)) { if (errmsg) { *errmsg =3D (errno =3D=3D EINVAL) ? "CONFIG_TRANSPARENT_HUGEPAGE is not set" : @@ -1272,13 +1263,17 @@ struct uffd_test_case_ops uffd_move_test_pmd_case_o= ps =3D { * Note that _UFFDIO_ZEROPAGE is tested separately in the zeropage test. */ static void -do_register_ioctls_test(uffd_test_args_t *args, bool miss, bool wp, bool m= inor) +do_register_ioctls_test(uffd_global_test_opts_t *gopts, + uffd_test_args_t *args, + bool miss, + bool wp, + bool minor) { uint64_t ioctls =3D 0, expected =3D BIT_ULL(_UFFDIO_WAKE); mem_type_t *mem_type =3D args->mem_type; int ret; =20 - ret =3D uffd_register_with_ioctls(uffd, area_dst, page_size, + ret =3D uffd_register_with_ioctls(gopts->uffd, gopts->area_dst, gopts->pa= ge_size, miss, wp, minor, &ioctls); =20 /* @@ -1309,18 +1304,18 @@ do_register_ioctls_test(uffd_test_args_t *args, boo= l miss, bool wp, bool minor) "(miss=3D%d, wp=3D%d, minor=3D%d): expected=3D0x%"PRIx64", " "returned=3D0x%"PRIx64, miss, wp, minor, expected, ioctls); =20 - if (uffd_unregister(uffd, area_dst, page_size)) + if (uffd_unregister(gopts->uffd, gopts->area_dst, gopts->page_size)) err("unregister"); } =20 -static void uffd_register_ioctls_test(uffd_test_args_t *args) +static void uffd_register_ioctls_test(uffd_global_test_opts_t *gopts, uffd= _test_args_t *args) { int miss, wp, minor; =20 for (miss =3D 0; miss <=3D 1; miss++) for (wp =3D 0; wp <=3D 1; wp++) for (minor =3D 0; minor <=3D 1; minor++) - do_register_ioctls_test(args, miss, wp, minor); + do_register_ioctls_test(gopts, args, miss, wp, minor); =20 uffd_test_pass(); } @@ -1532,6 +1527,27 @@ int main(int argc, char *argv[]) } for (j =3D 0; j < n_mems; j++) { mem_type =3D &mem_types[j]; + + // Initialize global test options + uffd_global_test_opts_t gopts; + + gopts.map_shared =3D mem_type->shared; + uffd_test_ops =3D mem_type->mem_ops; + uffd_test_case_ops =3D test->test_case_ops; + + if (mem_type->mem_flag & (MEM_HUGETLB_PRIVATE | MEM_HUGETLB)) + gopts.page_size =3D default_huge_page_size(); + else + gopts.page_size =3D psize(); + + /* Ensure we have at least 2 pages */ + gopts.nr_pages =3D MAX(UFFD_TEST_MEM_SIZE, gopts.page_size * 2) / gopts= .page_size; + /* TODO: remove this global var.. it's so ugly */ + gopts.nr_parallel =3D 1; + + /* Initialize test arguments */ + args.mem_type =3D mem_type; + if (!(test->mem_targets & mem_type->mem_flag)) continue; =20 @@ -1546,13 +1562,12 @@ int main(int argc, char *argv[]) uffd_test_skip("feature missing"); continue; } - if (uffd_setup_environment(&args, test, mem_type, - &errmsg)) { + if (uffd_test_ctx_init(&gopts, test->uffd_feature_required, &errmsg)) { uffd_test_skip(errmsg); continue; } - test->uffd_fn(&args); - uffd_test_ctx_clear(); + test->uffd_fn(&gopts, &args); + uffd_test_ctx_clear(&gopts); } } =20 --=20 2.20.1 From nobody Sat Feb 7 17:09:24 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F571214A9B; Thu, 1 May 2025 16:40:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117634; cv=none; b=Xt6Wpb3Umhme0W1j4k+C19aXzrg6DQavuMG+Oyafet51vkqw/geA6MT0cpl15SCEWPYjONGiCSKwkvIO7FAsQpGZ5POJ9Z6oea33ZdHzAiC/jMbisqfkbAwSaJUFXE+5P1KhkxpbWK+Q9/rQAlcgf21S1SvVQjlE6/Htx52n1ko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746117634; c=relaxed/simple; bh=ZLXpMNhjbMhJUyuguYDHAZXGWQC9HHtDncPQGBlHJ/E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lYrrI1ZanqQtLSzeqUbvlrZT9TQ/jQ+/9Z+D8ywwZyceh8UzHWBU6S3Q+z4JbIEWBWT7g5+OD6qNMgZbOhDcgA+P3WjkSdqJFlpLWGmv0aTJ5k+dYrXkbsv1kf8meXrBleesKeSSvr4WJHXNoUns/e194BWsnAxqUaFsiqWYCgg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=K4XogXnw; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="K4XogXnw" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-22c33677183so12862735ad.2; Thu, 01 May 2025 09:40:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746117631; x=1746722431; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vgPGTCQFUFW1kU3FlNW3yK7JtTBlo5vps0nfxBSj1K4=; b=K4XogXnwMsqfV9MVGYxmuwkX6V11HV3v1tQeN+vcb5n+v5ZNIjPeRCioE/VI5t0aro a8i2qRsi6KYyT6qlCS92EG9fWyFyDDT9xYViscH0Aps/l+CIoAH/DV1l3qI5yncQ/gjv 56d+5xKxMwZE648jpqj4BPvwy6RZES72CE3TG+9W2aAGlZrXD8v3D6eSTmF8qxUnVVd9 vZiLvtn4odUKPQfJ3qyk/hffAC0RHfrFtDBTK9JuKaJR1/Icd2WKxBauk6dn7H0ef8pX 2SafFyeKt6zYg6yo51DT9l/pl1Wj2chEVHq7nrKfi7lxeuPxQ0TAeXhvVTol/AY+HOmP Vtog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746117631; x=1746722431; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vgPGTCQFUFW1kU3FlNW3yK7JtTBlo5vps0nfxBSj1K4=; b=Aym6udv8GE2d108v6cpLcvi8xeOu5Lp1yESWKaKj5YVEhoMyD6ww/JTU06Ce99knn0 SdiphKuSLj38sfN/o2z1TrICsSYfRp4NmOF0tDmhcEtlBODRMNDPBbes+xBmPAoSrzk9 Qa7k6LIqZO2uaxQlGRURNnbvDdpMoehqA6H91jtNciCcjuPsFlV7A4wkn0xoloJoK9ZM 3oqLMnIVoQ8RNpLveLcDLRJywqhxeO3K8fB9LrgyjsAeSQRrR4064f+YhfZHetnHfeb3 Hoklb3uvtg8J6g3/AK6E2FelSEKqQ4VWNI57rXOHrURVuMtiPht+axPKdm4W4txszRzu elmw== X-Forwarded-Encrypted: i=1; AJvYcCUuRFP94617DASu9v2KHsblndlmOpD2OXA9OPTE7RrvxVE1DxHnq7+Yh+A7XsAJEVR3ZXf19WAWFp47v/Y=@vger.kernel.org, AJvYcCW3Zg7dmYRSfPIduv8I8IYlrAiPTkFc9icG0ztSKSn2dR7ggB0rnhM55moo3ZKGri+3GG0rtnajNJFHrDQbUUwO@vger.kernel.org X-Gm-Message-State: AOJu0YwQVX8plJ0B3cYHJTiG4/2CXYA8XnL9xhW9Wyi3nOMAssOi82pY hutq4SuHqegHT09TIOYeqIqzNYBzhj9ou/blIyQPEm/LQgnReQk= X-Gm-Gg: ASbGncufdcqQ85doPB/NCT81FbWjS7NJFPtDzZXiC51PP2/z88VWS+Z8nWpJHTLyuqh SLwGQx7QUoqkEHpvsAa2bHYoxYcbfd892iFRosu0go8sdh5M12ITK1ixa9G8mbxugv/xtE0twFu LrMNKFj82WkkibktC3cYAspRu9kWHXe9kZCh33qaIDO2kFrhhsxEYNBcS94eV1G/n45gWr269Rq jMZHmXNR+H/HN5Y+HgLW5o1KeEm+Fp8ZBDNwcZ+LG1yQ/2eGaMO7lZc6HNVHDleZJyQTmJmoKDR TSXrq1HShUSFFOQqUx8RahoArWwrimIB6GadykKUzx5pIwlTIseQNJgVHihi X-Google-Smtp-Source: AGHT+IGfA4+30n/veK7qPrC5roeakakzXs1CCl29OolxjuG2a7TLj+Et6zwpS35yrIvta2uRaadkVQ== X-Received: by 2002:a17:903:11d0:b0:223:53fb:e1dd with SMTP id d9443c01a7336-22e040b014cmr48138075ad.9.1746117630722; Thu, 01 May 2025 09:40:30 -0700 (PDT) Received: from debian.ujwal.com ([223.185.132.49]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-22e0bc6e678sm8797525ad.132.2025.05.01.09.40.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 May 2025 09:40:30 -0700 (PDT) From: Ujwal Kundur To: akpm@linux-foundation.org, peterx@redhat.com, shuah@kernel.org Cc: linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Ujwal Kundur Subject: [PATCH 4/4] selftests/mm/uffd: Swap global variables with global test opts Date: Thu, 1 May 2025 22:08:27 +0530 Message-Id: <20250501163827.2598-5-ujwal.kundur@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20250501163827.2598-1-ujwal.kundur@gmail.com> References: <20250501163827.2598-1-ujwal.kundur@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We use the refactored uffd_global_test_opts instead of global variables for uffd-wp-mremap tests. Signed-off-by: Ujwal Kundur Tested-by: Brendan Jackman --- tools/testing/selftests/mm/uffd-wp-mremap.c | 23 +++++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/tools/testing/selftests/mm/uffd-wp-mremap.c b/tools/testing/se= lftests/mm/uffd-wp-mremap.c index c2ba7d46c7b4..a24f35728ac3 100644 --- a/tools/testing/selftests/mm/uffd-wp-mremap.c +++ b/tools/testing/selftests/mm/uffd-wp-mremap.c @@ -157,7 +157,11 @@ static bool range_is_swapped(void *addr, size_t size) return true; } =20 -static void test_one_folio(size_t size, bool private, bool swapout, bool h= ugetlb) +static void test_one_folio(uffd_global_test_opts_t *gopts, + size_t size, + bool private, + bool swapout, + bool hugetlb) { struct uffdio_writeprotect wp_prms; uint64_t features =3D 0; @@ -181,21 +185,21 @@ static void test_one_folio(size_t size, bool private,= bool swapout, bool hugetlb } =20 /* Register range for uffd-wp. */ - if (userfaultfd_open(&features)) { + if (userfaultfd_open(gopts, &features)) { if (errno =3D=3D ENOENT) ksft_test_result_skip("userfaultfd not available\n"); else ksft_test_result_fail("userfaultfd_open() failed\n"); goto out; } - if (uffd_register(uffd, mem, size, false, true, false)) { + if (uffd_register(gopts->uffd, mem, size, false, true, false)) { ksft_test_result_fail("uffd_register() failed\n"); goto out; } wp_prms.mode =3D UFFDIO_WRITEPROTECT_MODE_WP; wp_prms.range.start =3D (uintptr_t)mem; wp_prms.range.len =3D size; - if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { + if (ioctl(gopts->uffd, UFFDIO_WRITEPROTECT, &wp_prms)) { ksft_test_result_fail("ioctl(UFFDIO_WRITEPROTECT) failed\n"); goto out; } @@ -242,9 +246,9 @@ static void test_one_folio(size_t size, bool private, b= ool swapout, bool hugetlb out: if (mem) munmap(mem, size); - if (uffd >=3D 0) { - close(uffd); - uffd =3D -1; + if (gopts->uffd >=3D 0) { + close(gopts->uffd); + gopts->uffd =3D -1; } } =20 @@ -336,6 +340,7 @@ static const struct testcase testcases[] =3D { =20 int main(int argc, char **argv) { + uffd_global_test_opts_t gopts; struct thp_settings settings; int i, j, plan =3D 0; =20 @@ -367,8 +372,8 @@ int main(int argc, char **argv) const struct testcase *tc =3D &testcases[i]; =20 for (j =3D 0; j < *tc->nr_sizes; j++) - test_one_folio(tc->sizes[j], tc->private, tc->swapout, - tc->hugetlb); + test_one_folio(&gopts, tc->sizes[j], tc->private, + tc->swapout, tc->hugetlb); } =20 /* If THP is supported, restore original THP settings. */ --=20 2.20.1