From nobody Thu Apr 9 12:06:26 2026 Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 558AA363C62 for ; Mon, 2 Mar 2026 11:47:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772452068; cv=none; b=KyCf6xn9cLl4dpIDoV0SBqAMurV76/duOLuzham4JxH3/53e/1zkXgBq180sWk4F2ETkZs4DFRqCnGI1RXIItffHSm2Tef07YOd1wy5XxXRkhcnO2Lo0rVyuberI1QaUrXuP95cpkdImt4h3+7IxW5C1tp5/rU6mCOUo75ycf4E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772452068; c=relaxed/simple; bh=gDMjY10zyyDSIzFnbHmbiyrlI3HQjt7H4HPJIQuPZuA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y4V2CQQv4A/b16ZZ96TnEWCqNVje+gXZVP2n3wXxy6dHiPCpesUSlPet8+w3NBvkbkQyRfISpzH/f2Pbao3z0oOBpkmzR3LWLArga2XynDYjDoTcS6s9LOgdQ6OXSJfT/KJv4R/IUCgVvLweo7W46d1Lb3UKc2calaFZRh5wvR0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GjYMeTm0; arc=none smtp.client-ip=209.85.208.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GjYMeTm0" Received: by mail-lj1-f177.google.com with SMTP id 38308e7fff4ca-3870902760cso67898471fa.3 for ; Mon, 02 Mar 2026 03:47:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772452063; x=1773056863; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HFo3Ih7QU9IZxDZbOcJePTkGiOdWme4psvs/TRTzUGA=; b=GjYMeTm047T6t1RFhL+1q/0wgQGE74G/eMtoOFPI6cjH797+QCec2fRuU3MXP8cCJ3 /JdGPdjmfmAkSXb61oOPO2rT+s2Fuz/TGeho7WBZuatLG9uFMR3RbrPr2hWLIdCHjo+W tJv6Kub6bNIyJsXVGi9miGTrxTYBGGunJPQnC2cwJVNGNez2e2cNwiTJq2WnxoEvo5LV BGqYtdE+FHruCjaUMQjpctmPvDtUUuOIs91iX2Nb2O09OCPmn+cdLGg7zU0RR6C4L8yg g4QfF1oKdhYs1xWIhbdOEHsuqgiXccxFhl8rI8Esz6c09pQB8QgrEt1CohC3O56wmo0b fXvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772452063; x=1773056863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=HFo3Ih7QU9IZxDZbOcJePTkGiOdWme4psvs/TRTzUGA=; b=Ggx4AgSOm5kMU+EQkwNFFlN6gT/dwruX/U3kHlKpD4hQQ8R73tmEScG/CdkVPelqzT vd2r2cAw8ayt9l8fBbLD6sn+hk4bXpk3f8PPOi9033osWefl0bWZqXzm2YboEIuDKD9r H1MN3kq9TnkqFuGaWXe8xgOcY51V6ft0cmjofjLeiRue9zbdmSav3ygksPZLKoSrAwN4 5++05FXaX4Y8tDvFJsF06FCR/wD/WDmLPjBj6slR0dspBbomp2y9K6gAXTAsSfvFeiHj pXDOQfreKJAjtNptZikTlv8kdfQAK3UbxxON0mPhuJOxTzxQIM0qWqEJFmQFJNZ4TacM QX2Q== X-Forwarded-Encrypted: i=1; AJvYcCUpbpt79Jpp2TrUL/JjyCAsBYp6jCf1RNlnJyXRB1JU21BUxEeqYjPvjR+xDeQB5tV8cPOjIUP7dWJF4tg=@vger.kernel.org X-Gm-Message-State: AOJu0YyoKjFGqQ5c1YEvabv0/B8mKekQJ4IRbLQI5kZ5MH9YVsz3cbIs M0WklZVuCvLXJtmBpi8bupC0jg2cbK7QH949nWBt5yxSk0iWCFmJZSFQ X-Gm-Gg: ATEYQzxm/PQfPeMPE7N/EzlVuPxGPSkk6a6R/biDj3ruNV5rL5TmzJA4iKpKAUIKYjO Jx2zXalCrtys8k5k0PmDY9s53LauUALJOR+DClJgAkebMLkd1Q2cJiHw41IBQ95UfEUx2kFjDe3 //iWfBgJUEoFveqUa6uIiY9B3HTaUwQOczgf7Rwx8UI3yo88K0iKFJSr2N1RjlEnIiTbj+Q/IuE nI1H0QWlLb4djHoZcM4RAI5vufL52KiiVXAWXxByAWmdHmTJUDPSFE8Do8JyJ2DfEm0JblSEVTn vhUR+DI13e4zQrUKtn/YPXWb9IZDuX464taS6adyZXbYfm2tXrNmokNOE9XUxWObNf0gI4GmpWT McoLZY2y/UQo3r9j97OlC/TTxMafm5sd1XJPB7T5j/Vpu2Z2pxPv2HOFuJxWGt9FkHqWFyzJugJ 7wXfcVOe63YLPCoGU= X-Received: by 2002:a05:651c:25d5:10b0:389:fc6b:943f with SMTP id 38308e7fff4ca-389ff119cebmr57137381fa.11.1772452063156; Mon, 02 Mar 2026 03:47:43 -0800 (PST) Received: from localhost.localdomain ([2001:9b1:d5a0:a500::24b]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5a115bca02asm1603412e87.25.2026.03.02.03.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2026 03:47:42 -0800 (PST) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: Michal Hocko , Mikulas Patocka , Vishal Moola , Baoquan He , LKML , Uladzislau Rezki Subject: [PATCH] vmalloc: support __GFP_RETRY_MAYFAIL and __GFP_NORETRY Date: Mon, 2 Mar 2026 12:47:40 +0100 Message-ID: <20260302114740.2668450-2-urezki@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260302114740.2668450-1-urezki@gmail.com> References: <20260302114740.2668450-1-urezki@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Michal Hocko __GFP_RETRY_MAYFAIL and __GFP_NORETRY haven't been supported so far because their semantic (i.e. to not trigger OOM killer) is not possible with the existing vmalloc page table allocation which is allowing for the OOM killer. Example: __vmalloc(size, GFP_KERNEL | __GFP_RETRY_MAYFAIL); vmalloc_test/55 invoked oom-killer: gfp_mask=3D0x40dc0( GFP_KERNEL|__GFP_ZERO|__GFP_COMP), order=3D0, oom_score_adj=3D0 active_anon:0 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:0 isolated_file:0 unevictable:0 dirty:0 writeback:0 slab_reclaimable:700 slab_unreclaimable:33708 mapped:0 shmem:0 pagetables:5174 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:850 free_pcp:319 free_cma:0 CPU: 4 UID: 0 PID: 639 Comm: vmalloc_test/55 ... Hardware name: QEMU Standard PC (i440FX + PIIX, ... Call Trace: dump_stack_lvl+0x5d/0x80 dump_header+0x43/0x1b3 out_of_memory.cold+0x8/0x78 __alloc_pages_slowpath.constprop.0+0xef5/0x1130 __alloc_frozen_pages_noprof+0x312/0x330 alloc_pages_mpol+0x7d/0x160 alloc_pages_noprof+0x50/0xa0 __pte_alloc_kernel+0x1e/0x1f0 ... There are usecases for these modifiers when a large allocation request should rather fail than trigger OOM killer which wouldn't be able to handle the situation anyway [1]. While we cannot change existing page table allocation code easily we can piggy back on scoped NOWAIT allocation for them that we already have in place. The rationale is that the bulk of the consumed memory is sitting in pages backing the vmalloc allocation. Page tables are only participating a tiny fraction. Moreover page tables for virtually allocated areas are never reclaimed so the longer the system runs to less likely they are. It makes sense to allow an approximation of __GFP_RETRY_MAYFAIL and __GFP_NORETRY even if the page table allocation part is much weaker. This doesn't break the failure mode while it allows for the no OOM semantic. [1] https://lore.kernel.org/all/32bd9bed-a939-69c4-696d-f7f9a5fe31d8@redhat= .com/T/#u Tested-by: Uladzislau Rezki (Sony) Signed-off-by: Michal Hocko Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a06f4b3ea367..975592b0ec89 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3798,6 +3798,8 @@ static void defer_vm_area_cleanup(struct vm_struct *a= rea) * non-blocking (no __GFP_DIRECT_RECLAIM) - memalloc_noreclaim_save() * GFP_NOFS - memalloc_nofs_save() * GFP_NOIO - memalloc_noio_save() + * __GFP_RETRY_MAYFAIL, __GFP_NORETRY - memalloc_noreclaim_save() + * to prevent OOMs * * Returns a flag cookie to pair with restore. */ @@ -3806,7 +3808,8 @@ memalloc_apply_gfp_scope(gfp_t gfp_mask) { unsigned int flags =3D 0; =20 - if (!gfpflags_allow_blocking(gfp_mask)) + if (!gfpflags_allow_blocking(gfp_mask) || + (gfp_mask & (__GFP_RETRY_MAYFAIL | __GFP_NORETRY))) flags =3D memalloc_noreclaim_save(); else if ((gfp_mask & (__GFP_FS | __GFP_IO)) =3D=3D __GFP_IO) flags =3D memalloc_nofs_save(); @@ -3940,7 +3943,8 @@ static void *__vmalloc_area_node(struct vm_struct *ar= ea, gfp_t gfp_mask, * GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP. */ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ - __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_NOFAIL | __GFP_ZERO |\ + __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP) =20 @@ -3971,12 +3975,15 @@ static gfp_t vmalloc_fix_flags(gfp_t flags) * virtual range with protection @prot. * * Supported GFP classes: %GFP_KERNEL, %GFP_ATOMIC, %GFP_NOWAIT, - * %GFP_NOFS and %GFP_NOIO. Zone modifiers are not supported. + * %__GFP_RETRY_MAYFAIL, %__GFP_NORETRY, %GFP_NOFS and %GFP_NOIO. + * Zone modifiers are not supported. * Please note %GFP_ATOMIC and %GFP_NOWAIT are supported only * by __vmalloc(). * - * Retry modifiers: only %__GFP_NOFAIL is supported; %__GFP_NORETRY - * and %__GFP_RETRY_MAYFAIL are not supported. + * Retry modifiers: only %__GFP_NOFAIL is fully supported; + * %__GFP_NORETRY and %__GFP_RETRY_MAYFAIL are supported with limitation, + * i.e. page tables are allocated with NOWAIT semantic so they might fail + * under moderate memory pressure. * * %__GFP_NOWARN can be used to suppress failure messages. * --=20 2.47.3