From nobody Mon Nov 25 22:33:49 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFE791CC150 for ; Wed, 23 Oct 2024 17:08:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703286; cv=none; b=u8EfTtzjbtUxRKpm+HGq4SrxUIchY+Sp1KZNNFsBQy4IBVeRkODL99TeQmcL2djHFfnZ2ADatdwSSd4gMUPsw0tvzjknXF7aRHaDq5AAnwRBzzB/Pr1s+ez2UlAJOdmBXPFd0fzE91+MMyM+vGAv+VhoFAQW75pt+su4zi3Xd4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703286; c=relaxed/simple; bh=oPiAZk7sAUwyc2ZRzUfKOKHVYouPvs61/fhZTZAzOCM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pE+Sj9Xt2x88R344zVb7JyQfkdnBtTkndqQOkyv1wy+/+5o1Oo383GiYt1aS1nBvfzkDTm9ptHXlxAAa06u9CT5pzLaWIE8id51GoEZQytZF6fsGQka9GAbiD0qENq3uDaSKM2BkeFGJYv1zrCgh95G7z+NtRi7SfrJ8QhDoOSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Qx652BjR; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Qx652BjR" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e2bd4b29887so28678276.1 for ; Wed, 23 Oct 2024 10:08:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703284; x=1730308084; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cy4VkAGTnueEjEvw1TSjR77Zk0lX1faSNj+Oljrwa2Y=; b=Qx652BjRJlLBDhiwtgP7rFltqcVCeOz64AMfJMi1lfmNwRT3W6yJF2h66vz/NOdKvE iZX2r/6r71iZd3H6wP+kW3t8a0HXMP2YOsXTg3tOJAbvPBlyVSndmovQp59wBX4nnbV8 jejqeBwwjT7qIeRe08oGq0Cp83lM1+FOvt43wByN6sfYVEa14DIljwjJONrccZ7wFFHv fZNNV++nuuSoZIeX9NJpihOasG9IF1MQ4QIyGQFQJs0TmnWguyVWNAywiZshv+JiuOUm SH/lNNTmYiapZtyvrkKBGRczlN8UQ2w9m85DNYwABykAZaiWY/T+pOmFff90ZJdc96hQ 4Vlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703284; x=1730308084; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cy4VkAGTnueEjEvw1TSjR77Zk0lX1faSNj+Oljrwa2Y=; b=rICtYytP+u0TvUMwUp6cxmlSYYqK7+YBFDHzzAp4qMVAgRbkTwyewOYmFLk1dbunTd FRf+cY46mZDOf5dMfZMPMF2GilREFR/ZTz/dQHbOTmrSAnxsrdIoM9g5wwAkoRJmVwCT 2DfVtS3N4rH/csaBsE7/NrI+hw5dyBiXDmsWss9Y2NvAfKHy2+Rpu9vvr4OZ9uYvtOAz sd8zscU0x1v5GS5vykceqI6AUwmXQNKeB9i4NsteIU3T1Dd+5wwafSHuYzJoHbQjogyS va85kQQ96UoJgCRb/ClTRpnrPkENNATlYrdlpojBOLbJwmoMQetqLqsuCl9M5abcogGq GiCg== X-Forwarded-Encrypted: i=1; AJvYcCW9FO9mXauJv51J7/TtmQ1tlbw4DwqH9BaM5Pf2IvcaoPcsuVXFGGWoeiOpI6DlDiqSzt0f/gMtpktVPV4=@vger.kernel.org X-Gm-Message-State: AOJu0Yx88+TXTQcngLmcdHS6aNv8vnzbqhUjP7IX5PxAeFAdCvro32Vc jKbhJBTWSYPb1ASe6+vdWqUTMvyaYwxgsfl9WX23uMYNvYITzaXHOumsGvad7I7SOu4i1FbZWLd NGg== X-Google-Smtp-Source: AGHT+IG5P3EOBq94xAhs93OihXu6fn1+vh8Hj0b9Xh0nro0ohDjLm9Tm7P85Bm8yXiSBBXuIOUCegUy3OSE= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:7241:0:b0:e2e:2b0f:19fc with SMTP id 3f1490d57ef6-e2e3a631bf2mr3476276.4.1729703283882; Wed, 23 Oct 2024 10:08:03 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:54 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-2-surenb@google.com> Subject: [PATCH v4 1/6] maple_tree: add mas_for_each_rev() helper From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add mas_for_each_rev() function to iterate maple tree nodes in reverse order. Suggested-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett Reviewed-by: Pasha Tatashin --- include/linux/maple_tree.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 61c236850ca8..cbbcd18d4186 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -592,6 +592,20 @@ static __always_inline void mas_reset(struct ma_state = *mas) #define mas_for_each(__mas, __entry, __max) \ while (((__entry) =3D mas_find((__mas), (__max))) !=3D NULL) =20 +/** + * mas_for_each_rev() - Iterate over a range of the maple tree in reverse = order. + * @__mas: Maple Tree operation state (maple_state) + * @__entry: Entry retrieved from the tree + * @__min: minimum index to retrieve from the tree + * + * When returned, mas->index and mas->last will hold the entire range for = the + * entry. + * + * Note: may return the zero entry. + */ +#define mas_for_each_rev(__mas, __entry, __min) \ + while (((__entry) =3D mas_find_rev((__mas), (__min))) !=3D NULL) + #ifdef CONFIG_DEBUG_MAPLE_TREE enum mt_dump_format { mt_dump_dec, --=20 2.47.0.105.g07ac214952-goog From nobody Mon Nov 25 22:33:49 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 283CB1CEEAD for ; Wed, 23 Oct 2024 17:08:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703288; cv=none; b=qnz/lWWyiIRt7kxad2PlBwZvtTQ2FRfoEx9fvuedCW+u4SjuJrtq4TiTsYmoIApuqPk8sbsmW8jtmhRrtRz6S56/cp9VhDg1FKRNW9LRoJ0be0bYDBfKxx4zR1GCHuTKrTuIMDwb94AYA1Cr6xe8rUhvah4p9LlIxa5opbwzMs8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703288; c=relaxed/simple; bh=ZiQDRmQrDQydODmBhplTHGUqS5BFAUWK779ZQGEAgqA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rT3rlM/oC6OjJhodH2U2Y5PhZmCBvlXINLfNr463d1d4Ndo4JYiHdfaNaQDuvxnaREK8kVE/XF0/86zgySDxQV5eRE7QDWmoJ1Y8EMBgs7996TOUl2tPXVJTf15ZaQPwkQZ4JLLldYpIdxi1PIyf/2i/AaLLE5F2gzBuQXj9OLk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=H9VJE0Ix; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="H9VJE0Ix" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e2bd3e0a26cso69785276.0 for ; Wed, 23 Oct 2024 10:08:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703286; x=1730308086; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qzUZQuDC5qa2V/vpTubJjhmN2aD40kQ8oHuQLi+u1Z4=; b=H9VJE0IxnL+nTpuDEfsO0joE5Y4h/z1jDdQWH90pxx78iQW0+5bC8VfzuKV24bUIYs JsgRVJHzYRaU4yx6NTIfs/fAgAnNj7fSlc/35MqQINhe8/EqMqbbaEH62DQmNMmE6gnL DaB95933+EXSIGWDW+BGhswMHRZvrC+TTIq6nrsH8ZbcjWCcNKR5lmKJM+IE4k3c6PW/ aEv4xTCJOGgygaMJmnwWlmD0NYZVx9Vh19W3WDDgy/NpHuctojMwISl4M/GdPUjTxgSw vHXGz8+HaWxNq0oFjvVSSStkHGbOT3S2lwkPCsjCOQPWkTzZ8ZXH3TMK6atlhFefzaSZ 9ZNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703286; x=1730308086; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qzUZQuDC5qa2V/vpTubJjhmN2aD40kQ8oHuQLi+u1Z4=; b=mr+wERZvlYP32eSaEsu6iETWSwtPTIlRO4A0fLS8JMPJobkVBIC+JKYRtST4P1D62i vAdwCVEmOdNvhxMw9EIW5imGwSlD9G8SLCUMp7nazOSRw8y1/bOxQapsv48Y5CRZpWMN VQaVDpiawTRU8oVEwj3JcULxgLIM9zvMM0fKPkY86BR+0bsdAdiT7D19+wcIl2Gp0DDS hEHPK/Y6qoD69CF+BMyYtKOz9m06UEgTtmXYMNqmynRgC2PC+7uGWnraCSPESU7JZzVm SGYFJ1bFlEWyS4iXII59/Z7Ll1HLOKN7Ur8Z7refRoAnmrjdx2e6A7hDM3Tyf3yryB3I HCVw== X-Forwarded-Encrypted: i=1; AJvYcCUBpi5ylhWgN1sFJjz6GxFkM8IDGA6eZo1YP6PCXxdm98+uigyf8m61j28VWr8tIHwgpsu0POJuN39xpV0=@vger.kernel.org X-Gm-Message-State: AOJu0YyfjtHwYpuGjEsHfVTwxSBMB8poBHozYbd7YnpVxUE+3ZCnkEGY a47AyxlQiLujyM19uNQp8N/CH+PIZOEma8J8EUfy0UlnU0t2Rs8beqe8yVBTlwjciVbvZiaLB7A 0og== X-Google-Smtp-Source: AGHT+IEf9XVjhAQxbJksRoGGus3EwGVaSTpLfWlLEYi7zdwXtn6qn+dayeJKMPCmDwrDuVs0NH6hwtuuxI8= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:8289:0:b0:e2e:3401:ea0f with SMTP id 3f1490d57ef6-e2e3a6bbdffmr7278276.7.1729703286023; Wed, 23 Oct 2024 10:08:06 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:55 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-3-surenb@google.com> Subject: [PATCH v4 2/6] alloc_tag: introduce shutdown_mem_profiling helper function From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement a helper function to disable memory allocation profiling and use it when creation of /proc/allocinfo fails. Ensure /proc/allocinfo does not get created when memory allocation profiling is disabled. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- lib/alloc_tag.c | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 81e5f9a70f22..435aa837e550 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -8,6 +8,14 @@ #include #include =20 +#define ALLOCINFO_FILE_NAME "allocinfo" + +#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT +static bool mem_profiling_support __meminitdata =3D true; +#else +static bool mem_profiling_support __meminitdata; +#endif + static struct codetag_type *alloc_tag_cttype; =20 DEFINE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); @@ -144,9 +152,26 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags,= size_t count, bool can_sl return nr; } =20 +static void __init shutdown_mem_profiling(void) +{ + if (mem_alloc_profiling_enabled()) + static_branch_disable(&mem_alloc_profiling_key); + + if (!mem_profiling_support) + return; + + mem_profiling_support =3D false; +} + static void __init procfs_init(void) { - proc_create_seq("allocinfo", 0400, NULL, &allocinfo_seq_op); + if (!mem_profiling_support) + return; + + if (!proc_create_seq(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op))= { + pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME); + shutdown_mem_profiling(); + } } =20 static bool alloc_tag_module_unload(struct codetag_type *cttype, @@ -174,12 +199,6 @@ static bool alloc_tag_module_unload(struct codetag_typ= e *cttype, return module_unused; } =20 -#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT -static bool mem_profiling_support __meminitdata =3D true; -#else -static bool mem_profiling_support __meminitdata; -#endif - static int __init setup_early_mem_profiling(char *str) { bool enable; --=20 2.47.0.105.g07ac214952-goog From nobody Mon Nov 25 22:33:49 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0E801CFEB8 for ; Wed, 23 Oct 2024 17:08:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703291; cv=none; b=Hi+CkJPhs0YxHVEDFLCu+WBwoxYmjdU/rvBTFf5zCkjEgYy44Tl8S9u+TuM6MbNXrclYxMkF2biwDWDe60bAbZtt8mxcqbcCHLFflaeEjce/mI9jombukcJbQJFM9WMA20jG3qHo0/p0BkoklbXWAgazTK81bClsgWZeT4twARU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703291; c=relaxed/simple; bh=Ju/mMo1I4q21dOJxY7i8P69ipvgzGOVpajNZqSso3Pg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=iyh5Jc2Uy3QZkJZXQIdVHDi3Hq1zwNe28cR/NPK5tCHUQW0+JMGTck5ryfOSuAwrRr4kKoWcWsJgmRvGBbD1IPcTlbxDOQ/JMyUNmLXElo9QW1y4fG76gmyYZSzCy5OzlEO3jvMwF2pjyPYh7fxfAPbVuy0Th02QF3R1+tCp2rI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=weSZPkSM; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="weSZPkSM" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e28fdb4f35fso54765276.0 for ; Wed, 23 Oct 2024 10:08:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703288; x=1730308088; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BDYzGeX72WULGwXnJI1q90ubF1+yTH7FuZO2allpANU=; b=weSZPkSMP7qHUt3YUK39jWkdl0wSepr88yKJ9CNvjiMhI5/8D8wkbnPxWXXMy0EZnA BInTrUgXv5bsxHazQKzUzjxYHseNMzERgJlghfwHrBKvgfqEBXHgUY1k+ZUe2Zwvz4R/ hevd6277d3LogY6YIwvm7M67yzhnyyiLI5M3tzOelZvnmo4lvEO8T5UJYVbAX5nW0K4h XgD33o1wD5LuQaU5DxBXKGG6OLmbCdLnF6GPEfX4u84gaktWIHVNx6HXE5QYqpu73Be5 LCvUrvUnllgA00zfCeZ77MzCq8AdTcYUOV7q540bDI/jQHcpln+Hw1GI6uBYggnehsaD vRRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703288; x=1730308088; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BDYzGeX72WULGwXnJI1q90ubF1+yTH7FuZO2allpANU=; b=v/O/Xp+aGcO1gEVw6p1R4x6sxXDNuNsVapuEariymEledagJp2hYfKRyb0gP1fDecs ytgew7qnCzHz/Lguep71k06Vo3RfPtBo6EGF8myKFF/AyaGjVCB+j/C5D3wAqFzPgiTC dSGwSDNGvmYcjRkUdtXdrJ7/AdfvH3PSv5rPabKFgi9Vl5jMTnxhkTh/ANiWI4ey2neg iXt+lZVIVZnN/9chvt+VEqLJArfpul6q8IgzZDU69UEKup6+iS+7mce7bnZDpQg4Dclk g6GE0oGkook0SihYVsaXoq2mRYyHEobZe8ZKQgypy/7WG9pi8gbwkId/ZRr3wdjKHNKv E0+g== X-Forwarded-Encrypted: i=1; AJvYcCXS/0VqgEEZMAmctpWxnvAhGsbLNp8ScBFzSiq6FPpsUX3lbnQIOwjST9hqR6NuRge1xxm2Y5zFFllcZS0=@vger.kernel.org X-Gm-Message-State: AOJu0YwexwMOWOkpEEwMDBf6ZaWzwnsjkIhzalvCbPJeSyOdwNGGhUdt d19CT8ZAMcX8Af0u7EuDzPRFokggCxB4fNyqU8OfM7Al/QM6enWWKmFVDsaAA6Bf7fiMOX9f5wh rtQ== X-Google-Smtp-Source: AGHT+IEUu6XihEhnY+t2e9Vxs/5iDORNtXg6aG1VSTYb7FdqL6TFnIMalLv7s/eH76D4Kj2pF+CEoHOIyug= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:868a:0:b0:e28:fc1b:66bb with SMTP id 3f1490d57ef6-e2e3a6b3017mr1697276.6.1729703287925; Wed, 23 Oct 2024 10:08:07 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:56 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-4-surenb@google.com> Subject: [PATCH v4 3/6] alloc_tag: load module tags into separate contiguous memory From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When a module gets unloaded there is a possibility that some of the allocations it made are still used and therefore the allocation tags corresponding to these allocations are still referenced. As such, the memory for these tags can't be freed. This is currently handled as an abnormal situation and module's data section is not being unloaded. To handle this situation without keeping module's data in memory, allow codetags with longer lifespan than the module to be loaded into their own separate memory. The in-use memory areas and gaps after module unloading in this separate memory are tracked using maple trees. Allocation tags arrange their separate memory so that it is virtually contiguous and that will allow simple allocation tag indexing later on in this patchset. The size of this virtually contiguous memory is set to store up to 100000 allocation tags. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/asm-generic/codetag.lds.h | 19 +++ include/linux/alloc_tag.h | 13 +- include/linux/codetag.h | 37 ++++- kernel/module/main.c | 80 ++++++---- lib/alloc_tag.c | 249 +++++++++++++++++++++++++++--- lib/codetag.c | 100 +++++++++++- scripts/module.lds.S | 5 +- 7 files changed, 441 insertions(+), 62 deletions(-) diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codeta= g.lds.h index 64f536b80380..372c320c5043 100644 --- a/include/asm-generic/codetag.lds.h +++ b/include/asm-generic/codetag.lds.h @@ -11,4 +11,23 @@ #define CODETAG_SECTIONS() \ SECTION_WITH_BOUNDARIES(alloc_tags) =20 +/* + * Module codetags which aren't used after module unload, therefore have t= he + * same lifespan as the module and can be safely unloaded with the module. + */ +#define MOD_CODETAG_SECTIONS() + +#define MOD_SEPARATE_CODETAG_SECTION(_name) \ + .codetag.##_name : { \ + SECTION_WITH_BOUNDARIES(_name) \ + } + +/* + * For codetags which might be used after module unload, therefore might s= tay + * longer in memory. Each such codetag type has its own section so that we= can + * unload them individually once unused. + */ +#define MOD_SEPARATE_CODETAG_SECTIONS() \ + MOD_SEPARATE_CODETAG_SECTION(alloc_tags) + #endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 1f0a9ff23a2c..7431757999c5 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,13 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); =20 +struct alloc_tag_module_section { + unsigned long start_addr; + unsigned long end_addr; + /* used size */ + unsigned long size; +}; + #ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG =20 #define CODETAG_EMPTY ((void *)1) @@ -54,6 +61,8 @@ static inline void set_codetag_empty(union codetag_ref *r= ef) {} =20 #ifdef CONFIG_MEM_ALLOC_PROFILING =20 +#define ALLOC_TAG_SECTION_NAME "alloc_tags" + struct codetag_bytes { struct codetag *ct; s64 bytes; @@ -76,7 +85,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_= tag); =20 #define DEFINE_ALLOC_TAG(_alloc_tag) \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") =3D { \ + __section(ALLOC_TAG_SECTION_NAME) =3D { \ .ct =3D CODE_TAG_INIT, \ .counters =3D &_shared_alloc_tag }; =20 @@ -85,7 +94,7 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_= tag); #define DEFINE_ALLOC_TAG(_alloc_tag) \ static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ static struct alloc_tag _alloc_tag __used __aligned(8) \ - __section("alloc_tags") =3D { \ + __section(ALLOC_TAG_SECTION_NAME) =3D { \ .ct =3D CODE_TAG_INIT, \ .counters =3D &_alloc_tag_cntr }; =20 diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c2a579ccd455..d10bd9810d32 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -35,8 +35,15 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - bool (*module_unload)(struct codetag_type *cttype, + void (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); +#ifdef CONFIG_MODULES + void (*module_replaced)(struct module *mod, struct module *new_mod); + bool (*needs_section_mem)(struct module *mod, unsigned long size); + void *(*alloc_section_mem)(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align); + void (*free_section_mem)(struct module *mod, bool used); +#endif }; =20 struct codetag_iterator { @@ -71,11 +78,31 @@ struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); =20 #if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) + +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size); +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align); +void codetag_free_module_sections(struct module *mod); +void codetag_module_replaced(struct module *mod, struct module *new_mod); void codetag_load_module(struct module *mod); -bool codetag_unload_module(struct module *mod); -#else +void codetag_unload_module(struct module *mod); + +#else /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ + +static inline bool +codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) { return false; } +static inline void * +codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) { return NULL; } +static inline void codetag_free_module_sections(struct module *mod) {} +static inline void codetag_module_replaced(struct module *mod, struct modu= le *new_mod) {} static inline void codetag_load_module(struct module *mod) {} -static inline bool codetag_unload_module(struct module *mod) { return true= ; } -#endif +static inline void codetag_unload_module(struct module *mod) {} + +#endif /* defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) */ =20 #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index ef54733bd7d2..1787686e5cae 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1254,22 +1254,17 @@ static int module_memory_alloc(struct module *mod, = enum mod_mem_type type) return 0; } =20 -static void module_memory_free(struct module *mod, enum mod_mem_type type, - bool unload_codetags) +static void module_memory_free(struct module *mod, enum mod_mem_type type) { struct module_memory *mem =3D &mod->mem[type]; - void *ptr =3D mem->base; =20 if (mem->is_rox) vfree(mem->rw_copy); =20 - if (!unload_codetags && mod_mem_type_is_core_data(type)) - return; - - execmem_free(ptr); + execmem_free(mem->base); } =20 -static void free_mod_mem(struct module *mod, bool unload_codetags) +static void free_mod_mem(struct module *mod) { for_each_mod_mem_type(type) { struct module_memory *mod_mem =3D &mod->mem[type]; @@ -1280,25 +1275,20 @@ static void free_mod_mem(struct module *mod, bool u= nload_codetags) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod, type, unload_codetags); + module_memory_free(mod, type); } =20 /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod, MOD_DATA, unload_codetags); + module_memory_free(mod, MOD_DATA); } =20 /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { - bool unload_codetags; - trace_module_free(mod); =20 - unload_codetags =3D codetag_unload_module(mod); - if (!unload_codetags) - pr_warn("%s: memory allocation(s) from the module still alive, cannot un= load cleanly\n", - mod->name); + codetag_unload_module(mod); =20 mod_sysfs_teardown(mod); =20 @@ -1341,7 +1331,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); =20 - free_mod_mem(mod, unload_codetags); + free_mod_mem(mod); } =20 void *__symbol_get(const char *symbol) @@ -1606,6 +1596,20 @@ static void __layout_sections(struct module *mod, st= ruct load_info *info, bool i if (WARN_ON_ONCE(type =3D=3D MOD_INVALID)) continue; =20 + /* + * Do not allocate codetag memory as we load it into + * preallocated contiguous memory. + */ + if (codetag_needs_module_section(mod, sname, s->sh_size)) { + /* + * s->sh_entsize won't be used but populate the + * type field to avoid confusion. + */ + s->sh_entsize =3D ((unsigned long)(type) & SH_ENTSIZE_TYPE_MASK) + << SH_ENTSIZE_TYPE_SHIFT; + continue; + } + s->sh_entsize =3D module_get_offset_and_type(mod, type, s, i); pr_debug("\t%s\n", sname); } @@ -2280,6 +2284,7 @@ static int move_module(struct module *mod, struct loa= d_info *info) int i; enum mod_mem_type t =3D 0; int ret =3D -ENOMEM; + bool codetag_section_found =3D false; =20 for_each_mod_mem_type(type) { if (!mod->mem[type].size) { @@ -2291,7 +2296,7 @@ static int move_module(struct module *mod, struct loa= d_info *info) ret =3D module_memory_alloc(mod, type); if (ret) { t =3D type; - goto out_enomem; + goto out_err; } } =20 @@ -2300,15 +2305,33 @@ static int move_module(struct module *mod, struct l= oad_info *info) for (i =3D 0; i < info->hdr->e_shnum; i++) { void *dest; Elf_Shdr *shdr =3D &info->sechdrs[i]; - enum mod_mem_type type =3D shdr->sh_entsize >> SH_ENTSIZE_TYPE_SHIFT; - unsigned long offset =3D shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK; + const char *sname; unsigned long addr; =20 if (!(shdr->sh_flags & SHF_ALLOC)) continue; =20 - addr =3D (unsigned long)mod->mem[type].base + offset; - dest =3D mod->mem[type].rw_copy + offset; + sname =3D info->secstrings + shdr->sh_name; + /* + * Load codetag sections separately as they might still be used + * after module unload. + */ + if (codetag_needs_module_section(mod, sname, shdr->sh_size)) { + dest =3D codetag_alloc_module_section(mod, sname, shdr->sh_size, + arch_mod_section_prepend(mod, i), shdr->sh_addralign); + if (IS_ERR(dest)) { + ret =3D PTR_ERR(dest); + goto out_err; + } + addr =3D (unsigned long)dest; + codetag_section_found =3D true; + } else { + enum mod_mem_type type =3D shdr->sh_entsize >> SH_ENTSIZE_TYPE_SHIFT; + unsigned long offset =3D shdr->sh_entsize & SH_ENTSIZE_OFFSET_MASK; + + addr =3D (unsigned long)mod->mem[type].base + offset; + dest =3D mod->mem[type].rw_copy + offset; + } =20 if (shdr->sh_type !=3D SHT_NOBITS) { /* @@ -2320,7 +2343,7 @@ static int move_module(struct module *mod, struct loa= d_info *info) if (i =3D=3D info->index.mod && (WARN_ON_ONCE(shdr->sh_size !=3D sizeof(struct module)))) { ret =3D -ENOEXEC; - goto out_enomem; + goto out_err; } memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size); } @@ -2336,9 +2359,12 @@ static int move_module(struct module *mod, struct lo= ad_info *info) } =20 return 0; -out_enomem: +out_err: for (t--; t >=3D 0; t--) - module_memory_free(mod, t, true); + module_memory_free(mod, t); + if (codetag_section_found) + codetag_free_module_sections(mod); + return ret; } =20 @@ -2459,6 +2485,8 @@ static struct module *layout_and_allocate(struct load= _info *info, int flags) /* Module has been copied to its final place now: return it. */ mod =3D (void *)info->sechdrs[info->index.mod].sh_addr; kmemleak_load_module(mod, info); + codetag_module_replaced(info->mod, mod); + return mod; } =20 @@ -2468,7 +2496,7 @@ static void module_deallocate(struct module *mod, str= uct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); =20 - free_mod_mem(mod, true); + free_mod_mem(mod); } =20 int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 435aa837e550..d9f51169ffeb 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include #include @@ -9,6 +10,7 @@ #include =20 #define ALLOCINFO_FILE_NAME "allocinfo" +#define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) =20 #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT static bool mem_profiling_support __meminitdata =3D true; @@ -174,31 +176,226 @@ static void __init procfs_init(void) } } =20 -static bool alloc_tag_module_unload(struct codetag_type *cttype, - struct codetag_module *cmod) +#ifdef CONFIG_MODULES + +static struct maple_tree mod_area_mt =3D MTREE_INIT(mod_area_mt, MT_FLAGS_= ALLOC_RANGE); +/* A dummy object used to indicate an unloaded module */ +static struct module unloaded_mod; +/* A dummy object used to indicate a module prepended area */ +static struct module prepend_mod; + +static struct alloc_tag_module_section module_tags; + +static bool needs_section_mem(struct module *mod, unsigned long size) { - struct codetag_iterator iter =3D codetag_get_ct_iter(cttype); - struct alloc_tag_counters counter; - bool module_unused =3D true; - struct alloc_tag *tag; - struct codetag *ct; + return size >=3D sizeof(struct alloc_tag); +} + +static struct alloc_tag *find_used_tag(struct alloc_tag *from, struct allo= c_tag *to) +{ + while (from <=3D to) { + struct alloc_tag_counters counter; =20 - for (ct =3D codetag_next_ct(&iter); ct; ct =3D codetag_next_ct(&iter)) { - if (iter.cmod !=3D cmod) + counter =3D alloc_tag_read(from); + if (counter.bytes) + return from; + from++; + } + + return NULL; +} + +/* Called with mod_area_mt locked */ +static void clean_unused_module_areas_locked(void) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; + + mas_for_each(&mas, val, module_tags.size) { + if (val !=3D &unloaded_mod) continue; =20 - tag =3D ct_to_alloc_tag(ct); - counter =3D alloc_tag_read(tag); + /* Release area if all tags are unused */ + if (!find_used_tag((struct alloc_tag *)(module_tags.start_addr + mas.ind= ex), + (struct alloc_tag *)(module_tags.start_addr + mas.last))) + mas_erase(&mas); + } +} + +/* Called with mod_area_mt locked */ +static bool find_aligned_area(struct ma_state *mas, unsigned long section_= size, + unsigned long size, unsigned int prepend, unsigned long align) +{ + bool cleanup_done =3D false; + +repeat: + /* Try finding exact size and hope the start is aligned */ + if (!mas_empty_area(mas, 0, section_size - 1, prepend + size)) { + if (IS_ALIGNED(mas->index + prepend, align)) + return true; + + /* Try finding larger area to align later */ + mas_reset(mas); + if (!mas_empty_area(mas, 0, section_size - 1, + size + prepend + align - 1)) + return true; + } =20 - if (WARN(counter.bytes, - "%s:%u module %s func:%s has %llu allocated at module unload", - ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) - module_unused =3D false; + /* No free area, try cleanup stale data and repeat the search once */ + if (!cleanup_done) { + clean_unused_module_areas_locked(); + cleanup_done =3D true; + mas_reset(mas); + goto repeat; } =20 - return module_unused; + return false; +} + +static void *reserve_module_tags(struct module *mod, unsigned long size, + unsigned int prepend, unsigned long align) +{ + unsigned long section_size =3D module_tags.end_addr - module_tags.start_a= ddr; + MA_STATE(mas, &mod_area_mt, 0, section_size - 1); + unsigned long offset; + void *ret =3D NULL; + + /* If no tags return NULL */ + if (size < sizeof(struct alloc_tag)) + return NULL; + + /* + * align is always power of 2, so we can use IS_ALIGNED and ALIGN. + * align 0 or 1 means no alignment, to simplify set to 1. + */ + if (!align) + align =3D 1; + + mas_lock(&mas); + if (!find_aligned_area(&mas, section_size, size, prepend, align)) { + ret =3D ERR_PTR(-ENOMEM); + goto unlock; + } + + /* Mark found area as reserved */ + offset =3D mas.index; + offset +=3D prepend; + offset =3D ALIGN(offset, align); + if (offset !=3D mas.index) { + unsigned long pad_start =3D mas.index; + + mas.last =3D offset - 1; + mas_store(&mas, &prepend_mod); + if (mas_is_err(&mas)) { + ret =3D ERR_PTR(xa_err(mas.node)); + goto unlock; + } + mas.index =3D offset; + mas.last =3D offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) { + mas.index =3D pad_start; + mas_erase(&mas); + ret =3D ERR_PTR(xa_err(mas.node)); + } + } else { + mas.last =3D offset + size - 1; + mas_store(&mas, mod); + if (mas_is_err(&mas)) + ret =3D ERR_PTR(xa_err(mas.node)); + } +unlock: + mas_unlock(&mas); + + if (IS_ERR(ret)) + return ret; + + if (module_tags.size < offset + size) + module_tags.size =3D offset + size; + + return (struct alloc_tag *)(module_tags.start_addr + offset); } =20 +static void release_module_tags(struct module *mod, bool used) +{ + MA_STATE(mas, &mod_area_mt, module_tags.size, module_tags.size); + struct alloc_tag *tag; + struct module *val; + + mas_lock(&mas); + mas_for_each_rev(&mas, val, 0) + if (val =3D=3D mod) + break; + + if (!val) /* module not found */ + goto out; + + if (!used) + goto release_area; + + /* Find out if the area is used */ + tag =3D find_used_tag((struct alloc_tag *)(module_tags.start_addr + mas.i= ndex), + (struct alloc_tag *)(module_tags.start_addr + mas.last)); + if (tag) { + struct alloc_tag_counters counter =3D alloc_tag_read(tag); + + pr_info("%s:%u module %s func:%s has %llu allocated at module unload\n", + tag->ct.filename, tag->ct.lineno, tag->ct.modname, + tag->ct.function, counter.bytes); + } else { + used =3D false; + } +release_area: + mas_store(&mas, used ? &unloaded_mod : NULL); + val =3D mas_prev_range(&mas, 0); + if (val =3D=3D &prepend_mod) + mas_store(&mas, NULL); +out: + mas_unlock(&mas); +} + +static void replace_module(struct module *mod, struct module *new_mod) +{ + MA_STATE(mas, &mod_area_mt, 0, module_tags.size); + struct module *val; + + mas_lock(&mas); + mas_for_each(&mas, val, module_tags.size) { + if (val !=3D mod) + continue; + + mas_store_gfp(&mas, new_mod, GFP_KERNEL); + break; + } + mas_unlock(&mas); +} + +static int __init alloc_mod_tags_mem(void) +{ + /* Allocate space to copy allocation tags */ + module_tags.start_addr =3D (unsigned long)execmem_alloc(EXECMEM_MODULE_DA= TA, + MODULE_ALLOC_TAG_VMAP_SIZE); + if (!module_tags.start_addr) + return -ENOMEM; + + module_tags.end_addr =3D module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_S= IZE; + + return 0; +} + +static void __init free_mod_tags_mem(void) +{ + execmem_free((void *)module_tags.start_addr); + module_tags.start_addr =3D 0; +} + +#else /* CONFIG_MODULES */ + +static inline int alloc_mod_tags_mem(void) { return 0; } +static inline void free_mod_tags_mem(void) {} + +#endif /* CONFIG_MODULES */ + static int __init setup_early_mem_profiling(char *str) { bool enable; @@ -274,14 +471,26 @@ static inline void sysctl_init(void) {} static int __init alloc_tag_init(void) { const struct codetag_type_desc desc =3D { - .section =3D "alloc_tags", - .tag_size =3D sizeof(struct alloc_tag), - .module_unload =3D alloc_tag_module_unload, + .section =3D ALLOC_TAG_SECTION_NAME, + .tag_size =3D sizeof(struct alloc_tag), +#ifdef CONFIG_MODULES + .needs_section_mem =3D needs_section_mem, + .alloc_section_mem =3D reserve_module_tags, + .free_section_mem =3D release_module_tags, + .module_replaced =3D replace_module, +#endif }; + int res; + + res =3D alloc_mod_tags_mem(); + if (res) + return res; =20 alloc_tag_cttype =3D codetag_register_type(&desc); - if (IS_ERR(alloc_tag_cttype)) + if (IS_ERR(alloc_tag_cttype)) { + free_mod_tags_mem(); return PTR_ERR(alloc_tag_cttype); + } =20 sysctl_init(); procfs_init(); diff --git a/lib/codetag.c b/lib/codetag.c index d1fbbb7c2ec3..654496952f86 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -207,6 +207,94 @@ static int codetag_module_init(struct codetag_type *ct= type, struct module *mod) } =20 #ifdef CONFIG_MODULES +#define CODETAG_SECTION_PREFIX ".codetag." + +/* Some codetag types need a separate module section */ +bool codetag_needs_module_section(struct module *mod, const char *name, + unsigned long size) +{ + const char *type_name; + struct codetag_type *cttype; + bool ret =3D false; + + if (strncmp(name, CODETAG_SECTION_PREFIX, strlen(CODETAG_SECTION_PREFIX))) + return false; + + type_name =3D name + strlen(CODETAG_SECTION_PREFIX); + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) =3D=3D 0) { + if (!cttype->desc.needs_section_mem) + break; + + down_write(&cttype->mod_lock); + ret =3D cttype->desc.needs_section_mem(mod, size); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void *codetag_alloc_module_section(struct module *mod, const char *name, + unsigned long size, unsigned int prepend, + unsigned long align) +{ + const char *type_name =3D name + strlen(CODETAG_SECTION_PREFIX); + struct codetag_type *cttype; + void *ret =3D NULL; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (strcmp(type_name, cttype->desc.section) =3D=3D 0) { + if (WARN_ON(!cttype->desc.alloc_section_mem)) + break; + + down_write(&cttype->mod_lock); + ret =3D cttype->desc.alloc_section_mem(mod, size, prepend, align); + up_write(&cttype->mod_lock); + break; + } + } + mutex_unlock(&codetag_lock); + + return ret; +} + +void codetag_free_module_sections(struct module *mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.free_section_mem) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.free_section_mem(mod, false); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + +void codetag_module_replaced(struct module *mod, struct module *new_mod) +{ + struct codetag_type *cttype; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + if (!cttype->desc.module_replaced) + continue; + + down_write(&cttype->mod_lock); + cttype->desc.module_replaced(mod, new_mod); + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + void codetag_load_module(struct module *mod) { struct codetag_type *cttype; @@ -220,13 +308,12 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } =20 -bool codetag_unload_module(struct module *mod) +void codetag_unload_module(struct module *mod) { struct codetag_type *cttype; - bool unload_ok =3D true; =20 if (!mod) - return true; + return; =20 /* await any module's kfree_rcu() operations to complete */ kvfree_rcu_barrier(); @@ -246,18 +333,17 @@ bool codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - if (!cttype->desc.module_unload(cttype, cmod)) - unload_ok =3D false; + cttype->desc.module_unload(cttype, cmod); =20 cttype->count -=3D range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); kfree(cmod); } up_write(&cttype->mod_lock); + if (found && cttype->desc.free_section_mem) + cttype->desc.free_section_mem(mod, true); } mutex_unlock(&codetag_lock); - - return unload_ok; } #endif /* CONFIG_MODULES */ =20 diff --git a/scripts/module.lds.S b/scripts/module.lds.S index 3f43edef813c..711c6e029936 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -50,7 +50,7 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } =20 .rodata : { @@ -59,9 +59,10 @@ SECTIONS { } #else .data : { - CODETAG_SECTIONS() + MOD_CODETAG_SECTIONS() } #endif + MOD_SEPARATE_CODETAG_SECTIONS() } =20 /* bring in arch-specific sections */ --=20 2.47.0.105.g07ac214952-goog From nobody Mon Nov 25 22:33:49 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF1D91D0173 for ; Wed, 23 Oct 2024 17:08:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703292; cv=none; b=mJc08jakGu2gPCABjKOvVXYc3jF0+d5CYMxF2Ca/i62JTBJ/6ysEmX+VxdkVuK3SxUcZlE0eqPAcZEa9bBtqkSyIWkRyUL2eCnnmfr3C8MC5PVmCQDFqvs1foRhNg0c/2uZpsNTrpxDT1xZTbn0/U5y2StLflHBnjfrbYwxXtZY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703292; c=relaxed/simple; bh=rSPNRF2fLgd2fBpGTA6Y9m3Pra5kFODk3NqDZER5+jw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=k7oVnToyCt+D7aT9pHUawQv95A4RCFsFA71TZnXz6eWHOKD+o9KBb2tScxRNhRXpTCoaftsPRyFyrBaAaTIQm7dke4A/HtxI9w5BDEpfe9hY/R7I2MxB+l9PMIw0cDe+IjJ+pgo4Fb7R+rWeCvPkupltio9b+hkTS97uKIAzhoE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Iq53X5Zt; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Iq53X5Zt" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e3204db795so104968077b3.2 for ; Wed, 23 Oct 2024 10:08:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703290; x=1730308090; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=Iq53X5ZtpUJMTtXIkO10di2uXOkdtqAqzUyER7tElcfV0RXgZJTp261b3h59k42JAa VSnCOr5T01ko8qcFIBUbq+Qn/pgU9PYGKIuFuo4t9Yl/jMWba7OM0xspPVAvEN/BqL3a kUkugYZYHIBsFWha6owlmPZ24SmYFe1buLhNw2KGzKTMF+hH9ZiUlhv5ugTIneUgFvxz rHGrHDuHHGgiE4z0l7ixQ9BaaRD/EPKlbz+U/YO6DT3XqnENGuxG0S/DI79JfJq9O3Ig piepwR9Ffk1CDAl5ZzxR4pWI58CSnyWNMLsSGNKJZUNlX8kZUGcSmYXXQAPgd43Swgrw umzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703290; x=1730308090; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H2ux1AvVUg/u2T/gNvjmDC0vmkvp+tl4Ubd3hF2zga4=; b=MXZPwSHiKe+3vT/5Xud5Y7mRGP5czZBzNSoeg1I16WDtq24KfkCqON7L5PwXagpdKt Tj602IPLXzXVU8l4ZEhzcedVLVMKaguhfPnQYj1+gj6fhI9UuIwj+BTTBAMpGpW9kZCm 23aubMXn/9rZmt9FBO0Bn+h8sDxCszmM6tXySdIoq2ZebWMPPPrTwYYZriFzP1/lNLC2 25BMGoIm0/sve4BsUNiwnuMUuAyi0G2K3mQuPjSbYnEiQCUXcCTrDYZEN1pYmnHkD1tL CsB4PDo5pWsyzdXTizavKGoqpxEbU1h1EsvwgIVaP1f+94Qp+2Xax+t1ua+Rp+XqlSxz +5ug== X-Forwarded-Encrypted: i=1; AJvYcCX8vQwGhs5Zsut/3B0EvF/kAFFtJnWUFpw8Fsk8tcDR2VthVps4qW4TSNcwJqB9VAHmAdBPQGoYDxZSefE=@vger.kernel.org X-Gm-Message-State: AOJu0YzlztuD2YDJ+mQBDscR8tgahToi04pGf7suE4pFE9SJybV90Vgl Rb1i1B40gHxwil3uzwFvdeHME2DghMdEbVqLUpk1FsAMFDmzYpzc4+KSB9noDqxb7XERGRtuTKb urA== X-Google-Smtp-Source: AGHT+IGGx5cwUMRbaaaRLBx9ozQm5nYkieFYXts2XJ8fpTDFGg4nxvLC1BvVBA4+mhBqSvprWgkBELkrSzs= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a05:690c:4b03:b0:6b0:d571:3540 with SMTP id 00721157ae682-6e7f0f97164mr929077b3.6.1729703289927; Wed, 23 Oct 2024 10:08:09 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:57 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-5-surenb@google.com> Subject: [PATCH v4 4/6] alloc_tag: populate memory for module tags as needed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The memory reserved for module tags does not need to be backed by physical pages until there are tags to store there. Change the way we reserve this memory to allocate only virtual area for the tags and populate it with physical pages as needed when we load a module. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/linux/execmem.h | 10 ++++++ include/linux/vmalloc.h | 3 ++ lib/alloc_tag.c | 73 ++++++++++++++++++++++++++++++++++++----- mm/execmem.c | 16 +++++++++ mm/internal.h | 6 ++++ mm/vmalloc.c | 4 +-- 6 files changed, 101 insertions(+), 11 deletions(-) diff --git a/include/linux/execmem.h b/include/linux/execmem.h index 1517fa196bf7..5a5e2917f870 100644 --- a/include/linux/execmem.h +++ b/include/linux/execmem.h @@ -139,6 +139,16 @@ void *execmem_alloc(enum execmem_type type, size_t siz= e); */ void execmem_free(void *ptr); =20 +/** + * execmem_vmap - create virtual mapping for EXECMEM_MODULE_DATA memory + * @size: size of the virtual mapping in bytes + * + * Maps virtually contiguous area in the range suitable for EXECMEM_MODULE= _DATA. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *execmem_vmap(size_t size); + /** * execmem_update_copy - copy an update to executable memory * @dst: destination address to update diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 27408f21e501..31e9ffd936e3 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -202,6 +202,9 @@ extern int remap_vmalloc_range_partial(struct vm_area_s= truct *vma, extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr, unsigned long pgoff); =20 +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, + struct page **pages, unsigned int page_shift); + /* * Architectures can set this mask to a combination of PGTBL_P?D_MODIFIED = values * and let generic vmalloc and ioremap code know when arch_sync_kernel_map= pings() diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index d9f51169ffeb..061e43196247 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -8,14 +8,15 @@ #include #include #include +#include =20 #define ALLOCINFO_FILE_NAME "allocinfo" #define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) =20 #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT -static bool mem_profiling_support __meminitdata =3D true; +static bool mem_profiling_support =3D true; #else -static bool mem_profiling_support __meminitdata; +static bool mem_profiling_support; #endif =20 static struct codetag_type *alloc_tag_cttype; @@ -154,7 +155,7 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, = size_t count, bool can_sl return nr; } =20 -static void __init shutdown_mem_profiling(void) +static void shutdown_mem_profiling(void) { if (mem_alloc_profiling_enabled()) static_branch_disable(&mem_alloc_profiling_key); @@ -179,6 +180,7 @@ static void __init procfs_init(void) #ifdef CONFIG_MODULES =20 static struct maple_tree mod_area_mt =3D MTREE_INIT(mod_area_mt, MT_FLAGS_= ALLOC_RANGE); +static struct vm_struct *vm_module_tags; /* A dummy object used to indicate an unloaded module */ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ @@ -252,6 +254,33 @@ static bool find_aligned_area(struct ma_state *mas, un= signed long section_size, return false; } =20 +static int vm_module_tags_populate(void) +{ + unsigned long phys_size =3D vm_module_tags->nr_pages << PAGE_SHIFT; + + if (phys_size < module_tags.size) { + struct page **next_page =3D vm_module_tags->pages + vm_module_tags->nr_p= ages; + unsigned long addr =3D module_tags.start_addr + phys_size; + unsigned long more_pages; + unsigned long nr; + + more_pages =3D ALIGN(module_tags.size - phys_size, PAGE_SIZE) >> PAGE_SH= IFT; + nr =3D alloc_pages_bulk_array_node(GFP_KERNEL | __GFP_NOWARN, + NUMA_NO_NODE, more_pages, next_page); + if (nr < more_pages || + vmap_pages_range(addr, addr + (nr << PAGE_SHIFT), PAGE_KERNEL, + next_page, PAGE_SHIFT) < 0) { + /* Clean up and error out */ + for (int i =3D 0; i < nr; i++) + __free_page(next_page[i]); + return -ENOMEM; + } + vm_module_tags->nr_pages +=3D nr; + } + + return 0; +} + static void *reserve_module_tags(struct module *mod, unsigned long size, unsigned int prepend, unsigned long align) { @@ -310,8 +339,18 @@ static void *reserve_module_tags(struct module *mod, u= nsigned long size, if (IS_ERR(ret)) return ret; =20 - if (module_tags.size < offset + size) + if (module_tags.size < offset + size) { + int grow_res; + module_tags.size =3D offset + size; + grow_res =3D vm_module_tags_populate(); + if (grow_res) { + shutdown_mem_profiling(); + pr_err("Failed to allocate memory for allocation tags in the module %s.= Memory allocation profiling is disabled!\n", + mod->name); + return ERR_PTR(grow_res); + } + } =20 return (struct alloc_tag *)(module_tags.start_addr + offset); } @@ -372,12 +411,23 @@ static void replace_module(struct module *mod, struct= module *new_mod) =20 static int __init alloc_mod_tags_mem(void) { - /* Allocate space to copy allocation tags */ - module_tags.start_addr =3D (unsigned long)execmem_alloc(EXECMEM_MODULE_DA= TA, - MODULE_ALLOC_TAG_VMAP_SIZE); - if (!module_tags.start_addr) + /* Map space to copy allocation tags */ + vm_module_tags =3D execmem_vmap(MODULE_ALLOC_TAG_VMAP_SIZE); + if (!vm_module_tags) { + pr_err("Failed to map %lu bytes for module allocation tags\n", + MODULE_ALLOC_TAG_VMAP_SIZE); + module_tags.start_addr =3D 0; return -ENOMEM; + } =20 + vm_module_tags->pages =3D kmalloc_array(get_vm_area_size(vm_module_tags) = >> PAGE_SHIFT, + sizeof(struct page *), GFP_KERNEL | __GFP_ZERO); + if (!vm_module_tags->pages) { + free_vm_area(vm_module_tags); + return -ENOMEM; + } + + module_tags.start_addr =3D (unsigned long)vm_module_tags->addr; module_tags.end_addr =3D module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_S= IZE; =20 return 0; @@ -385,8 +435,13 @@ static int __init alloc_mod_tags_mem(void) =20 static void __init free_mod_tags_mem(void) { - execmem_free((void *)module_tags.start_addr); + int i; + module_tags.start_addr =3D 0; + for (i =3D 0; i < vm_module_tags->nr_pages; i++) + __free_page(vm_module_tags->pages[i]); + kfree(vm_module_tags->pages); + free_vm_area(vm_module_tags); } =20 #else /* CONFIG_MODULES */ diff --git a/mm/execmem.c b/mm/execmem.c index 576a57e2161f..5c0f9f2d6f83 100644 --- a/mm/execmem.c +++ b/mm/execmem.c @@ -368,6 +368,22 @@ void execmem_free(void *ptr) vfree(ptr); } =20 +struct vm_struct *execmem_vmap(size_t size) +{ + struct execmem_range *range =3D &execmem_info->ranges[EXECMEM_MODULE_DATA= ]; + struct vm_struct *area; + + area =3D __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->start, range->end, NUMA_NO_NODE, + GFP_KERNEL, __builtin_return_address(0)); + if (!area && range->fallback_start) + area =3D __get_vm_area_node(size, range->alignment, PAGE_SHIFT, VM_ALLOC, + range->fallback_start, range->fallback_end, + NUMA_NO_NODE, GFP_KERNEL, __builtin_return_address(0)); + + return area; +} + void *execmem_update_copy(void *dst, const void *src, size_t size) { return text_poke_copy(dst, src, size); diff --git a/mm/internal.h b/mm/internal.h index 508f7802dd2b..f1ce0e10bed8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1219,6 +1219,12 @@ int numa_migrate_check(struct folio *folio, struct v= m_fault *vmf, void free_zone_device_folio(struct folio *folio); int migrate_device_coherent_folio(struct folio *folio); =20 +struct vm_struct *__get_vm_area_node(unsigned long size, + unsigned long align, unsigned long shift, + unsigned long flags, unsigned long start, + unsigned long end, int node, gfp_t gfp_mask, + const void *caller); + /* * mm/gup.c */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 74c0a5eae210..7ed39d104201 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -653,7 +653,7 @@ int vmap_pages_range_noflush(unsigned long addr, unsign= ed long end, * RETURNS: * 0 on success, -errno on failure. */ -static int vmap_pages_range(unsigned long addr, unsigned long end, +int vmap_pages_range(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) { int err; @@ -3106,7 +3106,7 @@ static void clear_vm_uninitialized_flag(struct vm_str= uct *vm) vm->flags &=3D ~VM_UNINITIALIZED; } =20 -static struct vm_struct *__get_vm_area_node(unsigned long size, +struct vm_struct *__get_vm_area_node(unsigned long size, unsigned long align, unsigned long shift, unsigned long flags, unsigned long start, unsigned long end, int node, gfp_t gfp_mask, const void *caller) --=20 2.47.0.105.g07ac214952-goog From nobody Mon Nov 25 22:33:49 2024 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C2941D0498 for ; Wed, 23 Oct 2024 17:08:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703296; cv=none; b=O6LLnZSrQwA2mxAHR+Zj9qJODaER7RmtbcFQu/CzyJDQH88+aXLRSiPMlPswEUIceqrYtpb87QJgFLHp4KYvtnIpG8qWY4a07y11BgFFY+kaetCgSMIEs8r3zAvVtgVNCrj657D6ze9Mv3Ryz4WPeaUgk9hNlwgSkfJaeu/Jfb8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703296; c=relaxed/simple; bh=iHvlDkaumSmSpNtZFfnLsYSuNLD36+oLYxkaNoBHbvg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ud+nUjbHxCVkxlVYdoEiQbPhBBj9aa9qUNQR1jGFHITabG6ayTlbCMHWVbX+0twvHOqbfu/8yNuOixrbAr27c9RS2ktWlcqYYCpF+uSUP1bYgZR+mICJ8WuPfhURdbPfVf/fsV0hsspTF8u2NNWW6v57neN8IVjrecNoitjHwx0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gZQFUqs3; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gZQFUqs3" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6e38fabff35so131148237b3.0 for ; Wed, 23 Oct 2024 10:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703292; x=1730308092; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8wnMIe1iCPC4H6xxK8PR5xB7BjuNdLAE9wA+yh8Gbwc=; b=gZQFUqs3rgaZi54MGRhEGKXQzFBzWq4LwckM9zSJSmUazWS28qFxI7YCwZIxZgp6xj YoxNhJq9aLTXpbHu3hu3nsEjxG47LFWrhUzsO8s/I3GJ49z8nYEoZGqgNT9lhew/YJIO bUwfk9+XibJmzpulvT30c9LNtz4S8JNSzmD2aObqKTY0iZ6++k8KPs8p2LbGSKnOZLsL PmT7gmqWp9EmP61NEVRY9isYF+N5LQn7PK8ekfS7adJICLfn9wzCOHK6E+jgWvVZ1fC2 hfDSUL+cDdE3yXEVVgj8E0fqQhdilcNUHJuYsLezin8jIRTU3O42eTiVYItZtGZd0X+S QPNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703292; x=1730308092; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8wnMIe1iCPC4H6xxK8PR5xB7BjuNdLAE9wA+yh8Gbwc=; b=vZOZ6ve81A9/6vbdmxhC1A7DHtBTQm4pGZ2TwI8vzUQa5UvqmkU5Hxkoq7wWryZeCK FHZAgU/Yidg+5L9i/8qAWCCMZyEuA0clIfZK/iX+ZULpN6pE6qMFdAODAB95y7PQvnEd 6mfsk3OKfTWb6XxQ7iqfnbU84OPhdUxGeozJZax6Z/VuaBjrTgQa6LH36jVFXQ8cMLu9 viQq8W7fFzbGYQzO6v0zssGI+5qE4jTo6kzkh217VvWn5OPBsIzLQsQMRi0OQf+4KG0r /x0vmawqI+s9+fP6vnUKyWeTsQGvQTQcpWODQY3euU9ShyU5Mvdzw12U+p/au3t7SFVF P9uA== X-Forwarded-Encrypted: i=1; AJvYcCWZ1o7PCSkqUg/aYC+9m0bxcr+xKTdBMdy7SwYdjlutNcKHHyHHg9XBHt7SyMdfC2/ccSR9/OvhXzQAI2g=@vger.kernel.org X-Gm-Message-State: AOJu0YyL6dWrt2Dw9GexWcVFYGwPUWXg6AeyTkMolqH2+mqHeyz8V1KJ I769sMmDKjy33g5etiPIoE13oOk4oSr2gbr1GAtWYfZwnHOtvLRlkLEb5s0GP5/sQBG6V6YCIba ApA== X-Google-Smtp-Source: AGHT+IHWTSefrY2mOEgvtjZrHcKSjJbm4CbIxnZltjP19nac0+xcMzP1s41Uk58cU65/uvtLl+YlVZeSEX4= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a5b:70b:0:b0:e28:e4a7:3206 with SMTP id 3f1490d57ef6-e2e3a6d62a7mr4485276.8.1729703291999; Wed, 23 Oct 2024 10:08:11 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:58 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-6-surenb@google.com> Subject: [PATCH v4 5/6] alloc_tag: introduce pgtag_ref_handle to abstract page tag references From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To simplify later changes to page tag references, introduce new pgtag_ref_handle type. This allows easy replacement of page_ext as a storage of page allocation tags. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/linux/mm.h | 25 +++++----- include/linux/pgalloc_tag.h | 92 ++++++++++++++++++++++--------------- 2 files changed, 67 insertions(+), 50 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5cd22303fbc0..8efb4a6a1a70 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4180,37 +4180,38 @@ static inline void pgalloc_tag_split(struct folio *= folio, int old_order, int new return; =20 for (i =3D nr_pages; i < (1 << old_order); i +=3D nr_pages) { - union codetag_ref *ref =3D get_page_tag_ref(folio_page(folio, i)); + union pgtag_ref_handle handle; + union codetag_ref ref; =20 - if (ref) { + if (get_page_tag_ref(folio_page(folio, i), &ref, &handle)) { /* Set new reference to point to the original tag */ - alloc_tag_ref_set(ref, tag); - put_page_tag_ref(ref); + alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } =20 static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) { + union pgtag_ref_handle handle; + union codetag_ref ref; struct alloc_tag *tag; - union codetag_ref *ref; =20 tag =3D pgalloc_tag_get(&old->page); if (!tag) return; =20 - ref =3D get_page_tag_ref(&new->page); - if (!ref) + if (!get_page_tag_ref(&new->page, &ref, &handle)) return; =20 /* Clear the old ref to the original allocation tag. */ clear_page_tag_ref(&old->page); /* Decrement the counters of the tag on get_new_folio. */ - alloc_tag_sub(ref, folio_nr_pages(new)); - - __alloc_tag_ref_set(ref, tag); - - put_page_tag_ref(ref); + alloc_tag_sub(&ref, folio_nr_pages(new)); + __alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } #else /* !CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_split(struct folio *folio, int old_order, i= nt new_order) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 59a3deb792a8..b13cd3313a88 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -11,46 +11,59 @@ =20 #include =20 +union pgtag_ref_handle { + union codetag_ref *ref; /* reference in page extension */ +}; + extern struct page_ext_operations page_alloc_tagging_ops; =20 -static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext= *page_ext) +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline bool get_page_tag_ref(struct page *page, union codetag_ref *= ref, + union pgtag_ref_handle *handle) { - return (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_o= ps); -} + struct page_ext *page_ext; + union codetag_ref *tmp; =20 -static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref= *ref) -{ - return (void *)ref - page_alloc_tagging_ops.offset; + if (!page) + return false; + + page_ext =3D page_ext_get(page); + if (!page_ext) + return false; + + tmp =3D (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_= ops); + ref->ct =3D tmp->ct; + handle->ref =3D tmp; + return true; } =20 -/* Should be called only if mem_alloc_profiling_enabled() */ -static inline union codetag_ref *get_page_tag_ref(struct page *page) +static inline void put_page_tag_ref(union pgtag_ref_handle handle) { - if (page) { - struct page_ext *page_ext =3D page_ext_get(page); + if (WARN_ON(!handle.ref)) + return; =20 - if (page_ext) - return codetag_ref_from_page_ext(page_ext); - } - return NULL; + page_ext_put((void *)handle.ref - page_alloc_tagging_ops.offset); } =20 -static inline void put_page_tag_ref(union codetag_ref *ref) +static inline void update_page_tag_ref(union pgtag_ref_handle handle, + union codetag_ref *ref) { - if (WARN_ON(!ref)) + if (WARN_ON(!handle.ref || !ref)) return; =20 - page_ext_put(page_ext_from_codetag_ref(ref)); + handle.ref->ct =3D ref->ct; } =20 static inline void clear_page_tag_ref(struct page *page) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref =3D get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; =20 - if (ref) { - set_codetag_empty(ref); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + set_codetag_empty(&ref); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -59,11 +72,13 @@ static inline void pgalloc_tag_add(struct page *page, s= truct task_struct *task, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref =3D get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; =20 - if (ref) { - alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE * nr); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -71,11 +86,13 @@ static inline void pgalloc_tag_add(struct page *page, s= truct task_struct *task, static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) { if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref =3D get_page_tag_ref(page); + union pgtag_ref_handle handle; + union codetag_ref ref; =20 - if (ref) { - alloc_tag_sub(ref, PAGE_SIZE * nr); - put_page_tag_ref(ref); + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub(&ref, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); } } } @@ -85,13 +102,14 @@ static inline struct alloc_tag *pgalloc_tag_get(struct= page *page) struct alloc_tag *tag =3D NULL; =20 if (mem_alloc_profiling_enabled()) { - union codetag_ref *ref =3D get_page_tag_ref(page); - - alloc_tag_sub_check(ref); - if (ref) { - if (ref->ct) - tag =3D ct_to_alloc_tag(ref->ct); - put_page_tag_ref(ref); + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub_check(&ref); + if (ref.ct) + tag =3D ct_to_alloc_tag(ref.ct); + put_page_tag_ref(handle); } } =20 @@ -106,8 +124,6 @@ static inline void pgalloc_tag_sub_pages(struct alloc_t= ag *tag, unsigned int nr) =20 #else /* CONFIG_MEM_ALLOC_PROFILING */ =20 -static inline union codetag_ref *get_page_tag_ref(struct page *page) { ret= urn NULL; } -static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void clear_page_tag_ref(struct page *page) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *= task, unsigned int nr) {} --=20 2.47.0.105.g07ac214952-goog From nobody Mon Nov 25 22:33:49 2024 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78FB01D0DD5 for ; Wed, 23 Oct 2024 17:08:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703298; cv=none; b=EmpM72FVx25H34dGMJzv+14lLZdimEUu7401ymRqvZmotWc3qddm6CC6VVKSKoQJ/cg5tqhjoBtjOkcUfxb4fnvhVf7sqJ+dHBM3IjoCDU9Jd7fJJvz6UU630FH0eM7SaFVV1z1XJAIDt/tdOaWk0VbhFPzzcxofayT73ooK+Gg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729703298; c=relaxed/simple; bh=GrkyMactMUumDxMQ8TJnIaREPJ1yXV+q+168Cd4Deg0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=R+unhJKG4e4Z4q3A7VNxONHD8fItBAZbok9QLxGHzX58lO47f+OakHwme7/9G/LD8px2/p/Napz8oPb5GnuawH3dF3/pCfCd29QyH/TbGtmIDWkM86qXFd0xPwIJqaS2apHT35dBQLns/XTZh+EMfy6sNWjdMyFKIAzDAj4oYrg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OpeM1YZi; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OpeM1YZi" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e293150c2c6so48778276.1 for ; Wed, 23 Oct 2024 10:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729703294; x=1730308094; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hUAPSrGkP1fndYxBbEXwkdRLC1ZGIzlRm3shVAxWj1o=; b=OpeM1YZiF3LkgDlCnLPckp1CQyhG9bDgBg11XJXv8h+4gEKDxmKRWMKY0WqI2ui0G6 tSf0DepCCFwGTTCnojfyX4W0HNllUu5h9XthBOdlQXj5LyQZ8t7XTfbVyJ5TAuYDRflk JqYjYOBE3e1IWHNJS4WvWoaANFg5FM7pXWfFLJPXEyEZS+NsSkOWjARSFGgInPxucgRv 0ODxrgGe099fY23Ogo+Dh1eaR9zHAf2UwSabhqSQN65wf+UQ4WK8wS5eD6B1Y9+Y7rNo OR5dTCHngWivqOF/xQ/mSha339V6dKu3FaILwAAYI2c5M6MGQWmp2XWsB48DGSFFyelz hIZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729703294; x=1730308094; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hUAPSrGkP1fndYxBbEXwkdRLC1ZGIzlRm3shVAxWj1o=; b=mbvEoAw8LkI0vimORqyuYU9HZX3Hc8iqcraRH2PrqDmPAal2ZGwie7gAL76TatderR QAEtZneXo+1xS2FFiiaNs9F4scGfG+kP0nozfmnu96mrhrzWnbZx8FbjrI9A66czkGSx fGBdeiapVFR5QJ0HpwdyHkXHb4WCKKCWoOPmCALnakSXRHI1pKE3Yv4NdpaUvZB1LaJ7 b5R9sFfO8QaBBNkFSz/WBi2kOsFJJQ8AJuHQTBWA6zD6E82vrB7bj084T663JOIgsSq6 26XH6SZqlAq8PVYVhxvwBGX/NFhS6z0P2pXA9cKKQilwLv4n1QS65dz/XpTpzhwEPCQy MEVQ== X-Forwarded-Encrypted: i=1; AJvYcCW+/YMA1MekqOIRngBbT98+ym3zpX0pDFd7vIkHkE0MUW+WMKPqKwPl36/Jv1B7uANkn8tRm8abyGy8qDA=@vger.kernel.org X-Gm-Message-State: AOJu0Yyl+nixdcFVK6mWfAbb8x5TG3DNrSj8NiF/LzKaHIqpyEPJvd/P BhHGehSc2Ur7Y4nLZLqinKVHqlhfwtOtJ7CavmAmRcgwTcGhafjIyV0H65JCySsYIhXlm8jUuCo qow== X-Google-Smtp-Source: AGHT+IGRE52J3Q/FMmtRptFBkvIdYNYKY+K4lu8CvvfN06S+6RGcE++e1CnN4bokrfVHywZPohucB5wHvBc= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:a087:59b9:198a:c44c]) (user=surenb job=sendgmr) by 2002:a25:9787:0:b0:e29:7cfa:9fbc with SMTP id 3f1490d57ef6-e2e3a6dc4d5mr1689276.11.1729703294360; Wed, 23 Oct 2024 10:08:14 -0700 (PDT) Date: Wed, 23 Oct 2024 10:07:59 -0700 In-Reply-To: <20241023170759.999909-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241023170759.999909-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241023170759.999909-7-surenb@google.com> Subject: [PATCH v4 6/6] alloc_tag: support for page allocation tag compression From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, corbet@lwn.net, arnd@arndb.de, mcgrof@kernel.org, rppt@kernel.org, paulmck@kernel.org, thuth@redhat.com, tglx@linutronix.de, bp@alien8.de, xiongwei.song@windriver.com, ardb@kernel.org, david@redhat.com, vbabka@suse.cz, mhocko@suse.com, hannes@cmpxchg.org, roman.gushchin@linux.dev, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, pasha.tatashin@soleen.com, souravpanda@google.com, keescook@chromium.org, dennis@kernel.org, jhubbard@nvidia.com, urezki@gmail.com, hch@infradead.org, petr.pavlu@suse.com, samitolvanen@google.com, da.gomez@samsung.com, yuzhao@google.com, vvvvvv@google.com, rostedt@goodmis.org, iamjoonsoo.kim@lge.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, maple-tree@lists.infradead.org, linux-modules@vger.kernel.org, kernel-team@android.com, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement support for storing page allocation tag references directly in the page flags instead of page extensions. sysctl.vm.mem_profiling boot parameter it extended to provide a way for a user to request this mode. Enabling compression eliminates memory overhead caused by page_ext and results in better performance for page allocations. However this mode will not work if the number of available page flag bits is insufficient to address all kernel allocations. Such condition can happen during boot or when loading a module. If this condition is detected, memory allocation profiling gets disabled with an appropriate warning. By default compression mode is disabled. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- Documentation/mm/allocation-profiling.rst | 7 +- include/linux/alloc_tag.h | 10 +- include/linux/codetag.h | 3 + include/linux/page-flags-layout.h | 7 ++ include/linux/pgalloc_tag.h | 145 +++++++++++++++++++--- lib/alloc_tag.c | 142 +++++++++++++++++++-- lib/codetag.c | 4 +- mm/mm_init.c | 5 +- 8 files changed, 290 insertions(+), 33 deletions(-) diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/a= llocation-profiling.rst index ffd6655b7be2..316311240e6a 100644 --- a/Documentation/mm/allocation-profiling.rst +++ b/Documentation/mm/allocation-profiling.rst @@ -18,12 +18,17 @@ kconfig options: missing annotation =20 Boot parameter: - sysctl.vm.mem_profiling=3D0|1|never + sysctl.vm.mem_profiling=3D{0|1|never}[,compressed] =20 When set to "never", memory allocation profiling overhead is minimized a= nd it cannot be enabled at runtime (sysctl becomes read-only). When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=3Dy, default value is= "1". When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=3Dn, default value is= "never". + "compressed" optional parameter will try to store page tag references in= a + compact format, avoiding page extensions. This results in improved perfo= rmance + and memory consumption, however it might fail depending on system config= uration. + If compression fails, a warning is issued and memory allocation profilin= g gets + disabled. =20 sysctl: /proc/sys/vm/mem_profiling diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 7431757999c5..4f811ec0ffe0 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,8 +30,16 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); =20 +struct alloc_tag_kernel_section { + struct alloc_tag *first_tag; + unsigned long count; +}; + struct alloc_tag_module_section { - unsigned long start_addr; + union { + unsigned long start_addr; + struct alloc_tag *first_tag; + }; unsigned long end_addr; /* used size */ unsigned long size; diff --git a/include/linux/codetag.h b/include/linux/codetag.h index d10bd9810d32..d14dbd26b370 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -13,6 +13,9 @@ struct codetag_module; struct seq_buf; struct module; =20 +#define CODETAG_SECTION_START_PREFIX "__start_" +#define CODETAG_SECTION_STOP_PREFIX "__stop_" + /* * An instance of this structure is created in a special ELF section at ev= ery * code location being tagged. At runtime, the special section is treated= as diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-l= ayout.h index 7d79818dc065..4f5c9e979bb9 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -111,5 +111,12 @@ ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \ NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH) =20 +#define NR_NON_PAGEFLAG_BITS (SECTIONS_WIDTH + NODES_WIDTH + ZONES_WIDTH += \ + LAST_CPUPID_SHIFT + KASAN_TAG_WIDTH + \ + LRU_GEN_WIDTH + LRU_REFS_WIDTH) + +#define NR_UNUSED_PAGEFLAG_BITS (BITS_PER_LONG - \ + (NR_NON_PAGEFLAG_BITS + NR_PAGEFLAGS)) + #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index b13cd3313a88..1fe63b52e5e5 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -11,29 +11,118 @@ =20 #include =20 +extern struct page_ext_operations page_alloc_tagging_ops; +extern unsigned long alloc_tag_ref_mask; +extern int alloc_tag_ref_offs; +extern struct alloc_tag_kernel_section kernel_tags; + +DECLARE_STATIC_KEY_FALSE(mem_profiling_compressed); + +typedef u16 pgalloc_tag_idx; + union pgtag_ref_handle { union codetag_ref *ref; /* reference in page extension */ + struct page *page; /* reference in page flags */ }; =20 -extern struct page_ext_operations page_alloc_tagging_ops; +/* Reserved indexes */ +#define CODETAG_ID_NULL 0 +#define CODETAG_ID_EMPTY 1 +#define CODETAG_ID_FIRST 2 + +#ifdef CONFIG_MODULES + +extern struct alloc_tag_module_section module_tags; + +static inline struct alloc_tag *module_idx_to_tag(pgalloc_tag_idx idx) +{ + return &module_tags.first_tag[idx - kernel_tags.count]; +} + +static inline pgalloc_tag_idx module_tag_to_idx(struct alloc_tag *tag) +{ + return CODETAG_ID_FIRST + kernel_tags.count + (tag - module_tags.first_ta= g); +} + +#else /* CONFIG_MODULES */ + +static inline struct alloc_tag *module_idx_to_tag(pgalloc_tag_idx idx) +{ + pr_warn("invalid page tag reference %lu\n", (unsigned long)idx); + return NULL; +} + +static inline pgalloc_tag_idx module_tag_to_idx(struct alloc_tag *tag) +{ + pr_warn("invalid page tag 0x%lx\n", (unsigned long)tag); + return CODETAG_ID_NULL; +} + +#endif /* CONFIG_MODULES */ + +static inline void idx_to_ref(pgalloc_tag_idx idx, union codetag_ref *ref) +{ + switch (idx) { + case (CODETAG_ID_NULL): + ref->ct =3D NULL; + break; + case (CODETAG_ID_EMPTY): + set_codetag_empty(ref); + break; + default: + idx -=3D CODETAG_ID_FIRST; + ref->ct =3D idx < kernel_tags.count ? + &kernel_tags.first_tag[idx].ct : + &module_idx_to_tag(idx)->ct; + break; + } +} + +static inline pgalloc_tag_idx ref_to_idx(union codetag_ref *ref) +{ + struct alloc_tag *tag; + + if (!ref->ct) + return CODETAG_ID_NULL; + + if (is_codetag_empty(ref)) + return CODETAG_ID_EMPTY; + + tag =3D ct_to_alloc_tag(ref->ct); + if (tag >=3D kernel_tags.first_tag && tag < kernel_tags.first_tag + kerne= l_tags.count) + return CODETAG_ID_FIRST + (tag - kernel_tags.first_tag); + + return module_tag_to_idx(tag); +} + + =20 /* Should be called only if mem_alloc_profiling_enabled() */ static inline bool get_page_tag_ref(struct page *page, union codetag_ref *= ref, union pgtag_ref_handle *handle) { - struct page_ext *page_ext; - union codetag_ref *tmp; - if (!page) return false; =20 - page_ext =3D page_ext_get(page); - if (!page_ext) - return false; + if (static_key_enabled(&mem_profiling_compressed)) { + pgalloc_tag_idx idx; + + idx =3D (page->flags >> alloc_tag_ref_offs) & alloc_tag_ref_mask; + idx_to_ref(idx, ref); + handle->page =3D page; + } else { + struct page_ext *page_ext; + union codetag_ref *tmp; + + page_ext =3D page_ext_get(page); + if (!page_ext) + return false; + + tmp =3D (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging= _ops); + ref->ct =3D tmp->ct; + handle->ref =3D tmp; + } =20 - tmp =3D (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_= ops); - ref->ct =3D tmp->ct; - handle->ref =3D tmp; return true; } =20 @@ -42,16 +131,35 @@ static inline void put_page_tag_ref(union pgtag_ref_ha= ndle handle) if (WARN_ON(!handle.ref)) return; =20 - page_ext_put((void *)handle.ref - page_alloc_tagging_ops.offset); + if (!static_key_enabled(&mem_profiling_compressed)) + page_ext_put((void *)handle.ref - page_alloc_tagging_ops.offset); } =20 -static inline void update_page_tag_ref(union pgtag_ref_handle handle, - union codetag_ref *ref) +static inline void update_page_tag_ref(union pgtag_ref_handle handle, unio= n codetag_ref *ref) { - if (WARN_ON(!handle.ref || !ref)) - return; - - handle.ref->ct =3D ref->ct; + if (static_key_enabled(&mem_profiling_compressed)) { + struct page *page =3D handle.page; + unsigned long old_flags; + unsigned long flags; + unsigned long idx; + + if (WARN_ON(!page || !ref)) + return; + + idx =3D (unsigned long)ref_to_idx(ref); + idx =3D (idx & alloc_tag_ref_mask) << alloc_tag_ref_offs; + do { + old_flags =3D READ_ONCE(page->flags); + flags =3D old_flags; + flags &=3D ~(alloc_tag_ref_mask << alloc_tag_ref_offs); + flags |=3D idx; + } while (unlikely(!try_cmpxchg(&page->flags, &old_flags, flags))); + } else { + if (WARN_ON(!handle.ref || !ref)) + return; + + handle.ref->ct =3D ref->ct; + } } =20 static inline void clear_page_tag_ref(struct page *page) @@ -122,6 +230,8 @@ static inline void pgalloc_tag_sub_pages(struct alloc_t= ag *tag, unsigned int nr) this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); } =20 +void __init alloc_tag_sec_init(void); + #else /* CONFIG_MEM_ALLOC_PROFILING */ =20 static inline void clear_page_tag_ref(struct page *page) {} @@ -130,6 +240,7 @@ static inline void pgalloc_tag_add(struct page *page, s= truct task_struct *task, static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { retur= n NULL; } static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned i= nt nr) {} +static inline void alloc_tag_sec_init(void) {} =20 #endif /* CONFIG_MEM_ALLOC_PROFILING */ =20 diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 061e43196247..a6f6f014461e 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -12,6 +13,8 @@ =20 #define ALLOCINFO_FILE_NAME "allocinfo" #define MODULE_ALLOC_TAG_VMAP_SIZE (100000UL * sizeof(struct alloc_tag)) +#define SECTION_START(NAME) (CODETAG_SECTION_START_PREFIX NAME) +#define SECTION_STOP(NAME) (CODETAG_SECTION_STOP_PREFIX NAME) =20 #ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT static bool mem_profiling_support =3D true; @@ -26,6 +29,11 @@ EXPORT_SYMBOL(_shared_alloc_tag); =20 DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, mem_alloc_profiling_key); +DEFINE_STATIC_KEY_FALSE(mem_profiling_compressed); + +struct alloc_tag_kernel_section kernel_tags =3D { NULL, 0 }; +unsigned long alloc_tag_ref_mask; +int alloc_tag_ref_offs; =20 struct allocinfo_private { struct codetag_iterator iter; @@ -155,7 +163,7 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, = size_t count, bool can_sl return nr; } =20 -static void shutdown_mem_profiling(void) +static void shutdown_mem_profiling(bool remove_file) { if (mem_alloc_profiling_enabled()) static_branch_disable(&mem_alloc_profiling_key); @@ -163,6 +171,8 @@ static void shutdown_mem_profiling(void) if (!mem_profiling_support) return; =20 + if (remove_file) + remove_proc_entry(ALLOCINFO_FILE_NAME, NULL); mem_profiling_support =3D false; } =20 @@ -173,10 +183,40 @@ static void __init procfs_init(void) =20 if (!proc_create_seq(ALLOCINFO_FILE_NAME, 0400, NULL, &allocinfo_seq_op))= { pr_err("Failed to create %s file\n", ALLOCINFO_FILE_NAME); - shutdown_mem_profiling(); + shutdown_mem_profiling(false); } } =20 +void __init alloc_tag_sec_init(void) +{ + struct alloc_tag *last_codetag; + + if (!mem_profiling_support) + return; + + if (!static_key_enabled(&mem_profiling_compressed)) + return; + + kernel_tags.first_tag =3D (struct alloc_tag *)kallsyms_lookup_name( + SECTION_START(ALLOC_TAG_SECTION_NAME)); + last_codetag =3D (struct alloc_tag *)kallsyms_lookup_name( + SECTION_STOP(ALLOC_TAG_SECTION_NAME)); + kernel_tags.count =3D last_codetag - kernel_tags.first_tag; + + /* Check if kernel tags fit into page flags */ + if (kernel_tags.count > (1UL << NR_UNUSED_PAGEFLAG_BITS)) { + shutdown_mem_profiling(false); /* allocinfo file does not exist yet */ + pr_err("%lu allocation tags cannot be references using %d available page= flag bits. Memory allocation profiling is disabled!\n", + kernel_tags.count, NR_UNUSED_PAGEFLAG_BITS); + return; + } + + alloc_tag_ref_offs =3D (LRU_REFS_PGOFF - NR_UNUSED_PAGEFLAG_BITS); + alloc_tag_ref_mask =3D ((1UL << NR_UNUSED_PAGEFLAG_BITS) - 1); + pr_debug("Memory allocation profiling compression is using %d page flag b= its!\n", + NR_UNUSED_PAGEFLAG_BITS); +} + #ifdef CONFIG_MODULES =20 static struct maple_tree mod_area_mt =3D MTREE_INIT(mod_area_mt, MT_FLAGS_= ALLOC_RANGE); @@ -186,10 +226,59 @@ static struct module unloaded_mod; /* A dummy object used to indicate a module prepended area */ static struct module prepend_mod; =20 -static struct alloc_tag_module_section module_tags; +struct alloc_tag_module_section module_tags; + +static inline unsigned long alloc_tag_align(unsigned long val) +{ + if (!static_key_enabled(&mem_profiling_compressed)) { + /* No alignment requirements when we are not indexing the tags */ + return val; + } + + if (val % sizeof(struct alloc_tag) =3D=3D 0) + return val; + return ((val / sizeof(struct alloc_tag)) + 1) * sizeof(struct alloc_tag); +} + +static bool ensure_alignment(unsigned long align, unsigned int *prepend) +{ + if (!static_key_enabled(&mem_profiling_compressed)) { + /* No alignment requirements when we are not indexing the tags */ + return true; + } + + /* + * If alloc_tag size is not a multiple of required alignment, tag + * indexing does not work. + */ + if (!IS_ALIGNED(sizeof(struct alloc_tag), align)) + return false; + + /* Ensure prepend consumes multiple of alloc_tag-sized blocks */ + if (*prepend) + *prepend =3D alloc_tag_align(*prepend); + + return true; +} + +static inline bool tags_addressable(void) +{ + unsigned long tag_idx_count; + + if (!static_key_enabled(&mem_profiling_compressed)) + return true; /* with page_ext tags are always addressable */ + + tag_idx_count =3D CODETAG_ID_FIRST + kernel_tags.count + + module_tags.size / sizeof(struct alloc_tag); + + return tag_idx_count < (1UL << NR_UNUSED_PAGEFLAG_BITS); +} =20 static bool needs_section_mem(struct module *mod, unsigned long size) { + if (!mem_profiling_support) + return false; + return size >=3D sizeof(struct alloc_tag); } =20 @@ -300,6 +389,13 @@ static void *reserve_module_tags(struct module *mod, u= nsigned long size, if (!align) align =3D 1; =20 + if (!ensure_alignment(align, &prepend)) { + shutdown_mem_profiling(true); + pr_err("%s: alignment %lu is incompatible with allocation tag indexing. = Memory allocation profiling is disabled!\n", + mod->name, align); + return ERR_PTR(-EINVAL); + } + mas_lock(&mas); if (!find_aligned_area(&mas, section_size, size, prepend, align)) { ret =3D ERR_PTR(-ENOMEM); @@ -343,9 +439,15 @@ static void *reserve_module_tags(struct module *mod, u= nsigned long size, int grow_res; =20 module_tags.size =3D offset + size; + if (mem_alloc_profiling_enabled() && !tags_addressable()) { + shutdown_mem_profiling(true); + pr_warn("With module %s there are too many tags to fit in %d page flag = bits. Memory allocation profiling is disabled!\n", + mod->name, NR_UNUSED_PAGEFLAG_BITS); + } + grow_res =3D vm_module_tags_populate(); if (grow_res) { - shutdown_mem_profiling(); + shutdown_mem_profiling(true); pr_err("Failed to allocate memory for allocation tags in the module %s.= Memory allocation profiling is disabled!\n", mod->name); return ERR_PTR(grow_res); @@ -429,6 +531,8 @@ static int __init alloc_mod_tags_mem(void) =20 module_tags.start_addr =3D (unsigned long)vm_module_tags->addr; module_tags.end_addr =3D module_tags.start_addr + MODULE_ALLOC_TAG_VMAP_S= IZE; + /* Ensure the base is alloc_tag aligned when required for indexing */ + module_tags.start_addr =3D alloc_tag_align(module_tags.start_addr); =20 return 0; } @@ -451,8 +555,10 @@ static inline void free_mod_tags_mem(void) {} =20 #endif /* CONFIG_MODULES */ =20 +/* See: Documentation/mm/allocation-profiling.rst */ static int __init setup_early_mem_profiling(char *str) { + bool compressed =3D false; bool enable; =20 if (!str || !str[0]) @@ -461,22 +567,37 @@ static int __init setup_early_mem_profiling(char *str) if (!strncmp(str, "never", 5)) { enable =3D false; mem_profiling_support =3D false; + pr_info("Memory allocation profiling is disabled!\n"); } else { - int res; + char *token =3D strsep(&str, ","); + + if (kstrtobool(token, &enable)) + return -EINVAL; =20 - res =3D kstrtobool(str, &enable); - if (res) - return res; + if (str) { =20 + if (strcmp(str, "compressed")) + return -EINVAL; + + compressed =3D true; + } mem_profiling_support =3D true; + pr_info("Memory allocation profiling is enabled %s compression and is tu= rned %s!\n", + compressed ? "with" : "without", enable ? "on" : "off"); } =20 - if (enable !=3D static_key_enabled(&mem_alloc_profiling_key)) { + if (enable !=3D mem_alloc_profiling_enabled()) { if (enable) static_branch_enable(&mem_alloc_profiling_key); else static_branch_disable(&mem_alloc_profiling_key); } + if (compressed !=3D static_key_enabled(&mem_profiling_compressed)) { + if (compressed) + static_branch_enable(&mem_profiling_compressed); + else + static_branch_disable(&mem_profiling_compressed); + } =20 return 0; } @@ -484,6 +605,9 @@ early_param("sysctl.vm.mem_profiling", setup_early_mem_= profiling); =20 static __init bool need_page_alloc_tagging(void) { + if (static_key_enabled(&mem_profiling_compressed)) + return false; + return mem_profiling_support; } =20 diff --git a/lib/codetag.c b/lib/codetag.c index 654496952f86..4949511b4933 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -149,8 +149,8 @@ static struct codetag_range get_section_range(struct mo= dule *mod, const char *section) { return (struct codetag_range) { - get_symbol(mod, "__start_", section), - get_symbol(mod, "__stop_", section), + get_symbol(mod, CODETAG_SECTION_START_PREFIX, section), + get_symbol(mod, CODETAG_SECTION_STOP_PREFIX, section), }; } =20 diff --git a/mm/mm_init.c b/mm/mm_init.c index 4ba5607aaf19..1c205b0a86ed 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -83,8 +83,7 @@ void __init mminit_verify_pageflags_layout(void) unsigned long or_mask, add_mask; =20 shift =3D BITS_PER_LONG; - width =3D shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - - LAST_CPUPID_SHIFT - KASAN_TAG_WIDTH - LRU_GEN_WIDTH - LRU_REFS_WIDTH; + width =3D shift - NR_NON_PAGEFLAG_BITS; mminit_dprintk(MMINIT_TRACE, "pageflags_layout_widths", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Fla= gs %d\n", SECTIONS_WIDTH, @@ -2639,7 +2638,7 @@ void __init mm_core_init(void) BUILD_BUG_ON(MAX_ZONELISTS > 2); build_all_zonelists(NULL); page_alloc_init_cpuhp(); - + alloc_tag_sec_init(); /* * page_ext requires contiguous pages, * bigger than MAX_PAGE_ORDER unless SPARSEMEM. --=20 2.47.0.105.g07ac214952-goog