From nobody Tue Feb 10 04:03:27 2026 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F265C3328F7 for ; Thu, 8 Jan 2026 20:39:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767904756; cv=none; b=oCWMSnlxsrPvMZ5C32M8cJT9nu4OOYj4aVM2mDYK0tKOjvDfLEEN+UHJ4MMzaVieBL5HgCcrLyvu5JlmDPIol22+NCXmcVf5mkfXlOuXcMWspeZTvSfOzp1iO/DfSUUu70wWnbxy3iace9zpzbzBI9mX75Lp2U/NYHK9viGrwSo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767904756; c=relaxed/simple; bh=Q6n6H6pGLwW6lg8o+yJkMzcp5ygcedJTXlkaT5WaISo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HySx8IQnja8rxCdRLvzqDrya0xv5jXf0VeCDgnonYR/8+7ZmVj4/SJxXtlx2u3lTL7DhNUqIVSBdp1J8V31ln9xtNbXaoIX3f0QmtqfnOGLCI+vAMnip34pxaYdKxVeuU2WFtwbrdkLnIviKBRUtTsASaBQ1IclUerjdtjYkbNQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=AfZQstJT; arc=none smtp.client-ip=209.85.160.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="AfZQstJT" Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-4ee13dc0c52so30322771cf.2 for ; Thu, 08 Jan 2026 12:39:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1767904750; x=1768509550; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5OTZsz9YEfzZ2R8Vdeqp1RCaOkKgwxKsmSu+J9FA9fQ=; b=AfZQstJTicHT8cU0FpbOpZQ226cG57nxuzO9CXkQR2+q1EpDMkbKf6rUuAVsxrGdTh v6crsNu+AYUpEvyGFd60KQYYnTuKPc9OACtbdGy0jGPVY91zv6lyHAUaabz5uhDce9gt iY7+Aa2Nt4/6bImyHuiist25XkTpKbeWo8KK6FajaQ6+ranNVv4zReewKrjXTfhiu+o0 n/wuib40Q1w5he3k9YH0trpvMh7uZy/SavVCwu1PyWvfBoqpOPGa/VuIcJRMGnkO82cM /7oqylfUe9lJIzHM3fNaOzy0rWjLtTtk7dU/bk+cn9qklC4SXwr+6sCGibE8ZCp7ZWeV OrKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767904750; x=1768509550; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5OTZsz9YEfzZ2R8Vdeqp1RCaOkKgwxKsmSu+J9FA9fQ=; b=R8E11dsRe6+pTD4S6uRsHfVrgLSw4jFD6f0BEHHtsZmhx5x0qzn5We/qqNSfHVPhYT jvgaUVKNkx9+Vmt1xriflY/AyuaeHb2FYlBu7TH8HMwsBD6Z709V08LFvWoVlfNNVakv QbxWphwaEieLVZMOuqFVD4igoNLpxb/fBPoonD7UrSwthf5QP5THVbSHNvDv9hxhY7Df G9HqSmaNSD0IXTeJrR1iTxF8PsLbDdQSWtB0VPsVNSLKeqSstfkyYclw2K1RYOvrd5Tg sWVa+sOpJ1QsCPK06IUPQS5LuuFBJR98zx3Z5VbTzckyq60+wcv98MLkBxXzItmNGMPx FNUg== X-Forwarded-Encrypted: i=1; AJvYcCV1cKdxKxS7+1p0afvEvqaKaSOCrus72rvlOSLL++XFZDYlg0DjgycvZLG+UqFl6kdEdzHy7Gvp6/RVzdc=@vger.kernel.org X-Gm-Message-State: AOJu0YynacNdJ04iGyPxCXMQtXXxkeTF8Zv0Kj4gCmI/I+M9CIja8s6w OuS6GacVppht/277pV5yJAKMjzl3r+MzcX3Pd0eNj8Yd+YV/krCAuOZQ4hCa93cGrTQ= X-Gm-Gg: AY/fxX6aouSBHTvE4sjLktLyjAnY6GgfizKz7g0UXPAGy+ZidozR+2o9pTT9Zr07qZ8 bRb+PYnRdXYMyYs5/lYVDUhet4C4jjLeiCrlITJ/MtDLE4mGwZt/GfYaK5lG98w6ztdEPA2zrdr UgtEW2PnhOogDegenZgmxhXrI4BIdt6iAwrc5ezQDSSJyS2rAIFP0XmNlWLWYGO64OwCvwwazwn GWrtjagneMoo2/Sr/04BArzZDLEtLo5J86FiegAp8DuHhiM3QTBrITUjIGP7qC+4+XLatlTGMVj d2XtX1CdlHGfAcy/uTAFdo3CKpAUpot/meiMrBsBPqtD6soj6BSauzzYhC8p6RBLd6IgiZcWffZ MnLVjN650nTzQi3pNavgpP32+6tQtz1jBtxwFyzbuJnMQhGsn0/q/fg+8/0I+Du9FXVhgOeZ3eG 8xKU6R20qQ1nFvix64yTkEVkOKvZdINySUe6fqAUCqDk3d2uIPA2uJYbL+2lJnHQ0rJ8MXE9yZn a8= X-Google-Smtp-Source: AGHT+IHjrJedObjQLnkqlATuK0iiTd2ReDzXEhDuZpiEE8iScOlIeLDoodxwtlrvEQjvQ0kp+C8/HQ== X-Received: by 2002:a05:622a:554:b0:4ee:26bd:13fa with SMTP id d75a77b69052e-4ffb4a38073mr98924131cf.80.1767904749843; Thu, 08 Jan 2026 12:39:09 -0800 (PST) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-138.washdc.ftas.verizon.net. [96.255.20.138]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-890770e472csm60483886d6.23.2026.01.08.12.39.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jan 2026 12:39:09 -0800 (PST) From: Gregory Price To: linux-mm@kvack.org, cgroups@vger.kernel.org, linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, kernel-team@meta.com, longman@redhat.com, tj@kernel.org, hannes@cmpxchg.org, mkoutny@suse.com, corbet@lwn.net, gregkh@linuxfoundation.org, rafael@kernel.org, dakr@kernel.org, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, akpm@linux-foundation.org, vbabka@suse.cz, surenb@google.com, mhocko@suse.com, jackmanb@google.com, ziy@nvidia.com, david@kernel.org, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, rppt@kernel.org, axelrasmussen@google.com, yuanchu@google.com, weixugc@google.com, yury.norov@gmail.com, linux@rasmusvillemoes.dk, rientjes@google.com, shakeel.butt@linux.dev, chrisl@kernel.org, kasong@tencent.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, bhe@redhat.com, baohua@kernel.org, yosry.ahmed@linux.dev, chengming.zhou@linux.dev, roman.gushchin@linux.dev, muchun.song@linux.dev, osalvador@suse.de, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, cl@gentwo.org, harry.yoo@oracle.com, zhengqi.arch@bytedance.com Subject: [RFC PATCH v3 7/8] mm/zswap: compressed ram direct integration Date: Thu, 8 Jan 2026 15:37:54 -0500 Message-ID: <20260108203755.1163107-8-gourry@gourry.net> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260108203755.1163107-1-gourry@gourry.net> References: <20260108203755.1163107-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If a private zswap-node is available, skip the entire software compression process and memcpy directly to a compressed memory folio, and store the newly allocated compressed memory page as the zswap entry->handle. On decompress we do the opposite: copy directly from the stored page to the destination, and free the compressed memory page. The driver callback is responsible for preventing run-away compression ratio failures by checking that the allocated page is safe to use (i.e. a compression ratio limit hasn't been crossed). Signed-off-by: Gregory Price --- include/linux/zswap.h | 5 ++ mm/zswap.c | 106 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 109 insertions(+), 2 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 30c193a1207e..4b52fe447e7e 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -35,6 +35,8 @@ void zswap_lruvec_state_init(struct lruvec *lruvec); void zswap_folio_swapin(struct folio *folio); bool zswap_is_enabled(void); bool zswap_never_enabled(void); +void zswap_add_direct_node(int nid); +void zswap_remove_direct_node(int nid); #else =20 struct zswap_lruvec_state {}; @@ -69,6 +71,9 @@ static inline bool zswap_never_enabled(void) return true; } =20 +static inline void zswap_add_direct_node(int nid) {} +static inline void zswap_remove_direct_node(int nid) {} + #endif =20 #endif /* _LINUX_ZSWAP_H */ diff --git a/mm/zswap.c b/mm/zswap.c index de8858ff1521..aada588c957e 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -35,6 +35,7 @@ #include #include #include +#include =20 #include "swap.h" #include "internal.h" @@ -190,6 +191,7 @@ struct zswap_entry { swp_entry_t swpentry; unsigned int length; bool referenced; + bool direct; struct zswap_pool *pool; unsigned long handle; struct obj_cgroup *objcg; @@ -199,6 +201,20 @@ struct zswap_entry { static struct xarray *zswap_trees[MAX_SWAPFILES]; static unsigned int nr_zswap_trees[MAX_SWAPFILES]; =20 +/* Nodemask for compressed RAM nodes used by zswap_compress_direct */ +static nodemask_t zswap_direct_nodes =3D NODE_MASK_NONE; + +void zswap_add_direct_node(int nid) +{ + node_set(nid, zswap_direct_nodes); +} + +void zswap_remove_direct_node(int nid) +{ + if (!node_online(nid)) + node_clear(nid, zswap_direct_nodes); +} + /* RCU-protected iteration */ static LIST_HEAD(zswap_pools); /* protects zswap_pools list modification */ @@ -716,7 +732,13 @@ static void zswap_entry_cache_free(struct zswap_entry = *entry) static void zswap_entry_free(struct zswap_entry *entry) { zswap_lru_del(&zswap_list_lru, entry); - zs_free(entry->pool->zs_pool, entry->handle); + if (entry->direct) { + struct page *page =3D (struct page *)entry->handle; + + node_private_freed(page); + __free_page(page); + } else + zs_free(entry->pool->zs_pool, entry->handle); zswap_pool_put(entry->pool); if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); @@ -849,6 +871,58 @@ static void acomp_ctx_put_unlock(struct crypto_acomp_c= tx *acomp_ctx) mutex_unlock(&acomp_ctx->mutex); } =20 +static struct page *zswap_compress_direct(struct page *src, + struct zswap_entry *entry) +{ + int nid; + struct page *dst; + gfp_t gfp; + nodemask_t tried_nodes =3D NODE_MASK_NONE; + + if (nodes_empty(zswap_direct_nodes)) + return NULL; + + gfp =3D GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE | + __GFP_THISNODE; + + for_each_node_mask(nid, zswap_direct_nodes) { + int ret; + + /* Skip nodes we've already tried and failed */ + if (node_isset(nid, tried_nodes)) + continue; + + dst =3D __alloc_pages(gfp, 0, nid, &zswap_direct_nodes); + if (!dst) + continue; + + /* + * Check with the device driver that this page is safe to use. + * If the device reports an error (e.g., compression ratio is + * too low and the page can't safely store data), free the page + * and try another node. + */ + ret =3D node_private_allocated(dst); + if (ret) { + __free_page(dst); + node_set(nid, tried_nodes); + continue; + } + + goto found; + } + + return NULL; + +found: + /* If we fail to copy at this point just fallback */ + if (copy_mc_highpage(dst, src)) { + __free_page(dst); + dst =3D NULL; + } + return dst; +} + static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -860,6 +934,17 @@ static bool zswap_compress(struct page *page, struct z= swap_entry *entry, gfp_t gfp; u8 *dst; bool mapped =3D false; + struct page *zpage; + + /* Try to shunt directly to compressed ram */ + zpage =3D zswap_compress_direct(page, entry); + if (zpage) { + entry->handle =3D (unsigned long)zpage; + entry->length =3D PAGE_SIZE; + entry->direct =3D true; + return true; + } + /* otherwise fallback to normal zswap */ =20 acomp_ctx =3D acomp_ctx_get_cpu_lock(pool); dst =3D acomp_ctx->buffer; @@ -913,6 +998,7 @@ static bool zswap_compress(struct page *page, struct zs= wap_entry *entry, zs_obj_write(pool->zs_pool, handle, dst, dlen); entry->handle =3D handle; entry->length =3D dlen; + entry->direct =3D false; =20 unlock: if (mapped) @@ -936,6 +1022,15 @@ static bool zswap_decompress(struct zswap_entry *entr= y, struct folio *folio) int decomp_ret =3D 0, dlen =3D PAGE_SIZE; u8 *src, *obj; =20 + /* compressed ram page */ + if (entry->direct) { + struct page *src =3D (struct page *)entry->handle; + struct folio *zfolio =3D page_folio(src); + + memcpy_folio(folio, 0, zfolio, 0, PAGE_SIZE); + goto direct_done; + } + acomp_ctx =3D acomp_ctx_get_cpu_lock(pool); obj =3D zs_obj_read_begin(pool->zs_pool, entry->handle, acomp_ctx->buffer= ); =20 @@ -969,6 +1064,7 @@ static bool zswap_decompress(struct zswap_entry *entry= , struct folio *folio) zs_obj_read_end(pool->zs_pool, entry->handle, obj); acomp_ctx_put_unlock(acomp_ctx); =20 +direct_done: if (!decomp_ret && dlen =3D=3D PAGE_SIZE) return true; =20 @@ -1483,7 +1579,13 @@ static bool zswap_store_page(struct page *page, return true; =20 store_failed: - zs_free(pool->zs_pool, entry->handle); + if (entry->direct) { + struct page *freepage =3D (struct page *)entry->handle; + + node_private_freed(freepage); + __free_page(freepage); + } else + zs_free(pool->zs_pool, entry->handle); compress_failed: zswap_entry_cache_free(entry); return false; --=20 2.52.0