From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F01B4194A54 for ; Mon, 18 Nov 2024 10:31:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925870; cv=none; b=u2hk+Kl7BTISXC6bHR0G0DK0hy8wUOAq9gQiG0ETeU41xMq/DBcGs5PkGs6zQ21gAvDIFmrzCh2z+rrRQOI6nq6voSF4xkXKWTW/zRbThaLZYbsBOAyBUsy+lgBcv4fO9KAxRQD+hobjIhZa/XStjvhwzh8IwLRRUP4Cnrx9KJc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925870; c=relaxed/simple; bh=Iv+4BLJ1ZpMsSNIr1yjl5CpmahtKfvS6PpzJl9HXyyE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M68yK+OjCkMtp/zv0kz+PeZ0fmTTmTfTnme/ht6TBRMxhGoPcsocOgtVhmKz23HRST0AqGKgYzCEJg0dnhCdL1LqCKpzFMBr7jpI1YC/K0BpqCJqo9VGOWhCNxFkCocWC8VDtzIFrffTSex9VP0cP0B01ewJBJY9NyUFRGbuBP0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Z76QeQVH; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Z76QeQVH" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-7240fa50694so2344087b3a.1 for ; Mon, 18 Nov 2024 02:31:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925868; x=1732530668; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mmtF7lyA0iLc/Yt6HZB9ijwUzmrAEUIH4YeSOeYoLmo=; b=Z76QeQVHlECt/XVLMxty9PEfeW8BmHuin7++b+zXsZoIRnDLVzXA77ElZ6wJ0ihNXg keDbO8T5CEpGgHVVlc5UJWJ07EHQphAoRb0VQQZjCtCdf5dJQheZyN4KPXt45PFyc8JS IG/jaXc2XDI8/FCTg5g3MVBnAVHkL2rPZ79YA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925868; x=1732530668; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mmtF7lyA0iLc/Yt6HZB9ijwUzmrAEUIH4YeSOeYoLmo=; b=BEhGQTApx/t17kntg0P9wuz9eyxJPBtSiNEwr23WNGTkvTHxW1bJ6lmr/gFAX772DL 15I/dx5FxCY9gzryp5oKMrJwy2VPwme9IkJfc7N3wbqJhrUTsEikpEeZvWfLCj9MOB3x rywbpvaH2++2FlR+YTTB3ypVxrinXqsr5IQaIT+nfAvHE08i4977dA0zyhW+cybKB6cD lRj5E6sVxHXPvS7YZw42GAK7qa4KN13rUuJR+lPD0roEHRvoBZnruATXAJc2JHJav9+z bNlZ+wHc0TBZxNHwnWvhhZXoOTfDcKdGj3cLtyS5Y0YH6iCbtku5boXhf1OdemWfP+dv d2aQ== X-Gm-Message-State: AOJu0YyCgZiqk/fFPMLf8Y3Kd8m8dU5b3KpPEeWA/q9xjaWp+DSJgnoC Q8eamwSQBbpP52d1yXLHYdB5vrpcrQmXaJAOBEa5uKcnwUjHj9ywSfUMFd/mbq+xX8xwzw9FG6w = X-Google-Smtp-Source: AGHT+IGAGA2exT/xw+8VVpdGTtxlUnBRppXPyEFCjjAjHw4kKWYrB2li3/2hnqjgbK6eOUVKVyb6fQ== X-Received: by 2002:a17:90b:250d:b0:2ea:2a8d:dd24 with SMTP id 98e67ed59e1d1-2ea2a8decb2mr11299390a91.24.1731925868295; Mon, 18 Nov 2024 02:31:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2ea7752f0a0sm1554913a91.50.2024.11.18.02.31.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:07 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 1/8] zram: cond_resched() in writeback loop Date: Mon, 18 Nov 2024 19:30:32 +0900 Message-ID: <20241118103054.3191944-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Writeback loop can run for quite a while (depending on wb device performance, compression algorithm and the number of entries we writeback), so we need to do cond_resched() there, similarly to what we do in recompress loop. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 3dee026988dc..882a32d46a75 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -883,6 +883,8 @@ static ssize_t writeback_store(struct device *dev, next: zram_slot_unlock(zram, index); release_pp_slot(zram, pps); + + cond_resched(); } =20 if (blk_idx) --=20 2.47.0.338.g60cca15819-goog From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C51EE194C8D for ; Mon, 18 Nov 2024 10:31:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925874; cv=none; b=XRERD8ZWDcvJFdhiKMrwkbjQkCgNQgSdPPvdzS8SXAzdPP4kCiC+O4ycKtM4eFAjW6j/rhg3Dlq3wfWywyXaJHsOk30N4SbWCl9xwEIXWkERTWdfQLBFQU5UzkyYGurIAttXFjzQiW2fvy5sgQECPUKzoZ9StbTlYoNoOqqNWvc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925874; c=relaxed/simple; bh=j1fz5OKcxtFtYCZ1AdIFbpu/JJQZ5mET+VDZ0WiyZxI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ADxfIlLzQu/QfvHb7tWSxRbFaG6zbrFjW0E2LWoZoB40BaUBOw/0rt7ni/SNaPhoHhVg7Doiv51ZL4gUqzkwtFuUMGWO0et9PNeoXo548a+PLGsPKEFVegxoA7E6Q4HwimJcCi5HMU9F0nJe2I+o3Yj1TzZcD5EcOfOUADt1T9s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=e4k+a7vY; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="e4k+a7vY" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-20cb47387ceso39722065ad.1 for ; Mon, 18 Nov 2024 02:31:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925872; x=1732530672; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I32VO0apGgp4lSLTII1ojr1S39dox5QX5dFI6JIsBCc=; b=e4k+a7vYHBa+NOO5LyWkplgpDnLAWSEWbBE1S3Y50S+SLtCZojy2QznE+12SaYzGzB PZZfVXBKfJp30nbhWOsbYq1AuSG59Ys+enhItUE2gPA4YtMbzgeyRMtw77OxQYcbOrOQ fnxXbhJTBSGRE2v1kMM5HhNR3CfhFLi/QI4ek= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925872; x=1732530672; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I32VO0apGgp4lSLTII1ojr1S39dox5QX5dFI6JIsBCc=; b=FG2OX1+tuARC1ZUzgChDHeSy0APz0+K4rph6W1S3AbpMFsui96dShOTwFQH8Wg7ruB oKpmX6xAZ3hpcxAVslWz762bvkQR3AKjFssTIuKx/FrYN25eqGoDlJraTBwkeZaK2ROJ sB7MwYMOSzWMW4Ti7BK7Hup5vTwakwK4LJNEiEjIyYBxSDuYtxKYWvEWgy2481/iC+vf Frt9aTC6iaLKPgViMCTioJx+Ry/dSJ98BkRFcZs1DkiI6ZH5x3OZSH6n4Ru4M+cn8R++ W2PrQgPvhZVVB2pQ7SmhYBwA84/VIQSEEr/yDZzQG3uKwJoa2b1jYx61KzItCEcRsh05 qRjg== X-Gm-Message-State: AOJu0YzUpE+5y3MZHc/Q5o2wD6e0jkNHGJYso7sWdA55VpXd4SdUFWI0 2CdHfJMgFw+cPlnq6A4OXxWjFSl88sO/X1B9az1ktwHRDKbvrhB/7aksKszAng== X-Google-Smtp-Source: AGHT+IFoGGct5z8epVdmpuexs7KBF7JTmI34F+LDla+ntNP+aR67ePG2UlBhF15sO6NZTrd1/iZTxA== X-Received: by 2002:a17:902:f710:b0:212:c9d:5f62 with SMTP id d9443c01a7336-2120c9d604amr74698155ad.52.1731925872289; Mon, 18 Nov 2024 02:31:12 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2ea56426954sm2895527a91.42.2024.11.18.02.31.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:11 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 2/8] zram: free slot memory early during write Date: Mon, 18 Nov 2024 19:30:33 +0900 Message-ID: <20241118103054.3191944-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the current implementation entry's previously allocated memory is released in the very last moment, when we already have allocated a new memory for new data. This, basically, temporarily increases memory usage for no good reason. For example, consider the case when both old (stale) and new entry data are incompressible so such entry will temporarily use two physical pages - one for stale (old) data and one for new data. We can release old memory as soon as we get a write request for entry. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 882a32d46a75..987d72f2249c 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1640,6 +1640,11 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) unsigned long element =3D 0; enum zram_pageflags flags =3D 0; =20 + /* First, free memory allocated to this slot (if any) */ + zram_slot_lock(zram, index); + zram_free_page(zram, index); + zram_slot_unlock(zram, index); + mem =3D kmap_local_page(page); if (page_same_filled(mem, &element)) { kunmap_local(mem); @@ -1728,13 +1733,7 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); out: - /* - * Free memory associated with this sector - * before overwriting unused sectors. - */ zram_slot_lock(zram, index); - zram_free_page(zram, index); - if (comp_len =3D=3D PAGE_SIZE) { zram_set_flag(zram, index, ZRAM_HUGE); atomic64_inc(&zram->stats.huge_pages); --=20 2.47.0.338.g60cca15819-goog From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B7221946A1 for ; Mon, 18 Nov 2024 10:31:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925879; cv=none; b=StL1qOFDOn5VTLDV7KswbRum7as3pb7m/oE9M/4KfoIENnwrsHLORqaK58U90ay4t3qQ8TbpscqkcOrRFJic3xAUy/MFPBMmr98yLvJeE213tCYKFasQ18EYQkUVRwfCAZ0VMxbO/uYXcz81Zc3OWzXxdTinPVqy3mQ1GVT41Ig= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925879; c=relaxed/simple; bh=Yz9Nexpl0EVIA2bJU+qi4XgK/SkBCABbsb+hclhAJIc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ao7vidVxTUhuB7VMDub0mkEYkxbzAOtf3zZvjiynoDzODgbqVNdVg9V3xk5iBQrcsewNkFVjfLpBLI7MYRUdfHGCjDWmxydsRWgd8TdcoV0chMG2JF1e3T+4niJj+Gm2908Emat5tc6iqZlPDvbNjXhGRjr8fNjPLGW72ETIetU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Y7B4ZCc3; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Y7B4ZCc3" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2120f9ec28eso6300235ad.1 for ; Mon, 18 Nov 2024 02:31:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925876; x=1732530676; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f+QCKNKqCnC0lwbQ04xjWB43TgDsU0jXExoWEZYBF2o=; b=Y7B4ZCc3nWT+OElFSEXQH+kTmi3UCTI96TotYwML+FFVizsdZlVZfFIKFtGc6kF9xO IH4YhNkkyMrbTWVt9xZ7vYItLbTotB0LzeJUkhfL7dWUy3LHyQDSt37fQ2FPKfo2w5N8 NmJeXL0B89qrkwNtgLfA1Gzb1PycbZhmca7eA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925876; x=1732530676; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f+QCKNKqCnC0lwbQ04xjWB43TgDsU0jXExoWEZYBF2o=; b=QrXltQ3X07LT1PX1ti1fsNY9mqJJJGD7ppOQPabRnzYW9SkC0mVerFYqkN2lcih5VS yhmoRWXSOAd47kcimAv14E+PcynTVQZQFWR6IsQYbdJIL5w+f8FeYU8/6+x7wnNt7cdS DBvsXtjC38lTNmrjtFsBfj38llZ6RZl0B1596iTebeVX9khDoqK1gw+sCR05Qh9OCtru 2kUXTKFyt91Q7jp6lPYu8UVz7uDliS6XLsK0mcSwFbDDikf5oqP4sVf9oofy//IouS+c 1bH0L1YN336k08JeMXEgV+Zk30cVdLyuza7lNP/BKbqZKODgWSCGLajFn0WxNU6tXzEe 87bw== X-Gm-Message-State: AOJu0YwwASsYmkkn7bosNWift/XNet5Yj0jlnbcFjjXC1sqgTUAHQ/xy DIAAbGhceXLVCTaCPfUo/CC3AdhjpO4sFAzpEwpzW60LVdRSo3EWx+ukcDzyTlAivX0mA7U0BNU = X-Google-Smtp-Source: AGHT+IFOjKJUn7Imh7y1iaNZa2OAFuXSL2vTdjy9vBoribW9ja/Y+3zukd9hwr07nuUCTisIT/gkvg== X-Received: by 2002:a17:902:d509:b0:212:4b3:1080 with SMTP id d9443c01a7336-21204b3158bmr76843935ad.13.1731925876498; Mon, 18 Nov 2024 02:31:16 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-7f8c1c644c8sm5600494a12.47.2024.11.18.02.31.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:15 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 3/8] zram: remove entry element member Date: Mon, 18 Nov 2024 19:30:34 +0900 Message-ID: <20241118103054.3191944-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Element is in the same anon union as handle and hence holds the same value, which makes code below sort of confusing handle =3D zram_get_handle() if (!handle) element =3D zram_get_element() Element doesn't really simplify the code, let's just remove it. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 23 +++++------------------ drivers/block/zram/zram_drv.h | 5 +---- 2 files changed, 6 insertions(+), 22 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 987d72f2249c..e80b4d15b74b 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -112,17 +112,6 @@ static void zram_clear_flag(struct zram *zram, u32 ind= ex, zram->table[index].flags &=3D ~BIT(flag); } =20 -static inline void zram_set_element(struct zram *zram, u32 index, - unsigned long element) -{ - zram->table[index].element =3D element; -} - -static unsigned long zram_get_element(struct zram *zram, u32 index) -{ - return zram->table[index].element; -} - static size_t zram_get_obj_size(struct zram *zram, u32 index) { return zram->table[index].flags & (BIT(ZRAM_FLAG_SHIFT) - 1); @@ -873,7 +862,7 @@ static ssize_t writeback_store(struct device *dev, =20 zram_free_page(zram, index); zram_set_flag(zram, index, ZRAM_WB); - zram_set_element(zram, index, blk_idx); + zram_set_handle(zram, index, blk_idx); blk_idx =3D 0; atomic64_inc(&zram->stats.pages_stored); spin_lock(&zram->wb_limit_lock); @@ -1496,7 +1485,7 @@ static void zram_free_page(struct zram *zram, size_t = index) =20 if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); - free_block_bdev(zram, zram_get_element(zram, index)); + free_block_bdev(zram, zram_get_handle(zram, index)); goto out; } =20 @@ -1540,12 +1529,10 @@ static int zram_read_from_zspool(struct zram *zram,= struct page *page, =20 handle =3D zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { - unsigned long value; void *mem; =20 - value =3D handle ? zram_get_element(zram, index) : 0; mem =3D kmap_local_page(page); - zram_fill_page(mem, PAGE_SIZE, value); + zram_fill_page(mem, PAGE_SIZE, handle); kunmap_local(mem); return 0; } @@ -1591,7 +1578,7 @@ static int zram_read_page(struct zram *zram, struct p= age *page, u32 index, */ zram_slot_unlock(zram, index); =20 - ret =3D read_from_bdev(zram, page, zram_get_element(zram, index), + ret =3D read_from_bdev(zram, page, zram_get_handle(zram, index), parent); } =20 @@ -1742,7 +1729,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) =20 if (flags) { zram_set_flag(zram, index, flags); - zram_set_element(zram, index, element); + zram_set_handle(zram, index, element); } else { zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 134be414e210..db78d7c01b9a 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -62,10 +62,7 @@ enum zram_pageflags { =20 /* Allocated for each disk page */ struct zram_table_entry { - union { - unsigned long handle; - unsigned long element; - }; + unsigned long handle; unsigned int flags; spinlock_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME --=20 2.47.0.338.g60cca15819-goog From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70933197552 for ; Mon, 18 Nov 2024 10:31:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925882; cv=none; b=WkURduHfKAxHeKpXjm+8lEv/FdPRC/0v0cFB7r4hk8HpTebeNPkopNu/QOQtINdmKkozjDbhBt181pxxrUdYTV3zOJMsDx6LxNTeNgV8HHQeAs0ASGQQKBisXT+tW27QbnXu4KvnO5yT4BCyz0fyVt3wqGoEoG3+qDIiKM0UIro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925882; c=relaxed/simple; bh=mIqXYsg/C2qrCITDcJd1YaJTS14sI9c5kfQAcRy8Hek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZyuvZap4R4zFVm0UFJV3Y/xJZm5nJM3LeRjow7lJIwuzRYoFKWb6JCjTMIYWJX6kt2Jq64j2xoQqzsH+eS6LvAJww7ZK79B8C7Xb7dC1M9tE52DeNzPyBKkgE84dGIMW9NX7zgxROi9VL9UObJPV8cQnDF7AdLPl/+X1cO1CPts= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=WVr1lEiV; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="WVr1lEiV" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-7ee7e87f6e4so1515888a12.2 for ; Mon, 18 Nov 2024 02:31:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925880; x=1732530680; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dkzCN3UhBBpnf4e6T+uWXcn8BnX32ULT9Gcxhq6EsGQ=; b=WVr1lEiVhOY4lCMKO05VQ7GCsuBhiYKD8X3eXOzAWYl26qgFTKjW67/Y4jBLJMGqCb Jjcd9DZP0bcqEVqV3Mnyssq/SjTkmWth51DWb488XKdrkT3y0/zlhirgYY8qtAYpXmS7 Tf2zPUAGqzlbCHHcxIGkFA7NzOiVD5ZBcV7qU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925880; x=1732530680; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dkzCN3UhBBpnf4e6T+uWXcn8BnX32ULT9Gcxhq6EsGQ=; b=WciXd03ouoVYXiFRYfcrte4DTs60WCRPWiGzBzfy6ISCyAXA+Mp1/WSjtqtNB6xl3B hvSSr4piiu19yywl3JuedwjawCjBt31z5swMSqZQwWnTshLDdf0sJk9lyl/ahO9A61Ma 6ypVhoggBkXmQl6vrIkd7Gys/Os3VwS+mQpd3qYVoGzXOUE6W6Aa7zv/qYO5mwGkxmnq DfrvA7cYWpP/nEappA7kkygAvdWYlTFsusWl5LmoJRtQ3U5x9pxa41geRgN4x3QjUYaj NHn/lpwf04Rh11mqSyE6RpXXGKchbv/vDwwSCsVi5v13lCt2RGZLhpHRfW9AhxguWlk7 +nrg== X-Gm-Message-State: AOJu0YwaaPpxgSIUJWsTcgYp1N79NjdZ80TGqPBPldS4i3Sw17f+tb32 TrgrNUzwI5R7++OiG6qy17ysg4ZFinAt8L/A5WqRlJ7UkfZOsdPr8AV2tE3iYg== X-Google-Smtp-Source: AGHT+IEfqhqURqDaD0G6JGPl35VcW+CAUQ5rdIfn+KXLAYTZYCO5p9vfNExEvOyytiZIAr03Fg1bqw== X-Received: by 2002:a05:6a20:9144:b0:1db:edb3:ed87 with SMTP id adf61e73a8af0-1dc90b482cbmr17702579637.22.1731925880281; Mon, 18 Nov 2024 02:31:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-7f8c1dce9afsm5653798a12.82.2024.11.18.02.31.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:19 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 4/8] zram: handle ZRAM_SAME immediately Date: Mon, 18 Nov 2024 19:30:35 +0900 Message-ID: <20241118103054.3191944-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Handling of ZRAM_SAME now uses a goto to the final stages of zram_write_page() plus it introduces a branch and flags variable, which is not making the code any simpler. In reality, we can handle ZRAM_SAME immediately when we detect such pages and remove a goto and a branch. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e80b4d15b74b..76ff74a75125 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1625,7 +1625,6 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) void *src, *dst, *mem; struct zcomp_strm *zstrm; unsigned long element =3D 0; - enum zram_pageflags flags =3D 0; =20 /* First, free memory allocated to this slot (if any) */ zram_slot_lock(zram, index); @@ -1635,10 +1634,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) mem =3D kmap_local_page(page); if (page_same_filled(mem, &element)) { kunmap_local(mem); - /* Free memory associated with this sector now. */ - flags =3D ZRAM_SAME; atomic64_inc(&zram->stats.same_pages); - goto out; + zram_set_flag(zram, index, ZRAM_SAME); + zram_set_handle(zram, index, element); + return 0; } kunmap_local(mem); =20 @@ -1719,7 +1718,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); -out: + zram_slot_lock(zram, index); if (comp_len =3D=3D PAGE_SIZE) { zram_set_flag(zram, index, ZRAM_HUGE); @@ -1727,13 +1726,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) atomic64_inc(&zram->stats.huge_pages_since); } =20 - if (flags) { - zram_set_flag(zram, index, flags); - zram_set_handle(zram, index, element); - } else { - zram_set_handle(zram, index, handle); - zram_set_obj_size(zram, index, comp_len); - } + zram_set_handle(zram, index, handle); + zram_set_obj_size(zram, index, comp_len); zram_slot_unlock(zram, index); =20 /* Update stats */ --=20 2.47.0.338.g60cca15819-goog From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69F47198845 for ; Mon, 18 Nov 2024 10:31:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925886; cv=none; b=D/3feC39ve0g1J6xAa1+9GQoE1IwdykMirwwdgwE7aWOa15+jsKC1zFIebUJaCqgch8G9yxdijdoJc3AkQf9+aLlkNCqXNtzERaAQgI5IbzUvetfDLtfnJv9ZBdYsoaWCmqetTxYBm+KHI48WKSTF67cSmzShmLnaWipHecW9Gc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925886; c=relaxed/simple; bh=4qnAzf0A9GYx/UXMtDh029xq1MIEq1qKhYC5v8zrURM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f+3Wcafwfi8Zy4ZgRjT41iiwQ1hM/jorpM9933YQOxr47i8kaRRKZquaJ0gC5A6fps+aHpRADf4Uu6A0ov3r1Kg/cfnHmq+qCprRm+Rgnmb7SAoS3OQzN8hIi677O2sG5hNFrtbbVVHTFepEPrhJp+aZiWmtHijMZlIAKjOdh/A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=OKxiDFRG; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="OKxiDFRG" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-7f8b37edeb7so2964102a12.0 for ; Mon, 18 Nov 2024 02:31:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925884; x=1732530684; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CyFU2BHIoGV0dtOXeqkkgnpAt0Qyo/99bFGxHgdr5lo=; b=OKxiDFRGR5rm9mDgB0A1faPudy2P1GUZbMVWgrOtFIEz0dzAcm1BYLLkYElhNPOCnO Gkq1KyAmEK+rP2I8ZjuoOCd3eW+JmNce48tOxX0UuRzPCTSpz1/WEj5bGoo6jKgINkwH p/TBN5u9n7alCzRpHSVE4lvPaBuvazUIRCZFA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925884; x=1732530684; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CyFU2BHIoGV0dtOXeqkkgnpAt0Qyo/99bFGxHgdr5lo=; b=lN5FtZysufBo3limnvsJiAA7GJaYXXQTDlf8+Dyaa65lYhw1cnYp9DqqcCU7ACKvkQ eq5QbRTxn4RtvcNCt0CwGsS2SJO+HyIvRPZKyAfCus2bS3hvJkzz+ItilZnbPjOmyNKu NoanR7iOPYP8vtBvesVbmD5i/7URWyWue2YGJhbPKFz7nxIhGiS/7D44otpYhitDZYG3 zgbCI2NKppDK4dKODL2RWNmDHEgSeNgv8uCD/Par2mRjL6VnlxjyswwliR6Aq4Ei8s89 +E9mdGiuPK399aVIVo37m9xghxZl1KPwDG0zRBAiu8/m9bK9h4VVu8JB0W4lmrDIwpSh DqzQ== X-Gm-Message-State: AOJu0YyVYug2zgO/Sh9zMLSTGFeTb89H1L5UfzFwpzFtug8mdgpVVZ9A lV6v22mIz7um9jxoMk0zZEnbRhtBzxUrWwLK/kwm6sL9QCVeBEyJNKe0oZY82deD3zL8pqJ2MpM = X-Google-Smtp-Source: AGHT+IE/bDybxqpTkSZHBlnC8w5UhDkGvZcj3Hi/AAd2MwYPMGiJDBe7j3AJX2xvHHra4twLVoYmkg== X-Received: by 2002:a05:6a21:339f:b0:1db:d869:9204 with SMTP id adf61e73a8af0-1dc90b28fbbmr17074348637.11.1731925883662; Mon, 18 Nov 2024 02:31:23 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-7f8c1c62ccfsm4802390a12.43.2024.11.18.02.31.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:23 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 5/8] zram: factor out ZRAM_HUGE write Date: Mon, 18 Nov 2024 19:30:36 +0900 Message-ID: <20241118103054.3191944-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zram_write_page() handles 3 different cases: ZRAM_SAME pages (which was already simplified in previous patches) writes, regular pages writes and ZRAM_HUGE pages writes. ZRAM_HUGE handling adds a significant amount of complexity: all those conditional src for copy_page(), etc., a conditional jump backward for the fallback handle allocation and so on. Instead, we can handle ZRAM_HUGE in a separate function and remove quite a bit of that complexity at a cost of minor code duplication (basically, only zram stats updates). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 140 +++++++++++++++++++++------------- 1 file changed, 85 insertions(+), 55 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 76ff74a75125..55ffd0534470 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -132,6 +132,30 @@ static inline bool zram_allocated(struct zram *zram, u= 32 index) zram_test_flag(zram, index, ZRAM_WB); } =20 +static inline void update_used_max(struct zram *zram, const unsigned long = pages) +{ + unsigned long cur_max =3D atomic_long_read(&zram->stats.max_used_pages); + + do { + if (cur_max >=3D pages) + return; + } while (!atomic_long_try_cmpxchg(&zram->stats.max_used_pages, + &cur_max, pages)); +} + +static bool zram_can_store_page(struct zram *zram) +{ + unsigned long alloced_pages; + + alloced_pages =3D zs_get_total_pages(zram->mem_pool); + update_used_max(zram, alloced_pages); + + if (!zram->limit_pages) + return true; + + return alloced_pages <=3D zram->limit_pages; +} + #if PAGE_SIZE !=3D 4096 static inline bool is_partial_io(struct bio_vec *bvec) { @@ -266,18 +290,6 @@ static struct zram_pp_slot *select_pp_slot(struct zram= _pp_ctl *ctl) } #endif =20 -static inline void update_used_max(struct zram *zram, - const unsigned long pages) -{ - unsigned long cur_max =3D atomic_long_read(&zram->stats.max_used_pages); - - do { - if (cur_max >=3D pages) - return; - } while (!atomic_long_try_cmpxchg(&zram->stats.max_used_pages, - &cur_max, pages)); -} - static inline void zram_fill_page(void *ptr, unsigned long len, unsigned long value) { @@ -1616,13 +1628,54 @@ static int zram_bvec_read(struct zram *zram, struct= bio_vec *bvec, return zram_read_page(zram, bvec->bv_page, index, bio); } =20 +static int zram_write_incompressible_page(struct zram *zram, struct page *= page, + u32 index) +{ + unsigned long handle; + void *src, *dst; + + /* + * This function is called from preemptible context so we don't need + * to do optimistic and fallback to pessimistic handle allocation, + * like we do for compressible pages. + */ + handle =3D zs_malloc(zram->mem_pool, PAGE_SIZE, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(handle)) + return PTR_ERR((void *)handle); + + if (!zram_can_store_page(zram)) { + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zs_free(zram->mem_pool, handle); + return -ENOMEM; + } + + dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); + src =3D kmap_local_page(page); + memcpy(dst, src, PAGE_SIZE); + kunmap_local(src); + zs_unmap_object(zram->mem_pool, handle); + + zram_slot_lock(zram, index); + zram_set_flag(zram, index, ZRAM_HUGE); + zram_set_handle(zram, index, handle); + zram_set_obj_size(zram, index, PAGE_SIZE); + zram_slot_unlock(zram, index); + + atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); + atomic64_inc(&zram->stats.huge_pages); + atomic64_inc(&zram->stats.huge_pages_since); + atomic64_inc(&zram->stats.pages_stored); + + return 0; +} + static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long alloced_pages; unsigned long handle =3D -ENOMEM; unsigned int comp_len =3D 0; - void *src, *dst, *mem; + void *dst, *mem; struct zcomp_strm *zstrm; unsigned long element =3D 0; =20 @@ -1643,10 +1696,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) =20 compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); - src =3D kmap_local_page(page); + mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, - src, &comp_len); - kunmap_local(src); + mem, &comp_len); + kunmap_local(mem); =20 if (unlikely(ret)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); @@ -1655,8 +1708,11 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return ret; } =20 - if (comp_len >=3D huge_class_size) - comp_len =3D PAGE_SIZE; + if (comp_len >=3D huge_class_size) { + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + return zram_write_incompressible_page(zram, page, index); + } + /* * handle allocation has 2 paths: * a) fast path is executed with preemption disabled (for @@ -1672,66 +1728,40 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) */ if (IS_ERR_VALUE(handle)) handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + __GFP_KSWAPD_RECLAIM | + __GFP_NOWARN | + __GFP_HIGHMEM | + __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) return PTR_ERR((void *)handle); =20 - if (comp_len !=3D PAGE_SIZE) - goto compress_again; - /* - * If the page is not compressible, you need to acquire the - * lock and execute the code below. The zcomp_stream_get() - * call is needed to disable the cpu hotplug and grab the - * zstrm buffer back. It is necessary that the dereferencing - * of the zstrm variable below occurs correctly. - */ - zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); + goto compress_again; } =20 - alloced_pages =3D zs_get_total_pages(zram->mem_pool); - update_used_max(zram, alloced_pages); - - if (zram->limit_pages && alloced_pages > zram->limit_pages) { + if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_free(zram->mem_pool, handle); return -ENOMEM; } =20 dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - src =3D zstrm->buffer; - if (comp_len =3D=3D PAGE_SIZE) - src =3D kmap_local_page(page); - memcpy(dst, src, comp_len); - if (comp_len =3D=3D PAGE_SIZE) - kunmap_local(src); - + memcpy(dst, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); - atomic64_add(comp_len, &zram->stats.compr_data_size); =20 zram_slot_lock(zram, index); - if (comp_len =3D=3D PAGE_SIZE) { - zram_set_flag(zram, index, ZRAM_HUGE); - atomic64_inc(&zram->stats.huge_pages); - atomic64_inc(&zram->stats.huge_pages_since); - } - zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); zram_slot_unlock(zram, index); =20 - /* Update stats */ atomic64_inc(&zram->stats.pages_stored); + atomic64_add(comp_len, &zram->stats.compr_data_size); + return ret; } =20 --=20 2.47.0.338.g60cca15819-goog From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1322198A35 for ; Mon, 18 Nov 2024 10:31:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925890; cv=none; b=RVBrnYxZuwO1qMGkNKGs+e+i/6uOtor3HMqGe/NLgGjHbWlaeahQnEImnU7lK/Mbr3fDMGlqsgRiYUhIIjdrhpAJa/5X5Fi29E9E5r4R4rQVnCt6tQLDJaTVm0XPVqlrfAOGgpMBs0463Yr2q1p+lEz8tC6Kkw3oKnOa9HLF6dE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925890; c=relaxed/simple; bh=URZDAqrp93EaHp3z5wCKmGin51pgZg/9Vvo9b4ImMZQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N+YcAf018iQ0zIUB1MSbr9Grw4+KWvugtF13s0HL/n9n4qDsVErPVFbuQFygtTkLniPCeqPwVkAiCMuGqyuYaV7jDcfIKXUu7NrbOjdeiOlk1t+5KMs05FDlU1QqxMgenZPBiMdI0jmMID8pN71lLw10mATFpJSGKp/3uIitKZ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=EwbzPl5y; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="EwbzPl5y" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-211a4682fcaso26290995ad.2 for ; Mon, 18 Nov 2024 02:31:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925887; x=1732530687; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=90w3pZT9FJvUmWsGdeMpl5t+GScBHGHvQ60IcDxysHU=; b=EwbzPl5y2Ewa8Z7T3CFoD6o4/YwVNxJH9+wPgkXjXROtjv/OfxKWvrSKXneqW5+w6C eLm8+CNBWF/fPBwy99ooXvDNnt258Uz8vLhes2pyy0lvZoh0/EfB/l+8KTcBl7ZhgzUl 7r/X7GiRFKDOEoYkzDuea9xfwVmU9D5HVeGvk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925887; x=1732530687; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=90w3pZT9FJvUmWsGdeMpl5t+GScBHGHvQ60IcDxysHU=; b=BBIaa6J+96+Y3V33O/LW+n1L1QhbIzyZbGFyF/acijCYbhZ5QV7mv8lu7CiFcN8AU1 qFSnnESiAR+G2YEqcrcnsoWblC3aOsvlGb7T3/R+bmpkXMZzXd66fi2BmLeClDFNDT4F Wivn6/msTlINwtoUuQN0Zk0cHNLIizYSOI96i3UbS4Rs2DlJstG96eE9ZYT6s76q/+gg p3MWvj7W8quapTDLuKj4nEwyu1msr+LeHvGsJjpXi6Z4eAwpcOr6cWWQPxbh5vqX78P4 jynfUZHS7M/K7tPzet6xAp1/zWpdJmldIC30CXTR+VWkPsDsAL1veLkmEk9dqGzwDnys j9uQ== X-Gm-Message-State: AOJu0YwjLX0YuItbT++KtiQT6vnd02EXj2qvNDkSYfCqWCji/Zpz4cxC tWoC67uQGgD848aGWOZ3UHAyue+yawWA3xM+n8z2efOpOQdxMEAZ/GmXQQz39bHXu2LcOCoQrJQ = X-Google-Smtp-Source: AGHT+IGx/uWi0Il9dEaPy65pYV4oOs3sdmOEUPIIfBWJ7G3iE5vWr6wc30DSLaD5I+/GGx6VPa0fcA== X-Received: by 2002:a17:903:32c7:b0:20d:2848:2bee with SMTP id d9443c01a7336-211d0d72d87mr172575205ad.16.1731925887044; Mon, 18 Nov 2024 02:31:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-211d0dc32acsm54573725ad.53.2024.11.18.02.31.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:26 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 6/8] zram: factor out ZRAM_HUGE read Date: Mon, 18 Nov 2024 19:30:37 +0900 Message-ID: <20241118103054.3191944-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similarly to write, move ZRAM_HUGE read handlig to a separate function. This will make more sense with introduction of multi-handle entries later in the series. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 71 ++++++++++++++++++++++------------- 1 file changed, 44 insertions(+), 27 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 55ffd0534470..745207597e23 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1525,6 +1525,46 @@ static void zram_free_page(struct zram *zram, size_t= index) zram_set_obj_size(zram, index, 0); } =20 +static int read_incompressible_page(struct zram *zram, struct page *page, + u32 index) +{ + unsigned long handle; + void *src, *dst; + + handle =3D zram_get_handle(zram, index); + src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + dst =3D kmap_local_page(page); + copy_page(dst, src); + kunmap_local(dst); + zs_unmap_object(zram->mem_pool, handle); + + return 0; +} + +static int read_compressible_page(struct zram *zram, struct page *page, + u32 index) +{ + struct zcomp_strm *zstrm; + unsigned long handle; + unsigned int size; + void *src, *dst; + int ret, prio; + + handle =3D zram_get_handle(zram, index); + size =3D zram_get_obj_size(zram, index); + prio =3D zram_get_priority(zram, index); + + zstrm =3D zcomp_stream_get(zram->comps[prio]); + src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + dst =3D kmap_local_page(page); + ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); + kunmap_local(dst); + zs_unmap_object(zram->mem_pool, handle); + zcomp_stream_put(zram->comps[prio]); + + return ret; +} + /* * Reads (decompresses if needed) a page from zspool (zsmalloc). * Corresponding ZRAM slot should be locked. @@ -1532,12 +1572,7 @@ static void zram_free_page(struct zram *zram, size_t= index) static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index) { - struct zcomp_strm *zstrm; unsigned long handle; - unsigned int size; - void *src, *dst; - u32 prio; - int ret; =20 handle =3D zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { @@ -1549,28 +1584,10 @@ static int zram_read_from_zspool(struct zram *zram,= struct page *page, return 0; } =20 - size =3D zram_get_obj_size(zram, index); - - if (size !=3D PAGE_SIZE) { - prio =3D zram_get_priority(zram, index); - zstrm =3D zcomp_stream_get(zram->comps[prio]); - } - - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); - if (size =3D=3D PAGE_SIZE) { - dst =3D kmap_local_page(page); - copy_page(dst, src); - kunmap_local(dst); - ret =3D 0; - } else { - dst =3D kmap_local_page(page); - ret =3D zcomp_decompress(zram->comps[prio], zstrm, - src, size, dst); - kunmap_local(dst); - zcomp_stream_put(zram->comps[prio]); - } - zs_unmap_object(zram->mem_pool, handle); - return ret; + if (!zram_test_flag(zram, index, ZRAM_HUGE)) + return read_compressible_page(zram, page, index); + else + return read_incompressible_page(zram, page, index); } =20 static int zram_read_page(struct zram *zram, struct page *page, u32 index, --=20 2.47.0.338.g60cca15819-goog From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0547198E9E for ; Mon, 18 Nov 2024 10:31:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925893; cv=none; b=beXInYwinQQx6+J1ZwqY2RigokoKxZVAo1hdOYKZFe8fkFHQUzJ/GwZjEJ1F3D2TSRiBA6HC/qqOhGjJ+GTtS7fUHLGY3upD2FIAHIGMCEQAonC3UWFiqC0B43nHk5WmOJukYNnFzx7JKGyiwde+71fS9I1ubFdGRqENbjuwdnI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925893; c=relaxed/simple; bh=xVzGpsMG9LnU1tZySVmIfVy5ZCYPebEJXiildOSUkyA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=F8cJMoUuEx5zlWc+Tv0V3na385tVZa0brvgSX1sdkN2JwB0acc802pkncHvZwYIy5U2KlWuI2Y0rOdhIk0tu2L1Kwh/SKYDfkqNQfPw6f5/3ikRiB7coGQvsl3WFDOthBJ34bMKR4/O74xqfvC3Sjdx3YA0YPSKiLLaY0aYzOtw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=n/N+bMQC; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="n/N+bMQC" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7ea8de14848so1447721a12.2 for ; Mon, 18 Nov 2024 02:31:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925890; x=1732530690; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dzBs8nZr7t4qyXsSXVR6Mvnn4ykaCUyDwpEIlMiVaNs=; b=n/N+bMQCYZtkta5/l9hPnLQZ87TgfOHKbhujL2DPkwyvh0gz6DxTiDlD/KUoK7TQny 111mWDiHJjWfobdv2vvsH7fQkZHgqL3YvuB0zg2EkkJWxSILOR/hG6Mo+jrReAcoPY7/ aPY6VClOcuOWUVUahFZvCtk4Ro02H6PCqirJs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925890; x=1732530690; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dzBs8nZr7t4qyXsSXVR6Mvnn4ykaCUyDwpEIlMiVaNs=; b=C63MRCcmwNGYMgzWP4FRxwbhsozzF6l+l/fg+zQQUGWzo98KDLdz7/nqbO7V0RsgK8 yy5D40pzGwrVYrU2y2vnaCS/5UvwT1/3S5umZXW634Uvj8sozdvedcPId45gg+RA8o3a PWcxxS9CdDR8sKxJZcM/m86adHsGtZu4F/f3ECDQEs8w1klber3jBwnP8PbmNmpx9Fvi MpT0J6Sfa8UwpXhOLRCrE9IJsVR9tUQEYEBKborU+KYabh4R0c7Mz0yeAFM+c79Ip347 d8k7G1obijbIAD/As0nS13so8PIU3T/GQR1UMqsRFALHbgPooEeLSmP0kkApl9l2sYv0 njEw== X-Gm-Message-State: AOJu0YwNKsvzpsOEo/L80XYC7uH5v/xcGiDh8GxGaBIGkixzwbC9+Vmm pIb+JidbDTQIpDAiszbdOyAsvX8PwS1L8PKWFtXRbKbN4tfzwXenvzWn+1ArW0mKq4sujH+7+7o = X-Google-Smtp-Source: AGHT+IFGhf9MPrSPXzUeQ74KTVU1NoLTG/jtQEg9tsUyi5MjDWNEBL8DjEi9nzYhqML2jb9KUmH1bQ== X-Received: by 2002:a05:6a20:258f:b0:1db:d998:513e with SMTP id adf61e73a8af0-1dc90b55109mr19012107637.22.1731925890544; Mon, 18 Nov 2024 02:31:30 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-7f8c1afa67bsm5703029a12.0.2024.11.18.02.31.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:30 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 7/8] zsmalloc: move ZS_HANDLE_SIZE to zmslloc header Date: Mon, 18 Nov 2024 19:30:38 +0900 Message-ID: <20241118103054.3191944-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It will be used in object's split size calculations. Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 2 ++ mm/zsmalloc.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..c17803da7f18 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -16,6 +16,8 @@ =20 #include =20 +#define ZS_HANDLE_SIZE (sizeof(unsigned long)) + /* * zsmalloc mapping modes * diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 64b66a4d3e6e..466d5f49eb91 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -78,8 +78,6 @@ */ #define ZS_ALIGN 8 =20 -#define ZS_HANDLE_SIZE (sizeof(unsigned long)) - /* * Object location (, ) is encoded as * a single (unsigned long) handle value. --=20 2.47.0.338.g60cca15819-goog From nobody Mon Feb 9 11:02:17 2026 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AABB91991B2 for ; Mon, 18 Nov 2024 10:31:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925896; cv=none; b=h7uP5p6VC+IJubaaLQWwmDIjSYyrNF16Fzh5+IA0/UuTRaTIEKNIEcU/qMTYvWaWILYCKZ3XFdrNuiAvaotQlDjtxc+ZxphxNctckfrssArZGK0rLsjtDkHG3kSpaBhq63O/Ql/REByTsjvK+4olZQvH5extwH9cLX5i13GZ6kI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731925896; c=relaxed/simple; bh=gUip9aJ2/OVtgELjoiXJFkzcAXqEnDZMqWYmRnkqFns=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b/H9wQxsb5yNJvL5qdOlq1sfI2VoP3d4yabxDe/Vwx+ggPdnQT82ZS0m+2/c4ltl3O5lBkWfzEbWggzPBx+286YNoDdHiCu4Bk9UED3Yp/WSvD/UtJTYgLjqgD8WFtuli6UInO+zAtse6dEyrMCr4FHRi5jw19z1Qq+f1eM9Nzw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=U3zpkexd; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="U3zpkexd" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-20cdbe608b3so38739045ad.1 for ; Mon, 18 Nov 2024 02:31:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1731925894; x=1732530694; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T3cWzzVgSnhEjQAJcgAKgDRaPSzxbgH2wfRRSRJOX8o=; b=U3zpkexdty4aoadogXm5/YdimFq6dPVk+M83oSB4cNk4bSsJTKqTq7FT3hG8nfawQa vihc8BUK9zVY1WQhJ9j8tZ8LQO1z6qh2KMRq7DcRDZlJHUHmnpj9RPYION8aq0nOp1EG yDVslqS+IpgXKn+Vy7UwbmnG/C3hS3jvf4EPA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731925894; x=1732530694; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T3cWzzVgSnhEjQAJcgAKgDRaPSzxbgH2wfRRSRJOX8o=; b=pdYIHbYZEy593zyZRDhs+QVYizR7kDkmhi7vRe7bdu+h5qe+tQ9AlgM/0ofRLIQ2aL TnKEkvKv0CViQeI8ZXFYiJydBx/yCGSXNQod5J0j4xoKhaaamV+xUP5g30TLAdFwl1AH BnQG9MKADmR9rkRAWOak0S79Gx49STQ0nI7LSOWRlMMEPd3eKJUPfJUV1Ghj4e5BU4yV Pc6IP9TTPx6TER8BPsJfzcyhMn62PZnixffLKYNUg3MnFyunvwc1JvenNXXVC4YkOykn v7e0orLRErzqJXdBGSUQ9jMqYDXXYqPipMDrXK/8PhCB1FqaYfm4574FE5lo9bUJhYQ9 MHiw== X-Gm-Message-State: AOJu0YxhSFENP/FofZq8S0Gs5O7Hq+4Cj6dKAqT0hxEfsD2/vDuqrJmV /i3DVMZqTn+HHRuioEWBAqllMGIHUL2+bt/L0p7F+Qyu7Vaudd5bNp0blBFelB+y1gSzpLGGKOg = X-Google-Smtp-Source: AGHT+IEB47AXatuKfeg8+OsULsp5pHzqsd3xxAxQri9iX5MnEv/ga1c6gsUQSTrgSoAShO5PjtJhTA== X-Received: by 2002:a17:902:f68c:b0:20c:ecca:432b with SMTP id d9443c01a7336-211d0ebc52amr173858025ad.35.1731925893927; Mon, 18 Nov 2024 02:31:33 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:8826:78b8:a8fe:1066]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-211d0f36e15sm54029875ad.116.2024.11.18.02.31.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 02:31:33 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 8/8] zram: introduce multi-handle entries Date: Mon, 18 Nov 2024 19:30:39 +0900 Message-ID: <20241118103054.3191944-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog In-Reply-To: <20241118103054.3191944-1-senozhatsky@chromium.org> References: <20241118103054.3191944-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zsmalloc size-classes store more than one compressed object per physical page, therefore internal fragmentation is expected and quite common. Internal fragmentation is completely normal, once the system gets low on memory zsmalloc attempts to defragment its pool and release empty zspage-s. However, even this does not guarantee 100% usage-ratio of pool memory due to the nature of allocators. ZRAM_HUGE objects, on the other hand, do not share physical pages with another objects, because each such object is stored raw (uncompressed) and occupies a whole physical page. We, in fact, can get advantage of zsmalloc's internal fragmentation. Instead of allocating a physical page for each huge object it is possible to split such objects into smaller objects and store them in regular size-classes, possibly using allocated but unused zspages' space. Given that huge objects are stored raw, both write and read of such objects require only memcpy() and don't need any extra temporary storage / buffers. Split ZRAM_HUGE objects into two 2048 objects are store those parts in regular size-classes. This now allocates and tracks two zsmalloc handles for such entries. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 122 ++++++++++++++++++++++++++-------- drivers/block/zram/zram_drv.h | 15 ++++- 2 files changed, 108 insertions(+), 29 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 745207597e23..997b9bbf74e9 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -37,6 +37,16 @@ =20 #include "zram_drv.h" =20 +/* + * This determines sizes of the ZRAM_HUGE object split. Currently we perf= orm + * a 2-way split. One part is soted in 2048 size-class and the other one = is + * stored in the size-class above 2048. + * + * To store an object in a target size-class we need to subtrac zsmalloc + * handle size, which is added to each store objects. + */ +#define ZRAM_MULTI_HANDLE_HEAD_SZ ((PAGE_SIZE) / 2 - ZS_HANDLE_SIZE) + static DEFINE_IDR(zram_index_idr); /* idr index must be protected */ static DEFINE_MUTEX(zram_index_mutex); @@ -93,6 +103,18 @@ static void zram_set_handle(struct zram *zram, u32 inde= x, unsigned long handle) zram->table[index].handle =3D handle; } =20 +static struct zram_multi_handle *zram_get_multi_handle(struct zram *zram, + u32 index) +{ + return zram->table[index].mhandle; +} + +static void zram_set_multi_handle(struct zram *zram, u32 index, + struct zram_multi_handle *mhandle) +{ + zram->table[index].mhandle =3D mhandle; +} + /* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) @@ -1479,8 +1501,6 @@ static bool zram_meta_alloc(struct zram *zram, u64 di= sksize) */ static void zram_free_page(struct zram *zram, size_t index) { - unsigned long handle; - #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME zram->table[index].ac_time =3D 0; #endif @@ -1490,11 +1510,6 @@ static void zram_free_page(struct zram *zram, size_t= index) zram_clear_flag(zram, index, ZRAM_PP_SLOT); zram_set_priority(zram, index, 0); =20 - if (zram_test_flag(zram, index, ZRAM_HUGE)) { - zram_clear_flag(zram, index, ZRAM_HUGE); - atomic64_dec(&zram->stats.huge_pages); - } - if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); free_block_bdev(zram, zram_get_handle(zram, index)); @@ -1511,11 +1526,26 @@ static void zram_free_page(struct zram *zram, size_= t index) goto out; } =20 - handle =3D zram_get_handle(zram, index); - if (!handle) - return; + if (zram_test_flag(zram, index, ZRAM_HUGE)) { + struct zram_multi_handle *handle; + + handle =3D zram_get_multi_handle(zram, index); + if (!handle) + return; =20 - zs_free(zram->mem_pool, handle); + zs_free(zram->mem_pool, handle->head); + zs_free(zram->mem_pool, handle->tail); + kfree(handle); + + zram_clear_flag(zram, index, ZRAM_HUGE); + atomic64_dec(&zram->stats.huge_pages); + } else { + unsigned long handle =3D zram_get_handle(zram, index); + + if (!handle) + return; + zs_free(zram->mem_pool, handle); + } =20 atomic64_sub(zram_get_obj_size(zram, index), &zram->stats.compr_data_size); @@ -1528,16 +1558,22 @@ static void zram_free_page(struct zram *zram, size_= t index) static int read_incompressible_page(struct zram *zram, struct page *page, u32 index) { - unsigned long handle; + struct zram_multi_handle *handle; void *src, *dst; =20 - handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + handle =3D zram_get_multi_handle(zram, index); dst =3D kmap_local_page(page); - copy_page(dst, src); - kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); =20 + src =3D zs_map_object(zram->mem_pool, handle->head, ZS_MM_RO); + memcpy(dst, src, ZRAM_MULTI_HANDLE_HEAD_SZ); + zs_unmap_object(zram->mem_pool, handle->head); + + src =3D zs_map_object(zram->mem_pool, handle->tail, ZS_MM_RO); + memcpy(dst + ZRAM_MULTI_HANDLE_HEAD_SZ, src, + PAGE_SIZE - ZRAM_MULTI_HANDLE_HEAD_SZ); + zs_unmap_object(zram->mem_pool, handle->tail); + + kunmap_local(dst); return 0; } =20 @@ -1648,34 +1684,56 @@ static int zram_bvec_read(struct zram *zram, struct= bio_vec *bvec, static int zram_write_incompressible_page(struct zram *zram, struct page *= page, u32 index) { - unsigned long handle; + struct zram_multi_handle *handle; void *src, *dst; + int ret; =20 /* * This function is called from preemptible context so we don't need * to do optimistic and fallback to pessimistic handle allocation, * like we do for compressible pages. */ - handle =3D zs_malloc(zram->mem_pool, PAGE_SIZE, - GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); + handle =3D kzalloc(sizeof(*handle), GFP_KERNEL); + if (!handle) + return -ENOMEM; + + handle->head =3D zs_malloc(zram->mem_pool, ZRAM_MULTI_HANDLE_HEAD_SZ, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(handle->head)) { + ret =3D PTR_ERR((void *)handle->head); + goto error; + } + + handle->tail =3D zs_malloc(zram->mem_pool, + PAGE_SIZE - ZRAM_MULTI_HANDLE_HEAD_SZ, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(handle->tail)) { + ret =3D PTR_ERR((void *)handle->tail); + goto error; + } =20 if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); - zs_free(zram->mem_pool, handle); - return -ENOMEM; + ret =3D -ENOMEM; + goto error; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + + dst =3D zs_map_object(zram->mem_pool, handle->head, ZS_MM_WO); + memcpy(dst, src, ZRAM_MULTI_HANDLE_HEAD_SZ); + zs_unmap_object(zram->mem_pool, handle->head); + + dst =3D zs_map_object(zram->mem_pool, handle->tail, ZS_MM_WO); + memcpy(dst, src + ZRAM_MULTI_HANDLE_HEAD_SZ, + PAGE_SIZE - ZRAM_MULTI_HANDLE_HEAD_SZ); + zs_unmap_object(zram->mem_pool, handle->tail); + kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); - zram_set_handle(zram, index, handle); + zram_set_multi_handle(zram, index, handle); zram_set_obj_size(zram, index, PAGE_SIZE); zram_slot_unlock(zram, index); =20 @@ -1685,6 +1743,14 @@ static int zram_write_incompressible_page(struct zra= m *zram, struct page *page, atomic64_inc(&zram->stats.pages_stored); =20 return 0; + +error: + if (!IS_ERR_VALUE(handle->head)) + zs_free(zram->mem_pool, handle->head); + if (!IS_ERR_VALUE(handle->tail)) + zs_free(zram->mem_pool, handle->tail); + kfree(handle); + return ret; } =20 static int zram_write_page(struct zram *zram, struct page *page, u32 index) diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..7bc7792c2fef 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -60,9 +60,22 @@ enum zram_pageflags { =20 /*-- Data structures */ =20 +/* + * Unlike regular zram table entries, ZRAM_HUGE entries are stored in zsma= lloc + * as smaller objects in multiple locations (size-classes). This keeps tr= acks + * of those locations. + */ +struct zram_multi_handle { + unsigned long head; + unsigned long tail; +}; + /* Allocated for each disk page */ struct zram_table_entry { - unsigned long handle; + union { + unsigned long handle; + struct zram_multi_handle *mhandle; + }; unsigned int flags; spinlock_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME --=20 2.47.0.338.g60cca15819-goog