From nobody Sat Nov 23 15:36:41 2024 Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3266C14F9E7 for ; Tue, 19 Nov 2024 07:21:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000869; cv=none; b=YnVAS5fYL97rVD45FTrhK6FIod1cQFUPXTe6tqnFNMsAMy6etcdGDxAB1N/a4tJJaGyAuBOLvgrg77TLD70V2S4k9KqmqU03Wl+6Y8Y3hEi9s8Qeghgg8j9jLUFrfZfZLb/uxf5PVYoQrUATFSZBxgOgtsKZGHRl3HRYKY7zhp4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000869; c=relaxed/simple; bh=55dqSCwZtJRWjRLShFkVUUIpivXMX5HWrsfMTgnd1pU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E6/EfFgzEyTeXDAsyOXkto4/wH7Y2i7VI4vy/lXdegeC53OCJdCY9+izYcsV62b8t1oshmZJf9SBTkOZptaRqDQ82LlAzhxOHAEXh/cWnJ+Gl6s91EmxRR4VqN6OMCdsnCNPSFxHV+XaDD7FWWoLXKtsx5W8plWHZbJmYQzrq8Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=DnUGRHx4; arc=none smtp.client-ip=209.85.167.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="DnUGRHx4" Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3e5f6e44727so1627682b6e.0 for ; Mon, 18 Nov 2024 23:21:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000867; x=1732605667; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6q/5L9SIu5rwldrEbUs2WhmscYirGWBWS/629Fd7FoQ=; b=DnUGRHx4qXqmXW0nxqo/3ZzM9M3ui9rE0RHozo8zaQn0Cego3pEKHy6TA1RZ21Iug0 dBP7vJDbuHokCmwt6jVq4t9vI4i0F6j5LXPDPJy+FCXdNJdiNiC03lKBlGNSkwXO865c +74VBxUbSAjincTxieHvz9LjXVlKOJHjFFCUQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000867; x=1732605667; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6q/5L9SIu5rwldrEbUs2WhmscYirGWBWS/629Fd7FoQ=; b=TdjbuaaplQEbvPps9A7vZDWpOOMaEKS8X8hQZ9PHK6aHHC8WKIBK+spToVjiU5rkrq PIJONqP3/i2Yy3X5gtJzQnjWLuZxOgppTUi8BoWvDWudY+0odK1WJtOt2PIQpjEvVMQ2 +Slwwh+MibBw7aj7h37ivwW6TDTORaivpRoBwCfkqDsd3I+kf7ckFCCKM/N4/oP5bWRP j+UN1Iqk9vZGUtdGKK/2yFeukkyxRf36kq6hFgSleI+YHhHfswZJLticsjgnkyk+Yd+p i8wFTN2iiM3wnqBUTC6gnGlPzjnXZuHZpvJrGP2lEYaOTCnMSgLDvpPn6vxo2rs+0Mwe pOvA== X-Gm-Message-State: AOJu0YzDNAqO8OXV0dTAQM3K/aWcMerr9D2DTvrFl+d1WVJUwYaDqk11 4eHX/4NWyFxd1y+lv29nsjqrgMXSWw4UGSmcz3xo0lNCk8sJZWp3YutglGnCZg== X-Google-Smtp-Source: AGHT+IGNx1cjKIAjK5ApVq0fmA/BaUlKaY6OH5RzqsFORvd5XDiMWUtUCYCTkQ6mYjoeJeseNXe0uA== X-Received: by 2002:a05:6808:14ca:b0:3e6:402:d228 with SMTP id 5614622812f47-3e7bc7c5116mr14045933b6e.13.1732000867333; Mon, 18 Nov 2024 23:21:07 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-7f8c1dd3f4csm6930821a12.75.2024.11.18.23.21.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:06 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 1/8] zram: cond_resched() in writeback loop Date: Tue, 19 Nov 2024 16:20:39 +0900 Message-ID: <20241119072057.3440039-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Writeback loop can run for quite a while (depending on wb device performance, compression algorithm and the number of entries we writeback), so we need to do cond_resched() there, similarly to what we do in recompress loop. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 3dee026988dc..882a32d46a75 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -883,6 +883,8 @@ static ssize_t writeback_store(struct device *dev, next: zram_slot_unlock(zram, index); release_pp_slot(zram, pps); + + cond_resched(); } =20 if (blk_idx) --=20 2.47.0.371.ga323438b13-goog From nobody Sat Nov 23 15:36:41 2024 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22102198E63 for ; Tue, 19 Nov 2024 07:21:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000873; cv=none; b=CdJUPBxRujXVj5EhxcbSBigC9eMWtRBLWcpK6Z86uTguT4fnW0NVc2P7BEaShuiAkqzWPHZXcA6bCw1TVXtDQqPHaDa/rE7JZIrXw/Ugxyz8TBTpykl1RfUtUu0UsUrFtGNOzcO5ypegUVzLUA5htYoFqUEKl3nGaePU8fETgQo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000873; c=relaxed/simple; bh=SczjJdKEBNX/mqKU8yTEr7MCNMXpYCdEbjooumU28p0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uFDh42gq1A3KjqzbuyfBSeRK14IhyZoQmreA+zqE3/KslTbWjEPsft46SnAJ1kAL/vCrGccYCuuSG2ZmAvFmjO9HZDzKZnBlI+rrT1HyyXY6lOvknld8xSLDfyptlctgpyvTvzwKtbki19WRTHFiZCY1Dy/19Mb/gXkiC6M0ZJw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=H0p99Gn5; arc=none smtp.client-ip=209.85.215.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="H0p99Gn5" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-7ee7e87f6e4so2392990a12.2 for ; Mon, 18 Nov 2024 23:21:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000871; x=1732605671; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dMQqiNmYmiFMft9yq6eBRmUzx8BPO2n9E5AN93b4Lcs=; b=H0p99Gn58PDrRsH5spu8M+ZJa6sslM0hBG8nh4JsgxL7FQYg1zJFdTHqYMn8m7z+gm xlcNyobxqEqGNxTtl0Blv9RvmrQo8ob+mirlHrmuEpJHOFmF2fwYBoBnF6QjCerCy/XU J4QalH70H6Q3h0ilzgBpWFakmlRsYTSRRsd2I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000871; x=1732605671; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dMQqiNmYmiFMft9yq6eBRmUzx8BPO2n9E5AN93b4Lcs=; b=JuGynkQlKaYOXg2I5W6ap1NYVeH6fKaajsichPermU+YWcyTV2SZEtBUZ/wWP1EAG6 OoJTwBRNuICRbCfwKA+xF5HtjzC+V4ZA9qh+4SI7UCZc67CC5c+7hNu1bBAQeePHNDLW S5gkJqAJn3+FOmly5Nu9nkS74rIpxLV8lmTtJxpOOAahNxfU1RQVd/0w+xGvvrznyWIN ki10+cLBTtNP1RHO+qsB+M6Ut1cw8ajjRK98p1zwEJWZiofFVZTyrWJpGNjJTxlf2GBW CDkj4AkyR5+uTdlHFSqHoOCwPWElbBbFFPo4z/ZfUbqyM21BMPi7Wbe88qbu8gR/p38c Bo1w== X-Gm-Message-State: AOJu0Yy+OnL8LvLSK06AxbahTKig6XbO8YTTNfzkGo1gYKN3+OCKTiqY X0auHM4wqXFt/WCurcCMylEVmn8ci6fvHhLPZJ+GxLKwGeuHmD0Ged5HhUf7RQ== X-Google-Smtp-Source: AGHT+IEqmtXnoRwOq40F0SEQzW7Ex0ubamKlsAciS92ROOQnLymwuA47V96kcMUoGX9IlyuX8DpbCA== X-Received: by 2002:a05:6a20:9152:b0:1db:ea00:8e91 with SMTP id adf61e73a8af0-1dc90afc661mr19853250637.2.1732000871488; Mon, 18 Nov 2024 23:21:11 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-724770ee8basm7460258b3a.7.2024.11.18.23.21.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:10 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 2/8] zram: free slot memory early during write Date: Tue, 19 Nov 2024 16:20:40 +0900 Message-ID: <20241119072057.3440039-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the current implementation entry's previously allocated memory is released in the very last moment, when we already have allocated a new memory for new data. This, basically, temporarily increases memory usage for no good reason. For example, consider the case when both old (stale) and new entry data are incompressible so such entry will temporarily use two physical pages - one for stale (old) data and one for new data. We can release old memory as soon as we get a write request for entry. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 882a32d46a75..987d72f2249c 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1640,6 +1640,11 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) unsigned long element =3D 0; enum zram_pageflags flags =3D 0; =20 + /* First, free memory allocated to this slot (if any) */ + zram_slot_lock(zram, index); + zram_free_page(zram, index); + zram_slot_unlock(zram, index); + mem =3D kmap_local_page(page); if (page_same_filled(mem, &element)) { kunmap_local(mem); @@ -1728,13 +1733,7 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); out: - /* - * Free memory associated with this sector - * before overwriting unused sectors. - */ zram_slot_lock(zram, index); - zram_free_page(zram, index); - if (comp_len =3D=3D PAGE_SIZE) { zram_set_flag(zram, index, ZRAM_HUGE); atomic64_inc(&zram->stats.huge_pages); --=20 2.47.0.371.ga323438b13-goog From nobody Sat Nov 23 15:36:41 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E602199FA1 for ; Tue, 19 Nov 2024 07:21:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000877; cv=none; b=PT/zwwZx2LE9WJR0ku5nhYcwft0RG5O3yilEiTfk6RfNt0b1wf5uAddEoO/UODGUr6s7vH/2YC+nnTAJZeVP4LY4rOYh6s5X6lVQsm/xYhzG6Zw3egAe3MBt2vV7TD8h4yVZPNKe52ZXURLkQP2jzQfBFyvmI01UFkjZMH4lxBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000877; c=relaxed/simple; bh=E9npFG0X9F8KtazdPCWKZz4Vb4U/GOdKlZTFjHDE72o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b9Z3drabVjXaEmB6lzaOewNLt2432hgIYNURpTIghCsk0Y4d1lVbedUN+WMvMvVwsap1188KTm1suuS/lCVpKtNqns8rxdl3uI8Naq0upQT3N7pY+ixBlKcpb+B7Zt2ZnGeVvFiFjgvUbzqm7Eyxr3eYXLzN0+jrFDw/u58OxIM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=fsBu3vpB; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="fsBu3vpB" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-20c70abba48so24131105ad.0 for ; Mon, 18 Nov 2024 23:21:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000875; x=1732605675; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Eyx+9in/HQFchdWYfQyVaWbKUcXPbHYEL1OWiH5aBP4=; b=fsBu3vpBqEL6lrivQVpzk151qgBelDx1KpVnJ+o9O/XUQX4EAIQ0Th5/YmNdHdr2p7 sBBBzaZlR1jWAbF0y3SrVPnvi/BSuTrfTJaW2Gs3Q+YJkRisBvbNWKB2casDbHyvHD08 geYYdawfgmdh+N+GUhuVcRHYPFgOfQn8xz/n0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000875; x=1732605675; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Eyx+9in/HQFchdWYfQyVaWbKUcXPbHYEL1OWiH5aBP4=; b=MN6InCcAydPCiGEW9V1SHPsMDBe8i0FoALFUgH9zYqRkq4NU3yxoBIl6fs5q88lUIX SDMVmO/PQziHHKuElyTzdlWAMDDTWYvFDtXF41AOqD8t3wTuFDzrFAen4CKurGfpw503 TJDHZIbKzQe/V69YSZEHPivDH2BEMruPnToWLlsGowMylpoSWHiB/webYkPEdQeMFBNU IHkKE0NVFzMLsIw++X3UUSLTBf1U04o4y8O2Ab/niGMx4W6IYgYrM994Our6/+aV5c7g ql6+MrO5OLIUKWFWXdnsQzZ7ameZvfbZTtPruWt1a34FOVFT7tQCtjGPaJpFvdcczEBK o9kw== X-Gm-Message-State: AOJu0YzxHv8SD0zcDHhNS7eX0q+PA4IiR2mBf8coqKJTXyay0p2sbc+4 J4Pi9bvp+N2A1UbQYq2RWsolJg4i2LKo9QHuMThioz2URYPeoxya53E9PLOc1A== X-Google-Smtp-Source: AGHT+IE8NzADorKVHeq5Bw9SIPLPudEHsKk/E9RPk636fHjChjYwBWoLJcm0pCwpZnFlUeKeomsUjw== X-Received: by 2002:a17:902:e54f:b0:212:37e:3fcd with SMTP id d9443c01a7336-212037e488fmr124581995ad.56.1732000874945; Mon, 18 Nov 2024 23:21:14 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-211d0f346d7sm67654955ad.138.2024.11.18.23.21.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:14 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 3/8] zram: remove entry element member Date: Tue, 19 Nov 2024 16:20:41 +0900 Message-ID: <20241119072057.3440039-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Element is in the same anon union as handle and hence holds the same value, which makes code below sort of confusing handle =3D zram_get_handle() if (!handle) element =3D zram_get_element() Element doesn't really simplify the code, let's just remove it. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 23 +++++------------------ drivers/block/zram/zram_drv.h | 5 +---- 2 files changed, 6 insertions(+), 22 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 987d72f2249c..e80b4d15b74b 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -112,17 +112,6 @@ static void zram_clear_flag(struct zram *zram, u32 ind= ex, zram->table[index].flags &=3D ~BIT(flag); } =20 -static inline void zram_set_element(struct zram *zram, u32 index, - unsigned long element) -{ - zram->table[index].element =3D element; -} - -static unsigned long zram_get_element(struct zram *zram, u32 index) -{ - return zram->table[index].element; -} - static size_t zram_get_obj_size(struct zram *zram, u32 index) { return zram->table[index].flags & (BIT(ZRAM_FLAG_SHIFT) - 1); @@ -873,7 +862,7 @@ static ssize_t writeback_store(struct device *dev, =20 zram_free_page(zram, index); zram_set_flag(zram, index, ZRAM_WB); - zram_set_element(zram, index, blk_idx); + zram_set_handle(zram, index, blk_idx); blk_idx =3D 0; atomic64_inc(&zram->stats.pages_stored); spin_lock(&zram->wb_limit_lock); @@ -1496,7 +1485,7 @@ static void zram_free_page(struct zram *zram, size_t = index) =20 if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); - free_block_bdev(zram, zram_get_element(zram, index)); + free_block_bdev(zram, zram_get_handle(zram, index)); goto out; } =20 @@ -1540,12 +1529,10 @@ static int zram_read_from_zspool(struct zram *zram,= struct page *page, =20 handle =3D zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { - unsigned long value; void *mem; =20 - value =3D handle ? zram_get_element(zram, index) : 0; mem =3D kmap_local_page(page); - zram_fill_page(mem, PAGE_SIZE, value); + zram_fill_page(mem, PAGE_SIZE, handle); kunmap_local(mem); return 0; } @@ -1591,7 +1578,7 @@ static int zram_read_page(struct zram *zram, struct p= age *page, u32 index, */ zram_slot_unlock(zram, index); =20 - ret =3D read_from_bdev(zram, page, zram_get_element(zram, index), + ret =3D read_from_bdev(zram, page, zram_get_handle(zram, index), parent); } =20 @@ -1742,7 +1729,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) =20 if (flags) { zram_set_flag(zram, index, flags); - zram_set_element(zram, index, element); + zram_set_handle(zram, index, element); } else { zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 134be414e210..db78d7c01b9a 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -62,10 +62,7 @@ enum zram_pageflags { =20 /* Allocated for each disk page */ struct zram_table_entry { - union { - unsigned long handle; - unsigned long element; - }; + unsigned long handle; unsigned int flags; spinlock_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME --=20 2.47.0.371.ga323438b13-goog From nobody Sat Nov 23 15:36:41 2024 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D446919067C for ; Tue, 19 Nov 2024 07:21:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000880; cv=none; b=tdQ0eIDHg+OZr8MebAr4UIiqjnXbYjqIhqC17XrOgCx2YFBYmLcyzbWmU/nW6GmiyRDw122L9PgFBIGv8ey3zM9pGHCfOGCowYgB4AMRk+oMgGn5jkWeGacD63ToyEelE7VP8y4ckr6VGWaSIvJ9X68soaOQKevsT6iprXohqF4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000880; c=relaxed/simple; bh=xS7ygz96qldG8ayzvqJio0pA3KvNFraN+8nnrp/FX8c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PNL4j0M5uw3xAp3Wkf34753n0ajMuo+tLV6FjxvHTufy+fi2dchRfSMHULq6tS/hGA6sI8qyAXmE7RAH3pfbMnAq5wYsEnV7JZ4Gw4bXUwcPwK8AgEhXS9uEUQ/kt2DJNiqlJI2fwZyf0dEWxxyocrkiOqYK+rm68jilrPbSXRs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=OX87otAt; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="OX87otAt" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-20cb7139d9dso5419615ad.1 for ; Mon, 18 Nov 2024 23:21:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000878; x=1732605678; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=enpKZhxs9rfiZ+ov5ec779bccoj8XP4Dk6HgEcENBDU=; b=OX87otAtkge1SSnZEZVW2zfQwZOlfZfnwz4cylTjyek/n5Oi/oHNYoCfm2OhgnVzob JkpQzH8d/2tAbi52tIRgHmoNx0pz7x2xw0N7Ov/cW7nbZfjUfoIO9yWfe04piNR6ppI1 7wW+4wSEz6cCdgyzunoufk3XDm3zkggqUT70E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000878; x=1732605678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=enpKZhxs9rfiZ+ov5ec779bccoj8XP4Dk6HgEcENBDU=; b=IsIgZb4b0GzycQ0jNggMZfyAS7Br/27G8UiX8UfMrxwZERh1vzV4lOm8kMSr/nsaLj lxCcMZNpciqi8B3aXBhhMjSEux2FR/gZAndf4vA4w74Zs53jWWL8/oLvJLzOmp4F6nRJ J9Q/A2KeXRQNInHw+y/EtGSpvv1V7+c58XC/aTFpvEqFIa7K7C3yX6pfQXXuWUCH/Qnw R6aJgk4z/BdE9l1HgnLKZLNZPLd8C3077R1MNXz8n9++aZov00z2HnwEOtVVgipBAew+ RD6e5oo9j+8SM/iVcjSMfaBhHKzI/MsBOTZkve6OvwnxCxIzck+eeUzck4ZdhjzTJqaR G5rg== X-Gm-Message-State: AOJu0YxnKpfUc6QAZ739VxfZORa4NUS3HgBdWJ65DmRcq+0afWZLXYp+ hKPdkV3GKX/QJx60RPI779CA1f+fRzMIIZzo131RUyD0ES73ovSn+yXt6fFa5XEnn6ntXnXy9Xo = X-Google-Smtp-Source: AGHT+IGKFu9r2P797y7Vi3Fn5YlQyisoXsV4b47g8ABIi7ZipWC2HqYQ6wonNKoQgYcGLeb5sTAqzg== X-Received: by 2002:a17:902:e852:b0:20c:98f8:e0fa with SMTP id d9443c01a7336-211d0d62aeemr201884045ad.11.1732000878245; Mon, 18 Nov 2024 23:21:18 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-211d0dc30cbsm67977765ad.25.2024.11.18.23.21.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:17 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 4/8] zram: factor out ZRAM_SAME write Date: Tue, 19 Nov 2024 16:20:42 +0900 Message-ID: <20241119072057.3440039-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Handling of ZRAM_SAME now uses a goto to the final stages of zram_write_page() plus it introduces a branch and flags variable, which is not making the code any simpler. In reality, we can handle ZRAM_SAME immediately when we detect such pages and remove a goto and a branch. Factor out ZRAM_SAME handling into a separate routine to simplify zram_write_page(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 37 ++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e80b4d15b74b..f89af45237c9 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1616,6 +1616,20 @@ static int zram_bvec_read(struct zram *zram, struct = bio_vec *bvec, return zram_read_page(zram, bvec->bv_page, index, bio); } =20 +static int zram_write_same_filled_page(struct zram *zram, unsigned long fi= ll, + u32 index) +{ + zram_slot_lock(zram, index); + zram_set_flag(zram, index, ZRAM_SAME); + zram_set_handle(zram, index, fill); + zram_slot_unlock(zram, index); + + atomic64_inc(&zram->stats.same_pages); + atomic64_inc(&zram->stats.pages_stored); + + return 0; +} + static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; @@ -1625,7 +1639,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) void *src, *dst, *mem; struct zcomp_strm *zstrm; unsigned long element =3D 0; - enum zram_pageflags flags =3D 0; + bool same_filled; =20 /* First, free memory allocated to this slot (if any) */ zram_slot_lock(zram, index); @@ -1633,14 +1647,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) zram_slot_unlock(zram, index); =20 mem =3D kmap_local_page(page); - if (page_same_filled(mem, &element)) { - kunmap_local(mem); - /* Free memory associated with this sector now. */ - flags =3D ZRAM_SAME; - atomic64_inc(&zram->stats.same_pages); - goto out; - } + same_filled =3D page_same_filled(mem, &element); kunmap_local(mem); + if (same_filled) + return zram_write_same_filled_page(zram, element, index); =20 compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); @@ -1719,7 +1729,7 @@ static int zram_write_page(struct zram *zram, struct = page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); -out: + zram_slot_lock(zram, index); if (comp_len =3D=3D PAGE_SIZE) { zram_set_flag(zram, index, ZRAM_HUGE); @@ -1727,13 +1737,8 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) atomic64_inc(&zram->stats.huge_pages_since); } =20 - if (flags) { - zram_set_flag(zram, index, flags); - zram_set_handle(zram, index, element); - } else { - zram_set_handle(zram, index, handle); - zram_set_obj_size(zram, index, comp_len); - } + zram_set_handle(zram, index, handle); + zram_set_obj_size(zram, index, comp_len); zram_slot_unlock(zram, index); =20 /* Update stats */ --=20 2.47.0.371.ga323438b13-goog From nobody Sat Nov 23 15:36:41 2024 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB90919AD8C for ; Tue, 19 Nov 2024 07:21:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000883; cv=none; b=AeqMLCsSuFmw+AwHKmz5B72LdEUa31dpW0gHU6xWBX+Vfqe7xhf6GrIs0rV0j9hgpXRaRoYHgJTNB0vH88KiEg8mDCtXSGBVT/xcamcRSUkmoByfu2PuWQ4iOT1UfvqyXTf/p0Sqk+CsCN8X/Vik4ncl69CNr7b0deV0wICEhgc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000883; c=relaxed/simple; bh=3MMMG4XKz7n2fVohXU7Jz1go5CS4olMTQboAbPLW3a8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nMzA+Sopczdfoe93TBOCaJqxljXO3MfwIODtZISkwfQePhMpusWcpa7st1gC/rK2/BjOiOj2M8wQsYhc1f6v7PaFBW83zMYjBpod3bAg+EY0xi3Z1Gmsp8zfReaTlKEscFcSGXssSsxWcXFxxhWW0/3ln5RvxupPDA7Tr1vztWU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=TBUH2XkM; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="TBUH2XkM" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2e91403950dso400730a91.3 for ; Mon, 18 Nov 2024 23:21:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000881; x=1732605681; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3r1BPbKg6+Hwje1rl36AJwmpN6gJmRKCRsZyk3dr8tE=; b=TBUH2XkMF1z//EaHEZS1RoL/sj88JsrR6n1+h7BBh2D8zls7ms2SOIIsUuDMAyHXwe phCqT82mbc77HD2YFJOmgh4M63Frfec4dFtVP5/enmYSToxHwWRRkgvidbllYnRayJOB 5mRkev5tbmgyaA4dBgib5LUygdG7BhJZtJ1RM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000881; x=1732605681; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3r1BPbKg6+Hwje1rl36AJwmpN6gJmRKCRsZyk3dr8tE=; b=ERp23XZGxvlZUZER3I6SHRu/I6BWiTehOheeurwOeG1tzJOLylAv+/OosPy4+SpzYQ zUE1PM/bkxGXKHG0MT4eTbj6MfPGI64vbKpNpL1v6ma3OYQcoygKhI/wyjB81pt2gTrh S4BP894mEWIRA4OWcPpeuprYNRJhUBkFVLdQAsGZWkunsdxXZQuKxOW7RqD+FM4RoAzn +jWMujxuPt456ULW2SgEKdld+R72zviNVssDsUZ3nd5t6OuASYvQIYzpRNybk0W/zxC3 YmmYwpWzAr8jN+3cJlJ6y9gf0t4V2glEBqqyQESjj5WASBPmn7bjC2HsMZ2skK16K864 fafw== X-Gm-Message-State: AOJu0Yw1QQa43DvA2Ii2mYYGXtH3v9b5J/DCk3N8Ba/qHKWu/gJH2iHi /hkXAIaYZx8h2gqxLAZXAjo3l7uwQJGNuVG7VYUlKYbe2eEo4qmSeysaVVE+KTj+YG2QI/QwwK4 = X-Google-Smtp-Source: AGHT+IGCzDUzZ9ZGQosjzl82UtKy4NubctNo+8zsr4rXK6+hylFusmKPLODB+qAGXXKd5SOAq7cN4g== X-Received: by 2002:a17:90b:3c02:b0:2ea:5c01:c1a4 with SMTP id 98e67ed59e1d1-2ea5c01c3b2mr10362501a91.20.1732000880991; Mon, 18 Nov 2024 23:21:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2ea024949a4sm8671620a91.11.2024.11.18.23.21.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:20 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 5/8] zram: factor out ZRAM_HUGE write Date: Tue, 19 Nov 2024 16:20:43 +0900 Message-ID: <20241119072057.3440039-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zram_write_page() handles 3 different cases: ZRAM_SAME pages (which was already simplified in previous patches) writes, regular pages writes and ZRAM_HUGE pages writes. ZRAM_HUGE handling adds a significant amount of complexity: all those conditional src for copy_page(), etc., a conditional jump backward for the fallback handle allocation and so on. Instead, we can handle ZRAM_HUGE in a separate function and remove quite a bit of that complexity at a cost of minor code duplication (basically, only zram stats updates). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 140 +++++++++++++++++++++------------- 1 file changed, 85 insertions(+), 55 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f89af45237c9..9101fd0c670f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -132,6 +132,30 @@ static inline bool zram_allocated(struct zram *zram, u= 32 index) zram_test_flag(zram, index, ZRAM_WB); } =20 +static inline void update_used_max(struct zram *zram, const unsigned long = pages) +{ + unsigned long cur_max =3D atomic_long_read(&zram->stats.max_used_pages); + + do { + if (cur_max >=3D pages) + return; + } while (!atomic_long_try_cmpxchg(&zram->stats.max_used_pages, + &cur_max, pages)); +} + +static bool zram_can_store_page(struct zram *zram) +{ + unsigned long alloced_pages; + + alloced_pages =3D zs_get_total_pages(zram->mem_pool); + update_used_max(zram, alloced_pages); + + if (!zram->limit_pages) + return true; + + return alloced_pages <=3D zram->limit_pages; +} + #if PAGE_SIZE !=3D 4096 static inline bool is_partial_io(struct bio_vec *bvec) { @@ -266,18 +290,6 @@ static struct zram_pp_slot *select_pp_slot(struct zram= _pp_ctl *ctl) } #endif =20 -static inline void update_used_max(struct zram *zram, - const unsigned long pages) -{ - unsigned long cur_max =3D atomic_long_read(&zram->stats.max_used_pages); - - do { - if (cur_max >=3D pages) - return; - } while (!atomic_long_try_cmpxchg(&zram->stats.max_used_pages, - &cur_max, pages)); -} - static inline void zram_fill_page(void *ptr, unsigned long len, unsigned long value) { @@ -1630,13 +1642,54 @@ static int zram_write_same_filled_page(struct zram = *zram, unsigned long fill, return 0; } =20 +static int zram_write_incompressible_page(struct zram *zram, struct page *= page, + u32 index) +{ + unsigned long handle; + void *src, *dst; + + /* + * This function is called from preemptible context so we don't need + * to do optimistic and fallback to pessimistic handle allocation, + * like we do for compressible pages. + */ + handle =3D zs_malloc(zram->mem_pool, PAGE_SIZE, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(handle)) + return PTR_ERR((void *)handle); + + if (!zram_can_store_page(zram)) { + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zs_free(zram->mem_pool, handle); + return -ENOMEM; + } + + dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); + src =3D kmap_local_page(page); + memcpy(dst, src, PAGE_SIZE); + kunmap_local(src); + zs_unmap_object(zram->mem_pool, handle); + + zram_slot_lock(zram, index); + zram_set_flag(zram, index, ZRAM_HUGE); + zram_set_handle(zram, index, handle); + zram_set_obj_size(zram, index, PAGE_SIZE); + zram_slot_unlock(zram, index); + + atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); + atomic64_inc(&zram->stats.huge_pages); + atomic64_inc(&zram->stats.huge_pages_since); + atomic64_inc(&zram->stats.pages_stored); + + return 0; +} + static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret =3D 0; - unsigned long alloced_pages; unsigned long handle =3D -ENOMEM; unsigned int comp_len =3D 0; - void *src, *dst, *mem; + void *dst, *mem; struct zcomp_strm *zstrm; unsigned long element =3D 0; bool same_filled; @@ -1654,10 +1707,10 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) =20 compress_again: zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); - src =3D kmap_local_page(page); + mem =3D kmap_local_page(page); ret =3D zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, - src, &comp_len); - kunmap_local(src); + mem, &comp_len); + kunmap_local(mem); =20 if (unlikely(ret)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); @@ -1666,8 +1719,11 @@ static int zram_write_page(struct zram *zram, struct= page *page, u32 index) return ret; } =20 - if (comp_len >=3D huge_class_size) - comp_len =3D PAGE_SIZE; + if (comp_len >=3D huge_class_size) { + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + return zram_write_incompressible_page(zram, page, index); + } + /* * handle allocation has 2 paths: * a) fast path is executed with preemption disabled (for @@ -1683,66 +1739,40 @@ static int zram_write_page(struct zram *zram, struc= t page *page, u32 index) */ if (IS_ERR_VALUE(handle)) handle =3D zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + __GFP_KSWAPD_RECLAIM | + __GFP_NOWARN | + __GFP_HIGHMEM | + __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); atomic64_inc(&zram->stats.writestall); handle =3D zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) return PTR_ERR((void *)handle); =20 - if (comp_len !=3D PAGE_SIZE) - goto compress_again; - /* - * If the page is not compressible, you need to acquire the - * lock and execute the code below. The zcomp_stream_get() - * call is needed to disable the cpu hotplug and grab the - * zstrm buffer back. It is necessary that the dereferencing - * of the zstrm variable below occurs correctly. - */ - zstrm =3D zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); + goto compress_again; } =20 - alloced_pages =3D zs_get_total_pages(zram->mem_pool); - update_used_max(zram, alloced_pages); - - if (zram->limit_pages && alloced_pages > zram->limit_pages) { + if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_free(zram->mem_pool, handle); return -ENOMEM; } =20 dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - src =3D zstrm->buffer; - if (comp_len =3D=3D PAGE_SIZE) - src =3D kmap_local_page(page); - memcpy(dst, src, comp_len); - if (comp_len =3D=3D PAGE_SIZE) - kunmap_local(src); - + memcpy(dst, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); - atomic64_add(comp_len, &zram->stats.compr_data_size); =20 zram_slot_lock(zram, index); - if (comp_len =3D=3D PAGE_SIZE) { - zram_set_flag(zram, index, ZRAM_HUGE); - atomic64_inc(&zram->stats.huge_pages); - atomic64_inc(&zram->stats.huge_pages_since); - } - zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); zram_slot_unlock(zram, index); =20 - /* Update stats */ atomic64_inc(&zram->stats.pages_stored); + atomic64_add(comp_len, &zram->stats.compr_data_size); + return ret; } =20 --=20 2.47.0.371.ga323438b13-goog From nobody Sat Nov 23 15:36:41 2024 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF73B19CD17 for ; Tue, 19 Nov 2024 07:21:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000886; cv=none; b=Cp2GbGzZwISF546dK7WVA7eYStHUY8VUtyRrJBNqNMWuxMwzqbfNYv95MS7run+6inFc6ry8+SBMqQdoWMNSRnHl0nZSEURWzoXTN2Kj+GLYBN8yg8k3RudtOOhTM5m6R/NHIbGP+wjnwXdl2xzeCbfwAfmsX7g6Bywuc68wqug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000886; c=relaxed/simple; bh=QZoBI0/NmZPZhWwcjQ/ZUK792oAETFLYk4pewXXZ+jU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ebZgYJJngxI5vnERSLh/W+aniMN9RmSL8i434iHyhKWuxiUM3wKzJYXiSNEsR2s90s4ToJkORvHmoLLhOJbofTvlZuGIJOomSP/B9K+jEiRSA12hDocc1Gw7AQnrjnh9rN9EKuP75WrsILulYUXmsULIEXoFiwdgp9BS0xzvQKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=cH1kBLfI; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="cH1kBLfI" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7ee7e87f6e4so2393078a12.2 for ; Mon, 18 Nov 2024 23:21:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000884; x=1732605684; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Hh0J+htCkr2Gj1s5kihWh8sqReBGwis+Zxl5bO+P3vk=; b=cH1kBLfI7S627hesLmBxpkhCZbuuLv54K6xY+1/lnxNgT5D+QcXR3sYSk54UBzBrD9 dfgYwG0AI+Sm0187mjSWSZ0Fn7X80VhMXVFxuTHBT52nBnYZAWYKB0sat4l+6iU2IL+n nZU8m5YCqPUH8/IabCl6CdBPaRdSoaO2QSU3s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000884; x=1732605684; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Hh0J+htCkr2Gj1s5kihWh8sqReBGwis+Zxl5bO+P3vk=; b=rZKUd9uhZqDh/2zo763pqx53FAHZldzLAq00oJefjbn5oLnvKX20K31BqrqBFVjHwS 3sAvTuyB4hSpSDlxBVSjBns+hZ27jM9S1C9+DHLUEQHG32stnGsNuv+FiRoSxVFzfFNP /gaYMK+DQ0g0QRlO+1GZ+xk/voTf8EmWTF7SNB+F7+GbSrq7VHhjAF/XLwix3tye55PE PeSXMBE2X1qomXwFLUqbG7qGDJYqXknqf5rH4yXgeVcweHnBIB/Spcy0+/3EscWE8+jS FuvAfDMbjslHqlyYFgQ0UsIQN2iRRLuShnGhSsh2aF2M4/hcOGstVc5pfltCa67Ha0+F SaaA== X-Gm-Message-State: AOJu0Yz2MuEU5VnfbDa+Utq2oi+21hC2Dl+ag3PtTCRJMgNyM/MNa9j0 /q5/4b45p8e1Xmv1rN+Ctjlw70iq63mbaiTj6wO4Lu1LV6fwe/C53rwb2GIGdw== X-Google-Smtp-Source: AGHT+IG1JymDLGg8WB/V8srdCrB9afMMKLVW52YenEFEkLMOBaJ6xJyBhfnZhwyK7PvnmGZWUg8a/g== X-Received: by 2002:a05:6a20:748f:b0:1cf:6c86:231c with SMTP id adf61e73a8af0-1dc90b5c503mr20907966637.26.1732000884119; Mon, 18 Nov 2024 23:21:24 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-724770eef34sm7475406b3a.8.2024.11.18.23.21.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:23 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 6/8] zram: factor out ZRAM_HUGE read Date: Tue, 19 Nov 2024 16:20:44 +0900 Message-ID: <20241119072057.3440039-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Similarly to write, move ZRAM_HUGE read handlig to a separate function. This will make more sense with introduction of multi-handle entries later in the series. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 71 ++++++++++++++++++++++------------- 1 file changed, 44 insertions(+), 27 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9101fd0c670f..2b20afcfbb94 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1525,6 +1525,46 @@ static void zram_free_page(struct zram *zram, size_t= index) zram_set_obj_size(zram, index, 0); } =20 +static int read_incompressible_page(struct zram *zram, struct page *page, + u32 index) +{ + unsigned long handle; + void *src, *dst; + + handle =3D zram_get_handle(zram, index); + src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + dst =3D kmap_local_page(page); + copy_page(dst, src); + kunmap_local(dst); + zs_unmap_object(zram->mem_pool, handle); + + return 0; +} + +static int read_compressible_page(struct zram *zram, struct page *page, + u32 index) +{ + struct zcomp_strm *zstrm; + unsigned long handle; + unsigned int size; + void *src, *dst; + int ret, prio; + + handle =3D zram_get_handle(zram, index); + size =3D zram_get_obj_size(zram, index); + prio =3D zram_get_priority(zram, index); + + zstrm =3D zcomp_stream_get(zram->comps[prio]); + src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + dst =3D kmap_local_page(page); + ret =3D zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); + kunmap_local(dst); + zs_unmap_object(zram->mem_pool, handle); + zcomp_stream_put(zram->comps[prio]); + + return ret; +} + /* * Reads (decompresses if needed) a page from zspool (zsmalloc). * Corresponding ZRAM slot should be locked. @@ -1532,12 +1572,7 @@ static void zram_free_page(struct zram *zram, size_t= index) static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index) { - struct zcomp_strm *zstrm; unsigned long handle; - unsigned int size; - void *src, *dst; - u32 prio; - int ret; =20 handle =3D zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { @@ -1549,28 +1584,10 @@ static int zram_read_from_zspool(struct zram *zram,= struct page *page, return 0; } =20 - size =3D zram_get_obj_size(zram, index); - - if (size !=3D PAGE_SIZE) { - prio =3D zram_get_priority(zram, index); - zstrm =3D zcomp_stream_get(zram->comps[prio]); - } - - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); - if (size =3D=3D PAGE_SIZE) { - dst =3D kmap_local_page(page); - copy_page(dst, src); - kunmap_local(dst); - ret =3D 0; - } else { - dst =3D kmap_local_page(page); - ret =3D zcomp_decompress(zram->comps[prio], zstrm, - src, size, dst); - kunmap_local(dst); - zcomp_stream_put(zram->comps[prio]); - } - zs_unmap_object(zram->mem_pool, handle); - return ret; + if (!zram_test_flag(zram, index, ZRAM_HUGE)) + return read_compressible_page(zram, page, index); + else + return read_incompressible_page(zram, page, index); } =20 static int zram_read_page(struct zram *zram, struct page *page, u32 index, --=20 2.47.0.371.ga323438b13-goog From nobody Sat Nov 23 15:36:41 2024 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C6611A705C for ; Tue, 19 Nov 2024 07:21:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000888; cv=none; b=A/RXouLWyTd6IlJhZtMaYEPcG0PlqbDdAZsGM1ar8zSnFvOvH7rABpQCn+aZwI9V9pB0yFY0sJtKWfPcsKNTbWt5+7gWKfD6038ri3rzsHYVmgqoxr6yAVJjYnKdpDixZ4MIHNDfrya/rj2Nq1b8Bd+G8rr2hC/bDMKshFTS00E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000888; c=relaxed/simple; bh=mrGv2qtF8ssBe0q/fGE8zoRY695b00OVbv4wzk1hbVY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M0hY4mZaXu7WKmiMlq3et/8RxqdnlmHrHI6Uve8rigC6qW2izxytANf60RvaYnd9OCnFjgwm+FrNhraol1WtwzzimDnG4Mo3+WWqBDm4T1ndpvJEpCU/OA+nueihOhKMfD/NsQqqyyr4/dhTVV+ysAKpMZ5azH6L5XYVXXnsWGw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=HfPqts38; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="HfPqts38" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-7f8b37edeb7so3838127a12.0 for ; Mon, 18 Nov 2024 23:21:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000887; x=1732605687; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ztq1HCHe5F7VoTpmdbLU74q1rBToVI1HubuTv8LWLRg=; b=HfPqts38K9E5NqA8CX8SSM2oO3rxQ8g9AYIUm14Nk7Hydxy+8oxFfGyl4/lv+aufEI 5Byde4tHTytfc8HCvPrk0Xlao9Vyyx0F9Zg+XHUT/3rxho1j0KxNaodSeefBcWJv10OG FLzp/KoKhrKzuvoopQgdNnaXwoN0hbaojg3iA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000887; x=1732605687; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ztq1HCHe5F7VoTpmdbLU74q1rBToVI1HubuTv8LWLRg=; b=pLni+qzUaRpVOSZr90YvQIKHkRy1EbcrfvvASQLDWx8CIevHlo8JY+ZvG7OeEaNKWU Z7VtV/B3R5uGnTwTQDy5MQwacMd1/pPYtLGVhBg7k6IeT1RIW99Ipoe4adBwWQeEoBj9 WMrEHQzdtzLt9MumfQp9Q/2TAXaHF0KfAYKTcxAGTTYs+zFhevjeagVUuoXqlletkInl i4JS3GuAtuHK5F09q2J3KW8wdSaVvixivxeXXydi95sJ8CeMAOjoUq0Uev+oO5FFtIYc dXHPDNWLnBr9DX5qhr/D/tc/w6Pl87bwdYDLIE55DMiIkGjYmp/T2qc67F6A/KOZRbwI P2Ig== X-Gm-Message-State: AOJu0Yz+Pqf/MAQ3A7OUFD/Q4wEFu5rwIXFFzA7968gAZZh79l3Vssga BGUhXA1+Sh4nO9UUoqE/wORChwRLwwT/MC/J9z1OoL9IJ3RjLq7UfAAuC/4AGA== X-Google-Smtp-Source: AGHT+IEgP0EdrJqNTLy/8qzn8Xs6Cj/FSp6/avrRE0kupevO3+it/G9+ostYbtoLlvUYVSzCeYQBDQ== X-Received: by 2002:a05:6a21:32a6:b0:1d9:78c:dcf2 with SMTP id adf61e73a8af0-1dc90c1b16bmr23377806637.43.1732000886863; Mon, 18 Nov 2024 23:21:26 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2ea777193b5sm3038784a91.45.2024.11.18.23.21.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:26 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 7/8] zsmalloc: move ZS_HANDLE_SIZE to zsmalloc header Date: Tue, 19 Nov 2024 16:20:45 +0900 Message-ID: <20241119072057.3440039-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It will be used in object's split size calculations. Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 2 ++ mm/zsmalloc.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..c17803da7f18 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -16,6 +16,8 @@ =20 #include =20 +#define ZS_HANDLE_SIZE (sizeof(unsigned long)) + /* * zsmalloc mapping modes * diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 64b66a4d3e6e..466d5f49eb91 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -78,8 +78,6 @@ */ #define ZS_ALIGN 8 =20 -#define ZS_HANDLE_SIZE (sizeof(unsigned long)) - /* * Object location (, ) is encoded as * a single (unsigned long) handle value. --=20 2.47.0.371.ga323438b13-goog From nobody Sat Nov 23 15:36:41 2024 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6AF01B85D3 for ; Tue, 19 Nov 2024 07:21:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000893; cv=none; b=mUDp+/Gpmb2gMGalyHvkayLUT5Jl1PbJP6a4vM1jxqYjiAj0iEPtHiTa1NvQEtW6ZMOTbAe5QU4KV0fWtVbrkZqOvqZZMOrhf6+SGJpLtHYLq1ANRkf8bYXS7h6XlYsPpfk2T+W7rZxCN58b5m7HuBCqasv9kVW0gyRcaYPeDQs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732000893; c=relaxed/simple; bh=4+bjv2d9eUx5fmy192m4VvtxaJyxp5nAXbZwhB5c9og=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ETCjkt6kPCkSyxiGSYvmuWu7Xp8iEIR7k2Pzd5DRyqQkIUxULnYGw8BiuH9kSdITTlwZ7cTZDXcyEWD4ggKTPn2ec/J7iy0Cokm+98K9gFAzUb7iJN0Q4u0y40d6MUiwzjkZ3TMbEyxysvCaqovkOupT9Ctx5snB1iLXNcQ3Uks= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=f9VeORgj; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="f9VeORgj" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-20cf6eea3c0so5683055ad.0 for ; Mon, 18 Nov 2024 23:21:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1732000891; x=1732605691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YPgvnxddRnh+JN7AYAi4qB46AptriyxUI0Yn0oo8khA=; b=f9VeORgj+mNiV09v7JLZ2CvvpAVePcLIg2Ue6mAt0tw3b73+zH3AJRXqx0tzFt0Cwq QwBIIzFxIYtnw4/+UJI76KfYsdcp3ckLkevie6RKhbG+I9ilXoPP2h+5pv8YZq/AE4x/ GId8PwoJy7X04G9xt9mC54RlHMAyhkBlB/kqY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732000891; x=1732605691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YPgvnxddRnh+JN7AYAi4qB46AptriyxUI0Yn0oo8khA=; b=r3OHLAr1Q4NGDhIO17kYZGrDhrlN1/sWHPAetBeXa8OSvu8bSP8BKTi0bSKMpEHfU9 nY4p0sM8doxXOKooLq0NRUyRsFcjMIP+3iKzY2mLTb50KrxNvDcZprQp6FOITU3NE0/6 Llpl6k01Fz8AYUuR9Ng6WnfrKTBmrf3cYBjNQM8eERChjnQFG5G8AWk7GIdifdMjrM7v 8F6bMH+2kx7FVJ+/Vd1+yEG7L3Z1cyS3pha89fM1M/YznGEqA6F8gATezLTFoDId23GA VaeJ88sBW/Y+MeTrCtKtRlGSC8S8Ond0isdwjRAQL/CAUskA27Bj3IqXW9JU+p4PZSa9 oPXw== X-Gm-Message-State: AOJu0YwD5qViSXdpQndl99kBEsbGGCQfABpMAtOSS0swe39DlOc5FiCp 1Q/ZiKffYugpsxmreJSh+/sGL3qtlkwk/WU+2LXHeLIU2k3Dj+HRO9BYLETMyQ== X-Google-Smtp-Source: AGHT+IG8enfxTdRo52pIptt7VuVfSVvJ5XWKTVi2HXnhOJ/BW+vLMlDZHkQDlv/qY3ZoJmRcAhpZeA== X-Received: by 2002:a17:903:2452:b0:20b:8776:4902 with SMTP id d9443c01a7336-211d0ebf325mr231639695ad.38.1732000890802; Mon, 18 Nov 2024 23:21:30 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:5534:f54c:1465:9438]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-211d0f34f3bsm68221295ad.137.2024.11.18.23.21.29 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Nov 2024 23:21:30 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv1 8/8] zram: introduce multi-handle entries Date: Tue, 19 Nov 2024 16:20:46 +0900 Message-ID: <20241119072057.3440039-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.47.0.371.ga323438b13-goog In-Reply-To: <20241119072057.3440039-1-senozhatsky@chromium.org> References: <20241119072057.3440039-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" zsmalloc size-classes store more than one compressed object per physical page, therefore internal fragmentation is expected and quite common. Internal fragmentation is completely normal, once the system gets low on memory zsmalloc attempts to defragment its pool and release empty zspage-s. However, even this does not guarantee 100% usage-ratio of pool memory due to the nature of allocators. ZRAM_HUGE objects, on the other hand, do not share physical pages with another objects, because each such object is stored raw (uncompressed) and occupies a whole physical page. We, in fact, can get advantage of zsmalloc's internal fragmentation. Instead of allocating a physical page for each huge object it is possible to split such objects into smaller objects and store them in regular size-classes, possibly using allocated but unused zspages' space. Given that huge objects are stored raw, both write and read of such objects require only memcpy() and don't need any extra temporary storage / buffers. Split ZRAM_HUGE objects into two 2048 objects are store those parts in regular size-classes. This now allocates and tracks two zsmalloc handles for such entries. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 120 ++++++++++++++++++++++++++-------- drivers/block/zram/zram_drv.h | 15 ++++- 2 files changed, 106 insertions(+), 29 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 2b20afcfbb94..7e29e204fccf 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -37,6 +37,17 @@ =20 #include "zram_drv.h" =20 +/* + * This determines sizes of the ZRAM_HUGE object split. Currently we perf= orm + * a 2-way split. One part is stored in 2048 size-class and the other one= is + * stored in the size-class above 2048. + * + * To store an object in a target size-class we need to sub zsmalloc + * handle size (which is added to every object by zsmalloc internally). + */ +#define ZRAM_MHANDLE_HEAD_LEN ((PAGE_SIZE) / 2 - ZS_HANDLE_SIZE) +#define ZRAM_MHANDLE_TAIL_LEN ((PAGE_SIZE) - ZRAM_MHANDLE_HEAD_LEN) + static DEFINE_IDR(zram_index_idr); /* idr index must be protected */ static DEFINE_MUTEX(zram_index_mutex); @@ -93,6 +104,18 @@ static void zram_set_handle(struct zram *zram, u32 inde= x, unsigned long handle) zram->table[index].handle =3D handle; } =20 +static struct zram_multi_handle *zram_get_multi_handle(struct zram *zram, + u32 index) +{ + return zram->table[index].mhandle; +} + +static void zram_set_multi_handle(struct zram *zram, u32 index, + struct zram_multi_handle *mhandle) +{ + zram->table[index].mhandle =3D mhandle; +} + /* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) @@ -1479,8 +1502,6 @@ static bool zram_meta_alloc(struct zram *zram, u64 di= sksize) */ static void zram_free_page(struct zram *zram, size_t index) { - unsigned long handle; - #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME zram->table[index].ac_time =3D 0; #endif @@ -1490,11 +1511,6 @@ static void zram_free_page(struct zram *zram, size_t= index) zram_clear_flag(zram, index, ZRAM_PP_SLOT); zram_set_priority(zram, index, 0); =20 - if (zram_test_flag(zram, index, ZRAM_HUGE)) { - zram_clear_flag(zram, index, ZRAM_HUGE); - atomic64_dec(&zram->stats.huge_pages); - } - if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); free_block_bdev(zram, zram_get_handle(zram, index)); @@ -1511,11 +1527,26 @@ static void zram_free_page(struct zram *zram, size_= t index) goto out; } =20 - handle =3D zram_get_handle(zram, index); - if (!handle) - return; + if (zram_test_flag(zram, index, ZRAM_HUGE)) { + struct zram_multi_handle *handle; + + handle =3D zram_get_multi_handle(zram, index); + if (!handle) + return; =20 - zs_free(zram->mem_pool, handle); + zs_free(zram->mem_pool, handle->head); + zs_free(zram->mem_pool, handle->tail); + kfree(handle); + + zram_clear_flag(zram, index, ZRAM_HUGE); + atomic64_dec(&zram->stats.huge_pages); + } else { + unsigned long handle =3D zram_get_handle(zram, index); + + if (!handle) + return; + zs_free(zram->mem_pool, handle); + } =20 atomic64_sub(zram_get_obj_size(zram, index), &zram->stats.compr_data_size); @@ -1528,16 +1559,21 @@ static void zram_free_page(struct zram *zram, size_= t index) static int read_incompressible_page(struct zram *zram, struct page *page, u32 index) { - unsigned long handle; + struct zram_multi_handle *handle; void *src, *dst; =20 - handle =3D zram_get_handle(zram, index); - src =3D zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + handle =3D zram_get_multi_handle(zram, index); dst =3D kmap_local_page(page); - copy_page(dst, src); - kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); =20 + src =3D zs_map_object(zram->mem_pool, handle->head, ZS_MM_RO); + memcpy(dst, src, ZRAM_MHANDLE_HEAD_LEN); + zs_unmap_object(zram->mem_pool, handle->head); + + src =3D zs_map_object(zram->mem_pool, handle->tail, ZS_MM_RO); + memcpy(dst + ZRAM_MHANDLE_HEAD_LEN, src, ZRAM_MHANDLE_TAIL_LEN); + zs_unmap_object(zram->mem_pool, handle->tail); + + kunmap_local(dst); return 0; } =20 @@ -1662,34 +1698,54 @@ static int zram_write_same_filled_page(struct zram = *zram, unsigned long fill, static int zram_write_incompressible_page(struct zram *zram, struct page *= page, u32 index) { - unsigned long handle; + struct zram_multi_handle *handle; void *src, *dst; + int ret; =20 /* * This function is called from preemptible context so we don't need * to do optimistic and fallback to pessimistic handle allocation, * like we do for compressible pages. */ - handle =3D zs_malloc(zram->mem_pool, PAGE_SIZE, - GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); + handle =3D kzalloc(sizeof(*handle), GFP_KERNEL); + if (!handle) + return -ENOMEM; + + handle->head =3D zs_malloc(zram->mem_pool, ZRAM_MHANDLE_HEAD_LEN, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(handle->head)) { + ret =3D PTR_ERR((void *)handle->head); + goto error; + } + + handle->tail =3D zs_malloc(zram->mem_pool, ZRAM_MHANDLE_TAIL_LEN, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(handle->tail)) { + ret =3D PTR_ERR((void *)handle->tail); + goto error; + } =20 if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); - zs_free(zram->mem_pool, handle); - return -ENOMEM; + ret =3D -ENOMEM; + goto error; } =20 - dst =3D zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src =3D kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + + dst =3D zs_map_object(zram->mem_pool, handle->head, ZS_MM_WO); + memcpy(dst, src, ZRAM_MHANDLE_HEAD_LEN); + zs_unmap_object(zram->mem_pool, handle->head); + + dst =3D zs_map_object(zram->mem_pool, handle->tail, ZS_MM_WO); + memcpy(dst, src + ZRAM_MHANDLE_HEAD_LEN, ZRAM_MHANDLE_TAIL_LEN); + zs_unmap_object(zram->mem_pool, handle->tail); + kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); =20 zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); - zram_set_handle(zram, index, handle); + zram_set_multi_handle(zram, index, handle); zram_set_obj_size(zram, index, PAGE_SIZE); zram_slot_unlock(zram, index); =20 @@ -1699,6 +1755,14 @@ static int zram_write_incompressible_page(struct zra= m *zram, struct page *page, atomic64_inc(&zram->stats.pages_stored); =20 return 0; + +error: + if (!IS_ERR_VALUE(handle->head)) + zs_free(zram->mem_pool, handle->head); + if (!IS_ERR_VALUE(handle->tail)) + zs_free(zram->mem_pool, handle->tail); + kfree(handle); + return ret; } =20 static int zram_write_page(struct zram *zram, struct page *page, u32 index) diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..7bc7792c2fef 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -60,9 +60,22 @@ enum zram_pageflags { =20 /*-- Data structures */ =20 +/* + * Unlike regular zram table entries, ZRAM_HUGE entries are stored in zsma= lloc + * as smaller objects in multiple locations (size-classes). This keeps tr= acks + * of those locations. + */ +struct zram_multi_handle { + unsigned long head; + unsigned long tail; +}; + /* Allocated for each disk page */ struct zram_table_entry { - unsigned long handle; + union { + unsigned long handle; + struct zram_multi_handle *mhandle; + }; unsigned int flags; spinlock_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME --=20 2.47.0.371.ga323438b13-goog