From nobody Fri Nov 29 17:50:02 2024 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8888815AF6 for ; Tue, 17 Sep 2024 02:10:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539031; cv=none; b=JVqwGXKp5rA2jlfLEbVSqAniH+loJ5viB6gLffsEu/MyjaPaskfA8SIZmCgnXvp7qdgIO7xCSaZUi2luaT77nZy9WGuuMvSBXe9R62DPwgqwdIf9AXgTAhmjYYSusdPL7anLPkaakQchYjx9vkNFkdXbc+14DPMrnhHOQDx0a4E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539031; c=relaxed/simple; bh=zs1xjkHS1qDR4NEj1PsKCOx0BbN2dHqRScEEUTjaEgs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NKk9pyQDYd3ejDjLAxJ6GnqpyLgmICJeWMCnxwXK+oJoNR+o0eHAFa1fQUyufGmd1QDtTFkcG5TvG5yO1jGODiONKNTRdIMgmbCqIMa5El0RB4MPhz6Ut77YgI9Q/nV1heDQhdOUnzDichzerTzmHkTytNbWcI04mwrZ/sO5wLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=hVol8mWf; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="hVol8mWf" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-20536dcc6e9so28607935ad.2 for ; Mon, 16 Sep 2024 19:10:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726539029; x=1727143829; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w8H7PhtZWFmbqhqP3UZN7g8qBsoFfPmUW3Wpglk6Y3U=; b=hVol8mWfSLvJ6aYtx8f7njHTqYXqRTXfVUkS/6tFuftIOPj41InA2FwrRFF7FwiYZy oF14+kGiO1jWFGAElW79sO5IZPzvHYandd5LerDDWNdMyPU5oj1mcZuXwX6e/8rxbhpf MCnT9qfzn/SnUsNOuYM978Hh3MTVaJ+FGRsJw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726539029; x=1727143829; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w8H7PhtZWFmbqhqP3UZN7g8qBsoFfPmUW3Wpglk6Y3U=; b=LFyv7sJu497CSu4jL/LDNa2oqIHm8MGUIK0yXZrc3L1iFO2F77I+/AiOdgjeV8Eouo oqPHYWZW+tMplEZJAKJHurGcwp2lL0Nl5XJCBZh9J1fgM1INhjP3FnzkbLzpNYOlJjON oYhgzqUvpffCkm5dCUCAYpiAPmkrWIwf9/S3xB0KaDdop0Ofp8OSg2MGh98Rxx/LOwco gC8ZaOoRB2XKgAJqlUxXpfEXigazQZqPdCOJOBK5U8WBx/aZjqvFRdhGbGAdIcTBHt00 5OKFJxVCKrDx0fK5XLHOP7vRzJQG1h+3u/seKkYEzTNZ/eDFa8UfLzP7Jm8kDi5q9kdF AnQw== X-Gm-Message-State: AOJu0Yy/navvqqnmkT6llvXr7bKP/nfoNqZyOUkQaYu8NFJ+8O3OichO I1ZliUJJ7/N3pXl+dyMHTuQsE/lPWf8X6XDE5b13c/iaOwhhhBLkrSignCAtpQ== X-Google-Smtp-Source: AGHT+IGW9izy2scT00BIEr9qBSc7odAOqerh23E67N1ckI9Ho82ST9EryEzRyvH3doY/o6Bdr7/Q8Q== X-Received: by 2002:a17:902:fc4e:b0:1fc:369b:c1dd with SMTP id d9443c01a7336-2078262ccccmr205190435ad.6.1726539028896; Mon, 16 Sep 2024 19:10:28 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:693c:b4a9:5e6e:c397]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2dbcfd9313fsm6037293a91.44.2024.09.16.19.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2024 19:10:28 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv5 1/7] zram: introduce ZRAM_PP_SLOT flag Date: Tue, 17 Sep 2024 11:09:06 +0900 Message-ID: <20240917021020.883356-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog In-Reply-To: <20240917021020.883356-1-senozhatsky@chromium.org> References: <20240917021020.883356-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This flag indicates that the slot was selected as a candidate slot for post-processing (pp) and was assigned to a pp bucket. It does not necessarily mean that the slot is currently under post-processing, but may mean so. The slot can loose its PP_SLOT flag, while still being in the pp-bucket, if it's accessed or slot_free-ed. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 2 ++ drivers/block/zram/zram_drv.h | 1 + 2 files changed, 3 insertions(+) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ee2a279c5f25..d170bf6cdcd8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -178,6 +178,7 @@ static inline u32 zram_get_priority(struct zram *zram, = u32 index) static void zram_accessed(struct zram *zram, u32 index) { zram_clear_flag(zram, index, ZRAM_IDLE); + zram_clear_flag(zram, index, ZRAM_PP_SLOT); #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME zram->table[index].ac_time =3D ktime_get_boottime(); #endif @@ -1354,6 +1355,7 @@ static void zram_free_page(struct zram *zram, size_t = index) zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE); =20 zram_set_priority(zram, index, 0); + zram_clear_flag(zram, index, ZRAM_PP_SLOT); =20 if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index cfc8c059db63..914cb6629969 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -48,6 +48,7 @@ enum zram_pageflags { ZRAM_SAME =3D ZRAM_FLAG_SHIFT, /* Page consists the same element */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_UNDER_WB, /* page is under writeback */ + ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ ZRAM_IDLE, /* not accessed page since last idle marking */ ZRAM_INCOMPRESSIBLE, /* none of the algorithms could compress it */ --=20 2.46.0.662.g92d0881bb0-goog From nobody Fri Nov 29 17:50:02 2024 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CA4B3B1A2 for ; Tue, 17 Sep 2024 02:10:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539033; cv=none; b=S2xM7y4BFPJ+scDmvj7xnjfhbC+b5EfW7UCG4VWHyIMZhMch3YwHt5FwosiL4jcJKYc402t8B4QQ0DdgBBVRg/6z0RC9ghdNEX7SPU1XDigCu3ohUGE8zQnLfaYfiKsOLoEYbRfvUWuKBQMaMdwDcIqhEYnBaSnlnhePw+HsPHg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539033; c=relaxed/simple; bh=pCNOlP9y5PFyA818TtLchVQPaUxrv0jSnsTmr9g9Z2s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oo/bwMScwFNERm7Y22rNPIkpI/nPzc3JE83fh1QU6Y74qdhbeFG1CRghzvM/Q3icSgzAqowLEGve/WN/LkVlLLTdi+ysk4PGqefrxMJc0JgzqELC9kpqVCO2CTK0tadT93Q/nbhLXMA/Inu06T4gusRKbxKHV7ruPW41wE+IZPQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Vqo9Zcd8; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Vqo9Zcd8" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2d88690837eso4151345a91.2 for ; Mon, 16 Sep 2024 19:10:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726539031; x=1727143831; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZhTBoAWfGdKQ585H98MwgnvKBszcs4TGX5DhJFeIfIQ=; b=Vqo9Zcd8SPAz6Ac7VUkwdYV09YFlc7GoxKzIvHcF+FnazAdNutixrHxZqHwGU6v862 bzsZQRcJKoDKpFUNRAfesgSJAZHXWGKKPiB9EVzFSIHV5+PpUfaUEYfLinb9d+JNjqWP AWqdxQ35DhFcuPHOON4ntU8cNgdwBRrkeEN2o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726539031; x=1727143831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZhTBoAWfGdKQ585H98MwgnvKBszcs4TGX5DhJFeIfIQ=; b=YDDKkqTPnJuvPz7Xwmje8HD6tEiOF5Ok+BJex5fOIRXpTpWg1AoEPy18/BRSZAj0UE g/3WTOi1yG7/e4HuQoX3y84k8oPZoYDG3qmNTyQ8UXw7YEybyTUamiWJ0k3PRZnjQOLe pjKuPXNnfu1oL+RBy8Uduj30BfP14SOTWQoBYA+AQSO5GPXVxKjJ03f7ry8Oi1k3kfLZ Ui82JzWCyM00NPTH3tQPUcLv4F+lfHX8Lc7efBANP9CiwVaHyucWHe5YIVQutbHxRNaG fLPq6D50KlXkm7FApEeWx5i1c6iiq+wIMyoQT6rToJpMiVvNRAah/KQWD2Bee9e85V2O vZhg== X-Gm-Message-State: AOJu0YyzlKdI+KieLSDwdelc6ZR+H2VZn+gxdrPGcAIGOKhf9Ay/wee0 Q2ZqygdNoyUEGapH1hO68XALpCl7QlfQxOngS7Y2gJ9xyS5yoWXqOJNPGNJvmQ== X-Google-Smtp-Source: AGHT+IGcqIskqlLQtZnytZWQ2RYuLUW45cV4WAfu6U9blGvncLxKnZKCgoGaid5OwQ0zaXO7zTpU/A== X-Received: by 2002:a17:90a:c915:b0:2d3:d063:bdb6 with SMTP id 98e67ed59e1d1-2db9ffa1624mr20734530a91.4.1726539031411; Mon, 16 Sep 2024 19:10:31 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:693c:b4a9:5e6e:c397]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2dbcfd9313fsm6037293a91.44.2024.09.16.19.10.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2024 19:10:31 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv5 2/7] zram: permit only one post-processing operation at a time Date: Tue, 17 Sep 2024 11:09:07 +0900 Message-ID: <20240917021020.883356-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog In-Reply-To: <20240917021020.883356-1-senozhatsky@chromium.org> References: <20240917021020.883356-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Both recompress and writeback soon will unlock slots during processing, which makes things too complex wrt possible race-conditions. We still want to clear PP_SLOT in slot_free, because this is how we figure out that slot that was selected for post-processing has been released under us and when we start post-processing we check if slot still has PP_SLOT set. At the same time, theoretically, we can have something like this: CPU0 CPU1 recompress scan slots set PP_SLOT unlock slot slot_free clear PP_SLOT allocate PP_SLOT writeback scan slots set PP_SLOT unlock slot select PP-slot test PP_SLOT So recompress will not detect that slot has been re-used and re-selected for concurrent writeback post-processing. Make sure that we only permit on post-processing operation at a time. So now recompress and writeback post-processing don't race against each other, we only need to handle slot re-use (slot_free and write), which is handled individually by each pp operation. Having recompress and writeback competing for the same slots is not exactly good anyway (can't imagine anyone doing that). Signed-off-by: Sergey Senozhatsky --- Documentation/admin-guide/blockdev/zram.rst | 2 ++ drivers/block/zram/zram_drv.c | 16 ++++++++++++++++ drivers/block/zram/zram_drv.h | 1 + 3 files changed, 19 insertions(+) diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/ad= min-guide/blockdev/zram.rst index 678d70d6e1c3..714a5171bfc0 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -47,6 +47,8 @@ The list of possible return codes: -ENOMEM zram was not able to allocate enough memory to fulfil your needs. -EINVAL invalid input has been provided. +-EAGAIN re-try operation later (e.g. when attempting to run recompress + and writeback simultaneously). =3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 If you use 'echo', the returned value is set by the 'echo' utility, diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index d170bf6cdcd8..37622268104e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -627,6 +627,12 @@ static ssize_t writeback_store(struct device *dev, goto release_init_lock; } =20 + /* Do not permit concurrent post-processing actions. */ + if (atomic_xchg(&zram->pp_in_progress, 1)) { + up_read(&zram->init_lock); + return -EAGAIN; + } + if (!zram->backing_dev) { ret =3D -ENODEV; goto release_init_lock; @@ -753,6 +759,7 @@ static ssize_t writeback_store(struct device *dev, free_block_bdev(zram, blk_idx); __free_page(page); release_init_lock: + atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); =20 return ret; @@ -1883,6 +1890,12 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 + /* Do not permit concurrent post-processing actions. */ + if (atomic_xchg(&zram->pp_in_progress, 1)) { + up_read(&zram->init_lock); + return -EAGAIN; + } + if (algo) { bool found =3D false; =20 @@ -1950,6 +1963,7 @@ static ssize_t recompress_store(struct device *dev, __free_page(page); =20 release_init_lock: + atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); return ret; } @@ -2146,6 +2160,7 @@ static void zram_reset_device(struct zram *zram) zram->disksize =3D 0; zram_destroy_comps(zram); memset(&zram->stats, 0, sizeof(zram->stats)); + atomic_set(&zram->pp_in_progress, 0); reset_bdev(zram); =20 comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); @@ -2383,6 +2398,7 @@ static int zram_add(void) zram->disk->fops =3D &zram_devops; zram->disk->private_data =3D zram; snprintf(zram->disk->disk_name, 16, "zram%d", device_id); + atomic_set(&zram->pp_in_progress, 0); =20 /* Actual capacity set using sysfs (/sys/block/zram/disksize */ set_capacity(zram->disk, 0); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 914cb6629969..73a9d47d76ba 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -140,5 +140,6 @@ struct zram { #ifdef CONFIG_ZRAM_MEMORY_TRACKING struct dentry *debugfs_dir; #endif + atomic_t pp_in_progress; }; #endif --=20 2.46.0.662.g92d0881bb0-goog From nobody Fri Nov 29 17:50:02 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6C5C487A5 for ; Tue, 17 Sep 2024 02:10:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539037; cv=none; b=eybPZi1UivkeXSQZPejoEwfnRerScu4CvEWFcIGizFRA+pA3Y4+v+nvcPHevk+wUKeBhD8M5bF+9bo7jtO8Np9S2f9z0taAWH0Uxw4M8D485O9QOb4UO5OOYbHaorMUuFBvNDwgRXdrRw2vEqX7Eh118szApIkyxQfiLSEJyWYs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539037; c=relaxed/simple; bh=8XiQVRaJH4ln3N1mxHOpbsElpJ0EPxnbzCxXtzZwTbI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Uwd35W+mVRaVTJV7V121ZoGK0kkdag5HmpTc8ZvDl+ksZzIivDVWXy0wpYVhpfmJ1l4S9AWwRh3fw9nsrKCx2IOVva1wlXyVp7PgMgU1DHlysYjW03lhpIgWrdtsZFndbTAJ5mOwY1BqPSo7XxIBbiWzM3kTyIHAlMB4g05BnDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=BUcTPJEk; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="BUcTPJEk" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2068acc8b98so45631925ad.3 for ; Mon, 16 Sep 2024 19:10:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726539035; x=1727143835; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ffF6dMQqehUmDqvR186gvSHc1Me3zwo9POTkpGbeCZg=; b=BUcTPJEk2LPUMc5UQxiJ7HHE5kcv9XfWljSfNz7U3AuOfNkropQtH3VtLvnHIhXXfc tt0JXSv7WC5NE3SKCATrdfBI6KYHL0d0FAkxENg1l2/ciJU6SI2A5ITUNjAtJhr4rxtQ JgMVkbW/wUJNHdVo9ctIgMMO7BbUdVVMccyig= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726539035; x=1727143835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ffF6dMQqehUmDqvR186gvSHc1Me3zwo9POTkpGbeCZg=; b=HIa7usqqIx6YY/F9ZF0ad9V1qDvmzzDNNrhwFtnk9xpawXy1kjyAQDxms0dKkHhttq JhAwp6LD/UGgbPohx9bFF3bjuhcA0vWTXZf0zR/wHY4s0SADsNGBDIi6l9JntbpyKDh9 s152TTK2QCxSsMO1zSGMik4inTKZPG+89Jmsb0OO5MYkq1qALz3JQi0p6WbacKXbpW9R JWzv5gT2n6Q6RqYm5690ErI3s+2WaHorgK0gW+8utXJZ3zOcPPCHa9PYPp2uFDLND38t 0YE67nfTBv7725sHwD+pC7il+Kb2aVeM/KTHoZ/suL9QrC27pUidXvsaPuL0Uar9fPVv EBtQ== X-Gm-Message-State: AOJu0YyP0RQPXjoYNvulIENJqrJ0bj8oq5Yj6vALbCsoriddzzPDxG3M hZZNA1r0rO5CESg/8bHEaAYb8sF2Tew/YxW+f7KwcYv2GIRpU7U/ytp0QXiRLg== X-Google-Smtp-Source: AGHT+IHHnE35X5FvcsbIX3uy/70uqZ2pFf4H1VjobdJZCi2odQms7Jk4fhbq+rkiH0RTQe9iwJkv0Q== X-Received: by 2002:a17:902:f68e:b0:205:7829:9d83 with SMTP id d9443c01a7336-2076e412fafmr198287575ad.38.1726539035111; Mon, 16 Sep 2024 19:10:35 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:693c:b4a9:5e6e:c397]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2dbcfd9313fsm6037293a91.44.2024.09.16.19.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2024 19:10:34 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv5 3/7] zram: rework recompress target selection strategy Date: Tue, 17 Sep 2024 11:09:08 +0900 Message-ID: <20240917021020.883356-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog In-Reply-To: <20240917021020.883356-1-senozhatsky@chromium.org> References: <20240917021020.883356-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Target slot selection for recompression is just a simple iteration over zram->table entries (stored pages) from slot 0 to max slot. Given that zram->table slots are written in random order and are not sorted by size, a simple iteration over slots selects suboptimal targets for recompression. This is not a problem if we recompress every single zram->table slot, but we never do that in reality. In reality we limit the number of slots we can recompress (via max_pages parameter) and hence proper slot selection becomes very important. The strategy is quite simple, suppose we have two candidate slots for recompression, one of size 48 bytes and one of size 2800 bytes, and we can recompress only one, then it certainly makes more sense to pick 2800 entry for recompression. Because even if we manage to compress 48 bytes objects even further the savings are going to be very small. Potential savings after good re-compression of 2800 bytes objects are much higher. This patch reworks slot selection and introduces the strategy described above: among candidate slots always select the biggest ones first. For that the patch introduces zram_pp_ctl (post-processing) structure which holds NUM_PP_BUCKETS pp buckets of slots. Slots are assigned to a particular group based on their sizes - the larger the size of the slot the higher the group index. This, basically, sorts slots by size in liner time (we still perform just one iteration over zram->table slots). When we select slot for recompression we always first lookup in higher pp buckets (those that hold the largest slots). Which achieves the desired behavior. TEST =3D=3D=3D=3D A very simple demonstration: zram is configured with zstd, and zstd with dict as a recompression stream. A limited (max 4096 pages) recompression is performed then, with a log of sizes of slots that were recompressed. You can see that patched zram selects slots for recompression in significantly different manner, which leads to higher memory savings (see column #2 of mm_stat output). BASE ---- *** initial state of zram device /sys/block/zram0/mm_stat 1750994944 504491413 514203648 0 514203648 1 0 3420= 4 34204 *** recompress idle max_pages=3D4096 /sys/block/zram0/mm_stat 1750994944 504262229 514953216 0 514203648 1 0 3420= 4 34204 Sizes of selected objects for recompression: ... 45 58 24 226 91 40 24 24 24 424 2104 93 2078 2078 2078 959 154 ... PATCHED ------- *** initial state of zram device /sys/block/zram0/mm_stat 1750982656 504492801 514170880 0 514170880 1 0 3420= 4 34204 *** recompress idle max_pages=3D4096 /sys/block/zram0/mm_stat 1750982656 503716710 517586944 0 514170880 1 0 3420= 4 34204 Sizes of selected objects for recompression: ... 3680 3694 3667 3590 3614 3553 3537 3548 3550 3542 3543 3537 ... Note, pp-slots are not strictly sorted, there is a PP_BUCKET_SIZE_RANGE variation of sizes within particular bucket. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 187 +++++++++++++++++++++++++++++----- 1 file changed, 160 insertions(+), 27 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 37622268104e..6688f70b2140 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -184,6 +184,99 @@ static void zram_accessed(struct zram *zram, u32 index) #endif } =20 +#ifdef CONFIG_ZRAM_MULTI_COMP +struct zram_pp_slot { + unsigned long index; + struct list_head entry; +}; + +/* + * A post-processing bucket is, essentially, a size class, this defines + * the range (in bytes) of pp-slots sizes in particular bucket. + */ +#define PP_BUCKET_SIZE_RANGE 64 +#define NUM_PP_BUCKETS ((PAGE_SIZE / PP_BUCKET_SIZE_RANGE) + 1) + +struct zram_pp_ctl { + struct list_head pp_buckets[NUM_PP_BUCKETS]; +}; + +static struct zram_pp_ctl *init_pp_ctl(void) +{ + struct zram_pp_ctl *ctl; + u32 idx; + + ctl =3D kmalloc(sizeof(*ctl), GFP_KERNEL); + if (!ctl) + return NULL; + + for (idx =3D 0; idx < NUM_PP_BUCKETS; idx++) + INIT_LIST_HEAD(&ctl->pp_buckets[idx]); + return ctl; +} + +static void release_pp_slot(struct zram *zram, struct zram_pp_slot *pps) +{ + list_del_init(&pps->entry); + + zram_slot_lock(zram, pps->index); + zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT); + zram_slot_unlock(zram, pps->index); + + kfree(pps); +} + +static void release_pp_ctl(struct zram *zram, struct zram_pp_ctl *ctl) +{ + u32 idx; + + if (!ctl) + return; + + for (idx =3D 0; idx < NUM_PP_BUCKETS; idx++) { + while (!list_empty(&ctl->pp_buckets[idx])) { + struct zram_pp_slot *pps; + + pps =3D list_first_entry(&ctl->pp_buckets[idx], + struct zram_pp_slot, + entry); + release_pp_slot(zram, pps); + } + } + + kfree(ctl); +} + +static void place_pp_slot(struct zram *zram, struct zram_pp_ctl *ctl, + struct zram_pp_slot *pps) +{ + u32 idx; + + idx =3D zram_get_obj_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE; + list_add(&pps->entry, &ctl->pp_buckets[idx]); + + zram_set_flag(zram, pps->index, ZRAM_PP_SLOT); +} + +static struct zram_pp_slot *select_pp_slot(struct zram_pp_ctl *ctl) +{ + struct zram_pp_slot *pps =3D NULL; + s32 idx =3D NUM_PP_BUCKETS - 1; + + /* The higher the bucket id the more optimal slot post-processing is */ + while (idx > 0) { + pps =3D list_first_entry_or_null(&ctl->pp_buckets[idx], + struct zram_pp_slot, + entry); + if (pps) + break; + + idx--; + } + return pps; +} +#endif + static inline void update_used_max(struct zram *zram, const unsigned long pages) { @@ -1657,6 +1750,52 @@ static int zram_bvec_write(struct zram *zram, struct= bio_vec *bvec, } =20 #ifdef CONFIG_ZRAM_MULTI_COMP +#define RECOMPRESS_IDLE (1 << 0) +#define RECOMPRESS_HUGE (1 << 1) + +static int scan_slots_for_recompress(struct zram *zram, u32 mode, + struct zram_pp_ctl *ctl) +{ + unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; + struct zram_pp_slot *pps =3D NULL; + unsigned long index; + + for (index =3D 0; index < nr_pages; index++) { + if (!pps) + pps =3D kmalloc(sizeof(*pps), GFP_KERNEL); + if (!pps) + return -ENOMEM; + + INIT_LIST_HEAD(&pps->entry); + + zram_slot_lock(zram, index); + if (!zram_allocated(zram, index)) + goto next; + + if (mode & RECOMPRESS_IDLE && + !zram_test_flag(zram, index, ZRAM_IDLE)) + goto next; + + if (mode & RECOMPRESS_HUGE && + !zram_test_flag(zram, index, ZRAM_HUGE)) + goto next; + + if (zram_test_flag(zram, index, ZRAM_WB) || + zram_test_flag(zram, index, ZRAM_SAME) || + zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) + goto next; + + pps->index =3D index; + place_pp_slot(zram, ctl, pps); + pps =3D NULL; +next: + zram_slot_unlock(zram, index); + } + + kfree(pps); + return 0; +} + /* * This function will decompress (unless it's ZRAM_HUGE) the page and then * attempt to compress it using provided compression algorithm priority @@ -1664,7 +1803,7 @@ static int zram_bvec_write(struct zram *zram, struct = bio_vec *bvec, * * Corresponding ZRAM slot should be locked. */ -static int zram_recompress(struct zram *zram, u32 index, struct page *page, +static int recompress_slot(struct zram *zram, u32 index, struct page *page, u64 *num_recomp_pages, u32 threshold, u32 prio, u32 prio_max) { @@ -1807,20 +1946,17 @@ static int zram_recompress(struct zram *zram, u32 i= ndex, struct page *page, return 0; } =20 -#define RECOMPRESS_IDLE (1 << 0) -#define RECOMPRESS_HUGE (1 << 1) - static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { u32 prio =3D ZRAM_SECONDARY_COMP, prio_max =3D ZRAM_MAX_COMPS; struct zram *zram =3D dev_to_zram(dev); - unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; char *args, *param, *val, *algo =3D NULL; u64 num_recomp_pages =3D ULLONG_MAX; + struct zram_pp_ctl *ctl =3D NULL; + struct zram_pp_slot *pps; u32 mode =3D 0, threshold =3D 0; - unsigned long index; struct page *page; ssize_t ret; =20 @@ -1922,36 +2058,32 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } =20 + ctl =3D init_pp_ctl(); + if (!ctl) { + ret =3D -ENOMEM; + goto release_init_lock; + } + + scan_slots_for_recompress(zram, mode, ctl); + ret =3D len; - for (index =3D 0; index < nr_pages; index++) { + while ((pps =3D select_pp_slot(ctl))) { int err =3D 0; =20 if (!num_recomp_pages) break; =20 - zram_slot_lock(zram, index); - - if (!zram_allocated(zram, index)) - goto next; - - if (mode & RECOMPRESS_IDLE && - !zram_test_flag(zram, index, ZRAM_IDLE)) + zram_slot_lock(zram, pps->index); + if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) goto next; =20 - if (mode & RECOMPRESS_HUGE && - !zram_test_flag(zram, index, ZRAM_HUGE)) - goto next; - - if (zram_test_flag(zram, index, ZRAM_WB) || - zram_test_flag(zram, index, ZRAM_UNDER_WB) || - zram_test_flag(zram, index, ZRAM_SAME) || - zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) - goto next; - - err =3D zram_recompress(zram, index, page, &num_recomp_pages, - threshold, prio, prio_max); + err =3D recompress_slot(zram, pps->index, page, + &num_recomp_pages, threshold, + prio, prio_max); next: - zram_slot_unlock(zram, index); + zram_slot_unlock(zram, pps->index); + release_pp_slot(zram, pps); + if (err) { ret =3D err; break; @@ -1963,6 +2095,7 @@ static ssize_t recompress_store(struct device *dev, __free_page(page); =20 release_init_lock: + release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); return ret; --=20 2.46.0.662.g92d0881bb0-goog From nobody Fri Nov 29 17:50:02 2024 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6610624B2F for ; Tue, 17 Sep 2024 02:10:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539040; cv=none; b=WejygIx/t+k/kci6mefPlwF90vq7EdrYzk0MUi9EX3pco2oA79DVwYP805LX5HIn2wdL7J51JoF3hk4HMDa3VzSH4T8IAzph4cOfg3u/7uL1sFz7F1l+WELY0J2UKdXL9gc9spxvoatuX0P58dgJ7m4BXBKwlOUCKKMypJgyUWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539040; c=relaxed/simple; bh=1UzoRtPzeopetqWERmEkqhfsk0wJWopivdLtREQqyrU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SEH5nrmTxaUAsK/48Q3o77dlbJ1HVS7tf1QKOAjkaqYo/LQmey87S90BP1hzboFW52T/lxVVM0Q1R06bQs9ClLDr6rlLVFhIJz/JCcOPrsWaWbdYu3KhvyM8Xz9F4PKOeUAyYLvPYvYXsg4Frq94ejj/HU6lBdnQo+hZzaDh0M0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=ayXVTbQL; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ayXVTbQL" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2d889207d1aso3492413a91.3 for ; Mon, 16 Sep 2024 19:10:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726539038; x=1727143838; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XBc6EPDKjhft2m2bi15OmMhHi0RMJoaT8lbBvZ0sG/I=; b=ayXVTbQLxuWvR9Ys6jvRiKFOa/QF55UsCcGSpwwHFGJC/aO1pnQRsdcAdCNEckDWHQ tMiuh1rd8a3FgfDv2RdMFL7d+roMkePJEx8ma3OkUZcnFI/GzVa0dVpBKI5euDcfCV87 zQKPJiYDSBvEqWNPEizh7qRaZ72gFZZYI0GP0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726539038; x=1727143838; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XBc6EPDKjhft2m2bi15OmMhHi0RMJoaT8lbBvZ0sG/I=; b=g3tUJeA+Z835VoD2bwQ3/i5LtpfJ+e2du2iqIlaDAVprHReEN8KF98jExfqNViGWst s4H8OoC1mYq7n9z8vqqBAafe9uLVm8gxjzul/VVd1udGXsngGLWARTky8rtGM8y8zFKB qL7P+TmI/BYkZcTOwDi/Qua+1clciprWmfbjftefumwzi1R7M0L0DFWnB5C5OPtofE7L svno3lt4jNTIR44Ys/hFAIUNDEr41m8oSJzWc4ydGPyIxGWTE7iAssOyyuEyq5aQQdYq eJuqEFSRTBJpELwdsA89FTEVt7c7uUTZ1BeBu/lgUoS9kX4jG0Vclb92gc/tNEviDcUo HblA== X-Gm-Message-State: AOJu0Yxe9aGQ3I88Q+RhSmOsFRrkccXwfVCFxl45ZYzwYLGVMuHAgTRv URYxwUmhNOlbmiVNpjK1KVnB7XE5QdVihYv01MNFPDl2pd9tbpLG20f4nGhCCQ== X-Google-Smtp-Source: AGHT+IETYO4Ocoh32I6uhW1JZ1pxZIbZ1lkYt++qfRNt6XtWAGlO4NVcZuS/nmUqg58ToWfiaa0Cig== X-Received: by 2002:a17:90b:1804:b0:2d4:924:8891 with SMTP id 98e67ed59e1d1-2dba008587emr17222522a91.38.1726539037517; Mon, 16 Sep 2024 19:10:37 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:693c:b4a9:5e6e:c397]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2dbcfd9313fsm6037293a91.44.2024.09.16.19.10.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2024 19:10:37 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv5 4/7] zram: rework writeback target selection strategy Date: Tue, 17 Sep 2024 11:09:09 +0900 Message-ID: <20240917021020.883356-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog In-Reply-To: <20240917021020.883356-1-senozhatsky@chromium.org> References: <20240917021020.883356-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Writeback suffers from the same problem as recompression did before - target slot selection for writeback is just a simple iteration over zram->table entries (stored pages) which selects suboptimal targets for writeback. This is especially problematic for writeback, because we uncompress objects before writeback so each of them takes 4K out of limited writeback storage. For example, when we take a 48 bytes slot and store it as a 4K object to writeback device we only save 48 bytes of memory (release from zsmalloc pool). We naturally want to pick the largest objects for writeback, because then each writeback will release the largest amount of memory. This patch applies the same solution and strategy as for recompression target selection: pp control (post-process) with 16 buckets of candidate pp slots. Slots are assigned to pp buckets based on sizes - the larger the slot the higher the group index. This gives us sorted by size lists of candidate slots (in linear time), so that among post-processing candidate slots we always select the largest ones first and maximize the memory saving. TEST =3D=3D=3D=3D A very simple demonstration: zram is configured with a writeback device. A limited writeback (wb_limit 2500 pages) is performed then, with a log of sizes of slots that were written back. You can see that patched zram selects slots for recompression in significantly different manner, which leads to higher memory savings (see column #2 of mm_stat output). BASE ---- *** initial state of zram device /sys/block/zram0/mm_stat 1750327296 619765836 631902208 0 631902208 1 0 3427= 8 34278 *** writeback idle wb_limit 2500 /sys/block/zram0/mm_stat 1750327296 617622333 631578624 0 631902208 1 0 3427= 8 34278 Sizes of selected objects for writeback: ... 193 349 46 46 46 46 852 1002 543 162 107 49 34 34 34 ... PATCHED ------- *** initial state of zram device /sys/block/zram0/mm_stat 1750319104 619760957 631992320 0 631992320 1 0 3427= 8 34278 *** writeback idle wb_limit 2500 /sys/block/zram0/mm_stat 1750319104 612672056 626135040 0 631992320 1 0 3427= 8 34278 Sizes of selected objects for writeback: ... 3667 3580 3581 3580 3581 3581 3581 3231 3211 3203 3231 3246 ... Note, pp-slots are not strictly sorted, there is a PP_BUCKET_SIZE_RANGE variation of sizes within particular bucket. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 83 +++++++++++++++++++++++++++-------- 1 file changed, 64 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 6688f70b2140..8ba994a5f30e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -184,7 +184,7 @@ static void zram_accessed(struct zram *zram, u32 index) #endif } =20 -#ifdef CONFIG_ZRAM_MULTI_COMP +#if defined CONFIG_ZRAM_WRITEBACK || defined CONFIG_ZRAM_MULTI_COMP struct zram_pp_slot { unsigned long index; struct list_head entry; @@ -681,11 +681,57 @@ static void read_from_bdev_async(struct zram *zram, s= truct page *page, #define IDLE_WRITEBACK (1<<1) #define INCOMPRESSIBLE_WRITEBACK (1<<2) =20 +static int scan_slots_for_writeback(struct zram *zram, u32 mode, + unsigned long nr_pages, + unsigned long index, + struct zram_pp_ctl *ctl) +{ + struct zram_pp_slot *pps =3D NULL; + + for (; nr_pages !=3D 0; index++, nr_pages--) { + if (!pps) + pps =3D kmalloc(sizeof(*pps), GFP_KERNEL); + if (!pps) + return -ENOMEM; + + INIT_LIST_HEAD(&pps->entry); + + zram_slot_lock(zram, index); + if (!zram_allocated(zram, index)) + goto next; + + if (zram_test_flag(zram, index, ZRAM_WB) || + zram_test_flag(zram, index, ZRAM_SAME)) + goto next; + + if (mode & IDLE_WRITEBACK && + !zram_test_flag(zram, index, ZRAM_IDLE)) + goto next; + if (mode & HUGE_WRITEBACK && + !zram_test_flag(zram, index, ZRAM_HUGE)) + goto next; + if (mode & INCOMPRESSIBLE_WRITEBACK && + !zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) + goto next; + + pps->index =3D index; + place_pp_slot(zram, ctl, pps); + pps =3D NULL; +next: + zram_slot_unlock(zram, index); + } + + kfree(pps); + return 0; +} + static ssize_t writeback_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { struct zram *zram =3D dev_to_zram(dev); unsigned long nr_pages =3D zram->disksize >> PAGE_SHIFT; + struct zram_pp_ctl *ctl =3D NULL; + struct zram_pp_slot *pps; unsigned long index =3D 0; struct bio bio; struct bio_vec bio_vec; @@ -737,7 +783,15 @@ static ssize_t writeback_store(struct device *dev, goto release_init_lock; } =20 - for (; nr_pages !=3D 0; index++, nr_pages--) { + ctl =3D init_pp_ctl(); + if (!ctl) { + ret =3D -ENOMEM; + goto release_init_lock; + } + + scan_slots_for_writeback(zram, mode, nr_pages, index, ctl); + + while ((pps =3D select_pp_slot(ctl))) { spin_lock(&zram->wb_limit_lock); if (zram->wb_limit_enable && !zram->bd_wb_limit) { spin_unlock(&zram->wb_limit_lock); @@ -754,25 +808,10 @@ static ssize_t writeback_store(struct device *dev, } } =20 + index =3D pps->index; zram_slot_lock(zram, index); - if (!zram_allocated(zram, index)) - goto next; - - if (zram_test_flag(zram, index, ZRAM_WB) || - zram_test_flag(zram, index, ZRAM_SAME) || - zram_test_flag(zram, index, ZRAM_UNDER_WB)) - goto next; - - if (mode & IDLE_WRITEBACK && - !zram_test_flag(zram, index, ZRAM_IDLE)) - goto next; - if (mode & HUGE_WRITEBACK && - !zram_test_flag(zram, index, ZRAM_HUGE)) - goto next; - if (mode & INCOMPRESSIBLE_WRITEBACK && - !zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) goto next; - /* * Clearing ZRAM_UNDER_WB is duty of caller. * IOW, zram_free_page never clear it. @@ -786,6 +825,8 @@ static ssize_t writeback_store(struct device *dev, zram_clear_flag(zram, index, ZRAM_UNDER_WB); zram_clear_flag(zram, index, ZRAM_IDLE); zram_slot_unlock(zram, index); + + release_pp_slot(zram, pps); continue; } =20 @@ -804,6 +845,8 @@ static ssize_t writeback_store(struct device *dev, zram_clear_flag(zram, index, ZRAM_UNDER_WB); zram_clear_flag(zram, index, ZRAM_IDLE); zram_slot_unlock(zram, index); + + release_pp_slot(zram, pps); /* * BIO errors are not fatal, we continue and simply * attempt to writeback the remaining objects (pages). @@ -846,12 +889,14 @@ static ssize_t writeback_store(struct device *dev, spin_unlock(&zram->wb_limit_lock); next: zram_slot_unlock(zram, index); + release_pp_slot(zram, pps); } =20 if (blk_idx) free_block_bdev(zram, blk_idx); __free_page(page); release_init_lock: + release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); =20 --=20 2.46.0.662.g92d0881bb0-goog From nobody Fri Nov 29 17:50:02 2024 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0F9355896 for ; Tue, 17 Sep 2024 02:10:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539042; cv=none; b=k519P+qYaH6eNvd1xh+O1shA0yD4RbIyVfQIYDlExgQuz4dROSvE69HZDdAU6DvOf3SZze4DXqWQP30L0ToPZF1oGD4tR8e8dILukY1NyKzjlFoqRLxWdLqUGwBcUk64vWBjkrZ/XzAePgVh97Ddv2y8e+hMssWzHc+yhjj8wg0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539042; c=relaxed/simple; bh=zhr17HUYcfU86krIL8CZe49IVeJYIRYESXrjqhMNHgk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YPr3B7pjlS3dhISDAt4qkNkeHnALzKRYb6VFXmEyMhNcgw4V6qp1apQ8SSYe5s0CwM/zbraJie9oXSRTTuOz7PUmPzzwp6vLdmwrj92pDqJ5yCr+gbRoRFhqRQhqjvyNjWBQPbL0wQ6A6A8wchZZztx+LObQZHR2TQmGAJx1CFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=ABZbuAzd; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ABZbuAzd" Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-7db4f323b12so1111951a12.1 for ; Mon, 16 Sep 2024 19:10:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726539040; x=1727143840; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M1LTK3ltYk3gaQ1Vi7PN62/nvHKdNc9jyl2jkjdvFYI=; b=ABZbuAzdFrtETETMaD+QqFaKP1jjBzbW3Pm6bxXPvHYFE/fU8aAu1MXOCvDY1r6SPX DO8lPNsbD9b7x5AcBKx8+Zc+SwHeu+V7P2++ECmNG6SSaLQT6q/wRRTqwut5UpIHFiag +GjpI8PgR15FfFrJbWcScb7/LOkYUjMb9+b2U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726539040; x=1727143840; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M1LTK3ltYk3gaQ1Vi7PN62/nvHKdNc9jyl2jkjdvFYI=; b=tA4yue3OjRpFgPB3OcP/GmqMFimx+RhKLQAYL8j3is+CicqeNFwLz/Wiq9eDvIPvwy m75vNgszERLizG4KFXPPWy1q7B6AA4TKry3ooYbVH+qYMTsHSHqH9fm/TZ9FF4Bkz+m0 I67YSkRjOzZrB0DyqbOPrPOS1eRzJXXM1oehZFNkldDczeP0ng0qHHYImcPONlCtwaQa s94Ul5F4VOKFlIaBNHxf1QYwLdvwUZbjNOJsW/gov3aq1SrprN1OU8LqxzpRfZUDdl1Y dKB/tCAszI4upI46XYebI5NOtBik4Ll8/jZiGKhOZsF2AHJAnWtcy2Rg6QCEDnHuEQHk H5AA== X-Gm-Message-State: AOJu0YzczFZelbMu0QAN3Z7IkWsTWhQJacCCRYOYNuCQTXabzHRz2Rpp LFzrLJInYdxLuJ4S7XMvWtM2IQNEmJXdzFr8jjKIH3H96WukMC0TaE0TAKhNkw== X-Google-Smtp-Source: AGHT+IFAeK4YGCqZpKr44hbfcgl2ZrRa2/PgseZ1YlmEhrarbBcDYBGp0r3tbRSXffjQjZHwatcFkg== X-Received: by 2002:a17:90a:2f64:b0:2d3:ce76:4af2 with SMTP id 98e67ed59e1d1-2db9ffee48amr19368053a91.18.1726539039904; Mon, 16 Sep 2024 19:10:39 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:693c:b4a9:5e6e:c397]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2dbcfd9313fsm6037293a91.44.2024.09.16.19.10.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2024 19:10:39 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv5 5/7] zram: do not mark idle slots that cannot be idle Date: Tue, 17 Sep 2024 11:09:10 +0900 Message-ID: <20240917021020.883356-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog In-Reply-To: <20240917021020.883356-1-senozhatsky@chromium.org> References: <20240917021020.883356-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ZRAM_SAME slots cannot be post-processed (writeback or recompress) so do not mark them ZRAM_IDLE. Same with ZRAM_WB slots, they cannot be ZRAM_IDLE because they are not in zsmalloc pool anymore. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 8ba994a5f30e..e48e2deec685 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -392,17 +392,28 @@ static void mark_idle(struct zram *zram, ktime_t cuto= ff) /* * Do not mark ZRAM_UNDER_WB slot as ZRAM_IDLE to close race. * See the comment in writeback_store. + * + * Also do not mark ZRAM_SAME slots as ZRAM_IDLE, because no + * post-processing (recompress, writeback) happens to the + * ZRAM_SAME slot. + * + * And ZRAM_WB slots simply cannot be ZRAM_IDLE. */ zram_slot_lock(zram, index); - if (zram_allocated(zram, index) && - !zram_test_flag(zram, index, ZRAM_UNDER_WB)) { + if (!zram_allocated(zram, index) || + zram_test_flag(zram, index, ZRAM_WB) || + zram_test_flag(zram, index, ZRAM_UNDER_WB) || + zram_test_flag(zram, index, ZRAM_SAME)) { + zram_slot_unlock(zram, index); + continue; + } + #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME - is_idle =3D !cutoff || ktime_after(cutoff, - zram->table[index].ac_time); + is_idle =3D !cutoff || + ktime_after(cutoff, zram->table[index].ac_time); #endif - if (is_idle) - zram_set_flag(zram, index, ZRAM_IDLE); - } + if (is_idle) + zram_set_flag(zram, index, ZRAM_IDLE); zram_slot_unlock(zram, index); } } --=20 2.46.0.662.g92d0881bb0-goog From nobody Fri Nov 29 17:50:02 2024 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8C527D07D for ; Tue, 17 Sep 2024 02:10:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539044; cv=none; b=iGDq/O2AwSt6WyCw/iOheV9qKzwd7jHM1gy6h9eRuR5AaEFhYC2zxbiZMIlIWWnbGEGI/TJJ0ANX42GT7dNKRLpQNuBdrFwXobBOqLOj2XwBSRwArgeTTo8mp2iJPOAyq9FOsV+mxctRBMJ4INm0xOzOJ6uFZQTc3pKQ6qME7xc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539044; c=relaxed/simple; bh=dO8Y8eFzKLlosbzRPB9JxoQ9y9QTo7ZYeJE0bQVxokk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nqOc9lpJvNYKfDDsB/jXVQXJ+zbWlTipwlu7cNm935lWUY2EqrCMrjyV1HXiAys3XBllt7LVb9eFgi5TSjrC0XLAn60IBgWTM78duA5ISd6nAjKJ8oEcQKUhCXR29vcNulMPDqTWAn9LIXzTmNJRltDfEBFyWHu8MJebtE+/cmg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Vz3oF84g; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Vz3oF84g" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2d8815ef6d2so4264242a91.0 for ; Mon, 16 Sep 2024 19:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726539042; x=1727143842; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZSqaX/BBQLqSgKJc8ElPkCPMmmR5Y/UU2/UCsSZ/N4c=; b=Vz3oF84gRn0P194qVTilS+Hd0nsUo7vK/KgA1p7bq4NYM2WESoMnaenbmCjLKJDBkT XdkTpVIS8BlcbAG8eJPJQ28AQtOImvqtlJxZJD/QD9HgUUA5RlKkJJNgX8w8LiIIqr/g 7j6F+lwQiZqycTcQJxXE5tP8+6roXNfnxEg7Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726539042; x=1727143842; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZSqaX/BBQLqSgKJc8ElPkCPMmmR5Y/UU2/UCsSZ/N4c=; b=Ii6+krKIypmTwopb+ccJcmVFONgGDYxRZYyBypYEVtzey/SbyJReHzjCGtOhEvqxbb 5ERPsVuxBU2AwPvFb8a9DqbgT7hLR6m/e7/OP77B47+7T5n50U3mBhS/Q8atIEfhGbzp hhsgsgk3ieUsIXmmt6eFQKq63uahkthTci7iE3kH0KWN2q6vc0FoUU3H4iQrr4Uf2BVl cylwARpydI5LQzunbnzOWQuti23yzHQur1GlreOlMFdRI3VbV2NCLrUvcxvCEJkEn2+l eH4Ld3O7wkBP4jizZHocLMBN5igHe58OjjRmI3v8L07XoDdx3FvY1GKSR5bAdRihaVZp cZmQ== X-Gm-Message-State: AOJu0YzsJUztPP7LqIsc1NpfkQlfQdw/pBysKRBDK9fmK6L1TlAFfmiA RHRDDxNle0pAB/u9m63vWLBPjSdJMVQgl1RRlD73jQrPzvyXeM0ODNJw4wtznA== X-Google-Smtp-Source: AGHT+IHvC+WDY6c3wdTTvUiPZig/zEMgFN8r/TTkevae4wokGJbRhTrWTypT3PUk2Jy6eciiBW6Mhw== X-Received: by 2002:a17:90a:fa45:b0:2d8:bbf7:bb7b with SMTP id 98e67ed59e1d1-2dba00681d6mr20980912a91.30.1726539042064; Mon, 16 Sep 2024 19:10:42 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:693c:b4a9:5e6e:c397]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2dbcfd9313fsm6037293a91.44.2024.09.16.19.10.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2024 19:10:41 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv5 6/7] zram: reshuffle zram_free_page() flags operations Date: Tue, 17 Sep 2024 11:09:11 +0900 Message-ID: <20240917021020.883356-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog In-Reply-To: <20240917021020.883356-1-senozhatsky@chromium.org> References: <20240917021020.883356-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop some redundant zram_test_flag() calls and re-order zram_clear_flag() calls. Plus two small trivial coding style fixes. No functional changes. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e48e2deec685..8f01dc1fc796 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1499,20 +1499,17 @@ static void zram_free_page(struct zram *zram, size_= t index) #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME zram->table[index].ac_time =3D 0; #endif - if (zram_test_flag(zram, index, ZRAM_IDLE)) - zram_clear_flag(zram, index, ZRAM_IDLE); + + zram_clear_flag(zram, index, ZRAM_IDLE); + zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE); + zram_clear_flag(zram, index, ZRAM_PP_SLOT); + zram_set_priority(zram, index, 0); =20 if (zram_test_flag(zram, index, ZRAM_HUGE)) { zram_clear_flag(zram, index, ZRAM_HUGE); atomic64_dec(&zram->stats.huge_pages); } =20 - if (zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) - zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE); - - zram_set_priority(zram, index, 0); - zram_clear_flag(zram, index, ZRAM_PP_SLOT); - if (zram_test_flag(zram, index, ZRAM_WB)) { zram_clear_flag(zram, index, ZRAM_WB); free_block_bdev(zram, zram_get_element(zram, index)); @@ -1536,13 +1533,12 @@ static void zram_free_page(struct zram *zram, size_= t index) zs_free(zram->mem_pool, handle); =20 atomic64_sub(zram_get_obj_size(zram, index), - &zram->stats.compr_data_size); + &zram->stats.compr_data_size); out: atomic64_dec(&zram->stats.pages_stored); zram_set_handle(zram, index, 0); zram_set_obj_size(zram, index, 0); - WARN_ON_ONCE(zram->table[index].flags & - ~(1UL << ZRAM_UNDER_WB)); + WARN_ON_ONCE(zram->table[index].flags & ~(1UL << ZRAM_UNDER_WB)); } =20 /* --=20 2.46.0.662.g92d0881bb0-goog From nobody Fri Nov 29 17:50:02 2024 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D801B136351 for ; Tue, 17 Sep 2024 02:10:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539046; cv=none; b=MQf8mk89oVUeynj+FFsc/bZtatw0/b+9A1DxBXCHdi0i3Gsfc0GNjYZL592CI3InlsBxBwrAccC/k7rn/mGfuamXB7JhAC8TUY/59KlstxJ9uedoajy4MVF2/VQV6W3HJr9h0lSAu/OthnXatZR8KBBZrovn3b9LKCH4/RHqH3c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726539046; c=relaxed/simple; bh=q4Xj0yUTh7FBoCT9cnIh139VqB03+wo1CTmdLv5xp7Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aWyo9dRT+U/lqw18PgexvTmcGLPwMX43aHYf4PfLLJMZVf/+kZXROUYnjpoMawXCm1H2RBJII9NReizaYjbfM+uSSqD7j0PEDvgKdl9q6+xgVJQELklhcmpGIuXuJXUcN+l20Mgtw+FJ8Wkytu2iFjltZhXlS60coisK/WkCjQs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=dQHUZN0+; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="dQHUZN0+" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2d87a1f0791so2672760a91.2 for ; Mon, 16 Sep 2024 19:10:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1726539044; x=1727143844; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tNimCjKQs/wejmpMYdUz38JPWuuYnJKQPKUcavQwiVg=; b=dQHUZN0+i5R4Mc5C68rdV27wkFVN99rLbHiUHZWQe7xPqP7Jt6rRsK4yeclVR003Wn HjYE2wu/UMIUvJzclYXgzW2tPQF2puxisThYGiwqZPhyMv209PWde4pDYoVSyaCPJSQa h6wtdjZmwff+z0lyE2hj0BQi8kZHxx8zHEyAQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726539044; x=1727143844; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tNimCjKQs/wejmpMYdUz38JPWuuYnJKQPKUcavQwiVg=; b=XyBF9cfUtpYhaeWqDLyXzP+sis7kNo8FU0jw4Z+VUqPB0Bn8rP6orQaoFo+kMKtRkX xY52PJ/MVp0nfMjucqqRW/gjdH88gyLju6COgDkso1He3X7Ea4Yg4bhByW05+s3eh5xe nX81suLShhyDsFFJRT2G1VZcJOElrkRrv3c0edujPQmoEQyOkCexOXdq0XNUI+4Ev0/h WqB0IsG9KzC2qnnV1gghcfRh0G8ltlwraqkFpFNkCAsbk9SJGYMOX8yRFEoeuqUbWd4S 4tWtyPr0fZiFt637EFReLQX1A3D6BNwwbrWcey3r6Eph0NoQhEwyK4/xdUD4l+Xm9s0t Up+g== X-Gm-Message-State: AOJu0YwP5V1IYepGpqSJMyE9pXaljTt2Cf8mcKFTbr7lWD/2YMSOennv oo+Xrag0BcBoH/iDWzESH+D8RANF9VIL6TQw2e0PvDzWnbE2JKSgXuB7Qxl6Bg== X-Google-Smtp-Source: AGHT+IHBYNBS+pXL9JgW1C5lPmU6MOzTHbb3cQsnM6y1w47vPQHBI+WwJNz8c/aIkMYhnRLoxhs+GQ== X-Received: by 2002:a17:90a:c395:b0:2d3:caeb:a9ad with SMTP id 98e67ed59e1d1-2dbb9f34e80mr15912452a91.31.1726539044166; Mon, 16 Sep 2024 19:10:44 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:693c:b4a9:5e6e:c397]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2dbcfd9313fsm6037293a91.44.2024.09.16.19.10.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Sep 2024 19:10:43 -0700 (PDT) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv5 7/7] zram: remove UNDER_WB and simplify writeback Date: Tue, 17 Sep 2024 11:09:12 +0900 Message-ID: <20240917021020.883356-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.46.0.662.g92d0881bb0-goog In-Reply-To: <20240917021020.883356-1-senozhatsky@chromium.org> References: <20240917021020.883356-1-senozhatsky@chromium.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We now have only one active post-processing at any time, so we don't have same race conditions that we had before. If slot selected for post-processing gets freed or freed and reallocated it loses its PP_SLOT flag and there is no way for such a slot to gain PP_SLOT flag again until current post-processing terminates. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 53 +++++++++++------------------------ drivers/block/zram/zram_drv.h | 1 - 2 files changed, 16 insertions(+), 38 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 8f01dc1fc796..69458f8afe7a 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -390,10 +390,7 @@ static void mark_idle(struct zram *zram, ktime_t cutof= f) =20 for (index =3D 0; index < nr_pages; index++) { /* - * Do not mark ZRAM_UNDER_WB slot as ZRAM_IDLE to close race. - * See the comment in writeback_store. - * - * Also do not mark ZRAM_SAME slots as ZRAM_IDLE, because no + * Do not mark ZRAM_SAME slots as ZRAM_IDLE, because no * post-processing (recompress, writeback) happens to the * ZRAM_SAME slot. * @@ -402,7 +399,6 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) zram_slot_lock(zram, index); if (!zram_allocated(zram, index) || zram_test_flag(zram, index, ZRAM_WB) || - zram_test_flag(zram, index, ZRAM_UNDER_WB) || zram_test_flag(zram, index, ZRAM_SAME)) { zram_slot_unlock(zram, index); continue; @@ -821,22 +817,17 @@ static ssize_t writeback_store(struct device *dev, =20 index =3D pps->index; zram_slot_lock(zram, index); - if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) - goto next; /* - * Clearing ZRAM_UNDER_WB is duty of caller. - * IOW, zram_free_page never clear it. + * scan_slots() sets ZRAM_PP_SLOT and relases slot lock, so + * slots can change in the meantime. If slots are accessed or + * freed they lose ZRAM_PP_SLOT flag and hence we don't + * post-process them. */ - zram_set_flag(zram, index, ZRAM_UNDER_WB); - /* Need for hugepage writeback racing */ - zram_set_flag(zram, index, ZRAM_IDLE); + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) + goto next; zram_slot_unlock(zram, index); - if (zram_read_page(zram, page, index, NULL)) { - zram_slot_lock(zram, index); - zram_clear_flag(zram, index, ZRAM_UNDER_WB); - zram_clear_flag(zram, index, ZRAM_IDLE); - zram_slot_unlock(zram, index); =20 + if (zram_read_page(zram, page, index, NULL)) { release_pp_slot(zram, pps); continue; } @@ -852,11 +843,6 @@ static ssize_t writeback_store(struct device *dev, */ err =3D submit_bio_wait(&bio); if (err) { - zram_slot_lock(zram, index); - zram_clear_flag(zram, index, ZRAM_UNDER_WB); - zram_clear_flag(zram, index, ZRAM_IDLE); - zram_slot_unlock(zram, index); - release_pp_slot(zram, pps); /* * BIO errors are not fatal, we continue and simply @@ -871,25 +857,19 @@ static ssize_t writeback_store(struct device *dev, } =20 atomic64_inc(&zram->stats.bd_writes); + zram_slot_lock(zram, index); /* - * We released zram_slot_lock so need to check if the slot was - * changed. If there is freeing for the slot, we can catch it - * easily by zram_allocated. - * A subtle case is the slot is freed/reallocated/marked as - * ZRAM_IDLE again. To close the race, idle_store doesn't - * mark ZRAM_IDLE once it found the slot was ZRAM_UNDER_WB. - * Thus, we could close the race by checking ZRAM_IDLE bit. + * Same as above, we release slot lock during writeback so + * slot can change under us: slot_free() or slot_free() and + * reallocation (zram_write_page()). In both cases slot loses + * ZRAM_PP_SLOT flag. No concurrent post-processing can set + * ZRAM_PP_SLOT on such slots until current post-processing + * finishes. */ - zram_slot_lock(zram, index); - if (!zram_allocated(zram, index) || - !zram_test_flag(zram, index, ZRAM_IDLE)) { - zram_clear_flag(zram, index, ZRAM_UNDER_WB); - zram_clear_flag(zram, index, ZRAM_IDLE); + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) goto next; - } =20 zram_free_page(zram, index); - zram_clear_flag(zram, index, ZRAM_UNDER_WB); zram_set_flag(zram, index, ZRAM_WB); zram_set_element(zram, index, blk_idx); blk_idx =3D 0; @@ -1538,7 +1518,6 @@ static void zram_free_page(struct zram *zram, size_t = index) atomic64_dec(&zram->stats.pages_stored); zram_set_handle(zram, index, 0); zram_set_obj_size(zram, index, 0); - WARN_ON_ONCE(zram->table[index].flags & ~(1UL << ZRAM_UNDER_WB)); } =20 /* diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 73a9d47d76ba..134be414e210 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -47,7 +47,6 @@ enum zram_pageflags { ZRAM_SAME =3D ZRAM_FLAG_SHIFT, /* Page consists the same element */ ZRAM_WB, /* page is stored on backing_device */ - ZRAM_UNDER_WB, /* page is under writeback */ ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ ZRAM_IDLE, /* not accessed page since last idle marking */ --=20 2.46.0.662.g92d0881bb0-goog