add static PMD zero page support

[PATCH v2 5/5] block: use largest_zero_folio in __blkdev_issue_zero_pages()

Posted by Pankaj Raghav (Samsung) 3 months ago

From: Pankaj Raghav <p.raghav@samsung.com>

Use largest_zero_folio() in __blkdev_issue_zero_pages().

On systems with CONFIG_STATIC_PMD_ZERO_PAGE enabled, we will end up
sending larger bvecs instead of multiple small ones.

Noticed a 4% increase in performance on a commercial NVMe SSD which does
not support OP_WRITE_ZEROES. The device's MDTS was 128K. The performance
gains might be bigger if the device supports bigger MDTS.

Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
---
 block/blk-lib.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/block/blk-lib.c b/block/blk-lib.c
index 4c9f20a689f7..70a5700b6717 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -196,6 +196,10 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev,
 		sector_t sector, sector_t nr_sects, gfp_t gfp_mask,
 		struct bio **biop, unsigned int flags)
 {
+	struct folio *zero_folio;
+
+	zero_folio = largest_zero_folio();
+
 	while (nr_sects) {
 		unsigned int nr_vecs = __blkdev_sectors_to_bio_pages(nr_sects);
 		struct bio *bio;
@@ -208,15 +212,14 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev,
 			break;
 
 		do {
-			unsigned int len, added;
+			unsigned int len;
 
-			len = min_t(sector_t,
-				PAGE_SIZE, nr_sects << SECTOR_SHIFT);
-			added = bio_add_page(bio, ZERO_PAGE(0), len, 0);
-			if (added < len)
+			len = min_t(sector_t, folio_size(zero_folio),
+				    nr_sects << SECTOR_SHIFT);
+			if (!bio_add_folio(bio, zero_folio, len, 0))
 				break;
-			nr_sects -= added >> SECTOR_SHIFT;
-			sector += added >> SECTOR_SHIFT;
+			nr_sects -= len >> SECTOR_SHIFT;
+			sector += len >> SECTOR_SHIFT;
 		} while (nr_sects);
 
 		*biop = bio_chain_and_submit(*biop, bio);
-- 
2.49.0

Re: [PATCH v2 5/5] block: use largest_zero_folio in __blkdev_issue_zero_pages()

Posted by Lorenzo Stoakes 2 months, 3 weeks ago

On Mon, Jul 07, 2025 at 04:23:19PM +0200, Pankaj Raghav (Samsung) wrote:
> From: Pankaj Raghav <p.raghav@samsung.com>
>
> Use largest_zero_folio() in __blkdev_issue_zero_pages().
>
> On systems with CONFIG_STATIC_PMD_ZERO_PAGE enabled, we will end up
> sending larger bvecs instead of multiple small ones.
>
> Noticed a 4% increase in performance on a commercial NVMe SSD which does
> not support OP_WRITE_ZEROES. The device's MDTS was 128K. The performance
> gains might be bigger if the device supports bigger MDTS.
>
> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> ---
>  block/blk-lib.c | 17 ++++++++++-------
>  1 file changed, 10 insertions(+), 7 deletions(-)
>
> diff --git a/block/blk-lib.c b/block/blk-lib.c
> index 4c9f20a689f7..70a5700b6717 100644
> --- a/block/blk-lib.c
> +++ b/block/blk-lib.c
> @@ -196,6 +196,10 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev,
>  		sector_t sector, sector_t nr_sects, gfp_t gfp_mask,
>  		struct bio **biop, unsigned int flags)
>  {
> +	struct folio *zero_folio;
> +
> +	zero_folio = largest_zero_folio();

Just assign this in the decl :)

> +
>  	while (nr_sects) {
>  		unsigned int nr_vecs = __blkdev_sectors_to_bio_pages(nr_sects);
>  		struct bio *bio;
> @@ -208,15 +212,14 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev,
>  			break;
>
>  		do {
> -			unsigned int len, added;
> +			unsigned int len;
>
> -			len = min_t(sector_t,
> -				PAGE_SIZE, nr_sects << SECTOR_SHIFT);
> -			added = bio_add_page(bio, ZERO_PAGE(0), len, 0);
> -			if (added < len)
> +			len = min_t(sector_t, folio_size(zero_folio),
> +				    nr_sects << SECTOR_SHIFT);
> +			if (!bio_add_folio(bio, zero_folio, len, 0))

Hmm, will this work if nr_sects << SECTOR_SHIFT size isn't PMD-aligned?

I guess it actually just copies individual pages in the folio as needed?

Does this actually result in a significant performance improvement? Do we
have numbers for this to justify the series?

>  				break;
> -			nr_sects -= added >> SECTOR_SHIFT;
> -			sector += added >> SECTOR_SHIFT;
> +			nr_sects -= len >> SECTOR_SHIFT;
> +			sector += len >> SECTOR_SHIFT;
>  		} while (nr_sects);
>
>  		*biop = bio_chain_and_submit(*biop, bio);
> --
> 2.49.0
>

Re: [PATCH v2 5/5] block: use largest_zero_folio in __blkdev_issue_zero_pages()

Posted by Pankaj Raghav (Samsung) 2 months, 3 weeks ago

On Tue, Jul 15, 2025 at 05:19:54PM +0100, Lorenzo Stoakes wrote:
> On Mon, Jul 07, 2025 at 04:23:19PM +0200, Pankaj Raghav (Samsung) wrote:
> > From: Pankaj Raghav <p.raghav@samsung.com>
> >
> > Use largest_zero_folio() in __blkdev_issue_zero_pages().
> >
> > On systems with CONFIG_STATIC_PMD_ZERO_PAGE enabled, we will end up
> > sending larger bvecs instead of multiple small ones.
> >
> > Noticed a 4% increase in performance on a commercial NVMe SSD which does
> > not support OP_WRITE_ZEROES. The device's MDTS was 128K. The performance
> > gains might be bigger if the device supports bigger MDTS.
> >
> > Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> > ---
> >  block/blk-lib.c | 17 ++++++++++-------
> >  1 file changed, 10 insertions(+), 7 deletions(-)
> >
> > diff --git a/block/blk-lib.c b/block/blk-lib.c
> > index 4c9f20a689f7..70a5700b6717 100644
> > --- a/block/blk-lib.c
> > +++ b/block/blk-lib.c
> > @@ -196,6 +196,10 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev,
> >  		sector_t sector, sector_t nr_sects, gfp_t gfp_mask,
> >  		struct bio **biop, unsigned int flags)
> >  {
> > +	struct folio *zero_folio;
> > +
> > +	zero_folio = largest_zero_folio();
> 
> Just assign this in the decl :)
Yeah!
> 
> > +
> >  	while (nr_sects) {
> >  		unsigned int nr_vecs = __blkdev_sectors_to_bio_pages(nr_sects);
> >  		struct bio *bio;
> > @@ -208,15 +212,14 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev,
> >  			break;
> >
> >  		do {
> > -			unsigned int len, added;
> > +			unsigned int len;
> >
> > -			len = min_t(sector_t,
> > -				PAGE_SIZE, nr_sects << SECTOR_SHIFT);
> > -			added = bio_add_page(bio, ZERO_PAGE(0), len, 0);
> > -			if (added < len)
> > +			len = min_t(sector_t, folio_size(zero_folio),
> > +				    nr_sects << SECTOR_SHIFT);
> > +			if (!bio_add_folio(bio, zero_folio, len, 0))
> 
> Hmm, will this work if nr_sects << SECTOR_SHIFT size isn't PMD-aligned?

Yeah, that should not be a problem as long as (nr_sects << SECTOR_SHIFT) < PMD_SIZED
folio.

> 
> I guess it actually just copies individual pages in the folio as needed?
> 
> Does this actually result in a significant performance improvement? Do we
> have numbers for this to justify the series?
I put it in my commit message:
```
Noticed a 4% increase in performance on a commercial NVMe SSD which does
not support OP_WRITE_ZEROES. The device's MDTS was 128K. The performance
gains might be bigger if the device supports bigger MDTS.
```

Even though it is more of a synthetic benchmark, but this goes to show the
effects of adding multiple bio_vecs with ZERO_PAGE instead of single
PMD_ZERO_PAGE.

--
Pankaj