From nobody Sat Apr 11 07:03:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01E1EC7EE36 for ; Thu, 2 Mar 2023 23:18:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229831AbjCBXSW (ORCPT ); Thu, 2 Mar 2023 18:18:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229461AbjCBXSP (ORCPT ); Thu, 2 Mar 2023 18:18:15 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86D47580F6 for ; Thu, 2 Mar 2023 15:17:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677799012; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1eE9Pafa530L5JjpUU74EsDdw6S9K1gsLvm9PBz92W8=; b=e5UibSOrWGxcxGg2l4E3lQMiSC1vlGR9bBNe4k4Xs0e4w/tEBHZWvaouNgCKh5KXj4bFMG 8Ys+EikQqRfQrcspmFb2sOoqBxSsAyqf0Poi0BA0SIjFAOdw7ZtJGPv/6kveQcpYEK8z1O vUeSrDSMlaDIn9EU6KQSSsgLkSZVI1E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-622-zwFLchJEPOeBqPxHgNImPw-1; Thu, 02 Mar 2023 18:16:47 -0500 X-MC-Unique: zwFLchJEPOeBqPxHgNImPw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 630DB800B23; Thu, 2 Mar 2023 23:16:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 54038440D9; Thu, 2 Mar 2023 23:16:44 +0000 (UTC) From: David Howells To: Linus Torvalds , Steve French Cc: David Howells , Vishal Moola , Shyam Prasad N , Rohith Surabattula , Tom Talpey , Stefan Metzmacher , Paulo Alcantara , Jeff Layton , Matthew Wilcox , Marc Dionne , linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Steve French , Andrew Morton , linux-mm@kvack.org Subject: [PATCH 1/3] mm: Add a function to get a single tagged folio from a file Date: Thu, 2 Mar 2023 23:16:36 +0000 Message-Id: <20230302231638.521280-2-dhowells@redhat.com> In-Reply-To: <20230302231638.521280-1-dhowells@redhat.com> References: <20230302231638.521280-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add a function to get a single tagged folio from a file rather than a batch for use in afs and cifs where, in the common case, the batch is likely to be rendered irrelevant by the {afs,cifs}_extend_writeback() function. For filemap_get_folios_tag() to be of use, the batch has to be passed down, and if it contains scattered, non-contiguous folios, these are likely to end up being pinned by the batch for significant periods of time whilst I/O is undertaken on earlier pages. Further, for write_cache_pages() to be useful, it would need to wait for PG_fscache which is used to indicate that I/O is in progress from a folio to the cache - but it can't do this unconditionally as some filesystems, such as btrfs, use PG_private_2 for other purposes. Signed-off-by: David Howells cc: Steve French cc: Linus Torvalds cc: "Vishal Moola (Oracle)" cc: "Matthew Wilcox (Oracle)" cc: Andrew Morton cc: linux-afs@lists.infradead.org cc: linux-cifs@vger.kernel.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/2214157.1677250083@warthog.procyon.org.uk/ --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 58 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0acb8e1fb7af..577535633006 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -741,6 +741,8 @@ unsigned filemap_get_folios_contig(struct address_space= *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch); unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *st= art, pgoff_t end, xa_mark_t tag, struct folio_batch *fbatch); +struct folio *filemap_get_folio_tag(struct address_space *mapping, pgoff_t= *start, + pgoff_t end, xa_mark_t tag); =20 struct page *grab_cache_page_write_begin(struct address_space *mapping, pgoff_t index); diff --git a/mm/filemap.c b/mm/filemap.c index 2723104cc06a..1b1e9c661018 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2339,6 +2339,64 @@ unsigned filemap_get_folios_tag(struct address_space= *mapping, pgoff_t *start, } EXPORT_SYMBOL(filemap_get_folios_tag); =20 +/** + * filemap_get_folio_tag - Get the first folio matching @tag + * @mapping: The address_space to search + * @start: The starting page index + * @end: The final page index (inclusive) + * @tag: The tag index + * + * Search for and return the first folios in the mapping starting at index + * @start and up to index @end (inclusive). The folio is returned with an + * elevated reference count. + * + * If a folio is returned, it may start before @start; if it does, it will + * contain @start. The folio may also extend beyond @end; if it does, it = will + * contain @end. If folios are added to or removed from the page cache wh= ile + * this is running, they may or may not be found by this call. + * + * Return: The folio that was found or NULL. @start is also updated to in= dex + * the next folio for the traversal or will be left pointing after @end. + */ +struct folio *filemap_get_folio_tag(struct address_space *mapping, pgoff_t= *start, + pgoff_t end, xa_mark_t tag) +{ + XA_STATE(xas, &mapping->i_pages, *start); + struct folio *folio; + + rcu_read_lock(); + while ((folio =3D find_get_entry(&xas, end, tag)) !=3D NULL) { + /* + * Shadow entries should never be tagged, but this iteration + * is lockless so there is a window for page reclaim to evict + * a page we saw tagged. Skip over it. + */ + if (xa_is_value(folio)) + continue; + + if (folio_test_hugetlb(folio)) + *start =3D folio->index + 1; + else + *start =3D folio_next_index(folio); + goto out; + } + + /* + * We come here when there is no page beyond @end. We take care to not + * overflow the index @start as it confuses some of the callers. This + * breaks the iteration when there is a page at index -1 but that is + * already broke anyway. + */ + if (end =3D=3D (pgoff_t)-1) + *start =3D (pgoff_t)-1; + else + *start =3D end + 1; +out: + rcu_read_unlock(); + return folio; +} +EXPORT_SYMBOL(filemap_get_folio_tag); + /* * CD/DVDs are error prone. When a medium error occurs, the driver may fail * a _large_ part of the i/o request. Imagine the worst scenario: From nobody Sat Apr 11 07:03:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 347D2C678D4 for ; Thu, 2 Mar 2023 23:18:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229907AbjCBXSZ (ORCPT ); Thu, 2 Mar 2023 18:18:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229482AbjCBXSP (ORCPT ); Thu, 2 Mar 2023 18:18:15 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C6FE5943D for ; Thu, 2 Mar 2023 15:17:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677799013; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nNYSQ/Y9B3aZRJt+LYUgV6EcG7vLla4aEHu0riyXHik=; b=ccQsJISbti997Ttdl2XBgh7bmwyhSQlSx3C9X5OHPsJbOIUfNYa1Yukvqv4NKoFngAu8op Bz2eL/FA15HwjbpruglQIHmYkTcjfHCvx/zLY35rXjJCDeuLyJxk4BH27S7w8UJfS3P+h0 3mbAYQoQlJbNC9w14z0LbgUZ5QaLBZI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-640-Ema7s1klNeSsDP6d3fPMHw-1; Thu, 02 Mar 2023 18:16:50 -0500 X-MC-Unique: Ema7s1klNeSsDP6d3fPMHw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2767F1C05AC1; Thu, 2 Mar 2023 23:16:49 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id 15443492C3E; Thu, 2 Mar 2023 23:16:46 +0000 (UTC) From: David Howells To: Linus Torvalds , Steve French Cc: David Howells , Vishal Moola , Shyam Prasad N , Rohith Surabattula , Tom Talpey , Stefan Metzmacher , Paulo Alcantara , Jeff Layton , Matthew Wilcox , Marc Dionne , linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Steve French , Andrew Morton , linux-mm@kvack.org Subject: [PATCH 2/3] afs: Partially revert and use filemap_get_folio_tag() Date: Thu, 2 Mar 2023 23:16:37 +0000 Message-Id: <20230302231638.521280-3-dhowells@redhat.com> In-Reply-To: <20230302231638.521280-1-dhowells@redhat.com> References: <20230302231638.521280-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Partially revert the changes made by: acc8d8588cb7e3e64b0d2fa611dad06574cd67b1. afs: convert afs_writepages_region() to use filemap_get_folios_tag() The issue is that filemap_get_folios_tag() gets a batch of pages at a time, and then afs_writepages_region() goes through them one at a time, extends each into an operation with as many pages as will fit using the loop in afs_extend_writeback() and submits it - but, in the common case, this means that the other pages in the batch already got annexed and processed in afs_extend_writeback() and we end up doing duplicate processing. Switching to write_cache_pages() isn't an immediate substitute as that doesn't take account of PG_fscache (and this bit is used in other ways by other filesystems). So go back to finding the next folio from the VM one at a time and then extending the op onwards. Fixes: acc8d8588cb7 ("afs: convert afs_writepages_region() to use filemap_g= et_folios_tag()") Signed-off-by: David Howells cc: Linus Torvalds cc: Marc Dionne cc: Steve French cc: "Vishal Moola (Oracle)" cc: "Matthew Wilcox (Oracle)" cc: Andrew Morton cc: linux-afs@lists.infradead.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/2214157.1677250083@warthog.procyon.org.uk/ --- fs/afs/write.c | 118 ++++++++++++++++++++++++------------------------- 1 file changed, 57 insertions(+), 61 deletions(-) diff --git a/fs/afs/write.c b/fs/afs/write.c index 571f3b9a417e..2ed76697be96 100644 --- a/fs/afs/write.c +++ b/fs/afs/write.c @@ -704,87 +704,83 @@ static int afs_writepages_region(struct address_space= *mapping, bool max_one_loop) { struct folio *folio; - struct folio_batch fbatch; ssize_t ret; - unsigned int i; - int n, skips =3D 0; + int skips =3D 0; =20 _enter("%llx,%llx,", start, end); - folio_batch_init(&fbatch); =20 do { pgoff_t index =3D start / PAGE_SIZE; =20 - n =3D filemap_get_folios_tag(mapping, &index, end / PAGE_SIZE, - PAGECACHE_TAG_DIRTY, &fbatch); - - if (!n) + folio =3D filemap_get_folio_tag(mapping, &index, end / PAGE_SIZE, + PAGECACHE_TAG_DIRTY); + if (!folio) break; - for (i =3D 0; i < n; i++) { - folio =3D fbatch.folios[i]; - start =3D folio_pos(folio); /* May regress with THPs */ =20 - _debug("wback %lx", folio_index(folio)); + start =3D folio_pos(folio); /* May regress with THPs */ =20 - /* At this point we hold neither the i_pages lock nor the - * page lock: the page may be truncated or invalidated - * (changing page->mapping to NULL), or even swizzled - * back from swapper_space to tmpfs file mapping - */ - if (wbc->sync_mode !=3D WB_SYNC_NONE) { - ret =3D folio_lock_killable(folio); - if (ret < 0) { - folio_batch_release(&fbatch); - return ret; - } - } else { - if (!folio_trylock(folio)) - continue; - } + _debug("wback %lx", folio_index(folio)); =20 - if (folio->mapping !=3D mapping || - !folio_test_dirty(folio)) { - start +=3D folio_size(folio); - folio_unlock(folio); - continue; + /* At this point we hold neither the i_pages lock nor the + * page lock: the page may be truncated or invalidated + * (changing page->mapping to NULL), or even swizzled + * back from swapper_space to tmpfs file mapping + */ + if (wbc->sync_mode !=3D WB_SYNC_NONE) { + ret =3D folio_lock_killable(folio); + if (ret < 0) { + folio_put(folio); + return ret; + } + } else { + if (!folio_trylock(folio)) { + folio_put(folio); + return 0; } + } =20 - if (folio_test_writeback(folio) || - folio_test_fscache(folio)) { - folio_unlock(folio); - if (wbc->sync_mode !=3D WB_SYNC_NONE) { - folio_wait_writeback(folio); + if (folio_mapping(folio) !=3D mapping || + !folio_test_dirty(folio)) { + start +=3D folio_size(folio); + folio_unlock(folio); + folio_put(folio); + continue; + } + + if (folio_test_writeback(folio) || + folio_test_fscache(folio)) { + folio_unlock(folio); + if (wbc->sync_mode !=3D WB_SYNC_NONE) { + folio_wait_writeback(folio); #ifdef CONFIG_AFS_FSCACHE - folio_wait_fscache(folio); + folio_wait_fscache(folio); #endif - } else { - start +=3D folio_size(folio); - } - if (wbc->sync_mode =3D=3D WB_SYNC_NONE) { - if (skips >=3D 5 || need_resched()) { - *_next =3D start; - _leave(" =3D 0 [%llx]", *_next); - return 0; - } - skips++; - } - continue; + } else { + start +=3D folio_size(folio); } - - if (!folio_clear_dirty_for_io(folio)) - BUG(); - ret =3D afs_write_back_from_locked_folio(mapping, wbc, - folio, start, end); - if (ret < 0) { - _leave(" =3D %zd", ret); - folio_batch_release(&fbatch); - return ret; + folio_put(folio); + if (wbc->sync_mode =3D=3D WB_SYNC_NONE) { + if (skips >=3D 5 || need_resched()) + break; + skips++; } + continue; + } =20 - start +=3D ret; + if (!folio_clear_dirty_for_io(folio)) + BUG(); + ret =3D afs_write_back_from_locked_folio(mapping, wbc, folio, start, end= ); + folio_put(folio); + if (ret < 0) { + _leave(" =3D %zd", ret); + return ret; } =20 - folio_batch_release(&fbatch); + start +=3D ret; + + if (max_one_loop) + break; + cond_resched(); } while (wbc->nr_to_write > 0); From nobody Sat Apr 11 07:03:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96606C678D4 for ; Thu, 2 Mar 2023 23:18:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229660AbjCBXSm (ORCPT ); Thu, 2 Mar 2023 18:18:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229844AbjCBXSW (ORCPT ); Thu, 2 Mar 2023 18:18:22 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EAFB59E47 for ; Thu, 2 Mar 2023 15:17:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677799014; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ix2GZROjqu1lNaAS5h0hXsC2zG3GiI8z8sp15KcpJfk=; b=LcvyD3ux3Y1/612Hm7SmO8jncvmUm10zvKPjdqMOQEdly2yg3E58rGsiP/9bYHXVNrF2X3 9HzILPVWJLDS64eA7BFCynWYb6eOdZ/OgBrAuaHsu/1UEmZr8feYK4ItmPoj7fIfs6zP8N n8ypbm9iBM9838fPlSFajkyIntRLcBc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-190-zQT8bSDzMRqgB_Iir98xAw-1; Thu, 02 Mar 2023 18:16:52 -0500 X-MC-Unique: zQT8bSDzMRqgB_Iir98xAw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C4BAA85A5A3; Thu, 2 Mar 2023 23:16:51 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6EE72166B26; Thu, 2 Mar 2023 23:16:49 +0000 (UTC) From: David Howells To: Linus Torvalds , Steve French Cc: David Howells , Vishal Moola , Shyam Prasad N , Rohith Surabattula , Tom Talpey , Stefan Metzmacher , Paulo Alcantara , Jeff Layton , Matthew Wilcox , Marc Dionne , linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Steve French , Andrew Morton , linux-mm@kvack.org Subject: [PATCH 3/3] cifs: Partially revert and use filemap_get_folio_tag() Date: Thu, 2 Mar 2023 23:16:38 +0000 Message-Id: <20230302231638.521280-4-dhowells@redhat.com> In-Reply-To: <20230302231638.521280-1-dhowells@redhat.com> References: <20230302231638.521280-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Mirror the changes made to afs to partially revert the changes made by: acc8d8588cb7e3e64b0d2fa611dad06574cd67b1. "afs: convert afs_writepages_region() to use filemap_get_folios_tag()" that were then mirrored into cifs. The issue is that filemap_get_folios_tag() gets a batch of pages at a time, and then cifs_writepages_region() goes through them one at a time, extends each into an operation with as many pages as will fit using the loop in cifs_extend_writeback() and submits it - but, in the common case, this means that the other pages in the batch already got annexed and processed in cifs_extend_writeback() and we end up doing duplicate processing. Switching to write_cache_pages() isn't an immediate substitute as that doesn't take account of PG_fscache (and this bit is used in other ways by other filesystems). So go back to finding the next folio from the VM one at a time and then extending the op onwards. Fixes: 3822a7c40997 ("Merge tag 'mm-stable-2023-02-20-13-37' of git://git.k= ernel.org/pub/scm/linux/kernel/git/akpm/mm") Signed-off-by: David Howells cc: Steve French cc: Linus Torvalds cc: Shyam Prasad N cc: Rohith Surabattula cc: Jeff Layton cc: Paulo Alcantara cc: "Vishal Moola (Oracle)" cc: "Matthew Wilcox (Oracle)" cc: Andrew Morton cc: linux-cifs@vger.kernel.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/2214157.1677250083@warthog.procyon.org.uk/ --- fs/cifs/file.c | 115 +++++++++++++++++++++---------------------------- 1 file changed, 49 insertions(+), 66 deletions(-) diff --git a/fs/cifs/file.c b/fs/cifs/file.c index 4d4a2d82636d..a3e89e741b42 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -2864,93 +2864,76 @@ static int cifs_writepages_region(struct address_sp= ace *mapping, struct writeback_control *wbc, loff_t start, loff_t end, loff_t *_next) { - struct folio_batch fbatch; + struct folio *folio; + ssize_t ret; int skips =3D 0; =20 - folio_batch_init(&fbatch); do { - int nr; pgoff_t index =3D start / PAGE_SIZE; =20 - nr =3D filemap_get_folios_tag(mapping, &index, end / PAGE_SIZE, - PAGECACHE_TAG_DIRTY, &fbatch); - if (!nr) + folio =3D filemap_get_folio_tag(mapping, &index, end / PAGE_SIZE, + PAGECACHE_TAG_DIRTY); + if (!folio) break; =20 - for (int i =3D 0; i < nr; i++) { - ssize_t ret; - struct folio *folio =3D fbatch.folios[i]; - -redo_folio: - start =3D folio_pos(folio); /* May regress with THPs */ + start =3D folio_pos(folio); /* May regress with THPs */ =20 - /* At this point we hold neither the i_pages lock nor the - * page lock: the page may be truncated or invalidated - * (changing page->mapping to NULL), or even swizzled - * back from swapper_space to tmpfs file mapping - */ - if (wbc->sync_mode !=3D WB_SYNC_NONE) { - ret =3D folio_lock_killable(folio); - if (ret < 0) - goto write_error; - } else { - if (!folio_trylock(folio)) - goto skip_write; + /* At this point we hold neither the i_pages lock nor the + * page lock: the page may be truncated or invalidated + * (changing page->mapping to NULL), or even swizzled + * back from swapper_space to tmpfs file mapping + */ + if (wbc->sync_mode !=3D WB_SYNC_NONE) { + ret =3D folio_lock_killable(folio); + if (ret < 0) { + folio_put(folio); + return ret; } - - if (folio_mapping(folio) !=3D mapping || - !folio_test_dirty(folio)) { - start +=3D folio_size(folio); - folio_unlock(folio); - continue; + } else { + if (!folio_trylock(folio)) { + folio_put(folio); + return 0; } + } =20 - if (folio_test_writeback(folio) || - folio_test_fscache(folio)) { - folio_unlock(folio); - if (wbc->sync_mode =3D=3D WB_SYNC_NONE) - goto skip_write; + if (folio_mapping(folio) !=3D mapping || + !folio_test_dirty(folio)) { + start +=3D folio_size(folio); + folio_unlock(folio); + folio_put(folio); + continue; + } =20 + if (folio_test_writeback(folio) || + folio_test_fscache(folio)) { + folio_unlock(folio); + if (wbc->sync_mode !=3D WB_SYNC_NONE) { folio_wait_writeback(folio); #ifdef CONFIG_CIFS_FSCACHE folio_wait_fscache(folio); #endif - goto redo_folio; + } else { + start +=3D folio_size(folio); } - - if (!folio_clear_dirty_for_io(folio)) - /* We hold the page lock - it should've been dirty. */ - WARN_ON(1); - - ret =3D cifs_write_back_from_locked_folio(mapping, wbc, folio, start, e= nd); - if (ret < 0) - goto write_error; - - start +=3D ret; - continue; - -write_error: - folio_batch_release(&fbatch); - *_next =3D start; - return ret; - -skip_write: - /* - * Too many skipped writes, or need to reschedule? - * Treat it as a write error without an error code. - */ - if (skips >=3D 5 || need_resched()) { - ret =3D 0; - goto write_error; + folio_put(folio); + if (wbc->sync_mode =3D=3D WB_SYNC_NONE) { + if (skips >=3D 5 || need_resched()) + break; + skips++; } - - /* Otherwise, just skip that folio and go on to the next */ - skips++; - start +=3D folio_size(folio); continue; } =20 - folio_batch_release(&fbatch); =09 + if (!folio_clear_dirty_for_io(folio)) + /* We hold the page lock - it should've been dirty. */ + WARN_ON(1); + + ret =3D cifs_write_back_from_locked_folio(mapping, wbc, folio, start, en= d); + folio_put(folio); + if (ret < 0) + return ret; + + start +=3D ret; cond_resched(); } while (wbc->nr_to_write > 0); From nobody Sat Apr 11 07:03:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F8D2C678D4 for ; Thu, 2 Mar 2023 23:37:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229565AbjCBXhx (ORCPT ); Thu, 2 Mar 2023 18:37:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229972AbjCBXhv (ORCPT ); Thu, 2 Mar 2023 18:37:51 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F35B1584BE for ; Thu, 2 Mar 2023 15:37:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677800223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CfB+4L/9Skhmwx3bsII6X8Q4qh3L+58o2DaPND89LWc=; b=bltaocvEATTM45BMeVr/PccQxnCP6+gJWCwCDm1RTzBFneL0eJIZr1tGm/wksSMSlOB43e 13mJtc00RiLNfdS6sef37Cuo1ZVn10dPFNbPIk2GOb7QMn/3tvBz9kVf4909YuVMiBpHFw 9YpGHHAZJ+th+qyxo/ca3XFLvWQeGtY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-404-TCftiIv7MNuWSIH4rQMgEw-1; Thu, 02 Mar 2023 18:36:52 -0500 X-MC-Unique: TCftiIv7MNuWSIH4rQMgEw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9B5DC87A382; Thu, 2 Mar 2023 23:36:51 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6ED2140EBF6; Thu, 2 Mar 2023 23:36:49 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 From: David Howells In-Reply-To: <20230302231638.521280-1-dhowells@redhat.com> References: <20230302231638.521280-1-dhowells@redhat.com> To: Linus Torvalds , Steve French Cc: dhowells@redhat.com, Vishal Moola , Shyam Prasad N , Rohith Surabattula , Tom Talpey , Stefan Metzmacher , Paulo Alcantara , Jeff Layton , Matthew Wilcox , Marc Dionne , linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: cifs test patch to convert to using write_cache_pages() MIME-Version: 1.0 Content-ID: <522330.1677800209.1@warthog.procyon.org.uk> Content-Transfer-Encoding: quoted-printable Date: Thu, 02 Mar 2023 23:36:49 +0000 Message-ID: <522331.1677800209@warthog.procyon.org.uk> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" David Howells wrote: > And then CIFS. ... >=20 > Base + write_cache_pages(): > WRITE: bw=3D457MiB/s (479MB/s), 114MiB/s-114MiB/s (120MB/s-120MB/s) > WRITE: bw=3D449MiB/s (471MB/s), 112MiB/s-113MiB/s (118MB/s-118MB/s) > WRITE: bw=3D459MiB/s (482MB/s), 115MiB/s-115MiB/s (120MB/s-121MB/s) Here's my patch to convert cifs to use write_cache_pages(). David --- fs/cifs/file.c | 400 ++++++++++++++++------------------------------------= ----- 1 file changed, 115 insertions(+), 285 deletions(-) diff --git a/fs/cifs/file.c b/fs/cifs/file.c index 3d304d4a54d6..04e2466609d9 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -2613,140 +2613,35 @@ static int cifs_partialpagewrite(struct page *page= , unsigned from, unsigned to) return rc; } =20 -/* - * Extend the region to be written back to include subsequent contiguously - * dirty pages if possible, but don't sleep while doing so. - */ -static void cifs_extend_writeback(struct address_space *mapping, - long *_count, - loff_t start, - int max_pages, - size_t max_len, - unsigned int *_len) -{ - struct folio_batch batch; - struct folio *folio; - unsigned int psize, nr_pages; - size_t len =3D *_len; - pgoff_t index =3D (start + len) / PAGE_SIZE; - bool stop =3D true; - unsigned int i; - XA_STATE(xas, &mapping->i_pages, index); - - folio_batch_init(&batch); - - do { - /* Firstly, we gather up a batch of contiguous dirty pages - * under the RCU read lock - but we can't clear the dirty flags - * there if any of those pages are mapped. - */ - rcu_read_lock(); - - xas_for_each(&xas, folio, ULONG_MAX) { - stop =3D true; - if (xas_retry(&xas, folio)) - continue; - if (xa_is_value(folio)) - break; - if (folio_index(folio) !=3D index) - break; - if (!folio_try_get_rcu(folio)) { - xas_reset(&xas); - continue; - } - nr_pages =3D folio_nr_pages(folio); - if (nr_pages > max_pages) - break; - - /* Has the page moved or been split? */ - if (unlikely(folio !=3D xas_reload(&xas))) { - folio_put(folio); - break; - } - - if (!folio_trylock(folio)) { - folio_put(folio); - break; - } - if (!folio_test_dirty(folio) || folio_test_writeback(folio)) { - folio_unlock(folio); - folio_put(folio); - break; - } - - psize =3D folio_size(folio); - len +=3D psize; - stop =3D false; - if (max_pages <=3D 0 || len >=3D max_len || *_count <=3D 0) - stop =3D true; - - index +=3D nr_pages; - if (!folio_batch_add(&batch, folio)) - break; - if (stop) - break; - } - - if (!stop) - xas_pause(&xas); - rcu_read_unlock(); - - /* Now, if we obtained any pages, we can shift them to being - * writable and mark them for caching. - */ - if (!folio_batch_count(&batch)) - break; - - for (i =3D 0; i < folio_batch_count(&batch); i++) { - folio =3D batch.folios[i]; - /* The folio should be locked, dirty and not undergoing - * writeback from the loop above. - */ - if (!folio_clear_dirty_for_io(folio)) - WARN_ON(1); - if (folio_start_writeback(folio)) - WARN_ON(1); - - *_count -=3D folio_nr_pages(folio); - folio_unlock(folio); - } - - folio_batch_release(&batch); - cond_resched(); - } while (!stop); - - *_len =3D len; -} +struct cifs_writepages_context { + struct cifs_writedata *wdata; + struct TCP_Server_Info *server; + struct cifs_credits credits; + unsigned long long start; + size_t len; + size_t wsize; + unsigned int xid; + bool begun; + bool caching; +}; =20 /* - * Write back the locked page and any subsequent non-locked dirty pages. + * Set up a writeback op. */ -static ssize_t cifs_write_back_from_locked_folio(struct address_space *map= ping, - struct writeback_control *wbc, - struct folio *folio, - loff_t start, loff_t end) +static int cifs_writeback_begin(struct address_space *mapping, + struct writeback_control *wbc, + unsigned long long start, + struct cifs_writepages_context *ctx) { struct inode *inode =3D mapping->host; struct TCP_Server_Info *server; struct cifs_writedata *wdata; struct cifs_sb_info *cifs_sb =3D CIFS_SB(inode->i_sb); - struct cifs_credits credits_on_stack; - struct cifs_credits *credits =3D &credits_on_stack; struct cifsFileInfo *cfile =3D NULL; - unsigned int xid, wsize, len; - loff_t i_size =3D i_size_read(inode); - size_t max_len; - long count =3D wbc->nr_to_write; + unsigned int wsize, len; int rc; =20 - /* The folio should be locked, dirty and not undergoing writeback. */ - if (folio_start_writeback(folio)) - WARN_ON(1); - - count -=3D folio_nr_pages(folio); - len =3D folio_size(folio); - - xid =3D get_xid(); + ctx->xid =3D get_xid(); server =3D cifs_pick_channel(cifs_sb_master_tcon(cifs_sb)->ses); =20 rc =3D cifs_get_writable_file(CIFS_I(inode), FIND_WR_ANY, &cfile); @@ -2756,7 +2651,7 @@ static ssize_t cifs_write_back_from_locked_folio(stru= ct address_space *mapping, } =20 rc =3D server->ops->wait_mtu_credits(server, cifs_sb->ctx->wsize, - &wsize, credits); + &wsize, &ctx->credits); if (rc !=3D 0) goto err_close; =20 @@ -2767,56 +2662,60 @@ static ssize_t cifs_write_back_from_locked_folio(st= ruct address_space *mapping, } =20 wdata->sync_mode =3D wbc->sync_mode; - wdata->offset =3D folio_pos(folio); + wdata->offset =3D start; wdata->pid =3D cfile->pid; - wdata->credits =3D credits_on_stack; + wdata->credits =3D ctx->credits; wdata->cfile =3D cfile; wdata->server =3D server; - cfile =3D NULL; - - /* Find all consecutive lockable dirty pages, stopping when we find a - * page that is not immediately lockable, is not dirty or is missing, - * or we reach the end of the range. - */ - if (start < i_size) { - /* Trim the write to the EOF; the extra data is ignored. Also - * put an upper limit on the size of a single storedata op. - */ - max_len =3D wsize; - max_len =3D min_t(unsigned long long, max_len, end - start + 1); - max_len =3D min_t(unsigned long long, max_len, i_size - start); - - if (len < max_len) { - int max_pages =3D INT_MAX; - -#ifdef CONFIG_CIFS_SMB_DIRECT - if (server->smbd_conn) - max_pages =3D server->smbd_conn->max_frmr_depth; -#endif - max_pages -=3D folio_nr_pages(folio); + ctx->wsize =3D wsize; + ctx->server =3D server; + ctx->wdata =3D wdata; + ctx->begun =3D true; + return 0; =20 - if (max_pages > 0) - cifs_extend_writeback(mapping, &count, start, - max_pages, max_len, &len); - } - len =3D min_t(loff_t, len, max_len); +err_uncredit: + add_credits_and_wake_if(server, &ctx->credits, 0); +err_close: + if (cfile) + cifsFileInfo_put(cfile); +err_xid: + free_xid(ctx->xid); + if (is_retryable_error(rc)) { + cifs_pages_write_redirty(inode, start, len); + } else { + cifs_pages_write_failed(inode, start, len); + mapping_set_error(mapping, rc); } + /* Indication to update ctime and mtime as close is deferred */ + set_bit(CIFS_INO_MODIFIED_ATTR, &CIFS_I(inode)->flags); + return rc; +} =20 - wdata->bytes =3D len; +/* + * Flush a block of pages to the server and the cache. + */ +static int cifs_writepages_submit(struct address_space *mapping, + struct writeback_control *wbc, + struct cifs_writepages_context *ctx) +{ + struct TCP_Server_Info *server =3D ctx->server; + struct cifs_writedata *wdata =3D ctx->wdata; + unsigned long long i_size =3D i_size_read(mapping->host); + int rc; =20 - /* We now have a contiguous set of dirty pages, each with writeback - * set; the first page is still locked at this point, but all the rest - * have been unlocked. + /* We now have a contiguous set of dirty pages, each with + * writeback set. */ - folio_unlock(folio); - - if (start < i_size) { - iov_iter_xarray(&wdata->iter, ITER_SOURCE, &mapping->i_pages, - start, len); + if (ctx->start < i_size) { + if (ctx->len > i_size - ctx->start) + ctx->len =3D i_size - ctx->start; + wdata->bytes =3D ctx->len; + iov_iter_xarray(&wdata->iter, ITER_SOURCE, + &mapping->i_pages, ctx->start, wdata->bytes); =20 rc =3D adjust_credits(wdata->server, &wdata->credits, wdata->bytes); if (rc) - goto err_wdata; + goto err; =20 if (wdata->cfile->invalidHandle) rc =3D -EAGAIN; @@ -2827,133 +2726,79 @@ static ssize_t cifs_write_back_from_locked_folio(s= truct address_space *mapping, kref_put(&wdata->refcount, cifs_writedata_release); goto err_close; } + } else { /* The dirty region was entirely beyond the EOF. */ - cifs_pages_written_back(inode, start, len); + cifs_pages_written_back(mapping->host, ctx->start, ctx->len); rc =3D 0; } =20 -err_wdata: +err: kref_put(&wdata->refcount, cifs_writedata_release); -err_uncredit: - add_credits_and_wake_if(server, credits, 0); + add_credits_and_wake_if(server, &ctx->credits, 0); err_close: - if (cfile) - cifsFileInfo_put(cfile); -err_xid: - free_xid(xid); + free_xid(ctx->xid); if (rc =3D=3D 0) { - wbc->nr_to_write =3D count; - rc =3D len; + rc =3D 0; } else if (is_retryable_error(rc)) { - cifs_pages_write_redirty(inode, start, len); + cifs_pages_write_redirty(mapping->host, ctx->start, ctx->len); } else { - cifs_pages_write_failed(inode, start, len); + cifs_pages_write_failed(mapping->host, ctx->start, ctx->len); mapping_set_error(mapping, rc); } + /* Indication to update ctime and mtime as close is deferred */ - set_bit(CIFS_INO_MODIFIED_ATTR, &CIFS_I(inode)->flags); + set_bit(CIFS_INO_MODIFIED_ATTR, &CIFS_I(mapping->host)->flags); + ctx->wdata =3D NULL; + ctx->begun =3D false; return rc; } =20 /* - * write a region of pages back to the server + * Add a page to the set and flush when large enough. */ -static int cifs_writepages_region(struct address_space *mapping, - struct writeback_control *wbc, - loff_t start, loff_t end, loff_t *_next) +static int cifs_writepages_add_folio(struct folio *folio, + struct writeback_control *wbc, void *data) { - struct folio_batch fbatch; - int skips =3D 0; - - folio_batch_init(&fbatch); - do { - int nr; - pgoff_t index =3D start / PAGE_SIZE; - - nr =3D filemap_get_folios_tag(mapping, &index, end / PAGE_SIZE, - PAGECACHE_TAG_DIRTY, &fbatch); - if (!nr) - break; - - for (int i =3D 0; i < nr; i++) { - ssize_t ret; - struct folio *folio =3D fbatch.folios[i]; - -redo_folio: - start =3D folio_pos(folio); /* May regress with THPs */ - - /* At this point we hold neither the i_pages lock nor the - * page lock: the page may be truncated or invalidated - * (changing page->mapping to NULL), or even swizzled - * back from swapper_space to tmpfs file mapping - */ - if (wbc->sync_mode !=3D WB_SYNC_NONE) { - ret =3D folio_lock_killable(folio); - if (ret < 0) - goto write_error; - } else { - if (!folio_trylock(folio)) - goto skip_write; - } - - if (folio_mapping(folio) !=3D mapping || - !folio_test_dirty(folio)) { - start +=3D folio_size(folio); - folio_unlock(folio); - continue; - } - - if (folio_test_writeback(folio) || - folio_test_fscache(folio)) { - folio_unlock(folio); - if (wbc->sync_mode =3D=3D WB_SYNC_NONE) - goto skip_write; - - folio_wait_writeback(folio); -#ifdef CONFIG_CIFS_FSCACHE - folio_wait_fscache(folio); -#endif - goto redo_folio; - } - - if (!folio_clear_dirty_for_io(folio)) - /* We hold the page lock - it should've been dirty. */ - WARN_ON(1); + struct cifs_writepages_context *ctx =3D data; + unsigned long long i_size =3D i_size_read(folio->mapping->host); + unsigned long long pos =3D folio_pos(folio); + size_t size =3D folio_size(folio); + int ret; =20 - ret =3D cifs_write_back_from_locked_folio(mapping, wbc, folio, start, e= nd); - if (ret < 0) - goto write_error; + if (pos < i_size && size > i_size - pos) + size =3D i_size - pos; =20 - start +=3D ret; - continue; - -write_error: - folio_batch_release(&fbatch); - *_next =3D start; + if (ctx->begun) { + if (pos =3D=3D ctx->start + ctx->len && + ctx->len + size <=3D ctx->wsize) + goto add; + ret =3D cifs_writepages_submit(folio->mapping, wbc, ctx); + if (ret < 0) { + ctx->begun =3D false; return ret; + } + } =20 -skip_write: - /* - * Too many skipped writes, or need to reschedule? - * Treat it as a write error without an error code. - */ - if (skips >=3D 5 || need_resched()) { - ret =3D 0; - goto write_error; - } + ret =3D cifs_writeback_begin(folio->mapping, wbc, pos, ctx); + if (ret < 0) + return ret; =20 - /* Otherwise, just skip that folio and go on to the next */ - skips++; - start +=3D folio_size(folio); - continue; - } + ctx->start =3D folio_pos(folio); + ctx->len =3D 0; +add: + ctx->len +=3D folio_size(folio); =20 - folio_batch_release(&fbatch); =09 - cond_resched(); - } while (wbc->nr_to_write > 0); + folio_wait_fscache(folio); + folio_start_writeback(folio); + folio_unlock(folio); =20 - *_next =3D start; + if (ctx->len >=3D ctx->wsize) { + ret =3D cifs_writepages_submit(folio->mapping, wbc, ctx); + if (ret < 0) + return ret; + ctx->begun =3D false; + } return 0; } =20 @@ -2963,7 +2808,7 @@ static int cifs_writepages_region(struct address_spac= e *mapping, static int cifs_writepages(struct address_space *mapping, struct writeback_control *wbc) { - loff_t start, next; + struct cifs_writepages_context ctx =3D {}; int ret; =20 /* We have to be careful as we can end up racing with setattr() @@ -2971,26 +2816,11 @@ static int cifs_writepages(struct address_space *ma= pping, * to prevent it. */ =20 - if (wbc->range_cyclic) { - start =3D mapping->writeback_index * PAGE_SIZE; - ret =3D cifs_writepages_region(mapping, wbc, start, LLONG_MAX, &next); - if (ret =3D=3D 0) { - mapping->writeback_index =3D next / PAGE_SIZE; - if (start > 0 && wbc->nr_to_write > 0) { - ret =3D cifs_writepages_region(mapping, wbc, 0, - start, &next); - if (ret =3D=3D 0) - mapping->writeback_index =3D - next / PAGE_SIZE; - } - } - } else if (wbc->range_start =3D=3D 0 && wbc->range_end =3D=3D LLONG_MAX) { - ret =3D cifs_writepages_region(mapping, wbc, 0, LLONG_MAX, &next); - if (wbc->nr_to_write > 0 && ret =3D=3D 0) - mapping->writeback_index =3D next / PAGE_SIZE; - } else { - ret =3D cifs_writepages_region(mapping, wbc, - wbc->range_start, wbc->range_end, &next); + ret =3D write_cache_pages(mapping, wbc, cifs_writepages_add_folio, &ctx); + if (ret >=3D 0 && ctx.begun) { + ret =3D cifs_writepages_submit(mapping, wbc, &ctx); + if (ret < 0) + return ret; } =20 return ret; From nobody Sat Apr 11 07:03:11 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E08AC678D4 for ; Thu, 2 Mar 2023 23:24:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229760AbjCBXYh (ORCPT ); Thu, 2 Mar 2023 18:24:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229615AbjCBXYe (ORCPT ); Thu, 2 Mar 2023 18:24:34 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8139E1689A for ; Thu, 2 Mar 2023 15:23:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677799427; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lnx3/NVhevje7xRs6jQfMiw8eOmAu/z4kjbV7YxLpFU=; b=K7wBrIGHXTNnFSKQN133ZGG7edabcBDAfPqewq1QjsUyvzqnKWamiZ4H46ZSmOitRnxpfZ sjPHWEfzMZ0fvsEpUBMGJApJL5fPWBDfBk0/FSnLkqv/VyFiwR9QBrBc1ynx54+ZpzSd46 6veVwvWkPQYLiCBzT3evUmHgke6IAZM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-164-LSj61p7LPEmXn6HCBRDYYg-1; Thu, 02 Mar 2023 18:23:44 -0500 X-MC-Unique: LSj61p7LPEmXn6HCBRDYYg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C2CE3811E6E; Thu, 2 Mar 2023 23:23:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.18]) by smtp.corp.redhat.com (Postfix) with ESMTP id E03432166B26; Thu, 2 Mar 2023 23:23:41 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 From: David Howells In-Reply-To: <20230302231638.521280-1-dhowells@redhat.com> References: <20230302231638.521280-1-dhowells@redhat.com> To: Linus Torvalds , Steve French Cc: dhowells@redhat.com, Vishal Moola , Shyam Prasad N , Rohith Surabattula , Tom Talpey , Stefan Metzmacher , Paulo Alcantara , Jeff Layton , Matthew Wilcox , Marc Dionne , linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Test patch to remove per-page dirty region tracking from afs MIME-Version: 1.0 Content-ID: <521670.1677799421.1@warthog.procyon.org.uk> Content-Transfer-Encoding: quoted-printable Date: Thu, 02 Mar 2023 23:23:41 +0000 Message-ID: <521671.1677799421@warthog.procyon.org.uk> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" David Howells wrote: > AFS firstly. ... >=20 > Base + Page-dirty-region removed: > WRITE: bw=3D301MiB/s (315MB/s), 70.4MiB/s-80.2MiB/s (73.8MB/s-84.1MB/s) > WRITE: bw=3D325MiB/s (341MB/s), 78.5MiB/s-87.1MiB/s (82.3MB/s-91.3MB/s) > WRITE: bw=3D320MiB/s (335MB/s), 71.6MiB/s-88.6MiB/s (75.0MB/s-92.9MB/s) Here's a patch to remove the use of page->private data to track the dirty p= art of a page from afs. David --- fs/afs/file.c | 68 ---------------- fs/afs/internal.h | 56 ------------- fs/afs/write.c | 187 +++++++++-------------------------------= ----- fs/cifs/file.c | 1=20 include/trace/events/afs.h | 14 --- 5 files changed, 45 insertions(+), 281 deletions(-) diff --git a/fs/afs/file.c b/fs/afs/file.c index 68d6d5dc608d..a2f3316fa174 100644 --- a/fs/afs/file.c +++ b/fs/afs/file.c @@ -402,80 +402,18 @@ int afs_write_inode(struct inode *inode, struct write= back_control *wbc) return 0; } =20 -/* - * Adjust the dirty region of the page on truncation or full invalidation, - * getting rid of the markers altogether if the region is entirely invalid= ated. - */ -static void afs_invalidate_dirty(struct folio *folio, size_t offset, - size_t length) -{ - struct afs_vnode *vnode =3D AFS_FS_I(folio_inode(folio)); - unsigned long priv; - unsigned int f, t, end =3D offset + length; - - priv =3D (unsigned long)folio_get_private(folio); - - /* we clean up only if the entire page is being invalidated */ - if (offset =3D=3D 0 && length =3D=3D folio_size(folio)) - goto full_invalidate; - - /* If the page was dirtied by page_mkwrite(), the PTE stays writable - * and we don't get another notification to tell us to expand it - * again. - */ - if (afs_is_folio_dirty_mmapped(priv)) - return; - - /* We may need to shorten the dirty region */ - f =3D afs_folio_dirty_from(folio, priv); - t =3D afs_folio_dirty_to(folio, priv); - - if (t <=3D offset || f >=3D end) - return; /* Doesn't overlap */ - - if (f < offset && t > end) - return; /* Splits the dirty region - just absorb it */ - - if (f >=3D offset && t <=3D end) - goto undirty; - - if (f < offset) - t =3D offset; - else - f =3D end; - if (f =3D=3D t) - goto undirty; - - priv =3D afs_folio_dirty(folio, f, t); - folio_change_private(folio, (void *)priv); - trace_afs_folio_dirty(vnode, tracepoint_string("trunc"), folio); - return; - -undirty: - trace_afs_folio_dirty(vnode, tracepoint_string("undirty"), folio); - folio_clear_dirty_for_io(folio); -full_invalidate: - trace_afs_folio_dirty(vnode, tracepoint_string("inval"), folio); - folio_detach_private(folio); -} - /* * invalidate part or all of a page * - release a page and clean up its private data if offset is 0 (indicati= ng * the entire page) */ static void afs_invalidate_folio(struct folio *folio, size_t offset, - size_t length) + size_t length) { - _enter("{%lu},%zu,%zu", folio->index, offset, length); - - BUG_ON(!folio_test_locked(folio)); - - if (folio_get_private(folio)) - afs_invalidate_dirty(folio, offset, length); + struct afs_vnode *vnode =3D AFS_FS_I(folio_inode(folio)); =20 + trace_afs_folio_dirty(vnode, tracepoint_string("inval"), folio); folio_wait_fscache(folio); - _leave(""); } =20 /* diff --git a/fs/afs/internal.h b/fs/afs/internal.h index ad8523d0d038..90d66b20ca8c 100644 --- a/fs/afs/internal.h +++ b/fs/afs/internal.h @@ -890,62 +890,6 @@ static inline void afs_invalidate_cache(struct afs_vno= de *vnode, unsigned int fl i_size_read(&vnode->netfs.inode), flags); } =20 -/* - * We use folio->private to hold the amount of the folio that we've writte= n to, - * splitting the field into two parts. However, we need to represent a ra= nge - * 0...FOLIO_SIZE, so we reduce the resolution if the size of the folio - * exceeds what we can encode. - */ -#ifdef CONFIG_64BIT -#define __AFS_FOLIO_PRIV_MASK 0x7fffffffUL -#define __AFS_FOLIO_PRIV_SHIFT 32 -#define __AFS_FOLIO_PRIV_MMAPPED 0x80000000UL -#else -#define __AFS_FOLIO_PRIV_MASK 0x7fffUL -#define __AFS_FOLIO_PRIV_SHIFT 16 -#define __AFS_FOLIO_PRIV_MMAPPED 0x8000UL -#endif - -static inline unsigned int afs_folio_dirty_resolution(struct folio *folio) -{ - int shift =3D folio_shift(folio) - (__AFS_FOLIO_PRIV_SHIFT - 1); - return (shift > 0) ? shift : 0; -} - -static inline size_t afs_folio_dirty_from(struct folio *folio, unsigned lo= ng priv) -{ - unsigned long x =3D priv & __AFS_FOLIO_PRIV_MASK; - - /* The lower bound is inclusive */ - return x << afs_folio_dirty_resolution(folio); -} - -static inline size_t afs_folio_dirty_to(struct folio *folio, unsigned long= priv) -{ - unsigned long x =3D (priv >> __AFS_FOLIO_PRIV_SHIFT) & __AFS_FOLIO_PRIV_M= ASK; - - /* The upper bound is immediately beyond the region */ - return (x + 1) << afs_folio_dirty_resolution(folio); -} - -static inline unsigned long afs_folio_dirty(struct folio *folio, size_t fr= om, size_t to) -{ - unsigned int res =3D afs_folio_dirty_resolution(folio); - from >>=3D res; - to =3D (to - 1) >> res; - return (to << __AFS_FOLIO_PRIV_SHIFT) | from; -} - -static inline unsigned long afs_folio_dirty_mmapped(unsigned long priv) -{ - return priv | __AFS_FOLIO_PRIV_MMAPPED; -} - -static inline bool afs_is_folio_dirty_mmapped(unsigned long priv) -{ - return priv & __AFS_FOLIO_PRIV_MMAPPED; -} - #include =20 /*************************************************************************= ****/ diff --git a/fs/afs/write.c b/fs/afs/write.c index 571f3b9a417e..d2f6623c8eab 100644 --- a/fs/afs/write.c +++ b/fs/afs/write.c @@ -14,11 +14,6 @@ #include #include "internal.h" =20 -static int afs_writepages_region(struct address_space *mapping, - struct writeback_control *wbc, - loff_t start, loff_t end, loff_t *_next, - bool max_one_loop); - static void afs_write_to_cache(struct afs_vnode *vnode, loff_t start, size= _t len, loff_t i_size, bool caching); =20 @@ -43,25 +38,6 @@ static void afs_folio_start_fscache(bool caching, struct= folio *folio) } #endif =20 -/* - * Flush out a conflicting write. This may extend the write to the surrou= nding - * pages if also dirty and contiguous to the conflicting region.. - */ -static int afs_flush_conflicting_write(struct address_space *mapping, - struct folio *folio) -{ - struct writeback_control wbc =3D { - .sync_mode =3D WB_SYNC_ALL, - .nr_to_write =3D LONG_MAX, - .range_start =3D folio_pos(folio), - .range_end =3D LLONG_MAX, - }; - loff_t next; - - return afs_writepages_region(mapping, &wbc, folio_pos(folio), LLONG_MAX, - &next, true); -} - /* * prepare to perform part of a write to a page */ @@ -71,10 +47,6 @@ int afs_write_begin(struct file *file, struct address_sp= ace *mapping, { struct afs_vnode *vnode =3D AFS_FS_I(file_inode(file)); struct folio *folio; - unsigned long priv; - unsigned f, from; - unsigned t, to; - pgoff_t index; int ret; =20 _enter("{%llx:%llu},%llx,%x", @@ -88,49 +60,17 @@ int afs_write_begin(struct file *file, struct address_s= pace *mapping, if (ret < 0) return ret; =20 - index =3D folio_index(folio); - from =3D pos - index * PAGE_SIZE; - to =3D from + len; - try_again: - /* See if this page is already partially written in a way that we can - * merge the new write with. - */ - if (folio_test_private(folio)) { - priv =3D (unsigned long)folio_get_private(folio); - f =3D afs_folio_dirty_from(folio, priv); - t =3D afs_folio_dirty_to(folio, priv); - ASSERTCMP(f, <=3D, t); - - if (folio_test_writeback(folio)) { - trace_afs_folio_dirty(vnode, tracepoint_string("alrdy"), folio); - folio_unlock(folio); - goto wait_for_writeback; - } - /* If the file is being filled locally, allow inter-write - * spaces to be merged into writes. If it's not, only write - * back what the user gives us. - */ - if (!test_bit(AFS_VNODE_NEW_CONTENT, &vnode->flags) && - (to < f || from > t)) - goto flush_conflicting_write; + if (folio_test_writeback(folio)) { + trace_afs_folio_dirty(vnode, tracepoint_string("alrdy"), folio); + folio_unlock(folio); + goto wait_for_writeback; } =20 *_page =3D folio_file_page(folio, pos / PAGE_SIZE); _leave(" =3D 0"); return 0; =20 - /* The previous write and this write aren't adjacent or overlapping, so - * flush the page out. - */ -flush_conflicting_write: - trace_afs_folio_dirty(vnode, tracepoint_string("confl"), folio); - folio_unlock(folio); - - ret =3D afs_flush_conflicting_write(mapping, folio); - if (ret < 0) - goto error; - wait_for_writeback: ret =3D folio_wait_writeback_killable(folio); if (ret < 0) @@ -156,9 +96,6 @@ int afs_write_end(struct file *file, struct address_spac= e *mapping, { struct folio *folio =3D page_folio(subpage); struct afs_vnode *vnode =3D AFS_FS_I(file_inode(file)); - unsigned long priv; - unsigned int f, from =3D offset_in_folio(folio, pos); - unsigned int t, to =3D from + copied; loff_t i_size, write_end_pos; =20 _enter("{%llx:%llu},{%lx}", @@ -188,25 +125,10 @@ int afs_write_end(struct file *file, struct address_s= pace *mapping, fscache_update_cookie(afs_vnode_cache(vnode), NULL, &write_end_pos); } =20 - if (folio_test_private(folio)) { - priv =3D (unsigned long)folio_get_private(folio); - f =3D afs_folio_dirty_from(folio, priv); - t =3D afs_folio_dirty_to(folio, priv); - if (from < f) - f =3D from; - if (to > t) - t =3D to; - priv =3D afs_folio_dirty(folio, f, t); - folio_change_private(folio, (void *)priv); - trace_afs_folio_dirty(vnode, tracepoint_string("dirty+"), folio); - } else { - priv =3D afs_folio_dirty(folio, from, to); - folio_attach_private(folio, (void *)priv); - trace_afs_folio_dirty(vnode, tracepoint_string("dirty"), folio); - } - if (folio_mark_dirty(folio)) - _debug("dirtied %lx", folio_index(folio)); + trace_afs_folio_dirty(vnode, tracepoint_string("dirty"), folio); + else + trace_afs_folio_dirty(vnode, tracepoint_string("dirty+"), folio); =20 out: folio_unlock(folio); @@ -465,18 +387,16 @@ static void afs_extend_writeback(struct address_space= *mapping, bool caching, unsigned int *_len) { - struct pagevec pvec; + struct folio_batch batch; struct folio *folio; - unsigned long priv; - unsigned int psize, filler =3D 0; - unsigned int f, t; + size_t psize; loff_t len =3D *_len; pgoff_t index =3D (start + len) / PAGE_SIZE; bool stop =3D true; unsigned int i; - XA_STATE(xas, &mapping->i_pages, index); - pagevec_init(&pvec); + + folio_batch_init(&batch); =20 do { /* Firstly, we gather up a batch of contiguous dirty pages @@ -493,7 +413,6 @@ static void afs_extend_writeback(struct address_space *= mapping, break; if (folio_index(folio) !=3D index) break; - if (!folio_try_get_rcu(folio)) { xas_reset(&xas); continue; @@ -518,24 +437,13 @@ static void afs_extend_writeback(struct address_space= *mapping, } =20 psize =3D folio_size(folio); - priv =3D (unsigned long)folio_get_private(folio); - f =3D afs_folio_dirty_from(folio, priv); - t =3D afs_folio_dirty_to(folio, priv); - if (f !=3D 0 && !new_content) { - folio_unlock(folio); - folio_put(folio); - break; - } - - len +=3D filler + t; - filler =3D psize - t; + len +=3D psize; + stop =3D false; if (len >=3D max_len || *_count <=3D 0) stop =3D true; - else if (t =3D=3D psize || new_content) - stop =3D false; =20 index +=3D folio_nr_pages(folio); - if (!pagevec_add(&pvec, &folio->page)) + if (!folio_batch_add(&batch, folio)) break; if (stop) break; @@ -548,11 +456,11 @@ static void afs_extend_writeback(struct address_space= *mapping, /* Now, if we obtained any pages, we can shift them to being * writable and mark them for caching. */ - if (!pagevec_count(&pvec)) + if (!folio_batch_count(&batch)) break; =20 - for (i =3D 0; i < pagevec_count(&pvec); i++) { - folio =3D page_folio(pvec.pages[i]); + for (i =3D 0; i < folio_batch_count(&batch); i++) { + folio =3D batch.folios[i]; trace_afs_folio_dirty(vnode, tracepoint_string("store+"), folio); =20 if (!folio_clear_dirty_for_io(folio)) @@ -565,7 +473,7 @@ static void afs_extend_writeback(struct address_space *= mapping, folio_unlock(folio); } =20 - pagevec_release(&pvec); + folio_batch_release(&batch); cond_resched(); } while (!stop); =20 @@ -583,8 +491,7 @@ static ssize_t afs_write_back_from_locked_folio(struct = address_space *mapping, { struct afs_vnode *vnode =3D AFS_FS_I(mapping->host); struct iov_iter iter; - unsigned long priv; - unsigned int offset, to, len, max_len; + unsigned int len, max_len; loff_t i_size =3D i_size_read(&vnode->netfs.inode); bool new_content =3D test_bit(AFS_VNODE_NEW_CONTENT, &vnode->flags); bool caching =3D fscache_cookie_enabled(afs_vnode_cache(vnode)); @@ -599,18 +506,14 @@ static ssize_t afs_write_back_from_locked_folio(struc= t address_space *mapping, =20 count -=3D folio_nr_pages(folio); =20 - /* Find all consecutive lockable dirty pages that have contiguous - * written regions, stopping when we find a page that is not - * immediately lockable, is not dirty or is missing, or we reach the - * end of the range. + /* Find all consecutive lockable dirty pages, stopping when we find a + * page that is not immediately lockable, is not dirty or is missing, + * or we reach the end of the range. */ - priv =3D (unsigned long)folio_get_private(folio); - offset =3D afs_folio_dirty_from(folio, priv); - to =3D afs_folio_dirty_to(folio, priv); trace_afs_folio_dirty(vnode, tracepoint_string("store"), folio); =20 - len =3D to - offset; - start +=3D offset; + len =3D folio_size(folio); + start =3D folio_pos(folio); if (start < i_size) { /* Trim the write to the EOF; the extra data is ignored. Also * put an upper limit on the size of a single storedata op. @@ -619,8 +522,7 @@ static ssize_t afs_write_back_from_locked_folio(struct = address_space *mapping, max_len =3D min_t(unsigned long long, max_len, end - start + 1); max_len =3D min_t(unsigned long long, max_len, i_size - start); =20 - if (len < max_len && - (to =3D=3D folio_size(folio) || new_content)) + if (len < max_len) afs_extend_writeback(mapping, vnode, &count, start, max_len, new_content, caching, &len); @@ -909,7 +811,6 @@ vm_fault_t afs_page_mkwrite(struct vm_fault *vmf) struct inode *inode =3D file_inode(file); struct afs_vnode *vnode =3D AFS_FS_I(inode); struct afs_file *af =3D file->private_data; - unsigned long priv; vm_fault_t ret =3D VM_FAULT_RETRY; =20 _enter("{{%llx:%llu}},{%lx}", vnode->fid.vid, vnode->fid.vnode, folio_ind= ex(folio)); @@ -942,15 +843,7 @@ vm_fault_t afs_page_mkwrite(struct vm_fault *vmf) goto out; } =20 - priv =3D afs_folio_dirty(folio, 0, folio_size(folio)); - priv =3D afs_folio_dirty_mmapped(priv); - if (folio_test_private(folio)) { - folio_change_private(folio, (void *)priv); - trace_afs_folio_dirty(vnode, tracepoint_string("mkwrite+"), folio); - } else { - folio_attach_private(folio, (void *)priv); - trace_afs_folio_dirty(vnode, tracepoint_string("mkwrite"), folio); - } + trace_afs_folio_dirty(vnode, tracepoint_string("mkwrite"), folio); file_update_time(file); =20 ret =3D VM_FAULT_LOCKED; @@ -992,33 +885,33 @@ void afs_prune_wb_keys(struct afs_vnode *vnode) */ int afs_launder_folio(struct folio *folio) { - struct afs_vnode *vnode =3D AFS_FS_I(folio_inode(folio)); + struct inode *inode =3D folio_inode(folio); + struct afs_vnode *vnode =3D AFS_FS_I(inode); struct iov_iter iter; struct bio_vec bv; - unsigned long priv; - unsigned int f, t; int ret =3D 0; =20 _enter("{%lx}", folio->index); =20 - priv =3D (unsigned long)folio_get_private(folio); if (folio_clear_dirty_for_io(folio)) { - f =3D 0; - t =3D folio_size(folio); - if (folio_test_private(folio)) { - f =3D afs_folio_dirty_from(folio, priv); - t =3D afs_folio_dirty_to(folio, priv); - } + unsigned long long i_size =3D i_size_read(inode); + unsigned long long pos =3D folio_pos(folio); + size_t size =3D folio_size(folio); =20 - bvec_set_folio(&bv, folio, t - f, f); - iov_iter_bvec(&iter, ITER_SOURCE, &bv, 1, bv.bv_len); + if (pos >=3D i_size) + goto out; + if (i_size - pos < size) + size =3D i_size - pos; + =09 + bvec_set_folio(&bv, folio, size, 0); + iov_iter_bvec(&iter, ITER_SOURCE, &bv, 1, size); =20 trace_afs_folio_dirty(vnode, tracepoint_string("launder"), folio); - ret =3D afs_store_data(vnode, &iter, folio_pos(folio) + f, true); + ret =3D afs_store_data(vnode, &iter, pos, true); } =20 +out: trace_afs_folio_dirty(vnode, tracepoint_string("laundered"), folio); - folio_detach_private(folio); folio_wait_fscache(folio); return ret; } diff --git a/fs/cifs/file.c b/fs/cifs/file.c index 4d4a2d82636d..3d304d4a54d6 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -2674,7 +2674,6 @@ static void cifs_extend_writeback(struct address_spac= e *mapping, break; } =20 - max_pages -=3D nr_pages; psize =3D folio_size(folio); len +=3D psize; stop =3D false; diff --git a/include/trace/events/afs.h b/include/trace/events/afs.h index e9d412d19dbb..4540aa801edd 100644 --- a/include/trace/events/afs.h +++ b/include/trace/events/afs.h @@ -1025,26 +1025,16 @@ TRACE_EVENT(afs_folio_dirty, __field(struct afs_vnode *, vnode ) __field(const char *, where ) __field(pgoff_t, index ) - __field(unsigned long, from ) - __field(unsigned long, to ) ), =20 TP_fast_assign( - unsigned long priv =3D (unsigned long)folio_get_private(folio); __entry->vnode =3D vnode; __entry->where =3D where; __entry->index =3D folio_index(folio); - __entry->from =3D afs_folio_dirty_from(folio, priv); - __entry->to =3D afs_folio_dirty_to(folio, priv); - __entry->to |=3D (afs_is_folio_dirty_mmapped(priv) ? - (1UL << (BITS_PER_LONG - 1)) : 0); ), =20 - TP_printk("vn=3D%p %lx %s %lx-%lx%s", - __entry->vnode, __entry->index, __entry->where, - __entry->from, - __entry->to & ~(1UL << (BITS_PER_LONG - 1)), - __entry->to & (1UL << (BITS_PER_LONG - 1)) ? " M" : "") + TP_printk("vn=3D%p %lx %s", + __entry->vnode, __entry->index, __entry->where) ); =20 TRACE_EVENT(afs_call_state,