From: Himanshu Madhani <himanshu.madhani@oracle.com>
Add RWF_ATOMIC flag description for pwritev2().
Signed-off-by: Himanshu Madhani <himanshu.madhani@oracle.com>
#jpg: complete rewrite
Signed-off-by: John Garry <john.g.garry@oracle.com>
---
man2/readv.2 | 45 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/man2/readv.2 b/man2/readv.2
index fa9b0e4e44a2..ff09f3bc9792 100644
--- a/man2/readv.2
+++ b/man2/readv.2
@@ -193,6 +193,51 @@ which provides lower latency, but may use additional resources.
.B O_DIRECT
flag.)
.TP
+.BR RWF_ATOMIC " (since Linux 6.7)"
+Allows block-based filesystems to indicate that write operations will be issued
+with torn-write protection. Torn-write protection means that for a power or any
+other hardware failure, all or none of the data from the write will be stored,
+but never a mix of old and new data. This flag is meaningful only for
+.BR pwritev2 (),
+and its effect applies only to the data range written by the system call.
+The total write length must be power-of-2 and must be sized between
+stx_atomic_write_unit_min and stx_atomic_write_unit_max, both inclusive. The
+write must be at a natural offset within the file with respect to the total
+write length. Torn-write protection only works with
+.B O_DIRECT
+flag, i.e. buffered writes are not supported. To guarantee consistency from
+the write between a file's in-core state with the storage device,
+.BR fdatasync (2)
+or
+.BR fsync (2)
+or
+.BR open (2)
+and
+.B O_SYNC
+or
+.B O_DSYNC
+or
+.B pwritev2 ()
+flag
+.B RWF_SYNC
+or
+.B RWF_DSYNC
+is required.
+For when regular files are opened with
+.BR open (2)
+but without
+.B O_SYNC
+or
+.B O_DSYNC
+and the
+.BR pwritev2()
+call is made without
+.B RWF_SYNC
+or
+.BR RWF_DSYNC
+set, the range metadata must already be flushed to storage and the data range
+must not be in unwritten state, shared, a preallocation, or a hole.
+.TP
.BR RWF_SYNC " (since Linux 4.7)"
.\" commit e864f39569f4092c2b2bc72c773b6e486c7e3bd9
Provide a per-write equivalent of the
--
2.31.1On Fri, Sep 29, 2023 at 09:37:15AM +0000, John Garry wrote: > From: Himanshu Madhani <himanshu.madhani@oracle.com> > > Add RWF_ATOMIC flag description for pwritev2(). > > Signed-off-by: Himanshu Madhani <himanshu.madhani@oracle.com> > #jpg: complete rewrite > Signed-off-by: John Garry <john.g.garry@oracle.com> > --- > man2/readv.2 | 45 +++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 45 insertions(+) > > diff --git a/man2/readv.2 b/man2/readv.2 > index fa9b0e4e44a2..ff09f3bc9792 100644 > --- a/man2/readv.2 > +++ b/man2/readv.2 > @@ -193,6 +193,51 @@ which provides lower latency, but may use additional resources. > .B O_DIRECT > flag.) > .TP > +.BR RWF_ATOMIC " (since Linux 6.7)" > +Allows block-based filesystems to indicate that write operations will be issued "Require regular file write operations to be issued with torn write protection." > +with torn-write protection. Torn-write protection means that for a power or any > +other hardware failure, all or none of the data from the write will be stored, > +but never a mix of old and new data. This flag is meaningful only for > +.BR pwritev2 (), > +and its effect applies only to the data range written by the system call. > +The total write length must be power-of-2 and must be sized between > +stx_atomic_write_unit_min and stx_atomic_write_unit_max, both inclusive. The > +write must be at a natural offset within the file with respect to the total What is a "natural" offset? That should be defined with more specificity. Does that mean that the position of a XX-KiB write must also be aligned to XX-KiB? e.g. a 32K untorn write can only start at a multiple of 32K? What if the device supports untorn writes between 4K and 64K, does that mean I /cannot/ issue a 32K untorn write at offset 48K? > +write length. Torn-write protection only works with > +.B O_DIRECT > +flag, i.e. buffered writes are not supported. To guarantee consistency from > +the write between a file's in-core state with the storage device, > +.BR fdatasync (2) > +or > +.BR fsync (2) > +or > +.BR open (2) > +and > +.B O_SYNC > +or > +.B O_DSYNC > +or > +.B pwritev2 () > +flag > +.B RWF_SYNC > +or > +.B RWF_DSYNC > +is required. I'm starting to think that this manpage shouldn't be restating durability information here. "Application programs with data or file integrity completion requirements must configure synchronous writes with the DSYNC or SYNC flags, as explained above." > +For when regular files are opened with > +.BR open (2) > +but without > +.B O_SYNC > +or > +.B O_DSYNC > +and the > +.BR pwritev2() > +call is made without > +.B RWF_SYNC > +or > +.BR RWF_DSYNC > +set, the range metadata must already be flushed to storage and the data range > +must not be in unwritten state, shared, a preallocation, or a hole. I think that we can drop all of these flags requirements, since the contiguous small space allocation requirement means that the fs can provide all-or-nothing writes even if metadata updates are needed: If the file range is allocated and marked unwritten (i.e. a preallocation), the ioend will clear the unwritten bit from the file mapping atomically. After a crash, the application sees either zeroes or all the data that was written. If the file range is shared, the ioend will map the COW staging extent into the file atomically. After a crash, the application sees either the old contents from the old blocks, or the new contents from the new blocks. If the file range is a sparse hole, the directio setup will allocate space and create an unwritten mapping before issuing the write bio. The rest of the process works the same as preallocations and has the same behaviors. If the file range is allocated and was previously written, the write is issued and that's all that's needed from the fs. After a crash, reads of the storage device produce the old contents or the new contents. Summarizing: An (ATOMIC|SYNC) request provides the strongest guarantees (data will not be torn, and all file metadata updates are persisted before the write is returned to userspace. Programs see either the old data or the new data, even if there's a crash. (ATOMIC|DSYNC) is less strong -- data will not be torn, and any file updates for just that region are persisted before the write is returned. (ATOMIC) is the least strong -- data will not be torn. Neither the filesystem nor the device make guarantees that anything ended up on stable storage, but if it does, programs see either the old data or the new data. Maybe we should rename the whole UAPI s/atomic/untorn/... --D > +.TP > .BR RWF_SYNC " (since Linux 4.7)" > .\" commit e864f39569f4092c2b2bc72c773b6e486c7e3bd9 > Provide a per-write equivalent of the > -- > 2.31.1 >
On 09/10/2023 18:44, Darrick J. Wong wrote: > On Fri, Sep 29, 2023 at 09:37:15AM +0000, John Garry wrote: >> From: Himanshu Madhani <himanshu.madhani@oracle.com> >> >> Add RWF_ATOMIC flag description for pwritev2(). >> >> Signed-off-by: Himanshu Madhani <himanshu.madhani@oracle.com> >> #jpg: complete rewrite >> Signed-off-by: John Garry <john.g.garry@oracle.com> >> --- >> man2/readv.2 | 45 +++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 45 insertions(+) >> >> diff --git a/man2/readv.2 b/man2/readv.2 >> index fa9b0e4e44a2..ff09f3bc9792 100644 >> --- a/man2/readv.2 >> +++ b/man2/readv.2 >> @@ -193,6 +193,51 @@ which provides lower latency, but may use additional resources. >> .B O_DIRECT >> flag.) >> .TP >> +.BR RWF_ATOMIC " (since Linux 6.7)" >> +Allows block-based filesystems to indicate that write operations will be issued > > "Require regular file write operations to be issued with torn write > protection." ok > >> +with torn-write protection. Torn-write protection means that for a power or any >> +other hardware failure, all or none of the data from the write will be stored, >> +but never a mix of old and new data. This flag is meaningful only for >> +.BR pwritev2 (), >> +and its effect applies only to the data range written by the system call. >> +The total write length must be power-of-2 and must be sized between >> +stx_atomic_write_unit_min and stx_atomic_write_unit_max, both inclusive. The >> +write must be at a natural offset within the file with respect to the total > > What is a "natural" offset? I really meant naturally-aligned offset > That should be defined with more > specificity. Does that mean that the position of a XX-KiB write must > also be aligned to XX-KiB? Yes > e.g. a 32K untorn write can only start at a > multiple of 32K? Correct > What if the device supports untorn writes between 4K > and 64K, does that mean I /cannot/ issue a 32K untorn write at offset > 48K? Correct Do you think that an example would help? > >> +write length. Torn-write protection only works with >> +.B O_DIRECT >> +flag, i.e. buffered writes are not supported. To guarantee consistency from >> +the write between a file's in-core state with the storage device, >> +.BR fdatasync (2) >> +or >> +.BR fsync (2) >> +or >> +.BR open (2) >> +and >> +.B O_SYNC >> +or >> +.B O_DSYNC >> +or >> +.B pwritev2 () >> +flag >> +.B RWF_SYNC >> +or >> +.B RWF_DSYNC >> +is required. > > I'm starting to think that this manpage shouldn't be restating > durability information here. > > "Application programs with data or file integrity completion > requirements must configure synchronous writes with the DSYNC > or SYNC flags, as explained above." ok > >> +For when regular files are opened with >> +.BR open (2) >> +but without >> +.B O_SYNC >> +or >> +.B O_DSYNC >> +and the >> +.BR pwritev2() >> +call is made without >> +.B RWF_SYNC >> +or >> +.BR RWF_DSYNC >> +set, the range metadata must already be flushed to storage and the data range >> +must not be in unwritten state, shared, a preallocation, or a hole. > > I think that we can drop all of these flags requirements, since the > contiguous small space allocation requirement means that the fs can > provide all-or-nothing writes even if metadata updates are needed: > > If the file range is allocated and marked unwritten (i.e. a > preallocation), the ioend will clear the unwritten bit from the file > mapping atomically. After a crash, the application sees either zeroes > or all the data that was written. > > If the file range is shared, the ioend will map the COW staging extent > into the file atomically. After a crash, the application sees either > the old contents from the old blocks, or the new contents from the new > blocks. > > If the file range is a sparse hole, the directio setup will allocate > space and create an unwritten mapping before issuing the write bio. The > rest of the process works the same as preallocations and has the same > behaviors. > > If the file range is allocated and was previously written, the write is > issued and that's all that's needed from the fs. After a crash, reads > of the storage device produce the old contents or the new contents. > > Summarizing: > > An (ATOMIC|SYNC) request provides the strongest guarantees (data > will not be torn, and all file metadata updates are persisted before > the write is returned to userspace. Programs see either the old data or > the new data, even if there's a crash. > > (ATOMIC|DSYNC) is less strong -- data will not be torn, and any file > updates for just that region are persisted before the write is returned. > > (ATOMIC) is the least strong -- data will not be torn. Neither the > filesystem nor the device make guarantees that anything ended up on > stable storage, but if it does, programs see either the old data or the > new data. > Will respond to later mail in thread. > Maybe we should rename the whole UAPI s/atomic/untorn/... > > --D > >> +.TP >> .BR RWF_SYNC " (since Linux 4.7)" >> .\" commit e864f39569f4092c2b2bc72c773b6e486c7e3bd9 >> Provide a per-write equivalent of the >> -- >> 2.31.1 >> Thanks, John
On Tue, Oct 24, 2023 at 01:30:03PM +0100, John Garry wrote: > On 09/10/2023 18:44, Darrick J. Wong wrote: > > On Fri, Sep 29, 2023 at 09:37:15AM +0000, John Garry wrote: > > > From: Himanshu Madhani <himanshu.madhani@oracle.com> > > > > > > Add RWF_ATOMIC flag description for pwritev2(). > > > > > > Signed-off-by: Himanshu Madhani <himanshu.madhani@oracle.com> > > > #jpg: complete rewrite > > > Signed-off-by: John Garry <john.g.garry@oracle.com> > > > --- > > > man2/readv.2 | 45 +++++++++++++++++++++++++++++++++++++++++++++ > > > 1 file changed, 45 insertions(+) > > > > > > diff --git a/man2/readv.2 b/man2/readv.2 > > > index fa9b0e4e44a2..ff09f3bc9792 100644 > > > --- a/man2/readv.2 > > > +++ b/man2/readv.2 > > > @@ -193,6 +193,51 @@ which provides lower latency, but may use additional resources. > > > .B O_DIRECT > > > flag.) > > > .TP > > > +.BR RWF_ATOMIC " (since Linux 6.7)" > > > +Allows block-based filesystems to indicate that write operations will be issued > > > > "Require regular file write operations to be issued with torn write > > protection." > > ok > > > > > > +with torn-write protection. Torn-write protection means that for a power or any > > > +other hardware failure, all or none of the data from the write will be stored, > > > +but never a mix of old and new data. This flag is meaningful only for > > > +.BR pwritev2 (), > > > +and its effect applies only to the data range written by the system call. > > > +The total write length must be power-of-2 and must be sized between > > > +stx_atomic_write_unit_min and stx_atomic_write_unit_max, both inclusive. The > > > +write must be at a natural offset within the file with respect to the total > > > > What is a "natural" offset? > > I really meant naturally-aligned offset > > > That should be defined with more > > specificity. Does that mean that the position of a XX-KiB write must > > also be aligned to XX-KiB? > > Yes > > > e.g. a 32K untorn write can only start at a > > multiple of 32K? > > Correct > > > What if the device supports untorn writes between 4K > > and 64K, does that mean I /cannot/ issue a 32K untorn write at offset > > 48K? > > Correct > > Do you think that an example would help? Yes. > > > > > +write length. Torn-write protection only works with > > > +.B O_DIRECT > > > +flag, i.e. buffered writes are not supported. To guarantee consistency from > > > +the write between a file's in-core state with the storage device, > > > +.BR fdatasync (2) > > > +or > > > +.BR fsync (2) > > > +or > > > +.BR open (2) > > > +and > > > +.B O_SYNC > > > +or > > > +.B O_DSYNC > > > +or > > > +.B pwritev2 () > > > +flag > > > +.B RWF_SYNC > > > +or > > > +.B RWF_DSYNC > > > +is required. > > > > I'm starting to think that this manpage shouldn't be restating > > durability information here. > > > > "Application programs with data or file integrity completion > > requirements must configure synchronous writes with the DSYNC > > or SYNC flags, as explained above." > > ok > > > > > > +For when regular files are opened with > > > +.BR open (2) > > > +but without > > > +.B O_SYNC > > > +or > > > +.B O_DSYNC > > > +and the > > > +.BR pwritev2() > > > +call is made without > > > +.B RWF_SYNC > > > +or > > > +.BR RWF_DSYNC > > > +set, the range metadata must already be flushed to storage and the data range > > > +must not be in unwritten state, shared, a preallocation, or a hole. > > > > I think that we can drop all of these flags requirements, since the > > contiguous small space allocation requirement means that the fs can > > provide all-or-nothing writes even if metadata updates are needed: > > > > If the file range is allocated and marked unwritten (i.e. a > > preallocation), the ioend will clear the unwritten bit from the file > > mapping atomically. After a crash, the application sees either zeroes > > or all the data that was written. > > > > If the file range is shared, the ioend will map the COW staging extent > > into the file atomically. After a crash, the application sees either > > the old contents from the old blocks, or the new contents from the new > > blocks. > > > > If the file range is a sparse hole, the directio setup will allocate > > space and create an unwritten mapping before issuing the write bio. The > > rest of the process works the same as preallocations and has the same > > behaviors. > > > > If the file range is allocated and was previously written, the write is > > issued and that's all that's needed from the fs. After a crash, reads > > of the storage device produce the old contents or the new contents. > > > > Summarizing: > > > > An (ATOMIC|SYNC) request provides the strongest guarantees (data > > will not be torn, and all file metadata updates are persisted before > > the write is returned to userspace. Programs see either the old data or > > the new data, even if there's a crash. > > > > (ATOMIC|DSYNC) is less strong -- data will not be torn, and any file > > updates for just that region are persisted before the write is returned. > > > > (ATOMIC) is the least strong -- data will not be torn. Neither the > > filesystem nor the device make guarantees that anything ended up on > > stable storage, but if it does, programs see either the old data or the > > new data. > > > > > Will respond to later mail in thread. Ok, thank you! --D > > Maybe we should rename the whole UAPI s/atomic/untorn/... > > > --D > > > > > +.TP > > > .BR RWF_SYNC " (since Linux 4.7)" > > > .\" commit e864f39569f4092c2b2bc72c773b6e486c7e3bd9 > > > Provide a per-write equivalent of the > > > -- > > > 2.31.1 > > > > > Thanks, > John >
On Mon, Oct 09, 2023 at 10:44:38AM -0700, Darrick J. Wong wrote: > On Fri, Sep 29, 2023 at 09:37:15AM +0000, John Garry wrote: > > From: Himanshu Madhani <himanshu.madhani@oracle.com> > > > > Add RWF_ATOMIC flag description for pwritev2(). > > > > Signed-off-by: Himanshu Madhani <himanshu.madhani@oracle.com> > > #jpg: complete rewrite > > Signed-off-by: John Garry <john.g.garry@oracle.com> > > --- > > man2/readv.2 | 45 +++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 45 insertions(+) .... > > +For when regular files are opened with > > +.BR open (2) > > +but without > > +.B O_SYNC > > +or > > +.B O_DSYNC > > +and the > > +.BR pwritev2() > > +call is made without > > +.B RWF_SYNC > > +or > > +.BR RWF_DSYNC > > +set, the range metadata must already be flushed to storage and the data range > > +must not be in unwritten state, shared, a preallocation, or a hole. > > I think that we can drop all of these flags requirements, since the > contiguous small space allocation requirement means that the fs can > provide all-or-nothing writes even if metadata updates are needed: > > If the file range is allocated and marked unwritten (i.e. a > preallocation), the ioend will clear the unwritten bit from the file > mapping atomically. After a crash, the application sees either zeroes > or all the data that was written. > > If the file range is shared, the ioend will map the COW staging extent > into the file atomically. After a crash, the application sees either > the old contents from the old blocks, or the new contents from the new > blocks. > > If the file range is a sparse hole, the directio setup will allocate > space and create an unwritten mapping before issuing the write bio. The > rest of the process works the same as preallocations and has the same > behaviors. > > If the file range is allocated and was previously written, the write is > issued and that's all that's needed from the fs. After a crash, reads > of the storage device produce the old contents or the new contents. This is exactly what I explained when reviewing the code that rejected RWF_ATOMIC without O_DSYNC on metadata dirty inodes. > Summarizing: > > An (ATOMIC|SYNC) request provides the strongest guarantees (data > will not be torn, and all file metadata updates are persisted before > the write is returned to userspace. Programs see either the old data or > the new data, even if there's a crash. > > (ATOMIC|DSYNC) is less strong -- data will not be torn, and any file > updates for just that region are persisted before the write is returned. > > (ATOMIC) is the least strong -- data will not be torn. Neither the > filesystem nor the device make guarantees that anything ended up on > stable storage, but if it does, programs see either the old data or the > new data. Yup, that makes sense to me. > Maybe we should rename the whole UAPI s/atomic/untorn/... Perhaps, though "torn writes" is nomenclature that nobody outside storage and filesystem developers really knows about. All I ever hear from userspace developers is "we want atomic/all-or-nothing data writes"... -Dave. -- Dave Chinner david@fromorbit.com
On Tue, Oct 10, 2023 at 07:39:17AM +1100, Dave Chinner wrote: > On Mon, Oct 09, 2023 at 10:44:38AM -0700, Darrick J. Wong wrote: > > On Fri, Sep 29, 2023 at 09:37:15AM +0000, John Garry wrote: > > > From: Himanshu Madhani <himanshu.madhani@oracle.com> > > > > > > Add RWF_ATOMIC flag description for pwritev2(). > > > > > > Signed-off-by: Himanshu Madhani <himanshu.madhani@oracle.com> > > > #jpg: complete rewrite > > > Signed-off-by: John Garry <john.g.garry@oracle.com> > > > --- > > > man2/readv.2 | 45 +++++++++++++++++++++++++++++++++++++++++++++ > > > 1 file changed, 45 insertions(+) > .... > > > +For when regular files are opened with > > > +.BR open (2) > > > +but without > > > +.B O_SYNC > > > +or > > > +.B O_DSYNC > > > +and the > > > +.BR pwritev2() > > > +call is made without > > > +.B RWF_SYNC > > > +or > > > +.BR RWF_DSYNC > > > +set, the range metadata must already be flushed to storage and the data range > > > +must not be in unwritten state, shared, a preallocation, or a hole. > > > > I think that we can drop all of these flags requirements, since the > > contiguous small space allocation requirement means that the fs can > > provide all-or-nothing writes even if metadata updates are needed: > > > > If the file range is allocated and marked unwritten (i.e. a > > preallocation), the ioend will clear the unwritten bit from the file > > mapping atomically. After a crash, the application sees either zeroes > > or all the data that was written. > > > > If the file range is shared, the ioend will map the COW staging extent > > into the file atomically. After a crash, the application sees either > > the old contents from the old blocks, or the new contents from the new > > blocks. > > > > If the file range is a sparse hole, the directio setup will allocate > > space and create an unwritten mapping before issuing the write bio. The > > rest of the process works the same as preallocations and has the same > > behaviors. > > > > If the file range is allocated and was previously written, the write is > > issued and that's all that's needed from the fs. After a crash, reads > > of the storage device produce the old contents or the new contents. > > This is exactly what I explained when reviewing the code that > rejected RWF_ATOMIC without O_DSYNC on metadata dirty inodes. I'm glad we agree. :) John, when you're back from vacation, can we get rid of this language and all those checks under _is_dsync() in the iomap patch? (That code is 100% the result of me handwaving and bellyaching 6 months ago when the team was trying to get all the atomic writes bits working prior to LSF and I was too burned out to think the xfs part through. As a result, I decided that we'd only support strict overwrites for the first iteration.) > > Summarizing: > > > > An (ATOMIC|SYNC) request provides the strongest guarantees (data > > will not be torn, and all file metadata updates are persisted before > > the write is returned to userspace. Programs see either the old data or > > the new data, even if there's a crash. > > > > (ATOMIC|DSYNC) is less strong -- data will not be torn, and any file > > updates for just that region are persisted before the write is returned. > > > > (ATOMIC) is the least strong -- data will not be torn. Neither the > > filesystem nor the device make guarantees that anything ended up on > > stable storage, but if it does, programs see either the old data or the > > new data. > > Yup, that makes sense to me. Perhaps this ^^ is what we should be documenting here. > > Maybe we should rename the whole UAPI s/atomic/untorn/... > > Perhaps, though "torn writes" is nomenclature that nobody outside > storage and filesystem developers really knows about. All I ever > hear from userspace developers is "we want atomic/all-or-nothing > data writes"... Fair 'enuf. --D > -Dave. > -- > Dave Chinner > david@fromorbit.com
On 09/10/2023 22:05, Darrick J. Wong wrote:
>>> If the file range is a sparse hole, the directio setup will allocate
>>> space and create an unwritten mapping before issuing the write bio. The
>>> rest of the process works the same as preallocations and has the same
>>> behaviors.
>>>
>>> If the file range is allocated and was previously written, the write is
>>> issued and that's all that's needed from the fs. After a crash, reads
>>> of the storage device produce the old contents or the new contents.
>> This is exactly what I explained when reviewing the code that
>> rejected RWF_ATOMIC without O_DSYNC on metadata dirty inodes.
> I'm glad we agree. 😄
>
> John, when you're back from vacation, can we get rid of this language
> and all those checks under _is_dsync() in the iomap patch?
>
> (That code is 100% the result of me handwaving and bellyaching 6 months
> ago when the team was trying to get all the atomic writes bits working
> prior to LSF and I was too burned out to think the xfs part through.
> As a result, I decided that we'd only support strict overwrites for the
> first iteration.)
So this following additive code in iomap_dio_bio_iter() should be dropped:
----8<-----
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -275,10 +275,11 @@ static inline blk_opf_t
iomap_dio_bio_opflags(struct iomap_dio *dio,
static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
struct iomap_dio *dio)
{
...
@@ -292,6 +293,13 @@ static loff_t iomap_dio_bio_iter(const struct
iomap_iter *iter,
!bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
return -EINVAL;
+ if (atomic_write && !iocb_is_dsync(dio->iocb)) {
+ if (iomap->flags & IOMAP_F_DIRTY)
+ return -EIO;
+ if (iomap->type != IOMAP_MAPPED)
+ return -EIO;
+ }
+
---->8-----
ok?
>
>>> Summarizing:
>>>
>>> An (ATOMIC|SYNC) request provides the strongest guarantees (data
>>> will not be torn, and all file metadata updates are persisted before
>>> the write is returned to userspace. Programs see either the old data or
>>> the new data, even if there's a crash.
>>>
>>> (ATOMIC|DSYNC) is less strong -- data will not be torn, and any file
>>> updates for just that region are persisted before the write is returned.
>>>
>>> (ATOMIC) is the least strong -- data will not be torn. Neither the
>>> filesystem nor the device make guarantees that anything ended up on
>>> stable storage, but if it does, programs see either the old data or the
>>> new data.
>> Yup, that makes sense to me.
> Perhaps this ^^ is what we should be documenting here.
>
>>> Maybe we should rename the whole UAPI s/atomic/untorn/...
>> Perhaps, though "torn writes" is nomenclature that nobody outside
>> storage and filesystem developers really knows about. All I ever
>> hear from userspace developers is "we want atomic/all-or-nothing
>> data writes"...
> Fair 'enuf.
Thanks,
John
On Tue, Oct 24, 2023 at 01:35:33PM +0100, John Garry wrote:
> On 09/10/2023 22:05, Darrick J. Wong wrote:
> > > > If the file range is a sparse hole, the directio setup will allocate
> > > > space and create an unwritten mapping before issuing the write bio. The
> > > > rest of the process works the same as preallocations and has the same
> > > > behaviors.
> > > >
> > > > If the file range is allocated and was previously written, the write is
> > > > issued and that's all that's needed from the fs. After a crash, reads
> > > > of the storage device produce the old contents or the new contents.
> > > This is exactly what I explained when reviewing the code that
> > > rejected RWF_ATOMIC without O_DSYNC on metadata dirty inodes.
> > I'm glad we agree. 😄
> >
> > John, when you're back from vacation, can we get rid of this language
> > and all those checks under _is_dsync() in the iomap patch?
> >
> > (That code is 100% the result of me handwaving and bellyaching 6 months
> > ago when the team was trying to get all the atomic writes bits working
> > prior to LSF and I was too burned out to think the xfs part through.
> > As a result, I decided that we'd only support strict overwrites for the
> > first iteration.)
>
> So this following additive code in iomap_dio_bio_iter() should be dropped:
>
> ----8<-----
>
> --- a/fs/iomap/direct-io.c
> +++ b/fs/iomap/direct-io.c
> @@ -275,10 +275,11 @@ static inline blk_opf_t iomap_dio_bio_opflags(struct
> iomap_dio *dio,
> static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter,
> struct iomap_dio *dio)
> {
>
> ...
>
> @@ -292,6 +293,13 @@ static loff_t iomap_dio_bio_iter(const struct
> iomap_iter *iter,
> !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter))
> return -EINVAL;
>
> + if (atomic_write && !iocb_is_dsync(dio->iocb)) {
> + if (iomap->flags & IOMAP_F_DIRTY)
> + return -EIO;
> + if (iomap->type != IOMAP_MAPPED)
> + return -EIO;
> + }
> +
>
> ---->8-----
>
> ok?
Yes.
> >
> > > > Summarizing:
> > > >
> > > > An (ATOMIC|SYNC) request provides the strongest guarantees (data
> > > > will not be torn, and all file metadata updates are persisted before
> > > > the write is returned to userspace. Programs see either the old data or
> > > > the new data, even if there's a crash.
> > > >
> > > > (ATOMIC|DSYNC) is less strong -- data will not be torn, and any file
> > > > updates for just that region are persisted before the write is returned.
> > > >
> > > > (ATOMIC) is the least strong -- data will not be torn. Neither the
> > > > filesystem nor the device make guarantees that anything ended up on
> > > > stable storage, but if it does, programs see either the old data or the
> > > > new data.
> > > Yup, that makes sense to me.
> > Perhaps this ^^ is what we should be documenting here.
> >
> > > > Maybe we should rename the whole UAPI s/atomic/untorn/...
> > > Perhaps, though "torn writes" is nomenclature that nobody outside
> > > storage and filesystem developers really knows about. All I ever
> > > hear from userspace developers is "we want atomic/all-or-nothing
> > > data writes"...
How about O_NOTEARS -> PWF_NOTEARS -> REQ_NOTEARS.
<obligatory "There's no crying in baseball" link, etc.>
--D
> > Fair 'enuf.
>
>
> Thanks,
> John
On 9/29/23 02:37, John Garry wrote: > +.BR RWF_ATOMIC " (since Linux 6.7)" > +Allows block-based filesystems to indicate that write operations will be issued > +with torn-write protection. Torn-write protection means that for a power or any > +other hardware failure, all or none of the data from the write will be stored, > +but never a mix of old and new data. This flag is meaningful only for > +.BR pwritev2 (), > +and its effect applies only to the data range written by the system call. > +The total write length must be power-of-2 and must be sized between > +stx_atomic_write_unit_min and stx_atomic_write_unit_max, both inclusive. The > +write must be at a natural offset within the file with respect to the total > +write length. Torn-write protection only works with > +.B O_DIRECT > +flag, i.e. buffered writes are not supported. To guarantee consistency from > +the write between a file's in-core state with the storage device, It seems wrong to me to start the first sentence with "Allows". Atomic behavior should be mandatory if RWF_ATOMIC has been set. Additionally, shouldn't it be documented what value will be stored in errno if the atomic write has been rejected? Thanks, Bart.
On 03/10/2023 20:25, Bart Van Assche wrote: > On 9/29/23 02:37, John Garry wrote: >> +.BR RWF_ATOMIC " (since Linux 6.7)" >> +Allows block-based filesystems to indicate that write operations will >> be issued >> +with torn-write protection. Torn-write protection means that for a >> power or any >> +other hardware failure, all or none of the data from the write will >> be stored, >> +but never a mix of old and new data. This flag is meaningful only for >> +.BR pwritev2 (), >> +and its effect applies only to the data range written by the system >> call. >> +The total write length must be power-of-2 and must be sized between >> +stx_atomic_write_unit_min and stx_atomic_write_unit_max, both >> inclusive. The >> +write must be at a natural offset within the file with respect to the >> total >> +write length. Torn-write protection only works with >> +.B O_DIRECT >> +flag, i.e. buffered writes are not supported. To guarantee >> consistency from >> +the write between a file's in-core state with the storage device, > > It seems wrong to me to start the first sentence with "Allows". Atomic > behavior should be mandatory if RWF_ATOMIC has been set. Yes, I agree that this has been poorly worded. Flag RWF_ATOMIC does not indicate anything. I will fix it. > > Additionally, shouldn't it be documented what value will be stored in > errno if the atomic write has been rejected? So I was treating all atomic writes errors which don't follow the "rules" as low-level I/O errors, which is -EIO. However, yes, I can document this. Further to that, based on description of an error for O_DIRECT, which is to return -EINVAL for misaligned, I think that -EINVAL may be better for any atomic write rule violations. OK? Thanks, John
On Wed, Oct 04, 2023 at 09:47:24AM +0100, John Garry wrote: > On 03/10/2023 20:25, Bart Van Assche wrote: > > On 9/29/23 02:37, John Garry wrote: > > > +.BR RWF_ATOMIC " (since Linux 6.7)" > > > +Allows block-based filesystems to indicate that write operations > > > will be issued > > > +with torn-write protection. Torn-write protection means that for a > > > power or any > > > +other hardware failure, all or none of the data from the write will > > > be stored, > > > +but never a mix of old and new data. This flag is meaningful only for > > > +.BR pwritev2 (), > > > +and its effect applies only to the data range written by the system > > > call. > > > +The total write length must be power-of-2 and must be sized between > > > +stx_atomic_write_unit_min and stx_atomic_write_unit_max, both > > > inclusive. The > > > +write must be at a natural offset within the file with respect to > > > the total > > > +write length. Torn-write protection only works with > > > +.B O_DIRECT > > > +flag, i.e. buffered writes are not supported. To guarantee > > > consistency from > > > +the write between a file's in-core state with the storage device, > > > > It seems wrong to me to start the first sentence with "Allows". Atomic > > behavior should be mandatory if RWF_ATOMIC has been set. > > Yes, I agree that this has been poorly worded. Flag RWF_ATOMIC does not > indicate anything. I will fix it. > > > > > Additionally, shouldn't it be documented what value will be stored in > > errno if the atomic write has been rejected? > > So I was treating all atomic writes errors which don't follow the "rules" as > low-level I/O errors, which is -EIO. However, yes, I can document this. > Further to that, based on description of an error for O_DIRECT, which is to > return -EINVAL for misaligned, I think that -EINVAL may be better for any > atomic write rule violations. OK? Agreed - I was going to make that comment myself about using EINVAL instead of EIO... -Dave. -- Dave Chinner david@fromorbit.com
On 10/4/23 01:47, John Garry wrote: > On 03/10/2023 20:25, Bart Van Assche wrote: >> Additionally, shouldn't it be documented what value will be stored in >> errno if the atomic write has been rejected? > > So I was treating all atomic writes errors which don't follow the > "rules" as low-level I/O errors, which is -EIO. However, yes, I can > document this. Further to that, based on description of an error for > O_DIRECT, which is to return -EINVAL for misaligned, I think that > -EINVAL may be better for any atomic write rule violations. OK? That sounds good to me. Thanks, Bart.
© 2016 - 2025 Red Hat, Inc.