This series allows overlayfs and casefolding to safely be used on the
same filesystem by providing exclusion to ensure that overlayfs never
has to deal with casefolded directories.

Currently, overlayfs can't be used _at all_ if a filesystem even
supports casefolding, which is really nasty for users.

Components:

- filesystem has to track, for each directory, "does any _descendent_
  have casefolding enabled"

- new inode flag to pass this to VFS layer

- new dcache methods for providing refs for overlayfs, and filesystem
  methods for safely clearing this flag

- new superblock flag for indicating to overlayfs & dcache "filesystem
  supports casefolding, it's safe to use provided new dcache methods are
  used"

Kent Overstreet (6):
  bcachefs: BCH_INODE_has_case_insensitive
  darray: lift from bcachefs
  fs: SB_CASEFOLD
  fs: dcache locking for exlusion between overlayfs, casefolding
  bcachefs: Hook up d_casefold_enable()
  overlayfs: Support casefolded filesystems

 MAINTAINERS                             |   7 +
 fs/bcachefs/Makefile                    |   1 -
 fs/bcachefs/bcachefs_format.h           |   3 +-
 fs/bcachefs/btree_node_scan_types.h     |   2 +-
 fs/bcachefs/btree_types.h               |   2 +-
 fs/bcachefs/btree_update.c              |   1 +
 fs/bcachefs/btree_write_buffer_types.h  |   2 +-
 fs/bcachefs/disk_accounting_types.h     |   2 +-
 fs/bcachefs/fs.c                        |  45 +++++-
 fs/bcachefs/fsck.c                      |  12 +-
 fs/bcachefs/inode.c                     |   8 +-
 fs/bcachefs/inode.h                     |   2 +-
 fs/bcachefs/inode_format.h              |   7 +-
 fs/bcachefs/journal_io.h                |   2 +-
 fs/bcachefs/journal_sb.c                |   2 +-
 fs/bcachefs/namei.c                     | 166 +++++++++++++++++++++-
 fs/bcachefs/namei.h                     |   5 +
 fs/bcachefs/rcu_pending.c               |   3 +-
 fs/bcachefs/sb-downgrade.c              |   9 +-
 fs/bcachefs/sb-errors_format.h          |   4 +-
 fs/bcachefs/sb-errors_types.h           |   2 +-
 fs/bcachefs/sb-members.h                |   3 +-
 fs/bcachefs/snapshot_types.h            |   3 +-
 fs/bcachefs/subvolume.h                 |   1 -
 fs/bcachefs/thread_with_file_types.h    |   2 +-
 fs/bcachefs/util.h                      |  28 +---
 fs/dcache.c                             | 177 ++++++++++++++++++++++++
 fs/libfs.c                              |   1 +
 fs/overlayfs/params.c                   |  20 ++-
 fs/overlayfs/util.c                     |  19 ++-
 {fs/bcachefs => include/linux}/darray.h |  70 +++++-----
 include/linux/darray_types.h            |  33 +++++
 include/linux/dcache.h                  |  10 ++
 include/linux/fs.h                      |   4 +
 lib/Makefile                            |   2 +-
 {fs/bcachefs => lib}/darray.c           |   9 +-
 36 files changed, 571 insertions(+), 98 deletions(-)
 rename {fs/bcachefs => include/linux}/darray.h (64%)
 create mode 100644 include/linux/darray_types.h
 rename {fs/bcachefs => lib}/darray.c (75%)

-- 
2.49.0

On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> This series allows overlayfs and casefolding to safely be used on the
> same filesystem by providing exclusion to ensure that overlayfs never
> has to deal with casefolded directories.
>
> Currently, overlayfs can't be used _at all_ if a filesystem even
> supports casefolding, which is really nasty for users.
>
> Components:
>
> - filesystem has to track, for each directory, "does any _descendent_
>   have casefolding enabled"
>
> - new inode flag to pass this to VFS layer
>
> - new dcache methods for providing refs for overlayfs, and filesystem
>   methods for safely clearing this flag
>
> - new superblock flag for indicating to overlayfs & dcache "filesystem
>   supports casefolding, it's safe to use provided new dcache methods are
>   used"
>

I don't think that this is really needed.

Too bad you did not ask before going through the trouble of this implementation.

I think it is enough for overlayfs to know the THIS directory has no
casefolding.

in ovl_lookup() that returns a merged directory, ovl_dentry_weird() would
result in -EIO if any of the real directories have casefolding and we can add
another sanotify in ovl_lookup_single() that the 'base' dentry is not weird()
to cover the case of casefolder changed on an underlying reference directory.

Obviously, if any of the overlayfs layer root dirs have casefolding enabled the
mount would fail.

w.r.t enabling casefolding underneath overlayfs, overlayfs documentation says:

"Changes to underlying filesystems
---------------------------------

Changes to the underlying filesystems while part of a mounted overlay
filesystem are not allowed.  If the underlying filesystem is changed,
the behavior of the overlay is undefined, though it will not result in
a crash or deadlock."

So why is enabling casefolding on underlying layers so special that we
should have specific protection for that?

From what I remember in ext4, enabling casefolding is only allowed
on empty directories.

Is this also the case for bcachefs?

If that is the case, then the situation is even simpler -
If filesystem can singal to vfs/ovl that directory is empty (i.e. S_EMPTYDIR)
then overlayfs can ignore this dir altogether when composing
the merged directory.

But again, I don't think there is a good reason to treat this case
of changing the underlying layer specially.

Please explain if I missed anything.

Thanks,
Amir.

On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
> On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > This series allows overlayfs and casefolding to safely be used on the
> > same filesystem by providing exclusion to ensure that overlayfs never
> > has to deal with casefolded directories.
> >
> > Currently, overlayfs can't be used _at all_ if a filesystem even
> > supports casefolding, which is really nasty for users.
> >
> > Components:
> >
> > - filesystem has to track, for each directory, "does any _descendent_
> >   have casefolding enabled"
> >
> > - new inode flag to pass this to VFS layer
> >
> > - new dcache methods for providing refs for overlayfs, and filesystem
> >   methods for safely clearing this flag
> >
> > - new superblock flag for indicating to overlayfs & dcache "filesystem
> >   supports casefolding, it's safe to use provided new dcache methods are
> >   used"
> >
> 
> I don't think that this is really needed.
> 
> Too bad you did not ask before going through the trouble of this implementation.
> 
> I think it is enough for overlayfs to know the THIS directory has no
> casefolding.

overlayfs works on trees, not directories...

On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
> > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
> > <kent.overstreet@linux.dev> wrote:
> > >
> > > This series allows overlayfs and casefolding to safely be used on the
> > > same filesystem by providing exclusion to ensure that overlayfs never
> > > has to deal with casefolded directories.
> > >
> > > Currently, overlayfs can't be used _at all_ if a filesystem even
> > > supports casefolding, which is really nasty for users.
> > >
> > > Components:
> > >
> > > - filesystem has to track, for each directory, "does any _descendent_
> > >   have casefolding enabled"
> > >
> > > - new inode flag to pass this to VFS layer
> > >
> > > - new dcache methods for providing refs for overlayfs, and filesystem
> > >   methods for safely clearing this flag
> > >
> > > - new superblock flag for indicating to overlayfs & dcache "filesystem
> > >   supports casefolding, it's safe to use provided new dcache methods are
> > >   used"
> > >
> >
> > I don't think that this is really needed.
> >
> > Too bad you did not ask before going through the trouble of this implementation.
> >
> > I think it is enough for overlayfs to know the THIS directory has no
> > casefolding.
>
> overlayfs works on trees, not directories...

I know how overlayfs works...

I've explained why I don't think that sanitizing the entire tree is needed
for creating overlayfs over a filesystem that may enable casefolding
on some of its directories.

Thanks,
Amir.

On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
> On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
> > > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
> > > <kent.overstreet@linux.dev> wrote:
> > > >
> > > > This series allows overlayfs and casefolding to safely be used on the
> > > > same filesystem by providing exclusion to ensure that overlayfs never
> > > > has to deal with casefolded directories.
> > > >
> > > > Currently, overlayfs can't be used _at all_ if a filesystem even
> > > > supports casefolding, which is really nasty for users.
> > > >
> > > > Components:
> > > >
> > > > - filesystem has to track, for each directory, "does any _descendent_
> > > >   have casefolding enabled"
> > > >
> > > > - new inode flag to pass this to VFS layer
> > > >
> > > > - new dcache methods for providing refs for overlayfs, and filesystem
> > > >   methods for safely clearing this flag
> > > >
> > > > - new superblock flag for indicating to overlayfs & dcache "filesystem
> > > >   supports casefolding, it's safe to use provided new dcache methods are
> > > >   used"
> > > >
> > >
> > > I don't think that this is really needed.
> > >
> > > Too bad you did not ask before going through the trouble of this implementation.
> > >
> > > I think it is enough for overlayfs to know the THIS directory has no
> > > casefolding.
> >
> > overlayfs works on trees, not directories...
> 
> I know how overlayfs works...
> 
> I've explained why I don't think that sanitizing the entire tree is needed
> for creating overlayfs over a filesystem that may enable casefolding
> on some of its directories.

So, you want to move error checking from mount time, where we _just_
did a massive API rework so that we can return errors in a way that
users will actually see them - to open/lookup, where all we have are a
small fixed set of error codes?

On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
> > On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
> > <kent.overstreet@linux.dev> wrote:
> > >
> > > On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
> > > > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
> > > > <kent.overstreet@linux.dev> wrote:
> > > > >
> > > > > This series allows overlayfs and casefolding to safely be used on the
> > > > > same filesystem by providing exclusion to ensure that overlayfs never
> > > > > has to deal with casefolded directories.
> > > > >
> > > > > Currently, overlayfs can't be used _at all_ if a filesystem even
> > > > > supports casefolding, which is really nasty for users.
> > > > >
> > > > > Components:
> > > > >
> > > > > - filesystem has to track, for each directory, "does any _descendent_
> > > > >   have casefolding enabled"
> > > > >
> > > > > - new inode flag to pass this to VFS layer
> > > > >
> > > > > - new dcache methods for providing refs for overlayfs, and filesystem
> > > > >   methods for safely clearing this flag
> > > > >
> > > > > - new superblock flag for indicating to overlayfs & dcache "filesystem
> > > > >   supports casefolding, it's safe to use provided new dcache methods are
> > > > >   used"
> > > > >
> > > >
> > > > I don't think that this is really needed.
> > > >
> > > > Too bad you did not ask before going through the trouble of this implementation.
> > > >
> > > > I think it is enough for overlayfs to know the THIS directory has no
> > > > casefolding.
> > >
> > > overlayfs works on trees, not directories...
> >
> > I know how overlayfs works...
> >
> > I've explained why I don't think that sanitizing the entire tree is needed
> > for creating overlayfs over a filesystem that may enable casefolding
> > on some of its directories.
>
> So, you want to move error checking from mount time, where we _just_
> did a massive API rework so that we can return errors in a way that
> users will actually see them - to open/lookup, where all we have are a
> small fixed set of error codes?

That's one way of putting it.

Please explain the use case.

When is overlayfs created over a subtree that is only partially case folded?
Is that really so common that a mount time error justifies all the vfs
infrastructure involved?

Thanks,
Amir.

On Tue, May 20, 2025 at 04:03:27PM +0200, Amir Goldstein wrote:
> On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
> > > On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
> > > <kent.overstreet@linux.dev> wrote:
> > > >
> > > > On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
> > > > > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
> > > > > <kent.overstreet@linux.dev> wrote:
> > > > > >
> > > > > > This series allows overlayfs and casefolding to safely be used on the
> > > > > > same filesystem by providing exclusion to ensure that overlayfs never
> > > > > > has to deal with casefolded directories.
> > > > > >
> > > > > > Currently, overlayfs can't be used _at all_ if a filesystem even
> > > > > > supports casefolding, which is really nasty for users.
> > > > > >
> > > > > > Components:
> > > > > >
> > > > > > - filesystem has to track, for each directory, "does any _descendent_
> > > > > >   have casefolding enabled"
> > > > > >
> > > > > > - new inode flag to pass this to VFS layer
> > > > > >
> > > > > > - new dcache methods for providing refs for overlayfs, and filesystem
> > > > > >   methods for safely clearing this flag
> > > > > >
> > > > > > - new superblock flag for indicating to overlayfs & dcache "filesystem
> > > > > >   supports casefolding, it's safe to use provided new dcache methods are
> > > > > >   used"
> > > > > >
> > > > >
> > > > > I don't think that this is really needed.
> > > > >
> > > > > Too bad you did not ask before going through the trouble of this implementation.
> > > > >
> > > > > I think it is enough for overlayfs to know the THIS directory has no
> > > > > casefolding.
> > > >
> > > > overlayfs works on trees, not directories...
> > >
> > > I know how overlayfs works...
> > >
> > > I've explained why I don't think that sanitizing the entire tree is needed
> > > for creating overlayfs over a filesystem that may enable casefolding
> > > on some of its directories.
> >
> > So, you want to move error checking from mount time, where we _just_
> > did a massive API rework so that we can return errors in a way that
> > users will actually see them - to open/lookup, where all we have are a
> > small fixed set of error codes?
> 
> That's one way of putting it.
> 
> Please explain the use case.
> 
> When is overlayfs created over a subtree that is only partially case folded?
> Is that really so common that a mount time error justifies all the vfs
> infrastructure involved?

Amir, you've got two widely used filesystem features that conflict and
can't be used on the same filesystem.

That's _broken_.

Users hate partitioning just for separate /boot and /home, having to
partition for different applications is horrible. And since overlay fs
is used under the hood by docker, and casefolding is used under the hood
for running Windows applications, this isn't something people can
predict in advance.

>>>>> "Kent" == Kent Overstreet <kent.overstreet@linux.dev> writes:

> On Tue, May 20, 2025 at 04:03:27PM +0200, Amir Goldstein wrote:
>> On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
>> <kent.overstreet@linux.dev> wrote:
>> >
>> > On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
>> > > On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
>> > > <kent.overstreet@linux.dev> wrote:
>> > > >
>> > > > On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
>> > > > > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
>> > > > > <kent.overstreet@linux.dev> wrote:
>> > > > > >
>> > > > > > This series allows overlayfs and casefolding to safely be used on the
>> > > > > > same filesystem by providing exclusion to ensure that overlayfs never
>> > > > > > has to deal with casefolded directories.
>> > > > > >
>> > > > > > Currently, overlayfs can't be used _at all_ if a filesystem even
>> > > > > > supports casefolding, which is really nasty for users.
>> > > > > >
>> > > > > > Components:
>> > > > > >
>> > > > > > - filesystem has to track, for each directory, "does any _descendent_
>> > > > > >   have casefolding enabled"
>> > > > > >
>> > > > > > - new inode flag to pass this to VFS layer
>> > > > > >
>> > > > > > - new dcache methods for providing refs for overlayfs, and filesystem
>> > > > > >   methods for safely clearing this flag
>> > > > > >
>> > > > > > - new superblock flag for indicating to overlayfs & dcache "filesystem
>> > > > > >   supports casefolding, it's safe to use provided new dcache methods are
>> > > > > >   used"
>> > > > > >
>> > > > >
>> > > > > I don't think that this is really needed.
>> > > > >
>> > > > > Too bad you did not ask before going through the trouble of this implementation.
>> > > > >
>> > > > > I think it is enough for overlayfs to know the THIS directory has no
>> > > > > casefolding.
>> > > >
>> > > > overlayfs works on trees, not directories...
>> > >
>> > > I know how overlayfs works...
>> > >
>> > > I've explained why I don't think that sanitizing the entire tree is needed
>> > > for creating overlayfs over a filesystem that may enable casefolding
>> > > on some of its directories.
>> >
>> > So, you want to move error checking from mount time, where we _just_
>> > did a massive API rework so that we can return errors in a way that
>> > users will actually see them - to open/lookup, where all we have are a
>> > small fixed set of error codes?
>> 
>> That's one way of putting it.
>> 
>> Please explain the use case.
>> 
>> When is overlayfs created over a subtree that is only partially case folded?
>> Is that really so common that a mount time error justifies all the vfs
>> infrastructure involved?

> Amir, you've got two widely used filesystem features that conflict and
> can't be used on the same filesystem.

Wait, what?  How many people use casefolding, on a per-directory
basis?  It's stupid.  Unix/Linux has used case-sensitive filesystems
for years.  Yes, linux supports other OSes which did do casefolding,
but yikes... per-directory support is just insane.  It should be
per-filesystem only at BEST.  

> That's _broken_.

So?  what about my cross mounting of VMS filesystems with "foo.txt;3"
version control so I can go back to previous versions?  Why can't I do
that from my Linux systems that's mounting that VMS image?   

Just because it's done doesn't mean it's not dumb.  

> Users hate partitioning just for separate /boot and /home, having to
> partition for different applications is horrible. And since overlay
> fs is used under the hood by docker, and casefolding is used under
> the hood for running Windows applications, this isn't something
> people can predict in advance.

Sure I can, I don't run windows applications to screw casefolding.
:-)

And I personally LIKE having a seperate /boot and /home, because it
gives isolation.  The world is not just single user laptops with
everything all on one disk or spread across a couple of disks using
LVM or RAID or all of the above.  

I also don't see any updates for the XFS tests, or any other
filesystem tests, that actually checks and confirms this decidedly
obtuse and dumb to implement idea.  


John

On 20/05/2025 20:49, John Stoffel wrote:
>>>>>> "Kent" == Kent Overstreet <kent.overstreet@linux.dev> writes:
>> On Tue, May 20, 2025 at 04:03:27PM +0200, Amir Goldstein wrote:
>>> On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
>>> <kent.overstreet@linux.dev> wrote:
>>>> On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
>>>>> On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
>>>>> <kent.overstreet@linux.dev> wrote:
>>>>>> On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
>>>>>>> On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
>>>>>>> <kent.overstreet@linux.dev> wrote:
>>>>>>>> This series allows overlayfs and casefolding to safely be used on the
>>>>>>>> same filesystem by providing exclusion to ensure that overlayfs never
>>>>>>>> has to deal with casefolded directories.
>>>>>>>>
>>>>>>>> Currently, overlayfs can't be used _at all_ if a filesystem even
>>>>>>>> supports casefolding, which is really nasty for users.
>>>>>>>>
>>>>>>>> Components:
>>>>>>>>
>>>>>>>> - filesystem has to track, for each directory, "does any _descendent_
>>>>>>>>   have casefolding enabled"
>>>>>>>>
>>>>>>>> - new inode flag to pass this to VFS layer
>>>>>>>>
>>>>>>>> - new dcache methods for providing refs for overlayfs, and filesystem
>>>>>>>>   methods for safely clearing this flag
>>>>>>>>
>>>>>>>> - new superblock flag for indicating to overlayfs & dcache "filesystem
>>>>>>>>   supports casefolding, it's safe to use provided new dcache methods are
>>>>>>>>   used"
>>>>>>>>
>>>>>>> I don't think that this is really needed.
>>>>>>>
>>>>>>> Too bad you did not ask before going through the trouble of this implementation.
>>>>>>>
>>>>>>> I think it is enough for overlayfs to know the THIS directory has no
>>>>>>> casefolding.
>>>>>> overlayfs works on trees, not directories...
>>>>> I know how overlayfs works...
>>>>>
>>>>> I've explained why I don't think that sanitizing the entire tree is needed
>>>>> for creating overlayfs over a filesystem that may enable casefolding
>>>>> on some of its directories.
>>>> So, you want to move error checking from mount time, where we _just_
>>>> did a massive API rework so that we can return errors in a way that
>>>> users will actually see them - to open/lookup, where all we have are a
>>>> small fixed set of error codes?
>>> That's one way of putting it.
>>>
>>> Please explain the use case.
>>>
>>> When is overlayfs created over a subtree that is only partially case folded?
>>> Is that really so common that a mount time error justifies all the vfs
>>> infrastructure involved?
>> Amir, you've got two widely used filesystem features that conflict and
>> can't be used on the same filesystem.
> Wait, what?  How many people use casefolding, on a per-directory
> basis?  It's stupid.  Unix/Linux has used case-sensitive filesystems
> for years.  Yes, linux supports other OSes which did do casefolding,
> but yikes... per-directory support is just insane.  It should be
> per-filesystem only at BEST.  
>
>> That's _broken_.
> So?  what about my cross mounting of VMS filesystems with "foo.txt;3"
> version control so I can go back to previous versions?  Why can't I do
> that from my Linux systems that's mounting that VMS image?   
>
> Just because it's done doesn't mean it's not dumb.  
>
>> Users hate partitioning just for separate /boot and /home, having to
>> partition for different applications is horrible. And since overlay
>> fs is used under the hood by docker, and casefolding is used under
>> the hood for running Windows applications, this isn't something
>> people can predict in advance.
> Sure I can, I don't run windows applications to screw casefolding.
> :-)
>
> And I personally LIKE having a seperate /boot and /home, because it
> gives isolation.  The world is not just single user laptops with
> everything all on one disk or spread across a couple of disks using
> LVM or RAID or all of the above.  
>
> I also don't see any updates for the XFS tests, or any other
> filesystem tests, that actually checks and confirms this decidedly
> obtuse and dumb to implement idea.  
>
>
> John
>
Hi there,

would you partition different subdirs of your /home? So there is
.local/share/containers where users put their container-stuff (at least
podman does). Then there is .wine where case-folding-craziness lives.
And then there is the mess that is Steam, which does all kinds of
containery case-foldy stuff. As much as I would like to keep these
things apart, it is not feasible. Not for me as a "power user", and
certainly far out of reach for average Joe user.

Just my 2 ct, greets

/Malte

On Wed May 21, 2025 at 4:26 AM PDT, Malte Schröder wrote:
> On 20/05/2025 20:49, John Stoffel wrote:
>>>>>>> "Kent" == Kent Overstreet <kent.overstreet@linux.dev> writes:
>>> On Tue, May 20, 2025 at 04:03:27PM +0200, Amir Goldstein wrote:
>>>> On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
>>>> <kent.overstreet@linux.dev> wrote:
>>>>> On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
>>>>>> On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
>>>>>> <kent.overstreet@linux.dev> wrote:
>>>>>>> On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
>>>>>>>> On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
>>>>>>>> <kent.overstreet@linux.dev> wrote:
>>>>>>>>> This series allows overlayfs and casefolding to safely be used on the
>>>>>>>>> same filesystem by providing exclusion to ensure that overlayfs never
>>>>>>>>> has to deal with casefolded directories.
>>>>>>>>>
>>>>>>>>> Currently, overlayfs can't be used _at all_ if a filesystem even
>>>>>>>>> supports casefolding, which is really nasty for users.
>>>>>>>>>
>>>>>>>>> Components:
>>>>>>>>>
>>>>>>>>> - filesystem has to track, for each directory, "does any _descendent_
>>>>>>>>>   have casefolding enabled"
>>>>>>>>>
>>>>>>>>> - new inode flag to pass this to VFS layer
>>>>>>>>>
>>>>>>>>> - new dcache methods for providing refs for overlayfs, and filesystem
>>>>>>>>>   methods for safely clearing this flag
>>>>>>>>>
>>>>>>>>> - new superblock flag for indicating to overlayfs & dcache "filesystem
>>>>>>>>>   supports casefolding, it's safe to use provided new dcache methods are
>>>>>>>>>   used"
>>>>>>>>>
>>>>>>>> I don't think that this is really needed.
>>>>>>>>
>>>>>>>> Too bad you did not ask before going through the trouble of this implementation.
>>>>>>>>
>>>>>>>> I think it is enough for overlayfs to know the THIS directory has no
>>>>>>>> casefolding.
>>>>>>> overlayfs works on trees, not directories...
>>>>>> I know how overlayfs works...
>>>>>>
>>>>>> I've explained why I don't think that sanitizing the entire tree is needed
>>>>>> for creating overlayfs over a filesystem that may enable casefolding
>>>>>> on some of its directories.
>>>>> So, you want to move error checking from mount time, where we _just_
>>>>> did a massive API rework so that we can return errors in a way that
>>>>> users will actually see them - to open/lookup, where all we have are a
>>>>> small fixed set of error codes?
>>>> That's one way of putting it.
>>>>
>>>> Please explain the use case.
>>>>
>>>> When is overlayfs created over a subtree that is only partially case folded?
>>>> Is that really so common that a mount time error justifies all the vfs
>>>> infrastructure involved?
>>> Amir, you've got two widely used filesystem features that conflict and
>>> can't be used on the same filesystem.
>> Wait, what?  How many people use casefolding, on a per-directory
>> basis?  It's stupid.  Unix/Linux has used case-sensitive filesystems
>> for years.  Yes, linux supports other OSes which did do casefolding,
>> but yikes... per-directory support is just insane.  It should be
>> per-filesystem only at BEST.  
>>
>>> That's _broken_.
>> So?  what about my cross mounting of VMS filesystems with "foo.txt;3"
>> version control so I can go back to previous versions?  Why can't I do
>> that from my Linux systems that's mounting that VMS image?   
>>
>> Just because it's done doesn't mean it's not dumb.  
>>
>>> Users hate partitioning just for separate /boot and /home, having to
>>> partition for different applications is horrible. And since overlay
>>> fs is used under the hood by docker, and casefolding is used under
>>> the hood for running Windows applications, this isn't something
>>> people can predict in advance.
>> Sure I can, I don't run windows applications to screw casefolding.
>> :-)
>>
>> And I personally LIKE having a seperate /boot and /home, because it
>> gives isolation.  The world is not just single user laptops with
>> everything all on one disk or spread across a couple of disks using
>> LVM or RAID or all of the above.  
>>
>> I also don't see any updates for the XFS tests, or any other
>> filesystem tests, that actually checks and confirms this decidedly
>> obtuse and dumb to implement idea.  
>>
>>
>> John
>>
> Hi there,
>
> would you partition different subdirs of your /home? So there is
> .local/share/containers where users put their container-stuff (at least
> podman does). Then there is .wine where case-folding-craziness lives.
> And then there is the mess that is Steam, which does all kinds of
> containery case-foldy stuff. As much as I would like to keep these
> things apart, it is not feasible. Not for me as a "power user", and
> certainly far out of reach for average Joe user.
>
> Just my 2 ct, greets

"But just disable it globally" How about no. Sure, ext4 has that flag,
but bcachefs by design does not. And a change that already made it into
6.15 has made it trip overlayfs as it is now, unconditionally. The
purpose of this new implementation is to make it work, and satisfy the
condition that overlayfs is guaranteed no casefolding in its tree, and
that nobody may create a new folder inside that tree and suddenly turn
on casefolding on it.

And this change also makes it possible to use it with ext4 with the
global casefolding flag enabled. It shouldn't be necessary to have a
global killswitch, these features should be able to live together on the
same filesystem as long as they're not touching each other, and aren't
allowed to touch each other.

>
> /Malte

On Tue, May 20, 2025 at 02:49:16PM -0400, John Stoffel wrote:
> >>>>> "Kent" == Kent Overstreet <kent.overstreet@linux.dev> writes:
> 
> > On Tue, May 20, 2025 at 04:03:27PM +0200, Amir Goldstein wrote:
> >> On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
> >> <kent.overstreet@linux.dev> wrote:
> >> >
> >> > On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
> >> > > On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
> >> > > <kent.overstreet@linux.dev> wrote:
> >> > > >
> >> > > > On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
> >> > > > > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
> >> > > > > <kent.overstreet@linux.dev> wrote:
> >> > > > > >
> >> > > > > > This series allows overlayfs and casefolding to safely be used on the
> >> > > > > > same filesystem by providing exclusion to ensure that overlayfs never
> >> > > > > > has to deal with casefolded directories.
> >> > > > > >
> >> > > > > > Currently, overlayfs can't be used _at all_ if a filesystem even
> >> > > > > > supports casefolding, which is really nasty for users.
> >> > > > > >
> >> > > > > > Components:
> >> > > > > >
> >> > > > > > - filesystem has to track, for each directory, "does any _descendent_
> >> > > > > >   have casefolding enabled"
> >> > > > > >
> >> > > > > > - new inode flag to pass this to VFS layer
> >> > > > > >
> >> > > > > > - new dcache methods for providing refs for overlayfs, and filesystem
> >> > > > > >   methods for safely clearing this flag
> >> > > > > >
> >> > > > > > - new superblock flag for indicating to overlayfs & dcache "filesystem
> >> > > > > >   supports casefolding, it's safe to use provided new dcache methods are
> >> > > > > >   used"
> >> > > > > >
> >> > > > >
> >> > > > > I don't think that this is really needed.
> >> > > > >
> >> > > > > Too bad you did not ask before going through the trouble of this implementation.
> >> > > > >
> >> > > > > I think it is enough for overlayfs to know the THIS directory has no
> >> > > > > casefolding.
> >> > > >
> >> > > > overlayfs works on trees, not directories...
> >> > >
> >> > > I know how overlayfs works...
> >> > >
> >> > > I've explained why I don't think that sanitizing the entire tree is needed
> >> > > for creating overlayfs over a filesystem that may enable casefolding
> >> > > on some of its directories.
> >> >
> >> > So, you want to move error checking from mount time, where we _just_
> >> > did a massive API rework so that we can return errors in a way that
> >> > users will actually see them - to open/lookup, where all we have are a
> >> > small fixed set of error codes?
> >> 
> >> That's one way of putting it.
> >> 
> >> Please explain the use case.
> >> 
> >> When is overlayfs created over a subtree that is only partially case folded?
> >> Is that really so common that a mount time error justifies all the vfs
> >> infrastructure involved?
> 
> > Amir, you've got two widely used filesystem features that conflict and
> > can't be used on the same filesystem.
> 
> Wait, what?  How many people use casefolding, on a per-directory
> basis?  It's stupid.  Unix/Linux has used case-sensitive filesystems
> for years.  Yes, linux supports other OSes which did do casefolding,
> but yikes... per-directory support is just insane.  It should be
> per-filesystem only at BEST.  

Quite a lot.

You may not realize this, but Valve has, quietly, behind the scenes,
been intelligently funding a ton of Linux work with an eye towards not
just gaming, but improving Linus on the desktop. And they've been
deploying it too, you can buy a Steam deck today.

And a significant fraction of desktop users like to play games - we're
not just all work. Windows ports need casefolding - alternatives have
been discussed and they're non viable.

(I fondly remember the days when I had time for such things).

Samba fileservers are a thing, too.

And for all us desktop/workstation users, not being able to use the same
filesystem for wine and docker is the kind of jankiness that makes
people say "maybe Linux isn't ready for the desktop after all".

Put aside your feelings on casefolding - this is about basic attention
to detail.

> > That's _broken_.
> 
> So?  what about my cross mounting of VMS filesystems with "foo.txt;3"
> version control so I can go back to previous versions?  Why can't I do
> that from my Linux systems that's mounting that VMS image?   
> 
> Just because it's done doesn't mean it's not dumb.  
> 
> > Users hate partitioning just for separate /boot and /home, having to
> > partition for different applications is horrible. And since overlay
> > fs is used under the hood by docker, and casefolding is used under
> > the hood for running Windows applications, this isn't something
> > people can predict in advance.
> 
> Sure I can, I don't run windows applications to screw casefolding.
> :-)
> 
> And I personally LIKE having a seperate /boot and /home, because it
> gives isolation.  The world is not just single user laptops with
> everything all on one disk or spread across a couple of disks using
> LVM or RAID or all of the above.  
> 
> I also don't see any updates for the XFS tests, or any other
> filesystem tests, that actually checks and confirms this decidedly
> obtuse and dumb to implement idea.  

Well, you certainly are making your personal feelings known :)

But for me, as the engineer designing and implementing this stuff, I
don't let my personal feelings dictate.

That's not my place.

My job is to write code that works, works reliably, solves real
problems, and lets users do the things they want with their machines.

All this drama over casefolding has been pure distraction, and I would
appreciate if you and everyone else could tone it down now.

>>>>> "Kent" == Kent Overstreet <kent.overstreet@linux.dev> writes:

> On Tue, May 20, 2025 at 02:49:16PM -0400, John Stoffel wrote:
>> >>>>> "Kent" == Kent Overstreet <kent.overstreet@linux.dev> writes:
>> 
>> > On Tue, May 20, 2025 at 04:03:27PM +0200, Amir Goldstein wrote:
>> >> On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
>> >> <kent.overstreet@linux.dev> wrote:
>> >> >
>> >> > On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
>> >> > > On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
>> >> > > <kent.overstreet@linux.dev> wrote:
>> >> > > >
>> >> > > > On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
>> >> > > > > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
>> >> > > > > <kent.overstreet@linux.dev> wrote:
>> >> > > > > >
>> >> > > > > > This series allows overlayfs and casefolding to safely be used on the
>> >> > > > > > same filesystem by providing exclusion to ensure that overlayfs never
>> >> > > > > > has to deal with casefolded directories.
>> >> > > > > >
>> >> > > > > > Currently, overlayfs can't be used _at all_ if a filesystem even
>> >> > > > > > supports casefolding, which is really nasty for users.
>> >> > > > > >
>> >> > > > > > Components:
>> >> > > > > >
>> >> > > > > > - filesystem has to track, for each directory, "does any _descendent_
>> >> > > > > >   have casefolding enabled"
>> >> > > > > >
>> >> > > > > > - new inode flag to pass this to VFS layer
>> >> > > > > >
>> >> > > > > > - new dcache methods for providing refs for overlayfs, and filesystem
>> >> > > > > >   methods for safely clearing this flag
>> >> > > > > >
>> >> > > > > > - new superblock flag for indicating to overlayfs & dcache "filesystem
>> >> > > > > >   supports casefolding, it's safe to use provided new dcache methods are
>> >> > > > > >   used"
>> >> > > > > >
>> >> > > > >
>> >> > > > > I don't think that this is really needed.
>> >> > > > >
>> >> > > > > Too bad you did not ask before going through the trouble of this implementation.
>> >> > > > >
>> >> > > > > I think it is enough for overlayfs to know the THIS directory has no
>> >> > > > > casefolding.
>> >> > > >
>> >> > > > overlayfs works on trees, not directories...
>> >> > >
>> >> > > I know how overlayfs works...
>> >> > >
>> >> > > I've explained why I don't think that sanitizing the entire tree is needed
>> >> > > for creating overlayfs over a filesystem that may enable casefolding
>> >> > > on some of its directories.
>> >> >
>> >> > So, you want to move error checking from mount time, where we _just_
>> >> > did a massive API rework so that we can return errors in a way that
>> >> > users will actually see them - to open/lookup, where all we have are a
>> >> > small fixed set of error codes?
>> >> 
>> >> That's one way of putting it.
>> >> 
>> >> Please explain the use case.
>> >> 
>> >> When is overlayfs created over a subtree that is only partially case folded?
>> >> Is that really so common that a mount time error justifies all the vfs
>> >> infrastructure involved?
>> 
>> > Amir, you've got two widely used filesystem features that conflict and
>> > can't be used on the same filesystem.
>> 
>> Wait, what?  How many people use casefolding, on a per-directory
>> basis?  It's stupid.  Unix/Linux has used case-sensitive filesystems
>> for years.  Yes, linux supports other OSes which did do casefolding,
>> but yikes... per-directory support is just insane.  It should be
>> per-filesystem only at BEST.  

> Quite a lot.

As I'm learning!  :-)    And sorry for the late reply... life got in
the way...

> You may not realize this, but Valve has, quietly, behind the scenes,
> been intelligently funding a ton of Linux work with an eye towards
> not just gaming, but improving Linus on the desktop. And they've
> been deploying it too, you can buy a Steam deck today.

So... this arguement that you're improving the desktop by doing
case-folding on filesystems smells bogus to me.  I think it's more
correct to say "Valve wants windows application support on top of
linux, and since windows does case-folding, linux should too."

Which is more an arguement for supporting legacy backwards
implementations.  But for the future?  God, why? 

> And a significant fraction of desktop users like to play games - we're
> not just all work. Windows ports need casefolding - alternatives have
> been discussed and they're non viable.

> (I fondly remember the days when I had time for such things).

Heh heh.  I'm not a Windows gamer at all, which is why I haven't run
into this at all.  

> Samba fileservers are a thing, too.

Sure.  And I understand that Windows (even Windows 11!) is still does
casefolding by default.  Sigh... which sucks.  

> And for all us desktop/workstation users, not being able to use the same
> filesystem for wine and docker is the kind of jankiness that makes
> people say "maybe Linux isn't ready for the desktop after all".

I dunno... I just wonder why overlayfs (or some other level) can't
just hide this and leave the lower levels alone.  Yes, when you have
'foo' and 'Foo' on Linux, how do you handle it?  It's painful.

> Put aside your feelings on casefolding - this is about basic attention
> to detail.

Heh.  I'm getting an education here.  And disliking all the hoops
filesystems have to go through to support this (IMHO) dumb feature of
windows.  

>> > That's _broken_.
>> 
>> So?  what about my cross mounting of VMS filesystems with "foo.txt;3"
>> version control so I can go back to previous versions?  Why can't I do
>> that from my Linux systems that's mounting that VMS image?   
>> 
>> Just because it's done doesn't mean it's not dumb.  
>> 
>> > Users hate partitioning just for separate /boot and /home, having to
>> > partition for different applications is horrible. And since overlay
>> > fs is used under the hood by docker, and casefolding is used under
>> > the hood for running Windows applications, this isn't something
>> > people can predict in advance.
>> 
>> Sure I can, I don't run windows applications to screw casefolding.
>> :-)
>> 
>> And I personally LIKE having a seperate /boot and /home, because it
>> gives isolation.  The world is not just single user laptops with
>> everything all on one disk or spread across a couple of disks using
>> LVM or RAID or all of the above.  
>> 
>> I also don't see any updates for the XFS tests, or any other
>> filesystem tests, that actually checks and confirms this decidedly
>> obtuse and dumb to implement idea.  

> Well, you certainly are making your personal feelings known :)

> But for me, as the engineer designing and implementing this stuff, I
> don't let my personal feelings dictate.

> That's not my place.

> My job is to write code that works, works reliably, solves real
> problems, and lets users do the things they want with their machines.

> All this drama over casefolding has been pure distraction, and I would
> appreciate if you and everyone else could tone it down now.

On Tue, May 20, 2025 at 4:12 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Tue, May 20, 2025 at 04:03:27PM +0200, Amir Goldstein wrote:
> > On Tue, May 20, 2025 at 2:43 PM Kent Overstreet
> > <kent.overstreet@linux.dev> wrote:
> > >
> > > On Tue, May 20, 2025 at 02:40:07PM +0200, Amir Goldstein wrote:
> > > > On Tue, May 20, 2025 at 2:25 PM Kent Overstreet
> > > > <kent.overstreet@linux.dev> wrote:
> > > > >
> > > > > On Tue, May 20, 2025 at 10:05:14AM +0200, Amir Goldstein wrote:
> > > > > > On Tue, May 20, 2025 at 7:16 AM Kent Overstreet
> > > > > > <kent.overstreet@linux.dev> wrote:
> > > > > > >
> > > > > > > This series allows overlayfs and casefolding to safely be used on the
> > > > > > > same filesystem by providing exclusion to ensure that overlayfs never
> > > > > > > has to deal with casefolded directories.
> > > > > > >
> > > > > > > Currently, overlayfs can't be used _at all_ if a filesystem even
> > > > > > > supports casefolding, which is really nasty for users.
> > > > > > >
> > > > > > > Components:
> > > > > > >
> > > > > > > - filesystem has to track, for each directory, "does any _descendent_
> > > > > > >   have casefolding enabled"
> > > > > > >
> > > > > > > - new inode flag to pass this to VFS layer
> > > > > > >
> > > > > > > - new dcache methods for providing refs for overlayfs, and filesystem
> > > > > > >   methods for safely clearing this flag
> > > > > > >
> > > > > > > - new superblock flag for indicating to overlayfs & dcache "filesystem
> > > > > > >   supports casefolding, it's safe to use provided new dcache methods are
> > > > > > >   used"
> > > > > > >
> > > > > >
> > > > > > I don't think that this is really needed.
> > > > > >
> > > > > > Too bad you did not ask before going through the trouble of this implementation.
> > > > > >
> > > > > > I think it is enough for overlayfs to know the THIS directory has no
> > > > > > casefolding.
> > > > >
> > > > > overlayfs works on trees, not directories...
> > > >
> > > > I know how overlayfs works...
> > > >
> > > > I've explained why I don't think that sanitizing the entire tree is needed
> > > > for creating overlayfs over a filesystem that may enable casefolding
> > > > on some of its directories.
> > >
> > > So, you want to move error checking from mount time, where we _just_
> > > did a massive API rework so that we can return errors in a way that
> > > users will actually see them - to open/lookup, where all we have are a
> > > small fixed set of error codes?
> >
> > That's one way of putting it.
> >
> > Please explain the use case.
> >
> > When is overlayfs created over a subtree that is only partially case folded?
> > Is that really so common that a mount time error justifies all the vfs
> > infrastructure involved?
>
> Amir, you've got two widely used filesystem features that conflict and
> can't be used on the same filesystem.
>
> That's _broken_.

Correct.

I am saying that IMO a smaller impact (and less user friendly) fix is more
appropriate way to deal with this problem.

>
> Users hate partitioning just for separate /boot and /home, having to
> partition for different applications is horrible. And since overlay fs
> is used under the hood by docker, and casefolding is used under the hood
> for running Windows applications, this isn't something people can
> predict in advance.

Right, I am not expecting users to partition by application,
but my question was this:

When is overlayfs created over a subtree that is only partially case-folded?

Obviously, docker would create overlayfs on parts of the fs
and smbd/cygwin could create a case folder subtree on another
part of the fs.
I just don't see a common use case when these sections overlap.

Perhaps I am wrong (please present real world use cases),
but my claim is that this case is not common enough and therefore,
a suboptimal EIO error from lookup is good enough to prevert crossing
over into the case folded zone by mistake, just as EIO on lookup is
enough to deal with the unsupported use case of modifying
overlayfs underlying layers with overlay is mounted.

BTW, it is not enough to claim that there is no case folding for the
entire subtree to allow the mount.
For overlayfs to allow d_hash()/d_compare() fs must claim that
these implementations are the default implementation in all subtree
or at least that all layers share the same implementation.

Thanks,
Amir.

On Tue, May 20, 2025 at 04:33:14PM +0200, Amir Goldstein wrote:
> I am saying that IMO a smaller impact (and less user friendly) fix is more
> appropriate way to deal with this problem.

What do you think about doing your approach as a stopgap?

It seems this is hitting a lot of people, something we can backport to
6.15 would be good to have.

On Fri, May 23, 2025 at 4:10 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Tue, May 20, 2025 at 04:33:14PM +0200, Amir Goldstein wrote:
> > I am saying that IMO a smaller impact (and less user friendly) fix is more
> > appropriate way to deal with this problem.
>
> What do you think about doing your approach as a stopgap?
>
> It seems this is hitting a lot of people, something we can backport to
> 6.15 would be good to have.

Yes, I think I can do that.
Will try to get to it next week.

Thanks,
Amir.

On Fri, May 23, 2025 at 7:14 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Fri, May 23, 2025 at 4:10 PM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Tue, May 20, 2025 at 04:33:14PM +0200, Amir Goldstein wrote:
> > > I am saying that IMO a smaller impact (and less user friendly) fix is more
> > > appropriate way to deal with this problem.
> >
> > What do you think about doing your approach as a stopgap?
> >
> > It seems this is hitting a lot of people, something we can backport to
> > 6.15 would be good to have.
>
> Yes, I think I can do that.
> Will try to get to it next week.
>

On second look, here is a compile-basic-sanity-tested patch.

Care to test if it does the job for you?

Thanks,
Amir.

On Fri, May 23, 2025 at 10:30:16PM +0200, Amir Goldstein wrote:

That makes fstests generic/631 pass.

We really should be logging something in dmesg on error due to
casefolded directory, though.

On Fri, May 23, 2025 at 11:10 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Fri, May 23, 2025 at 10:30:16PM +0200, Amir Goldstein wrote:
>
> That makes fstests generic/631 pass.

Yes, that is not very surprising.
I meant if you could help test that:

1. mounting case folder upperdir/lowerdir fails
2. lookup a case folder subdir fails
3. lookup in a dir that was empty and became case folder while ovl was
mounted fails

For me, I do not have any setup with case folding subtrees
so testing those use cases would take me time and
I think that you must have tested all those scenarios with your patch set?
and maybe already have some fstests for them?

>
> We really should be logging something in dmesg on error due to
> casefolded directory, though.

No problem.
Attached v2 with warnings.

For example, here is the new warning in test overlay/065:

[  131.815110] overlayfs: failed lookup in lower (/lower,
name='upper', err=-40): overlapping layers

Thanks,
Amir.

On Sat, May 24, 2025 at 03:01:44PM +0200, Amir Goldstein wrote:
> On Fri, May 23, 2025 at 11:10 PM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Fri, May 23, 2025 at 10:30:16PM +0200, Amir Goldstein wrote:
> >
> > That makes fstests generic/631 pass.
> 
> Yes, that is not very surprising.
> I meant if you could help test that:
> 
> 1. mounting case folder upperdir/lowerdir fails
> 2. lookup a case folder subdir fails
> 3. lookup in a dir that was empty and became case folder while ovl was
> mounted fails
> 
> For me, I do not have any setup with case folding subtrees
> so testing those use cases would take me time and
> I think that you must have tested all those scenarios with your patch set?
> and maybe already have some fstests for them?

Unmount fauls after I test an overlayfs with a casefold subdir:

Testing an overlayfs on a casefold fs with non-casefolded dirs
Test using casefolded dir - should fail
overlayfs: failed to resolve '/mnt/casefold': -2
mount: /mnt/merged: special device overlay does not exist.
       dmesg(1) may have more information after failed mount system call.
Test using a dir with a casefold subdir - should mount
overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
ls: cannot access '/mnt/merged/dir/casefold': No such file or directory
umount: /mnt: target is busy.

https://evilpiepirate.org/git/ktest.git/commit/?id=47d1f2a04d79bc4cbc843f81e71eb7d821fb8384

On Sun, May 25, 2025 at 8:27 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Sat, May 24, 2025 at 03:01:44PM +0200, Amir Goldstein wrote:
> > On Fri, May 23, 2025 at 11:10 PM Kent Overstreet
> > <kent.overstreet@linux.dev> wrote:
> > >
> > > On Fri, May 23, 2025 at 10:30:16PM +0200, Amir Goldstein wrote:
> > >
> > > That makes fstests generic/631 pass.
> >
> > Yes, that is not very surprising.
> > I meant if you could help test that:
> >
> > 1. mounting case folder upperdir/lowerdir fails
> > 2. lookup a case folder subdir fails
> > 3. lookup in a dir that was empty and became case folder while ovl was
> > mounted fails
> >
> > For me, I do not have any setup with case folding subtrees
> > so testing those use cases would take me time and
> > I think that you must have tested all those scenarios with your patch set?
> > and maybe already have some fstests for them?
>
> Unmount fauls after I test an overlayfs with a casefold subdir:
>
> Testing an overlayfs on a casefold fs with non-casefolded dirs
> Test using casefolded dir - should fail
> overlayfs: failed to resolve '/mnt/casefold': -2
> mount: /mnt/merged: special device overlay does not exist.
>        dmesg(1) may have more information after failed mount system call.

Test is using the wrong path:


+    echo "Test using casefolded dir - should fail"
+    ! mount -t overlay -o
lowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/work overlay
/mnt/merged
+    ! mount -t overlay -o
lowerdir=/mnt/casefold,upperdir=/mnt/casefold,workdir=/mnt/work
overlay /mnt/merged

There is no "/mnt/casefold"

> Test using a dir with a casefold subdir - should mount
> overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
> overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.

Those warnings are because you have a stray mount command above:
+    echo "Test using casefolded dir - should fail"
+    ! mount -t overlay -o
lowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/work overlay
/mnt/merged

So a mount already exists. leftover?

> ls: cannot access '/mnt/merged/dir/casefold': No such file or directory
> umount: /mnt: target is busy.

Not sure about that, but could be due to the aforementioned stay mount.

>
> https://evilpiepirate.org/git/ktest.git/commit/?id=47d1f2a04d79bc4cbc843f81e71eb7d821fb8384

Please fix the test and report Tested-by if all works as expected.
Please include dmesg so we can see the new warnings.

Thanks,
Amir.

On Tue, May 27, 2025 at 10:57:05AM +0200, Amir Goldstein wrote:
> On Sun, May 25, 2025 at 8:27 PM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Sat, May 24, 2025 at 03:01:44PM +0200, Amir Goldstein wrote:
> > > On Fri, May 23, 2025 at 11:10 PM Kent Overstreet
> > > <kent.overstreet@linux.dev> wrote:
> > > >
> > > > On Fri, May 23, 2025 at 10:30:16PM +0200, Amir Goldstein wrote:
> > > >
> > > > That makes fstests generic/631 pass.
> > >
> > > Yes, that is not very surprising.
> > > I meant if you could help test that:
> > >
> > > 1. mounting case folder upperdir/lowerdir fails
> > > 2. lookup a case folder subdir fails
> > > 3. lookup in a dir that was empty and became case folder while ovl was
> > > mounted fails
> > >
> > > For me, I do not have any setup with case folding subtrees
> > > so testing those use cases would take me time and
> > > I think that you must have tested all those scenarios with your patch set?
> > > and maybe already have some fstests for them?
> >
> > Unmount fauls after I test an overlayfs with a casefold subdir:
> >
> > Testing an overlayfs on a casefold fs with non-casefolded dirs
> > Test using casefolded dir - should fail
> > overlayfs: failed to resolve '/mnt/casefold': -2
> > mount: /mnt/merged: special device overlay does not exist.
> >        dmesg(1) may have more information after failed mount system call.
> 
> Test is using the wrong path:
> 
> 
> +    echo "Test using casefolded dir - should fail"
> +    ! mount -t overlay -o
> lowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/work overlay
> /mnt/merged
> +    ! mount -t overlay -o
> lowerdir=/mnt/casefold,upperdir=/mnt/casefold,workdir=/mnt/work
> overlay /mnt/merged
> 
> There is no "/mnt/casefold"

*nod*

> > Test using a dir with a casefold subdir - should mount
> > overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
> > overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
> 
> Those warnings are because you have a stray mount command above:
> +    echo "Test using casefolded dir - should fail"
> +    ! mount -t overlay -o
> lowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/work overlay
> /mnt/merged
> 
> So a mount already exists. leftover?

*snort* - -ENOCAFFEINE, I presume.

The new warning doesn't fire after the last mount.

ls: cannot access '/mnt/merged/casefold': Object is remote

But nothing in dmesg. Adding a printk to ovl_mount_dir_check() shows
that it's never called for the 'ls /mnt/merged/casefold' call.

On Tue, May 20, 2025 at 04:33:14PM +0200, Amir Goldstein wrote:
> On Tue, May 20, 2025 at 4:12 PM Kent Overstreet
> > Amir, you've got two widely used filesystem features that conflict and
> > can't be used on the same filesystem.
> >
> > That's _broken_.
> 
> Correct.
> 
> I am saying that IMO a smaller impact (and less user friendly) fix is more
> appropriate way to deal with this problem.

Less user friendly is an understatement.

Obscure errors that only get reported via overloaded standard error
codes is a massive problem today, for _developers_ - have you never had
a day of swearing over trying to track down where in a massive subsystem
an -EINVAL is coming from?

It's even worse for end users that don't know to check the dmesg log.

And I support my code, so these would turn into bug reports coming
across my desk - no thanks; I already get enough weird shit from other
subsystems that I have to look at and at least triage.

> > Users hate partitioning just for separate /boot and /home, having to
> > partition for different applications is horrible. And since overlay fs
> > is used under the hood by docker, and casefolding is used under the hood
> > for running Windows applications, this isn't something people can
> > predict in advance.
> 
> Right, I am not expecting users to partition by application,
> but my question was this:
> 
> When is overlayfs created over a subtree that is only partially case-folded?
> 
> Obviously, docker would create overlayfs on parts of the fs
> and smbd/cygwin could create a case folder subtree on another
> part of the fs.
> I just don't see a common use case when these sections overlap.

Today, you cannot user docker and casefolding on _different parts of_
the same filesystem.

So yees, today users do have to partition by application, or only use
one feature or the other.

This isn't about allowing casefolding and overlayfs to fix on the same
subtree, that would be a bigger project.

> Perhaps I am wrong (please present real world use cases),
> but my claim is that this case is not common enough and therefore,
> a suboptimal EIO error from lookup is good enough to prevert crossing
> over into the case folded zone by mistake, just as EIO on lookup is
> enough to deal with the unsupported use case of modifying
> overlayfs underlying layers with overlay is mounted.
> 
> BTW, it is not enough to claim that there is no case folding for the
> entire subtree to allow the mount.
> For overlayfs to allow d_hash()/d_compare() fs must claim that
> these implementations are the default implementation in all subtree
> or at least that all layers share the same implementation.

I honestly don't know what you're claiming here.

On Tue, May 20, 2025 at 4:44 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Tue, May 20, 2025 at 04:33:14PM +0200, Amir Goldstein wrote:
> > On Tue, May 20, 2025 at 4:12 PM Kent Overstreet
> > > Amir, you've got two widely used filesystem features that conflict and
> > > can't be used on the same filesystem.
> > >
> > > That's _broken_.
> >
> > Correct.
> >
> > I am saying that IMO a smaller impact (and less user friendly) fix is more
> > appropriate way to deal with this problem.
>
> Less user friendly is an understatement.
>
> Obscure errors that only get reported via overloaded standard error
> codes is a massive problem today, for _developers_ - have you never had
> a day of swearing over trying to track down where in a massive subsystem
> an -EINVAL is coming from?
>
> It's even worse for end users that don't know to check the dmesg log.
>
> And I support my code, so these would turn into bug reports coming
> across my desk - no thanks; I already get enough weird shit from other
> subsystems that I have to look at and at least triage.
>
> > > Users hate partitioning just for separate /boot and /home, having to
> > > partition for different applications is horrible. And since overlay fs
> > > is used under the hood by docker, and casefolding is used under the hood
> > > for running Windows applications, this isn't something people can
> > > predict in advance.
> >
> > Right, I am not expecting users to partition by application,
> > but my question was this:
> >
> > When is overlayfs created over a subtree that is only partially case-folded?
> >
> > Obviously, docker would create overlayfs on parts of the fs
> > and smbd/cygwin could create a case folder subtree on another
> > part of the fs.
> > I just don't see a common use case when these sections overlap.
>
> Today, you cannot user docker and casefolding on _different parts of_
> the same filesystem.
>
> So yees, today users do have to partition by application, or only use
> one feature or the other.
>

Didn't say there was no problem.

Argued that your fix is a big gun and not worth the added complexity.

Let's see what Miklos thinks.

> This isn't about allowing casefolding and overlayfs to fix on the same
> subtree, that would be a bigger project.
>
> > Perhaps I am wrong (please present real world use cases),
> > but my claim is that this case is not common enough and therefore,
> > a suboptimal EIO error from lookup is good enough to prevert crossing
> > over into the case folded zone by mistake, just as EIO on lookup is
> > enough to deal with the unsupported use case of modifying
> > overlayfs underlying layers with overlay is mounted.
> >
> > BTW, it is not enough to claim that there is no case folding for the
> > entire subtree to allow the mount.
> > For overlayfs to allow d_hash()/d_compare() fs must claim that
> > these implementations are the default implementation in all subtree
> > or at least that all layers share the same implementation.
>

Nevermind. Misread patch 6.

Thanks,
Amir.

On Tue, May 20, 2025 at 05:13:37PM +0200, Amir Goldstein wrote:
> On Tue, May 20, 2025 at 4:44 PM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Tue, May 20, 2025 at 04:33:14PM +0200, Amir Goldstein wrote:
> > > On Tue, May 20, 2025 at 4:12 PM Kent Overstreet
> > > > Amir, you've got two widely used filesystem features that conflict and
> > > > can't be used on the same filesystem.
> > > >
> > > > That's _broken_.
> > >
> > > Correct.
> > >
> > > I am saying that IMO a smaller impact (and less user friendly) fix is more
> > > appropriate way to deal with this problem.
> >
> > Less user friendly is an understatement.
> >
> > Obscure errors that only get reported via overloaded standard error
> > codes is a massive problem today, for _developers_ - have you never had
> > a day of swearing over trying to track down where in a massive subsystem
> > an -EINVAL is coming from?
> >
> > It's even worse for end users that don't know to check the dmesg log.
> >
> > And I support my code, so these would turn into bug reports coming
> > across my desk - no thanks; I already get enough weird shit from other
> > subsystems that I have to look at and at least triage.
> >
> > > > Users hate partitioning just for separate /boot and /home, having to
> > > > partition for different applications is horrible. And since overlay fs
> > > > is used under the hood by docker, and casefolding is used under the hood
> > > > for running Windows applications, this isn't something people can
> > > > predict in advance.
> > >
> > > Right, I am not expecting users to partition by application,
> > > but my question was this:
> > >
> > > When is overlayfs created over a subtree that is only partially case-folded?
> > >
> > > Obviously, docker would create overlayfs on parts of the fs
> > > and smbd/cygwin could create a case folder subtree on another
> > > part of the fs.
> > > I just don't see a common use case when these sections overlap.
> >
> > Today, you cannot user docker and casefolding on _different parts of_
> > the same filesystem.
> >
> > So yees, today users do have to partition by application, or only use
> > one feature or the other.
> >
> 
> Didn't say there was no problem.
> 
> Argued that your fix is a big gun and not worth the added complexity.
> 
> Let's see what Miklos thinks.
> 
> > This isn't about allowing casefolding and overlayfs to fix on the same
> > subtree, that would be a bigger project.
> >
> > > Perhaps I am wrong (please present real world use cases),
> > > but my claim is that this case is not common enough and therefore,
> > > a suboptimal EIO error from lookup is good enough to prevert crossing
> > > over into the case folded zone by mistake, just as EIO on lookup is
> > > enough to deal with the unsupported use case of modifying
> > > overlayfs underlying layers with overlay is mounted.
> > >
> > > BTW, it is not enough to claim that there is no case folding for the
> > > entire subtree to allow the mount.
> > > For overlayfs to allow d_hash()/d_compare() fs must claim that
> > > these implementations are the default implementation in all subtree
> > > or at least that all layers share the same implementation.
> >
> 
> Nevermind. Misread patch 6.

Since you were asking for use cases - docker & related are pretty widely
used for deploying things that are "unwieldy" within the normal packgae
manager ecosystem - and wine is case study #1 in that, where these days
people want to ship a specific version of wine with applications being
emulated (that's been tested with that application).

But wine wants casefolding, so - hapless user deploys docker image where
casefolding is enabled _but only on the subdir that holds Windows data_,
not the whole image.

Docker mounts the image, but then everything explodes when you try to
use it with what look to the user like impenetrable IO errors.

That's a bad day for someone, or more likely a lot of someones.

On Tue, 20 May 2025 at 17:21, Kent Overstreet <kent.overstreet@linux.dev> wrote:

> Docker mounts the image, but then everything explodes when you try to
> use it with what look to the user like impenetrable IO errors.
>
> That's a bad day for someone, or more likely a lot of someones.

Wouldn't it be docker's responsibility to know that that won't work
with overlayfs?

Any error, whether at startup or during operation is not something the
user will like.

What am I missing?

Thanks,
Miklos

On Tue, May 20, 2025 at 06:40:35PM +0200, Miklos Szeredi wrote:
> On Tue, 20 May 2025 at 17:21, Kent Overstreet <kent.overstreet@linux.dev> wrote:
> 
> > Docker mounts the image, but then everything explodes when you try to
> > use it with what look to the user like impenetrable IO errors.
> >
> > That's a bad day for someone, or more likely a lot of someones.
> 
> Wouldn't it be docker's responsibility to know that that won't work
> with overlayfs?

Why would it be?

> Any error, whether at startup or during operation is not something the
> user will like.
> 
> What am I missing?

A _mount_ error due to misconfiguration is expected operation, and we've
built mechanisms for reporting those errors in a way that users can
understand what's going on.

That's not true for normal file accesses post mount.