On 2025/3/10 17:54, Gao Xiang wrote:
> Hi folks,
>
> The current 32-bit block addressing limits EROFS to a 16TiB maximum
> volume size with 4KiB blocks. However, several new use cases now
> require larger capacity support:
> - Massive datasets for model training to boost random sampling
> performance for each epoch;
> - Object storage clients using EROFS direct passthrough.
>
> This extends core on-disk structures to support 48-bit block addressing,
> such as inodes, device slots, and inode chunks.
>
> In addition, it introduces an mtime field to 32-byte compact inodes for
> basic timestamp support, as well as expands the superblock root NID to
> an 8-byte rootnid_8b for out-of-place update incremental builds.
>
> In order to support larger images beyond 32-bit block addressing and
> efficient indexing of large compression units for compressed data, and
> to better support popular compression algorithms (mainly Zstd) that
> still lack native fixed-sized output compression support, introduce
> byte-oriented encoded extents, so that these compressors are allowed
> to retain their current methods.
>
> Therefore, it speeds up Zstd image building a lot by using:
> Processor: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz * 96
> Dataset: enwik9
> Build time Size Type Command Line
> 3m52.339s 266653696 FO -C524288 -zzstd,22
> 3m48.549s 266174464 FO -E48bit -C524288 -zzstd,22
> 0m12.821s 272134144 FI -E48bit -C1048576 --max-extent-bytes=1048576 -zzstd,22
>
> It has been stress-tested on my local setup for a while without any
> strange happens.
Cool, good work! For this serial,
Acked-by: Chao Yu <chao@kernel.org>
Hoping to grab continuous free slots to check the details, then we can
change it to rvb status before merge window. :)
Thanks,
>
> Thanks,
> Gao Xiang
>
> Gao Xiang (10):
> erofs: get rid of erofs_map_blocks_flatmode()
> erofs: simplify erofs_{read,fill}_inode()
> erofs: add 48-bit block addressing on-disk support
> erofs: implement 48-bit block addressing for unencoded inodes
> erofs: support dot-omitted directories
> erofs: initialize decompression early
> erofs: add encoded extent on-disk definition
> erofs: implement encoded extent metadata
> erofs: support unaligned encoded data
> erofs: enable 48-bit layout support
>
> fs/erofs/Kconfig | 14 +--
> fs/erofs/data.c | 133 +++++++++++-------------
> fs/erofs/decompressor.c | 2 +-
> fs/erofs/dir.c | 7 +-
> fs/erofs/erofs_fs.h | 191 ++++++++++++++++-------------------
> fs/erofs/inode.c | 126 +++++++++++------------
> fs/erofs/internal.h | 30 +++---
> fs/erofs/super.c | 49 ++++-----
> fs/erofs/sysfs.c | 2 +
> fs/erofs/zdata.c | 96 +++++++++---------
> fs/erofs/zmap.c | 166 +++++++++++++++++++++++++-----
> include/trace/events/erofs.h | 2 +-
> 12 files changed, 455 insertions(+), 363 deletions(-)
>