So there's a lot of new stuff since the first posting:

 - Printbufs have been broken up into multiple patches that each add distinct
   functionality - this is intended to make it easier to review and to see
   what's used for what

 - Printbufs now support both auto-heap allocated buffers, and external/static
   buffers: this was required for the vsprintf.c refactoring, and means they're
   (almost) a direct replacement for seq_buf

 - The big thing: a new %pf(%p) format string extension for calling pretty
   printers from printf (idea from Matthew Wilcox)

   This is intended to replace most of our other format string extensions: e.g.
   instead of writing
     printf("%pg", bdev);

   You'd now write
     printf("%pf(%p)", bdev_name, bdev);

   The advantage of this is that pretty printers no longer have to live in
   lib/vsprintf.c, and they're much more discoverable - you can cscope to the
   pretty printer!

   And my hope is that this will help induce people to write lots more pretty
   printers; since they can now live with the code they're printing and don't
   require touching code in vsprintf.c, there's less friction to creating new
   ones.

   We hope to standardize this extension as %(%p), but since gcc's printf format
   checking doesn't yet understand that we're going with %pf(%p) for now.

   Currently, we only support pointer arguments to pretty-printers. I think we
   can improve this in the future to support at least integer arguments as well,
   i.e. "%(%u)" will eventually work. This will require using libffi to do it
   correctly, but it looks like libffi is nearly suitable for in kernel use (it
   supports all linux architectures, and configured with the features we want it
   compiles down to practically nothing).

 - Massive vsprintf.c refactoring

   printbufs are now the core data structure used by vsprintf.c - we're not
   passing around a bunch of raw char pointers anymore! yay!

   This gets us a sane standard calling convention for pretty printers - i.e.,
   we need this for %pf(%p).

   Couple notes on the refactoring:

   - printf_spec has become a dumping ground of state, passed everywhere and
     used inconsistently. The refactoring attempts to improve this, and
     centralize printf_spec handling as much as possible near/in the top level
     code that handles format strings. Some %p extensions use
     field/width/precision in nonstandard ways; the refactoring patches make it
     clearer where this is going on.

   - a _lot_ of pretty printers were allocating secondary buffers on the stack,
     mainly to avoid ever writing past the terminating null in the output
     buffer. There was a test that checked for this, but it had no documentation
     where the requirement came from nor does that requirement make any sense,
     so I deleted it (if anyone knows anything about it, speak up!). The code
     now takes the approach of just writing to the output buffer and then
     truncating afterwards if required by the precision argument.

     Yay, less stack usage!

   - format string parsing is still a mess: I'd like to consolidate that to only
     happen in one place, but that's going to be a much more involved
     refactoring - and if we just switch to new-style calling pretty printers
     directly, we'll be able to just delete all that code.

 - More seq_buf conversions

   Using printbufs to clean up vsprintf.c meant adding a second, non
   heap-allocated mode, so printbufs now do almost everything seq_buf does -
   seq_buf has a read_pos member for some reason (tracing?) that I didn't get
   into.

   So now seq_buf is just used by the tracing code, and that can also probably
   be converted to printbuf, but seq_buf is Steven's thing so I'll let him take
   a look before getting into that.

Kent Overstreet (28):
  lib/printbuf: New data structure for printing strings
  vsprintf: Convert to printbuf
  vsprintf: %pf(%p)
  lib/string_helpers: string_get_size() now returns characters wrote
  lib/printbuf: Heap allocation
  lib/printbuf: Tabstops, indenting
  lib/printbuf: Unit specifiers
  lib/pretty-printers: pr_string_option(), pr_bitflags()
  vsprintf: Improve number()
  vsprintf: pr_u64_minwidth(), pr_u64()
  vsprintf: Lift pr_hex_bytes() out from hex_string()
  test_printf: Drop requirement that sprintf not write past nul
  vsprintf: Start consolidating printf_spec handling
  vsprintf: Refactor resource_string()
  vsprintf: Refactor fourcc_string()
  vsprintf: Refactor ip_addr_string()
  vsprintf: Refactor mac_address_string()
  vsprintf: time_and_date() no longer takes printf_spec
  vsprintf: flags_string() no longer takes printf_spec
  vsprintf: Refactor device_node_string, fwnode_string
  vsprintf: Refactor hex_string, bitmap_string_list, bitmap_string
  Input/joystick/analog: Convert from seq_buf -> printbuf
  mm/memcontrol.c: Convert to printbuf
  clk: tegra: bpmp: Convert to printbuf
  tools/testing/nvdimm: Convert to printbuf
  powerpc: Convert to printbuf
  x86/resctrl: Convert to printbuf
  PCI/P2PDMA: Convert to printbuf

 Documentation/core-api/printk-formats.rst |   19 +
 arch/powerpc/kernel/process.c             |   16 +-
 arch/powerpc/kernel/security.c            |   75 +-
 arch/powerpc/platforms/pseries/papr_scm.c |   34 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    |   16 +-
 drivers/clk/tegra/clk-bpmp.c              |   21 +-
 drivers/input/joystick/analog.c           |   37 +-
 drivers/pci/p2pdma.c                      |   17 +-
 include/linux/kernel.h                    |    4 +
 include/linux/pretty-printers.h           |   11 +
 include/linux/printbuf.h                  |  225 +++
 include/linux/string_helpers.h            |    8 +-
 lib/Makefile                              |    2 +-
 lib/pretty-printers.c                     |   81 ++
 lib/printbuf.c                            |  252 ++++
 lib/string_helpers.c                      |   18 +-
 lib/test_printf.c                         |   23 +-
 lib/vsprintf.c                            | 1612 ++++++++++-----------
 mm/memcontrol.c                           |   68 +-
 tools/testing/nvdimm/test/ndtest.c        |   22 +-
 20 files changed, 1527 insertions(+), 1034 deletions(-)
 create mode 100644 include/linux/pretty-printers.h
 create mode 100644 include/linux/printbuf.h
 create mode 100644 lib/pretty-printers.c
 create mode 100644 lib/printbuf.c

-- 
2.36.0

On Thu 2022-05-19 13:23:53, Kent Overstreet wrote:
> So there's a lot of new stuff since the first posting:

>  - Printbufs have been broken up into multiple patches that each add distinct
>    functionality - this is intended to make it easier to review and to see
>    what's used for what

It is great that it is split. Also it is great to see all the ideas.
But I would really prefer to somehow split it to make it easier
for review and rebasing.

I see the following "independent" parts:

  1. Add simple API that allows to replace @len, @buf, @end in vsprintf.c
     by @printbuf. I agree that the code looks better and more safe.

  2. Clean up of printf_spec. It would be great. But I do not like
     some parts. For example, si_units, human_readable_units
     are not property of the buffer. They are specific for a
     particular substring.

  3. New %p(%p) format. It really needs deep thinking. It is a
     ticket for potential big troubles. It is one patch that
     might be introduced and discussed anytime once we have
     the simple buffer API.

  3. Replace seq_buf. Steven Rostedt has to agree with it. Honestly,
     I do not see any improvement. The patches mostly do 1:1 replacement
     of one API with another.

  4. Heap allocated buffer. I am not sure if it is really needed.
     The patchset adds 3 users. IMHO, small static buffer would be
     perfectly fine for 2 of them. I personally do not like the error
     handling and the need to call exit.

  5. All the fancy stuff (pr_tab(), pr_string_option()). The patchset
    does not add any user for them.


I am going to comment the particular patches. It might take some
time. The patchset is really huge. It would really help to split it.

Best Regards,
Petr

On Thu, May 26, 2022 at 04:44:20PM +0200, Petr Mladek wrote:
> On Thu 2022-05-19 13:23:53, Kent Overstreet wrote:
> > So there's a lot of new stuff since the first posting:
> 
> >  - Printbufs have been broken up into multiple patches that each add distinct
> >    functionality - this is intended to make it easier to review and to see
> >    what's used for what
> 
> It is great that it is split. Also it is great to see all the ideas.
> But I would really prefer to somehow split it to make it easier
> for review and rebasing.
> 
> I see the following "independent" parts:
> 
>   1. Add simple API that allows to replace @len, @buf, @end in vsprintf.c
>      by @printbuf. I agree that the code looks better and more safe.
> 
>   2. Clean up of printf_spec. It would be great. But I do not like
>      some parts. For example, si_units, human_readable_units
>      are not property of the buffer. They are specific for a
>      particular substring.

Not in conventional usage - these are properties that are set globally when
building up a string: consider an -h flag to a userspace utility.

> 
>   3. New %p(%p) format. It really needs deep thinking. It is a
>      ticket for potential big troubles. It is one patch that
>      might be introduced and discussed anytime once we have
>      the simple buffer API.

It's split out into a separate patch - discuss away!

>   3. Replace seq_buf. Steven Rostedt has to agree with it. Honestly,
>      I do not see any improvement. The patches mostly do 1:1 replacement
>      of one API with another.

It was necessary to avoid code duplication, which Christoph didn't like, and
seq_buf wasn't quite right for what I was trying to do and Steven didn't want to
change it. (read_pos is tracing specific and doesn't have anything to do with
printing, some of the semantics had to be tweaked to support snprintf, and the
name was wrong... :)

>   4. Heap allocated buffer. I am not sure if it is really needed.
>      The patchset adds 3 users. IMHO, small static buffer would be
>      perfectly fine for 2 of them. I personally do not like the error
>      handling and the need to call exit.
> 
>   5. All the fancy stuff (pr_tab(), pr_string_option()). The patchset
>     does not add any user for them.

Heap allocated buffers and tabstops: these are things that I've been using in
bcachefs, and have quickly become necessities - I included them because I think
others will soon find them valuable (e.g. /proc has a lot of files that would be
more readable if formatted with tabstops).

pr_string_option(): this isn't anything really new, we've got a lot of pretty
printers and string helpers of this nature that need to be better organized and
given a common calling convention, I included this as code I've got that's been
useful and I think does it cleanly. I have more future plans for pretty printer
cleanup.