Dear Jonathan,

I wanted to clean up the development-process documentation. There is
however no easy way to break the ice here:

The elephant in the room is that there is some unclear relation between
5.Posting.rst, 6.Followthrough.rst and submitting-patches.rst.
(Yes, I know each document has its own history...; but let us put the
history aside for now.)

Submitting-patches.rst contains information largely put together from
different initial starting points and is partly outdated due to common
workflows with git format-patch and git send-email.

Often when new additions are made, they are somehow added to both
documents with slightly different wording.

Also, looking at the word count (with wc -l) on next-20240227:

  2917 5.Posting.rst
  2136 6.Followthrough.rst
  5878 submitting-patches.rst

So, from the numbers, you would expect that submitting-patches.rst is a
detailing of Posting and Followthrough.
However, it is really difficult to see how submitting-patches.rst would
be a refinement of Posting and Followthrough or if and where it is not.
First, at the moment, the different initial starting points and different
ordering somehow makes it difficult to judge.

Also, the factor of 20% more words really does not indicate much more
content in submitting-patches compared to Posting and Followthrough.

For a simple experiment, I moved the larger parts on the tags
(signed-off-by, co-developed-by, acked-by, reported-by, etc.) into a
separate document and then ran the numbers on submitting-patches again:

  4329 submitting-patches.rst

Nowt, the size of submitting-patches is actually below Posting and
Followthrough.

So, the difficult task to reach a coherent process description is to see
some relation between these documents and then go through the editorial
changes. I have come up with this kind of vision:

Phase 1: Clean up submitting patches

  Topics/Statements that can be easily cleaned up first do not get in
  the way (at least mentally) when trying to understand the next steps.
  
  E.g., as an experiment I moved the details on tags into a separate
  document.
  
Phase 2: Make submitting-patches have one clear temporal flow.

  The top-level structure should basically be along the temporal order of
  things: Prepare a patch, Post a patch, Respond to review, Send reworked
  patches, Be patient before resending

  "No MIME, no links, no compression, no attachments. Just plain text"
  needs to be reworked into "Send your patch".
  
  The content from the canonical patch format needs to evaluated on
  relevance with a git workflow and the important pieces need to be
  included into the temporal flow.

  In other words, it is worth describing "the canonical patch format" much
  more from what the submitter may add where and in which format rather
  than explaining the purpose of some things that git format-patch does by
  default and usually nobody would play around with.

  Probably, prepare a patch will need to be broken down a bit in some
  smaller steps. The "Send reworked patch" section is new and needs to
  include content distributed throughout submitting-patches.
  
Phase 3: Merge the pieces of content from Posting and Followthrough into
submitting patches if it adds something to that document.

  When both documents roughly cover the topics of similar depth, we look
  fine-grained into how to construct the one document that has the best
  from both documents.
  
Phase 4: Remove Posting and Followthrough and simply replace it in the
process description with submitting patches.
  
  In some way, when both documents cover the topic in similar depth either
  document could be deleted.
  However, the name "submitting patches" is probably already stuck too much
  with the community; I have seen many presentations referring to
  submitting patches, but I have not seen anyone referring to '5.Posting'
  in any presentation.

  Also, the number of references---excluding translations---to Posting and
  Followthrough, submitting_patches.rst is 3, 1 and 41, respectively.
  In fact, the two references in handling-regressions mention Posting just
  as further reference next to submitting patches, clearly just indicating
  this kind of duplication. So, submitting-patches is much more stuck with
  the community at the current state and once the content from Posting is
  added to submitting-patches, Posting will not be missed.

  Further, this requires to rewrite the process description intro
  and the intro of submitting patches a bit, such that if readers:

  - just jump into submitting-patches from the top page,
  - or are going through the development-process from cover to cover
    (coming from its section 4 and moving on to section 7)

  they see a roughly consistent flow of thought and suitable introduction.
  But I think this should be feasible.

Let us see how long it takes me to work through this and convince the
reviewers and future readers that we are moving a good direction.

So, here are some first changes to Phase 1 and Phase 2.

Current state:

I spend roughly two days of work---besides the usual distractions---on
this topic, digging into the documents and trying to make a plan.

Obviously, the final goal is not reached with this series, but I would
like to get first feedback and hope that we can get the first patches
generally accepted (but not necessarily included into the repository yet
if only truly accepted when seeing the full picture of changes) to
continue the rework with some backing confidence that this is not all in
vain.

Please let me know if this is going in the right direction and if some
patches would already be accepted to be included upfront to lower the
risks and conflicts when continuing the editorial work.

Well, long text... some short simple patches for now.


Best regards,

Lukas


Lukas Bulwahn (3):
  docs: submitting-patches: divert focus from PATCH in the subject line
  docs: submitting-patches: move split_changes before describe_change
  docs: submitting-patches: move backtraces to patch description

 Documentation/process/submitting-patches.rst | 119 +++++++++----------
 1 file changed, 58 insertions(+), 61 deletions(-)

-- 
2.43.2

Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:

> Dear Jonathan,
>
> I wanted to clean up the development-process documentation. There is
> however no easy way to break the ice here:
>
> The elephant in the room is that there is some unclear relation between
> 5.Posting.rst, 6.Followthrough.rst and submitting-patches.rst.
> (Yes, I know each document has its own history...; but let us put the
> history aside for now.)

FWIW, the objective of those two documents is quite different; one is a
high-level overview of how the development process as a whole works, the
other is a detailed guide to submitting work for consideration.

> Submitting-patches.rst contains information largely put together from
> different initial starting points and is partly outdated due to common
> workflows with git format-patch and git send-email.

You should have seen it before I thrashed it a few years back :)

> For a simple experiment, I moved the larger parts on the tags
> (signed-off-by, co-developed-by, acked-by, reported-by, etc.) into a
> separate document and then ran the numbers on submitting-patches again:
>
>   4329 submitting-patches.rst
>
> Nowt, the size of submitting-patches is actually below Posting and
> Followthrough.

I don't think we should be driven by word counts.  I do think that
moving a bunch of information on tags to its own document could make
sense.

> So, the difficult task to reach a coherent process description is to see
> some relation between these documents and then go through the editorial
> changes. I have come up with this kind of vision:
>
> Phase 1: Clean up submitting patches
>
>   Topics/Statements that can be easily cleaned up first do not get in
>   the way (at least mentally) when trying to understand the next steps.
>   
>   E.g., as an experiment I moved the details on tags into a separate
>   document.

Fine.

> Phase 2: Make submitting-patches have one clear temporal flow.
>
>   The top-level structure should basically be along the temporal order of
>   things: Prepare a patch, Post a patch, Respond to review, Send reworked
>   patches, Be patient before resending

This makes sense as well.  I wonder if splitting the document along some
of those lines might also be a good idea, with submitting-patches.rst
becoming a relatively short overview deferring details to the others.
This is one of the most important docs we have, and it's far too much
for people to engage with all at once.

> Phase 3: Merge the pieces of content from Posting and Followthrough into
> submitting patches if it adds something to that document.
>
>   When both documents roughly cover the topics of similar depth, we look
>   fine-grained into how to construct the one document that has the best
>   from both documents.
>   
> Phase 4: Remove Posting and Followthrough and simply replace it in the
> process description with submitting patches.

In broad terms, this seems like a good direction to me.

Again, let's remember the different purposes of these documents.  The
development-process document is an overall description of the process,
so it doesn't need the details.  But when you say:

> Posting will not be missed.

I don't entirely agree.  But I don't doubt it could be a fraction of
what it is now.

> So, here are some first changes to Phase 1 and Phase 2.

At a first glance, these changes seem fine.  I think I'll hold them
until after the merge window so that others can think about what you're
up to, but I suspect there will be no reason not to apply this first set
then.

Thanks for working on this material; it's some of the most important we
have and it definitely needs some attention.

jon

On Sun, Mar 3, 2024 at 5:31 PM Jonathan Corbet <corbet@lwn.net> wrote:
>
> Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:
>
> > Dear Jonathan,
> >
> > I wanted to clean up the development-process documentation. There is
> > however no easy way to break the ice here:
> >
> > The elephant in the room is that there is some unclear relation between
> > 5.Posting.rst, 6.Followthrough.rst and submitting-patches.rst.
> > (Yes, I know each document has its own history...; but let us put the
> > history aside for now.)
>
> FWIW, the objective of those two documents is quite different; one is a
> high-level overview of how the development process as a whole works, the
> other is a detailed guide to submitting work for consideration.
>

Yes, that _objective_ is clear when reading the documents.
However, unfortunately, the detailed guide to submitting work for
consideration in submitting-patches.rst really is not that much more
detailed than what 5.Posting and 6.Followthrough already recommend.
A lot of the "details" in submitting-patches.rst is then also just
details on topics that are much more an explanation than actual
recommendation for specific actions.

Let me clean things up in submitting-patches, and then start a proper
comparison.

> > Submitting-patches.rst contains information largely put together from
> > different initial starting points and is partly outdated due to common
> > workflows with git format-patch and git send-email.
>
> You should have seen it before I thrashed it a few years back :)
>
> > For a simple experiment, I moved the larger parts on the tags
> > (signed-off-by, co-developed-by, acked-by, reported-by, etc.) into a
> > separate document and then ran the numbers on submitting-patches again:
> >
> >   4329 submitting-patches.rst
> >
> > Nowt, the size of submitting-patches is actually below Posting and
> > Followthrough.
>
> I don't think we should be driven by word counts.  I do think that
> moving a bunch of information on tags to its own document could make
> sense.
>
> > So, the difficult task to reach a coherent process description is to see
> > some relation between these documents and then go through the editorial
> > changes. I have come up with this kind of vision:
> >
> > Phase 1: Clean up submitting patches
> >
> >   Topics/Statements that can be easily cleaned up first do not get in
> >   the way (at least mentally) when trying to understand the next steps.
> >
> >   E.g., as an experiment I moved the details on tags into a separate
> >   document.
>
> Fine.
>
> > Phase 2: Make submitting-patches have one clear temporal flow.
> >
> >   The top-level structure should basically be along the temporal order of
> >   things: Prepare a patch, Post a patch, Respond to review, Send reworked
> >   patches, Be patient before resending
>
> This makes sense as well.  I wonder if splitting the document along some
> of those lines might also be a good idea, with submitting-patches.rst
> becoming a relatively short overview deferring details to the others.
> This is one of the most important docs we have, and it's far too much
> for people to engage with all at once.
>

I understand that people nowadays do not read prose from top to
bottom, as soon as it exceeds a certain length. So, for sure, we can
consider splitting the current content into multiple pieces and add
links between them. However, I also want to avoid that we have say 15
documents of a hundred lines, and you are always jumping
back-and-forth in your web browser while reading. I think the split is
going to be into two or three documents if at all.

I will do some experiments and suggest some splitting.

> > Phase 3: Merge the pieces of content from Posting and Followthrough into
> > submitting patches if it adds something to that document.
> >
> >   When both documents roughly cover the topics of similar depth, we look
> >   fine-grained into how to construct the one document that has the best
> >   from both documents.
> >
> > Phase 4: Remove Posting and Followthrough and simply replace it in the
> > process description with submitting patches.
>
> In broad terms, this seems like a good direction to me.
>
> Again, let's remember the different purposes of these documents.  The
> development-process document is an overall description of the process,
> so it doesn't need the details.  But when you say:
>
> > Posting will not be missed.
>
> I don't entirely agree.  But I don't doubt it could be a fraction of
> what it is now.
>

When I say "Posting will not be missed", I mean the name
"5.Posting.rst" will not be missed, as the future submitting-patches,
partially existent on my hard disk right now, includes the best of
5.Posting.rst as it is now, namely the natural flow of the
explanation, the good style of writing, being precise and concise and
the ability to address all audiences with a suitable text, e.g.,
newcomers and experienced kernel developers enjoy reading it. Some
important information in 5.Posting.rst should really also be mentioned
in submitting-patches.rst.

I think if submitting-patches.rst is structured and written well, the
development process description can go from 4. Getting the code right
to "5.Submitting patches" and the readers would not even notice that
they once originated from very different sources and authors.

> > So, here are some first changes to Phase 1 and Phase 2.
>
> At a first glance, these changes seem fine.  I think I'll hold them
> until after the merge window so that others can think about what you're
> up to, but I suspect there will be no reason not to apply this first set
> then.
>
> Thanks for working on this material; it's some of the most important we
> have and it definitely needs some attention.
>

I will continue working on it and see what I consider stable enough in
moving around that it deserves to be posted to the mailing list. While
working on the document, it is unfortunately a lot of temporary
movement back and forth, or huge changes at once and it is a bit
difficult to then extract the next natural change to propose, but I
will see how I can present this best piece by piece.


Lukas

On 03.03.24 17:31, Jonathan Corbet wrote:
> Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:
>> I wanted to clean up the development-process documentation. There is
>> however no easy way to break the ice here:
>>
>> The elephant in the room is that there is some unclear relation between
>> 5.Posting.rst, 6.Followthrough.rst and submitting-patches.rst.
>> (Yes, I know each document has its own history...; but let us put the
>> history aside for now.)
> 
> FWIW, the objective of those two documents is quite different; one is a
> high-level overview of how the development process as a whole works, the
> other is a detailed guide to submitting work for consideration.

Sorry, I'm slightly confused here, so I have to ask: which is which?

Due to the "*essential*" in the headline of submitting-patches.rst and
its "For *detailed* information on how the kernel development process
works, see Documentation/process/development-process.rst" in the intro
make it sounds to me like submitting-patches.rst should be the one with
the high-level overview. But...

> Again, let's remember the different purposes of these documents.  The
> development-process document is an overall description of the process,
> so it doesn't need the details.

...this makes it sounds like you consider it the other way around. And
for me that feels the wrong, as why describe the overall process in
detail, but leave the most important part of the process to some other
document?

/me wonders what he is missing

Ciao, Thorsten

Thorsten Leemhuis <linux@leemhuis.info> writes:

> On 03.03.24 17:31, Jonathan Corbet wrote:
>> Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:
>>> I wanted to clean up the development-process documentation. There is
>>> however no easy way to break the ice here:
>>>
>>> The elephant in the room is that there is some unclear relation between
>>> 5.Posting.rst, 6.Followthrough.rst and submitting-patches.rst.
>>> (Yes, I know each document has its own history...; but let us put the
>>> history aside for now.)
>> 
>> FWIW, the objective of those two documents is quite different; one is a
>> high-level overview of how the development process as a whole works, the
>> other is a detailed guide to submitting work for consideration.
>
> Sorry, I'm slightly confused here, so I have to ask: which is which?
>
> Due to the "*essential*" in the headline of submitting-patches.rst and
> its "For *detailed* information on how the kernel development process
> works, see Documentation/process/development-process.rst" in the intro
> make it sounds to me like submitting-patches.rst should be the one with
> the high-level overview. But...
>
>> Again, let's remember the different purposes of these documents.  The
>> development-process document is an overall description of the process,
>> so it doesn't need the details.
>
> ...this makes it sounds like you consider it the other way around. And
> for me that feels the wrong, as why describe the overall process in
> detail, but leave the most important part of the process to some other
> document?
>
> /me wonders what he is missing

The series of files starting with Documentation/process/1.Intro.rst was
meant to describe the whole of the development process to a wider
audience; I originally wrote it as a project for the Linux Foundation.
It covers far more than the business of putting up patches for
consideration - development cycles and all that.

submitting-patches.rst, instead, covers the details of getting code
considered for merging; it is intended to be read by the people actually
trying to do that work.

One document describes what the pieces of the car are and how they work
together to get you to the pub.  The other gives all of the steps for
working on the brakes without causing accidents.  They both fit as part
of a larger body of documentation, but they are definitely not the same
document.

Make sense?

jon

On 05.03.24 13:59, Jonathan Corbet wrote:
> Thorsten Leemhuis <linux@leemhuis.info> writes:
>> On 03.03.24 17:31, Jonathan Corbet wrote:
>>> Lukas Bulwahn <lukas.bulwahn@gmail.com> writes:
>>>> I wanted to clean up the development-process documentation. There is
>>>> however no easy way to break the ice here:
>>>>
>>>> The elephant in the room is that there is some unclear relation between
>>>> 5.Posting.rst, 6.Followthrough.rst and submitting-patches.rst.
>>>> (Yes, I know each document has its own history...; but let us put the
>>>> history aside for now.)
>>>
>>> FWIW, the objective of those two documents is quite different; one is a
>>> high-level overview of how the development process as a whole works, the
>>> other is a detailed guide to submitting work for consideration.
>>
>> Sorry, I'm slightly confused here, so I have to ask: which is which?
>>
>> Due to the "*essential*" in the headline of submitting-patches.rst and
>> its "For *detailed* information on how the kernel development process
>> works, see Documentation/process/development-process.rst" in the intro
>> make it sounds to me like submitting-patches.rst should be the one with
>> the high-level overview. But...
>>
>>> Again, let's remember the different purposes of these documents.  The
>>> development-process document is an overall description of the process,
>>> so it doesn't need the details.
>>
>> ...this makes it sounds like you consider it the other way around. And
>> for me that feels the wrong, as why describe the overall process in
>> detail, but leave the most important part of the process to some other
>> document?
>>
>> /me wonders what he is missing
> 
> The series of files starting with Documentation/process/1.Intro.rst was
> meant to describe the whole of the development process to a wider
> audience; I originally wrote it as a project for the Linux Foundation.
> It covers far more than the business of putting up patches for
> consideration - development cycles and all that.
> 
> submitting-patches.rst, instead, covers the details of getting code
> considered for merging; it is intended to be read by the people actually
> trying to do that work.
> 
> One document describes what the pieces of the car are and how they work
> together to get you to the pub.  The other gives all of the steps for
> working on the brakes without causing accidents.  They both fit as part
> of a larger body of documentation, but they are definitely not the same
> document.

Thx for the clarification. And of course both fit in a larger body. It
seems some of the word used in the intros of both documents made me
assume it was the other way around at some point in the past and then it
stuck. Wondering if that's just me or if that happened to others as
well. Whatever, if Lukas will realize his plans I guess the different
target audiences will become more obvious over time.

Thx again! Ciao, Thorsten