[PATCH v6 00/11] kbuild, kconfig: Add support for conflict resolution

Ole Schuerks posted 11 patches 1 year, 3 months ago
Documentation/kbuild/kconfig.rst    |   56 +
scripts/include/list.h              |   49 +
scripts/kconfig/.gitignore          |    1 +
scripts/kconfig/Makefile            |   11 +-
scripts/kconfig/cf_constraints.c    | 1779 ++++++++++++++++++++++++
scripts/kconfig/cf_constraints.h    |   24 +
scripts/kconfig/cf_defs.h           |  391 ++++++
scripts/kconfig/cf_expr.c           | 2003 +++++++++++++++++++++++++++
scripts/kconfig/cf_expr.h           |  181 +++
scripts/kconfig/cf_rangefix.c       | 1136 +++++++++++++++
scripts/kconfig/cf_rangefix.h       |   21 +
scripts/kconfig/cf_utils.c          |  980 +++++++++++++
scripts/kconfig/cf_utils.h          |  112 ++
scripts/kconfig/cfoutconfig.c       |  149 ++
scripts/kconfig/configfix.c         |  351 +++++
scripts/kconfig/configfix.h         |   31 +
scripts/kconfig/expr.h              |   17 +
scripts/kconfig/loader.gif          |  Bin 0 -> 4177 bytes
scripts/kconfig/picosat_functions.c |   74 +
scripts/kconfig/picosat_functions.h |   35 +
scripts/kconfig/qconf.cc            |  676 ++++++++-
scripts/kconfig/qconf.h             |  111 ++
22 files changed, 8184 insertions(+), 4 deletions(-)
create mode 100644 scripts/kconfig/cf_constraints.c
create mode 100644 scripts/kconfig/cf_constraints.h
create mode 100644 scripts/kconfig/cf_defs.h
create mode 100644 scripts/kconfig/cf_expr.c
create mode 100644 scripts/kconfig/cf_expr.h
create mode 100644 scripts/kconfig/cf_rangefix.c
create mode 100644 scripts/kconfig/cf_rangefix.h
create mode 100644 scripts/kconfig/cf_utils.c
create mode 100644 scripts/kconfig/cf_utils.h
create mode 100644 scripts/kconfig/cfoutconfig.c
create mode 100644 scripts/kconfig/configfix.c
create mode 100644 scripts/kconfig/configfix.h
create mode 100644 scripts/kconfig/loader.gif
create mode 100644 scripts/kconfig/picosat_functions.c
create mode 100644 scripts/kconfig/picosat_functions.h
[PATCH v6 00/11] kbuild, kconfig: Add support for conflict resolution
Posted by Ole Schuerks 1 year, 3 months ago
Hi,

Configuring a kernel requires a forward enabling approach where one enables
each option one needs at a time. If one enables an option that selects
other options, these options are no longer de-selectable by design.
Likewise, if one has enabled an option which creates a conflict with a
secondary option one wishes to enable, one cannot easily enable that
secondary option, unless one is willing to spend time analyzing the
dependencies that led to this conflict. Sometimes, these conflicts are not
easy to understand [0,1].

This patch series (for linux-next) provides support to enable users to
express their desired target configuration and display possible resolutions
to their conflicts. This support is provided within xconfig.

Conflict resolution is provided by translating kconfig's configuration
option tree to a propositional formula, and then allowing our resolution
algorithm, which uses a SAT solver (picosat, implemented in C) calculate
the possible fixes for an expressed target kernel configuration.

New UI extensions are made to xconfig with panes and buttons to allow users
to express new desired target options, calculate fixes, and apply any of
found solutions.

We created a separate test infrastructure that we used to validate the
correctness of the suggestions made. It shows that our resolution algorithm
resolves around 95% of the conflicts. We plan to incorporate this with a
later patch series.

We envision that our translation of the kconfig option tree into a
propositional formula could potentially also later be repurposed to address
other problems. An example is checking the consistency between the use of
ifdefs and logic expressed in kconfig files. We suspect that this could,
for example, help avoid invalid kconfig configurations and help with ifdef
maintenance.

You can see a YouTube video demonstrating this work [2]. This effort is
part of the kernelnewbies Kconfig-SAT project [3], the approach and effort
is also explained in detail in our paper [4]. The results from the
evaluation have significantly improved since then: Around 80 % of the
conflicts could be resolved, and 99.9 % of the generated fixes resolved the
conflict. It is also our attempt at contributing back to the kernel
community, whose configurator researchers studied a lot.

Patches applicable to next-20241025.

[0] https://gsd.uwaterloo.ca/sites/default/files/vamos12-survey.pdf
[1] https://www.linux-magazine.com/Issues/2021/244/Kconfig-Deep-Dive
[2] https://www.youtube.com/watch?v=vn2JgK_PTbc
[3] https://kernelnewbies.org/KernelProjects/kconfig-sat
[4] http://www.cse.chalmers.se/~bergert/paper/2021-icseseip-configfix.pdf

Thanks from the team! (and thanks to Luis Chamberlain for guiding us here)

Co-developed-by: Patrick Franz <deltaone@debian.org>
Signed-off-by: Patrick Franz <deltaone@debian.org>
Co-developed-by: Ibrahim Fayaz <phayax@gmail.com>
Signed-off-by: Ibrahim Fayaz <phayax@gmail.com>
Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>
Tested-by: Evgeny Groshev <eugene.groshev@gmail.com>
Suggested-by: Sarah Nadi <nadi@ualberta.ca>
Suggested-by: Thorsten Berger <thorsten.berger@rub.de>
Signed-off-by: Thorsten Berger <thorsten.berger@rub.de>
Signed-off-by: Ole Schuerks <ole0811sch@gmail.com>

Changelog v6:
* remove script for manually installing PicoSAT
* adapt documentation and help text in the GUI accordingly
* convert Qt signal/slot connects to new style

Changelog v5:
* use lists from scripts/include/list.h
* use PicoSAT as a dynamically loaded library
* Fix GUI bug that made the displayed tables editable
* Allow cycling through the desired values of a symbol in the conflict view
  in the GUI by clicking on the cell
* Fix usage of "NO" instead of "N" etc. in some places in the GUI
* Improve function naming
* Add documentation
* Simlify xcalloc to xmalloc in some places
* Fix allocation bug in fexpr_add_to_satmap() and init_data()
* Remove functions pexpr_eliminate_dups() and print_expr()

Ole Schuerks (11):
  kconfig: Add PicoSAT interface
  kbuild: Add list_size, list_at_index, list_for_each_from
  kconfig: Add definitions
  kconfig: Add files for building constraints
  kconfig: Add files for handling expressions
  kconfig: Add files for RangeFix
  kconfig: Add files with utility functions
  kconfig: Add tools
  kconfig: Add xconfig-modifications
  kconfig: Add loader.gif
  kconfig: Add documentation for the conflict resolver

 Documentation/kbuild/kconfig.rst    |   56 +
 scripts/include/list.h              |   49 +
 scripts/kconfig/.gitignore          |    1 +
 scripts/kconfig/Makefile            |   11 +-
 scripts/kconfig/cf_constraints.c    | 1779 ++++++++++++++++++++++++
 scripts/kconfig/cf_constraints.h    |   24 +
 scripts/kconfig/cf_defs.h           |  391 ++++++
 scripts/kconfig/cf_expr.c           | 2003 +++++++++++++++++++++++++++
 scripts/kconfig/cf_expr.h           |  181 +++
 scripts/kconfig/cf_rangefix.c       | 1136 +++++++++++++++
 scripts/kconfig/cf_rangefix.h       |   21 +
 scripts/kconfig/cf_utils.c          |  980 +++++++++++++
 scripts/kconfig/cf_utils.h          |  112 ++
 scripts/kconfig/cfoutconfig.c       |  149 ++
 scripts/kconfig/configfix.c         |  351 +++++
 scripts/kconfig/configfix.h         |   31 +
 scripts/kconfig/expr.h              |   17 +
 scripts/kconfig/loader.gif          |  Bin 0 -> 4177 bytes
 scripts/kconfig/picosat_functions.c |   74 +
 scripts/kconfig/picosat_functions.h |   35 +
 scripts/kconfig/qconf.cc            |  676 ++++++++-
 scripts/kconfig/qconf.h             |  111 ++
 22 files changed, 8184 insertions(+), 4 deletions(-)
 create mode 100644 scripts/kconfig/cf_constraints.c
 create mode 100644 scripts/kconfig/cf_constraints.h
 create mode 100644 scripts/kconfig/cf_defs.h
 create mode 100644 scripts/kconfig/cf_expr.c
 create mode 100644 scripts/kconfig/cf_expr.h
 create mode 100644 scripts/kconfig/cf_rangefix.c
 create mode 100644 scripts/kconfig/cf_rangefix.h
 create mode 100644 scripts/kconfig/cf_utils.c
 create mode 100644 scripts/kconfig/cf_utils.h
 create mode 100644 scripts/kconfig/cfoutconfig.c
 create mode 100644 scripts/kconfig/configfix.c
 create mode 100644 scripts/kconfig/configfix.h
 create mode 100644 scripts/kconfig/loader.gif
 create mode 100644 scripts/kconfig/picosat_functions.c
 create mode 100644 scripts/kconfig/picosat_functions.h

-- 
2.39.5
Re: [PATCH v6 00/11] kbuild, kconfig: Add support for conflict resolution
Posted by Luis Chamberlain 1 year, 2 months ago
On Mon, Oct 28, 2024 at 04:49:38AM +0100, Ole Schuerks wrote:
> Hi,
> 
> Configuring a kernel requires a forward enabling approach where one enables
> each option one needs at a time. If one enables an option that selects
> other options, these options are no longer de-selectable by design.
> Likewise, if one has enabled an option which creates a conflict with a
> secondary option one wishes to enable, one cannot easily enable that
> secondary option, unless one is willing to spend time analyzing the
> dependencies that led to this conflict. Sometimes, these conflicts are not
> easy to understand [0,1].

This paragraph is a bit too researchy, move it to:

https://kernelnewbies.org/KernelProjects/kconfig-sat

> This patch series (for linux-next) provides support to enable users to
> express their desired target configuration and display possible resolutions
> to their conflicts. This support is provided within xconfig.

This should be your next cover letter's first paragraph.

> Conflict resolution is provided by translating kconfig's configuration
> option tree to a propositional formula, and then allowing our resolution
> algorithm, which uses a SAT solver (picosat, implemented in C) calculate
> the possible fixes for an expressed target kernel configuration.

Just say something like:

Conflict resolution is provided by translating kconfig's configuration
option tree to a propositional formula and allowing our resolution
algorithm and picosat to calculate the possible fixes for an expressed
target kernel configuration. Picosat is the smallest and most robust C
SAT solver we could find which is also GPL compatible. <insert
information about the efforts done to also provide debian packages
for picosat and briefly mention how picosat is used as library>.

> New UI extensions are made to xconfig with panes and buttons to allow users
> to express new desired target options, calculate fixes, and apply any of
> found solutions.

This can be the third paragraph.

> We created a separate test infrastructure that we used to validate the
> correctness of the suggestions made. It shows that our resolution algorithm
> resolves around 95% of the conflicts. We plan to incorporate this with a
> later patch series.
> 
> We envision that our translation of the kconfig option tree into a
> propositional formula could potentially also later be repurposed to address
> other problems. An example is checking the consistency between the use of
> ifdefs and logic expressed in kconfig files. We suspect that this could,
> for example, help avoid invalid kconfig configurations and help with ifdef
> maintenance.

I think these two paragraphs are very forward lookjng and can be just
put into the wiki for now.

> You can see a YouTube video demonstrating this work [2]. This effort is
> part of the kernelnewbies Kconfig-SAT project [3], the approach and effort
> is also explained in detail in our paper [4]. The results from the
> evaluation have significantly improved since then: Around 80 % of the
> conflicts could be resolved, and 99.9 % of the generated fixes resolved the
> conflict. It is also our attempt at contributing back to the kernel
> community, whose configurator researchers studied a lot.

I think this is a nice final paragraph summary for the research to
include.
 
> Patches applicable to next-20241025.
> 
> [0] https://gsd.uwaterloo.ca/sites/default/files/vamos12-survey.pdf
> [1] https://www.linux-magazine.com/Issues/2021/244/Kconfig-Deep-Dive
> [2] https://www.youtube.com/watch?v=vn2JgK_PTbc
> [3] https://kernelnewbies.org/KernelProjects/kconfig-sat
> [4] http://www.cse.chalmers.se/~bergert/paper/2021-icseseip-configfix.pdf
> 
> Thanks from the team! (and thanks to Luis Chamberlain for guiding us here)

Sure.

> Co-developed-by: Patrick Franz <deltaone@debian.org>
> Signed-off-by: Patrick Franz <deltaone@debian.org>
> Co-developed-by: Ibrahim Fayaz <phayax@gmail.com>
> Signed-off-by: Ibrahim Fayaz <phayax@gmail.com>
> Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>
> Tested-by: Evgeny Groshev <eugene.groshev@gmail.com>
> Suggested-by: Sarah Nadi <nadi@ualberta.ca>
> Suggested-by: Thorsten Berger <thorsten.berger@rub.de>
> Signed-off-by: Thorsten Berger <thorsten.berger@rub.de>
> Signed-off-by: Ole Schuerks <ole0811sch@gmail.com>

This all can be removed, patch tags have no meaning for cover letters.

> Changelog v6:
> * remove script for manually installing PicoSAT
> * adapt documentation and help text in the GUI accordingly
> * convert Qt signal/slot connects to new style

It is wonderful you are keeping tabs of the changes on the cover letter,
keep it up!

  Luis
Re: [PATCH v6 00/11] kbuild, kconfig: Add support for conflict resolution
Posted by Luis Chamberlain 1 year, 2 months ago
OK now that I've gone through the entire series again, I think this
is looking so much better. Just address those things I mentioned and
rebase on to v6.13-rc1:

  * use checkpatch
  * simplify commt logs and be more imperative and to the point

But from my perspective hopefully on the next respin, my hope is that
this will be ready.

  Luis
Re: [PATCH v6 01/11] kconfig: Add PicoSAT interface
Posted by Brendan Jackman 1 year ago
Hi all, this looks cool!

> This support is provided within xconfig.

Sorry I haven't read properly enough to be able to answer this for myself but I
suspect I'm not the only one idly wondering...

Would it be easy enough to provide this support as a standalone binary that can
be called from scripts/config too?

Personally, the only reason I ever use xconfig (well, I use menuconfig, but same
thing) is to navigate "conflicts". If that problem went away I'd have no more
reason to open it except to access the cool SAT thing, so it would be even more
convenient to cut out the middleman.
Re: [PATCH v6 01/11] kconfig: Add PicoSAT interface
Posted by Luis Chamberlain 1 year ago
On Thu, Jan 09, 2025 at 01:28:17PM +0000, Brendan Jackman wrote:
> Hi all, this looks cool!
> 
> > This support is provided within xconfig.
> 
> Sorry I haven't read properly enough to be able to answer this for myself but I
> suspect I'm not the only one idly wondering...
> 
> Would it be easy enough to provide this support as a standalone binary that can
> be called from scripts/config too?

I suspect this begs more the question of how could this be useful
outside of xconfig. But one step at a time, this can be done *after*
this initial patch set gets merged, but certainly its good to think
about and project if the current architecture would need to change to
address that later. I suspect not much yet.

  Luis
Re: [PATCH v6 01/11] kconfig: Add PicoSAT interface
Posted by Brendan Jackman 1 year ago
On Fri, 10 Jan 2025 at 20:15, Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Thu, Jan 09, 2025 at 01:28:17PM +0000, Brendan Jackman wrote:
> > Would it be easy enough to provide this support as a standalone binary that can
> > be called from scripts/config too?
>
> I suspect this begs more the question of how could this be useful
> outside of xconfig.

I was thinking of something like like this:

$ scripts/sat_config --interactive --enable KVM_AMD

This would then prompt the user with a range of solutions for getting
KVM_AMD enabled.

Then there might be an "I'm feeling lucky" variant like:

$ scripts/sat_config --first-solution --enable KVM_AMD

Where instead of prompting the user it just picks some arbitrary or
"minimal" set of changes that leads to KVM_AMD being enabled.

> But one step at a time, this can be done *after*
> this initial patch set gets merged,

Totally, please consider this a "yes, and" not a "but".
Re: [PATCH v6 01/11] kconfig: Add PicoSAT interface
Posted by Thorsten Berger 12 months ago
Hi Brendan,

Yes, that's a great idea. It reminds me about the eCos configurator 
configtool developed by the Cygnus Solutions developers, which had such 
a capability. One could manually create a partial specification and ask 
it to complete the configuration. We should be able to create such a 
support soon.

We've just created the final set of patches, with the requested changes 
(also some small naming change in the fix generation to avoid 
confusion). Ole is sending them in.

Best, Thorsten



On 13/01/2025 17:29, Brendan Jackman wrote:
> On Fri, 10 Jan 2025 at 20:15, Luis Chamberlain <mcgrof@kernel.org> wrote:
>> On Thu, Jan 09, 2025 at 01:28:17PM +0000, Brendan Jackman wrote:
>>> Would it be easy enough to provide this support as a standalone binary that can
>>> be called from scripts/config too?
>> I suspect this begs more the question of how could this be useful
>> outside of xconfig.
> I was thinking of something like like this:
>
> $ scripts/sat_config --interactive --enable KVM_AMD
>
> This would then prompt the user with a range of solutions for getting
> KVM_AMD enabled.
>
> Then there might be an "I'm feeling lucky" variant like:
>
> $ scripts/sat_config --first-solution --enable KVM_AMD
>
> Where instead of prompting the user it just picks some arbitrary or
> "minimal" set of changes that leads to KVM_AMD being enabled.
>
>> But one step at a time, this can be done *after*
>> this initial patch set gets merged,
> Totally, please consider this a "yes, and" not a "but".