See comment for details.  Works around a GCC-9 bug which breaks the build on
Ubuntu.

Reported-by: Jason Andryuk <jandryuk@gmail.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Jan Beulich <JBeulich@suse.com>
CC: Wei Liu <wl@xen.org>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jason Andryuk <jandryuk@gmail.com>
CC: Stefan Bader <stefan.bader@canonical.com>

Sorry for messing you around with how to fix this.  I'd neglected to consider
the CONFIG_LIVEPATCH interaction.  With that extra observation, there is no
point having the extra complexity given that the result with CET-IBT and
Retpoline still isn't usable.
---
 xen/arch/x86/arch.mk         | 9 +++++++++
 xen/arch/x86/boot/build32.mk | 1 +
 2 files changed, 10 insertions(+)

diff --git a/xen/arch/x86/arch.mk b/xen/arch/x86/arch.mk
index 2a51553edb..93e30e4bea 100644
--- a/xen/arch/x86/arch.mk
+++ b/xen/arch/x86/arch.mk
@@ -67,6 +67,15 @@ CFLAGS-$(CONFIG_INDIRECT_THUNK) += -mindirect-branch=thunk-extern
 CFLAGS-$(CONFIG_INDIRECT_THUNK) += -mindirect-branch-register
 CFLAGS-$(CONFIG_INDIRECT_THUNK) += -fno-jump-tables
 
+# Xen doesn't support CET-IBT yet.  At a minimum, logic is required to
+# enable it for supervisor use, but the Livepatch functionality needs
+# to learn not to overwrite ENDBR64 instructions.
+#
+# Furthermore, Ubuntu enables -fcf-protection by default, along with a
+# buggy version of GCC-9 which objects to it in combination with
+# -mindirect-branch=thunk-extern (Fixed in GCC 10, 9.4).
+$(call cc-option-add,CFLAGS,CC,-fcf-protection=none)
+
 # If supported by the compiler, reduce stack alignment to 8 bytes. But allow
 # this to be overridden elsewhere.
 $(call cc-option-add,CFLAGS-stack-boundary,CC,-mpreferred-stack-boundary=3)
diff --git a/xen/arch/x86/boot/build32.mk b/xen/arch/x86/boot/build32.mk
index 48c7407c00..5a00755512 100644
--- a/xen/arch/x86/boot/build32.mk
+++ b/xen/arch/x86/boot/build32.mk
@@ -3,6 +3,7 @@ CFLAGS =
 include $(XEN_ROOT)/Config.mk
 
 $(call cc-options-add,CFLAGS,CC,$(EMBEDDED_EXTRA_CFLAGS))
+$(call cc-option-add,CFLAGS,CC,-fcf-protection=none)
 
 CFLAGS += -Werror -fno-asynchronous-unwind-tables -fno-builtin -g0 -msoft-float
 CFLAGS += -I$(XEN_ROOT)/xen/include
-- 
2.11.0

On Tue, May 12, 2020 at 3:11 PM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
> See comment for details.  Works around a GCC-9 bug which breaks the build on
> Ubuntu.
>
> Reported-by: Jason Andryuk <jandryuk@gmail.com>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Tested-by: Jason Andryuk <jandryuk@gmail.com>
Reviewed-by: Jason Andryuk <jandryuk@gmail.com>

> diff --git a/xen/arch/x86/arch.mk b/xen/arch/x86/arch.mk
> index 2a51553edb..93e30e4bea 100644
> --- a/xen/arch/x86/arch.mk
> +++ b/xen/arch/x86/arch.mk
> @@ -67,6 +67,15 @@ CFLAGS-$(CONFIG_INDIRECT_THUNK) += -mindirect-branch=thunk-extern
>  CFLAGS-$(CONFIG_INDIRECT_THUNK) += -mindirect-branch-register
>  CFLAGS-$(CONFIG_INDIRECT_THUNK) += -fno-jump-tables
>
> +# Xen doesn't support CET-IBT yet.  At a minimum, logic is required to
> +# enable it for supervisor use, but the Livepatch functionality needs
> +# to learn not to overwrite ENDBR64 instructions.

Is the problem that existing functions start with ENDBR64, but the
livepatch overwrites with a "real" instruction?

Regards,
Jason

On 13/05/2020 03:35, Jason Andryuk wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
>
> On Tue, May 12, 2020 at 3:11 PM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> See comment for details.  Works around a GCC-9 bug which breaks the build on
>> Ubuntu.
>>
>> Reported-by: Jason Andryuk <jandryuk@gmail.com>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Tested-by: Jason Andryuk <jandryuk@gmail.com>
> Reviewed-by: Jason Andryuk <jandryuk@gmail.com>
>
>> diff --git a/xen/arch/x86/arch.mk b/xen/arch/x86/arch.mk
>> index 2a51553edb..93e30e4bea 100644
>> --- a/xen/arch/x86/arch.mk
>> +++ b/xen/arch/x86/arch.mk
>> @@ -67,6 +67,15 @@ CFLAGS-$(CONFIG_INDIRECT_THUNK) += -mindirect-branch=thunk-extern
>>  CFLAGS-$(CONFIG_INDIRECT_THUNK) += -mindirect-branch-register
>>  CFLAGS-$(CONFIG_INDIRECT_THUNK) += -fno-jump-tables
>>
>> +# Xen doesn't support CET-IBT yet.  At a minimum, logic is required to
>> +# enable it for supervisor use, but the Livepatch functionality needs
>> +# to learn not to overwrite ENDBR64 instructions.
> Is the problem that existing functions start with ENDBR64, but the
> livepatch overwrites with a "real" instruction?

We livepatch by creating a new complete copy of the function, and
putting `jmp new` at the head of the old one.

This means we don't need to patch every callsite and track every
function pointer to the old function, and we can fully revert by
replacing the 5 bytes which became `jmp new`.

With CET-IBT in the mix, livepatch will have to learn to spot an ENDBR64
instruction and leave it intact, patching instead the next 5 bytes, so
an old function pointer still lands on the ENDBR64 instruction.

~Andrew

On Wed, May 13, 2020 at 7:01 AM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
> On 13/05/2020 03:35, Jason Andryuk wrote:
> > [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
> >
> > On Tue, May 12, 2020 at 3:11 PM Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> >> +# Xen doesn't support CET-IBT yet.  At a minimum, logic is required to
> >> +# enable it for supervisor use, but the Livepatch functionality needs
> >> +# to learn not to overwrite ENDBR64 instructions.
> > Is the problem that existing functions start with ENDBR64, but the
> > livepatch overwrites with a "real" instruction?
>
> We livepatch by creating a new complete copy of the function, and
> putting `jmp new` at the head of the old one.
>
> This means we don't need to patch every callsite and track every
> function pointer to the old function, and we can fully revert by
> replacing the 5 bytes which became `jmp new`.
>
> With CET-IBT in the mix, livepatch will have to learn to spot an ENDBR64
> instruction and leave it intact, patching instead the next 5 bytes, so
> an old function pointer still lands on the ENDBR64 instruction.

Ah, okay.  Thanks for the explanation.

-Jason

On 12.05.2020 21:11, Andrew Cooper wrote:
> See comment for details.  Works around a GCC-9 bug which breaks the build on
> Ubuntu.
> 
> Reported-by: Jason Andryuk <jandryuk@gmail.com>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Jan Beulich <jbeulich@suse.com>