[PATCH v10 07/30] x86/resctrl: Check all domains are offline in resctrl_exit()

James Morse posted 30 patches 7 months, 1 week ago
There is a newer version of this series
[PATCH v10 07/30] x86/resctrl: Check all domains are offline in resctrl_exit()
Posted by James Morse 7 months, 1 week ago
resctrl_exit() removes things like the resctrl mount point directory
and unregisters the filesystem prior to freeing data structures that
were allocated during resctrl_init().

This assumes that there are no online domains when resctrl_exit() is
called. If any domain were online, the limbo or overflow handler could
be scheduled to run.

Add a check for any online control or monitor domains, and document that
the architecture code is required to offline all monitor and control
domains before calling resctrl_exit().

Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: James Morse <james.morse@arm.com>
Tested-by: Babu Moger <babu.moger@amd.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
---
Changes since v9:
 * Clarified commit message.

Changes since v8:
 * This patch is new.
---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 88197afbbb8a..f617ac97758b 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -4420,8 +4420,32 @@ int __init resctrl_init(void)
 	return ret;
 }
 
+static bool __exit resctrl_online_domains_exist(void)
+{
+	struct rdt_resource *r;
+
+	for_each_rdt_resource(r) {
+		if (!list_empty(&r->ctrl_domains) || !list_empty(&r->mon_domains))
+			return true;
+	}
+
+	return false;
+}
+
+/*
+ * resctrl_exit() - Remove the resctrl filesystem and free resources.
+ *
+ * When called by the architecture code, all CPUs and resctrl domains must be
+ * offline. This ensures the limbo and overflow handlers are not scheduled to
+ * run, meaning the data structures they access can be freed by
+ * resctrl_mon_resource_exit().
+ */
 void __exit resctrl_exit(void)
 {
+	cpus_read_lock();
+	WARN_ON_ONCE(resctrl_online_domains_exist());
+	cpus_read_unlock();
+
 	debugfs_remove_recursive(debugfs_resctrl);
 	unregister_filesystem(&rdt_fs_type);
 	sysfs_remove_mount_point(fs_kobj, "resctrl");
-- 
2.39.5
Re: [PATCH v10 07/30] x86/resctrl: Check all domains are offline in resctrl_exit()
Posted by Reinette Chatre 7 months, 1 week ago
Hi James,

On 5/8/25 10:18 AM, James Morse wrote:
> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> index 88197afbbb8a..f617ac97758b 100644
> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> @@ -4420,8 +4420,32 @@ int __init resctrl_init(void)
>  	return ret;
>  }
>  
> +static bool __exit resctrl_online_domains_exist(void)
> +{
> +	struct rdt_resource *r;
> +
> +	for_each_rdt_resource(r) {
> +		if (!list_empty(&r->ctrl_domains) || !list_empty(&r->mon_domains))
> +			return true;
> +	}
> +
> +	return false;
> +}

This looks the same as before. Did you notice my comment in [1] about this list_empty() usage?

Reinette

[1] https://lore.kernel.org/lkml/91c31b70-3d41-40cb-b00b-aa39cbd07bc9@intel.com/
Re: [PATCH v10 07/30] x86/resctrl: Check all domains are offline in resctrl_exit()
Posted by Catalin Marinas 7 months, 1 week ago
Hi Reinette,

Thanks for the reviews.

On Thu, May 08, 2025 at 10:51:23AM -0700, Reinette Chatre wrote:
> On 5/8/25 10:18 AM, James Morse wrote:
> > diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> > index 88197afbbb8a..f617ac97758b 100644
> > --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> > +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
> > @@ -4420,8 +4420,32 @@ int __init resctrl_init(void)
> >  	return ret;
> >  }
> >  
> > +static bool __exit resctrl_online_domains_exist(void)
> > +{
> > +	struct rdt_resource *r;
> > +
> > +	for_each_rdt_resource(r) {
> > +		if (!list_empty(&r->ctrl_domains) || !list_empty(&r->mon_domains))
> > +			return true;
> > +	}
> > +
> > +	return false;
> > +}
> 
> This looks the same as before. Did you notice my comment in [1] about this list_empty() usage?

James may have missed that comment (and he's off today). Copying your
comment here:

> A list needs to be initialized for list_empty() to behave as intended. A list within
> an uninitialized or "kzalloc()'ed" struct will not be considered empty.
> resctrl_arch_get_resource() as used by for_each_rdt_resource() already establishes
> that if an architecture does not support a particular resource then it can (should?)
> return a "dummy/not-capable" resource. I do not think resctrl should require
> anything additionally like initializing the lists of a dummy/not-capable resource
> to support things like this loop.

I agree the description of the resctrl_arch_get_resource() semantics
doesn't specifically mention that the list_heads should be initialised
in dummy resources. IIUC, x86 should be safe for now since the
rdt_resources_all[] array elements have the ctrl_domains and mon_domains
list_heads initialised.

> Considering this, could this be made more specific? For example,
> 
> 	for_each_alloc_capable_rdt_resource(r) {
> 		if (!list_empty(&r->ctrl_domains))
> 			return true;
> 	}
> 
> 	for_each_mon_capable_rdt_resource(r) {
> 		if (!list_empty(&r->mon_domains))
> 			return true;
> 	}

If not-capable resources can be only partially initialised, your
proposal makes (alternatively, keep a single loop but only check
list_empty() if {mon,alloc}_capable; maybe the compiler is smart enough
to squash the two loops).

That said, if Boris plans to take this series, I don't think it's worth
a respin (assuming my understanding is correct and there are no other
issues with the series). The fix can be added subsequently on top before
other architectures gain support for resctrl (well, most likely Mon/Tue
when James is back).

Thanks.

-- 
Catalin
Re: [PATCH v10 07/30] x86/resctrl: Check all domains are offline in resctrl_exit()
Posted by Reinette Chatre 7 months, 1 week ago
Hi Catalin,

On 5/9/25 4:21 AM, Catalin Marinas wrote:
> Hi Reinette,
> 
> Thanks for the reviews.
> 
> On Thu, May 08, 2025 at 10:51:23AM -0700, Reinette Chatre wrote:
>> On 5/8/25 10:18 AM, James Morse wrote:
>>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>> index 88197afbbb8a..f617ac97758b 100644
>>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>> @@ -4420,8 +4420,32 @@ int __init resctrl_init(void)
>>>  	return ret;
>>>  }
>>>  
>>> +static bool __exit resctrl_online_domains_exist(void)
>>> +{
>>> +	struct rdt_resource *r;
>>> +
>>> +	for_each_rdt_resource(r) {
>>> +		if (!list_empty(&r->ctrl_domains) || !list_empty(&r->mon_domains))
>>> +			return true;
>>> +	}
>>> +
>>> +	return false;
>>> +}
>>
>> This looks the same as before. Did you notice my comment in [1] about this list_empty() usage?
> 
> James may have missed that comment (and he's off today). Copying your
> comment here:
> 
>> A list needs to be initialized for list_empty() to behave as intended. A list within
>> an uninitialized or "kzalloc()'ed" struct will not be considered empty.
>> resctrl_arch_get_resource() as used by for_each_rdt_resource() already establishes
>> that if an architecture does not support a particular resource then it can (should?)
>> return a "dummy/not-capable" resource. I do not think resctrl should require
>> anything additionally like initializing the lists of a dummy/not-capable resource
>> to support things like this loop.
> 
> I agree the description of the resctrl_arch_get_resource() semantics
> doesn't specifically mention that the list_heads should be initialised
> in dummy resources. IIUC, x86 should be safe for now since the
> rdt_resources_all[] array elements have the ctrl_domains and mon_domains
> list_heads initialised.

Not all x86 resources support both control and monitoring. For these resources x86
only initializes the arrays it uses. For example, the L2 resource only supports control
and thus only the ctrl_domains list is initialized. When the above loop runs on a resource
like this it will return "true" because it believes there are monitoring domains
online.

Your conclusion that x86 is safe for now may (reason for use of "may" follows) still
be valid since resctrl_online_domains_exist() is only called by x86 from its
resctrl_arch_exit() that is within the "exit.text" section. Here is where I am not
entirely certain of the impact (hence the earlier use of "may") since from what I can
tell the linker does not actually discard "exit.text" on x86 because it defines
RUNTIME_DISCARD_EXIT.

> 
>> Considering this, could this be made more specific? For example,
>>
>> 	for_each_alloc_capable_rdt_resource(r) {
>> 		if (!list_empty(&r->ctrl_domains))
>> 			return true;
>> 	}
>>
>> 	for_each_mon_capable_rdt_resource(r) {
>> 		if (!list_empty(&r->mon_domains))
>> 			return true;
>> 	}
> 
> If not-capable resources can be only partially initialised, your
> proposal makes (alternatively, keep a single loop but only check
> list_empty() if {mon,alloc}_capable; maybe the compiler is smart enough
> to squash the two loops).

Yes, this can be done with a single loop also.

> That said, if Boris plans to take this series, I don't think it's worth
> a respin (assuming my understanding is correct and there are no other
> issues with the series). The fix can be added subsequently on top before
> other architectures gain support for resctrl (well, most likely Mon/Tue
> when James is back).

I cannot speak to Boris's plans or requirements related to this work.

Reinette
Re: [PATCH v10 07/30] x86/resctrl: Check all domains are offline in resctrl_exit()
Posted by Reinette Chatre 7 months, 1 week ago
Hi Catalin,

On 5/9/25 9:17 AM, Reinette Chatre wrote:
> Hi Catalin,
> 
> On 5/9/25 4:21 AM, Catalin Marinas wrote:
>> Hi Reinette,
>>
>> Thanks for the reviews.
>>
>> On Thu, May 08, 2025 at 10:51:23AM -0700, Reinette Chatre wrote:
>>> On 5/8/25 10:18 AM, James Morse wrote:
>>>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>>> index 88197afbbb8a..f617ac97758b 100644
>>>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>>> @@ -4420,8 +4420,32 @@ int __init resctrl_init(void)
>>>>  	return ret;
>>>>  }
>>>>  
>>>> +static bool __exit resctrl_online_domains_exist(void)
>>>> +{
>>>> +	struct rdt_resource *r;
>>>> +
>>>> +	for_each_rdt_resource(r) {
>>>> +		if (!list_empty(&r->ctrl_domains) || !list_empty(&r->mon_domains))
>>>> +			return true;
>>>> +	}
>>>> +
>>>> +	return false;
>>>> +}
>>>
>>> This looks the same as before. Did you notice my comment in [1] about this list_empty() usage?
>>
>> James may have missed that comment (and he's off today). Copying your
>> comment here:
>>
>>> A list needs to be initialized for list_empty() to behave as intended. A list within
>>> an uninitialized or "kzalloc()'ed" struct will not be considered empty.
>>> resctrl_arch_get_resource() as used by for_each_rdt_resource() already establishes
>>> that if an architecture does not support a particular resource then it can (should?)
>>> return a "dummy/not-capable" resource. I do not think resctrl should require
>>> anything additionally like initializing the lists of a dummy/not-capable resource
>>> to support things like this loop.
>>
>> I agree the description of the resctrl_arch_get_resource() semantics
>> doesn't specifically mention that the list_heads should be initialised
>> in dummy resources. IIUC, x86 should be safe for now since the
>> rdt_resources_all[] array elements have the ctrl_domains and mon_domains
>> list_heads initialised.
> 
> Not all x86 resources support both control and monitoring. For these resources x86
> only initializes the arrays it uses. For example, the L2 resource only supports control
> and thus only the ctrl_domains list is initialized. When the above loop runs on a resource
> like this it will return "true" because it believes there are monitoring domains
> online.
> 
> Your conclusion that x86 is safe for now may (reason for use of "may" follows) still
> be valid since resctrl_online_domains_exist() is only called by x86 from its
> resctrl_arch_exit() that is within the "exit.text" section. Here is where I am not
> entirely certain of the impact (hence the earlier use of "may") since from what I can
> tell the linker does not actually discard "exit.text" on x86 because it defines
> RUNTIME_DISCARD_EXIT.

If you are not familiar with the details then I should add for completeness that resctrl
is not currently capable of being compiled as a module.

Reinette
Re: [PATCH v10 07/30] x86/resctrl: Check all domains are offline in resctrl_exit()
Posted by James Morse 7 months ago
Hi Reinette,

On 09/05/2025 17:28, Reinette Chatre wrote:
> On 5/9/25 9:17 AM, Reinette Chatre wrote:
>> On 5/9/25 4:21 AM, Catalin Marinas wrote:
>>> On Thu, May 08, 2025 at 10:51:23AM -0700, Reinette Chatre wrote:
>>>> On 5/8/25 10:18 AM, James Morse wrote:
>>>>> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>>>> index 88197afbbb8a..f617ac97758b 100644
>>>>> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>>>> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
>>>>> @@ -4420,8 +4420,32 @@ int __init resctrl_init(void)
>>>>>  	return ret;
>>>>>  }
>>>>>  
>>>>> +static bool __exit resctrl_online_domains_exist(void)
>>>>> +{
>>>>> +	struct rdt_resource *r;
>>>>> +
>>>>> +	for_each_rdt_resource(r) {
>>>>> +		if (!list_empty(&r->ctrl_domains) || !list_empty(&r->mon_domains))
>>>>> +			return true;
>>>>> +	}
>>>>> +
>>>>> +	return false;
>>>>> +}
>>>>
>>>> This looks the same as before. Did you notice my comment in [1] about this list_empty() usage?
>>>
>>> James may have missed that comment (and he's off today). Copying your
>>> comment here:
>>>
>>>> A list needs to be initialized for list_empty() to behave as intended. A list within
>>>> an uninitialized or "kzalloc()'ed" struct will not be considered empty.
>>>> resctrl_arch_get_resource() as used by for_each_rdt_resource() already establishes
>>>> that if an architecture does not support a particular resource then it can (should?)
>>>> return a "dummy/not-capable" resource. I do not think resctrl should require
>>>> anything additionally like initializing the lists of a dummy/not-capable resource
>>>> to support things like this loop.
>>>
>>> I agree the description of the resctrl_arch_get_resource() semantics
>>> doesn't specifically mention that the list_heads should be initialised
>>> in dummy resources. IIUC, x86 should be safe for now since the
>>> rdt_resources_all[] array elements have the ctrl_domains and mon_domains
>>> list_heads initialised.

I had missed this. The logic makes sense - I'll spin this as a v11.


>> Not all x86 resources support both control and monitoring. For these resources x86
>> only initializes the arrays it uses. For example, the L2 resource only supports control
>> and thus only the ctrl_domains list is initialized. When the above loop runs on a resource
>> like this it will return "true" because it believes there are monitoring domains
>> online.
>>
>> Your conclusion that x86 is safe for now may (reason for use of "may" follows) still
>> be valid since resctrl_online_domains_exist() is only called by x86 from its
>> resctrl_arch_exit() that is within the "exit.text" section. Here is where I am not
>> entirely certain of the impact (hence the earlier use of "may") since from what I can
>> tell the linker does not actually discard "exit.text" on x86 because it defines
>> RUNTIME_DISCARD_EXIT.

Aha - I'd not spotted that one before.


> If you are not familiar with the details then I should add for completeness that resctrl
> is not currently capable of being compiled as a module.


Thanks,

James