rdt_put_mon_l3_config() is called via the architecture's
resctrl_arch_exit() call, and appears to free the rmid_ptrs[]
and closid_num_dirty_rmid[] arrays. In reality this code is marked
__exit, and is removed by the linker as resctl can't be built
as a module.

To separate the filesystem and architecture parts of resctrl,
this free()ing work needs to be triggered by the filesystem,
as these structures belong to the filesystem code.

Rename rdt_put_mon_l3_config() resctrl_mon_resource_exit()
and call it from resctrl_exit(). The kfree() is currently
dependent on r->mon_capable. resctrl_mon_resource_init()
takes no arguments, so resctrl_mon_resource_exit() shouldn't
take any either. Add the check to dom_data_exit(), making it
take the resource as an argument. This makes it more symmetrical
with dom_data_init().

Signed-off-by: James Morse <james.morse@arm.com>
Tested-by: Peter Newman <peternewman@google.com>
Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
---
Changes since v2:
 * Dropped __exit as needed in the next patch.

Change since v1:
 * [Commit message only] Typo fixes:
   s/restrl/resctrl/g
   s/resctl/resctrl/g

 * [Commit message only] Reword second paragraph to remove reference to
   the MPAM error interrupt, which provides background rationale for a
   later patch rather than for this patch, and so it is not really
   relevant here.
---
 arch/x86/kernel/cpu/resctrl/core.c     |  5 -----
 arch/x86/kernel/cpu/resctrl/internal.h |  2 +-
 arch/x86/kernel/cpu/resctrl/monitor.c  | 12 ++++++++----
 arch/x86/kernel/cpu/resctrl/rdtgroup.c |  2 ++
 4 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 9ad660b2b097..2540a7cb11b0 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -950,14 +950,9 @@ late_initcall(resctrl_arch_late_init);
 
 static void __exit resctrl_arch_exit(void)
 {
-	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
-
 	cpuhp_remove_state(rdt_online);
 
 	resctrl_exit();
-
-	if (r->mon_capable)
-		rdt_put_mon_l3_config();
 }
 
 __exitcall(resctrl_arch_exit);
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 7ede340b1301..9aa7f587484c 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -535,7 +535,7 @@ void closid_free(int closid);
 int alloc_rmid(u32 closid);
 void free_rmid(u32 closid, u32 rmid);
 int rdt_get_mon_l3_config(struct rdt_resource *r);
-void __exit rdt_put_mon_l3_config(void);
+void resctrl_mon_resource_exit(void);
 bool __init rdt_cpu_has(int flag);
 void mon_event_count(void *info);
 int rdtgroup_mondata_show(struct seq_file *m, void *arg);
diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
index 3e5375c365e6..7d6aebce75c1 100644
--- a/arch/x86/kernel/cpu/resctrl/monitor.c
+++ b/arch/x86/kernel/cpu/resctrl/monitor.c
@@ -965,10 +965,12 @@ static int dom_data_init(struct rdt_resource *r)
 	return err;
 }
 
-static void __exit dom_data_exit(void)
+static void dom_data_exit(struct rdt_resource *r)
 {
-	mutex_lock(&rdtgroup_mutex);
+	if (!r->mon_capable)
+		return;
 
+	mutex_lock(&rdtgroup_mutex);
 	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
 		kfree(closid_num_dirty_rmid);
 		closid_num_dirty_rmid = NULL;
@@ -1075,9 +1077,11 @@ int __init rdt_get_mon_l3_config(struct rdt_resource *r)
 	return 0;
 }
 
-void __exit rdt_put_mon_l3_config(void)
+void resctrl_mon_resource_exit(void)
 {
-	dom_data_exit();
+	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
+
+	dom_data_exit(r);
 }
 
 void __init intel_rdt_mbm_apply_quirk(void)
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 82d64885c6c0..8c380f389b93 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -4212,4 +4212,6 @@ void __exit resctrl_exit(void)
 	debugfs_remove_recursive(debugfs_resctrl);
 	unregister_filesystem(&rdt_fs_type);
 	sysfs_remove_mount_point(fs_kobj, "resctrl");
+
+	resctrl_mon_resource_exit();
 }
-- 
2.39.2

James Morse <james.morse@arm.com> writes:
> __exit, and is removed by the linker as resctl can't be built

Typo fix: resctl -> resctrl

Also in the commit message subject line:

Typo fix: restrl -> resctrl

> Change since v1:
>  * [Commit message only] Typo fixes:
>    s/restrl/resctrl/g
>    s/resctl/resctrl/g

Oh? I'm not sure how convinced I am. :-)

-Carl

Hi James,

On 6/14/24 8:00 AM, James Morse wrote:
> rdt_put_mon_l3_config() is called via the architecture's
> resctrl_arch_exit() call, and appears to free the rmid_ptrs[]
> and closid_num_dirty_rmid[] arrays. In reality this code is marked
> __exit, and is removed by the linker as resctl can't be built
> as a module.
> 
> To separate the filesystem and architecture parts of resctrl,
> this free()ing work needs to be triggered by the filesystem,
> as these structures belong to the filesystem code.
> 
> Rename rdt_put_mon_l3_config() resctrl_mon_resource_exit()
> and call it from resctrl_exit(). The kfree() is currently
> dependent on r->mon_capable. resctrl_mon_resource_init()

resctrl_mon_resource_init() does not exist at this point making
this motivation difficult to follow.

> takes no arguments, so resctrl_mon_resource_exit() shouldn't
> take any either. Add the check to dom_data_exit(), making it
> take the resource as an argument. This makes it more symmetrical
> with dom_data_init().
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> Tested-by: Peter Newman <peternewman@google.com>
> Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
> ---
> Changes since v2:
>   * Dropped __exit as needed in the next patch.
> 
> Change since v1:
>   * [Commit message only] Typo fixes:
>     s/restrl/resctrl/g
>     s/resctl/resctrl/g

Something went wrong here since the subject and changelog still contains
the terms that were intended to be replaced.

> 
>   * [Commit message only] Reword second paragraph to remove reference to
>     the MPAM error interrupt, which provides background rationale for a
>     later patch rather than for this patch, and so it is not really
>     relevant here.
> ---
>   arch/x86/kernel/cpu/resctrl/core.c     |  5 -----
>   arch/x86/kernel/cpu/resctrl/internal.h |  2 +-
>   arch/x86/kernel/cpu/resctrl/monitor.c  | 12 ++++++++----
>   arch/x86/kernel/cpu/resctrl/rdtgroup.c |  2 ++
>   4 files changed, 11 insertions(+), 10 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 9ad660b2b097..2540a7cb11b0 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -950,14 +950,9 @@ late_initcall(resctrl_arch_late_init);
>   
>   static void __exit resctrl_arch_exit(void)
>   {
> -	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
> -
>   	cpuhp_remove_state(rdt_online);
>   
>   	resctrl_exit();
> -
> -	if (r->mon_capable)
> -		rdt_put_mon_l3_config();
>   }
>   
>   __exitcall(resctrl_arch_exit);
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 7ede340b1301..9aa7f587484c 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -535,7 +535,7 @@ void closid_free(int closid);
>   int alloc_rmid(u32 closid);
>   void free_rmid(u32 closid, u32 rmid);
>   int rdt_get_mon_l3_config(struct rdt_resource *r);
> -void __exit rdt_put_mon_l3_config(void);
> +void resctrl_mon_resource_exit(void);
>   bool __init rdt_cpu_has(int flag);
>   void mon_event_count(void *info);
>   int rdtgroup_mondata_show(struct seq_file *m, void *arg);
> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
> index 3e5375c365e6..7d6aebce75c1 100644
> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
> @@ -965,10 +965,12 @@ static int dom_data_init(struct rdt_resource *r)
>   	return err;
>   }
>   
> -static void __exit dom_data_exit(void)
> +static void dom_data_exit(struct rdt_resource *r)
>   {
> -	mutex_lock(&rdtgroup_mutex);
> +	if (!r->mon_capable)
> +		return;
>   
> +	mutex_lock(&rdtgroup_mutex);

I know there has been a bit of back&forth on whether the mutex is needed
here. With this change moving dom_data_exit() out from __exit I think
the locking should aim to be consistent with existing runtime
and thus the check of r->mon_capable should be with mutex held. Having
this little snippet outside mutex will just cause confusion. Do you
have motivation for needing this be done outside of mutex? I think it
ended up this way with this patch aiming to keep existing flow exactly,
but that ended up as convenience in a flow where mutex was not really
needed at all.

Reinette

Hi Reinette,

On 28/06/2024 17:46, Reinette Chatre wrote:
> On 6/14/24 8:00 AM, James Morse wrote:
>> rdt_put_mon_l3_config() is called via the architecture's
>> resctrl_arch_exit() call, and appears to free the rmid_ptrs[]
>> and closid_num_dirty_rmid[] arrays. In reality this code is marked
>> __exit, and is removed by the linker as resctl can't be built
>> as a module.
>>
>> To separate the filesystem and architecture parts of resctrl,
>> this free()ing work needs to be triggered by the filesystem,
>> as these structures belong to the filesystem code.
>>
>> Rename rdt_put_mon_l3_config() resctrl_mon_resource_exit()
>> and call it from resctrl_exit(). The kfree() is currently
>> dependent on r->mon_capable. resctrl_mon_resource_init()

> resctrl_mon_resource_init() does not exist at this point making
> this motivation difficult to follow.

Right - I re-ordered the patches to make the diffs simpler. I'll drop this paragraph.


>> takes no arguments, so resctrl_mon_resource_exit() shouldn't
>> take any either. Add the check to dom_data_exit(), making it
>> take the resource as an argument. This makes it more symmetrical
>> with dom_data_init().

>> diff --git a/arch/x86/kernel/cpu/resctrl/monitor.c b/arch/x86/kernel/cpu/resctrl/monitor.c
>> index 3e5375c365e6..7d6aebce75c1 100644
>> --- a/arch/x86/kernel/cpu/resctrl/monitor.c
>> +++ b/arch/x86/kernel/cpu/resctrl/monitor.c
>> @@ -965,10 +965,12 @@ static int dom_data_init(struct rdt_resource *r)
>>       return err;
>>   }
>>   -static void __exit dom_data_exit(void)
>> +static void dom_data_exit(struct rdt_resource *r)
>>   {
>> -    mutex_lock(&rdtgroup_mutex);
>> +    if (!r->mon_capable)
>> +        return;
>>   +    mutex_lock(&rdtgroup_mutex);
> 
> I know there has been a bit of back&forth on whether the mutex is needed
> here. With this change moving dom_data_exit() out from __exit I think
> the locking should aim to be consistent with existing runtime
> and thus the check of r->mon_capable should be with mutex held. Having
> this little snippet outside mutex will just cause confusion.

> Do you have motivation for needing this be done outside of mutex?

Just to avoid sleeping - then returning having done nothing.


> I think it
> ended up this way with this patch aiming to keep existing flow exactly,
> but that ended up as convenience in a flow where mutex was not really
> needed at all.

Sure - I don't think it matters either way. Its not a path where performance matters.
The property is read-only from when resctrl_init() is called. I'll make it look like
resctrl_offline_domain().


Thanks,

James