Hi all,

In the last MISRA meeting Stefano told me about the Rule 10.3 and that we have a lot of violations,
but GCC has a way to list them as written in rules.rst.

So the issue was just that we have a lot of violations, most of them coming from headers and the
Developer might feel a bit lost since the number is very high, so I’ve developed this script to take
the GCC build log and create a list of unique occurrences of the violations sorted in descending
order, so that who has a bit of time can try to fix some of those and maybe can start from the top
of the list to fix as many as possible occurrences.

I attach the script as patch file.

Cheers,
Luca

From 0035fc693f8c97fb41ff4e5697b9321042484b36 Mon Sep 17 00:00:00 2001
From: Luca Fancellu <luca.fancellu@arm.com>
Date: Thu, 28 Sep 2023 08:48:33 +0100
Subject: [PATCH] Script to analyse the rule 10.3 GCC output

Signed-off-by: Luca Fancellu <luca.fancellu@arm.com>
---
This script is not meant to be part of Xen, I'm sending it just as a reference.
---
 xen/scripts/rule_10_3.py | 111 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 111 insertions(+)
 create mode 100755 xen/scripts/rule_10_3.py

diff --git a/xen/scripts/rule_10_3.py b/xen/scripts/rule_10_3.py
new file mode 100755
index 000000000000..40b34b470808
--- /dev/null
+++ b/xen/scripts/rule_10_3.py
@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+import re
+from argparse import ArgumentParser
+
+# This script analyse the build output having the GCC conversion warning
+# enabled for what concerns the rule 10.3, it provides a report with unique
+# occurences for a violation listed as:
+# file path:line number:column number (count)
+# The list is sorted in descending count order, so that fixing the violations
+# at the beginning will provide a better impact on the total count of violations
+# Build Xen with the command line suggested in rules.rst and generate an output
+# file, an example here building for arm64:
+# CFLAGS="-Wconversion -Wno-error=sign-conversion -Wno-error=conversion" \
+# make -C xen CROSS_COMPILE="aarch64-linux-gnu-" XEN_TARGET_ARCH="arm64" \
+# 2> >(tee -a ../build-arm64.txt >&2)
+
+class ViolationList:
+    class ListElement:
+        def __init__(self, file_path, line_num, col_num):
+            self.path = file_path
+            self.line = str(line_num)
+            self.col = str(col_num)
+            self.count = 1
+
+        def __eq__(self, other):
+            if self.path == other.path and self.line == other.line \
+               and self.col == other.col:
+                return True
+
+            return False
+
+    def __init__(self):
+        # __list is a dictionary with this format:
+        # key         -> value
+        # [file path]    [array of items with line number and col number]
+        self.__list = {}
+
+    def add_element(self, file_path, line_num, col_num):
+        entry = self.ListElement(file_path, line_num, col_num)
+        if file_path in self.__list.keys():
+            if entry in self.__list[file_path]:
+                for el in self.__list[file_path]:
+                    if el == entry:
+                        el.count += 1
+            else:
+                self.__list[file_path].append(entry)
+        else:
+            self.__list[file_path] = [entry]
+
+    def to_list(self):
+        report_list = []
+        for _, entries in self.__list.items():
+            for entry in entries:
+                report_list.append(entry)
+
+        report_list.sort(reverse=True, key=lambda x: x.count)
+        return report_list
+
+    def __str__(self):
+        ret = ""
+        total = 0
+        for entry in self.to_list():
+            ret += entry.path + ":" + entry.line + ":" + entry.col + " (" \
+                   + str(entry.count) + ")\n"
+            total += entry.count
+
+        ret += "\n\nTotal count: " + str(total) + "\n"
+
+        return ret
+
+
+def main(argv):
+    parser = ArgumentParser(prog="rule_10_3.py")
+    parser.add_argument("-l", "--log", required=True, type=str,
+                        help="Path to the build log.")
+    args = parser.parse_args()
+
+    log_path = os.path.realpath(args.log)
+    if log_path == "":
+        print("Please pass the log.")
+        sys.exit(1)
+
+    try:
+        with open(log_path, "rt") as infile:
+            log_lines = infile.readlines()
+    except OSError as e:
+        print("Issue with reading file {}: {}".format(log_path, e))
+        sys.exit(1)
+
+    violation_entry = re.compile(r'^(.*):(\d+):(\d+):.*$')
+    violation_list = ViolationList()
+
+    for line in log_lines:
+        if ("[-Wsign-conversion]" in line) or ("[-Wconversion]" in line):
+            entry = violation_entry.match(line)
+            if entry and entry.group(1) and entry.group(2) and entry.group(3):
+                file_path = entry.group(1)
+                line_number = int(entry.group(2))
+                col_number = int(entry.group(3))
+                violation_list.add_element(file_path, line_number, col_number)
+            else:
+                print("Malformed report entry in file {}:\n{}"
+                      .format(log_path, line))
+    print(violation_list)
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
-- 
2.34.1

On Thu, 28 Sep 2023, Luca Fancellu wrote:
> Hi all,
> 
> In the last MISRA meeting Stefano told me about the Rule 10.3 and that we have a lot of violations,
> but GCC has a way to list them as written in rules.rst.
> 
> So the issue was just that we have a lot of violations, most of them coming from headers and the
> Developer might feel a bit lost since the number is very high, so I’ve developed this script to take
> the GCC build log and create a list of unique occurrences of the violations sorted in descending
> order, so that who has a bit of time can try to fix some of those and maybe can start from the top
> of the list to fix as many as possible occurrences.
> 
> I attach the script as patch file.

Thanks Luca! Actually I think this is great! The script makes it much
easier to use the GCC output to address 10.3 violations. I think we
should have it in the xen.git tree.

Thanks to the script I was able to quickly address the top violation in
bitmap.h.

> On 29 Sep 2023, at 00:20, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Thu, 28 Sep 2023, Luca Fancellu wrote:
>> Hi all,
>> 
>> In the last MISRA meeting Stefano told me about the Rule 10.3 and that we have a lot of violations,
>> but GCC has a way to list them as written in rules.rst.
>> 
>> So the issue was just that we have a lot of violations, most of them coming from headers and the
>> Developer might feel a bit lost since the number is very high, so I’ve developed this script to take
>> the GCC build log and create a list of unique occurrences of the violations sorted in descending
>> order, so that who has a bit of time can try to fix some of those and maybe can start from the top
>> of the list to fix as many as possible occurrences.
>> 
>> I attach the script as patch file.

Hi Stefano,

> 
> Thanks Luca! Actually I think this is great! The script makes it much
> easier to use the GCC output to address 10.3 violations. I think we
> should have it in the xen.git tree.

Ok I can refine it a bit in that case and push it to the ML.

> 
> Thanks to the script I was able to quickly address the top violation in
> bitmap.h.

I’m wondering if we need to remove from the list the excluded files, in this
first shot I didn’t do that but should be easy.

On Fri, 29 Sep 2023, Luca Fancellu wrote:
> > On 29 Sep 2023, at 00:20, Stefano Stabellini <sstabellini@kernel.org> wrote:
> > 
> > On Thu, 28 Sep 2023, Luca Fancellu wrote:
> >> Hi all,
> >> 
> >> In the last MISRA meeting Stefano told me about the Rule 10.3 and that we have a lot of violations,
> >> but GCC has a way to list them as written in rules.rst.
> >> 
> >> So the issue was just that we have a lot of violations, most of them coming from headers and the
> >> Developer might feel a bit lost since the number is very high, so I’ve developed this script to take
> >> the GCC build log and create a list of unique occurrences of the violations sorted in descending
> >> order, so that who has a bit of time can try to fix some of those and maybe can start from the top
> >> of the list to fix as many as possible occurrences.
> >> 
> >> I attach the script as patch file.
> 
> Hi Stefano,
> 
> > 
> > Thanks Luca! Actually I think this is great! The script makes it much
> > easier to use the GCC output to address 10.3 violations. I think we
> > should have it in the xen.git tree.
> 
> Ok I can refine it a bit in that case and push it to the ML.
> 
> > 
> > Thanks to the script I was able to quickly address the top violation in
> > bitmap.h.
> 
> I’m wondering if we need to remove from the list the excluded files, in this
> first shot I didn’t do that but should be easy.

Uhm, maybe with a command line option?

On 28/09/2023 09:34, Luca Fancellu wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> Hi all,
Hi Luca,
>
> In the last MISRA meeting Stefano told me about the Rule 10.3 and that we have a lot of violations,
> but GCC has a way to list them as written in rules.rst.
>
> So the issue was just that we have a lot of violations, most of them coming from headers and the
> Developer might feel a bit lost since the number is very high, so I’ve developed this script to take
> the GCC build log and create a list of unique occurrences of the violations sorted in descending
> order, so that who has a bit of time can try to fix some of those and maybe can start from the top
> of the list to fix as many as possible occurrences.
>
> I attach the script as patch file.

This is an excellent script to see the violations in a neat way. For eg, 
I see this

arch/arm/domain_build.c:3883:28 (1)
arch/arm/domain_build.c:3910:36 (1)
./include/xen/pfn.h:6:23 (1)
./include/xen/macros.h:4:40 (1)
arch/arm/setup.c:221:10 (1)
arch/arm/setup.c:227:59 (1)

I think once in a while, we have been approached by newbies wanting to 
contribute to Xen project.

For eg "Beginner looking to contribute to project in archived list"

Sometimes people reach out individually as well.

Do you have such similar tasks (like this) within or beyond the scope of 
MISRA which will require some basic programming skills and is a great 
first step to learn and contribute to Xen project ?

This can be things which students/folks can contribute on an ad-hoc basis.

I can work with Kelly to see if we can put them in a wiki page.

I could see https://wiki.xenproject.org/wiki/Outreach_Program_Projects , 
but don't know if it is up to date.

- Ayan

>
> Cheers,
> Luca
>

> Hi Luca,

Hi Ayan,

>> 
>> In the last MISRA meeting Stefano told me about the Rule 10.3 and that we have a lot of violations,
>> but GCC has a way to list them as written in rules.rst.
>> 
>> So the issue was just that we have a lot of violations, most of them coming from headers and the
>> Developer might feel a bit lost since the number is very high, so I’ve developed this script to take
>> the GCC build log and create a list of unique occurrences of the violations sorted in descending
>> order, so that who has a bit of time can try to fix some of those and maybe can start from the top
>> of the list to fix as many as possible occurrences.
>> 
>> I attach the script as patch file.
> 
> This is an excellent script to see the violations in a neat way. For eg, I see this
> 
> arch/arm/domain_build.c:3883:28 (1)
> arch/arm/domain_build.c:3910:36 (1)
> ./include/xen/pfn.h:6:23 (1)
> ./include/xen/macros.h:4:40 (1)
> arch/arm/setup.c:221:10 (1)
> arch/arm/setup.c:227:59 (1)
> 
> I think once in a while, we have been approached by newbies wanting to contribute to Xen project.
> 
> For eg "Beginner looking to contribute to project in archived list"
> 
> Sometimes people reach out individually as well.
> 
> Do you have such similar tasks (like this) within or beyond the scope of MISRA which will require some basic programming skills and is a great first step to learn and contribute to Xen project ?
> 
> This can be things which students/folks can contribute on an ad-hoc basis.

Yes indeed, these are the kind of fixes that could be handled by beginners contributors,
given the amount of them we need some help for sure!


> 
> I can work with Kelly to see if we can put them in a wiki page.
> 
> I could see https://wiki.xenproject.org/wiki/Outreach_Program_Projects , but don't know if it is up to date.


Sure we can sync with Kelly to see what we can do!

Cheers,
Luca

On Thu, 28 Sep 2023, Ayan Kumar Halder wrote:
> On 28/09/2023 09:34, Luca Fancellu wrote:
> > CAUTION: This message has originated from an External Source. Please use
> > proper judgment and caution when opening attachments, clicking links, or
> > responding to this email.
> > 
> > 
> > Hi all,
> Hi Luca,
> > 
> > In the last MISRA meeting Stefano told me about the Rule 10.3 and that we
> > have a lot of violations,
> > but GCC has a way to list them as written in rules.rst.
> > 
> > So the issue was just that we have a lot of violations, most of them coming
> > from headers and the
> > Developer might feel a bit lost since the number is very high, so I’ve
> > developed this script to take
> > the GCC build log and create a list of unique occurrences of the violations
> > sorted in descending
> > order, so that who has a bit of time can try to fix some of those and maybe
> > can start from the top
> > of the list to fix as many as possible occurrences.
> > 
> > I attach the script as patch file.
> 
> This is an excellent script to see the violations in a neat way. For eg, I see
> this
> 
> arch/arm/domain_build.c:3883:28 (1)
> arch/arm/domain_build.c:3910:36 (1)
> ./include/xen/pfn.h:6:23 (1)
> ./include/xen/macros.h:4:40 (1)
> arch/arm/setup.c:221:10 (1)
> arch/arm/setup.c:227:59 (1)
> 
> I think once in a while, we have been approached by newbies wanting to
> contribute to Xen project.
> 
> For eg "Beginner looking to contribute to project in archived list"
> 
> Sometimes people reach out individually as well.
> 
> Do you have such similar tasks (like this) within or beyond the scope of MISRA
> which will require some basic programming skills and is a great first step to
> learn and contribute to Xen project ?
> 
> This can be things which students/folks can contribute on an ad-hoc basis.
> 
> I can work with Kelly to see if we can put them in a wiki page.

That's a great idea


> I could see https://wiki.xenproject.org/wiki/Outreach_Program_Projects , but
> don't know if it is up to date.

I don't think it is