Following the recent discussion, we had on IRC and the action I had in 
the March community call, this file provides a file format that 
enables writing an automated test to check whether files are out of sync. 

Unlike in the IRC discussion, which suggested a single line for all
information, I broke out the repository into a separate statement for
- Better readability (aka shorter lines)
- Better maintainability if a repo URL changes

The list of files that need to be included are

Once the file format is agree, I will write a test or script.

I also need some more correct test data, aka entries in the file from
committers looking after the following files
[Jan]
xen/arch/x86/cpu/mwait-idle.c 
[Stefano, Julien - this has to be finalized]
xen/drivers/passthrough/arm/smmu.c
xen/arch/arm/vgic/*
xen/include/asm-arm/div64.h
xen/drivers/char/meson-uart.c
xen/arch/arm/arm32/lib/*
xen/arch/arm/arm64/lib/*
xen/arch/arm/arm64/cache.S
xen/arch/arm/arm64/bpi.S
xen/include/asm-arm/system.h
xen/arch/arm/arm64/insn.c
[Others?]
xen/common/rbtree.c

Note that in some cases Linux has diverged and some Linux files have disappeared. 
Julien also raised the point, that in some cases only a subset of code from Linux Xen files was applied or that only some functions get moved across to Xen.

I believe that is entirely OK. The workflow would be in most cases that:
- We use a Linux (source) commit as a benchmark and record the commit ID
- If there is a change in Linux the test will fail
- The committer looks at the diff and either
  - Decides to ignore it and bumps the commit ID in this file
  - Decides the change is needed, integrates it into Xen and then 
    bumps the commit ID in this file

Signed-off-by: Lars Kurth <lars.kurth@citrix.com>
CC: committers@xenproject.org
---
 TRACKING.IMPORTS | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
 create mode 100644 TRACKING.IMPORTS

diff --git a/TRACKING.IMPORTS b/TRACKING.IMPORTS
new file mode 100644
index 0000000000..39829e078c
--- /dev/null
+++ b/TRACKING.IMPORTS
@@ -0,0 +1,40 @@
+# This file contains information about source files that have been
+# copied from other sources and need to be tracked
+#
+# The file may contain lines starting with ...
+# --------------------------------------------
+# version: of file format
+# repo: repository definition
+# auto|manual: a mapping to track files
+#
+# Repository Definitions are of the following format
+# --------------------------------------------------
+# repo name-of-source-repo git|svn https-url-of-source-repo
+#
+# For example:
+#   repo linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+#
+# Mappings to track files are of the following format
+# ---------------------------------------------------
+# manual|auto xen-file name-of-original-repo original-file commit-id
+#
+# auto:
+#   The xen-file needs to track the the original-file exactly
+#   In other words, we can automatically update the file using a script
+#
+# manual:
+#   A developer needs to make a decision whether a
+#   specific change is applied or ignored and update the last commit id
+#   accordingly
+#
+# name-of-original-repo:
+#   A reference to a source repository defined by *repo* keyword
+#
+# commit id:
+#   Last commit id of source file that was deemed to be ok
+#   and either imported into the tree or rejected
+#
+# For example:
+#   manual xen/drivers/passthrough/arm/smmu.c linux-torvalds linux/drivers/iommu/arm-smmu.c b77cf11f094136
+
+version 1
-- 
2.13.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

>>> On 16.05.19 at 00:18, <lars.kurth@citrix.com> wrote:
> --- /dev/null
> +++ b/TRACKING.IMPORTS
> @@ -0,0 +1,40 @@
> +# This file contains information about source files that have been
> +# copied from other sources and need to be tracked
> +#
> +# The file may contain lines starting with ...
> +# --------------------------------------------
> +# version: of file format
> +# repo: repository definition
> +# auto|manual: a mapping to track files
> +#
> +# Repository Definitions are of the following format
> +# --------------------------------------------------
> +# repo name-of-source-repo git|svn https-url-of-source-repo
> +#
> +# For example:
> +#   repo linux-torvalds git https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 
> +#
> +# Mappings to track files are of the following format
> +# ---------------------------------------------------
> +# manual|auto xen-file name-of-original-repo original-file commit-id
> +#
> +# auto:
> +#   The xen-file needs to track the the original-file exactly
> +#   In other words, we can automatically update the file using a script

Do we have _any_ example of this? I can't even imagine one, due
to e.g. our includes all starting with xen/ whereas Linux'es (just to
take as example) all start with linux/. Perhaps "auto" needs to
include sed expressions that need to be applied before actually
applying the original change to our tree?

> +# manual:
> +#   A developer needs to make a decision whether a
> +#   specific change is applied or ignored and update the last commit id
> +#   accordingly
> +#
> +# name-of-original-repo:
> +#   A reference to a source repository defined by *repo* keyword
> +#
> +# commit id:
> +#   Last commit id of source file that was deemed to be ok
> +#   and either imported into the tree or rejected
> +#
> +# For example:
> +#   manual xen/drivers/passthrough/arm/smmu.c linux-torvalds linux/drivers/iommu/arm-smmu.c b77cf11f094136
> +
> +version 1

Perhaps it wouldn't hurt to include the colons in the actual entries as
well? I also don't think examples are needed once we get the first
real entries. Hence I'd move them to the commit message or a
post-commit message remark.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

On 16/05/2019, 04:47, "Jan Beulich" <JBeulich@suse.com> wrote:

    >>> On 16.05.19 at 00:18, <lars.kurth@citrix.com> wrote:
    > +# Mappings to track files are of the following format
    > +# ---------------------------------------------------
    > +# manual|auto xen-file name-of-original-repo original-file commit-id
    > +#
    > +# auto:
    > +#   The xen-file needs to track the the original-file exactly
    > +#   In other words, we can automatically update the file using a script
    
    Do we have _any_ example of this? I can't even imagine one, due
    to e.g. our includes all starting with xen/ whereas Linux'es (just to
    take as example) all start with linux/. Perhaps "auto" needs to
    include sed expressions that need to be applied before actually
    applying the original change to our tree?

I am not sure I fully understand your concern. 
This was intended for the case where say we would exactly track xen/.../foo.bar with linux/.../foo.bar
In other words, auto only applies to the content of a file: the filename isn't relevant, because all the information that would be needed to do this is in the file.

But if there is no need for it, we can drop it. And if needed, we can add in future.
@Julien, @Stefano, @Jan: are any of the files you listed fall into the "should be tracked exactly" category?
    
    > +# manual:
    > +#   A developer needs to make a decision whether a
    > +#   specific change is applied or ignored and update the last commit id
    > +#   accordingly
    > +#
    > +# name-of-original-repo:
    > +#   A reference to a source repository defined by *repo* keyword
    > +#
    > +# commit id:
    > +#   Last commit id of source file that was deemed to be ok
    > +#   and either imported into the tree or rejected
    > +#
    > +# For example:
    > +#   manual xen/drivers/passthrough/arm/smmu.c linux-torvalds linux/drivers/iommu/arm-smmu.c b77cf11f094136
    > +
    > +version 1
    
    Perhaps it wouldn't hurt to include the colons in the actual entries as
    well? 

I am not sure what you mean, which colons? Are you saying, the format should be
version: 1
repo: ...

I think the confusion comes because I used colons after statements in the comments. 
I think that "version: 1" is slightly more human-readable, so I would be OK with that

    I also don't think examples are needed once we get the first
    real entries. Hence I'd move them to the commit message or a
    post-commit message remark.

Agreed.

Lars


    
    

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

>>> On 16.05.19 at 17:54, <lars.kurth@citrix.com> wrote:

> 
> On 16/05/2019, 04:47, "Jan Beulich" <JBeulich@suse.com> wrote:
> 
>     >>> On 16.05.19 at 00:18, <lars.kurth@citrix.com> wrote:
>     > +# Mappings to track files are of the following format
>     > +# ---------------------------------------------------
>     > +# manual|auto xen-file name-of-original-repo original-file commit-id
>     > +#
>     > +# auto:
>     > +#   The xen-file needs to track the the original-file exactly
>     > +#   In other words, we can automatically update the file using a script
>     
>     Do we have _any_ example of this? I can't even imagine one, due
>     to e.g. our includes all starting with xen/ whereas Linux'es (just to
>     take as example) all start with linux/. Perhaps "auto" needs to
>     include sed expressions that need to be applied before actually
>     applying the original change to our tree?
> 
> I am not sure I fully understand your concern. 
> This was intended for the case where say we would exactly track 
> xen/.../foo.bar with linux/.../foo.bar
> In other words, auto only applies to the content of a file: the filename 
> isn't relevant, because all the information that would be needed to do this 
> is in the file.

When talking about file names in my reply, I referred to C language
#include directives inside the file in question, as a (pretty important)
example. There was no talk about the cloned/copied file's name itself.
Hence the suggestion to accompany auto: with a set of sed
expressions, which could then e.g. transform #include <linux/...>
into #include <xen/...>.

> @Julien, @Stefano, @Jan: are any of the files you listed fall into the 
> "should be tracked exactly" category?

As I've said before - I can't even imagine such a file to exist.

>     > +# manual:
>     > +#   A developer needs to make a decision whether a
>     > +#   specific change is applied or ignored and update the last commit id
>     > +#   accordingly
>     > +#
>     > +# name-of-original-repo:
>     > +#   A reference to a source repository defined by *repo* keyword
>     > +#
>     > +# commit id:
>     > +#   Last commit id of source file that was deemed to be ok
>     > +#   and either imported into the tree or rejected
>     > +#
>     > +# For example:
>     > +#   manual xen/drivers/passthrough/arm/smmu.c linux-torvalds 
> linux/drivers/iommu/arm-smmu.c b77cf11f094136
>     > +
>     > +version 1
>     
>     Perhaps it wouldn't hurt to include the colons in the actual entries as
>     well? 
> 
> I am not sure what you mean, which colons? Are you saying, the format should be
> version: 1
> repo: ...

Yes. This would make it even more prominent that these are tags of
some sort. But this was only a thought of mine, it's in no way meant
to be a requirement I have.

> I think the confusion comes because I used colons after statements in the 
> comments. 

Right, that's how I got there.

> I think that "version: 1" is slightly more human-readable, so I would be OK 
> with that

A well defined non-blank separator also allows machine processing
to notice more easily if there's a malformed line. Plus (if need be)
it would permit tags with blanks in their names.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

On 17/05/2019, 01:34, "Jan Beulich" <JBeulich@suse.com> wrote:

    >>> On 16.05.19 at 17:54, <lars.kurth@citrix.com> wrote:
    
    > 
    > On 16/05/2019, 04:47, "Jan Beulich" <JBeulich@suse.com> wrote:
    > 
    >     >>> On 16.05.19 at 00:18, <lars.kurth@citrix.com> wrote:
    >     > +# Mappings to track files are of the following format
    >     > +# ---------------------------------------------------
    >     > +# manual|auto xen-file name-of-original-repo original-file commit-id
    >     > +#
    >     > +# auto:
    >     > +#   The xen-file needs to track the the original-file exactly
    >     > +#   In other words, we can automatically update the file using a script
    >     
    >     Do we have _any_ example of this? I can't even imagine one, due
    >     to e.g. our includes all starting with xen/ whereas Linux'es (just to
    >     take as example) all start with linux/. Perhaps "auto" needs to
    >     include sed expressions that need to be applied before actually
    >     applying the original change to our tree?
    > 
    > I am not sure I fully understand your concern. 
    > This was intended for the case where say we would exactly track 
    > xen/.../foo.bar with linux/.../foo.bar
    > In other words, auto only applies to the content of a file: the filename 
    > isn't relevant, because all the information that would be needed to do this 
    > is in the file.
    
    When talking about file names in my reply, I referred to C language
    #include directives inside the file in question, as a (pretty important)
    example. There was no talk about the cloned/copied file's name itself.
    Hence the suggestion to accompany auto: with a set of sed
    expressions, which could then e.g. transform #include <linux/...>
    into #include <xen/...>.

That makes perfect sense now. In that case, I tend to agree that "auto" is probably not needed. Would be quite happy to drop it.
    
    > @Julien, @Stefano, @Jan: are any of the files you listed fall into the 
    > "should be tracked exactly" category?
    
    As I've said before - I can't even imagine such a file to exist.
    
    >     > +# manual:
    >     > +#   A developer needs to make a decision whether a
    >     > +#   specific change is applied or ignored and update the last commit id
    >     > +#   accordingly
    >     > +#
    >     > +# name-of-original-repo:
    >     > +#   A reference to a source repository defined by *repo* keyword
    >     > +#
    >     > +# commit id:
    >     > +#   Last commit id of source file that was deemed to be ok
    >     > +#   and either imported into the tree or rejected
    >     > +#
    >     > +# For example:
    >     > +#   manual xen/drivers/passthrough/arm/smmu.c linux-torvalds 
    > linux/drivers/iommu/arm-smmu.c b77cf11f094136
    >     > +
    >     > +version 1
    >     
    >     Perhaps it wouldn't hurt to include the colons in the actual entries as
    >     well? 
    > 
    > I am not sure what you mean, which colons? Are you saying, the format should be
    > version: 1
    > repo: ...
    
    Yes. This would make it even more prominent that these are tags of
    some sort. But this was only a thought of mine, it's in no way meant
    to be a requirement I have.
    
    > I think the confusion comes because I used colons after statements in the 
    > comments. 
    
    Right, that's how I got there.
    
    > I think that "version: 1" is slightly more human-readable, so I would be OK 
    > with that
    
    A well defined non-blank separator also allows machine processing
    to notice more easily if there's a malformed line. Plus (if need be)
    it would permit tags with blanks in their names.
    
I can do that. No problem.

Any other comments from anyone, before sending version 2?

Lars    
    

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

This is going in the right direction IMO.

Lars Kurth writes ("Re: [PATCH] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream"):
> That makes perfect sense now. In that case, I tend to agree that "auto" is probably not needed. Would be quite happy to drop it.

It will considerably complicate things to add a way to define
seddery.  Let us leave that to a future extension.

That suggests that `manual' should become `file:'.

As for delimiters

> On 17/05/2019, 01:34, "Jan Beulich" <JBeulich@suse.com> wrote:
>     > I am not sure what you mean, which colons? Are you saying, the format should be
>     > version: 1
>     > repo: ...
>     
>     Yes. This would make it even more prominent that these are tags of
>     some sort. But this was only a thought of mine, it's in no way meant
>     to be a requirement I have.

It will make writing a parser easier if each entry is a single line
with the fields in a defined order and if we can say that a `repo:'
must precede every `file:' that mentions it.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

On 20/05/2019, 10:37, "Ian Jackson" <ian.jackson@citrix.com> wrote:

    This is going in the right direction IMO.
    
    Lars Kurth writes ("Re: [PATCH] Add TRACKING.IMPORTS to xen.git to more easily manage imported files that need to be kept in sync with an upstream"):
    > That makes perfect sense now. In that case, I tend to agree that "auto" is probably not needed. Would be quite happy to drop it.
    
    It will considerably complicate things to add a way to define
    seddery.  Let us leave that to a future extension.
    
    That suggests that `manual' should become `file:'.
    
    As for delimiters
    
    > On 17/05/2019, 01:34, "Jan Beulich" <JBeulich@suse.com> wrote:
    >     > I am not sure what you mean, which colons? Are you saying, the format should be
    >     > version: 1
    >     > repo: ...
    >     
    >     Yes. This would make it even more prominent that these are tags of
    >     some sort. But this was only a thought of mine, it's in no way meant
    >     to be a requirement I have.
    
    It will make writing a parser easier if each entry is a single line
    with the fields in a defined order and if we can say that a `repo:'
    must precede every `file:' that mentions it.
    
That sounds perfectly sensible
Version 2 on its way, sometimes next week

Lars
    

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel