Adapt the configuration of redirects from the server hosting libvirt.org

The redirect to the 'libvirt-console-proxy' Golang module is not adapted
as it doesn't exist on the current server.

NOTES:
 - The redirects are currently configured for hosting via the
   'gitlab.io/libvirt' URI. For hosting via custom domain it will need
   to be rewritten to drop the '/libvirt' prefixes.

 - gitlab pages doesn't currently support redirects to outside content,
   thus most of the redirects don't actually work:

     https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601

 - The redirects file is only installed in the webpage job but is not
   actually distributed.

 - The validity of the redirects can be checked by visiting:

     https://libvirt.gitlab.io/_redirects

   Having them installed allows us to validate them before we'll be
   switching to use gitlab pages completely.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
---
 .gitlab-ci.yml         |  1 +
 docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)
 create mode 100644 docs/.gitlab_redirects

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6c7d54c311..1ee945aa0a 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -130,6 +130,7 @@ website_prebuilt_env_legacy:
   after_script:
     - test "$CI_JOB_STATUS" != "success" && exit 1;
     - mv install/usr/share/doc/libvirt/html/ public
+    - cp docs/.gitlab_redirects public/_redirects
   artifacts:
     expose_as: 'pages'
     name: 'pages'
diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects
new file mode 100644
index 0000000000..26e64563ee
--- /dev/null
+++ b/docs/.gitlab_redirects
@@ -0,0 +1,54 @@
+# Redirects to golang module pages
+/libvirt/libvirt-go /libvirt-go.html 200
+/libvirt/libvirt-go-xml /libvirt-go-xml.html 200
+/libvirt/go/libvirt /go/libvirt.html 200
+/libvirt/go/libvirtxml /go/libvirtxml.html 200
+
+# Redirects to the download server
+/libvirt/sources/* https://download.libvirt.org/:splat 301
+/libvirt/ruby/download/* https://download.libvirt.org/ruby/:splat 301
+/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
+
+# Redirects to subproject pages
+/libvirt/ruby/* https://ruby.libvirt.org/:splat 301
+/libvirt/ocaml/* https://ocaml.libvirt.org/:splat 301
+/libvirt/php/* https://php.libvirt.org/:splat 301
+/libvirt/libvirt-appdev-guide-python/en-US/html/* https://libvirt.gitlab.io/libvirt-appdev-guide-python/:splat: 301
+/libvirt/java.html https://java.libvirt.org 301
+# Redirect to the proper javadoc directory on the subproject page
+/libvirt/sources/java/javadoc/* https://java.libvirt.org/javadoc/:splat 301
+
+# Redirects from old gitweb location (see below)
+/libvirt/git https://gitlab.com/libvirt/ 301
+
+# The above rules are adapted from the following set of 'mod_rewrite' rules used
+# originally on libvirt.org:
+#
+#    RewriteRule ^/libvirt-go$ /libvirt-go.html [L]
+#    RewriteRule ^/libvirt-go-xml$ /libvirt-go-xml.html [L]
+#    RewriteRule ^/go/libvirt$ /go/libvirt.html [L]
+#    RewriteRule ^/go/libvirtxml$ /go/libvirtxml.html [L]
+#    RewriteRule ^/sources/(.*) https://download.libvirt.org/$1 [L]
+#    RewriteRule ^/ruby/download/(.*) https://download.libvirt.org/ruby/$1 [L]
+#    RewriteRule ^/(maven2/.*) https://download.libvirt.org/$1 [L]
+#    RewriteRule ^/ocaml/(.*) https://ocaml.libvirt.org/$1 [L]
+#    RewriteRule ^/ruby/(.*) https://ruby.libvirt.org/$1 [L]
+#    RewriteRule ^/php/(.*) https://php.libvirt.org/$1 [L]
+#    RewriteRule ^/java.html https://java.libvirt.org [L]
+#    RewriteRule ^/docs/libvirt-appdev-guide-python/en-US/html/(.*) https://libvirt.gitlab.io/libvirt-appdev-guide-python/$1 [L]
+#    RewriteRule ^/git https://gitlab.com/libvirt/ [L]
+#
+#  Redirect replacing 'gitweb'. The 'gitweb' interface was originally replaced
+#  by the following redirect condition:
+#
+#    RewriteCond %{QUERY_STRING} p=([-a-zA-Z0-9]+).git
+#    RewriteRule ^/git/$ https://gitlab.com/libvirt/%1 [L]
+#
+# That unfortunately can't be represented in gitlab redirects as it doesn't
+# support redirects based on query strings. Given that the above redirect broke
+# most gitweb links anyways, due to handling only the 'p=' argument, git gitlab
+# redirect will break the rest of them.
+#
+#  The following rule was dropped as the page never existed:
+#
+#    RewriteRule ^/libvirt-console-proxy$ /libvirt-console-proxy.html [L]
-- 
2.43.0
_______________________________________________
Devel mailing list -- devel@lists.libvirt.org
To unsubscribe send an email to devel-leave@lists.libvirt.org

On Wed, Jan 10, 2024 at 02:30:23PM +0100, Peter Krempa wrote:
> Adapt the configuration of redirects from the server hosting libvirt.org
> 
> The redirect to the 'libvirt-console-proxy' Golang module is not adapted
> as it doesn't exist on the current server.
> 
> NOTES:
>  - The redirects are currently configured for hosting via the
>    'gitlab.io/libvirt' URI. For hosting via custom domain it will need
>    to be rewritten to drop the '/libvirt' prefixes.
> 
>  - gitlab pages doesn't currently support redirects to outside content,
>    thus most of the redirects don't actually work:
> 
>      https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
> 
>  - The redirects file is only installed in the webpage job but is not
>    actually distributed.
> 
>  - The validity of the redirects can be checked by visiting:
> 
>      https://libvirt.gitlab.io/_redirects
> 
>    Having them installed allows us to validate them before we'll be
>    switching to use gitlab pages completely.
> 
> Signed-off-by: Peter Krempa <pkrempa@redhat.com>
> ---
>  .gitlab-ci.yml         |  1 +
>  docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 55 insertions(+)
>  create mode 100644 docs/.gitlab_redirects
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 6c7d54c311..1ee945aa0a 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -130,6 +130,7 @@ website_prebuilt_env_legacy:
>    after_script:
>      - test "$CI_JOB_STATUS" != "success" && exit 1;
>      - mv install/usr/share/doc/libvirt/html/ public
> +    - cp docs/.gitlab_redirects public/_redirects
>    artifacts:
>      expose_as: 'pages'
>      name: 'pages'
> diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects
> new file mode 100644
> index 0000000000..26e64563ee
> --- /dev/null
> +++ b/docs/.gitlab_redirects
> @@ -0,0 +1,54 @@

> +/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301

Maven has this unhelpful behaviour where by when you add libvirt.org
as a maven repository host, it'll now consult libvirt.org for *all*
Java JARs, not solely the libvirt JARs.

For all of last week we were serving about 1 million hits a day
for /maven, of which the libvirt JAR requests were about 0.5%,
the rest were 404s ! This is why libvirt.org was painfully slow
for most of last week.

This was all made worse by us redirecting from libvirt.org to
download.libvirt.org, needlessly doubling the number of requests
had to serve. 

I changed the /maven2/* redirect to /maven2/org/libvirt/* which
lets libvirt.org service an immediate 404, avoiding hitting
download.libvirt.org, cutting our requests by 50%. Fortunately
traffic has died down to more normal levels after the "back
to work mass code recompile"  spike has gone.

Anyway lets do the same for this gitlab redirect to keep the
bogus maven traffic for non-existent files off download.libvirt.org


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
_______________________________________________
Devel mailing list -- devel@lists.libvirt.org
To unsubscribe send an email to devel-leave@lists.libvirt.org

On Wed, Jan 10, 2024 at 15:02:25 +0000, Daniel P. Berrangé wrote:
> On Wed, Jan 10, 2024 at 02:30:23PM +0100, Peter Krempa wrote:
> > Adapt the configuration of redirects from the server hosting libvirt.org
> > 
> > The redirect to the 'libvirt-console-proxy' Golang module is not adapted
> > as it doesn't exist on the current server.
> > 
> > NOTES:
> >  - The redirects are currently configured for hosting via the
> >    'gitlab.io/libvirt' URI. For hosting via custom domain it will need
> >    to be rewritten to drop the '/libvirt' prefixes.
> > 
> >  - gitlab pages doesn't currently support redirects to outside content,
> >    thus most of the redirects don't actually work:
> > 
> >      https://gitlab.com/gitlab-org/gitlab-pages/-/issues/601
> > 
> >  - The redirects file is only installed in the webpage job but is not
> >    actually distributed.
> > 
> >  - The validity of the redirects can be checked by visiting:
> > 
> >      https://libvirt.gitlab.io/_redirects
> > 
> >    Having them installed allows us to validate them before we'll be
> >    switching to use gitlab pages completely.
> > 
> > Signed-off-by: Peter Krempa <pkrempa@redhat.com>
> > ---
> >  .gitlab-ci.yml         |  1 +
> >  docs/.gitlab_redirects | 54 ++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 55 insertions(+)
> >  create mode 100644 docs/.gitlab_redirects
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 6c7d54c311..1ee945aa0a 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -130,6 +130,7 @@ website_prebuilt_env_legacy:
> >    after_script:
> >      - test "$CI_JOB_STATUS" != "success" && exit 1;
> >      - mv install/usr/share/doc/libvirt/html/ public
> > +    - cp docs/.gitlab_redirects public/_redirects
> >    artifacts:
> >      expose_as: 'pages'
> >      name: 'pages'
> > diff --git a/docs/.gitlab_redirects b/docs/.gitlab_redirects
> > new file mode 100644
> > index 0000000000..26e64563ee
> > --- /dev/null
> > +++ b/docs/.gitlab_redirects
> > @@ -0,0 +1,54 @@
> 
> > +/libvirt/maven2/* https://download.libvirt.org/maven2/:splat 301
> 
> Maven has this unhelpful behaviour where by when you add libvirt.org
> as a maven repository host, it'll now consult libvirt.org for *all*
> Java JARs, not solely the libvirt JARs.
> 
> For all of last week we were serving about 1 million hits a day
> for /maven, of which the libvirt JAR requests were about 0.5%,
> the rest were 404s ! This is why libvirt.org was painfully slow
> for most of last week.
> 
> This was all made worse by us redirecting from libvirt.org to
> download.libvirt.org, needlessly doubling the number of requests
> had to serve. 
> 
> I changed the /maven2/* redirect to /maven2/org/libvirt/* which
> lets libvirt.org service an immediate 404, avoiding hitting
> download.libvirt.org, cutting our requests by 50%. Fortunately
> traffic has died down to more normal levels after the "back
> to work mass code recompile"  spike has gone.
> 
> Anyway lets do the same for this gitlab redirect to keep the
> bogus maven traffic for non-existent files off download.libvirt.org

Ah, right. I've captured the redirect list before the holliday break
(actually the original version is from April) so it didn't have this
one and didn't bother to check once more.

The final check needs to be done before switching over anyways.

I've changed the rule in my branch to:

/libvirt/maven2/org/libvirt* https://download.libvirt.org/maven2/org/libvirt/:splat 301
_______________________________________________
Devel mailing list -- devel@lists.libvirt.org
To unsubscribe send an email to devel-leave@lists.libvirt.org