[Patchew-devel] [PATCH 0/5] next.patchew.org doesn't use blobs anymore!

Paolo Bonzini posted 5 patches 2 years, 6 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/patchew-ci tags/patchew/20220407084704.78533-1-pbonzini@redhat.com
api/blobs.py                                  | 38 ------------------
api/migrations/0030_deblob_properties.py      |  4 +-
api/migrations/0061_message_mbox_bytes.py     | 18 +++++++++
api/migrations/0062_deblob_messages.py        | 40 +++++++++++++++++++
api/migrations/0063_remove_broken_messages.py | 19 +++++++++
api/migrations/0064_auto_20220407_0742.py     | 18 +++++++++
api/migrations/__init__.py                    | 24 +++++++++--
api/models.py                                 | 20 ++++------
8 files changed, 125 insertions(+), 56 deletions(-)
delete mode 100644 api/blobs.py
create mode 100644 api/migrations/0061_message_mbox_bytes.py
create mode 100644 api/migrations/0062_deblob_messages.py
create mode 100644 api/migrations/0063_remove_broken_messages.py
create mode 100644 api/migrations/0064_auto_20220407_0742.py
[Patchew-devel] [PATCH 0/5] next.patchew.org doesn't use blobs anymore!
Posted by Paolo Bonzini 2 years, 6 months ago
I have finished the (small) amount of work needed to remove blobs completely
from patchew and deployed the change to next.patchew.org.  The bulk of the work
can be done while the server is up, thanks to a migration script that can
run within "./manage.py shell" and to a bit of live database hacking. :)

next.patchew.org seems to be a lot snappier than before, and my worries
about consuming more disk space (because .xz compression is removed) were
unfounded.  A directory with a million files is a bit heavy on the
filesystem, evidently.

The process to deploy these patches is as follows:

- push the first patch, and run from the shell the following command
  (which will take a long time)

    from scripts import blob_migrate
    blob_migrate.doit()

- backup what's left of the blobs directory

- push the second and third patch (no more slow chown step!)

- do the queries in the commit messages of the fourth patch

- push the last two patches as well.

I plan to do this sometime next week on patchew.org.

Paolo

Paolo Bonzini (5):
  models: start moving away from placing mboxes in the filesystem
  models: change ad hoc deblobbing script to migration
  models: confine blob handling to migrations
  models: remove messages for which the blobs were lost
  models: make mbox_bytes non-null

 api/blobs.py                                  | 38 ------------------
 api/migrations/0030_deblob_properties.py      |  4 +-
 api/migrations/0061_message_mbox_bytes.py     | 18 +++++++++
 api/migrations/0062_deblob_messages.py        | 40 +++++++++++++++++++
 api/migrations/0063_remove_broken_messages.py | 19 +++++++++
 api/migrations/0064_auto_20220407_0742.py     | 18 +++++++++
 api/migrations/__init__.py                    | 24 +++++++++--
 api/models.py                                 | 20 ++++------
 8 files changed, 125 insertions(+), 56 deletions(-)
 delete mode 100644 api/blobs.py
 create mode 100644 api/migrations/0061_message_mbox_bytes.py
 create mode 100644 api/migrations/0062_deblob_messages.py
 create mode 100644 api/migrations/0063_remove_broken_messages.py
 create mode 100644 api/migrations/0064_auto_20220407_0742.py

-- 
2.35.1

_______________________________________________
Patchew-devel mailing list
Patchew-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/patchew-devel