From nobody Fri May 10 07:20:13 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=quarantine dis=quarantine) header.from=virtuozzo.com Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1645716164724291.0242406135949; Thu, 24 Feb 2022 07:22:44 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-63-x1iqq_AzMcatXWpxDqIRnA-1; Thu, 24 Feb 2022 10:22:41 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8807780D6AB; Thu, 24 Feb 2022 15:22:33 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5C1E17DE3E; Thu, 24 Feb 2022 15:22:33 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id E0C5A1809CB2; Thu, 24 Feb 2022 15:22:32 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 21OFMV8v008856 for ; Thu, 24 Feb 2022 10:22:31 -0500 Received: by smtp.corp.redhat.com (Postfix) id 05B1C2026D64; Thu, 24 Feb 2022 15:22:31 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast09.extmail.prod.ext.rdu2.redhat.com [10.11.55.25]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 00F782026985 for ; Thu, 24 Feb 2022 15:22:26 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 71BAB2816CEF for ; Thu, 24 Feb 2022 15:22:26 +0000 (UTC) Received: from relay.sw.ru (130.117.225.111 [130.117.225.111]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-222-cHukNfu3MR2480N09p3aag-1; Thu, 24 Feb 2022 10:22:24 -0500 Received: from vz-out.virtuozzo.com ([185.231.240.5] helo=vzdev-s01.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.94.2) (envelope-from ) id 1nNFhB-009IxS-CS for libvir-list@redhat.com; Thu, 24 Feb 2022 16:06:36 +0100 X-MC-Unique: x1iqq_AzMcatXWpxDqIRnA-1 X-MC-Unique: cHukNfu3MR2480N09p3aag-1 From: Nikolay Shirokovskiy To: libvir-list@redhat.com Subject: [RFC PATCH 1/2] logging: touch opened files periodically Date: Thu, 24 Feb 2022 18:06:17 +0300 Message-Id: <20220224150618.47404-2-nshirokovskiy@virtuozzo.com> In-Reply-To: <20220224150618.47404-1-nshirokovskiy@virtuozzo.com> References: <20220224150618.47404-1-nshirokovskiy@virtuozzo.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1645716165833100001 Content-Type: text/plain; charset="utf-8" This will protect log files from being deleted by virtlogcleaner even if log is not being written active currently. --- src/logging/log_handler.c | 113 +++++++++++++++++++++++++++++++++----- 1 file changed, 98 insertions(+), 15 deletions(-) diff --git a/src/logging/log_handler.c b/src/logging/log_handler.c index 5c3df37415..fee4567911 100644 --- a/src/logging/log_handler.c +++ b/src/logging/log_handler.c @@ -34,6 +34,8 @@ #include #include #include +#include +#include =20 #include "configmake.h" =20 @@ -43,6 +45,8 @@ VIR_LOG_INIT("logging.log_handler"); =20 #define DEFAULT_MODE 0600 =20 +#define LOG_HANDLER_TOUCH_TIMEOUT (24 * 3600 * 1000) + typedef struct _virLogHandlerLogFile virLogHandlerLogFile; struct _virLogHandlerLogFile { virRotatingFileWriter *file; @@ -65,6 +69,8 @@ struct _virLogHandler { virLogHandlerLogFile **files; size_t nfiles; =20 + int timer; + virLogHandlerShutdownInhibitor inhibitor; void *opaque; }; @@ -102,6 +108,17 @@ virLogHandlerLogFileFree(virLogHandlerLogFile *file) } =20 =20 +static void +virLogHandlerCleanupTimer(virLogHandler *handler) +{ + if (handler->nfiles > 0 || handler->timer =3D=3D 0) + return; + + virEventRemoveTimeout(handler->timer); + handler->timer =3D 0; +} + + static void virLogHandlerLogFileClose(virLogHandler *handler, virLogHandlerLogFile *file) @@ -115,6 +132,8 @@ virLogHandlerLogFileClose(virLogHandler *handler, break; } } + + virLogHandlerCleanupTimer(handler); } =20 =20 @@ -209,6 +228,30 @@ virLogHandlerNew(bool privileged, } =20 =20 +/* + * This helper aims to handle races with file deleting by log file cleaner. + * Cleaner can unlink file right after we open it for write. In this case + * let's just recreate it. + * + */ +static virRotatingFileWriter * +virLogHandlerNewWriter(const char *path, + off_t maxlen, + size_t maxbackup, + bool trunc, + mode_t mode) +{ + virRotatingFileWriter *writer; + + writer =3D virRotatingFileWriterNew(path, maxlen, maxbackup, trunc, mo= de); + if (virFileExists(path)) + return writer; + + virRotatingFileWriterFree(writer); + return virRotatingFileWriterNew(path, maxlen, maxbackup, trunc, mode); +} + + static virLogHandlerLogFile * virLogHandlerLogFilePostExecRestart(virLogHandler *handler, virJSONValue *object) @@ -253,11 +296,11 @@ virLogHandlerLogFilePostExecRestart(virLogHandler *ha= ndler, goto error; } =20 - if ((file->file =3D virRotatingFileWriterNew(path, - handler->max_size, - handler->max_backups, - false, - DEFAULT_MODE)) =3D=3D NULL) + if ((file->file =3D virLogHandlerNewWriter(path, + handler->max_size, + handler->max_backups, + false, + DEFAULT_MODE)) =3D=3D NULL) goto error; =20 if (virJSONValueObjectGetNumberInt(object, "pipefd", &file->pipefd) < = 0) { @@ -280,6 +323,26 @@ virLogHandlerLogFilePostExecRestart(virLogHandler *han= dler, } =20 =20 +static void +virLogHandlerTimeout(int timer G_GNUC_UNUSED, + void *opaque) +{ + virLogHandler *handler =3D opaque; + size_t i; + + virObjectLock(handler); + + for (i =3D 0; i < handler->nfiles; i++) { + const char *path =3D virRotatingFileWriterGetPath(handler->files[i= ]->file); + + if (utime(path, NULL) < 0) + VIR_WARN("utime(%s) error: %s", path, g_strerror(errno)); + } + + virObjectUnlock(handler); +} + + virLogHandler * virLogHandlerNewPostExecRestart(virJSONValue *object, bool privileged, @@ -330,6 +393,11 @@ virLogHandlerNewPostExecRestart(virJSONValue *object, } } =20 + if (handler->nfiles > 0 && + (handler->timer =3D virEventAddTimeout(LOG_HANDLER_TOUCH_TIMEOUT, + virLogHandlerTimeout, + handler, NULL) <=3D 0)) + goto error; =20 return handler; =20 @@ -349,7 +417,10 @@ virLogHandlerDispose(void *obj) handler->inhibitor(false, handler->opaque); virLogHandlerLogFileFree(handler->files[i]); } + g_free(handler->files); + handler->nfiles =3D 0; + virLogHandlerCleanupTimer(handler); } =20 =20 @@ -393,11 +464,21 @@ virLogHandlerDomainOpenLogFile(virLogHandler *handler, file->driver =3D g_strdup(driver); file->domname =3D g_strdup(domname); =20 - if ((file->file =3D virRotatingFileWriterNew(path, - handler->max_size, - handler->max_backups, - trunc, - DEFAULT_MODE)) =3D=3D NULL) + /* + * Touch log files every day to prevent from removing by log files + * cleaner. + */ + if (handler->nfiles =3D=3D 0 && + (handler->timer =3D virEventAddTimeout(LOG_HANDLER_TOUCH_TIMEOUT, + virLogHandlerTimeout, + handler, NULL) <=3D 0)) + goto error; + + if ((file->file =3D virLogHandlerNewWriter(path, + handler->max_size, + handler->max_backups, + trunc, + DEFAULT_MODE)) =3D=3D NULL) goto error; =20 VIR_APPEND_ELEMENT_COPY(handler->files, handler->nfiles, file); @@ -418,6 +499,8 @@ virLogHandlerDomainOpenLogFile(virLogHandler *handler, return pipefd[1]; =20 error: + virLogHandlerCleanupTimer(handler); + VIR_FORCE_CLOSE(pipefd[0]); VIR_FORCE_CLOSE(pipefd[1]); handler->inhibitor(false, handler->opaque); @@ -579,11 +662,11 @@ virLogHandlerDomainAppendLogFile(virLogHandler *handl= er, } =20 if (!writer) { - if (!(newwriter =3D virRotatingFileWriterNew(path, - handler->max_size, - handler->max_backups, - false, - DEFAULT_MODE))) + if (!(newwriter =3D virLogHandlerNewWriter(path, + handler->max_size, + handler->max_backups, + false, + DEFAULT_MODE))) goto cleanup; =20 writer =3D newwriter; --=20 2.31.1 From nobody Fri May 10 07:20:13 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=quarantine dis=quarantine) header.from=virtuozzo.com Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1645716271775225.29040894170578; Thu, 24 Feb 2022 07:24:31 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-553-_F1BFDGJNLee3YHQJZMqnQ-1; Thu, 24 Feb 2022 10:24:25 -0500 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 345718145F5; Thu, 24 Feb 2022 15:24:20 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0E531106F777; Thu, 24 Feb 2022 15:24:20 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id BF1FE1809CB2; Thu, 24 Feb 2022 15:24:19 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 21OFOInE009107 for ; Thu, 24 Feb 2022 10:24:18 -0500 Received: by smtp.corp.redhat.com (Postfix) id 0ED3D40CFD11; Thu, 24 Feb 2022 15:24:18 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast10.extmail.prod.ext.rdu2.redhat.com [10.11.55.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0AA8940CFD15 for ; Thu, 24 Feb 2022 15:24:18 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E5B931C168EB for ; Thu, 24 Feb 2022 15:24:17 +0000 (UTC) Received: from relay.sw.ru (130.117.225.111 [130.117.225.111]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-eB6VF5SCN7iujRr4apjAig-1; Thu, 24 Feb 2022 10:24:14 -0500 Received: from vz-out.virtuozzo.com ([185.231.240.5] helo=vzdev-s01.qa.sw.ru) by relay.sw.ru with esmtp (Exim 4.94.2) (envelope-from ) id 1nNFhB-009IxS-JU for libvir-list@redhat.com; Thu, 24 Feb 2022 16:06:36 +0100 X-MC-Unique: _F1BFDGJNLee3YHQJZMqnQ-1 X-MC-Unique: eB6VF5SCN7iujRr4apjAig-1 From: Nikolay Shirokovskiy To: libvir-list@redhat.com Subject: [RFC PATCH 2/2] logging: add virtlogcleaner service Date: Thu, 24 Feb 2022 18:06:18 +0300 Message-Id: <20220224150618.47404-3-nshirokovskiy@virtuozzo.com> In-Reply-To: <20220224150618.47404-1-nshirokovskiy@virtuozzo.com> References: <20220224150618.47404-1-nshirokovskiy@virtuozzo.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1645716272723100001 Content-Type: text/plain; charset="utf-8" It will clean log files of qemu processes that are outdated. Here it means that VM where not active for given amount time (currently hardcoded to a month). --- src/logging/meson.build | 15 +++++++++++++++ src/logging/virtlogcleaner.service.in | 7 +++++++ src/logging/virtlogcleaner.sh | 9 +++++++++ src/logging/virtlogcleaner.timer | 8 ++++++++ src/logging/virtlogd.service.in | 1 + 5 files changed, 40 insertions(+) create mode 100644 src/logging/virtlogcleaner.service.in create mode 100755 src/logging/virtlogcleaner.sh create mode 100644 src/logging/virtlogcleaner.timer diff --git a/src/logging/meson.build b/src/logging/meson.build index 7066f16fad..d23f51b9fd 100644 --- a/src/logging/meson.build +++ b/src/logging/meson.build @@ -101,6 +101,21 @@ if conf.has('WITH_LIBVIRTD') 'name': 'virtlogd', 'in_file': files('virtlogd.init.in'), } + + if init_script =3D=3D 'systemd' + systemd_unit_dir =3D prefix / 'lib' / 'systemd' / 'system' + + configure_file( + input: 'virtlogcleaner.service.in', + output: 'virtlogcleaner.service', + configuration: configuration_data({'sbindir': sbindir}), + install: true, + install_dir: systemd_unit_dir, + ) + + install_data('virtlogcleaner.timer', install_dir: systemd_unit_dir) + install_data('virtlogcleaner.sh', install_dir: sbindir) + endif endif =20 log_inc_dir =3D include_directories('.') diff --git a/src/logging/virtlogcleaner.service.in b/src/logging/virtlogcle= aner.service.in new file mode 100644 index 0000000000..1d1ff2b121 --- /dev/null +++ b/src/logging/virtlogcleaner.service.in @@ -0,0 +1,7 @@ +[Unit] +Description=3DVirtual machine log cleaner +Documentation=3Dhttps://libvirt.org + +[Service] +Type=3Doneshot +ExecStart=3D@sbindir@/virtlogcleaner.sh diff --git a/src/logging/virtlogcleaner.sh b/src/logging/virtlogcleaner.sh new file mode 100755 index 0000000000..21ddefb15a --- /dev/null +++ b/src/logging/virtlogcleaner.sh @@ -0,0 +1,9 @@ +#!/bin/sh + +logdir=3D/var/log/libvirt/qemu + +cd "$logdir" +find . -mtime +30 -name "*.log" | while read file +do + find . -regex "$file.[0-9]*" -delete && rm "$file" +done diff --git a/src/logging/virtlogcleaner.timer b/src/logging/virtlogcleaner.= timer new file mode 100644 index 0000000000..a916f05f87 --- /dev/null +++ b/src/logging/virtlogcleaner.timer @@ -0,0 +1,8 @@ +[Unit] +Description=3DPeriodic cleanup of virtual machine logs +Documentation=3Dhttps://libvirt.org + +[Timer] +OnCalendar=3Ddaily +AccuracySec=3D1h +Persistent=3Dtrue diff --git a/src/logging/virtlogd.service.in b/src/logging/virtlogd.service= .in index 8ab5478517..e1883b73be 100644 --- a/src/logging/virtlogd.service.in +++ b/src/logging/virtlogd.service.in @@ -2,6 +2,7 @@ Description=3DVirtual machine log manager Requires=3Dvirtlogd.socket Requires=3Dvirtlogd-admin.socket +Requires=3Dvirtlogcleaner.timer Before=3Dlibvirtd.service Documentation=3Dman:virtlogd(8) Documentation=3Dhttps://libvirt.org --=20 2.31.1