From nobody Tue Feb 10 15:01:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=quarantine dis=quarantine) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1671045983; cv=none; d=zohomail.com; s=zohoarc; b=MRYZfhxYxkd3xYsQvuFCI7E6KNVg8YckESvkj2fM6S+merwqTf681zOyKwpYaBkFEAou1GQQpIb8+uN7EjhIKQ6h84DsfkCwGhyxoYpsRvbdOW5HwVwJ8RFj6ajvURa/FABEh6LCyxtvajsrojCGEgIO6HxrEVOrAmPoRj14xF0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1671045983; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=sw9NRhWswb8z5id/w7lCrwjbDdhapaAFwgQm48Ft/wE=; b=amKUH/gqsKuA9Pgk5MYPam4Cy9Bou8U1YU8OAdZxOKMvMDi18Y5+oGylXSPRFr9ZR1HQZTuBk7RU4+yoY3Y+Pv6AGZWKQhoc60kDWRB6Yi5Jncpci65KEetLwwaB91LeyCxa5ViMMrfxsv0+V8Izr+GySv+x2em7c9/k/yl6rfE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail header.from= (p=quarantine dis=quarantine) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1671045983046257.9763959547888; Wed, 14 Dec 2022 11:26:23 -0800 (PST) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-509-zYhSX9SKNraxL39IydM53w-1; Wed, 14 Dec 2022 14:26:20 -0500 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 58654805564; Wed, 14 Dec 2022 19:26:13 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 407AB492C14; Wed, 14 Dec 2022 19:26:13 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id A9D511946A66; Wed, 14 Dec 2022 19:26:12 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 4AA4D1946A4E for ; Wed, 14 Dec 2022 19:26:12 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 3D07F14171C0; Wed, 14 Dec 2022 19:26:12 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast09.extmail.prod.ext.rdu2.redhat.com [10.11.55.25]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 351F714171BE for ; Wed, 14 Dec 2022 19:26:12 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 11C1629A9CA0 for ; Wed, 14 Dec 2022 19:26:12 +0000 (UTC) Received: from relay.virtuozzo.com (relay.virtuozzo.com [130.117.225.111]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-618-U-wYZb7aO_mtXFYjB2cucg-1; Wed, 14 Dec 2022 14:26:10 -0500 Received: from [192.168.16.26] (helo=dv.sw.ru) by relay.virtuozzo.com with esmtp (Exim 4.95) (envelope-from ) id 1p5XOK-00EGSm-Pz; Wed, 14 Dec 2022 20:26:09 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671045982; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=sw9NRhWswb8z5id/w7lCrwjbDdhapaAFwgQm48Ft/wE=; b=Tn/X3CeECD4TqcSJTCgZ7qV2XUI6JrA5LdgdxTc+7he93SV7XGPi6O4qcVlJeUlehBs5YW 4Jt9zypTNVGoQhn1cfAKC2OFbnECQYLlK5hXkUZe96Typ/QYXq9qbTTtZmhJ6hQ0XqxiYw hDUM/AB0DH/D+Ys/G3a6b90AmGdVk28= X-MC-Unique: zYhSX9SKNraxL39IydM53w-1 X-Original-To: libvir-list@listman.corp.redhat.com X-MC-Unique: U-wYZb7aO_mtXFYjB2cucg-1 From: Oleg Vasilev To: libvir-list@redhat.com Subject: [PATCH v2 4/5] logging: add log cleanup for obsolete domains Date: Thu, 15 Dec 2022 01:25:49 +0600 Message-Id: <20221214192550.149417-5-oleg.vasilev@virtuozzo.com> In-Reply-To: <20221214192550.149417-1-oleg.vasilev@virtuozzo.com> References: <20221214192550.149417-1-oleg.vasilev@virtuozzo.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Oleg Vasilev , Den Lunev Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1671045983339100003 Content-Type: text/plain; charset="utf-8"; x-default="true" Before, logs from deleted machines have been piling up, since there were no garbage collection mechanism. Now, virtlogd can be configured to periodically scan the log folder for orphan logs with no recent modificatio= ns and delete it. A single chain of recent and rotated logs is deleted in a single transactio= n. Signed-off-by: Oleg Vasilev --- po/POTFILES | 1 + src/logging/log_cleaner.c | 276 ++++++++++++++++++++++++++++++++++++++ src/logging/log_cleaner.h | 29 ++++ src/logging/log_handler.h | 2 + src/logging/meson.build | 1 + 5 files changed, 309 insertions(+) create mode 100644 src/logging/log_cleaner.c create mode 100644 src/logging/log_cleaner.h diff --git a/po/POTFILES b/po/POTFILES index 169e2a41dc..2fb6d18e27 100644 --- a/po/POTFILES +++ b/po/POTFILES @@ -123,6 +123,7 @@ src/locking/lock_driver_lockd.c src/locking/lock_driver_sanlock.c src/locking/lock_manager.c src/locking/sanlock_helper.c +src/logging/log_cleaner.c src/logging/log_daemon.c src/logging/log_daemon_dispatch.c src/logging/log_handler.c diff --git a/src/logging/log_cleaner.c b/src/logging/log_cleaner.c new file mode 100644 index 0000000000..03de9c17ee --- /dev/null +++ b/src/logging/log_cleaner.c @@ -0,0 +1,276 @@ +/* + * log_cleaner.c: cleans obsolete log files + * + * Copyright (C) 2022 Virtuozzo International GmbH + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; If not, see + * . + */ + +#include + +#include "log_cleaner.h" +#include "log_handler.h" + +#include "virerror.h" +#include "virobject.h" +#include "virfile.h" +#include "viralloc.h" +#include "virlog.h" +#include "virrotatingfile.h" +#include "virstring.h" + +#define VIR_FROM_THIS VIR_FROM_LOGGING + +/* Cleanup log root (/var/log/libvirt) and all subfolders (e.g. /var/log/l= ibvirt/qemu) */ +#define CLEANER_LOG_DEPTH 1 +#define CLEANER_LOG_TIMEOUT_MS (24 * 3600 * 1000) /* One day */ +#define MAX_TIME ((time_t) G_MAXINT64) + +static GRegex *log_regex; + +typedef struct _virLogCleanerChain virLogCleanerChain; +struct _virLogCleanerChain { + int rotated_max_index; + time_t last_modified; +}; + +typedef struct _virLogCleanerData virLogCleanerData; +struct _virLogCleanerData { + virLogHandler *handler; + time_t oldest_to_keep; + GHashTable *chains; +}; + +static const char* +virLogCleanerParseFilename(const char* path, + int* rotated_index) +{ + g_autoptr(GMatchInfo) matchInfo =3D NULL; + g_autofree const char* rotated_index_str =3D NULL; + g_autofree const char *clear_path =3D NULL; + const char* chain_prefix =3D NULL; + + clear_path =3D realpath(path, NULL); + if (!clear_path) { + virReportSystemError(errno, + _("Failed to resolve path '%s'"), + path); + return NULL; + } + + if (!g_regex_match(log_regex, path, 0, &matchInfo)) { + return NULL; + } + + chain_prefix =3D g_match_info_fetch(matchInfo, 1); + if (!rotated_index) + return chain_prefix; + + *rotated_index =3D 0; + rotated_index_str =3D g_match_info_fetch(matchInfo, 2); + + if (!rotated_index_str) + return chain_prefix; + + if (virStrToLong_i(rotated_index_str+1, NULL, 10, rotated_index) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Failed to parse rotated index from '%s'"), + rotated_index_str); + return NULL; + } + return chain_prefix; +} + +static void +virLogCleanerProcessFile(virLogCleanerData* data, + const char* path, + struct stat* sb) +{ + int rotated_index =3D 0; + g_autofree const char *chain_prefix =3D NULL; + virLogCleanerChain *chain; + + if (!S_ISREG(sb->st_mode)) { + return; + } + + chain_prefix =3D virLogCleanerParseFilename(path, &rotated_index); + + if (!chain_prefix) { + return; + } + + if (rotated_index > data->handler->config->max_backups) { + if (unlink(path) < 0) { + virReportSystemError(errno, _("Unable to delete %s"), path); + } + return; + } + + chain =3D g_hash_table_lookup(data->chains, chain_prefix); + + if (!chain) { + chain =3D g_new0(virLogCleanerChain, 1); + g_hash_table_insert(data->chains, (void *) chain_prefix, chain); + chain_prefix =3D NULL; /* Moved to hash table */ + } + + chain->last_modified =3D MAX(chain->last_modified, sb->st_mtime); + chain->rotated_max_index =3D MAX(chain->rotated_max_index, + rotated_index); +} + +static GHashTable* +virLogCleanerCreateTable(virLogHandler *handler) +{ + /* (const char*) chain_prefix -> (virLogCleanerChain*) chain */ + GHashTable *chains =3D g_hash_table_new_full(g_str_hash, g_str_equal, + g_free, g_free); + size_t i; + virLogHandlerLogFile *file; + const char *chain_prefix; + virLogCleanerChain* chain; + VIR_LOCK_GUARD lock =3D virObjectLockGuard(handler); + + for (i =3D 0; i < handler->nfiles; i++) { + file =3D handler->files[i]; + chain_prefix =3D virLogCleanerParseFilename(virRotatingFileWriterG= etPath(file->file), + NULL); + if (!chain_prefix) { + continue; + } + + chain =3D g_new0(virLogCleanerChain, 1); + chain->last_modified =3D (time_t) G_MAXINT64; + g_hash_table_insert(chains, (void *) chain_prefix, chain); + } + + return chains; +} + +static void +virLogCleanerProcessFolder(virLogCleanerData *data, + const char *path, + int depth_left) +{ + DIR *dir; + struct dirent *entry; + struct stat sb; + + if (virDirOpenIfExists(&dir, path) < 0) + return; + + while (virDirRead(dir, &entry, path) > 0) { + g_autofree char *newpath =3D NULL; + if (STREQ(entry->d_name, ".")) + continue; + if (STREQ(entry->d_name, "..")) + continue; + + newpath =3D g_strdup_printf("%s/%s", path, entry->d_name); + + if (stat(newpath, &sb) < 0) { + virReportSystemError(errno, _("Unable to stat %s"), newpath); + continue; + } + + if (S_ISDIR(sb.st_mode)) { + if (depth_left > 0) + virLogCleanerProcessFolder(data, newpath, depth_left - 1); + continue; + } + + virLogCleanerProcessFile(data, newpath, &sb); + } + + virDirClose(dir); +} + +static void +virLogCleanerChainCB(gpointer key, + gpointer value, + gpointer user_data) +{ + const char* chain_prefix =3D key; + virLogCleanerChain* chain =3D value; + virLogCleanerData* data =3D user_data; + size_t i; + + if (chain->last_modified > data->oldest_to_keep) { + return; + } + + for (i =3D 0; i <=3D chain->rotated_max_index; i++) { + g_autofree char* path =3D NULL; + if (i =3D=3D 0) { + path =3D g_strdup_printf("%s.log", chain_prefix); + } else { + path =3D g_strdup_printf("%s.log.%ld", chain_prefix, i); + } + if (unlink(path) < 0) { + virReportSystemError(errno, _("Unable to delete %s"), path); + } + } +} + +static void +virLogCleanerTimer(int timer G_GNUC_UNUSED, void* opaque) +{ + virLogHandler *handler =3D opaque; + virLogCleanerData data =3D { + .handler =3D handler, + .oldest_to_keep =3D time(NULL) - 3600 * 24 * handler->config->max_= age_days, + .chains =3D virLogCleanerCreateTable(handler), + }; + + /* First prepare the hashmap of chains to delete */ + virLogCleanerProcessFolder(&data, + handler->config->log_root, + CLEANER_LOG_DEPTH); + g_hash_table_foreach(data.chains, virLogCleanerChainCB, &data); + g_hash_table_destroy(data.chains); +} + +int +virLogCleanerInit(virLogHandler *handler) +{ + if (handler->config->max_age_days <=3D 0) { + return 0; + } + + log_regex =3D g_regex_new("^(.*)\\.log(\\.(\\d+))?$", 0, 0, NULL); + if (!log_regex) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Unable to compile regex")); + return -1; + } + + handler->cleanup_log_timer =3D virEventAddTimeout(CLEANER_LOG_TIMEOUT_= MS, + virLogCleanerTimer, + handler, NULL); + return handler->cleanup_log_timer; +} + +void +virLogCleanerShutdown(virLogHandler *handler) +{ + if (handler->cleanup_log_timer !=3D -1) { + virEventRemoveTimeout(handler->cleanup_log_timer); + handler->cleanup_log_timer =3D -1; + } + + g_regex_unref(log_regex); + log_regex =3D NULL; +} diff --git a/src/logging/log_cleaner.h b/src/logging/log_cleaner.h new file mode 100644 index 0000000000..1a9a94b68d --- /dev/null +++ b/src/logging/log_cleaner.h @@ -0,0 +1,29 @@ +/* + * log_cleaner.h: cleans obsolete log files + * + * Copyright (C) 2022 Virtuozzo International GmbH + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; If not, see + * . + */ + +#pragma once + +#include "log_handler.h" + +int +virLogCleanerInit(virLogHandler *handler); + +void +virLogCleanerShutdown(virLogHandler *handler); diff --git a/src/logging/log_handler.h b/src/logging/log_handler.h index d473a19fc6..97dad27eda 100644 --- a/src/logging/log_handler.h +++ b/src/logging/log_handler.h @@ -48,6 +48,8 @@ struct _virLogHandler { bool privileged; virLogDaemonConfig *config; =20 + int cleanup_log_timer; + virLogHandlerLogFile **files; size_t nfiles; =20 diff --git a/src/logging/meson.build b/src/logging/meson.build index 7066f16fad..fa6af50fa6 100644 --- a/src/logging/meson.build +++ b/src/logging/meson.build @@ -30,6 +30,7 @@ log_daemon_sources =3D files( 'log_daemon_config.c', 'log_daemon_dispatch.c', 'log_handler.c', + 'log_cleaner.c', ) =20 if conf.has('WITH_REMOTE') --=20 2.38.1