From nobody Fri Apr 19 11:09:44 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.39 as permitted sender) client-ip=209.132.183.39; envelope-from=libvir-list-bounces@redhat.com; helo=mx6-phx2.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.39 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) by mx.zohomail.com with SMTPS id 1486734941552915.8737449252932; Fri, 10 Feb 2017 05:55:41 -0800 (PST) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v1ADq6QL063023; Fri, 10 Feb 2017 08:52:07 -0500 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v1ADq5AP030234 for ; Fri, 10 Feb 2017 08:52:05 -0500 Received: from mx1.redhat.com (ext-mx10.extmail.prod.ext.phx2.redhat.com [10.5.110.39]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v1ADq4Us030991 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Fri, 10 Feb 2017 08:52:05 -0500 Received: from relay.sw.ru (mailhub.sw.ru [195.214.232.25]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0EABC61D20 for ; Fri, 10 Feb 2017 13:52:02 +0000 (UTC) Received: from localhost.localdomain (msk-vpn.virtuozzo.com [195.214.232.6]) by relay.sw.ru (8.13.4/8.13.4) with ESMTP id v1ADV86n004137; Fri, 10 Feb 2017 16:31:09 +0300 (MSK) From: Derbyshev Dmitriy To: libvir-list@redhat.com Date: Fri, 10 Feb 2017 16:29:27 +0300 Message-Id: <1486733367-9316-1-git-send-email-dderbyshev@virtuozzo.com> X-Greylist: Delayed for 00:20:47 by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Fri, 10 Feb 2017 13:52:03 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Fri, 10 Feb 2017 13:52:03 +0000 (UTC) for IP:'195.214.232.25' DOMAIN:'mailhub.sw.ru' HELO:'relay.sw.ru' FROM:'dderbyshev@virtuozzo.com' RCPT:'' X-RedHat-Spam-Score: 0.799 (BAYES_50, SPF_PASS) 195.214.232.25 mailhub.sw.ru 195.214.232.25 mailhub.sw.ru X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Scanned-By: MIMEDefang 2.78 on 10.5.110.39 X-loop: libvir-list@redhat.com Cc: redkoi@virtuozzo.com, Derbyshev Dmitry , den@virtuozzo.com, kneumoin@virtuozzo.com, mnestratov@virtuozzo.com, nshirokovskiy@virtuozzo.com Subject: [libvirt] [RFC] Substitute poll by epoll in virEventPollRunOnce X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" From: Derbyshev Dmitry This makes it possible to avoid allocations in virEventPollMakePollFDs, as well as generally reduces time spent in kernel if handles remain the same, which should generally be true. __________________________________________ Questions: 1) What to do with probe in virEventPollRunOnce? 2) Are ifdef an acceptable solution to epoll avaliability issues? 3) Is using MAX_POLL_EVENTS_AT_ONCE constant a valid solution? 4) some fds are sometimes added twice, but only once with events!=3D0? This= complicates virEventPoll*Handle fuctions. At the very least, it is called = twice in _dbus_watch_list_set_functions. Signed-off-by: Derbyshev Dmitry --- configure.ac | 26 +++++ src/util/vireventpoll.c | 205 +++++++++++++++++++++++++++++++++++= +--- src/util/vireventpoll.h | 5 + tests/commanddata/test3epoll.log | 15 +++ tests/commandtest.c | 4 + 5 files changed, 242 insertions(+), 13 deletions(-) create mode 100644 tests/commanddata/test3epoll.log diff --git a/configure.ac b/configure.ac index dfc536f..c985ccc 100644 --- a/configure.ac +++ b/configure.ac @@ -2695,6 +2695,32 @@ AC_DEFINE_UNQUOTED([isbase64],[libvirt_gl_isbase64],= [Hack to avoid symbol clash] AC_DEFINE_UNQUOTED([base64_encode],[libvirt_gl_base64_encode],[Hack to avo= id symbol clash]) AC_DEFINE_UNQUOTED([base64_encode_alloc],[libvirt_gl_base64_encode_alloc],= [Hack to avoid symbol clash]) =20 +AC_ARG_ENABLE([epoll], + [AS_HELP_STRING([--enable-epoll],[use epoll(4) on Linux])], + [enable_epoll=3D$enableval], [enable_epoll=3Dauto]) +if test x$enable_epoll =3D xno; then + have_linux_epoll=3Dno +else + AC_MSG_CHECKING([for Linux epoll(4)]) + AC_LINK_IFELSE([AC_LANG_PROGRAM( + [ + #ifndef __linux__ + #error This is not Linux + #endif + #include + ], + [epoll_create1 (EPOLL_CLOEXEC);])], + [have_linux_epoll=3Dyes], + [have_linux_epoll=3Dno]) + AC_MSG_RESULT([$have_linux_epoll]) +fi +if test x$enable_epoll,$have_linux_epoll =3D xyes,no; then + AC_MSG_ERROR([epoll support explicitly enabled but not available]) +fi +if test "x$have_linux_epoll" =3D xyes; then + AC_DEFINE_UNQUOTED([HAVE_LINUX_EPOLL], 1, [whether epoll is supported]) +fi + AC_CONFIG_FILES([run], [chmod +x,-w run]) AC_CONFIG_FILES([\ diff --git a/src/util/vireventpoll.c b/src/util/vireventpoll.c index 81ecab4..1541706 100644 --- a/src/util/vireventpoll.c +++ b/src/util/vireventpoll.c @@ -25,7 +25,11 @@ =20 #include #include -#include +#ifdef HAVE_LINUX_EPOLL +# include +#else +# include +#endif #include #include #include @@ -87,6 +91,9 @@ struct virEventPollLoop { size_t timeoutsCount; size_t timeoutsAlloc; struct virEventPollTimeout *timeouts; +#ifdef HAVE_LINUX_EPOLL + int epollfd; +#endif }; =20 /* Only have one event loop */ @@ -109,6 +116,11 @@ int virEventPollAddHandle(int fd, int events, virFreeCallback ff) { int watch; +#ifdef HAVE_LINUX_EPOLL + size_t i; + struct epoll_event ev; +#endif + int native =3D virEventPollToNativeEvents(events); virMutexLock(&eventLoop.lock); if (eventLoop.handlesCount =3D=3D eventLoop.handlesAlloc) { EVENT_DEBUG("Used %zu handle slots, adding at least %d more", @@ -124,8 +136,7 @@ int virEventPollAddHandle(int fd, int events, =20 eventLoop.handles[eventLoop.handlesCount].watch =3D watch; eventLoop.handles[eventLoop.handlesCount].fd =3D fd; - eventLoop.handles[eventLoop.handlesCount].events =3D - virEventPollToNativeEvents(events= ); + eventLoop.handles[eventLoop.handlesCount].events =3D native; eventLoop.handles[eventLoop.handlesCount].cb =3D cb; eventLoop.handles[eventLoop.handlesCount].ff =3D ff; eventLoop.handles[eventLoop.handlesCount].opaque =3D opaque; @@ -133,7 +144,30 @@ int virEventPollAddHandle(int fd, int events, =20 eventLoop.handlesCount++; =20 +#ifdef HAVE_LINUX_EPOLL + ev.events =3D native; + ev.data.fd =3D fd; + if (epoll_ctl(eventLoop.epollfd, EPOLL_CTL_ADD, fd, &ev) < 0) { + if (errno =3D=3D EEXIST) { + for (i =3D 0; i < eventLoop.handlesCount; i++) { + if (eventLoop.handles[i].fd =3D=3D fd && + !eventLoop.handles[i].deleted) { + ev.events |=3D eventLoop.handles[i].events; + } + } + if (epoll_ctl(eventLoop.epollfd, EPOLL_CTL_MOD, fd, &ev) < 0) { + virMutexUnlock(&eventLoop.lock); + return -1; + } + } + else { + virMutexUnlock(&eventLoop.lock); + return -1; + } + } +#else virEventPollInterruptLocked(); +#endif =20 PROBE(EVENT_POLL_ADD_HANDLE, "watch=3D%d fd=3D%d events=3D%d cb=3D%p opaque=3D%p ff=3D%p", @@ -145,8 +179,14 @@ int virEventPollAddHandle(int fd, int events, =20 void virEventPollUpdateHandle(int watch, int events) { +#ifdef HAVE_LINUX_EPOLL + struct epoll_event ev; + size_t i, j; +#else size_t i; +#endif bool found =3D false; + int native =3D virEventPollToNativeEvents(events); PROBE(EVENT_POLL_UPDATE_HANDLE, "watch=3D%d events=3D%d", watch, events); @@ -159,13 +199,34 @@ void virEventPollUpdateHandle(int watch, int events) virMutexLock(&eventLoop.lock); for (i =3D 0; i < eventLoop.handlesCount; i++) { if (eventLoop.handles[i].watch =3D=3D watch) { - eventLoop.handles[i].events =3D - virEventPollToNativeEvents(events); + eventLoop.handles[i].events =3D native; +#ifndef HAVE_LINUX_EPOLL virEventPollInterruptLocked(); +#endif found =3D true; break; } } + +#ifdef HAVE_LINUX_EPOLL + ev.events =3D native; + for (j =3D 0; j < eventLoop.handlesCount; j++) { + if (eventLoop.handles[i].fd =3D=3D eventLoop.handles[i].fd && + !eventLoop.handles[i].deleted && + i !=3D j) { + ev.events |=3D eventLoop.handles[i].events; + } + } + ev.data.fd =3D eventLoop.handles[i].fd; + + if (found && epoll_ctl(eventLoop.epollfd, EPOLL_CTL_MOD, + eventLoop.handles[i].fd, &ev) < 0) { + VIR_WARN("Update for existing handle watch %d failed", watch); + virMutexUnlock(&eventLoop.lock); + return; + } +#endif + virMutexUnlock(&eventLoop.lock); =20 if (!found) @@ -180,7 +241,12 @@ void virEventPollUpdateHandle(int watch, int events) */ int virEventPollRemoveHandle(int watch) { +#ifdef HAVE_LINUX_EPOLL + struct epoll_event ev; + size_t i, j; +#else size_t i; +#endif PROBE(EVENT_POLL_REMOVE_HANDLE, "watch=3D%d", watch); @@ -196,15 +262,47 @@ int virEventPollRemoveHandle(int watch) continue; =20 if (eventLoop.handles[i].watch =3D=3D watch) { - EVENT_DEBUG("mark delete %zu %d", i, eventLoop.handles[i].fd); - eventLoop.handles[i].deleted =3D 1; - virEventPollInterruptLocked(); + break; + } + } + + if (i =3D=3D eventLoop.handlesCount) { + virMutexUnlock(&eventLoop.lock); + return -1; + } + +#ifdef HAVE_LINUX_EPOLL + ev.events =3D 0; + for (j =3D 0; j < eventLoop.handlesCount; j++) { + if (eventLoop.handles[j].fd =3D=3D eventLoop.handles[i].fd && + !eventLoop.handles[j].deleted && + i !=3D j) { + ev.events |=3D eventLoop.handles[i].events; + } + } + + ev.data.fd =3D eventLoop.handles[i].fd; + if (ev.events) { + if (epoll_ctl(eventLoop.epollfd, EPOLL_CTL_MOD, eventLoop.handles[= i].fd, + &ev) < 0) { virMutexUnlock(&eventLoop.lock); - return 0; + return -1; } } + else { + if (epoll_ctl(eventLoop.epollfd, EPOLL_CTL_DEL, eventLoop.handles[= i].fd, + &ev) < 0) { + virMutexUnlock(&eventLoop.lock); + return -1; + } + } +#endif + + EVENT_DEBUG("mark delete %zu %d", i, eventLoop.handles[i].fd); + eventLoop.handles[i].deleted =3D 1; + virEventPollInterruptLocked(); virMutexUnlock(&eventLoop.lock); - return -1; + return 0; } =20 =20 @@ -373,6 +471,7 @@ static int virEventPollCalculateTimeout(int *timeout) return 0; } =20 +#ifndef HAVE_LINUX_EPOLL /* * Allocate a pollfd array containing data for all registered * file handles. The caller must free the returned data struct @@ -409,6 +508,7 @@ static struct pollfd *virEventPollMakePollFDs(int *nfds= ) { =20 return fds; } +#endif =20 =20 /* @@ -472,7 +572,11 @@ static int virEventPollDispatchTimeouts(void) * * Returns 0 upon success, -1 if an error occurred */ +#ifdef HAVE_LINUX_EPOLL +static int virEventPollDispatchHandles(int nfds, struct epoll_event *event= s) +#else static int virEventPollDispatchHandles(int nfds, struct pollfd *fds) +#endif { size_t i, n; VIR_DEBUG("Dispatch %d", nfds); @@ -482,7 +586,11 @@ static int virEventPollDispatchHandles(int nfds, struc= t pollfd *fds) * in the fds array we've got */ for (i =3D 0, n =3D 0; n < nfds && i < eventLoop.handlesCount; n++) { while (i < eventLoop.handlesCount && +#ifdef HAVE_LINUX_EPOLL + (eventLoop.handles[i].fd !=3D events[n].data.fd || +#else (eventLoop.handles[i].fd !=3D fds[n].fd || +#endif eventLoop.handles[i].events =3D=3D 0)) { i++; } @@ -496,16 +604,25 @@ static int virEventPollDispatchHandles(int nfds, stru= ct pollfd *fds) continue; } =20 +#ifdef HAVE_LINUX_EPOLL + if (events[n].events) { + int hEvents =3D virEventPollFromNativeEvents(events[n].events); +#else if (fds[n].revents) { + int hEvents =3D virEventPollFromNativeEvents(fds[n].revents); +#endif virEventHandleCallback cb =3D eventLoop.handles[i].cb; int watch =3D eventLoop.handles[i].watch; void *opaque =3D eventLoop.handles[i].opaque; - int hEvents =3D virEventPollFromNativeEvents(fds[n].revents); PROBE(EVENT_POLL_DISPATCH_HANDLE, "watch=3D%d events=3D%d", watch, hEvents); virMutexUnlock(&eventLoop.lock); +#ifdef HAVE_LINUX_EPOLL + (cb)(watch, events[n].data.fd, hEvents, opaque); +#else (cb)(watch, fds[n].fd, hEvents, opaque); +#endif virMutexLock(&eventLoop.lock); } } @@ -618,7 +735,11 @@ static void virEventPollCleanupHandles(void) */ int virEventPollRunOnce(void) { +#ifdef HAVE_LINUX_EPOLL + struct epoll_event events[MAX_POLL_EVENTS_AT_ONCE]; +#else struct pollfd *fds =3D NULL; +#endif int ret, timeout, nfds; =20 virMutexLock(&eventLoop.lock); @@ -628,17 +749,29 @@ int virEventPollRunOnce(void) virEventPollCleanupTimeouts(); virEventPollCleanupHandles(); =20 - if (!(fds =3D virEventPollMakePollFDs(&nfds)) || - virEventPollCalculateTimeout(&timeout) < 0) +#ifndef HAVE_LINUX_EPOLL + if (!(fds =3D virEventPollMakePollFDs(&nfds))) + goto error; +#endif + if (virEventPollCalculateTimeout(&timeout) < 0) goto error; =20 virMutexUnlock(&eventLoop.lock); =20 retry: +#ifdef HAVE_LINUX_EPOLL +//FIXME: PROBE handles + PROBE(EVENT_POLL_RUN, + "nhandles=3D%d timeout=3D%d", + 42, timeout); + nfds =3D ret =3D epoll_wait(eventLoop.epollfd, events, + MAX_POLL_EVENTS_AT_ONCE, timeout); +#else PROBE(EVENT_POLL_RUN, "nhandles=3D%d timeout=3D%d", nfds, timeout); ret =3D poll(fds, nfds, timeout); +#endif if (ret < 0) { EVENT_DEBUG("Poll got error event %d", errno); if (errno =3D=3D EINTR || errno =3D=3D EAGAIN) @@ -653,22 +786,32 @@ int virEventPollRunOnce(void) if (virEventPollDispatchTimeouts() < 0) goto error; =20 +#ifdef HAVE_LINUX_EPOLL + if (ret > 0 && + virEventPollDispatchHandles(nfds, events) < 0) + goto error; +#else if (ret > 0 && virEventPollDispatchHandles(nfds, fds) < 0) goto error; +#endif =20 virEventPollCleanupTimeouts(); virEventPollCleanupHandles(); =20 eventLoop.running =3D 0; virMutexUnlock(&eventLoop.lock); +#ifndef HAVE_LINUX_EPOLL VIR_FREE(fds); +#endif return 0; =20 error: virMutexUnlock(&eventLoop.lock); error_unlocked: +#ifndef HAVE_LINUX_EPOLL VIR_FREE(fds); +#endif return -1; } =20 @@ -698,6 +841,17 @@ int virEventPollInit(void) return -1; } =20 +#ifdef HAVE_LINUX_EPOLL + eventLoop.epollfd =3D epoll_create1(0); + if (eventLoop.epollfd < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Unable to initialize epoll")); + VIR_FORCE_CLOSE(eventLoop.wakeupfd[0]); + VIR_FORCE_CLOSE(eventLoop.wakeupfd[1]); + return -1; + } +#endif + if (virEventPollAddHandle(eventLoop.wakeupfd[0], VIR_EVENT_HANDLE_READABLE, virEventPollHandleWakeup, NULL, NULL) < 0) { @@ -706,6 +860,9 @@ int virEventPollInit(void) eventLoop.wakeupfd[0]); VIR_FORCE_CLOSE(eventLoop.wakeupfd[0]); VIR_FORCE_CLOSE(eventLoop.wakeupfd[1]); +#ifdef HAVE_LINUX_EPOLL + VIR_FORCE_CLOSE(eventLoop.epollfd); +#endif return -1; } =20 @@ -742,6 +899,16 @@ int virEventPollToNativeEvents(int events) { int ret =3D 0; +#ifdef HAVE_LINUX_EPOLL + if (events & VIR_EVENT_HANDLE_READABLE) + ret |=3D EPOLLIN; + if (events & VIR_EVENT_HANDLE_WRITABLE) + ret |=3D EPOLLOUT; + if (events & VIR_EVENT_HANDLE_ERROR) + ret |=3D EPOLLERR; + if (events & VIR_EVENT_HANDLE_HANGUP) + ret |=3D EPOLLHUP; +#else if (events & VIR_EVENT_HANDLE_READABLE) ret |=3D POLLIN; if (events & VIR_EVENT_HANDLE_WRITABLE) @@ -750,6 +917,7 @@ virEventPollToNativeEvents(int events) ret |=3D POLLERR; if (events & VIR_EVENT_HANDLE_HANGUP) ret |=3D POLLHUP; +#endif return ret; } =20 @@ -757,6 +925,16 @@ int virEventPollFromNativeEvents(int events) { int ret =3D 0; +#ifdef HAVE_LINUX_EPOLL + if (events & EPOLLIN) + ret |=3D VIR_EVENT_HANDLE_READABLE; + if (events & EPOLLOUT) + ret |=3D VIR_EVENT_HANDLE_WRITABLE; + if (events & EPOLLERR) + ret |=3D VIR_EVENT_HANDLE_ERROR; + if (events & EPOLLHUP) + ret |=3D VIR_EVENT_HANDLE_HANGUP; +#else if (events & POLLIN) ret |=3D VIR_EVENT_HANDLE_READABLE; if (events & POLLOUT) @@ -767,5 +945,6 @@ virEventPollFromNativeEvents(int events) ret |=3D VIR_EVENT_HANDLE_ERROR; if (events & POLLHUP) ret |=3D VIR_EVENT_HANDLE_HANGUP; +#endif return ret; } diff --git a/src/util/vireventpoll.h b/src/util/vireventpoll.h index 8844c9c..172009a 100644 --- a/src/util/vireventpoll.h +++ b/src/util/vireventpoll.h @@ -26,6 +26,11 @@ =20 # include "internal.h" =20 +# ifdef HAVE_LINUX_EPOLL +/* Maximum number of events that are returned by epoll in virEventPollRunO= nce */ +# define MAX_POLL_EVENTS_AT_ONCE 10 +# endif + /** * virEventPollAddHandle: register a callback for monitoring file handle e= vents * diff --git a/tests/commanddata/test3epoll.log b/tests/commanddata/test3epol= l.log new file mode 100644 index 0000000..e4619b3 --- /dev/null +++ b/tests/commanddata/test3epoll.log @@ -0,0 +1,15 @@ +ENV:DISPLAY=3D:0.0 +ENV:HOME=3D/home/test +ENV:HOSTNAME=3Dtest +ENV:LANG=3DC +ENV:LOGNAME=3DtestTMPDIR=3D/tmp +ENV:PATH=3D/usr/bin:/bin +ENV:USER=3Dtest +FD:0 +FD:1 +FD:2 +FD:6 +FD:8 +DAEMON:no +CWD:/tmp +UMASK:0022 diff --git a/tests/commandtest.c b/tests/commandtest.c index 7bf5447..6b23659 100644 --- a/tests/commandtest.c +++ b/tests/commandtest.c @@ -227,7 +227,11 @@ static int test3(const void *unused ATTRIBUTE_UNUSED) goto cleanup; } =20 +#ifdef HAVE_LINUX_EPOLL + ret =3D checkoutput("test3epoll", NULL); +#else ret =3D checkoutput("test3", NULL); +#endif =20 cleanup: virCommandFree(cmd); --=20 1.9.5.msysgit.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list