[Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

dhbarboza82@gmail.com posted 1 patch 4 years, 2 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/kimchi-wok tags/patchew/20170224132202.31465-1-dhbarboza82@gmail.com
Test Unit Tests passed
src/wok/model/notifications.py |  18 +++++-
src/wok/pushserver.py          | 132 +++++++++++++++++++++++++++++++++++++++++
src/wok/reqlogger.py           |   7 ++-
src/wok/server.py              |   8 ++-
src/wok/websocket.py           | 123 ++++++++++++++++++++++++++++++++++++++
ui/js/src/wok.main.js          |  38 ++++++++++++
ui/js/wok.user-log.js          |   6 ++
7 files changed, 327 insertions(+), 5 deletions(-)
create mode 100644 src/wok/pushserver.py
create mode 100644 src/wok/websocket.py

[Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by dhbarboza82@gmail.com 4 years, 2 months ago
From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>

This patch makes backend and UI changes to implement the asynchronous
UI notification in WoK.

- Backend:

A push server was implemented from scratch to manage the opened websocket
connections. The push server connects to the /run/woknotifications
UNIX socket and broadcasts all messages to all connections.

The websocket module is the same module that exists in the Kimchi
plug-in. The idea is to remove the module from Kimchi and make it
use the module from WoK. ws_proxy initialization was also added
in src/wok/server.py.

- Frontend:

In ui/js/wok.main.js two new functions were added to help the
usage of asynchronous notifications in the frontend. The idea:
a single websocket is opened per session. This opened websocket
will broadcast all incoming messages to all listeners registered.
Listeners can be added by the new wok.addNotificationListener()
method. This method will clean up any registered listener by
itself when the user changes tabs/URL.

The single websocket sends heartbeats to the backend side each
30 seconds. No reply from the backend is issued or expected. This
heartbeat is just a way to ensure that the browser does not
close the connection due to inactivity. This behavior varies from
browser to browser but this 30 second heartbeat is more than enough
to ensure that the websocket is kept alive.

- Working example in User Log:

A simple usage is provided in this patch. A change was made in
src/wok/reqlogger.py to send an asynchronous notification each
time a new log entry is created. In ui/js/wok.user-log.js a
websocket listener is added using wok.addNotificationListener()
and, for each message that indicates a new user log entry, a
refresh in the listing is issued.

Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
---
 src/wok/model/notifications.py |  18 +++++-
 src/wok/pushserver.py          | 132 +++++++++++++++++++++++++++++++++++++++++
 src/wok/reqlogger.py           |   7 ++-
 src/wok/server.py              |   8 ++-
 src/wok/websocket.py           | 123 ++++++++++++++++++++++++++++++++++++++
 ui/js/src/wok.main.js          |  38 ++++++++++++
 ui/js/wok.user-log.js          |   6 ++
 7 files changed, 327 insertions(+), 5 deletions(-)
 create mode 100644 src/wok/pushserver.py
 create mode 100644 src/wok/websocket.py

diff --git a/src/wok/model/notifications.py b/src/wok/model/notifications.py
index bdb7c78..597eac5 100644
--- a/src/wok/model/notifications.py
+++ b/src/wok/model/notifications.py
@@ -1,7 +1,7 @@
 #
 # Project Wok
 #
-# Copyright IBM Corp, 2016
+# Copyright IBM Corp, 2016-2017
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -19,12 +19,22 @@
 
 from datetime import datetime
 
+from wok import config
 from wok.exception import NotFoundError, OperationFailed
 from wok.message import WokMessage
+from wok.pushserver import PushServer
 from wok.utils import wok_log
 
 
 notificationsStore = {}
+push_server = None
+
+
+def send_websocket_notification(message):
+    global push_server
+
+    if push_server:
+        push_server.send_notification(message)
 
 
 def add_notification(code, args=None, plugin_name=None):
@@ -57,7 +67,11 @@ def del_notification(code):
 
 class NotificationsModel(object):
     def __init__(self, **kargs):
-        pass
+        global push_server
+
+        test_mode = config.config.get('server', 'test').lower() == 'true'
+        if not test_mode:
+            push_server = PushServer()
 
     def get_list(self):
         global notificationsStore
diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
new file mode 100644
index 0000000..8993f00
--- /dev/null
+++ b/src/wok/pushserver.py
@@ -0,0 +1,132 @@
+#
+# Project Wok
+#
+# Copyright IBM Corp, 2017
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
+#
+
+import cherrypy
+import os
+import select
+import socket
+import threading
+
+import websocket
+from utils import wok_log
+
+
+BASE_DIRECTORY = '/run'
+TOKEN_NAME = 'woknotifications'
+
+
+class PushServer(object):
+
+    def set_socket_file(self):
+        if not os.path.isdir(BASE_DIRECTORY):
+            try:
+                os.mkdir(BASE_DIRECTORY)
+            except OSError:
+                raise RuntimeError('PushServer base UNIX socket dir %s '
+                                   'not found.' % BASE_DIRECTORY)
+
+        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
+
+        if os.path.exists(self.server_addr):
+            try:
+                os.remove(self.server_addr)
+            except:
+                raise RuntimeError('There is an existing connection in %s' %
+                                   self.server_addr)
+
+    def __init__(self):
+        self.set_socket_file()
+
+        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
+
+        self.connections = []
+
+        self.server_running = True
+        self.server_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        self.server_socket.setsockopt(socket.SOL_SOCKET,
+                                      socket.SO_REUSEADDR, 1)
+        self.server_socket.bind(self.server_addr)
+        self.server_socket.listen(10)
+        wok_log.info('Push server created on address %s' % self.server_addr)
+
+        self.connections.append(self.server_socket)
+        cherrypy.engine.subscribe('stop', self.close_server, 1)
+
+        server_loop = threading.Thread(target=self.listen)
+        server_loop.start()
+
+    def listen(self):
+        try:
+            while self.server_running:
+                read_ready, _, _ = select.select(self.connections,
+                                                 [], [], 1)
+                for sock in read_ready:
+                    if not self.server_running:
+                        break
+
+                    if sock == self.server_socket:
+
+                        new_socket, addr = self.server_socket.accept()
+                        self.connections.append(new_socket)
+                    else:
+                        try:
+                            data = sock.recv(4096)
+                        except:
+                            try:
+                                self.connections.remove(sock)
+                            except ValueError:
+                                pass
+
+                            continue
+                        if data and data == 'CLOSE':
+                            sock.send('ACK')
+                            try:
+                                self.connections.remove(sock)
+                            except ValueError:
+                                pass
+                            sock.close()
+
+        except Exception as e:
+            raise RuntimeError('Exception ocurred in listen() of pushserver '
+                               'module: %s' % e.message)
+
+    def send_notification(self, message):
+        for sock in self.connections:
+            if sock != self.server_socket:
+                try:
+                    sock.send(message)
+                except IOError as e:
+                    if 'Broken pipe' in str(e):
+                        sock.close()
+                        try:
+                            self.connections.remove(sock)
+                        except ValueError:
+                            pass
+
+    def close_server(self):
+        try:
+            self.server_running = False
+            self.server_socket.shutdown(socket.SHUT_RDWR)
+            self.server_socket.close()
+            os.remove(self.server_addr)
+        except:
+            pass
+        finally:
+            cherrypy.engine.unsubscribe('stop', self.close_server)
diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
index 92e155d..1b774e2 100644
--- a/src/wok/reqlogger.py
+++ b/src/wok/reqlogger.py
@@ -1,7 +1,7 @@
 #
 # Project Wok
 #
-# Copyright IBM Corp, 2016
+# Copyright IBM Corp, 2016-2017
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -34,6 +34,7 @@ from wok.auth import USER_NAME
 from wok.config import get_log_download_path, paths
 from wok.exception import InvalidParameter, OperationFailed
 from wok.message import WokMessage
+from wok.model.notifications import send_websocket_notification
 from wok.stringutils import ascii_dict
 from wok.utils import remove_old_files
 
@@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
 # AsyncTask handling
 ASYNCTASK_REQUEST_METHOD = 'TASK'
 
+NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
+
 
 def log_request(code, params, exception, method, status, app=None, user=None,
                 ip=None):
@@ -114,6 +117,8 @@ def log_request(code, params, exception, method, status, app=None, user=None,
         ip=ip
     ).log()
 
+    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
+
     return log_id
 
 
diff --git a/src/wok/server.py b/src/wok/server.py
index fc2e167..2d823c9 100644
--- a/src/wok/server.py
+++ b/src/wok/server.py
@@ -25,8 +25,7 @@ import logging
 import logging.handlers
 import os
 
-from wok import auth
-from wok import config
+from wok import auth, config, websocket
 from wok.config import config as configParser
 from wok.config import WokConfig
 from wok.control import sub_nodes
@@ -159,6 +158,11 @@ class Server(object):
         cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
                             options.server_root, self.configObj)
 
+        test_mode = config.config.get('server', 'test').lower() == 'true'
+        if not test_mode:
+            ws_proxy = websocket.new_ws_proxy()
+            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
+
         self._load_plugins()
         cherrypy.lib.sessions.init()
 
diff --git a/src/wok/websocket.py b/src/wok/websocket.py
new file mode 100644
index 0000000..5d7fb91
--- /dev/null
+++ b/src/wok/websocket.py
@@ -0,0 +1,123 @@
+#!/usr/bin/env python2
+#
+# Project Wok
+#
+# Copyright IBM Corp, 2017
+#
+# Code derived from Project Kimchi
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
+
+import base64
+import errno
+import os
+
+from multiprocessing import Process
+from websockify import WebSocketProxy
+
+from config import config, PluginPaths
+
+
+try:
+    from websockify.token_plugins import TokenFile
+    tokenFile = True
+except ImportError:
+    tokenFile = False
+
+try:
+    from websockify import ProxyRequestHandler as request_proxy
+except:
+    from websockify import WebSocketProxy as request_proxy
+
+
+WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 'ws-tokens')
+
+
+class CustomHandler(request_proxy):
+
+    def get_target(self, target_plugin, path):
+        if issubclass(CustomHandler, object):
+            target = super(CustomHandler, self).get_target(target_plugin,
+                                                           path)
+        else:
+            target = request_proxy.get_target(self, target_plugin, path)
+
+        if target[0] == 'unix_socket':
+            try:
+                self.server.unix_target = target[1]
+            except:
+                self.unix_target = target[1]
+        else:
+            try:
+                self.server.unix_target = None
+            except:
+                self.unix_target = None
+        return target
+
+
+def new_ws_proxy():
+    try:
+        os.makedirs(WS_TOKENS_DIR, mode=0755)
+    except OSError as e:
+        if e.errno == errno.EEXIST:
+            pass
+
+    params = {'listen_host': '127.0.0.1',
+              'listen_port': config.get('server', 'websockets_port'),
+              'ssl_only': False}
+
+    # old websockify: do not use TokenFile
+    if not tokenFile:
+        params['target_cfg'] = WS_TOKENS_DIR
+
+    # websockify 0.7 and higher: use TokenFile
+    else:
+        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
+
+    def start_proxy():
+        try:
+            server = WebSocketProxy(RequestHandlerClass=CustomHandler,
+                                    **params)
+        except TypeError:
+            server = CustomHandler(**params)
+
+        server.start_server()
+
+    proc = Process(target=start_proxy)
+    proc.start()
+    return proc
+
+
+def add_proxy_token(name, port, is_unix_socket=False):
+    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
+        """
+        From python documentation base64.urlsafe_b64encode(s)
+        substitutes - instead of + and _ instead of / in the
+        standard Base64 alphabet, BUT the result can still
+        contain = which is not safe in a URL query component.
+        So remove it when needed as base64 can work well without it.
+        """
+        name = base64.urlsafe_b64encode(name).rstrip('=')
+        if is_unix_socket:
+            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), port))
+        else:
+            f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
+
+
+def remove_proxy_token(name):
+    try:
+        os.unlink(os.path.join(WS_TOKENS_DIR, name))
+    except OSError:
+        pass
diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
index 20c017e..f5031ce 100644
--- a/ui/js/src/wok.main.js
+++ b/ui/js/src/wok.main.js
@@ -29,6 +29,41 @@ wok.getConfig(function(result) {
     wok.config = {};
 });
 
+
+wok.notificationListeners = {};
+wok.addNotificationListener = function(name, func) {
+    wok.notificationListeners[name] = func;
+    $(window).one("hashchange", function() {
+        delete wok.notificationListeners[name];
+    });
+};
+
+wok.notificationsWebSocket = undefined;
+wok.startNotificationWebSocket = function () {
+    var addr = window.location.hostname + ':' + window.location.port;
+    var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
+    var url = 'wss://' + addr + '/websockify?token=' + token;
+    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
+
+    wok.notificationsWebSocket.onmessage = function(event) {
+        var message = window.atob(event.data);
+        for (name in wok.notificationListeners) {
+            func = wok.notificationListeners[name];
+            func(message);
+        }
+    };
+
+    sessionStorage.setItem('wokNotificationWebSocket', 'true');
+    var heartbeat = setInterval(function() {
+        wok.notificationsWebSocket.send(window.btoa('heartbeat'));
+    }, 30000);
+
+    wok.notificationsWebSocket.onclose = function() {
+        clearInterval(heartbeat);
+    };
+};
+
+
 wok.main = function() {
     wok.isLoggingOut = false;
     wok.popable();
@@ -395,6 +430,9 @@ wok.main = function() {
 
         // Set handler for help button
         $('#btn-help').on('click', wok.openHelp);
+
+        // start WebSocket
+        wok.startNotificationWebSocket();
     };
 
     var initUI = function() {
diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
index 0e8fb09..083b6c3 100644
--- a/ui/js/wok.user-log.js
+++ b/ui/js/wok.user-log.js
@@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
         $("#user-log-grid").bootgrid("search");
         wok.initUserLogConfigGridData();
     });
+
+    wok.addNotificationListener('userlog', function(message) {
+        if (message === 'new_log_entry') {
+            $("#refresh-button").click();
+        }
+    });
 };
 
 wok.initUserLogWindow = function() {
-- 
2.9.3

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/24/2017 10:22 AM, dhbarboza82@gmail.com wrote:
> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>
> This patch makes backend and UI changes to implement the asynchronous
> UI notification in WoK.
>
> - Backend:
>
> A push server was implemented from scratch to manage the opened websocket
> connections. The push server connects to the /run/woknotifications
> UNIX socket and broadcasts all messages to all connections.
>
> The websocket module is the same module that exists in the Kimchi
> plug-in. The idea is to remove the module from Kimchi and make it
> use the module from WoK. ws_proxy initialization was also added
> in src/wok/server.py.
>
> - Frontend:
>
> In ui/js/wok.main.js two new functions were added to help the
> usage of asynchronous notifications in the frontend. The idea:
> a single websocket is opened per session. This opened websocket
> will broadcast all incoming messages to all listeners registered.
> Listeners can be added by the new wok.addNotificationListener()
> method. This method will clean up any registered listener by
> itself when the user changes tabs/URL.
>
> The single websocket sends heartbeats to the backend side each
> 30 seconds. No reply from the backend is issued or expected. This
> heartbeat is just a way to ensure that the browser does not
> close the connection due to inactivity. This behavior varies from
> browser to browser but this 30 second heartbeat is more than enough
> to ensure that the websocket is kept alive.
>
> - Working example in User Log:
>
> A simple usage is provided in this patch. A change was made in
> src/wok/reqlogger.py to send an asynchronous notification each
> time a new log entry is created. In ui/js/wok.user-log.js a
> websocket listener is added using wok.addNotificationListener()
> and, for each message that indicates a new user log entry, a
> refresh in the listing is issued.
>
> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
> ---
>   src/wok/model/notifications.py |  18 +++++-
>   src/wok/pushserver.py          | 132 +++++++++++++++++++++++++++++++++++++++++
>   src/wok/reqlogger.py           |   7 ++-
>   src/wok/server.py              |   8 ++-
>   src/wok/websocket.py           | 123 ++++++++++++++++++++++++++++++++++++++
>   ui/js/src/wok.main.js          |  38 ++++++++++++
>   ui/js/wok.user-log.js          |   6 ++
>   7 files changed, 327 insertions(+), 5 deletions(-)
>   create mode 100644 src/wok/pushserver.py
>   create mode 100644 src/wok/websocket.py
>
> diff --git a/src/wok/model/notifications.py b/src/wok/model/notifications.py
> index bdb7c78..597eac5 100644
> --- a/src/wok/model/notifications.py
> +++ b/src/wok/model/notifications.py
> @@ -1,7 +1,7 @@
>   #
>   # Project Wok
>   #
> -# Copyright IBM Corp, 2016
> +# Copyright IBM Corp, 2016-2017
>   #
>   # This library is free software; you can redistribute it and/or
>   # modify it under the terms of the GNU Lesser General Public
> @@ -19,12 +19,22 @@
>
>   from datetime import datetime
>
> +from wok import config
>   from wok.exception import NotFoundError, OperationFailed
>   from wok.message import WokMessage
> +from wok.pushserver import PushServer
>   from wok.utils import wok_log
>
>
>   notificationsStore = {}
> +push_server = None
> +
> +
> +def send_websocket_notification(message):
> +    global push_server
> +
> +    if push_server:
> +        push_server.send_notification(message)
>
>
>   def add_notification(code, args=None, plugin_name=None):
> @@ -57,7 +67,11 @@ def del_notification(code):
>
>   class NotificationsModel(object):
>       def __init__(self, **kargs):
> -        pass
> +        global push_server
> +

> +        test_mode = config.config.get('server', 'test').lower() == 'true'
> +        if not test_mode:
> +            push_server = PushServer()

Why distinguish test mode here? While running on test mode, aka, using 
MockModel, the server should behave the same as on real model.

>
>       def get_list(self):
>           global notificationsStore
> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
> new file mode 100644
> index 0000000..8993f00
> --- /dev/null
> +++ b/src/wok/pushserver.py
> @@ -0,0 +1,132 @@
> +#
> +# Project Wok
> +#
> +# Copyright IBM Corp, 2017
> +#
> +# This library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +#
> +# This library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +#
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with this library; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
> +#
> +
> +import cherrypy
> +import os
> +import select
> +import socket
> +import threading
> +

> +import websocket
> +from utils import wok_log
> +

Use full import path.

import wok.websocket
from wok.utils import wok_log

> +
> +BASE_DIRECTORY = '/run'

I think this directory depends on distro settings. Some ones uses /var/run
You will need to get it on build settings. Check config.py.in for details.

> +TOKEN_NAME = 'woknotifications'
> +

Maybe only 'notifications' under a wok directory is better.
Just to place all the wok files related to the same directory. I am not 
sure if others files is places on /run but i think there are.

> +
> +class PushServer(object):
> +
> +    def set_socket_file(self):
> +        if not os.path.isdir(BASE_DIRECTORY):
> +            try:
> +                os.mkdir(BASE_DIRECTORY)
> +            except OSError:
> +                raise RuntimeError('PushServer base UNIX socket dir %s '
> +                                   'not found.' % BASE_DIRECTORY)
> +
> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
> +
> +        if os.path.exists(self.server_addr):
> +            try:
> +                os.remove(self.server_addr)
> +            except:
> +                raise RuntimeError('There is an existing connection in %s' %
> +                                   self.server_addr)
> +
> +    def __init__(self):
> +        self.set_socket_file()
> +
> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
> +
> +        self.connections = []
> +
> +        self.server_running = True
> +        self.server_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
> +                                      socket.SO_REUSEADDR, 1)
> +        self.server_socket.bind(self.server_addr)
> +        self.server_socket.listen(10)
> +        wok_log.info('Push server created on address %s' % self.server_addr)
> +
> +        self.connections.append(self.server_socket)
> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
> +
> +        server_loop = threading.Thread(target=self.listen)
> +        server_loop.start()
> +
> +    def listen(self):
> +        try:
> +            while self.server_running:
> +                read_ready, _, _ = select.select(self.connections,
> +                                                 [], [], 1)
> +                for sock in read_ready:
> +                    if not self.server_running:
> +                        break
> +
> +                    if sock == self.server_socket:
> +
> +                        new_socket, addr = self.server_socket.accept()
> +                        self.connections.append(new_socket)
> +                    else:
> +                        try:
> +                            data = sock.recv(4096)
> +                        except:
> +                            try:
> +                                self.connections.remove(sock)
> +                            except ValueError:
> +                                pass
> +
> +                            continue
> +                        if data and data == 'CLOSE':
> +                            sock.send('ACK')
> +                            try:
> +                                self.connections.remove(sock)
> +                            except ValueError:
> +                                pass
> +                            sock.close()
> +
> +        except Exception as e:
> +            raise RuntimeError('Exception ocurred in listen() of pushserver '
> +                               'module: %s' % e.message)
> +
> +    def send_notification(self, message):
> +        for sock in self.connections:
> +            if sock != self.server_socket:
> +                try:
> +                    sock.send(message)
> +                except IOError as e:
> +                    if 'Broken pipe' in str(e):
> +                        sock.close()
> +                        try:
> +                            self.connections.remove(sock)
> +                        except ValueError:
> +                            pass
> +
> +    def close_server(self):
> +        try:
> +            self.server_running = False
> +            self.server_socket.shutdown(socket.SHUT_RDWR)
> +            self.server_socket.close()
> +            os.remove(self.server_addr)
> +        except:
> +            pass
> +        finally:
> +            cherrypy.engine.unsubscribe('stop', self.close_server)
> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
> index 92e155d..1b774e2 100644
> --- a/src/wok/reqlogger.py
> +++ b/src/wok/reqlogger.py
> @@ -1,7 +1,7 @@
>   #
>   # Project Wok
>   #
> -# Copyright IBM Corp, 2016
> +# Copyright IBM Corp, 2016-2017
>   #
>   # This library is free software; you can redistribute it and/or
>   # modify it under the terms of the GNU Lesser General Public
> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>   from wok.config import get_log_download_path, paths
>   from wok.exception import InvalidParameter, OperationFailed
>   from wok.message import WokMessage
> +from wok.model.notifications import send_websocket_notification
>   from wok.stringutils import ascii_dict
>   from wok.utils import remove_old_files
>
> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>   # AsyncTask handling
>   ASYNCTASK_REQUEST_METHOD = 'TASK'
>
> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
> +
>
>   def log_request(code, params, exception, method, status, app=None, user=None,
>                   ip=None):
> @@ -114,6 +117,8 @@ def log_request(code, params, exception, method, status, app=None, user=None,
>           ip=ip
>       ).log()
>
> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
> +
>       return log_id
>
>
> diff --git a/src/wok/server.py b/src/wok/server.py
> index fc2e167..2d823c9 100644
> --- a/src/wok/server.py
> +++ b/src/wok/server.py
> @@ -25,8 +25,7 @@ import logging
>   import logging.handlers
>   import os
>
> -from wok import auth
> -from wok import config
> +from wok import auth, config, websocket
>   from wok.config import config as configParser
>   from wok.config import WokConfig
>   from wok.control import sub_nodes
> @@ -159,6 +158,11 @@ class Server(object):
>           cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>                               options.server_root, self.configObj)

> +        test_mode = config.config.get('server', 'test').lower() == 'true'
> +        if not test_mode:
> +            ws_proxy = websocket.new_ws_proxy()
> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
> +

The same question I did before.

>           self._load_plugins()
>           cherrypy.lib.sessions.init()
>
> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
> new file mode 100644
> index 0000000..5d7fb91
> --- /dev/null
> +++ b/src/wok/websocket.py
> @@ -0,0 +1,123 @@
> +#!/usr/bin/env python2
> +#
> +# Project Wok
> +#
> +# Copyright IBM Corp, 2017
> +#
> +# Code derived from Project Kimchi
> +#
> +# This library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +#
> +# This library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +#
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with this library; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
> +
> +import base64
> +import errno
> +import os
> +
> +from multiprocessing import Process
> +from websockify import WebSocketProxy
> +
> +from config import config, PluginPaths
> +
> +
> +try:
> +    from websockify.token_plugins import TokenFile
> +    tokenFile = True
> +except ImportError:
> +    tokenFile = False
> +
> +try:
> +    from websockify import ProxyRequestHandler as request_proxy
> +except:
> +    from websockify import WebSocketProxy as request_proxy
> +
> +
> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 'ws-tokens')
> +
> +
> +class CustomHandler(request_proxy):
> +
> +    def get_target(self, target_plugin, path):
> +        if issubclass(CustomHandler, object):
> +            target = super(CustomHandler, self).get_target(target_plugin,
> +                                                           path)
> +        else:
> +            target = request_proxy.get_target(self, target_plugin, path)
> +
> +        if target[0] == 'unix_socket':
> +            try:
> +                self.server.unix_target = target[1]
> +            except:
> +                self.unix_target = target[1]
> +        else:
> +            try:
> +                self.server.unix_target = None
> +            except:
> +                self.unix_target = None
> +        return target
> +
> +
> +def new_ws_proxy():
> +    try:
> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
> +    except OSError as e:
> +        if e.errno == errno.EEXIST:
> +            pass
> +
> +    params = {'listen_host': '127.0.0.1',
> +              'listen_port': config.get('server', 'websockets_port'),
> +              'ssl_only': False}
> +
> +    # old websockify: do not use TokenFile
> +    if not tokenFile:
> +        params['target_cfg'] = WS_TOKENS_DIR
> +
> +    # websockify 0.7 and higher: use TokenFile
> +    else:
> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
> +
> +    def start_proxy():
> +        try:
> +            server = WebSocketProxy(RequestHandlerClass=CustomHandler,
> +                                    **params)
> +        except TypeError:
> +            server = CustomHandler(**params)
> +
> +        server.start_server()
> +
> +    proc = Process(target=start_proxy)
> +    proc.start()
> +    return proc
> +
> +
> +def add_proxy_token(name, port, is_unix_socket=False):
> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
> +        """
> +        From python documentation base64.urlsafe_b64encode(s)
> +        substitutes - instead of + and _ instead of / in the
> +        standard Base64 alphabet, BUT the result can still
> +        contain = which is not safe in a URL query component.
> +        So remove it when needed as base64 can work well without it.
> +        """
> +        name = base64.urlsafe_b64encode(name).rstrip('=')
> +        if is_unix_socket:
> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), port))
> +        else:
> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
> +
> +
> +def remove_proxy_token(name):
> +    try:
> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
> +    except OSError:
> +        pass
> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
> index 20c017e..f5031ce 100644
> --- a/ui/js/src/wok.main.js
> +++ b/ui/js/src/wok.main.js
> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>       wok.config = {};
>   });
>
> +
> +wok.notificationListeners = {};
> +wok.addNotificationListener = function(name, func) {
> +    wok.notificationListeners[name] = func;
> +    $(window).one("hashchange", function() {
> +        delete wok.notificationListeners[name];
> +    });
> +};
> +
> +wok.notificationsWebSocket = undefined;
> +wok.startNotificationWebSocket = function () {
> +    var addr = window.location.hostname + ':' + window.location.port;
> +    var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
> +    var url = 'wss://' + addr + '/websockify?token=' + token;
> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
> +
> +    wok.notificationsWebSocket.onmessage = function(event) {
> +        var message = window.atob(event.data);
> +        for (name in wok.notificationListeners) {
> +            func = wok.notificationListeners[name];
> +            func(message);
> +        }
> +    };
> +
> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
> +    var heartbeat = setInterval(function() {
> +        wok.notificationsWebSocket.send(window.btoa('heartbeat'));
> +    }, 30000);
> +
> +    wok.notificationsWebSocket.onclose = function() {
> +        clearInterval(heartbeat);
> +    };
> +};
> +
> +
>   wok.main = function() {
>       wok.isLoggingOut = false;
>       wok.popable();
> @@ -395,6 +430,9 @@ wok.main = function() {
>
>           // Set handler for help button
>           $('#btn-help').on('click', wok.openHelp);
> +
> +        // start WebSocket
> +        wok.startNotificationWebSocket();
>       };
>
>       var initUI = function() {
> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
> index 0e8fb09..083b6c3 100644
> --- a/ui/js/wok.user-log.js
> +++ b/ui/js/wok.user-log.js
> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>           $("#user-log-grid").bootgrid("search");
>           wok.initUserLogConfigGridData();
>       });
> +

> +    wok.addNotificationListener('userlog', function(message) {
> +        if (message === 'new_log_entry') {
> +            $("#refresh-button").click();
> +        }
> +    });

Backend is sending a notification with 'new_log_entry' and you are 
registering a notification for 'userlog'.
How is it working?

Shouldn't the 'name' required by addNotificationListener be 
'new_log_entry'? So wok would be redirect to the function automatically?
Why do you need to distinguish the 'new_log_entry' later?

Also, we will need to create a pattern message for notification to avoid 
having a plugin overwriting a notification from another plugin.

That said, my suggestion is: <plugin>_<resource>_<action> which will 
result in something like:

wok_log_create
kimchi_vms_create
kimchi_vm_update
kimchi_vm_start
ginger_backups_create
ginger_backup_restore

and so on.

That way, we can automatically add the notification on control/base.py 
which will work for everyone. And then, the plugin just need to register 
the listener on UI.
Something like:

@model/base.py
#Collection.index and Resource.index (need to cover both)
def index(...):
(...)
     # on sucsess
     notification = "%s_%s_%s" % (plugin_name, resource/collection, action)
     send_websocket_notification(notification)

def _generate_action_handler_base(...):
     (...)
     # on success
     notification = "%s_%s_%s" % (plugin_name, resource/collection, action)
     send_websocket_notification(notification)

What do you think?

>   };
>
>   wok.initUserLogWindow = function() {

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago

On 02/27/2017 11:23 AM, Aline Manera wrote:
>
>
> On 02/24/2017 10:22 AM, dhbarboza82@gmail.com wrote:
>> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>
>> This patch makes backend and UI changes to implement the asynchronous
>> UI notification in WoK.
>>
>> - Backend:
>>
>> A push server was implemented from scratch to manage the opened 
>> websocket
>> connections. The push server connects to the /run/woknotifications
>> UNIX socket and broadcasts all messages to all connections.
>>
>> The websocket module is the same module that exists in the Kimchi
>> plug-in. The idea is to remove the module from Kimchi and make it
>> use the module from WoK. ws_proxy initialization was also added
>> in src/wok/server.py.
>>
>> - Frontend:
>>
>> In ui/js/wok.main.js two new functions were added to help the
>> usage of asynchronous notifications in the frontend. The idea:
>> a single websocket is opened per session. This opened websocket
>> will broadcast all incoming messages to all listeners registered.
>> Listeners can be added by the new wok.addNotificationListener()
>> method. This method will clean up any registered listener by
>> itself when the user changes tabs/URL.
>>
>> The single websocket sends heartbeats to the backend side each
>> 30 seconds. No reply from the backend is issued or expected. This
>> heartbeat is just a way to ensure that the browser does not
>> close the connection due to inactivity. This behavior varies from
>> browser to browser but this 30 second heartbeat is more than enough
>> to ensure that the websocket is kept alive.
>>
>> - Working example in User Log:
>>
>> A simple usage is provided in this patch. A change was made in
>> src/wok/reqlogger.py to send an asynchronous notification each
>> time a new log entry is created. In ui/js/wok.user-log.js a
>> websocket listener is added using wok.addNotificationListener()
>> and, for each message that indicates a new user log entry, a
>> refresh in the listing is issued.
>>
>> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>> ---
>>   src/wok/model/notifications.py |  18 +++++-
>>   src/wok/pushserver.py          | 132 
>> +++++++++++++++++++++++++++++++++++++++++
>>   src/wok/reqlogger.py           |   7 ++-
>>   src/wok/server.py              |   8 ++-
>>   src/wok/websocket.py           | 123 
>> ++++++++++++++++++++++++++++++++++++++
>>   ui/js/src/wok.main.js          |  38 ++++++++++++
>>   ui/js/wok.user-log.js          |   6 ++
>>   7 files changed, 327 insertions(+), 5 deletions(-)
>>   create mode 100644 src/wok/pushserver.py
>>   create mode 100644 src/wok/websocket.py
>>
>> diff --git a/src/wok/model/notifications.py 
>> b/src/wok/model/notifications.py
>> index bdb7c78..597eac5 100644
>> --- a/src/wok/model/notifications.py
>> +++ b/src/wok/model/notifications.py
>> @@ -1,7 +1,7 @@
>>   #
>>   # Project Wok
>>   #
>> -# Copyright IBM Corp, 2016
>> +# Copyright IBM Corp, 2016-2017
>>   #
>>   # This library is free software; you can redistribute it and/or
>>   # modify it under the terms of the GNU Lesser General Public
>> @@ -19,12 +19,22 @@
>>
>>   from datetime import datetime
>>
>> +from wok import config
>>   from wok.exception import NotFoundError, OperationFailed
>>   from wok.message import WokMessage
>> +from wok.pushserver import PushServer
>>   from wok.utils import wok_log
>>
>>
>>   notificationsStore = {}
>> +push_server = None
>> +
>> +
>> +def send_websocket_notification(message):
>> +    global push_server
>> +
>> +    if push_server:
>> +        push_server.send_notification(message)
>>
>>
>>   def add_notification(code, args=None, plugin_name=None):
>> @@ -57,7 +67,11 @@ def del_notification(code):
>>
>>   class NotificationsModel(object):
>>       def __init__(self, **kargs):
>> -        pass
>> +        global push_server
>> +
>
>> +        test_mode = config.config.get('server', 'test').lower() == 
>> 'true'
>> +        if not test_mode:
>> +            push_server = PushServer()
>
> Why distinguish test mode here? While running on test mode, aka, using 
> MockModel, the server should behave the same as on real model.

It makes no sense to enable the push server without websocket support. As
it is now, on Kimchi code, the websocket proxy doesn't initialize in
test_mode. This same design was kept when moving the initialization
to WoK.

I've also experienced conflicts when running the unit tests with the 
push server.
The multiple instancing of test_model led to the multiple instancing of
the push_server, causing conflict because all push server instances read
from the same unix socket.

>
>>
>>       def get_list(self):
>>           global notificationsStore
>> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
>> new file mode 100644
>> index 0000000..8993f00
>> --- /dev/null
>> +++ b/src/wok/pushserver.py
>> @@ -0,0 +1,132 @@
>> +#
>> +# Project Wok
>> +#
>> +# Copyright IBM Corp, 2017
>> +#
>> +# This library is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU Lesser General Public
>> +# License as published by the Free Software Foundation; either
>> +# version 2.1 of the License, or (at your option) any later version.
>> +#
>> +# This library is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> +# Lesser General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU Lesser General Public
>> +# License along with this library; if not, write to the Free Software
>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>> 02110-1301 USA
>> +#
>> +
>> +import cherrypy
>> +import os
>> +import select
>> +import socket
>> +import threading
>> +
>
>> +import websocket
>> +from utils import wok_log
>> +
>
> Use full import path.
>
> import wok.websocket
> from wok.utils import wok_log

ok
>
>> +
>> +BASE_DIRECTORY = '/run'
>
> I think this directory depends on distro settings. Some ones uses 
> /var/run
> You will need to get it on build settings. Check config.py.in for 
> details.

I've copied this line/logic from Kimchi's serialconsole.py assuming it was a
good way to do it. I'll see if using config.py.in in this case is possible.

We can also discuss if putting it under /run is the choice and perhaps 
choose
another path entirely.

>
>> +TOKEN_NAME = 'woknotifications'
>> +
>
> Maybe only 'notifications' under a wok directory is better.
> Just to place all the wok files related to the same directory. I am 
> not sure if others files is places on /run but i think there are.

As I've said above, I'm open to suggestions of where to put this unix 
socket.

>
>> +
>> +class PushServer(object):
>> +
>> +    def set_socket_file(self):
>> +        if not os.path.isdir(BASE_DIRECTORY):
>> +            try:
>> +                os.mkdir(BASE_DIRECTORY)
>> +            except OSError:
>> +                raise RuntimeError('PushServer base UNIX socket dir 
>> %s '
>> +                                   'not found.' % BASE_DIRECTORY)
>> +
>> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
>> +
>> +        if os.path.exists(self.server_addr):
>> +            try:
>> +                os.remove(self.server_addr)
>> +            except:
>> +                raise RuntimeError('There is an existing connection 
>> in %s' %
>> +                                   self.server_addr)
>> +
>> +    def __init__(self):
>> +        self.set_socket_file()
>> +
>> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
>> +
>> +        self.connections = []
>> +
>> +        self.server_running = True
>> +        self.server_socket = socket.socket(socket.AF_UNIX, 
>> socket.SOCK_STREAM)
>> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
>> +                                      socket.SO_REUSEADDR, 1)
>> +        self.server_socket.bind(self.server_addr)
>> +        self.server_socket.listen(10)
>> +        wok_log.info('Push server created on address %s' % 
>> self.server_addr)
>> +
>> +        self.connections.append(self.server_socket)
>> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
>> +
>> +        server_loop = threading.Thread(target=self.listen)
>> +        server_loop.start()
>> +
>> +    def listen(self):
>> +        try:
>> +            while self.server_running:
>> +                read_ready, _, _ = select.select(self.connections,
>> +                                                 [], [], 1)
>> +                for sock in read_ready:
>> +                    if not self.server_running:
>> +                        break
>> +
>> +                    if sock == self.server_socket:
>> +
>> +                        new_socket, addr = self.server_socket.accept()
>> +                        self.connections.append(new_socket)
>> +                    else:
>> +                        try:
>> +                            data = sock.recv(4096)
>> +                        except:
>> +                            try:
>> +                                self.connections.remove(sock)
>> +                            except ValueError:
>> +                                pass
>> +
>> +                            continue
>> +                        if data and data == 'CLOSE':
>> +                            sock.send('ACK')
>> +                            try:
>> +                                self.connections.remove(sock)
>> +                            except ValueError:
>> +                                pass
>> +                            sock.close()
>> +
>> +        except Exception as e:
>> +            raise RuntimeError('Exception ocurred in listen() of 
>> pushserver '
>> +                               'module: %s' % e.message)
>> +
>> +    def send_notification(self, message):
>> +        for sock in self.connections:
>> +            if sock != self.server_socket:
>> +                try:
>> +                    sock.send(message)
>> +                except IOError as e:
>> +                    if 'Broken pipe' in str(e):
>> +                        sock.close()
>> +                        try:
>> +                            self.connections.remove(sock)
>> +                        except ValueError:
>> +                            pass
>> +
>> +    def close_server(self):
>> +        try:
>> +            self.server_running = False
>> +            self.server_socket.shutdown(socket.SHUT_RDWR)
>> +            self.server_socket.close()
>> +            os.remove(self.server_addr)
>> +        except:
>> +            pass
>> +        finally:
>> +            cherrypy.engine.unsubscribe('stop', self.close_server)
>> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
>> index 92e155d..1b774e2 100644
>> --- a/src/wok/reqlogger.py
>> +++ b/src/wok/reqlogger.py
>> @@ -1,7 +1,7 @@
>>   #
>>   # Project Wok
>>   #
>> -# Copyright IBM Corp, 2016
>> +# Copyright IBM Corp, 2016-2017
>>   #
>>   # This library is free software; you can redistribute it and/or
>>   # modify it under the terms of the GNU Lesser General Public
>> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>>   from wok.config import get_log_download_path, paths
>>   from wok.exception import InvalidParameter, OperationFailed
>>   from wok.message import WokMessage
>> +from wok.model.notifications import send_websocket_notification
>>   from wok.stringutils import ascii_dict
>>   from wok.utils import remove_old_files
>>
>> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>>   # AsyncTask handling
>>   ASYNCTASK_REQUEST_METHOD = 'TASK'
>>
>> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
>> +
>>
>>   def log_request(code, params, exception, method, status, app=None, 
>> user=None,
>>                   ip=None):
>> @@ -114,6 +117,8 @@ def log_request(code, params, exception, method, 
>> status, app=None, user=None,
>>           ip=ip
>>       ).log()
>>
>> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
>> +
>>       return log_id
>>
>>
>> diff --git a/src/wok/server.py b/src/wok/server.py
>> index fc2e167..2d823c9 100644
>> --- a/src/wok/server.py
>> +++ b/src/wok/server.py
>> @@ -25,8 +25,7 @@ import logging
>>   import logging.handlers
>>   import os
>>
>> -from wok import auth
>> -from wok import config
>> +from wok import auth, config, websocket
>>   from wok.config import config as configParser
>>   from wok.config import WokConfig
>>   from wok.control import sub_nodes
>> @@ -159,6 +158,11 @@ class Server(object):
>>           cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>>                               options.server_root, self.configObj)
>
>> +        test_mode = config.config.get('server', 'test').lower() == 
>> 'true'
>> +        if not test_mode:
>> +            ws_proxy = websocket.new_ws_proxy()
>> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
>> +
>
> The same question I did before.

See answer above.

>
>>           self._load_plugins()
>>           cherrypy.lib.sessions.init()
>>
>> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
>> new file mode 100644
>> index 0000000..5d7fb91
>> --- /dev/null
>> +++ b/src/wok/websocket.py
>> @@ -0,0 +1,123 @@
>> +#!/usr/bin/env python2
>> +#
>> +# Project Wok
>> +#
>> +# Copyright IBM Corp, 2017
>> +#
>> +# Code derived from Project Kimchi
>> +#
>> +# This library is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU Lesser General Public
>> +# License as published by the Free Software Foundation; either
>> +# version 2.1 of the License, or (at your option) any later version.
>> +#
>> +# This library is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> +# Lesser General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU Lesser General Public
>> +# License along with this library; if not, write to the Free Software
>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>> 02110-1301 USA
>> +
>> +import base64
>> +import errno
>> +import os
>> +
>> +from multiprocessing import Process
>> +from websockify import WebSocketProxy
>> +
>> +from config import config, PluginPaths
>> +
>> +
>> +try:
>> +    from websockify.token_plugins import TokenFile
>> +    tokenFile = True
>> +except ImportError:
>> +    tokenFile = False
>> +
>> +try:
>> +    from websockify import ProxyRequestHandler as request_proxy
>> +except:
>> +    from websockify import WebSocketProxy as request_proxy
>> +
>> +
>> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 
>> 'ws-tokens')
>> +
>> +
>> +class CustomHandler(request_proxy):
>> +
>> +    def get_target(self, target_plugin, path):
>> +        if issubclass(CustomHandler, object):
>> +            target = super(CustomHandler, 
>> self).get_target(target_plugin,
>> + path)
>> +        else:
>> +            target = request_proxy.get_target(self, target_plugin, 
>> path)
>> +
>> +        if target[0] == 'unix_socket':
>> +            try:
>> +                self.server.unix_target = target[1]
>> +            except:
>> +                self.unix_target = target[1]
>> +        else:
>> +            try:
>> +                self.server.unix_target = None
>> +            except:
>> +                self.unix_target = None
>> +        return target
>> +
>> +
>> +def new_ws_proxy():
>> +    try:
>> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
>> +    except OSError as e:
>> +        if e.errno == errno.EEXIST:
>> +            pass
>> +
>> +    params = {'listen_host': '127.0.0.1',
>> +              'listen_port': config.get('server', 'websockets_port'),
>> +              'ssl_only': False}
>> +
>> +    # old websockify: do not use TokenFile
>> +    if not tokenFile:
>> +        params['target_cfg'] = WS_TOKENS_DIR
>> +
>> +    # websockify 0.7 and higher: use TokenFile
>> +    else:
>> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
>> +
>> +    def start_proxy():
>> +        try:
>> +            server = WebSocketProxy(RequestHandlerClass=CustomHandler,
>> +                                    **params)
>> +        except TypeError:
>> +            server = CustomHandler(**params)
>> +
>> +        server.start_server()
>> +
>> +    proc = Process(target=start_proxy)
>> +    proc.start()
>> +    return proc
>> +
>> +
>> +def add_proxy_token(name, port, is_unix_socket=False):
>> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
>> +        """
>> +        From python documentation base64.urlsafe_b64encode(s)
>> +        substitutes - instead of + and _ instead of / in the
>> +        standard Base64 alphabet, BUT the result can still
>> +        contain = which is not safe in a URL query component.
>> +        So remove it when needed as base64 can work well without it.
>> +        """
>> +        name = base64.urlsafe_b64encode(name).rstrip('=')
>> +        if is_unix_socket:
>> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), 
>> port))
>> +        else:
>> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
>> +
>> +
>> +def remove_proxy_token(name):
>> +    try:
>> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
>> +    except OSError:
>> +        pass
>> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
>> index 20c017e..f5031ce 100644
>> --- a/ui/js/src/wok.main.js
>> +++ b/ui/js/src/wok.main.js
>> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>>       wok.config = {};
>>   });
>>
>> +
>> +wok.notificationListeners = {};
>> +wok.addNotificationListener = function(name, func) {
>> +    wok.notificationListeners[name] = func;
>> +    $(window).one("hashchange", function() {
>> +        delete wok.notificationListeners[name];
>> +    });
>> +};
>> +
>> +wok.notificationsWebSocket = undefined;
>> +wok.startNotificationWebSocket = function () {
>> +    var addr = window.location.hostname + ':' + window.location.port;
>> +    var token = 
>> wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>> +    var url = 'wss://' + addr + '/websockify?token=' + token;
>> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
>> +
>> +    wok.notificationsWebSocket.onmessage = function(event) {
>> +        var message = window.atob(event.data);
>> +        for (name in wok.notificationListeners) {
>> +            func = wok.notificationListeners[name];
>> +            func(message);
>> +        }
>> +    };
>> +
>> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
>> +    var heartbeat = setInterval(function() {
>> + wok.notificationsWebSocket.send(window.btoa('heartbeat'));
>> +    }, 30000);
>> +
>> +    wok.notificationsWebSocket.onclose = function() {
>> +        clearInterval(heartbeat);
>> +    };
>> +};
>> +
>> +
>>   wok.main = function() {
>>       wok.isLoggingOut = false;
>>       wok.popable();
>> @@ -395,6 +430,9 @@ wok.main = function() {
>>
>>           // Set handler for help button
>>           $('#btn-help').on('click', wok.openHelp);
>> +
>> +        // start WebSocket
>> +        wok.startNotificationWebSocket();
>>       };
>>
>>       var initUI = function() {
>> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
>> index 0e8fb09..083b6c3 100644
>> --- a/ui/js/wok.user-log.js
>> +++ b/ui/js/wok.user-log.js
>> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>>           $("#user-log-grid").bootgrid("search");
>>           wok.initUserLogConfigGridData();
>>       });
>> +
>
>> + wok.addNotificationListener('userlog', function(message) {
>> +        if (message === 'new_log_entry') {
>> +            $("#refresh-button").click();
>> +        }
>> +    });
>
> Backend is sending a notification with 'new_log_entry' and you are 
> registering a notification for 'userlog'.
> How is it working?

'userlog' is the identifier of the listener that will be used in the hash
wok.notificationListeners.

>
> Shouldn't the 'name' required by addNotificationListener be 
> 'new_log_entry'? So wok would be redirect to the function automatically?
> Why do you need to distinguish the 'new_log_entry' later?
>
> Also, we will need to create a pattern message for notification to 
> avoid having a plugin overwriting a notification from another plugin.

Agree. I was aware that this 'new_log_entry' was going to be changed by
something more formal.

>
> That said, my suggestion is: <plugin>_<resource>_<action> which will 
> result in something like:
>
> wok_log_create
> kimchi_vms_create
> kimchi_vm_update
> kimchi_vm_start
> ginger_backups_create
> ginger_backup_restore
>
> and so on.
>
> That way, we can automatically add the notification on control/base.py 
> which will work for everyone. And then, the plugin just need to 
> register the listener on UI.
> Something like:
>
> @model/base.py
> #Collection.index and Resource.index (need to cover both)
> def index(...):
> (...)
>     # on sucsess
>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
> action)
>     send_websocket_notification(notification)
>
> def _generate_action_handler_base(...):
>     (...)
>     # on success
>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
> action)
>     send_websocket_notification(notification)
>
> What do you think?

I think we can work with this plugin_name/entity/action format, perhaps 
adding
an 'args' field to it. In the case of this patch we would have 
'wok/userlog/create'
or something similar.

We need to also consider task handling in this context. We can change 
the default
success/failure handles to also automatically report the status to the 
UI via
websocket.

In fact, we need to consider a lot of standardization with this feature. 
For this patch,
I we can start by doing the action handler changes you've proposed to 
implement the
user log changes. I have no problems putting the notification under the 
'index' of
both Collection and Resource models too.


Daniel
>
>>   };
>>
>>   wok.initUserLogWindow = function() {
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/27/2017 05:58 PM, Daniel Henrique Barboza wrote:
>
>
> On 02/27/2017 11:23 AM, Aline Manera wrote:
>>
>>
>> On 02/24/2017 10:22 AM, dhbarboza82@gmail.com wrote:
>>> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>>
>>> This patch makes backend and UI changes to implement the asynchronous
>>> UI notification in WoK.
>>>
>>> - Backend:
>>>
>>> A push server was implemented from scratch to manage the opened 
>>> websocket
>>> connections. The push server connects to the /run/woknotifications
>>> UNIX socket and broadcasts all messages to all connections.
>>>
>>> The websocket module is the same module that exists in the Kimchi
>>> plug-in. The idea is to remove the module from Kimchi and make it
>>> use the module from WoK. ws_proxy initialization was also added
>>> in src/wok/server.py.
>>>
>>> - Frontend:
>>>
>>> In ui/js/wok.main.js two new functions were added to help the
>>> usage of asynchronous notifications in the frontend. The idea:
>>> a single websocket is opened per session. This opened websocket
>>> will broadcast all incoming messages to all listeners registered.
>>> Listeners can be added by the new wok.addNotificationListener()
>>> method. This method will clean up any registered listener by
>>> itself when the user changes tabs/URL.
>>>
>>> The single websocket sends heartbeats to the backend side each
>>> 30 seconds. No reply from the backend is issued or expected. This
>>> heartbeat is just a way to ensure that the browser does not
>>> close the connection due to inactivity. This behavior varies from
>>> browser to browser but this 30 second heartbeat is more than enough
>>> to ensure that the websocket is kept alive.
>>>
>>> - Working example in User Log:
>>>
>>> A simple usage is provided in this patch. A change was made in
>>> src/wok/reqlogger.py to send an asynchronous notification each
>>> time a new log entry is created. In ui/js/wok.user-log.js a
>>> websocket listener is added using wok.addNotificationListener()
>>> and, for each message that indicates a new user log entry, a
>>> refresh in the listing is issued.
>>>
>>> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>> ---
>>>   src/wok/model/notifications.py |  18 +++++-
>>>   src/wok/pushserver.py          | 132 
>>> +++++++++++++++++++++++++++++++++++++++++
>>>   src/wok/reqlogger.py           |   7 ++-
>>>   src/wok/server.py              |   8 ++-
>>>   src/wok/websocket.py           | 123 
>>> ++++++++++++++++++++++++++++++++++++++
>>>   ui/js/src/wok.main.js          |  38 ++++++++++++
>>>   ui/js/wok.user-log.js          |   6 ++
>>>   7 files changed, 327 insertions(+), 5 deletions(-)
>>>   create mode 100644 src/wok/pushserver.py
>>>   create mode 100644 src/wok/websocket.py
>>>
>>> diff --git a/src/wok/model/notifications.py 
>>> b/src/wok/model/notifications.py
>>> index bdb7c78..597eac5 100644
>>> --- a/src/wok/model/notifications.py
>>> +++ b/src/wok/model/notifications.py
>>> @@ -1,7 +1,7 @@
>>>   #
>>>   # Project Wok
>>>   #
>>> -# Copyright IBM Corp, 2016
>>> +# Copyright IBM Corp, 2016-2017
>>>   #
>>>   # This library is free software; you can redistribute it and/or
>>>   # modify it under the terms of the GNU Lesser General Public
>>> @@ -19,12 +19,22 @@
>>>
>>>   from datetime import datetime
>>>
>>> +from wok import config
>>>   from wok.exception import NotFoundError, OperationFailed
>>>   from wok.message import WokMessage
>>> +from wok.pushserver import PushServer
>>>   from wok.utils import wok_log
>>>
>>>
>>>   notificationsStore = {}
>>> +push_server = None
>>> +
>>> +
>>> +def send_websocket_notification(message):
>>> +    global push_server
>>> +
>>> +    if push_server:
>>> +        push_server.send_notification(message)
>>>
>>>
>>>   def add_notification(code, args=None, plugin_name=None):
>>> @@ -57,7 +67,11 @@ def del_notification(code):
>>>
>>>   class NotificationsModel(object):
>>>       def __init__(self, **kargs):
>>> -        pass
>>> +        global push_server
>>> +
>>
>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>> 'true'
>>> +        if not test_mode:
>>> +            push_server = PushServer()
>>
>> Why distinguish test mode here? While running on test mode, aka, 
>> using MockModel, the server should behave the same as on real model.
>
> It makes no sense to enable the push server without websocket support. As
> it is now, on Kimchi code, the websocket proxy doesn't initialize in
> test_mode. This same design was kept when moving the initialization
> to WoK.
>
> I've also experienced conflicts when running the unit tests with the 
> push server.
> The multiple instancing of test_model led to the multiple instancing of
> the push_server, causing conflict because all push server instances read
> from the same unix socket.
>

Maybe initializing the websockets only on non-test mode on Kimchi makes 
sense, ie, the libvirt test driver may not be allowed to streaming the 
console.
But for a Wok perspective, the websockets is now being used for the 
whole application, so there is no need to distinguish the test mode.

Thinking about Selenium tests which will run on test mode (aka, using 
MockModel) we need to have the same UI behavior as running on real model 
to validate the tests so the async notification should work as well.

Do not mistake tests cases with the wok test mode. The test mode is a 
dry run environment to user tries Wok without affecting the system (aka, 
using the MockModel).
We should not deliver a partial implementation to production mode due 
the tests cases.

Also, if you move the Push Server initialization to Server (as I 
commented in another reply), the Push Server will be only initialized 
when Server is, ie, the test_model* which does not initialize the server 
will not have the Push Server and maybe the problems you saw will go away.

>>
>>>
>>>       def get_list(self):
>>>           global notificationsStore
>>> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
>>> new file mode 100644
>>> index 0000000..8993f00
>>> --- /dev/null
>>> +++ b/src/wok/pushserver.py
>>> @@ -0,0 +1,132 @@
>>> +#
>>> +# Project Wok
>>> +#
>>> +# Copyright IBM Corp, 2017
>>> +#
>>> +# This library is free software; you can redistribute it and/or
>>> +# modify it under the terms of the GNU Lesser General Public
>>> +# License as published by the Free Software Foundation; either
>>> +# version 2.1 of the License, or (at your option) any later version.
>>> +#
>>> +# This library is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> +# Lesser General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU Lesser General Public
>>> +# License along with this library; if not, write to the Free Software
>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>> 02110-1301 USA
>>> +#
>>> +
>>> +import cherrypy
>>> +import os
>>> +import select
>>> +import socket
>>> +import threading
>>> +
>>
>>> +import websocket
>>> +from utils import wok_log
>>> +
>>
>> Use full import path.
>>
>> import wok.websocket
>> from wok.utils import wok_log
>
> ok
>>
>>> +
>>> +BASE_DIRECTORY = '/run'
>>
>> I think this directory depends on distro settings. Some ones uses 
>> /var/run
>> You will need to get it on build settings. Check config.py.in for 
>> details.
>
> I've copied this line/logic from Kimchi's serialconsole.py assuming it 
> was a
> good way to do it. I'll see if using config.py.in in this case is 
> possible.
>
> We can also discuss if putting it under /run is the choice and perhaps 
> choose
> another path entirely.
>

OK.

>>
>>> +TOKEN_NAME = 'woknotifications'
>>> +
>>
>> Maybe only 'notifications' under a wok directory is better.
>> Just to place all the wok files related to the same directory. I am 
>> not sure if others files is places on /run but i think there are.
>
> As I've said above, I'm open to suggestions of where to put this unix 
> socket.
>
>>
>>> +
>>> +class PushServer(object):
>>> +
>>> +    def set_socket_file(self):
>>> +        if not os.path.isdir(BASE_DIRECTORY):
>>> +            try:
>>> +                os.mkdir(BASE_DIRECTORY)
>>> +            except OSError:
>>> +                raise RuntimeError('PushServer base UNIX socket dir 
>>> %s '
>>> +                                   'not found.' % BASE_DIRECTORY)
>>> +
>>> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
>>> +
>>> +        if os.path.exists(self.server_addr):
>>> +            try:
>>> +                os.remove(self.server_addr)
>>> +            except:
>>> +                raise RuntimeError('There is an existing connection 
>>> in %s' %
>>> +                                   self.server_addr)
>>> +
>>> +    def __init__(self):
>>> +        self.set_socket_file()
>>> +
>>> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
>>> +
>>> +        self.connections = []
>>> +
>>> +        self.server_running = True
>>> +        self.server_socket = socket.socket(socket.AF_UNIX, 
>>> socket.SOCK_STREAM)
>>> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
>>> +                                      socket.SO_REUSEADDR, 1)
>>> +        self.server_socket.bind(self.server_addr)
>>> +        self.server_socket.listen(10)
>>> +        wok_log.info('Push server created on address %s' % 
>>> self.server_addr)
>>> +
>>> +        self.connections.append(self.server_socket)
>>> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
>>> +
>>> +        server_loop = threading.Thread(target=self.listen)
>>> +        server_loop.start()
>>> +
>>> +    def listen(self):
>>> +        try:
>>> +            while self.server_running:
>>> +                read_ready, _, _ = select.select(self.connections,
>>> +                                                 [], [], 1)
>>> +                for sock in read_ready:
>>> +                    if not self.server_running:
>>> +                        break
>>> +
>>> +                    if sock == self.server_socket:
>>> +
>>> +                        new_socket, addr = self.server_socket.accept()
>>> +                        self.connections.append(new_socket)
>>> +                    else:
>>> +                        try:
>>> +                            data = sock.recv(4096)
>>> +                        except:
>>> +                            try:
>>> +                                self.connections.remove(sock)
>>> +                            except ValueError:
>>> +                                pass
>>> +
>>> +                            continue
>>> +                        if data and data == 'CLOSE':
>>> +                            sock.send('ACK')
>>> +                            try:
>>> +                                self.connections.remove(sock)
>>> +                            except ValueError:
>>> +                                pass
>>> +                            sock.close()
>>> +
>>> +        except Exception as e:
>>> +            raise RuntimeError('Exception ocurred in listen() of 
>>> pushserver '
>>> +                               'module: %s' % e.message)
>>> +
>>> +    def send_notification(self, message):
>>> +        for sock in self.connections:
>>> +            if sock != self.server_socket:
>>> +                try:
>>> +                    sock.send(message)
>>> +                except IOError as e:
>>> +                    if 'Broken pipe' in str(e):
>>> +                        sock.close()
>>> +                        try:
>>> +                            self.connections.remove(sock)
>>> +                        except ValueError:
>>> +                            pass
>>> +
>>> +    def close_server(self):
>>> +        try:
>>> +            self.server_running = False
>>> +            self.server_socket.shutdown(socket.SHUT_RDWR)
>>> +            self.server_socket.close()
>>> +            os.remove(self.server_addr)
>>> +        except:
>>> +            pass
>>> +        finally:
>>> +            cherrypy.engine.unsubscribe('stop', self.close_server)
>>> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
>>> index 92e155d..1b774e2 100644
>>> --- a/src/wok/reqlogger.py
>>> +++ b/src/wok/reqlogger.py
>>> @@ -1,7 +1,7 @@
>>>   #
>>>   # Project Wok
>>>   #
>>> -# Copyright IBM Corp, 2016
>>> +# Copyright IBM Corp, 2016-2017
>>>   #
>>>   # This library is free software; you can redistribute it and/or
>>>   # modify it under the terms of the GNU Lesser General Public
>>> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>>>   from wok.config import get_log_download_path, paths
>>>   from wok.exception import InvalidParameter, OperationFailed
>>>   from wok.message import WokMessage
>>> +from wok.model.notifications import send_websocket_notification
>>>   from wok.stringutils import ascii_dict
>>>   from wok.utils import remove_old_files
>>>
>>> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>>>   # AsyncTask handling
>>>   ASYNCTASK_REQUEST_METHOD = 'TASK'
>>>
>>> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
>>> +
>>>
>>>   def log_request(code, params, exception, method, status, app=None, 
>>> user=None,
>>>                   ip=None):
>>> @@ -114,6 +117,8 @@ def log_request(code, params, exception, method, 
>>> status, app=None, user=None,
>>>           ip=ip
>>>       ).log()
>>>
>>> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
>>> +
>>>       return log_id
>>>
>>>
>>> diff --git a/src/wok/server.py b/src/wok/server.py
>>> index fc2e167..2d823c9 100644
>>> --- a/src/wok/server.py
>>> +++ b/src/wok/server.py
>>> @@ -25,8 +25,7 @@ import logging
>>>   import logging.handlers
>>>   import os
>>>
>>> -from wok import auth
>>> -from wok import config
>>> +from wok import auth, config, websocket
>>>   from wok.config import config as configParser
>>>   from wok.config import WokConfig
>>>   from wok.control import sub_nodes
>>> @@ -159,6 +158,11 @@ class Server(object):
>>>           cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>>>                               options.server_root, self.configObj)
>>
>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>> 'true'
>>> +        if not test_mode:
>>> +            ws_proxy = websocket.new_ws_proxy()
>>> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
>>> +
>>
>> The same question I did before.
>
> See answer above.
>
>>
>>>           self._load_plugins()
>>>           cherrypy.lib.sessions.init()
>>>
>>> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
>>> new file mode 100644
>>> index 0000000..5d7fb91
>>> --- /dev/null
>>> +++ b/src/wok/websocket.py
>>> @@ -0,0 +1,123 @@
>>> +#!/usr/bin/env python2
>>> +#
>>> +# Project Wok
>>> +#
>>> +# Copyright IBM Corp, 2017
>>> +#
>>> +# Code derived from Project Kimchi
>>> +#
>>> +# This library is free software; you can redistribute it and/or
>>> +# modify it under the terms of the GNU Lesser General Public
>>> +# License as published by the Free Software Foundation; either
>>> +# version 2.1 of the License, or (at your option) any later version.
>>> +#
>>> +# This library is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> +# Lesser General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU Lesser General Public
>>> +# License along with this library; if not, write to the Free Software
>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>> 02110-1301 USA
>>> +
>>> +import base64
>>> +import errno
>>> +import os
>>> +
>>> +from multiprocessing import Process
>>> +from websockify import WebSocketProxy
>>> +
>>> +from config import config, PluginPaths
>>> +
>>> +
>>> +try:
>>> +    from websockify.token_plugins import TokenFile
>>> +    tokenFile = True
>>> +except ImportError:
>>> +    tokenFile = False
>>> +
>>> +try:
>>> +    from websockify import ProxyRequestHandler as request_proxy
>>> +except:
>>> +    from websockify import WebSocketProxy as request_proxy
>>> +
>>> +
>>> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 
>>> 'ws-tokens')
>>> +
>>> +
>>> +class CustomHandler(request_proxy):
>>> +
>>> +    def get_target(self, target_plugin, path):
>>> +        if issubclass(CustomHandler, object):
>>> +            target = super(CustomHandler, 
>>> self).get_target(target_plugin,
>>> + path)
>>> +        else:
>>> +            target = request_proxy.get_target(self, target_plugin, 
>>> path)
>>> +
>>> +        if target[0] == 'unix_socket':
>>> +            try:
>>> +                self.server.unix_target = target[1]
>>> +            except:
>>> +                self.unix_target = target[1]
>>> +        else:
>>> +            try:
>>> +                self.server.unix_target = None
>>> +            except:
>>> +                self.unix_target = None
>>> +        return target
>>> +
>>> +
>>> +def new_ws_proxy():
>>> +    try:
>>> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
>>> +    except OSError as e:
>>> +        if e.errno == errno.EEXIST:
>>> +            pass
>>> +
>>> +    params = {'listen_host': '127.0.0.1',
>>> +              'listen_port': config.get('server', 'websockets_port'),
>>> +              'ssl_only': False}
>>> +
>>> +    # old websockify: do not use TokenFile
>>> +    if not tokenFile:
>>> +        params['target_cfg'] = WS_TOKENS_DIR
>>> +
>>> +    # websockify 0.7 and higher: use TokenFile
>>> +    else:
>>> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
>>> +
>>> +    def start_proxy():
>>> +        try:
>>> +            server = WebSocketProxy(RequestHandlerClass=CustomHandler,
>>> +                                    **params)
>>> +        except TypeError:
>>> +            server = CustomHandler(**params)
>>> +
>>> +        server.start_server()
>>> +
>>> +    proc = Process(target=start_proxy)
>>> +    proc.start()
>>> +    return proc
>>> +
>>> +
>>> +def add_proxy_token(name, port, is_unix_socket=False):
>>> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
>>> +        """
>>> +        From python documentation base64.urlsafe_b64encode(s)
>>> +        substitutes - instead of + and _ instead of / in the
>>> +        standard Base64 alphabet, BUT the result can still
>>> +        contain = which is not safe in a URL query component.
>>> +        So remove it when needed as base64 can work well without it.
>>> +        """
>>> +        name = base64.urlsafe_b64encode(name).rstrip('=')
>>> +        if is_unix_socket:
>>> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), 
>>> port))
>>> +        else:
>>> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
>>> +
>>> +
>>> +def remove_proxy_token(name):
>>> +    try:
>>> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
>>> +    except OSError:
>>> +        pass
>>> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
>>> index 20c017e..f5031ce 100644
>>> --- a/ui/js/src/wok.main.js
>>> +++ b/ui/js/src/wok.main.js
>>> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>>>       wok.config = {};
>>>   });
>>>
>>> +
>>> +wok.notificationListeners = {};
>>> +wok.addNotificationListener = function(name, func) {
>>> +    wok.notificationListeners[name] = func;
>>> +    $(window).one("hashchange", function() {
>>> +        delete wok.notificationListeners[name];
>>> +    });
>>> +};
>>> +
>>> +wok.notificationsWebSocket = undefined;
>>> +wok.startNotificationWebSocket = function () {
>>> +    var addr = window.location.hostname + ':' + window.location.port;
>>> +    var token = 
>>> wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>>> +    var url = 'wss://' + addr + '/websockify?token=' + token;
>>> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
>>> +
>>> +    wok.notificationsWebSocket.onmessage = function(event) {
>>> +        var message = window.atob(event.data);
>>> +        for (name in wok.notificationListeners) {
>>> +            func = wok.notificationListeners[name];
>>> +            func(message);
>>> +        }
>>> +    };
>>> +
>>> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
>>> +    var heartbeat = setInterval(function() {
>>> + wok.notificationsWebSocket.send(window.btoa('heartbeat'));
>>> +    }, 30000);
>>> +
>>> +    wok.notificationsWebSocket.onclose = function() {
>>> +        clearInterval(heartbeat);
>>> +    };
>>> +};
>>> +
>>> +
>>>   wok.main = function() {
>>>       wok.isLoggingOut = false;
>>>       wok.popable();
>>> @@ -395,6 +430,9 @@ wok.main = function() {
>>>
>>>           // Set handler for help button
>>>           $('#btn-help').on('click', wok.openHelp);
>>> +
>>> +        // start WebSocket
>>> +        wok.startNotificationWebSocket();
>>>       };
>>>
>>>       var initUI = function() {
>>> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
>>> index 0e8fb09..083b6c3 100644
>>> --- a/ui/js/wok.user-log.js
>>> +++ b/ui/js/wok.user-log.js
>>> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>>>           $("#user-log-grid").bootgrid("search");
>>>           wok.initUserLogConfigGridData();
>>>       });
>>> +
>>
>>> + wok.addNotificationListener('userlog', function(message) {
>>> +        if (message === 'new_log_entry') {
>>> +            $("#refresh-button").click();
>>> +        }
>>> +    });
>>
>> Backend is sending a notification with 'new_log_entry' and you are 
>> registering a notification for 'userlog'.
>> How is it working?
>
> 'userlog' is the identifier of the listener that will be used in the hash
> wok.notificationListeners.
>

Where does the backend make reference to it? On backend, I am only 
seeing the message as 'new_log_entry'. How does the UI assumes 'userlog' 
as the identifier?

>>
>> Shouldn't the 'name' required by addNotificationListener be 
>> 'new_log_entry'? So wok would be redirect to the function automatically?
>> Why do you need to distinguish the 'new_log_entry' later?
>>
>> Also, we will need to create a pattern message for notification to 
>> avoid having a plugin overwriting a notification from another plugin.
>
> Agree. I was aware that this 'new_log_entry' was going to be changed by
> something more formal.
>
>>
>> That said, my suggestion is: <plugin>_<resource>_<action> which will 
>> result in something like:
>>
>> wok_log_create
>> kimchi_vms_create
>> kimchi_vm_update
>> kimchi_vm_start
>> ginger_backups_create
>> ginger_backup_restore
>>
>> and so on.
>>
>> That way, we can automatically add the notification on 
>> control/base.py which will work for everyone. And then, the plugin 
>> just need to register the listener on UI.
>> Something like:
>>
>> @model/base.py
>> #Collection.index and Resource.index (need to cover both)
>> def index(...):
>> (...)
>>     # on sucsess
>>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
>> action)
>>     send_websocket_notification(notification)
>>
>> def _generate_action_handler_base(...):
>>     (...)
>>     # on success
>>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
>> action)
>>     send_websocket_notification(notification)
>>
>> What do you think?
>
> I think we can work with this plugin_name/entity/action format, 
> perhaps adding
> an 'args' field to it. 

Which 'args'?

> In the case of this patch we would have 'wok/userlog/create'
> or something similar.
>
> We need to also consider task handling in this context. We can change 
> the default
> success/failure handles to also automatically report the status to the 
> UI via
> websocket.
>

It already exists today for the user log.
So we just need to link the log_request() to the 
send_notification_message() and we are all covered.

> In fact, we need to consider a lot of standardization with this 
> feature. For this patch,
> I we can start by doing the action handler changes you've proposed to 
> implement the
> user log changes. I have no problems putting the notification under 
> the 'index' of
> both Collection and Resource models too.
>
>
> Daniel
>>
>>>   };
>>>
>>>   wok.initUserLogWindow = function() {
>>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago

On 02/28/2017 10:20 AM, Aline Manera wrote:
>
>
> On 02/27/2017 05:58 PM, Daniel Henrique Barboza wrote:
>>
>>
>> On 02/27/2017 11:23 AM, Aline Manera wrote:
>>>
>>>
>>> On 02/24/2017 10:22 AM, dhbarboza82@gmail.com wrote:
>>>> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>>>
>>>> This patch makes backend and UI changes to implement the asynchronous
>>>> UI notification in WoK.
>>>>
>>>> - Backend:
>>>>
>>>> A push server was implemented from scratch to manage the opened 
>>>> websocket
>>>> connections. The push server connects to the /run/woknotifications
>>>> UNIX socket and broadcasts all messages to all connections.
>>>>
>>>> The websocket module is the same module that exists in the Kimchi
>>>> plug-in. The idea is to remove the module from Kimchi and make it
>>>> use the module from WoK. ws_proxy initialization was also added
>>>> in src/wok/server.py.
>>>>
>>>> - Frontend:
>>>>
>>>> In ui/js/wok.main.js two new functions were added to help the
>>>> usage of asynchronous notifications in the frontend. The idea:
>>>> a single websocket is opened per session. This opened websocket
>>>> will broadcast all incoming messages to all listeners registered.
>>>> Listeners can be added by the new wok.addNotificationListener()
>>>> method. This method will clean up any registered listener by
>>>> itself when the user changes tabs/URL.
>>>>
>>>> The single websocket sends heartbeats to the backend side each
>>>> 30 seconds. No reply from the backend is issued or expected. This
>>>> heartbeat is just a way to ensure that the browser does not
>>>> close the connection due to inactivity. This behavior varies from
>>>> browser to browser but this 30 second heartbeat is more than enough
>>>> to ensure that the websocket is kept alive.
>>>>
>>>> - Working example in User Log:
>>>>
>>>> A simple usage is provided in this patch. A change was made in
>>>> src/wok/reqlogger.py to send an asynchronous notification each
>>>> time a new log entry is created. In ui/js/wok.user-log.js a
>>>> websocket listener is added using wok.addNotificationListener()
>>>> and, for each message that indicates a new user log entry, a
>>>> refresh in the listing is issued.
>>>>
>>>> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>>> ---
>>>>   src/wok/model/notifications.py |  18 +++++-
>>>>   src/wok/pushserver.py          | 132 
>>>> +++++++++++++++++++++++++++++++++++++++++
>>>>   src/wok/reqlogger.py           |   7 ++-
>>>>   src/wok/server.py              |   8 ++-
>>>>   src/wok/websocket.py           | 123 
>>>> ++++++++++++++++++++++++++++++++++++++
>>>>   ui/js/src/wok.main.js          |  38 ++++++++++++
>>>>   ui/js/wok.user-log.js          |   6 ++
>>>>   7 files changed, 327 insertions(+), 5 deletions(-)
>>>>   create mode 100644 src/wok/pushserver.py
>>>>   create mode 100644 src/wok/websocket.py
>>>>
>>>> diff --git a/src/wok/model/notifications.py 
>>>> b/src/wok/model/notifications.py
>>>> index bdb7c78..597eac5 100644
>>>> --- a/src/wok/model/notifications.py
>>>> +++ b/src/wok/model/notifications.py
>>>> @@ -1,7 +1,7 @@
>>>>   #
>>>>   # Project Wok
>>>>   #
>>>> -# Copyright IBM Corp, 2016
>>>> +# Copyright IBM Corp, 2016-2017
>>>>   #
>>>>   # This library is free software; you can redistribute it and/or
>>>>   # modify it under the terms of the GNU Lesser General Public
>>>> @@ -19,12 +19,22 @@
>>>>
>>>>   from datetime import datetime
>>>>
>>>> +from wok import config
>>>>   from wok.exception import NotFoundError, OperationFailed
>>>>   from wok.message import WokMessage
>>>> +from wok.pushserver import PushServer
>>>>   from wok.utils import wok_log
>>>>
>>>>
>>>>   notificationsStore = {}
>>>> +push_server = None
>>>> +
>>>> +
>>>> +def send_websocket_notification(message):
>>>> +    global push_server
>>>> +
>>>> +    if push_server:
>>>> +        push_server.send_notification(message)
>>>>
>>>>
>>>>   def add_notification(code, args=None, plugin_name=None):
>>>> @@ -57,7 +67,11 @@ def del_notification(code):
>>>>
>>>>   class NotificationsModel(object):
>>>>       def __init__(self, **kargs):
>>>> -        pass
>>>> +        global push_server
>>>> +
>>>
>>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>>> 'true'
>>>> +        if not test_mode:
>>>> +            push_server = PushServer()
>>>
>>> Why distinguish test mode here? While running on test mode, aka, 
>>> using MockModel, the server should behave the same as on real model.
>>
>> It makes no sense to enable the push server without websocket 
>> support. As
>> it is now, on Kimchi code, the websocket proxy doesn't initialize in
>> test_mode. This same design was kept when moving the initialization
>> to WoK.
>>
>> I've also experienced conflicts when running the unit tests with the 
>> push server.
>> The multiple instancing of test_model led to the multiple instancing of
>> the push_server, causing conflict because all push server instances read
>> from the same unix socket.
>>
>
> Maybe initializing the websockets only on non-test mode on Kimchi 
> makes sense, ie, the libvirt test driver may not be allowed to 
> streaming the console.
> But for a Wok perspective, the websockets is now being used for the 
> whole application, so there is no need to distinguish the test mode.
>
> Thinking about Selenium tests which will run on test mode (aka, using 
> MockModel) we need to have the same UI behavior as running on real 
> model to validate the tests so the async notification should work as 
> well.
>
> Do not mistake tests cases with the wok test mode. The test mode is a 
> dry run environment to user tries Wok without affecting the system 
> (aka, using the MockModel).
> We should not deliver a partial implementation to production mode due 
> the tests cases.
>
> Also, if you move the Push Server initialization to Server (as I 
> commented in another reply), the Push Server will be only initialized 
> when Server is, ie, the test_model* which does not initialize the 
> server will not have the Push Server and maybe the problems you saw 
> will go away.
>
>>>
>>>>
>>>>       def get_list(self):
>>>>           global notificationsStore
>>>> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
>>>> new file mode 100644
>>>> index 0000000..8993f00
>>>> --- /dev/null
>>>> +++ b/src/wok/pushserver.py
>>>> @@ -0,0 +1,132 @@
>>>> +#
>>>> +# Project Wok
>>>> +#
>>>> +# Copyright IBM Corp, 2017
>>>> +#
>>>> +# This library is free software; you can redistribute it and/or
>>>> +# modify it under the terms of the GNU Lesser General Public
>>>> +# License as published by the Free Software Foundation; either
>>>> +# version 2.1 of the License, or (at your option) any later version.
>>>> +#
>>>> +# This library is distributed in the hope that it will be useful,
>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>>> +# Lesser General Public License for more details.
>>>> +#
>>>> +# You should have received a copy of the GNU Lesser General Public
>>>> +# License along with this library; if not, write to the Free Software
>>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>>> 02110-1301 USA
>>>> +#
>>>> +
>>>> +import cherrypy
>>>> +import os
>>>> +import select
>>>> +import socket
>>>> +import threading
>>>> +
>>>
>>>> +import websocket
>>>> +from utils import wok_log
>>>> +
>>>
>>> Use full import path.
>>>
>>> import wok.websocket
>>> from wok.utils import wok_log
>>
>> ok
>>>
>>>> +
>>>> +BASE_DIRECTORY = '/run'
>>>
>>> I think this directory depends on distro settings. Some ones uses 
>>> /var/run
>>> You will need to get it on build settings. Check config.py.in for 
>>> details.
>>
>> I've copied this line/logic from Kimchi's serialconsole.py assuming 
>> it was a
>> good way to do it. I'll see if using config.py.in in this case is 
>> possible.
>>
>> We can also discuss if putting it under /run is the choice and 
>> perhaps choose
>> another path entirely.
>>
>
> OK.
>
>>>
>>>> +TOKEN_NAME = 'woknotifications'
>>>> +
>>>
>>> Maybe only 'notifications' under a wok directory is better.
>>> Just to place all the wok files related to the same directory. I am 
>>> not sure if others files is places on /run but i think there are.
>>
>> As I've said above, I'm open to suggestions of where to put this unix 
>> socket.
>>
>>>
>>>> +
>>>> +class PushServer(object):
>>>> +
>>>> +    def set_socket_file(self):
>>>> +        if not os.path.isdir(BASE_DIRECTORY):
>>>> +            try:
>>>> +                os.mkdir(BASE_DIRECTORY)
>>>> +            except OSError:
>>>> +                raise RuntimeError('PushServer base UNIX socket 
>>>> dir %s '
>>>> +                                   'not found.' % BASE_DIRECTORY)
>>>> +
>>>> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
>>>> +
>>>> +        if os.path.exists(self.server_addr):
>>>> +            try:
>>>> +                os.remove(self.server_addr)
>>>> +            except:
>>>> +                raise RuntimeError('There is an existing 
>>>> connection in %s' %
>>>> +                                   self.server_addr)
>>>> +
>>>> +    def __init__(self):
>>>> +        self.set_socket_file()
>>>> +
>>>> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
>>>> +
>>>> +        self.connections = []
>>>> +
>>>> +        self.server_running = True
>>>> +        self.server_socket = socket.socket(socket.AF_UNIX, 
>>>> socket.SOCK_STREAM)
>>>> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
>>>> +                                      socket.SO_REUSEADDR, 1)
>>>> +        self.server_socket.bind(self.server_addr)
>>>> +        self.server_socket.listen(10)
>>>> +        wok_log.info('Push server created on address %s' % 
>>>> self.server_addr)
>>>> +
>>>> +        self.connections.append(self.server_socket)
>>>> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
>>>> +
>>>> +        server_loop = threading.Thread(target=self.listen)
>>>> +        server_loop.start()
>>>> +
>>>> +    def listen(self):
>>>> +        try:
>>>> +            while self.server_running:
>>>> +                read_ready, _, _ = select.select(self.connections,
>>>> +                                                 [], [], 1)
>>>> +                for sock in read_ready:
>>>> +                    if not self.server_running:
>>>> +                        break
>>>> +
>>>> +                    if sock == self.server_socket:
>>>> +
>>>> +                        new_socket, addr = 
>>>> self.server_socket.accept()
>>>> +                        self.connections.append(new_socket)
>>>> +                    else:
>>>> +                        try:
>>>> +                            data = sock.recv(4096)
>>>> +                        except:
>>>> +                            try:
>>>> + self.connections.remove(sock)
>>>> +                            except ValueError:
>>>> +                                pass
>>>> +
>>>> +                            continue
>>>> +                        if data and data == 'CLOSE':
>>>> +                            sock.send('ACK')
>>>> +                            try:
>>>> + self.connections.remove(sock)
>>>> +                            except ValueError:
>>>> +                                pass
>>>> +                            sock.close()
>>>> +
>>>> +        except Exception as e:
>>>> +            raise RuntimeError('Exception ocurred in listen() of 
>>>> pushserver '
>>>> +                               'module: %s' % e.message)
>>>> +
>>>> +    def send_notification(self, message):
>>>> +        for sock in self.connections:
>>>> +            if sock != self.server_socket:
>>>> +                try:
>>>> +                    sock.send(message)
>>>> +                except IOError as e:
>>>> +                    if 'Broken pipe' in str(e):
>>>> +                        sock.close()
>>>> +                        try:
>>>> +                            self.connections.remove(sock)
>>>> +                        except ValueError:
>>>> +                            pass
>>>> +
>>>> +    def close_server(self):
>>>> +        try:
>>>> +            self.server_running = False
>>>> +            self.server_socket.shutdown(socket.SHUT_RDWR)
>>>> +            self.server_socket.close()
>>>> +            os.remove(self.server_addr)
>>>> +        except:
>>>> +            pass
>>>> +        finally:
>>>> +            cherrypy.engine.unsubscribe('stop', self.close_server)
>>>> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
>>>> index 92e155d..1b774e2 100644
>>>> --- a/src/wok/reqlogger.py
>>>> +++ b/src/wok/reqlogger.py
>>>> @@ -1,7 +1,7 @@
>>>>   #
>>>>   # Project Wok
>>>>   #
>>>> -# Copyright IBM Corp, 2016
>>>> +# Copyright IBM Corp, 2016-2017
>>>>   #
>>>>   # This library is free software; you can redistribute it and/or
>>>>   # modify it under the terms of the GNU Lesser General Public
>>>> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>>>>   from wok.config import get_log_download_path, paths
>>>>   from wok.exception import InvalidParameter, OperationFailed
>>>>   from wok.message import WokMessage
>>>> +from wok.model.notifications import send_websocket_notification
>>>>   from wok.stringutils import ascii_dict
>>>>   from wok.utils import remove_old_files
>>>>
>>>> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>>>>   # AsyncTask handling
>>>>   ASYNCTASK_REQUEST_METHOD = 'TASK'
>>>>
>>>> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
>>>> +
>>>>
>>>>   def log_request(code, params, exception, method, status, 
>>>> app=None, user=None,
>>>>                   ip=None):
>>>> @@ -114,6 +117,8 @@ def log_request(code, params, exception, 
>>>> method, status, app=None, user=None,
>>>>           ip=ip
>>>>       ).log()
>>>>
>>>> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
>>>> +
>>>>       return log_id
>>>>
>>>>
>>>> diff --git a/src/wok/server.py b/src/wok/server.py
>>>> index fc2e167..2d823c9 100644
>>>> --- a/src/wok/server.py
>>>> +++ b/src/wok/server.py
>>>> @@ -25,8 +25,7 @@ import logging
>>>>   import logging.handlers
>>>>   import os
>>>>
>>>> -from wok import auth
>>>> -from wok import config
>>>> +from wok import auth, config, websocket
>>>>   from wok.config import config as configParser
>>>>   from wok.config import WokConfig
>>>>   from wok.control import sub_nodes
>>>> @@ -159,6 +158,11 @@ class Server(object):
>>>>           cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>>>>                               options.server_root, self.configObj)
>>>
>>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>>> 'true'
>>>> +        if not test_mode:
>>>> +            ws_proxy = websocket.new_ws_proxy()
>>>> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
>>>> +
>>>
>>> The same question I did before.
>>
>> See answer above.
>>
>>>
>>>>           self._load_plugins()
>>>>           cherrypy.lib.sessions.init()
>>>>
>>>> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
>>>> new file mode 100644
>>>> index 0000000..5d7fb91
>>>> --- /dev/null
>>>> +++ b/src/wok/websocket.py
>>>> @@ -0,0 +1,123 @@
>>>> +#!/usr/bin/env python2
>>>> +#
>>>> +# Project Wok
>>>> +#
>>>> +# Copyright IBM Corp, 2017
>>>> +#
>>>> +# Code derived from Project Kimchi
>>>> +#
>>>> +# This library is free software; you can redistribute it and/or
>>>> +# modify it under the terms of the GNU Lesser General Public
>>>> +# License as published by the Free Software Foundation; either
>>>> +# version 2.1 of the License, or (at your option) any later version.
>>>> +#
>>>> +# This library is distributed in the hope that it will be useful,
>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>>> +# Lesser General Public License for more details.
>>>> +#
>>>> +# You should have received a copy of the GNU Lesser General Public
>>>> +# License along with this library; if not, write to the Free Software
>>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>>> 02110-1301 USA
>>>> +
>>>> +import base64
>>>> +import errno
>>>> +import os
>>>> +
>>>> +from multiprocessing import Process
>>>> +from websockify import WebSocketProxy
>>>> +
>>>> +from config import config, PluginPaths
>>>> +
>>>> +
>>>> +try:
>>>> +    from websockify.token_plugins import TokenFile
>>>> +    tokenFile = True
>>>> +except ImportError:
>>>> +    tokenFile = False
>>>> +
>>>> +try:
>>>> +    from websockify import ProxyRequestHandler as request_proxy
>>>> +except:
>>>> +    from websockify import WebSocketProxy as request_proxy
>>>> +
>>>> +
>>>> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 
>>>> 'ws-tokens')
>>>> +
>>>> +
>>>> +class CustomHandler(request_proxy):
>>>> +
>>>> +    def get_target(self, target_plugin, path):
>>>> +        if issubclass(CustomHandler, object):
>>>> +            target = super(CustomHandler, 
>>>> self).get_target(target_plugin,
>>>> + path)
>>>> +        else:
>>>> +            target = request_proxy.get_target(self, target_plugin, 
>>>> path)
>>>> +
>>>> +        if target[0] == 'unix_socket':
>>>> +            try:
>>>> +                self.server.unix_target = target[1]
>>>> +            except:
>>>> +                self.unix_target = target[1]
>>>> +        else:
>>>> +            try:
>>>> +                self.server.unix_target = None
>>>> +            except:
>>>> +                self.unix_target = None
>>>> +        return target
>>>> +
>>>> +
>>>> +def new_ws_proxy():
>>>> +    try:
>>>> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
>>>> +    except OSError as e:
>>>> +        if e.errno == errno.EEXIST:
>>>> +            pass
>>>> +
>>>> +    params = {'listen_host': '127.0.0.1',
>>>> +              'listen_port': config.get('server', 'websockets_port'),
>>>> +              'ssl_only': False}
>>>> +
>>>> +    # old websockify: do not use TokenFile
>>>> +    if not tokenFile:
>>>> +        params['target_cfg'] = WS_TOKENS_DIR
>>>> +
>>>> +    # websockify 0.7 and higher: use TokenFile
>>>> +    else:
>>>> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
>>>> +
>>>> +    def start_proxy():
>>>> +        try:
>>>> +            server = 
>>>> WebSocketProxy(RequestHandlerClass=CustomHandler,
>>>> +                                    **params)
>>>> +        except TypeError:
>>>> +            server = CustomHandler(**params)
>>>> +
>>>> +        server.start_server()
>>>> +
>>>> +    proc = Process(target=start_proxy)
>>>> +    proc.start()
>>>> +    return proc
>>>> +
>>>> +
>>>> +def add_proxy_token(name, port, is_unix_socket=False):
>>>> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
>>>> +        """
>>>> +        From python documentation base64.urlsafe_b64encode(s)
>>>> +        substitutes - instead of + and _ instead of / in the
>>>> +        standard Base64 alphabet, BUT the result can still
>>>> +        contain = which is not safe in a URL query component.
>>>> +        So remove it when needed as base64 can work well without it.
>>>> +        """
>>>> +        name = base64.urlsafe_b64encode(name).rstrip('=')
>>>> +        if is_unix_socket:
>>>> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), 
>>>> port))
>>>> +        else:
>>>> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), 
>>>> port))
>>>> +
>>>> +
>>>> +def remove_proxy_token(name):
>>>> +    try:
>>>> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
>>>> +    except OSError:
>>>> +        pass
>>>> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
>>>> index 20c017e..f5031ce 100644
>>>> --- a/ui/js/src/wok.main.js
>>>> +++ b/ui/js/src/wok.main.js
>>>> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>>>>       wok.config = {};
>>>>   });
>>>>
>>>> +
>>>> +wok.notificationListeners = {};
>>>> +wok.addNotificationListener = function(name, func) {
>>>> +    wok.notificationListeners[name] = func;
>>>> +    $(window).one("hashchange", function() {
>>>> +        delete wok.notificationListeners[name];
>>>> +    });
>>>> +};
>>>> +
>>>> +wok.notificationsWebSocket = undefined;
>>>> +wok.startNotificationWebSocket = function () {
>>>> +    var addr = window.location.hostname + ':' + window.location.port;
>>>> +    var token = 
>>>> wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>>>> +    var url = 'wss://' + addr + '/websockify?token=' + token;
>>>> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
>>>> +
>>>> +    wok.notificationsWebSocket.onmessage = function(event) {
>>>> +        var message = window.atob(event.data);
>>>> +        for (name in wok.notificationListeners) {
>>>> +            func = wok.notificationListeners[name];
>>>> +            func(message);
>>>> +        }
>>>> +    };
>>>> +
>>>> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
>>>> +    var heartbeat = setInterval(function() {
>>>> + wok.notificationsWebSocket.send(window.btoa('heartbeat'));
>>>> +    }, 30000);
>>>> +
>>>> +    wok.notificationsWebSocket.onclose = function() {
>>>> +        clearInterval(heartbeat);
>>>> +    };
>>>> +};
>>>> +
>>>> +
>>>>   wok.main = function() {
>>>>       wok.isLoggingOut = false;
>>>>       wok.popable();
>>>> @@ -395,6 +430,9 @@ wok.main = function() {
>>>>
>>>>           // Set handler for help button
>>>>           $('#btn-help').on('click', wok.openHelp);
>>>> +
>>>> +        // start WebSocket
>>>> +        wok.startNotificationWebSocket();
>>>>       };
>>>>
>>>>       var initUI = function() {
>>>> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
>>>> index 0e8fb09..083b6c3 100644
>>>> --- a/ui/js/wok.user-log.js
>>>> +++ b/ui/js/wok.user-log.js
>>>> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>>>>           $("#user-log-grid").bootgrid("search");
>>>>           wok.initUserLogConfigGridData();
>>>>       });
>>>> +
>>>
>>>> + wok.addNotificationListener('userlog', function(message) {
>>>> +        if (message === 'new_log_entry') {
>>>> +            $("#refresh-button").click();
>>>> +        }
>>>> +    });
>>>
>>> Backend is sending a notification with 'new_log_entry' and you are 
>>> registering a notification for 'userlog'.
>>> How is it working?
>>
>> 'userlog' is the identifier of the listener that will be used in the 
>> hash
>> wok.notificationListeners.
>>
>
> Where does the backend make reference to it? On backend, I am only 
> seeing the message as 'new_log_entry'. How does the UI assumes 
> 'userlog' as the identifier?

There is no connection to the backend. This is simply an listener 
identifier of the
tab "user log". It is an UI information only.

The backend can send the information in the format we've discussed. the 
UI can
name its event listener freely. Or course we can standardize the name. 
Perhaps
using the tab_id or something.

Do you want me to split this contribution in two patches, one for 
backend and
other for frontend, to make it clearer what is backend or frontend only 
code?



>
>>>
>>> Shouldn't the 'name' required by addNotificationListener be 
>>> 'new_log_entry'? So wok would be redirect to the function 
>>> automatically?
>>> Why do you need to distinguish the 'new_log_entry' later?
>>>
>>> Also, we will need to create a pattern message for notification to 
>>> avoid having a plugin overwriting a notification from another plugin.
>>
>> Agree. I was aware that this 'new_log_entry' was going to be changed by
>> something more formal.
>>
>>>
>>> That said, my suggestion is: <plugin>_<resource>_<action> which will 
>>> result in something like:
>>>
>>> wok_log_create
>>> kimchi_vms_create
>>> kimchi_vm_update
>>> kimchi_vm_start
>>> ginger_backups_create
>>> ginger_backup_restore
>>>
>>> and so on.
>>>
>>> That way, we can automatically add the notification on 
>>> control/base.py which will work for everyone. And then, the plugin 
>>> just need to register the listener on UI.
>>> Something like:
>>>
>>> @model/base.py
>>> #Collection.index and Resource.index (need to cover both)
>>> def index(...):
>>> (...)
>>>     # on sucsess
>>>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
>>> action)
>>>     send_websocket_notification(notification)
>>>
>>> def _generate_action_handler_base(...):
>>>     (...)
>>>     # on success
>>>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
>>> action)
>>>     send_websocket_notification(notification)
>>>
>>> What do you think?
>>
>> I think we can work with this plugin_name/entity/action format, 
>> perhaps adding
>> an 'args' field to it. 
>
> Which 'args'?

For example, /ginger/Users/create. Which user was created? This is 
something we would
like to inform too.

>
>> In the case of this patch we would have 'wok/userlog/create'
>> or something similar.
>>
>> We need to also consider task handling in this context. We can change 
>> the default
>> success/failure handles to also automatically report the status to 
>> the UI via
>> websocket.
>>
>
> It already exists today for the user log.
> So we just need to link the log_request() to the 
> send_notification_message() and we are all covered.
>
>> In fact, we need to consider a lot of standardization with this 
>> feature. For this patch,
>> I we can start by doing the action handler changes you've proposed to 
>> implement the
>> user log changes. I have no problems putting the notification under 
>> the 'index' of
>> both Collection and Resource models too.
>>
>>
>> Daniel
>>>
>>>>   };
>>>>
>>>>   wok.initUserLogWindow = function() {
>>>
>>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago
>>>>>       var initUI = function() {
>>>>> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
>>>>> index 0e8fb09..083b6c3 100644
>>>>> --- a/ui/js/wok.user-log.js
>>>>> +++ b/ui/js/wok.user-log.js
>>>>> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>>>>>           $("#user-log-grid").bootgrid("search");
>>>>>           wok.initUserLogConfigGridData();
>>>>>       });
>>>>> +
>>>>
>>>>> + wok.addNotificationListener('userlog', function(message) {
>>>>> +        if (message === 'new_log_entry') {
>>>>> +            $("#refresh-button").click();
>>>>> +        }
>>>>> +    });
>>>>
>>>> Backend is sending a notification with 'new_log_entry' and you are 
>>>> registering a notification for 'userlog'.
>>>> How is it working?
>>>
>>> 'userlog' is the identifier of the listener that will be used in the 
>>> hash
>>> wok.notificationListeners.
>>>
>>
>> Where does the backend make reference to it? On backend, I am only 
>> seeing the message as 'new_log_entry'. How does the UI assumes 
>> 'userlog' as the identifier?
>
> There is no connection to the backend. This is simply an listener 
> identifier of the
> tab "user log". It is an UI information only.
>
> The backend can send the information in the format we've discussed. 
> the UI can
> name its event listener freely. Or course we can standardize the name. 
> Perhaps
> using the tab_id or something.
>

Why do we need a second identifier? Why isn't the message id 
(new_log_entry) enough?

> Do you want me to split this contribution in two patches, one for 
> backend and
> other for frontend, to make it clearer what is backend or frontend 
> only code?
>
>

It is up to you. I am only trying to understand the whole flow.

>
>>
>>>>
>>>> Shouldn't the 'name' required by addNotificationListener be 
>>>> 'new_log_entry'? So wok would be redirect to the function 
>>>> automatically?
>>>> Why do you need to distinguish the 'new_log_entry' later?
>>>>
>>>> Also, we will need to create a pattern message for notification to 
>>>> avoid having a plugin overwriting a notification from another plugin.
>>>
>>> Agree. I was aware that this 'new_log_entry' was going to be changed by
>>> something more formal.
>>>
>>>>
>>>> That said, my suggestion is: <plugin>_<resource>_<action> which 
>>>> will result in something like:
>>>>
>>>> wok_log_create
>>>> kimchi_vms_create
>>>> kimchi_vm_update
>>>> kimchi_vm_start
>>>> ginger_backups_create
>>>> ginger_backup_restore
>>>>
>>>> and so on.
>>>>
>>>> That way, we can automatically add the notification on 
>>>> control/base.py which will work for everyone. And then, the plugin 
>>>> just need to register the listener on UI.
>>>> Something like:
>>>>
>>>> @model/base.py
>>>> #Collection.index and Resource.index (need to cover both)
>>>> def index(...):
>>>> (...)
>>>>     # on sucsess
>>>>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
>>>> action)
>>>>     send_websocket_notification(notification)
>>>>
>>>> def _generate_action_handler_base(...):
>>>>     (...)
>>>>     # on success
>>>>     notification = "%s_%s_%s" % (plugin_name, resource/collection, 
>>>> action)
>>>>     send_websocket_notification(notification)
>>>>
>>>> What do you think?
>>>
>>> I think we can work with this plugin_name/entity/action format, 
>>> perhaps adding
>>> an 'args' field to it. 
>>
>> Which 'args'?
>
> For example, /ginger/Users/create. Which user was created? This is 
> something we would
> like to inform too.
>

Really? Why?

On an notification received the UI will refresh the whole page. I don't 
think knowing the user will make difference.

>>
>>> In the case of this patch we would have 'wok/userlog/create'
>>> or something similar.
>>>
>>> We need to also consider task handling in this context. We can 
>>> change the default
>>> success/failure handles to also automatically report the status to 
>>> the UI via
>>> websocket.
>>>
>>
>> It already exists today for the user log.
>> So we just need to link the log_request() to the 
>> send_notification_message() and we are all covered.
>>
>>> In fact, we need to consider a lot of standardization with this 
>>> feature. For this patch,
>>> I we can start by doing the action handler changes you've proposed 
>>> to implement the
>>> user log changes. I have no problems putting the notification under 
>>> the 'index' of
>>> both Collection and Resource models too.
>>>
>>>
>>> Daniel
>>>>
>>>>>   };
>>>>>
>>>>>   wok.initUserLogWindow = function() {
>>>>
>>>
>>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Lucio Correia 4 years, 2 months ago
Hi Daniel, that is great feature, see my comments below.

On 24/02/2017 10:22, dhbarboza82@gmail.com wrote:
> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>
> This patch makes backend and UI changes to implement the asynchronous
> UI notification in WoK.
>
> - Backend:
>
> A push server was implemented from scratch to manage the opened websocket
> connections. The push server connects to the /run/woknotifications
> UNIX socket and broadcasts all messages to all connections.
>
> The websocket module is the same module that exists in the Kimchi
> plug-in. The idea is to remove the module from Kimchi and make it
> use the module from WoK. ws_proxy initialization was also added
> in src/wok/server.py.
>
> - Frontend:
>
> In ui/js/wok.main.js two new functions were added to help the
> usage of asynchronous notifications in the frontend. The idea:
> a single websocket is opened per session. This opened websocket
> will broadcast all incoming messages to all listeners registered.
> Listeners can be added by the new wok.addNotificationListener()
> method. This method will clean up any registered listener by
> itself when the user changes tabs/URL.
>
> The single websocket sends heartbeats to the backend side each
> 30 seconds. No reply from the backend is issued or expected. This
> heartbeat is just a way to ensure that the browser does not
> close the connection due to inactivity. This behavior varies from
> browser to browser but this 30 second heartbeat is more than enough
> to ensure that the websocket is kept alive.
>
> - Working example in User Log:
>
> A simple usage is provided in this patch. A change was made in
> src/wok/reqlogger.py to send an asynchronous notification each
> time a new log entry is created. In ui/js/wok.user-log.js a
> websocket listener is added using wok.addNotificationListener()
> and, for each message that indicates a new user log entry, a
> refresh in the listing is issued.
>
> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
> ---
>  src/wok/model/notifications.py |  18 +++++-
>  src/wok/pushserver.py          | 132 +++++++++++++++++++++++++++++++++++++++++
>  src/wok/reqlogger.py           |   7 ++-
>  src/wok/server.py              |   8 ++-
>  src/wok/websocket.py           | 123 ++++++++++++++++++++++++++++++++++++++
>  ui/js/src/wok.main.js          |  38 ++++++++++++
>  ui/js/wok.user-log.js          |   6 ++
>  7 files changed, 327 insertions(+), 5 deletions(-)
>  create mode 100644 src/wok/pushserver.py
>  create mode 100644 src/wok/websocket.py
>
> diff --git a/src/wok/model/notifications.py b/src/wok/model/notifications.py
> index bdb7c78..597eac5 100644
> --- a/src/wok/model/notifications.py
> +++ b/src/wok/model/notifications.py
> @@ -1,7 +1,7 @@
>  #
>  # Project Wok
>  #
> -# Copyright IBM Corp, 2016
> +# Copyright IBM Corp, 2016-2017
>  #
>  # This library is free software; you can redistribute it and/or
>  # modify it under the terms of the GNU Lesser General Public
> @@ -19,12 +19,22 @@
>
>  from datetime import datetime
>
> +from wok import config
>  from wok.exception import NotFoundError, OperationFailed
>  from wok.message import WokMessage
> +from wok.pushserver import PushServer
>  from wok.utils import wok_log
>
>
>  notificationsStore = {}
> +push_server = None
> +
> +
> +def send_websocket_notification(message):
> +    global push_server
> +
> +    if push_server:
> +        push_server.send_notification(message)
>
>
>  def add_notification(code, args=None, plugin_name=None):
> @@ -57,7 +67,11 @@ def del_notification(code):
>
>  class NotificationsModel(object):
>      def __init__(self, **kargs):
> -        pass
> +        global push_server
> +
> +        test_mode = config.config.get('server', 'test').lower() == 'true'

This will cause some tests to initialize pushserver, since 'test' option 
is now tied to which model runs (mockmodel or model). Problem is that 
now wok tests run without sudo (patch in ML), and pushserver's 
BASE_DIRECTORY is only writable with sudo permission. Gave a suggestion 
there.


> +        if not test_mode:
> +            push_server = PushServer()
All 'users' of functionality will have their own instance of pushserver 
(each with a "while True" running)? Why not a single instance used by 
everybody?


>
>      def get_list(self):
>          global notificationsStore
> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
> new file mode 100644
> index 0000000..8993f00
> --- /dev/null
> +++ b/src/wok/pushserver.py
> @@ -0,0 +1,132 @@
> +#
> +# Project Wok
> +#
> +# Copyright IBM Corp, 2017
> +#
> +# This library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +#
> +# This library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +#
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with this library; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
> +#
> +
> +import cherrypy
> +import os
> +import select
> +import socket
> +import threading
> +
> +import websocket
> +from utils import wok_log
> +
> +
> +BASE_DIRECTORY = '/run'
Suggestion to use:
os.path.join('/run/user', str(os.getuid()))
in order tests may be run with root.

> +TOKEN_NAME = 'woknotifications'
> +
> +
> +class PushServer(object):
> +
> +    def set_socket_file(self):
> +        if not os.path.isdir(BASE_DIRECTORY):
> +            try:
> +                os.mkdir(BASE_DIRECTORY)
> +            except OSError:
> +                raise RuntimeError('PushServer base UNIX socket dir %s '
> +                                   'not found.' % BASE_DIRECTORY)
> +
> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
> +
> +        if os.path.exists(self.server_addr):
> +            try:
> +                os.remove(self.server_addr)
> +            except:
> +                raise RuntimeError('There is an existing connection in %s' %
> +                                   self.server_addr)
> +
> +    def __init__(self):
> +        self.set_socket_file()
> +
> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
> +
> +        self.connections = []
> +
> +        self.server_running = True
> +        self.server_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
> +                                      socket.SO_REUSEADDR, 1)
> +        self.server_socket.bind(self.server_addr)
> +        self.server_socket.listen(10)
> +        wok_log.info('Push server created on address %s' % self.server_addr)
> +
> +        self.connections.append(self.server_socket)
> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
> +
> +        server_loop = threading.Thread(target=self.listen)
> +        server_loop.start()
> +
> +    def listen(self):
> +        try:
> +            while self.server_running:
> +                read_ready, _, _ = select.select(self.connections,
> +                                                 [], [], 1)
> +                for sock in read_ready:
> +                    if not self.server_running:
> +                        break
> +
> +                    if sock == self.server_socket:
> +
> +                        new_socket, addr = self.server_socket.accept()
> +                        self.connections.append(new_socket)
> +                    else:
> +                        try:
> +                            data = sock.recv(4096)
> +                        except:
> +                            try:
> +                                self.connections.remove(sock)
> +                            except ValueError:
> +                                pass
> +
> +                            continue
> +                        if data and data == 'CLOSE':
> +                            sock.send('ACK')
> +                            try:
> +                                self.connections.remove(sock)
> +                            except ValueError:
> +                                pass
> +                            sock.close()
> +
> +        except Exception as e:
> +            raise RuntimeError('Exception ocurred in listen() of pushserver '
> +                               'module: %s' % e.message)
> +
> +    def send_notification(self, message):
> +        for sock in self.connections:
> +            if sock != self.server_socket:
> +                try:
> +                    sock.send(message)
> +                except IOError as e:
> +                    if 'Broken pipe' in str(e):
> +                        sock.close()
> +                        try:
> +                            self.connections.remove(sock)
> +                        except ValueError:
> +                            pass
> +
> +    def close_server(self):
> +        try:
> +            self.server_running = False
> +            self.server_socket.shutdown(socket.SHUT_RDWR)
> +            self.server_socket.close()
> +            os.remove(self.server_addr)
> +        except:
> +            pass
> +        finally:
> +            cherrypy.engine.unsubscribe('stop', self.close_server)
> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
> index 92e155d..1b774e2 100644
> --- a/src/wok/reqlogger.py
> +++ b/src/wok/reqlogger.py
> @@ -1,7 +1,7 @@
>  #
>  # Project Wok
>  #
> -# Copyright IBM Corp, 2016
> +# Copyright IBM Corp, 2016-2017
>  #
>  # This library is free software; you can redistribute it and/or
>  # modify it under the terms of the GNU Lesser General Public
> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>  from wok.config import get_log_download_path, paths
>  from wok.exception import InvalidParameter, OperationFailed
>  from wok.message import WokMessage
> +from wok.model.notifications import send_websocket_notification
>  from wok.stringutils import ascii_dict
>  from wok.utils import remove_old_files
>
> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>  # AsyncTask handling
>  ASYNCTASK_REQUEST_METHOD = 'TASK'
>
> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
> +
>
>  def log_request(code, params, exception, method, status, app=None, user=None,
>                  ip=None):
> @@ -114,6 +117,8 @@ def log_request(code, params, exception, method, status, app=None, user=None,
>          ip=ip
>      ).log()
>
> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
> +
>      return log_id
>
>
> diff --git a/src/wok/server.py b/src/wok/server.py
> index fc2e167..2d823c9 100644
> --- a/src/wok/server.py
> +++ b/src/wok/server.py
> @@ -25,8 +25,7 @@ import logging
>  import logging.handlers
>  import os
>
> -from wok import auth
> -from wok import config
> +from wok import auth, config, websocket
>  from wok.config import config as configParser
>  from wok.config import WokConfig
>  from wok.control import sub_nodes
> @@ -159,6 +158,11 @@ class Server(object):
>          cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>                              options.server_root, self.configObj)
>
> +        test_mode = config.config.get('server', 'test').lower() == 'true'
> +        if not test_mode:
> +            ws_proxy = websocket.new_ws_proxy()
> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
> +
>          self._load_plugins()
>          cherrypy.lib.sessions.init()
>
> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
> new file mode 100644
> index 0000000..5d7fb91
> --- /dev/null
> +++ b/src/wok/websocket.py
> @@ -0,0 +1,123 @@
> +#!/usr/bin/env python2
> +#
> +# Project Wok
> +#
> +# Copyright IBM Corp, 2017
> +#
> +# Code derived from Project Kimchi
> +#
> +# This library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +#
> +# This library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +#
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with this library; if not, write to the Free Software
> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
> +
> +import base64
> +import errno
> +import os
> +
> +from multiprocessing import Process
> +from websockify import WebSocketProxy
Add dep on websockify to Wok


> +
> +from config import config, PluginPaths
> +
> +
> +try:
> +    from websockify.token_plugins import TokenFile
> +    tokenFile = True
> +except ImportError:
> +    tokenFile = False
> +
> +try:
> +    from websockify import ProxyRequestHandler as request_proxy
> +except:
> +    from websockify import WebSocketProxy as request_proxy
> +
> +
> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 'ws-tokens')
This should be wok path now, and it would be nice to be inside /run/user 
as commented above.


> +
> +
> +class CustomHandler(request_proxy):
> +
> +    def get_target(self, target_plugin, path):
> +        if issubclass(CustomHandler, object):
> +            target = super(CustomHandler, self).get_target(target_plugin,
> +                                                           path)
> +        else:
> +            target = request_proxy.get_target(self, target_plugin, path)
> +
> +        if target[0] == 'unix_socket':
> +            try:
> +                self.server.unix_target = target[1]
> +            except:
> +                self.unix_target = target[1]
> +        else:
> +            try:
> +                self.server.unix_target = None
> +            except:
> +                self.unix_target = None
> +        return target
> +
> +
> +def new_ws_proxy():
> +    try:
> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
> +    except OSError as e:
> +        if e.errno == errno.EEXIST:
> +            pass
> +
> +    params = {'listen_host': '127.0.0.1',
> +              'listen_port': config.get('server', 'websockets_port'),
> +              'ssl_only': False}
> +
> +    # old websockify: do not use TokenFile
> +    if not tokenFile:
> +        params['target_cfg'] = WS_TOKENS_DIR
> +
> +    # websockify 0.7 and higher: use TokenFile
> +    else:
> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
> +
> +    def start_proxy():
> +        try:
> +            server = WebSocketProxy(RequestHandlerClass=CustomHandler,
> +                                    **params)
> +        except TypeError:
> +            server = CustomHandler(**params)
> +
> +        server.start_server()
> +
> +    proc = Process(target=start_proxy)
> +    proc.start()
> +    return proc
> +
> +
> +def add_proxy_token(name, port, is_unix_socket=False):
> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
> +        """
> +        From python documentation base64.urlsafe_b64encode(s)
> +        substitutes - instead of + and _ instead of / in the
> +        standard Base64 alphabet, BUT the result can still
> +        contain = which is not safe in a URL query component.
> +        So remove it when needed as base64 can work well without it.
> +        """
> +        name = base64.urlsafe_b64encode(name).rstrip('=')
> +        if is_unix_socket:
> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), port))
> +        else:
> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
> +
> +
> +def remove_proxy_token(name):
> +    try:
> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
> +    except OSError:
> +        pass
> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
> index 20c017e..f5031ce 100644
> --- a/ui/js/src/wok.main.js
> +++ b/ui/js/src/wok.main.js
> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>      wok.config = {};
>  });
>
> +
> +wok.notificationListeners = {};
> +wok.addNotificationListener = function(name, func) {
> +    wok.notificationListeners[name] = func;
> +    $(window).one("hashchange", function() {
> +        delete wok.notificationListeners[name];
> +    });
> +};
> +
> +wok.notificationsWebSocket = undefined;
> +wok.startNotificationWebSocket = function () {
> +    var addr = window.location.hostname + ':' + window.location.port;
> +    var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
> +    var url = 'wss://' + addr + '/websockify?token=' + token;
> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
> +
> +    wok.notificationsWebSocket.onmessage = function(event) {
> +        var message = window.atob(event.data);
> +        for (name in wok.notificationListeners) {
> +            func = wok.notificationListeners[name];
> +            func(message);
> +        }
> +    };
> +
> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
> +    var heartbeat = setInterval(function() {
> +        wok.notificationsWebSocket.send(window.btoa('heartbeat'));
> +    }, 30000);
> +
> +    wok.notificationsWebSocket.onclose = function() {
> +        clearInterval(heartbeat);
> +    };
> +};
> +
> +
>  wok.main = function() {
>      wok.isLoggingOut = false;
>      wok.popable();
> @@ -395,6 +430,9 @@ wok.main = function() {
>
>          // Set handler for help button
>          $('#btn-help').on('click', wok.openHelp);
> +
> +        // start WebSocket
> +        wok.startNotificationWebSocket();
>      };
>
>      var initUI = function() {
> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
> index 0e8fb09..083b6c3 100644
> --- a/ui/js/wok.user-log.js
> +++ b/ui/js/wok.user-log.js
> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>          $("#user-log-grid").bootgrid("search");
>          wok.initUserLogConfigGridData();
>      });
> +
> +    wok.addNotificationListener('userlog', function(message) {
> +        if (message === 'new_log_entry') {
> +            $("#refresh-button").click();
> +        }
> +    });
>  };
>
>  wok.initUserLogWindow = function() {
>


-- 
Lucio Correia

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/27/2017 11:35 AM, Lucio Correia wrote:
> Hi Daniel, that is great feature, see my comments below.
>
> On 24/02/2017 10:22, dhbarboza82@gmail.com wrote:
>> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>
>> This patch makes backend and UI changes to implement the asynchronous
>> UI notification in WoK.
>>
>> - Backend:
>>
>> A push server was implemented from scratch to manage the opened 
>> websocket
>> connections. The push server connects to the /run/woknotifications
>> UNIX socket and broadcasts all messages to all connections.
>>
>> The websocket module is the same module that exists in the Kimchi
>> plug-in. The idea is to remove the module from Kimchi and make it
>> use the module from WoK. ws_proxy initialization was also added
>> in src/wok/server.py.
>>
>> - Frontend:
>>
>> In ui/js/wok.main.js two new functions were added to help the
>> usage of asynchronous notifications in the frontend. The idea:
>> a single websocket is opened per session. This opened websocket
>> will broadcast all incoming messages to all listeners registered.
>> Listeners can be added by the new wok.addNotificationListener()
>> method. This method will clean up any registered listener by
>> itself when the user changes tabs/URL.
>>
>> The single websocket sends heartbeats to the backend side each
>> 30 seconds. No reply from the backend is issued or expected. This
>> heartbeat is just a way to ensure that the browser does not
>> close the connection due to inactivity. This behavior varies from
>> browser to browser but this 30 second heartbeat is more than enough
>> to ensure that the websocket is kept alive.
>>
>> - Working example in User Log:
>>
>> A simple usage is provided in this patch. A change was made in
>> src/wok/reqlogger.py to send an asynchronous notification each
>> time a new log entry is created. In ui/js/wok.user-log.js a
>> websocket listener is added using wok.addNotificationListener()
>> and, for each message that indicates a new user log entry, a
>> refresh in the listing is issued.
>>
>> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>> ---
>>  src/wok/model/notifications.py |  18 +++++-
>>  src/wok/pushserver.py          | 132 
>> +++++++++++++++++++++++++++++++++++++++++
>>  src/wok/reqlogger.py           |   7 ++-
>>  src/wok/server.py              |   8 ++-
>>  src/wok/websocket.py           | 123 
>> ++++++++++++++++++++++++++++++++++++++
>>  ui/js/src/wok.main.js          |  38 ++++++++++++
>>  ui/js/wok.user-log.js          |   6 ++
>>  7 files changed, 327 insertions(+), 5 deletions(-)
>>  create mode 100644 src/wok/pushserver.py
>>  create mode 100644 src/wok/websocket.py
>>
>> diff --git a/src/wok/model/notifications.py 
>> b/src/wok/model/notifications.py
>> index bdb7c78..597eac5 100644
>> --- a/src/wok/model/notifications.py
>> +++ b/src/wok/model/notifications.py
>> @@ -1,7 +1,7 @@
>>  #
>>  # Project Wok
>>  #
>> -# Copyright IBM Corp, 2016
>> +# Copyright IBM Corp, 2016-2017
>>  #
>>  # This library is free software; you can redistribute it and/or
>>  # modify it under the terms of the GNU Lesser General Public
>> @@ -19,12 +19,22 @@
>>
>>  from datetime import datetime
>>
>> +from wok import config
>>  from wok.exception import NotFoundError, OperationFailed
>>  from wok.message import WokMessage
>> +from wok.pushserver import PushServer
>>  from wok.utils import wok_log
>>
>>
>>  notificationsStore = {}
>> +push_server = None
>> +
>> +
>> +def send_websocket_notification(message):
>> +    global push_server
>> +
>> +    if push_server:
>> +        push_server.send_notification(message)
>>
>>
>>  def add_notification(code, args=None, plugin_name=None):
>> @@ -57,7 +67,11 @@ def del_notification(code):
>>
>>  class NotificationsModel(object):
>>      def __init__(self, **kargs):
>> -        pass
>> +        global push_server
>> +
>> +        test_mode = config.config.get('server', 'test').lower() == 
>> 'true'
>
> This will cause some tests to initialize pushserver, since 'test' 
> option is now tied to which model runs (mockmodel or model). Problem 
> is that now wok tests run without sudo (patch in ML), and pushserver's 
> BASE_DIRECTORY is only writable with sudo permission. Gave a 
> suggestion there.
>
>
>> +        if not test_mode:
>> +            push_server = PushServer()
> All 'users' of functionality will have their own instance of 
> pushserver (each with a "while True" running)? Why not a single 
> instance used by everybody?
>

Good point! Maybe it is better to move it do Server() to initiate the 
websockets connections to everyone. Also there is no need to distinguish 
test mode.

>
>>
>>      def get_list(self):
>>          global notificationsStore
>> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
>> new file mode 100644
>> index 0000000..8993f00
>> --- /dev/null
>> +++ b/src/wok/pushserver.py
>> @@ -0,0 +1,132 @@
>> +#
>> +# Project Wok
>> +#
>> +# Copyright IBM Corp, 2017
>> +#
>> +# This library is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU Lesser General Public
>> +# License as published by the Free Software Foundation; either
>> +# version 2.1 of the License, or (at your option) any later version.
>> +#
>> +# This library is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> +# Lesser General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU Lesser General Public
>> +# License along with this library; if not, write to the Free Software
>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>> 02110-1301 USA
>> +#
>> +
>> +import cherrypy
>> +import os
>> +import select
>> +import socket
>> +import threading
>> +
>> +import websocket
>> +from utils import wok_log
>> +
>> +
>> +BASE_DIRECTORY = '/run'
> Suggestion to use:
> os.path.join('/run/user', str(os.getuid()))
> in order tests may be run with root.
>
>> +TOKEN_NAME = 'woknotifications'
>> +
>> +
>> +class PushServer(object):
>> +
>> +    def set_socket_file(self):
>> +        if not os.path.isdir(BASE_DIRECTORY):
>> +            try:
>> +                os.mkdir(BASE_DIRECTORY)
>> +            except OSError:
>> +                raise RuntimeError('PushServer base UNIX socket dir 
>> %s '
>> +                                   'not found.' % BASE_DIRECTORY)
>> +
>> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
>> +
>> +        if os.path.exists(self.server_addr):
>> +            try:
>> +                os.remove(self.server_addr)
>> +            except:
>> +                raise RuntimeError('There is an existing connection 
>> in %s' %
>> +                                   self.server_addr)
>> +
>> +    def __init__(self):
>> +        self.set_socket_file()
>> +
>> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
>> +
>> +        self.connections = []
>> +
>> +        self.server_running = True
>> +        self.server_socket = socket.socket(socket.AF_UNIX, 
>> socket.SOCK_STREAM)
>> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
>> +                                      socket.SO_REUSEADDR, 1)
>> +        self.server_socket.bind(self.server_addr)
>> +        self.server_socket.listen(10)
>> +        wok_log.info('Push server created on address %s' % 
>> self.server_addr)
>> +
>> +        self.connections.append(self.server_socket)
>> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
>> +
>> +        server_loop = threading.Thread(target=self.listen)
>> +        server_loop.start()
>> +
>> +    def listen(self):
>> +        try:
>> +            while self.server_running:
>> +                read_ready, _, _ = select.select(self.connections,
>> +                                                 [], [], 1)
>> +                for sock in read_ready:
>> +                    if not self.server_running:
>> +                        break
>> +
>> +                    if sock == self.server_socket:
>> +
>> +                        new_socket, addr = self.server_socket.accept()
>> +                        self.connections.append(new_socket)
>> +                    else:
>> +                        try:
>> +                            data = sock.recv(4096)
>> +                        except:
>> +                            try:
>> +                                self.connections.remove(sock)
>> +                            except ValueError:
>> +                                pass
>> +
>> +                            continue
>> +                        if data and data == 'CLOSE':
>> +                            sock.send('ACK')
>> +                            try:
>> +                                self.connections.remove(sock)
>> +                            except ValueError:
>> +                                pass
>> +                            sock.close()
>> +
>> +        except Exception as e:
>> +            raise RuntimeError('Exception ocurred in listen() of 
>> pushserver '
>> +                               'module: %s' % e.message)
>> +
>> +    def send_notification(self, message):
>> +        for sock in self.connections:
>> +            if sock != self.server_socket:
>> +                try:
>> +                    sock.send(message)
>> +                except IOError as e:
>> +                    if 'Broken pipe' in str(e):
>> +                        sock.close()
>> +                        try:
>> +                            self.connections.remove(sock)
>> +                        except ValueError:
>> +                            pass
>> +
>> +    def close_server(self):
>> +        try:
>> +            self.server_running = False
>> +            self.server_socket.shutdown(socket.SHUT_RDWR)
>> +            self.server_socket.close()
>> +            os.remove(self.server_addr)
>> +        except:
>> +            pass
>> +        finally:
>> +            cherrypy.engine.unsubscribe('stop', self.close_server)
>> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
>> index 92e155d..1b774e2 100644
>> --- a/src/wok/reqlogger.py
>> +++ b/src/wok/reqlogger.py
>> @@ -1,7 +1,7 @@
>>  #
>>  # Project Wok
>>  #
>> -# Copyright IBM Corp, 2016
>> +# Copyright IBM Corp, 2016-2017
>>  #
>>  # This library is free software; you can redistribute it and/or
>>  # modify it under the terms of the GNU Lesser General Public
>> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>>  from wok.config import get_log_download_path, paths
>>  from wok.exception import InvalidParameter, OperationFailed
>>  from wok.message import WokMessage
>> +from wok.model.notifications import send_websocket_notification
>>  from wok.stringutils import ascii_dict
>>  from wok.utils import remove_old_files
>>
>> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>>  # AsyncTask handling
>>  ASYNCTASK_REQUEST_METHOD = 'TASK'
>>
>> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
>> +
>>
>>  def log_request(code, params, exception, method, status, app=None, 
>> user=None,
>>                  ip=None):
>> @@ -114,6 +117,8 @@ def log_request(code, params, exception, method, 
>> status, app=None, user=None,
>>          ip=ip
>>      ).log()
>>
>> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
>> +
>>      return log_id
>>
>>
>> diff --git a/src/wok/server.py b/src/wok/server.py
>> index fc2e167..2d823c9 100644
>> --- a/src/wok/server.py
>> +++ b/src/wok/server.py
>> @@ -25,8 +25,7 @@ import logging
>>  import logging.handlers
>>  import os
>>
>> -from wok import auth
>> -from wok import config
>> +from wok import auth, config, websocket
>>  from wok.config import config as configParser
>>  from wok.config import WokConfig
>>  from wok.control import sub_nodes
>> @@ -159,6 +158,11 @@ class Server(object):
>>          cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>>                              options.server_root, self.configObj)
>>
>> +        test_mode = config.config.get('server', 'test').lower() == 
>> 'true'
>> +        if not test_mode:
>> +            ws_proxy = websocket.new_ws_proxy()
>> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
>> +
>>          self._load_plugins()
>>          cherrypy.lib.sessions.init()
>>
>> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
>> new file mode 100644
>> index 0000000..5d7fb91
>> --- /dev/null
>> +++ b/src/wok/websocket.py
>> @@ -0,0 +1,123 @@
>> +#!/usr/bin/env python2
>> +#
>> +# Project Wok
>> +#
>> +# Copyright IBM Corp, 2017
>> +#
>> +# Code derived from Project Kimchi
>> +#
>> +# This library is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU Lesser General Public
>> +# License as published by the Free Software Foundation; either
>> +# version 2.1 of the License, or (at your option) any later version.
>> +#
>> +# This library is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> +# Lesser General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU Lesser General Public
>> +# License along with this library; if not, write to the Free Software
>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>> 02110-1301 USA
>> +
>> +import base64
>> +import errno
>> +import os
>> +
>> +from multiprocessing import Process
>> +from websockify import WebSocketProxy
> Add dep on websockify to Wok
>
>
>> +
>> +from config import config, PluginPaths
>> +
>> +
>> +try:
>> +    from websockify.token_plugins import TokenFile
>> +    tokenFile = True
>> +except ImportError:
>> +    tokenFile = False
>> +
>> +try:
>> +    from websockify import ProxyRequestHandler as request_proxy
>> +except:
>> +    from websockify import WebSocketProxy as request_proxy
>> +
>> +
>> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 
>> 'ws-tokens')
> This should be wok path now, and it would be nice to be inside 
> /run/user as commented above.
>
>
>> +
>> +
>> +class CustomHandler(request_proxy):
>> +
>> +    def get_target(self, target_plugin, path):
>> +        if issubclass(CustomHandler, object):
>> +            target = super(CustomHandler, 
>> self).get_target(target_plugin,
>> + path)
>> +        else:
>> +            target = request_proxy.get_target(self, target_plugin, 
>> path)
>> +
>> +        if target[0] == 'unix_socket':
>> +            try:
>> +                self.server.unix_target = target[1]
>> +            except:
>> +                self.unix_target = target[1]
>> +        else:
>> +            try:
>> +                self.server.unix_target = None
>> +            except:
>> +                self.unix_target = None
>> +        return target
>> +
>> +
>> +def new_ws_proxy():
>> +    try:
>> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
>> +    except OSError as e:
>> +        if e.errno == errno.EEXIST:
>> +            pass
>> +
>> +    params = {'listen_host': '127.0.0.1',
>> +              'listen_port': config.get('server', 'websockets_port'),
>> +              'ssl_only': False}
>> +
>> +    # old websockify: do not use TokenFile
>> +    if not tokenFile:
>> +        params['target_cfg'] = WS_TOKENS_DIR
>> +
>> +    # websockify 0.7 and higher: use TokenFile
>> +    else:
>> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
>> +
>> +    def start_proxy():
>> +        try:
>> +            server = WebSocketProxy(RequestHandlerClass=CustomHandler,
>> +                                    **params)
>> +        except TypeError:
>> +            server = CustomHandler(**params)
>> +
>> +        server.start_server()
>> +
>> +    proc = Process(target=start_proxy)
>> +    proc.start()
>> +    return proc
>> +
>> +
>> +def add_proxy_token(name, port, is_unix_socket=False):
>> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
>> +        """
>> +        From python documentation base64.urlsafe_b64encode(s)
>> +        substitutes - instead of + and _ instead of / in the
>> +        standard Base64 alphabet, BUT the result can still
>> +        contain = which is not safe in a URL query component.
>> +        So remove it when needed as base64 can work well without it.
>> +        """
>> +        name = base64.urlsafe_b64encode(name).rstrip('=')
>> +        if is_unix_socket:
>> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), 
>> port))
>> +        else:
>> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
>> +
>> +
>> +def remove_proxy_token(name):
>> +    try:
>> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
>> +    except OSError:
>> +        pass
>> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
>> index 20c017e..f5031ce 100644
>> --- a/ui/js/src/wok.main.js
>> +++ b/ui/js/src/wok.main.js
>> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>>      wok.config = {};
>>  });
>>
>> +
>> +wok.notificationListeners = {};
>> +wok.addNotificationListener = function(name, func) {
>> +    wok.notificationListeners[name] = func;
>> +    $(window).one("hashchange", function() {
>> +        delete wok.notificationListeners[name];
>> +    });
>> +};
>> +
>> +wok.notificationsWebSocket = undefined;
>> +wok.startNotificationWebSocket = function () {
>> +    var addr = window.location.hostname + ':' + window.location.port;
>> +    var token = 
>> wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>> +    var url = 'wss://' + addr + '/websockify?token=' + token;
>> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
>> +
>> +    wok.notificationsWebSocket.onmessage = function(event) {
>> +        var message = window.atob(event.data);
>> +        for (name in wok.notificationListeners) {
>> +            func = wok.notificationListeners[name];
>> +            func(message);
>> +        }
>> +    };
>> +
>> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
>> +    var heartbeat = setInterval(function() {
>> + wok.notificationsWebSocket.send(window.btoa('heartbeat'));
>> +    }, 30000);
>> +
>> +    wok.notificationsWebSocket.onclose = function() {
>> +        clearInterval(heartbeat);
>> +    };
>> +};
>> +
>> +
>>  wok.main = function() {
>>      wok.isLoggingOut = false;
>>      wok.popable();
>> @@ -395,6 +430,9 @@ wok.main = function() {
>>
>>          // Set handler for help button
>>          $('#btn-help').on('click', wok.openHelp);
>> +
>> +        // start WebSocket
>> +        wok.startNotificationWebSocket();
>>      };
>>
>>      var initUI = function() {
>> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
>> index 0e8fb09..083b6c3 100644
>> --- a/ui/js/wok.user-log.js
>> +++ b/ui/js/wok.user-log.js
>> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>>          $("#user-log-grid").bootgrid("search");
>>          wok.initUserLogConfigGridData();
>>      });
>> +
>> +    wok.addNotificationListener('userlog', function(message) {
>> +        if (message === 'new_log_entry') {
>> +            $("#refresh-button").click();
>> +        }
>> +    });
>>  };
>>
>>  wok.initUserLogWindow = function() {
>>
>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago

On 02/27/2017 11:41 AM, Aline Manera wrote:
>
>
> On 02/27/2017 11:35 AM, Lucio Correia wrote:
>> Hi Daniel, that is great feature, see my comments below.
>>
>> On 24/02/2017 10:22, dhbarboza82@gmail.com wrote:
>>> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>>
>>> This patch makes backend and UI changes to implement the asynchronous
>>> UI notification in WoK.
>>>
>>> - Backend:
>>>
>>> A push server was implemented from scratch to manage the opened 
>>> websocket
>>> connections. The push server connects to the /run/woknotifications
>>> UNIX socket and broadcasts all messages to all connections.
>>>
>>> The websocket module is the same module that exists in the Kimchi
>>> plug-in. The idea is to remove the module from Kimchi and make it
>>> use the module from WoK. ws_proxy initialization was also added
>>> in src/wok/server.py.
>>>
>>> - Frontend:
>>>
>>> In ui/js/wok.main.js two new functions were added to help the
>>> usage of asynchronous notifications in the frontend. The idea:
>>> a single websocket is opened per session. This opened websocket
>>> will broadcast all incoming messages to all listeners registered.
>>> Listeners can be added by the new wok.addNotificationListener()
>>> method. This method will clean up any registered listener by
>>> itself when the user changes tabs/URL.
>>>
>>> The single websocket sends heartbeats to the backend side each
>>> 30 seconds. No reply from the backend is issued or expected. This
>>> heartbeat is just a way to ensure that the browser does not
>>> close the connection due to inactivity. This behavior varies from
>>> browser to browser but this 30 second heartbeat is more than enough
>>> to ensure that the websocket is kept alive.
>>>
>>> - Working example in User Log:
>>>
>>> A simple usage is provided in this patch. A change was made in
>>> src/wok/reqlogger.py to send an asynchronous notification each
>>> time a new log entry is created. In ui/js/wok.user-log.js a
>>> websocket listener is added using wok.addNotificationListener()
>>> and, for each message that indicates a new user log entry, a
>>> refresh in the listing is issued.
>>>
>>> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>> ---
>>>  src/wok/model/notifications.py |  18 +++++-
>>>  src/wok/pushserver.py          | 132 
>>> +++++++++++++++++++++++++++++++++++++++++
>>>  src/wok/reqlogger.py           |   7 ++-
>>>  src/wok/server.py              |   8 ++-
>>>  src/wok/websocket.py           | 123 
>>> ++++++++++++++++++++++++++++++++++++++
>>>  ui/js/src/wok.main.js          |  38 ++++++++++++
>>>  ui/js/wok.user-log.js          |   6 ++
>>>  7 files changed, 327 insertions(+), 5 deletions(-)
>>>  create mode 100644 src/wok/pushserver.py
>>>  create mode 100644 src/wok/websocket.py
>>>
>>> diff --git a/src/wok/model/notifications.py 
>>> b/src/wok/model/notifications.py
>>> index bdb7c78..597eac5 100644
>>> --- a/src/wok/model/notifications.py
>>> +++ b/src/wok/model/notifications.py
>>> @@ -1,7 +1,7 @@
>>>  #
>>>  # Project Wok
>>>  #
>>> -# Copyright IBM Corp, 2016
>>> +# Copyright IBM Corp, 2016-2017
>>>  #
>>>  # This library is free software; you can redistribute it and/or
>>>  # modify it under the terms of the GNU Lesser General Public
>>> @@ -19,12 +19,22 @@
>>>
>>>  from datetime import datetime
>>>
>>> +from wok import config
>>>  from wok.exception import NotFoundError, OperationFailed
>>>  from wok.message import WokMessage
>>> +from wok.pushserver import PushServer
>>>  from wok.utils import wok_log
>>>
>>>
>>>  notificationsStore = {}
>>> +push_server = None
>>> +
>>> +
>>> +def send_websocket_notification(message):
>>> +    global push_server
>>> +
>>> +    if push_server:
>>> +        push_server.send_notification(message)
>>>
>>>
>>>  def add_notification(code, args=None, plugin_name=None):
>>> @@ -57,7 +67,11 @@ def del_notification(code):
>>>
>>>  class NotificationsModel(object):
>>>      def __init__(self, **kargs):
>>> -        pass
>>> +        global push_server
>>> +
>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>> 'true'
>>
>> This will cause some tests to initialize pushserver, since 'test' 
>> option is now tied to which model runs (mockmodel or model). Problem 
>> is that now wok tests run without sudo (patch in ML), and 
>> pushserver's BASE_DIRECTORY is only writable with sudo permission. 
>> Gave a suggestion there.

Current design is that the websocket_proxy isn't initiated in test_mode.


>>
>>
>>> +        if not test_mode:
>>> +            push_server = PushServer()
>> All 'users' of functionality will have their own instance of 
>> pushserver (each with a "while True" running)? Why not a single 
>> instance used by everybody?
>>

One WoK instance will have it's own push server. The constructor of 
NotificationsModel is called
only once per WoK init.

Also, I haven't prepared this feature to be run in a scenario of 
multiple WoK instances running at the
same time - since it's always the same unix socket used, multiples 
instances can't be launched at
once.

To allow multiple push servers to be started simultaneously I would need 
to make the unix socket
randomly generated. This means that the websocket URL would change. The 
UI then would need
a way to discover the current websocket URL to be used, probably using 
the /config API. It is feasible,
but more changes would need to be made.

>
> Good point! Maybe it is better to move it do Server() to initiate the 
> websockets connections to everyone. Also there is no need to 
> distinguish test mode.

Websockets connections are available to everyone as is.

As I said in my previous reply, test_mode is already being distinguished 
in the websocket initialization
of Kimchi:

root.py line 48:

         # When running on test mode, specify the objectstore location to
         # remove the file on server shutting down. That way, the system 
will
         # not suffer any change while running on test mode
         if wok_options.test and (wok_options.test is True or
                                  wok_options.test.lower() == 'true'):
             self.objectstore_loc = tempfile.mktemp()
             self.model = mockmodel.MockModel(self.objectstore_loc)

             def remove_objectstore():
                 if os.path.exists(self.objectstore_loc):
                     os.unlink(self.objectstore_loc)
             cherrypy.engine.subscribe('exit', remove_objectstore)
         else:
             self.model = kimchiModel.Model()
             ws_proxy = websocket.new_ws_proxy()
             cherrypy.engine.subscribe('exit', ws_proxy.terminate)


I assumed that this design was intended and I haven't thought of any
good reason to change it, so this design was considered in the
push_server and also in the websocket initiation in WoK.


>
>>
>>>
>>>      def get_list(self):
>>>          global notificationsStore
>>> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
>>> new file mode 100644
>>> index 0000000..8993f00
>>> --- /dev/null
>>> +++ b/src/wok/pushserver.py
>>> @@ -0,0 +1,132 @@
>>> +#
>>> +# Project Wok
>>> +#
>>> +# Copyright IBM Corp, 2017
>>> +#
>>> +# This library is free software; you can redistribute it and/or
>>> +# modify it under the terms of the GNU Lesser General Public
>>> +# License as published by the Free Software Foundation; either
>>> +# version 2.1 of the License, or (at your option) any later version.
>>> +#
>>> +# This library is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> +# Lesser General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU Lesser General Public
>>> +# License along with this library; if not, write to the Free Software
>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>> 02110-1301 USA
>>> +#
>>> +
>>> +import cherrypy
>>> +import os
>>> +import select
>>> +import socket
>>> +import threading
>>> +
>>> +import websocket
>>> +from utils import wok_log
>>> +
>>> +
>>> +BASE_DIRECTORY = '/run'
>> Suggestion to use:
>> os.path.join('/run/user', str(os.getuid()))
>> in order tests may be run with root.

I would prefer to choose a path that can be written by anyone else to 
allow the push_server
to be started without root.


>>
>>> +TOKEN_NAME = 'woknotifications'
>>> +
>>> +
>>> +class PushServer(object):
>>> +
>>> +    def set_socket_file(self):
>>> +        if not os.path.isdir(BASE_DIRECTORY):
>>> +            try:
>>> +                os.mkdir(BASE_DIRECTORY)
>>> +            except OSError:
>>> +                raise RuntimeError('PushServer base UNIX socket dir 
>>> %s '
>>> +                                   'not found.' % BASE_DIRECTORY)
>>> +
>>> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
>>> +
>>> +        if os.path.exists(self.server_addr):
>>> +            try:
>>> +                os.remove(self.server_addr)
>>> +            except:
>>> +                raise RuntimeError('There is an existing connection 
>>> in %s' %
>>> +                                   self.server_addr)
>>> +
>>> +    def __init__(self):
>>> +        self.set_socket_file()
>>> +
>>> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
>>> +
>>> +        self.connections = []
>>> +
>>> +        self.server_running = True
>>> +        self.server_socket = socket.socket(socket.AF_UNIX, 
>>> socket.SOCK_STREAM)
>>> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
>>> +                                      socket.SO_REUSEADDR, 1)
>>> +        self.server_socket.bind(self.server_addr)
>>> +        self.server_socket.listen(10)
>>> +        wok_log.info('Push server created on address %s' % 
>>> self.server_addr)
>>> +
>>> +        self.connections.append(self.server_socket)
>>> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
>>> +
>>> +        server_loop = threading.Thread(target=self.listen)
>>> +        server_loop.start()
>>> +
>>> +    def listen(self):
>>> +        try:
>>> +            while self.server_running:
>>> +                read_ready, _, _ = select.select(self.connections,
>>> +                                                 [], [], 1)
>>> +                for sock in read_ready:
>>> +                    if not self.server_running:
>>> +                        break
>>> +
>>> +                    if sock == self.server_socket:
>>> +
>>> +                        new_socket, addr = self.server_socket.accept()
>>> +                        self.connections.append(new_socket)
>>> +                    else:
>>> +                        try:
>>> +                            data = sock.recv(4096)
>>> +                        except:
>>> +                            try:
>>> +                                self.connections.remove(sock)
>>> +                            except ValueError:
>>> +                                pass
>>> +
>>> +                            continue
>>> +                        if data and data == 'CLOSE':
>>> +                            sock.send('ACK')
>>> +                            try:
>>> +                                self.connections.remove(sock)
>>> +                            except ValueError:
>>> +                                pass
>>> +                            sock.close()
>>> +
>>> +        except Exception as e:
>>> +            raise RuntimeError('Exception ocurred in listen() of 
>>> pushserver '
>>> +                               'module: %s' % e.message)
>>> +
>>> +    def send_notification(self, message):
>>> +        for sock in self.connections:
>>> +            if sock != self.server_socket:
>>> +                try:
>>> +                    sock.send(message)
>>> +                except IOError as e:
>>> +                    if 'Broken pipe' in str(e):
>>> +                        sock.close()
>>> +                        try:
>>> +                            self.connections.remove(sock)
>>> +                        except ValueError:
>>> +                            pass
>>> +
>>> +    def close_server(self):
>>> +        try:
>>> +            self.server_running = False
>>> +            self.server_socket.shutdown(socket.SHUT_RDWR)
>>> +            self.server_socket.close()
>>> +            os.remove(self.server_addr)
>>> +        except:
>>> +            pass
>>> +        finally:
>>> +            cherrypy.engine.unsubscribe('stop', self.close_server)
>>> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
>>> index 92e155d..1b774e2 100644
>>> --- a/src/wok/reqlogger.py
>>> +++ b/src/wok/reqlogger.py
>>> @@ -1,7 +1,7 @@
>>>  #
>>>  # Project Wok
>>>  #
>>> -# Copyright IBM Corp, 2016
>>> +# Copyright IBM Corp, 2016-2017
>>>  #
>>>  # This library is free software; you can redistribute it and/or
>>>  # modify it under the terms of the GNU Lesser General Public
>>> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>>>  from wok.config import get_log_download_path, paths
>>>  from wok.exception import InvalidParameter, OperationFailed
>>>  from wok.message import WokMessage
>>> +from wok.model.notifications import send_websocket_notification
>>>  from wok.stringutils import ascii_dict
>>>  from wok.utils import remove_old_files
>>>
>>> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>>>  # AsyncTask handling
>>>  ASYNCTASK_REQUEST_METHOD = 'TASK'
>>>
>>> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
>>> +
>>>
>>>  def log_request(code, params, exception, method, status, app=None, 
>>> user=None,
>>>                  ip=None):
>>> @@ -114,6 +117,8 @@ def log_request(code, params, exception, method, 
>>> status, app=None, user=None,
>>>          ip=ip
>>>      ).log()
>>>
>>> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
>>> +
>>>      return log_id
>>>
>>>
>>> diff --git a/src/wok/server.py b/src/wok/server.py
>>> index fc2e167..2d823c9 100644
>>> --- a/src/wok/server.py
>>> +++ b/src/wok/server.py
>>> @@ -25,8 +25,7 @@ import logging
>>>  import logging.handlers
>>>  import os
>>>
>>> -from wok import auth
>>> -from wok import config
>>> +from wok import auth, config, websocket
>>>  from wok.config import config as configParser
>>>  from wok.config import WokConfig
>>>  from wok.control import sub_nodes
>>> @@ -159,6 +158,11 @@ class Server(object):
>>>          cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>>>                              options.server_root, self.configObj)
>>>
>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>> 'true'
>>> +        if not test_mode:
>>> +            ws_proxy = websocket.new_ws_proxy()
>>> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
>>> +
>>>          self._load_plugins()
>>>          cherrypy.lib.sessions.init()
>>>
>>> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
>>> new file mode 100644
>>> index 0000000..5d7fb91
>>> --- /dev/null
>>> +++ b/src/wok/websocket.py
>>> @@ -0,0 +1,123 @@
>>> +#!/usr/bin/env python2
>>> +#
>>> +# Project Wok
>>> +#
>>> +# Copyright IBM Corp, 2017
>>> +#
>>> +# Code derived from Project Kimchi
>>> +#
>>> +# This library is free software; you can redistribute it and/or
>>> +# modify it under the terms of the GNU Lesser General Public
>>> +# License as published by the Free Software Foundation; either
>>> +# version 2.1 of the License, or (at your option) any later version.
>>> +#
>>> +# This library is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> +# Lesser General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU Lesser General Public
>>> +# License along with this library; if not, write to the Free Software
>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>> 02110-1301 USA
>>> +
>>> +import base64
>>> +import errno
>>> +import os
>>> +
>>> +from multiprocessing import Process
>>> +from websockify import WebSocketProxy
>> Add dep on websockify to Wok

Good catch. I'll add it in v2.

>>
>>
>>> +
>>> +from config import config, PluginPaths
>>> +
>>> +
>>> +try:
>>> +    from websockify.token_plugins import TokenFile
>>> +    tokenFile = True
>>> +except ImportError:
>>> +    tokenFile = False
>>> +
>>> +try:
>>> +    from websockify import ProxyRequestHandler as request_proxy
>>> +except:
>>> +    from websockify import WebSocketProxy as request_proxy
>>> +
>>> +
>>> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 
>>> 'ws-tokens')
>> This should be wok path now, and it would be nice to be inside 
>> /run/user as commented above.
>>
>>
Good catch again. I'll change the 'Kimchi' specific code in v2.

>>> +
>>> +
>>> +class CustomHandler(request_proxy):
>>> +
>>> +    def get_target(self, target_plugin, path):
>>> +        if issubclass(CustomHandler, object):
>>> +            target = super(CustomHandler, 
>>> self).get_target(target_plugin,
>>> + path)
>>> +        else:
>>> +            target = request_proxy.get_target(self, target_plugin, 
>>> path)
>>> +
>>> +        if target[0] == 'unix_socket':
>>> +            try:
>>> +                self.server.unix_target = target[1]
>>> +            except:
>>> +                self.unix_target = target[1]
>>> +        else:
>>> +            try:
>>> +                self.server.unix_target = None
>>> +            except:
>>> +                self.unix_target = None
>>> +        return target
>>> +
>>> +
>>> +def new_ws_proxy():
>>> +    try:
>>> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
>>> +    except OSError as e:
>>> +        if e.errno == errno.EEXIST:
>>> +            pass
>>> +
>>> +    params = {'listen_host': '127.0.0.1',
>>> +              'listen_port': config.get('server', 'websockets_port'),
>>> +              'ssl_only': False}
>>> +
>>> +    # old websockify: do not use TokenFile
>>> +    if not tokenFile:
>>> +        params['target_cfg'] = WS_TOKENS_DIR
>>> +
>>> +    # websockify 0.7 and higher: use TokenFile
>>> +    else:
>>> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
>>> +
>>> +    def start_proxy():
>>> +        try:
>>> +            server = WebSocketProxy(RequestHandlerClass=CustomHandler,
>>> +                                    **params)
>>> +        except TypeError:
>>> +            server = CustomHandler(**params)
>>> +
>>> +        server.start_server()
>>> +
>>> +    proc = Process(target=start_proxy)
>>> +    proc.start()
>>> +    return proc
>>> +
>>> +
>>> +def add_proxy_token(name, port, is_unix_socket=False):
>>> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
>>> +        """
>>> +        From python documentation base64.urlsafe_b64encode(s)
>>> +        substitutes - instead of + and _ instead of / in the
>>> +        standard Base64 alphabet, BUT the result can still
>>> +        contain = which is not safe in a URL query component.
>>> +        So remove it when needed as base64 can work well without it.
>>> +        """
>>> +        name = base64.urlsafe_b64encode(name).rstrip('=')
>>> +        if is_unix_socket:
>>> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), 
>>> port))
>>> +        else:
>>> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), port))
>>> +
>>> +
>>> +def remove_proxy_token(name):
>>> +    try:
>>> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
>>> +    except OSError:
>>> +        pass
>>> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
>>> index 20c017e..f5031ce 100644
>>> --- a/ui/js/src/wok.main.js
>>> +++ b/ui/js/src/wok.main.js
>>> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>>>      wok.config = {};
>>>  });
>>>
>>> +
>>> +wok.notificationListeners = {};
>>> +wok.addNotificationListener = function(name, func) {
>>> +    wok.notificationListeners[name] = func;
>>> +    $(window).one("hashchange", function() {
>>> +        delete wok.notificationListeners[name];
>>> +    });
>>> +};
>>> +
>>> +wok.notificationsWebSocket = undefined;
>>> +wok.startNotificationWebSocket = function () {
>>> +    var addr = window.location.hostname + ':' + window.location.port;
>>> +    var token = 
>>> wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>>> +    var url = 'wss://' + addr + '/websockify?token=' + token;
>>> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
>>> +
>>> +    wok.notificationsWebSocket.onmessage = function(event) {
>>> +        var message = window.atob(event.data);
>>> +        for (name in wok.notificationListeners) {
>>> +            func = wok.notificationListeners[name];
>>> +            func(message);
>>> +        }
>>> +    };
>>> +
>>> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
>>> +    var heartbeat = setInterval(function() {
>>> + wok.notificationsWebSocket.send(window.btoa('heartbeat'));
>>> +    }, 30000);
>>> +
>>> +    wok.notificationsWebSocket.onclose = function() {
>>> +        clearInterval(heartbeat);
>>> +    };
>>> +};
>>> +
>>> +
>>>  wok.main = function() {
>>>      wok.isLoggingOut = false;
>>>      wok.popable();
>>> @@ -395,6 +430,9 @@ wok.main = function() {
>>>
>>>          // Set handler for help button
>>>          $('#btn-help').on('click', wok.openHelp);
>>> +
>>> +        // start WebSocket
>>> +        wok.startNotificationWebSocket();
>>>      };
>>>
>>>      var initUI = function() {
>>> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
>>> index 0e8fb09..083b6c3 100644
>>> --- a/ui/js/wok.user-log.js
>>> +++ b/ui/js/wok.user-log.js
>>> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>>>          $("#user-log-grid").bootgrid("search");
>>>          wok.initUserLogConfigGridData();
>>>      });
>>> +
>>> +    wok.addNotificationListener('userlog', function(message) {
>>> +        if (message === 'new_log_entry') {
>>> +            $("#refresh-button").click();
>>> +        }
>>> +    });
>>>  };
>>>
>>>  wok.initUserLogWindow = function() {
>>>
>>
>>
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Lucio Correia 4 years, 2 months ago
On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>> +BASE_DIRECTORY = '/run'
>>> Suggestion to use:
>>> os.path.join('/run/user', str(os.getuid()))
>>> in order tests may be run with root.
>
> I would prefer to choose a path that can be written by anyone else to
> allow the push_server
> to be started without root.

Sorry, I meant *without* root.
/run/user/<UID> allows for that, it's writable by the user that started 
wokd, be it root or not.

-- 
Lucio Correia

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago

On 02/27/2017 06:18 PM, Lucio Correia wrote:
> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>> +BASE_DIRECTORY = '/run'
>>>> Suggestion to use:
>>>> os.path.join('/run/user', str(os.getuid()))
>>>> in order tests may be run with root.
>>
>> I would prefer to choose a path that can be written by anyone else to
>> allow the push_server
>> to be started without root.
>
> Sorry, I meant *without* root.
> /run/user/<UID> allows for that, it's writable by the user that 
> started wokd, be it root or not.
>
This change would require further UI changes to allow the /config API 
(or other) to inform
the UI of the current websocket URL. This will not solve the problems 
I've seen with the unit tests
though - multiple instances of the push_server will not be possible and 
the unit tests will
break.

We need to discuss an alternative where:

- any user can start the push_server, as you require

- if we're to allow the push_server to be run in test_mode*, we need to 
think in a way of
generating random paths that can be written by any user as well. You are 
working closely
in Debian changes. What system dir can be used that can be written by 
any user and,
preferably, exists in RPM distros too?


Daniel


_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Lucio Correia 4 years, 2 months ago
On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>
>
> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>> +BASE_DIRECTORY = '/run'
>>>>> Suggestion to use:
>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>> in order tests may be run with root.
>>>
>>> I would prefer to choose a path that can be written by anyone else to
>>> allow the push_server
>>> to be started without root.
>>
>> Sorry, I meant *without* root.
>> /run/user/<UID> allows for that, it's writable by the user that
>> started wokd, be it root or not.
>>
> This change would require further UI changes to allow the /config API
> (or other) to inform
> the UI of the current websocket URL. This will not solve the problems
> I've seen with the unit tests
> though - multiple instances of the push_server will not be possible and
> the unit tests will
> break.

I'm not seeing the relation between that BASE_DIR and the URL used by 
UI. Anyway, I believe we both agree that just one server and one URL is 
fine.

My only concern is the directory to save that woknotifications file and 
the permissions required for it.


>
> We need to discuss an alternative where:
>
> - any user can start the push_server, as you require

I'm not requiring it. Just saying that if it will be started by tests, 
it needs to run without sudo (i.e. use paths in filesystems that allow 
for that).

As I mentioned earlier, testing for 'test' option is not enough to avoid 
PushServer to be started during tests, since as of now 'test' option 
only governs which model will be used.


>
> - if we're to allow the push_server to be run in test_mode*, we need to
> think in a way of
> generating random paths that can be written by any user as well. You are
> working closely
> in Debian changes. What system dir can be used that can be written by
> any user and,
> preferably, exists in RPM distros too?

On Fedora 25, /run/user/<UID>/libvirt is the default dir used to store 
runtime data by libvirt when it is started with "qemu:///session" url 
(regular user session).

In fact it seems to be default place for that kind of usage in both distros:

Ubuntu 16.04:
$ ls /run/user/1000/
dbus-session  gnome-shell  gvfs-burn  pulse              systemd 
upstart-dbus-bridge.4451.pid  upstart-udev-bridge.2946.pid
dconf         gvfs         keyring    speech-dispatcher  upstart 
upstart-file-bridge.4451.pid  upstart-udev-bridge.4451.pid

Fedora 25:
$ ls /run/user/1000/
bus  libvirt  systemd


-- 
Lucio Correia

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago

On 02/28/2017 10:02 AM, Lucio Correia wrote:
> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>
>>
>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>> Suggestion to use:
>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>> in order tests may be run with root.
>>>>
>>>> I would prefer to choose a path that can be written by anyone else to
>>>> allow the push_server
>>>> to be started without root.
>>>
>>> Sorry, I meant *without* root.
>>> /run/user/<UID> allows for that, it's writable by the user that
>>> started wokd, be it root or not.
>>>
>> This change would require further UI changes to allow the /config API
>> (or other) to inform
>> the UI of the current websocket URL. This will not solve the problems
>> I've seen with the unit tests
>> though - multiple instances of the push_server will not be possible and
>> the unit tests will
>> break.
>
> I'm not seeing the relation between that BASE_DIR and the URL used by 
> UI. Anyway, I believe we both agree that just one server and one URL 
> is fine.
The unix socket, now /run/woknotifications, is used by the push server 
to send
notifications. The websocket creates a proxy between this unix socket and a
common TCP socket that the UI can read/write to.

This connection is made by the following URL:

<host:port>/websockify?token=<token>

The UI knows what token to use to connect to the unix socket because the
token is based on the file path of the unix socket. This is the relevant UI
code:

var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");


This means that if I change the file path to /run/user_id, the token 
changes too. If the UI can't
predict what unix socket is being used in WoK then we need to supply 
this info to the UI
in another way.

>
> My only concern is the directory to save that woknotifications file 
> and the permissions required for it.
>
>
>>
>> We need to discuss an alternative where:
>>
>> - any user can start the push_server, as you require
>
> I'm not requiring it. Just saying that if it will be started by tests, 
> it needs to run without sudo (i.e. use paths in filesystems that allow 
> for that).
>
> As I mentioned earlier, testing for 'test' option is not enough to 
> avoid PushServer to be started during tests, since as of now 'test' 
> option only governs which model will be used.

Strange, it worked for me. I think you're referring to WoK changes that 
were made
right after/before I sent this patch set. I'll re-test it and see how I 
can prevent it
from running on test mode.

>
>
>>
>> - if we're to allow the push_server to be run in test_mode*, we need to
>> think in a way of
>> generating random paths that can be written by any user as well. You are
>> working closely
>> in Debian changes. What system dir can be used that can be written by
>> any user and,
>> preferably, exists in RPM distros too?
>
> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to store 
> runtime data by libvirt when it is started with "qemu:///session" url 
> (regular user session).
>
> In fact it seems to be default place for that kind of usage in both 
> distros:
>
> Ubuntu 16.04:
> $ ls /run/user/1000/
> dbus-session  gnome-shell  gvfs-burn  pulse              systemd 
> upstart-dbus-bridge.4451.pid  upstart-udev-bridge.2946.pid
> dconf         gvfs         keyring    speech-dispatcher  upstart 
> upstart-file-bridge.4451.pid  upstart-udev-bridge.4451.pid
>
> Fedora 25:
> $ ls /run/user/1000/
> bus  libvirt  systemd

I see. Perhaps changing the dir to /run/user/<UID> is the right thing to 
do in
this case. This means that we can't avoid "breaking" the current design 
- we'll
need to use an additional API to inform the UI of the current unix 
socket (token)
being used.


>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/28/2017 10:24 AM, Daniel Henrique Barboza wrote:
>
>
> On 02/28/2017 10:02 AM, Lucio Correia wrote:
>> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>>
>>>
>>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>>> Suggestion to use:
>>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>>> in order tests may be run with root.
>>>>>
>>>>> I would prefer to choose a path that can be written by anyone else to
>>>>> allow the push_server
>>>>> to be started without root.
>>>>
>>>> Sorry, I meant *without* root.
>>>> /run/user/<UID> allows for that, it's writable by the user that
>>>> started wokd, be it root or not.
>>>>
>>> This change would require further UI changes to allow the /config API
>>> (or other) to inform
>>> the UI of the current websocket URL. This will not solve the problems
>>> I've seen with the unit tests
>>> though - multiple instances of the push_server will not be possible and
>>> the unit tests will
>>> break.
>>
>> I'm not seeing the relation between that BASE_DIR and the URL used by 
>> UI. Anyway, I believe we both agree that just one server and one URL 
>> is fine.
> The unix socket, now /run/woknotifications, is used by the push server 
> to send
> notifications. The websocket creates a proxy between this unix socket 
> and a
> common TCP socket that the UI can read/write to.
>
> This connection is made by the following URL:
>
> <host:port>/websockify?token=<token>
>
> The UI knows what token to use to connect to the unix socket because the
> token is based on the file path of the unix socket. This is the 
> relevant UI
> code:
>
> var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>
>
> This means that if I change the file path to /run/user_id, the token 
> changes too. If the UI can't
> predict what unix socket is being used in WoK then we need to supply 
> this info to the UI
> in another way.

Daniel, for the novnc/spice/serial console, the token is the virtual 
machine name, ie, it is not related to the real path in the system.
The websocket knows on which directory to look for the token and it is all.

Why is it different for the notifications? Shouldn't the directory be 
known by the server and the UI only request a token? The same as 
novnc/spice/serial does.

>
>>
>> My only concern is the directory to save that woknotifications file 
>> and the permissions required for it.
>>
>>
>>>
>>> We need to discuss an alternative where:
>>>
>>> - any user can start the push_server, as you require
>>
>> I'm not requiring it. Just saying that if it will be started by 
>> tests, it needs to run without sudo (i.e. use paths in filesystems 
>> that allow for that).
>>
>> As I mentioned earlier, testing for 'test' option is not enough to 
>> avoid PushServer to be started during tests, since as of now 'test' 
>> option only governs which model will be used.
>
> Strange, it worked for me. I think you're referring to WoK changes 
> that were made
> right after/before I sent this patch set. I'll re-test it and see how 
> I can prevent it
> from running on test mode.
>
>>
>>
>>>
>>> - if we're to allow the push_server to be run in test_mode*, we need to
>>> think in a way of
>>> generating random paths that can be written by any user as well. You 
>>> are
>>> working closely
>>> in Debian changes. What system dir can be used that can be written by
>>> any user and,
>>> preferably, exists in RPM distros too?
>>
>> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to 
>> store runtime data by libvirt when it is started with 
>> "qemu:///session" url (regular user session).
>>
>> In fact it seems to be default place for that kind of usage in both 
>> distros:
>>
>> Ubuntu 16.04:
>> $ ls /run/user/1000/
>> dbus-session  gnome-shell  gvfs-burn  pulse              systemd 
>> upstart-dbus-bridge.4451.pid  upstart-udev-bridge.2946.pid
>> dconf         gvfs         keyring    speech-dispatcher  upstart 
>> upstart-file-bridge.4451.pid  upstart-udev-bridge.4451.pid
>>
>> Fedora 25:
>> $ ls /run/user/1000/
>> bus  libvirt  systemd
>
> I see. Perhaps changing the dir to /run/user/<UID> is the right thing 
> to do in
> this case. This means that we can't avoid "breaking" the current 
> design - we'll
> need to use an additional API to inform the UI of the current unix 
> socket (token)
> being used.
>
>
>>
>>
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago

On 02/28/2017 10:31 AM, Aline Manera wrote:
>
>
> On 02/28/2017 10:24 AM, Daniel Henrique Barboza wrote:
>>
>>
>> On 02/28/2017 10:02 AM, Lucio Correia wrote:
>>> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>>>
>>>>
>>>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>>>> Suggestion to use:
>>>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>>>> in order tests may be run with root.
>>>>>>
>>>>>> I would prefer to choose a path that can be written by anyone 
>>>>>> else to
>>>>>> allow the push_server
>>>>>> to be started without root.
>>>>>
>>>>> Sorry, I meant *without* root.
>>>>> /run/user/<UID> allows for that, it's writable by the user that
>>>>> started wokd, be it root or not.
>>>>>
>>>> This change would require further UI changes to allow the /config API
>>>> (or other) to inform
>>>> the UI of the current websocket URL. This will not solve the problems
>>>> I've seen with the unit tests
>>>> though - multiple instances of the push_server will not be possible 
>>>> and
>>>> the unit tests will
>>>> break.
>>>
>>> I'm not seeing the relation between that BASE_DIR and the URL used 
>>> by UI. Anyway, I believe we both agree that just one server and one 
>>> URL is fine.
>> The unix socket, now /run/woknotifications, is used by the push 
>> server to send
>> notifications. The websocket creates a proxy between this unix socket 
>> and a
>> common TCP socket that the UI can read/write to.
>>
>> This connection is made by the following URL:
>>
>> <host:port>/websockify?token=<token>
>>
>> The UI knows what token to use to connect to the unix socket because the
>> token is based on the file path of the unix socket. This is the 
>> relevant UI
>> code:
>>
>> var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, 
>> "");
>>
>>
>> This means that if I change the file path to /run/user_id, the token 
>> changes too. If the UI can't
>> predict what unix socket is being used in WoK then we need to supply 
>> this info to the UI
>> in another way.
>
> Daniel, for the novnc/spice/serial console, the token is the virtual 
> machine name, ie, it is not related to the real path in the system.
> The websocket knows on which directory to look for the token and it is 
> all.
>
> Why is it different for the notifications? Shouldn't the directory be 
> known by the server and the UI only request a token? The same as 
> novnc/spice/serial does.

In these cases the UI doesn't request a token, the UI already knows it. 
It is embedded in the
URL.

kimchi.api.js:

     serialToVM : function(vm) {
                (...)
                 url = 'https://' + location.hostname + ':' + proxy_port;
                 url += server_root;
                 url += "/plugins/kimchi/serial/html/serial.html";
                 url += "?port=" + proxy_port;
                 url += "&path=" + server_root + "/websockify";
                 url += "?token=" + 
wok.urlSafeB64Encode(vm+'-console').replace(/=*$/g, "");
                 url += '&encrypt=1';


(same thing with novnc and spice, check vncToVM and spiceToVM in the same
file)


In serial.html the token is retrieved as follows:


             var url = 'wss://' + window.location.hostname + ':' + 
params.get('port');
             url += '/' + params.get('path');
             url += '?token=' + params.get('token');
             var socket = new WebSocket(url, ['base64']);



Note that for serial,novnc and spice the token is already knew by the UI 
due to the
URL. In the situation we're describing with the push server this will 
not be case - we
can't predict the current user information and embed it in the URL (not 
in the frontend
level, at least).


This is why I am saying that we need to deliver this information in 
another fashion to
the UI, perhaps using the /config API.


>
>>
>>>
>>> My only concern is the directory to save that woknotifications file 
>>> and the permissions required for it.
>>>
>>>
>>>>
>>>> We need to discuss an alternative where:
>>>>
>>>> - any user can start the push_server, as you require
>>>
>>> I'm not requiring it. Just saying that if it will be started by 
>>> tests, it needs to run without sudo (i.e. use paths in filesystems 
>>> that allow for that).
>>>
>>> As I mentioned earlier, testing for 'test' option is not enough to 
>>> avoid PushServer to be started during tests, since as of now 'test' 
>>> option only governs which model will be used.
>>
>> Strange, it worked for me. I think you're referring to WoK changes 
>> that were made
>> right after/before I sent this patch set. I'll re-test it and see how 
>> I can prevent it
>> from running on test mode.
>>
>>>
>>>
>>>>
>>>> - if we're to allow the push_server to be run in test_mode*, we 
>>>> need to
>>>> think in a way of
>>>> generating random paths that can be written by any user as well. 
>>>> You are
>>>> working closely
>>>> in Debian changes. What system dir can be used that can be written by
>>>> any user and,
>>>> preferably, exists in RPM distros too?
>>>
>>> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to 
>>> store runtime data by libvirt when it is started with 
>>> "qemu:///session" url (regular user session).
>>>
>>> In fact it seems to be default place for that kind of usage in both 
>>> distros:
>>>
>>> Ubuntu 16.04:
>>> $ ls /run/user/1000/
>>> dbus-session  gnome-shell  gvfs-burn  pulse systemd 
>>> upstart-dbus-bridge.4451.pid upstart-udev-bridge.2946.pid
>>> dconf         gvfs         keyring    speech-dispatcher upstart 
>>> upstart-file-bridge.4451.pid upstart-udev-bridge.4451.pid
>>>
>>> Fedora 25:
>>> $ ls /run/user/1000/
>>> bus  libvirt  systemd
>>
>> I see. Perhaps changing the dir to /run/user/<UID> is the right thing 
>> to do in
>> this case. This means that we can't avoid "breaking" the current 
>> design - we'll
>> need to use an additional API to inform the UI of the current unix 
>> socket (token)
>> being used.
>>
>>
>>>
>>>
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/28/2017 10:58 AM, Daniel Henrique Barboza wrote:
>
>
> On 02/28/2017 10:31 AM, Aline Manera wrote:
>>
>>
>> On 02/28/2017 10:24 AM, Daniel Henrique Barboza wrote:
>>>
>>>
>>> On 02/28/2017 10:02 AM, Lucio Correia wrote:
>>>> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>>>>
>>>>>
>>>>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>>>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>>>>> Suggestion to use:
>>>>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>>>>> in order tests may be run with root.
>>>>>>>
>>>>>>> I would prefer to choose a path that can be written by anyone 
>>>>>>> else to
>>>>>>> allow the push_server
>>>>>>> to be started without root.
>>>>>>
>>>>>> Sorry, I meant *without* root.
>>>>>> /run/user/<UID> allows for that, it's writable by the user that
>>>>>> started wokd, be it root or not.
>>>>>>
>>>>> This change would require further UI changes to allow the /config API
>>>>> (or other) to inform
>>>>> the UI of the current websocket URL. This will not solve the problems
>>>>> I've seen with the unit tests
>>>>> though - multiple instances of the push_server will not be 
>>>>> possible and
>>>>> the unit tests will
>>>>> break.
>>>>
>>>> I'm not seeing the relation between that BASE_DIR and the URL used 
>>>> by UI. Anyway, I believe we both agree that just one server and one 
>>>> URL is fine.
>>> The unix socket, now /run/woknotifications, is used by the push 
>>> server to send
>>> notifications. The websocket creates a proxy between this unix 
>>> socket and a
>>> common TCP socket that the UI can read/write to.
>>>
>>> This connection is made by the following URL:
>>>
>>> <host:port>/websockify?token=<token>
>>>
>>> The UI knows what token to use to connect to the unix socket because 
>>> the
>>> token is based on the file path of the unix socket. This is the 
>>> relevant UI
>>> code:
>>>
>>> var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, 
>>> "");
>>>
>>>
>>> This means that if I change the file path to /run/user_id, the token 
>>> changes too. If the UI can't
>>> predict what unix socket is being used in WoK then we need to supply 
>>> this info to the UI
>>> in another way.
>>
>> Daniel, for the novnc/spice/serial console, the token is the virtual 
>> machine name, ie, it is not related to the real path in the system.
>> The websocket knows on which directory to look for the token and it 
>> is all.
>>
>> Why is it different for the notifications? Shouldn't the directory be 
>> known by the server and the UI only request a token? The same as 
>> novnc/spice/serial does.
>
> In these cases the UI doesn't request a token, the UI already knows 
> it. It is embedded in the
> URL.
>
> kimchi.api.js:
>
>     serialToVM : function(vm) {
>                (...)
>                 url = 'https://' + location.hostname + ':' + proxy_port;
>                 url += server_root;
>                 url += "/plugins/kimchi/serial/html/serial.html";
>                 url += "?port=" + proxy_port;
>                 url += "&path=" + server_root + "/websockify";
>                 url += "?token=" + 
> wok.urlSafeB64Encode(vm+'-console').replace(/=*$/g, "");
>                 url += '&encrypt=1';
>
>
> (same thing with novnc and spice, check vncToVM and spiceToVM in the same
> file)
>
>
> In serial.html the token is retrieved as follows:
>
>
>             var url = 'wss://' + window.location.hostname + ':' + 
> params.get('port');
>             url += '/' + params.get('path');
>             url += '?token=' + params.get('token');
>             var socket = new WebSocket(url, ['base64']);
>
>
>
> Note that for serial,novnc and spice the token is already knew by the 
> UI due to the
> URL. In the situation we're describing with the push server this will 
> not be case - we
> can't predict the current user information and embed it in the URL 
> (not in the frontend
> level, at least).
>

I don't think it is true!
The UI just need to provide a token. The token is the file name. There 
is no relation with the websockets directory.

Please, do a test! Change the directory on backend, proper set it on 
websocket, keep the file name the same "woknotifications" and test the 
async notification.

>
> This is why I am saying that we need to deliver this information in 
> another fashion to
> the UI, perhaps using the /config API.
>

It is not needed.

>
>>
>>>
>>>>
>>>> My only concern is the directory to save that woknotifications file 
>>>> and the permissions required for it.
>>>>
>>>>
>>>>>
>>>>> We need to discuss an alternative where:
>>>>>
>>>>> - any user can start the push_server, as you require
>>>>
>>>> I'm not requiring it. Just saying that if it will be started by 
>>>> tests, it needs to run without sudo (i.e. use paths in filesystems 
>>>> that allow for that).
>>>>
>>>> As I mentioned earlier, testing for 'test' option is not enough to 
>>>> avoid PushServer to be started during tests, since as of now 'test' 
>>>> option only governs which model will be used.
>>>
>>> Strange, it worked for me. I think you're referring to WoK changes 
>>> that were made
>>> right after/before I sent this patch set. I'll re-test it and see 
>>> how I can prevent it
>>> from running on test mode.
>>>
>>>>
>>>>
>>>>>
>>>>> - if we're to allow the push_server to be run in test_mode*, we 
>>>>> need to
>>>>> think in a way of
>>>>> generating random paths that can be written by any user as well. 
>>>>> You are
>>>>> working closely
>>>>> in Debian changes. What system dir can be used that can be written by
>>>>> any user and,
>>>>> preferably, exists in RPM distros too?
>>>>
>>>> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to 
>>>> store runtime data by libvirt when it is started with 
>>>> "qemu:///session" url (regular user session).
>>>>
>>>> In fact it seems to be default place for that kind of usage in both 
>>>> distros:
>>>>
>>>> Ubuntu 16.04:
>>>> $ ls /run/user/1000/
>>>> dbus-session  gnome-shell  gvfs-burn  pulse systemd 
>>>> upstart-dbus-bridge.4451.pid upstart-udev-bridge.2946.pid
>>>> dconf         gvfs         keyring    speech-dispatcher upstart 
>>>> upstart-file-bridge.4451.pid upstart-udev-bridge.4451.pid
>>>>
>>>> Fedora 25:
>>>> $ ls /run/user/1000/
>>>> bus  libvirt  systemd
>>>
>>> I see. Perhaps changing the dir to /run/user/<UID> is the right 
>>> thing to do in
>>> this case. This means that we can't avoid "breaking" the current 
>>> design - we'll
>>> need to use an additional API to inform the UI of the current unix 
>>> socket (token)
>>> being used.
>>>
>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Kimchi-devel mailing list
>>> Kimchi-devel@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>>>
>>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Lucio Correia 4 years, 2 months ago
On 28/02/2017 10:24, Daniel Henrique Barboza wrote:
>
>
> On 02/28/2017 10:02 AM, Lucio Correia wrote:
>> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>>
>>>
>>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>>> Suggestion to use:
>>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>>> in order tests may be run with root.
>>>>>
>>>>> I would prefer to choose a path that can be written by anyone else to
>>>>> allow the push_server
>>>>> to be started without root.
>>>>
>>>> Sorry, I meant *without* root.
>>>> /run/user/<UID> allows for that, it's writable by the user that
>>>> started wokd, be it root or not.
>>>>
>>> This change would require further UI changes to allow the /config API
>>> (or other) to inform
>>> the UI of the current websocket URL. This will not solve the problems
>>> I've seen with the unit tests
>>> though - multiple instances of the push_server will not be possible and
>>> the unit tests will
>>> break.
>>
>> I'm not seeing the relation between that BASE_DIR and the URL used by
>> UI. Anyway, I believe we both agree that just one server and one URL
>> is fine.
> The unix socket, now /run/woknotifications, is used by the push server
> to send
> notifications. The websocket creates a proxy between this unix socket and a
> common TCP socket that the UI can read/write to.
>
> This connection is made by the following URL:
>
> <host:port>/websockify?token=<token>
>
> The UI knows what token to use to connect to the unix socket because the
> token is based on the file path of the unix socket. This is the relevant UI
> code:
>
> var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>
>
> This means that if I change the file path to /run/user_id, the token
> changes too. If the UI can't
> predict what unix socket is being used in WoK then we need to supply
> this info to the UI
> in another way.

I see, perhaps a getWebsocketURI API could solve it.


>
>>
>> My only concern is the directory to save that woknotifications file
>> and the permissions required for it.
>>
>>
>>>
>>> We need to discuss an alternative where:
>>>
>>> - any user can start the push_server, as you require
>>
>> I'm not requiring it. Just saying that if it will be started by tests,
>> it needs to run without sudo (i.e. use paths in filesystems that allow
>> for that).
>>
>> As I mentioned earlier, testing for 'test' option is not enough to
>> avoid PushServer to be started during tests, since as of now 'test'
>> option only governs which model will be used.
>
> Strange, it worked for me. I think you're referring to WoK changes that
> were made
> right after/before I sent this patch set. I'll re-test it and see how I
> can prevent it
> from running on test mode.

Those changes were made by Aline's patches last week or the week before.
We had a specific model parameter which was removed and now the model is 
governed by test_mode.

>
>>
>>
>>>
>>> - if we're to allow the push_server to be run in test_mode*, we need to
>>> think in a way of
>>> generating random paths that can be written by any user as well. You are
>>> working closely
>>> in Debian changes. What system dir can be used that can be written by
>>> any user and,
>>> preferably, exists in RPM distros too?
>>
>> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to store
>> runtime data by libvirt when it is started with "qemu:///session" url
>> (regular user session).
>>
>> In fact it seems to be default place for that kind of usage in both
>> distros:
>>
>> Ubuntu 16.04:
>> $ ls /run/user/1000/
>> dbus-session  gnome-shell  gvfs-burn  pulse              systemd
>> upstart-dbus-bridge.4451.pid  upstart-udev-bridge.2946.pid
>> dconf         gvfs         keyring    speech-dispatcher  upstart
>> upstart-file-bridge.4451.pid  upstart-udev-bridge.4451.pid
>>
>> Fedora 25:
>> $ ls /run/user/1000/
>> bus  libvirt  systemd
>
> I see. Perhaps changing the dir to /run/user/<UID> is the right thing to
> do in
> this case. This means that we can't avoid "breaking" the current design
> - we'll
> need to use an additional API to inform the UI of the current unix
> socket (token)
> being used.
>
>
Yes, the URI will still be "fixed" in the sense it's defined on server 
startup (based on the user which started it) and does not change until 
wok server is stopped.


-- 
Lucio Correia

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/28/2017 10:43 AM, Lucio Correia wrote:
> On 28/02/2017 10:24, Daniel Henrique Barboza wrote:
>>
>>
>> On 02/28/2017 10:02 AM, Lucio Correia wrote:
>>> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>>>
>>>>
>>>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>>>> Suggestion to use:
>>>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>>>> in order tests may be run with root.
>>>>>>
>>>>>> I would prefer to choose a path that can be written by anyone 
>>>>>> else to
>>>>>> allow the push_server
>>>>>> to be started without root.
>>>>>
>>>>> Sorry, I meant *without* root.
>>>>> /run/user/<UID> allows for that, it's writable by the user that
>>>>> started wokd, be it root or not.
>>>>>
>>>> This change would require further UI changes to allow the /config API
>>>> (or other) to inform
>>>> the UI of the current websocket URL. This will not solve the problems
>>>> I've seen with the unit tests
>>>> though - multiple instances of the push_server will not be possible 
>>>> and
>>>> the unit tests will
>>>> break.
>>>
>>> I'm not seeing the relation between that BASE_DIR and the URL used by
>>> UI. Anyway, I believe we both agree that just one server and one URL
>>> is fine.
>> The unix socket, now /run/woknotifications, is used by the push server
>> to send
>> notifications. The websocket creates a proxy between this unix socket 
>> and a
>> common TCP socket that the UI can read/write to.
>>
>> This connection is made by the following URL:
>>
>> <host:port>/websockify?token=<token>
>>
>> The UI knows what token to use to connect to the unix socket because the
>> token is based on the file path of the unix socket. This is the 
>> relevant UI
>> code:
>>
>> var token = wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, 
>> "");
>>
>>
>> This means that if I change the file path to /run/user_id, the token
>> changes too. If the UI can't
>> predict what unix socket is being used in WoK then we need to supply
>> this info to the UI
>> in another way.
>
> I see, perhaps a getWebsocketURI API could solve it.
>
>

No! The token is only the file name. It is not related to the directory! 
If we keep the file name the same and only change the directory and 
letting the websocket knows it is enough.

>>
>>>
>>> My only concern is the directory to save that woknotifications file
>>> and the permissions required for it.
>>>
>>>
>>>>
>>>> We need to discuss an alternative where:
>>>>
>>>> - any user can start the push_server, as you require
>>>
>>> I'm not requiring it. Just saying that if it will be started by tests,
>>> it needs to run without sudo (i.e. use paths in filesystems that allow
>>> for that).
>>>
>>> As I mentioned earlier, testing for 'test' option is not enough to
>>> avoid PushServer to be started during tests, since as of now 'test'
>>> option only governs which model will be used.
>>
>> Strange, it worked for me. I think you're referring to WoK changes that
>> were made
>> right after/before I sent this patch set. I'll re-test it and see how I
>> can prevent it
>> from running on test mode.
>
> Those changes were made by Aline's patches last week or the week before.
> We had a specific model parameter which was removed and now the model 
> is governed by test_mode.
>
>>
>>>
>>>
>>>>
>>>> - if we're to allow the push_server to be run in test_mode*, we 
>>>> need to
>>>> think in a way of
>>>> generating random paths that can be written by any user as well. 
>>>> You are
>>>> working closely
>>>> in Debian changes. What system dir can be used that can be written by
>>>> any user and,
>>>> preferably, exists in RPM distros too?
>>>
>>> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to store
>>> runtime data by libvirt when it is started with "qemu:///session" url
>>> (regular user session).
>>>
>>> In fact it seems to be default place for that kind of usage in both
>>> distros:
>>>
>>> Ubuntu 16.04:
>>> $ ls /run/user/1000/
>>> dbus-session  gnome-shell  gvfs-burn  pulse systemd
>>> upstart-dbus-bridge.4451.pid  upstart-udev-bridge.2946.pid
>>> dconf         gvfs         keyring    speech-dispatcher upstart
>>> upstart-file-bridge.4451.pid  upstart-udev-bridge.4451.pid
>>>
>>> Fedora 25:
>>> $ ls /run/user/1000/
>>> bus  libvirt  systemd
>>
>> I see. Perhaps changing the dir to /run/user/<UID> is the right thing to
>> do in
>> this case. This means that we can't avoid "breaking" the current design
>> - we'll
>> need to use an additional API to inform the UI of the current unix
>> socket (token)
>> being used.
>>
>>
> Yes, the URI will still be "fixed" in the sense it's defined on server 
> startup (based on the user which started it) and does not change until 
> wok server is stopped.
>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/28/2017 10:02 AM, Lucio Correia wrote:
> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>
>>
>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>> Suggestion to use:
>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>> in order tests may be run with root.
>>>>
>>>> I would prefer to choose a path that can be written by anyone else to
>>>> allow the push_server
>>>> to be started without root.
>>>
>>> Sorry, I meant *without* root.
>>> /run/user/<UID> allows for that, it's writable by the user that
>>> started wokd, be it root or not.
>>>
>> This change would require further UI changes to allow the /config API
>> (or other) to inform
>> the UI of the current websocket URL. This will not solve the problems
>> I've seen with the unit tests
>> though - multiple instances of the push_server will not be possible and
>> the unit tests will
>> break.
>
> I'm not seeing the relation between that BASE_DIR and the URL used by 
> UI. Anyway, I believe we both agree that just one server and one URL 
> is fine.
>
> My only concern is the directory to save that woknotifications file 
> and the permissions required for it.
>
>

If the websockets directory path would be returned by a function in 
config.py (for example) the tests may overwrite that to set a /tmp dir 
and that way allowing it to run without root permissions.

Daniel helped me to use @mock.patch and seems it can be a good way to go 
to solve it.

>>
>> We need to discuss an alternative where:
>>
>> - any user can start the push_server, as you require
>
> I'm not requiring it. Just saying that if it will be started by tests, 
> it needs to run without sudo (i.e. use paths in filesystems that allow 
> for that).
>
> As I mentioned earlier, testing for 'test' option is not enough to 
> avoid PushServer to be started during tests, since as of now 'test' 
> option only governs which model will be used.
>
>
>>
>> - if we're to allow the push_server to be run in test_mode*, we need to
>> think in a way of
>> generating random paths that can be written by any user as well. You are
>> working closely
>> in Debian changes. What system dir can be used that can be written by
>> any user and,
>> preferably, exists in RPM distros too?
>
> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to store 
> runtime data by libvirt when it is started with "qemu:///session" url 
> (regular user session).
>
> In fact it seems to be default place for that kind of usage in both 
> distros:
>
> Ubuntu 16.04:
> $ ls /run/user/1000/
> dbus-session  gnome-shell  gvfs-burn  pulse              systemd 
> upstart-dbus-bridge.4451.pid  upstart-udev-bridge.2946.pid
> dconf         gvfs         keyring    speech-dispatcher  upstart 
> upstart-file-bridge.4451.pid  upstart-udev-bridge.4451.pid
>
> Fedora 25:
> $ ls /run/user/1000/
> bus  libvirt  systemd
>
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago
Just realized studying the code that, perhaps, the websockify token 
considers
only the file name, not the file path. This could mean that we can choose
the dir of the unix socket freely, as long as the canonical name stays
the same. So /run/woknotifications and /run/user/<u-id>/woknotifications
would be accessed by the same URL. I'll investigate and see if it's true.

Let me summarize here what I am planning for v2 reading all the replies:

- Change the socket dir with as little backend changes as possible. 
Assess if
maintaining the same file name allows for the same token (URI) in the UI,
avoiding using another API to deliver the token;

- Change the push server initialization. Move it to server.py to avoid 
execution
in the unit tests, but allow execution in the test mode. Do the same 
treatment
with the ws_proxy initialization;

- Simplify the UI code. I think there's a way to get rid of the listener 
identifier
without losing functionality;

- Insert notifications in base model level. Put code in base WoK classes 
to allow notifications
for every model event without the need of adding it manually;

- Standardization of messages sent. Send messages in the pattern 
plugin/entity/action;

- Minor code fixes (imports, remove kimchi references, etc).


Let me know if I am missing something.


Daniel


On 02/28/2017 10:27 AM, Aline Manera wrote:
>
>
> On 02/28/2017 10:02 AM, Lucio Correia wrote:
>> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>>
>>>
>>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>>> Suggestion to use:
>>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>>> in order tests may be run with root.
>>>>>
>>>>> I would prefer to choose a path that can be written by anyone else to
>>>>> allow the push_server
>>>>> to be started without root.
>>>>
>>>> Sorry, I meant *without* root.
>>>> /run/user/<UID> allows for that, it's writable by the user that
>>>> started wokd, be it root or not.
>>>>
>>> This change would require further UI changes to allow the /config API
>>> (or other) to inform
>>> the UI of the current websocket URL. This will not solve the problems
>>> I've seen with the unit tests
>>> though - multiple instances of the push_server will not be possible and
>>> the unit tests will
>>> break.
>>
>> I'm not seeing the relation between that BASE_DIR and the URL used by 
>> UI. Anyway, I believe we both agree that just one server and one URL 
>> is fine.
>>
>> My only concern is the directory to save that woknotifications file 
>> and the permissions required for it.
>>
>>
>
> If the websockets directory path would be returned by a function in 
> config.py (for example) the tests may overwrite that to set a /tmp dir 
> and that way allowing it to run without root permissions.
>
> Daniel helped me to use @mock.patch and seems it can be a good way to 
> go to solve it.
>
>>>
>>> We need to discuss an alternative where:
>>>
>>> - any user can start the push_server, as you require
>>
>> I'm not requiring it. Just saying that if it will be started by 
>> tests, it needs to run without sudo (i.e. use paths in filesystems 
>> that allow for that).
>>
>> As I mentioned earlier, testing for 'test' option is not enough to 
>> avoid PushServer to be started during tests, since as of now 'test' 
>> option only governs which model will be used.
>>
>>
>>>
>>> - if we're to allow the push_server to be run in test_mode*, we need to
>>> think in a way of
>>> generating random paths that can be written by any user as well. You 
>>> are
>>> working closely
>>> in Debian changes. What system dir can be used that can be written by
>>> any user and,
>>> preferably, exists in RPM distros too?
>>
>> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to 
>> store runtime data by libvirt when it is started with 
>> "qemu:///session" url (regular user session).
>>
>> In fact it seems to be default place for that kind of usage in both 
>> distros:
>>
>> Ubuntu 16.04:
>> $ ls /run/user/1000/
>> dbus-session  gnome-shell  gvfs-burn  pulse              systemd 
>> upstart-dbus-bridge.4451.pid  upstart-udev-bridge.2946.pid
>> dconf         gvfs         keyring    speech-dispatcher  upstart 
>> upstart-file-bridge.4451.pid  upstart-udev-bridge.4451.pid
>>
>> Fedora 25:
>> $ ls /run/user/1000/
>> bus  libvirt  systemd
>>
>>
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Lucio Correia 4 years, 2 months ago
On 28/02/2017 11:12, Daniel Henrique Barboza wrote:
> Just realized studying the code that, perhaps, the websockify token
> considers
> only the file name, not the file path. This could mean that we can choose
> the dir of the unix socket freely, as long as the canonical name stays
> the same. So /run/woknotifications and /run/user/<u-id>/woknotifications
> would be accessed by the same URL. I'll investigate and see if it's true.
>
> Let me summarize here what I am planning for v2 reading all the replies:
>
> - Change the socket dir with as little backend changes as possible.
> Assess if
> maintaining the same file name allows for the same token (URI) in the UI,
> avoiding using another API to deliver the token;
>
> - Change the push server initialization. Move it to server.py to avoid
> execution
> in the unit tests, but allow execution in the test mode. Do the same
> treatment
> with the ws_proxy initialization;
>
> - Simplify the UI code. I think there's a way to get rid of the listener
> identifier
> without losing functionality;
>
> - Insert notifications in base model level. Put code in base WoK classes
> to allow notifications
> for every model event without the need of adding it manually;
>
> - Standardization of messages sent. Send messages in the pattern
> plugin/entity/action;
>
> - Minor code fixes (imports, remove kimchi references, etc).
>
>
> Let me know if I am missing something.
>
>
> Daniel

+1

-- 
Lucio Correia

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Daniel Henrique Barboza 4 years, 2 months ago
In time:

On 02/28/2017 11:15 AM, Lucio Correia wrote:
> On 28/02/2017 11:12, Daniel Henrique Barboza wrote:
>> Just realized studying the code that, perhaps, the websockify token
>> considers
>> only the file name, not the file path. This could mean that we can 
>> choose
>> the dir of the unix socket freely, as long as the canonical name stays
>> the same. So /run/woknotifications and /run/user/<u-id>/woknotifications
>> would be accessed by the same URL. I'll investigate and see if it's 
>> true.
>>
>> Let me summarize here what I am planning for v2 reading all the replies:
>>
>> - Change the socket dir with as little backend changes as possible.
>> Assess if
>> maintaining the same file name allows for the same token (URI) in the 
>> UI,
>> avoiding using another API to deliver the token;
Amend in this item: consider using config.py.in to retrieve the dir, 
avoiding
hardcoding the backend with the directory.


>>
>> - Change the push server initialization. Move it to server.py to avoid
>> execution
>> in the unit tests, but allow execution in the test mode. Do the same
>> treatment
>> with the ws_proxy initialization;
>>
>> - Simplify the UI code. I think there's a way to get rid of the listener
>> identifier
>> without losing functionality;
>>
>> - Insert notifications in base model level. Put code in base WoK classes
>> to allow notifications
>> for every model event without the need of adding it manually;
>>
>> - Standardization of messages sent. Send messages in the pattern
>> plugin/entity/action;
>>
>> - Minor code fixes (imports, remove kimchi references, etc).
>>
>>
>> Let me know if I am missing something.
>>
>>
>> Daniel
>
> +1
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago
+1

On 02/28/2017 11:12 AM, Daniel Henrique Barboza wrote:
> Just realized studying the code that, perhaps, the websockify token 
> considers
> only the file name, not the file path. This could mean that we can choose
> the dir of the unix socket freely, as long as the canonical name stays
> the same. So /run/woknotifications and /run/user/<u-id>/woknotifications
> would be accessed by the same URL. I'll investigate and see if it's true.
>
> Let me summarize here what I am planning for v2 reading all the replies:
>
> - Change the socket dir with as little backend changes as possible. 
> Assess if
> maintaining the same file name allows for the same token (URI) in the UI,
> avoiding using another API to deliver the token;
>
> - Change the push server initialization. Move it to server.py to avoid 
> execution
> in the unit tests, but allow execution in the test mode. Do the same 
> treatment
> with the ws_proxy initialization;
>
> - Simplify the UI code. I think there's a way to get rid of the 
> listener identifier
> without losing functionality;
>
> - Insert notifications in base model level. Put code in base WoK 
> classes to allow notifications
> for every model event without the need of adding it manually;
>
> - Standardization of messages sent. Send messages in the pattern 
> plugin/entity/action;
>
> - Minor code fixes (imports, remove kimchi references, etc).
>
>
> Let me know if I am missing something.
>
>
> Daniel
>
>
> On 02/28/2017 10:27 AM, Aline Manera wrote:
>>
>>
>> On 02/28/2017 10:02 AM, Lucio Correia wrote:
>>> On 27/02/2017 18:30, Daniel Henrique Barboza wrote:
>>>>
>>>>
>>>> On 02/27/2017 06:18 PM, Lucio Correia wrote:
>>>>> On 27/02/2017 18:12, Daniel Henrique Barboza wrote:
>>>>>>>>> +BASE_DIRECTORY = '/run'
>>>>>>>> Suggestion to use:
>>>>>>>> os.path.join('/run/user', str(os.getuid()))
>>>>>>>> in order tests may be run with root.
>>>>>>
>>>>>> I would prefer to choose a path that can be written by anyone 
>>>>>> else to
>>>>>> allow the push_server
>>>>>> to be started without root.
>>>>>
>>>>> Sorry, I meant *without* root.
>>>>> /run/user/<UID> allows for that, it's writable by the user that
>>>>> started wokd, be it root or not.
>>>>>
>>>> This change would require further UI changes to allow the /config API
>>>> (or other) to inform
>>>> the UI of the current websocket URL. This will not solve the problems
>>>> I've seen with the unit tests
>>>> though - multiple instances of the push_server will not be possible 
>>>> and
>>>> the unit tests will
>>>> break.
>>>
>>> I'm not seeing the relation between that BASE_DIR and the URL used 
>>> by UI. Anyway, I believe we both agree that just one server and one 
>>> URL is fine.
>>>
>>> My only concern is the directory to save that woknotifications file 
>>> and the permissions required for it.
>>>
>>>
>>
>> If the websockets directory path would be returned by a function in 
>> config.py (for example) the tests may overwrite that to set a /tmp 
>> dir and that way allowing it to run without root permissions.
>>
>> Daniel helped me to use @mock.patch and seems it can be a good way to 
>> go to solve it.
>>
>>>>
>>>> We need to discuss an alternative where:
>>>>
>>>> - any user can start the push_server, as you require
>>>
>>> I'm not requiring it. Just saying that if it will be started by 
>>> tests, it needs to run without sudo (i.e. use paths in filesystems 
>>> that allow for that).
>>>
>>> As I mentioned earlier, testing for 'test' option is not enough to 
>>> avoid PushServer to be started during tests, since as of now 'test' 
>>> option only governs which model will be used.
>>>
>>>
>>>>
>>>> - if we're to allow the push_server to be run in test_mode*, we 
>>>> need to
>>>> think in a way of
>>>> generating random paths that can be written by any user as well. 
>>>> You are
>>>> working closely
>>>> in Debian changes. What system dir can be used that can be written by
>>>> any user and,
>>>> preferably, exists in RPM distros too?
>>>
>>> On Fedora 25, /run/user/<UID>/libvirt is the default dir used to 
>>> store runtime data by libvirt when it is started with 
>>> "qemu:///session" url (regular user session).
>>>
>>> In fact it seems to be default place for that kind of usage in both 
>>> distros:
>>>
>>> Ubuntu 16.04:
>>> $ ls /run/user/1000/
>>> dbus-session  gnome-shell  gvfs-burn  pulse systemd 
>>> upstart-dbus-bridge.4451.pid upstart-udev-bridge.2946.pid
>>> dconf         gvfs         keyring    speech-dispatcher upstart 
>>> upstart-file-bridge.4451.pid upstart-udev-bridge.4451.pid
>>>
>>> Fedora 25:
>>> $ ls /run/user/1000/
>>> bus  libvirt  systemd
>>>
>>>
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel

Re: [Kimchi-devel] [PATCH] [WoK] Asynchronous UI notification implementation

Posted by Aline Manera 4 years, 2 months ago

On 02/27/2017 06:12 PM, Daniel Henrique Barboza wrote:
>
>
> On 02/27/2017 11:41 AM, Aline Manera wrote:
>>
>>
>> On 02/27/2017 11:35 AM, Lucio Correia wrote:
>>> Hi Daniel, that is great feature, see my comments below.
>>>
>>> On 24/02/2017 10:22, dhbarboza82@gmail.com wrote:
>>>> From: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>>>
>>>> This patch makes backend and UI changes to implement the asynchronous
>>>> UI notification in WoK.
>>>>
>>>> - Backend:
>>>>
>>>> A push server was implemented from scratch to manage the opened 
>>>> websocket
>>>> connections. The push server connects to the /run/woknotifications
>>>> UNIX socket and broadcasts all messages to all connections.
>>>>
>>>> The websocket module is the same module that exists in the Kimchi
>>>> plug-in. The idea is to remove the module from Kimchi and make it
>>>> use the module from WoK. ws_proxy initialization was also added
>>>> in src/wok/server.py.
>>>>
>>>> - Frontend:
>>>>
>>>> In ui/js/wok.main.js two new functions were added to help the
>>>> usage of asynchronous notifications in the frontend. The idea:
>>>> a single websocket is opened per session. This opened websocket
>>>> will broadcast all incoming messages to all listeners registered.
>>>> Listeners can be added by the new wok.addNotificationListener()
>>>> method. This method will clean up any registered listener by
>>>> itself when the user changes tabs/URL.
>>>>
>>>> The single websocket sends heartbeats to the backend side each
>>>> 30 seconds. No reply from the backend is issued or expected. This
>>>> heartbeat is just a way to ensure that the browser does not
>>>> close the connection due to inactivity. This behavior varies from
>>>> browser to browser but this 30 second heartbeat is more than enough
>>>> to ensure that the websocket is kept alive.
>>>>
>>>> - Working example in User Log:
>>>>
>>>> A simple usage is provided in this patch. A change was made in
>>>> src/wok/reqlogger.py to send an asynchronous notification each
>>>> time a new log entry is created. In ui/js/wok.user-log.js a
>>>> websocket listener is added using wok.addNotificationListener()
>>>> and, for each message that indicates a new user log entry, a
>>>> refresh in the listing is issued.
>>>>
>>>> Signed-off-by: Daniel Henrique Barboza <danielhb@linux.vnet.ibm.com>
>>>> ---
>>>>  src/wok/model/notifications.py |  18 +++++-
>>>>  src/wok/pushserver.py          | 132 
>>>> +++++++++++++++++++++++++++++++++++++++++
>>>>  src/wok/reqlogger.py           |   7 ++-
>>>>  src/wok/server.py              |   8 ++-
>>>>  src/wok/websocket.py           | 123 
>>>> ++++++++++++++++++++++++++++++++++++++
>>>>  ui/js/src/wok.main.js          |  38 ++++++++++++
>>>>  ui/js/wok.user-log.js          |   6 ++
>>>>  7 files changed, 327 insertions(+), 5 deletions(-)
>>>>  create mode 100644 src/wok/pushserver.py
>>>>  create mode 100644 src/wok/websocket.py
>>>>
>>>> diff --git a/src/wok/model/notifications.py 
>>>> b/src/wok/model/notifications.py
>>>> index bdb7c78..597eac5 100644
>>>> --- a/src/wok/model/notifications.py
>>>> +++ b/src/wok/model/notifications.py
>>>> @@ -1,7 +1,7 @@
>>>>  #
>>>>  # Project Wok
>>>>  #
>>>> -# Copyright IBM Corp, 2016
>>>> +# Copyright IBM Corp, 2016-2017
>>>>  #
>>>>  # This library is free software; you can redistribute it and/or
>>>>  # modify it under the terms of the GNU Lesser General Public
>>>> @@ -19,12 +19,22 @@
>>>>
>>>>  from datetime import datetime
>>>>
>>>> +from wok import config
>>>>  from wok.exception import NotFoundError, OperationFailed
>>>>  from wok.message import WokMessage
>>>> +from wok.pushserver import PushServer
>>>>  from wok.utils import wok_log
>>>>
>>>>
>>>>  notificationsStore = {}
>>>> +push_server = None
>>>> +
>>>> +
>>>> +def send_websocket_notification(message):
>>>> +    global push_server
>>>> +
>>>> +    if push_server:
>>>> +        push_server.send_notification(message)
>>>>
>>>>
>>>>  def add_notification(code, args=None, plugin_name=None):
>>>> @@ -57,7 +67,11 @@ def del_notification(code):
>>>>
>>>>  class NotificationsModel(object):
>>>>      def __init__(self, **kargs):
>>>> -        pass
>>>> +        global push_server
>>>> +
>>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>>> 'true'
>>>
>>> This will cause some tests to initialize pushserver, since 'test' 
>>> option is now tied to which model runs (mockmodel or model). Problem 
>>> is that now wok tests run without sudo (patch in ML), and 
>>> pushserver's BASE_DIRECTORY is only writable with sudo permission. 
>>> Gave a suggestion there.
>
> Current design is that the websocket_proxy isn't initiated in test_mode.
>

I have replied to another thread. But briefly, the websocket_proxy could 
be only required on non-test mode for Kimchi, but now we are 
implementing a feature which needs to be available on both mode.

>
>>>
>>>
>>>> +        if not test_mode:
>>>> +            push_server = PushServer()
>>> All 'users' of functionality will have their own instance of 
>>> pushserver (each with a "while True" running)? Why not a single 
>>> instance used by everybody?
>>>
>
> One WoK instance will have it's own push server. The constructor of 
> NotificationsModel is called
> only once per WoK init.
>

It is a backend-UI server, so it only makes sense when the UI is up, ie, 
when the web server is running. So adding it to the NotificationsModel 
may lead a model initialization without the server which does not make 
sense. So moving it to Server initialization may fix that and avoid 
problems on test_model*

> Also, I haven't prepared this feature to be run in a scenario of 
> multiple WoK instances running at the
> same time - since it's always the same unix socket used, multiples 
> instances can't be launched at
> once.
>

I don't think we need to worry about that. Only one instance may be running.

> To allow multiple push servers to be started simultaneously I would 
> need to make the unix socket
> randomly generated. This means that the websocket URL would change. 
> The UI then would need
> a way to discover the current websocket URL to be used, probably using 
> the /config API. It is feasible,
> but more changes would need to be made.
>
>>
>> Good point! Maybe it is better to move it do Server() to initiate the 
>> websockets connections to everyone. Also there is no need to 
>> distinguish test mode.
>
> Websockets connections are available to everyone as is.
>
> As I said in my previous reply, test_mode is already being 
> distinguished in the websocket initialization
> of Kimchi:
>
> root.py line 48:
>
>         # When running on test mode, specify the objectstore location to
>         # remove the file on server shutting down. That way, the 
> system will
>         # not suffer any change while running on test mode
>         if wok_options.test and (wok_options.test is True or
>                                  wok_options.test.lower() == 'true'):
>             self.objectstore_loc = tempfile.mktemp()
>             self.model = mockmodel.MockModel(self.objectstore_loc)
>
>             def remove_objectstore():
>                 if os.path.exists(self.objectstore_loc):
>                     os.unlink(self.objectstore_loc)
>             cherrypy.engine.subscribe('exit', remove_objectstore)
>         else:
>             self.model = kimchiModel.Model()
>             ws_proxy = websocket.new_ws_proxy()
>             cherrypy.engine.subscribe('exit', ws_proxy.terminate)
>
>
> I assumed that this design was intended and I haven't thought of any
> good reason to change it, so this design was considered in the
> push_server and also in the websocket initiation in WoK.
>
>
>>
>>>
>>>>
>>>>      def get_list(self):
>>>>          global notificationsStore
>>>> diff --git a/src/wok/pushserver.py b/src/wok/pushserver.py
>>>> new file mode 100644
>>>> index 0000000..8993f00
>>>> --- /dev/null
>>>> +++ b/src/wok/pushserver.py
>>>> @@ -0,0 +1,132 @@
>>>> +#
>>>> +# Project Wok
>>>> +#
>>>> +# Copyright IBM Corp, 2017
>>>> +#
>>>> +# This library is free software; you can redistribute it and/or
>>>> +# modify it under the terms of the GNU Lesser General Public
>>>> +# License as published by the Free Software Foundation; either
>>>> +# version 2.1 of the License, or (at your option) any later version.
>>>> +#
>>>> +# This library is distributed in the hope that it will be useful,
>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>>> +# Lesser General Public License for more details.
>>>> +#
>>>> +# You should have received a copy of the GNU Lesser General Public
>>>> +# License along with this library; if not, write to the Free Software
>>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>>> 02110-1301 USA
>>>> +#
>>>> +
>>>> +import cherrypy
>>>> +import os
>>>> +import select
>>>> +import socket
>>>> +import threading
>>>> +
>>>> +import websocket
>>>> +from utils import wok_log
>>>> +
>>>> +
>>>> +BASE_DIRECTORY = '/run'
>>> Suggestion to use:
>>> os.path.join('/run/user', str(os.getuid()))
>>> in order tests may be run with root.
>
> I would prefer to choose a path that can be written by anyone else to 
> allow the push_server
> to be started without root.
>
>
>>>
>>>> +TOKEN_NAME = 'woknotifications'
>>>> +
>>>> +
>>>> +class PushServer(object):
>>>> +
>>>> +    def set_socket_file(self):
>>>> +        if not os.path.isdir(BASE_DIRECTORY):
>>>> +            try:
>>>> +                os.mkdir(BASE_DIRECTORY)
>>>> +            except OSError:
>>>> +                raise RuntimeError('PushServer base UNIX socket 
>>>> dir %s '
>>>> +                                   'not found.' % BASE_DIRECTORY)
>>>> +
>>>> +        self.server_addr = os.path.join(BASE_DIRECTORY, TOKEN_NAME)
>>>> +
>>>> +        if os.path.exists(self.server_addr):
>>>> +            try:
>>>> +                os.remove(self.server_addr)
>>>> +            except:
>>>> +                raise RuntimeError('There is an existing 
>>>> connection in %s' %
>>>> +                                   self.server_addr)
>>>> +
>>>> +    def __init__(self):
>>>> +        self.set_socket_file()
>>>> +
>>>> +        websocket.add_proxy_token(TOKEN_NAME, self.server_addr, True)
>>>> +
>>>> +        self.connections = []
>>>> +
>>>> +        self.server_running = True
>>>> +        self.server_socket = socket.socket(socket.AF_UNIX, 
>>>> socket.SOCK_STREAM)
>>>> +        self.server_socket.setsockopt(socket.SOL_SOCKET,
>>>> +                                      socket.SO_REUSEADDR, 1)
>>>> +        self.server_socket.bind(self.server_addr)
>>>> +        self.server_socket.listen(10)
>>>> +        wok_log.info('Push server created on address %s' % 
>>>> self.server_addr)
>>>> +
>>>> +        self.connections.append(self.server_socket)
>>>> +        cherrypy.engine.subscribe('stop', self.close_server, 1)
>>>> +
>>>> +        server_loop = threading.Thread(target=self.listen)
>>>> +        server_loop.start()
>>>> +
>>>> +    def listen(self):
>>>> +        try:
>>>> +            while self.server_running:
>>>> +                read_ready, _, _ = select.select(self.connections,
>>>> +                                                 [], [], 1)
>>>> +                for sock in read_ready:
>>>> +                    if not self.server_running:
>>>> +                        break
>>>> +
>>>> +                    if sock == self.server_socket:
>>>> +
>>>> +                        new_socket, addr = 
>>>> self.server_socket.accept()
>>>> +                        self.connections.append(new_socket)
>>>> +                    else:
>>>> +                        try:
>>>> +                            data = sock.recv(4096)
>>>> +                        except:
>>>> +                            try:
>>>> + self.connections.remove(sock)
>>>> +                            except ValueError:
>>>> +                                pass
>>>> +
>>>> +                            continue
>>>> +                        if data and data == 'CLOSE':
>>>> +                            sock.send('ACK')
>>>> +                            try:
>>>> + self.connections.remove(sock)
>>>> +                            except ValueError:
>>>> +                                pass
>>>> +                            sock.close()
>>>> +
>>>> +        except Exception as e:
>>>> +            raise RuntimeError('Exception ocurred in listen() of 
>>>> pushserver '
>>>> +                               'module: %s' % e.message)
>>>> +
>>>> +    def send_notification(self, message):
>>>> +        for sock in self.connections:
>>>> +            if sock != self.server_socket:
>>>> +                try:
>>>> +                    sock.send(message)
>>>> +                except IOError as e:
>>>> +                    if 'Broken pipe' in str(e):
>>>> +                        sock.close()
>>>> +                        try:
>>>> +                            self.connections.remove(sock)
>>>> +                        except ValueError:
>>>> +                            pass
>>>> +
>>>> +    def close_server(self):
>>>> +        try:
>>>> +            self.server_running = False
>>>> +            self.server_socket.shutdown(socket.SHUT_RDWR)
>>>> +            self.server_socket.close()
>>>> +            os.remove(self.server_addr)
>>>> +        except:
>>>> +            pass
>>>> +        finally:
>>>> +            cherrypy.engine.unsubscribe('stop', self.close_server)
>>>> diff --git a/src/wok/reqlogger.py b/src/wok/reqlogger.py
>>>> index 92e155d..1b774e2 100644
>>>> --- a/src/wok/reqlogger.py
>>>> +++ b/src/wok/reqlogger.py
>>>> @@ -1,7 +1,7 @@
>>>>  #
>>>>  # Project Wok
>>>>  #
>>>> -# Copyright IBM Corp, 2016
>>>> +# Copyright IBM Corp, 2016-2017
>>>>  #
>>>>  # This library is free software; you can redistribute it and/or
>>>>  # modify it under the terms of the GNU Lesser General Public
>>>> @@ -34,6 +34,7 @@ from wok.auth import USER_NAME
>>>>  from wok.config import get_log_download_path, paths
>>>>  from wok.exception import InvalidParameter, OperationFailed
>>>>  from wok.message import WokMessage
>>>> +from wok.model.notifications import send_websocket_notification
>>>>  from wok.stringutils import ascii_dict
>>>>  from wok.utils import remove_old_files
>>>>
>>>> @@ -68,6 +69,8 @@ WOK_REQUEST_LOGGER = "wok_request_logger"
>>>>  # AsyncTask handling
>>>>  ASYNCTASK_REQUEST_METHOD = 'TASK'
>>>>
>>>> +NEW_LOG_ENTRY_MESSAGE = 'new_log_entry'
>>>> +
>>>>
>>>>  def log_request(code, params, exception, method, status, app=None, 
>>>> user=None,
>>>>                  ip=None):
>>>> @@ -114,6 +117,8 @@ def log_request(code, params, exception, 
>>>> method, status, app=None, user=None,
>>>>          ip=ip
>>>>      ).log()
>>>>
>>>> +    send_websocket_notification(NEW_LOG_ENTRY_MESSAGE)
>>>> +
>>>>      return log_id
>>>>
>>>>
>>>> diff --git a/src/wok/server.py b/src/wok/server.py
>>>> index fc2e167..2d823c9 100644
>>>> --- a/src/wok/server.py
>>>> +++ b/src/wok/server.py
>>>> @@ -25,8 +25,7 @@ import logging
>>>>  import logging.handlers
>>>>  import os
>>>>
>>>> -from wok import auth
>>>> -from wok import config
>>>> +from wok import auth, config, websocket
>>>>  from wok.config import config as configParser
>>>>  from wok.config import WokConfig
>>>>  from wok.control import sub_nodes
>>>> @@ -159,6 +158,11 @@ class Server(object):
>>>>          cherrypy.tree.mount(WokRoot(model.Model(), dev_env),
>>>>                              options.server_root, self.configObj)
>>>>
>>>> +        test_mode = config.config.get('server', 'test').lower() == 
>>>> 'true'
>>>> +        if not test_mode:
>>>> +            ws_proxy = websocket.new_ws_proxy()
>>>> +            cherrypy.engine.subscribe('exit', ws_proxy.terminate)
>>>> +
>>>>          self._load_plugins()
>>>>          cherrypy.lib.sessions.init()
>>>>
>>>> diff --git a/src/wok/websocket.py b/src/wok/websocket.py
>>>> new file mode 100644
>>>> index 0000000..5d7fb91
>>>> --- /dev/null
>>>> +++ b/src/wok/websocket.py
>>>> @@ -0,0 +1,123 @@
>>>> +#!/usr/bin/env python2
>>>> +#
>>>> +# Project Wok
>>>> +#
>>>> +# Copyright IBM Corp, 2017
>>>> +#
>>>> +# Code derived from Project Kimchi
>>>> +#
>>>> +# This library is free software; you can redistribute it and/or
>>>> +# modify it under the terms of the GNU Lesser General Public
>>>> +# License as published by the Free Software Foundation; either
>>>> +# version 2.1 of the License, or (at your option) any later version.
>>>> +#
>>>> +# This library is distributed in the hope that it will be useful,
>>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>>> +# Lesser General Public License for more details.
>>>> +#
>>>> +# You should have received a copy of the GNU Lesser General Public
>>>> +# License along with this library; if not, write to the Free Software
>>>> +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  
>>>> 02110-1301 USA
>>>> +
>>>> +import base64
>>>> +import errno
>>>> +import os
>>>> +
>>>> +from multiprocessing import Process
>>>> +from websockify import WebSocketProxy
>>> Add dep on websockify to Wok
>
> Good catch. I'll add it in v2.
>
>>>
>>>
>>>> +
>>>> +from config import config, PluginPaths
>>>> +
>>>> +
>>>> +try:
>>>> +    from websockify.token_plugins import TokenFile
>>>> +    tokenFile = True
>>>> +except ImportError:
>>>> +    tokenFile = False
>>>> +
>>>> +try:
>>>> +    from websockify import ProxyRequestHandler as request_proxy
>>>> +except:
>>>> +    from websockify import WebSocketProxy as request_proxy
>>>> +
>>>> +
>>>> +WS_TOKENS_DIR = os.path.join(PluginPaths('kimchi').state_dir, 
>>>> 'ws-tokens')
>>> This should be wok path now, and it would be nice to be inside 
>>> /run/user as commented above.
>>>
>>>
> Good catch again. I'll change the 'Kimchi' specific code in v2.
>
>>>> +
>>>> +
>>>> +class CustomHandler(request_proxy):
>>>> +
>>>> +    def get_target(self, target_plugin, path):
>>>> +        if issubclass(CustomHandler, object):
>>>> +            target = super(CustomHandler, 
>>>> self).get_target(target_plugin,
>>>> + path)
>>>> +        else:
>>>> +            target = request_proxy.get_target(self, target_plugin, 
>>>> path)
>>>> +
>>>> +        if target[0] == 'unix_socket':
>>>> +            try:
>>>> +                self.server.unix_target = target[1]
>>>> +            except:
>>>> +                self.unix_target = target[1]
>>>> +        else:
>>>> +            try:
>>>> +                self.server.unix_target = None
>>>> +            except:
>>>> +                self.unix_target = None
>>>> +        return target
>>>> +
>>>> +
>>>> +def new_ws_proxy():
>>>> +    try:
>>>> +        os.makedirs(WS_TOKENS_DIR, mode=0755)
>>>> +    except OSError as e:
>>>> +        if e.errno == errno.EEXIST:
>>>> +            pass
>>>> +
>>>> +    params = {'listen_host': '127.0.0.1',
>>>> +              'listen_port': config.get('server', 'websockets_port'),
>>>> +              'ssl_only': False}
>>>> +
>>>> +    # old websockify: do not use TokenFile
>>>> +    if not tokenFile:
>>>> +        params['target_cfg'] = WS_TOKENS_DIR
>>>> +
>>>> +    # websockify 0.7 and higher: use TokenFile
>>>> +    else:
>>>> +        params['token_plugin'] = TokenFile(src=WS_TOKENS_DIR)
>>>> +
>>>> +    def start_proxy():
>>>> +        try:
>>>> +            server = 
>>>> WebSocketProxy(RequestHandlerClass=CustomHandler,
>>>> +                                    **params)
>>>> +        except TypeError:
>>>> +            server = CustomHandler(**params)
>>>> +
>>>> +        server.start_server()
>>>> +
>>>> +    proc = Process(target=start_proxy)
>>>> +    proc.start()
>>>> +    return proc
>>>> +
>>>> +
>>>> +def add_proxy_token(name, port, is_unix_socket=False):
>>>> +    with open(os.path.join(WS_TOKENS_DIR, name), 'w') as f:
>>>> +        """
>>>> +        From python documentation base64.urlsafe_b64encode(s)
>>>> +        substitutes - instead of + and _ instead of / in the
>>>> +        standard Base64 alphabet, BUT the result can still
>>>> +        contain = which is not safe in a URL query component.
>>>> +        So remove it when needed as base64 can work well without it.
>>>> +        """
>>>> +        name = base64.urlsafe_b64encode(name).rstrip('=')
>>>> +        if is_unix_socket:
>>>> +            f.write('%s: unix_socket:%s' % (name.encode('utf-8'), 
>>>> port))
>>>> +        else:
>>>> +            f.write('%s: localhost:%s' % (name.encode('utf-8'), 
>>>> port))
>>>> +
>>>> +
>>>> +def remove_proxy_token(name):
>>>> +    try:
>>>> +        os.unlink(os.path.join(WS_TOKENS_DIR, name))
>>>> +    except OSError:
>>>> +        pass
>>>> diff --git a/ui/js/src/wok.main.js b/ui/js/src/wok.main.js
>>>> index 20c017e..f5031ce 100644
>>>> --- a/ui/js/src/wok.main.js
>>>> +++ b/ui/js/src/wok.main.js
>>>> @@ -29,6 +29,41 @@ wok.getConfig(function(result) {
>>>>      wok.config = {};
>>>>  });
>>>>
>>>> +
>>>> +wok.notificationListeners = {};
>>>> +wok.addNotificationListener = function(name, func) {
>>>> +    wok.notificationListeners[name] = func;
>>>> +    $(window).one("hashchange", function() {
>>>> +        delete wok.notificationListeners[name];
>>>> +    });
>>>> +};
>>>> +
>>>> +wok.notificationsWebSocket = undefined;
>>>> +wok.startNotificationWebSocket = function () {
>>>> +    var addr = window.location.hostname + ':' + window.location.port;
>>>> +    var token = 
>>>> wok.urlSafeB64Encode('woknotifications').replace(/=*$/g, "");
>>>> +    var url = 'wss://' + addr + '/websockify?token=' + token;
>>>> +    wok.notificationsWebSocket = new WebSocket(url, ['base64']);
>>>> +
>>>> +    wok.notificationsWebSocket.onmessage = function(event) {
>>>> +        var message = window.atob(event.data);
>>>> +        for (name in wok.notificationListeners) {
>>>> +            func = wok.notificationListeners[name];
>>>> +            func(message);
>>>> +        }
>>>> +    };
>>>> +
>>>> +    sessionStorage.setItem('wokNotificationWebSocket', 'true');
>>>> +    var heartbeat = setInterval(function() {
>>>> + wok.notificationsWebSocket.send(window.btoa('heartbeat'));
>>>> +    }, 30000);
>>>> +
>>>> +    wok.notificationsWebSocket.onclose = function() {
>>>> +        clearInterval(heartbeat);
>>>> +    };
>>>> +};
>>>> +
>>>> +
>>>>  wok.main = function() {
>>>>      wok.isLoggingOut = false;
>>>>      wok.popable();
>>>> @@ -395,6 +430,9 @@ wok.main = function() {
>>>>
>>>>          // Set handler for help button
>>>>          $('#btn-help').on('click', wok.openHelp);
>>>> +
>>>> +        // start WebSocket
>>>> +        wok.startNotificationWebSocket();
>>>>      };
>>>>
>>>>      var initUI = function() {
>>>> diff --git a/ui/js/wok.user-log.js b/ui/js/wok.user-log.js
>>>> index 0e8fb09..083b6c3 100644
>>>> --- a/ui/js/wok.user-log.js
>>>> +++ b/ui/js/wok.user-log.js
>>>> @@ -153,6 +153,12 @@ wok.initUserLogContent = function() {
>>>>          $("#user-log-grid").bootgrid("search");
>>>>          wok.initUserLogConfigGridData();
>>>>      });
>>>> +
>>>> +    wok.addNotificationListener('userlog', function(message) {
>>>> +        if (message === 'new_log_entry') {
>>>> +            $("#refresh-button").click();
>>>> +        }
>>>> +    });
>>>>  };
>>>>
>>>>  wok.initUserLogWindow = function() {
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Kimchi-devel mailing list
>> Kimchi-devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>
> _______________________________________________
> Kimchi-devel mailing list
> Kimchi-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/kimchi-devel
>

_______________________________________________
Kimchi-devel mailing list
Kimchi-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/kimchi-devel