Unlike the ansi2html module from PyPI, this implementation focuses on
tracking the formatting within a line.  As a result, it provides nice
HTML output for e.g. carriage returns or tabs.  There are many test
cases that were checked against vte, and a huge repetitive CSS (which was
generated by a small program), so don't let the diffstats scare you. :)

Since the sample database does not have ANSI control codes, I took a
Travis log and placed a sample conversion here:
- text source: http://people.redhat.com/pbonzini/log.txt
- HTML output: http://people.redhat.com/pbonzini/log.html

Note that the HTML output is 20% the size of the source, because carriage
returns are optimized out!

Paolo

Paolo Bonzini (9):
  ansi2html: stub implementation
  ansi2html: add basic parsing and cursor movement support
  ansi2html: optimize lines without any control characters
  ansi2html: handle CR-LF like LF
  ansi2html: add support for CSIs other than ESC[m
  ansi2html: add logic to track formatting per-character
  ansi2html: add support for text attributes
  ansi2html: add support for 16-color palette
  ansi2html: add support for 256-color palette

 patchew/logviewer.py     | 391 +++++++++++++++++++++++++++++++++-
 static/css/ansi2html.css | 545 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/test_ansi2html.py  | 295 +++++++++++++++++++++++++
 3 files changed, 1227 insertions(+), 4 deletions(-)
 create mode 100644 static/css/ansi2html.css
 create mode 100644 tests/test_ansi2html.py

-- 
2.14.3

On Mon, 02/26 12:27, Paolo Bonzini wrote:
> Unlike the ansi2html module from PyPI, this implementation focuses on
> tracking the formatting within a line.  As a result, it provides nice
> HTML output for e.g. carriage returns or tabs.  There are many test
> cases that were checked against vte, and a huge repetitive CSS (which was
> generated by a small program), so don't let the diffstats scare you. :)
> 
> Since the sample database does not have ANSI control codes, I took a
> Travis log and placed a sample conversion here:
> - text source: http://people.redhat.com/pbonzini/log.txt
> - HTML output: http://people.redhat.com/pbonzini/log.html
> 
> Note that the HTML output is 20% the size of the source, because carriage
> returns are optimized out!

Now I'm done, thanks.  I am not able to review line by line but I'll trust the
tests and your demo
page. :)

I'm not sure if you want any change for the comment I replied to, so I'll leave
it for you to push the code:

Reviewed-by: Fam Zheng <famz@redhat.com>

On Mon, 02/26 12:27, Paolo Bonzini wrote:
> Unlike the ansi2html module from PyPI, this implementation focuses on
> tracking the formatting within a line.  As a result, it provides nice
> HTML output for e.g. carriage returns or tabs.  There are many test
> cases that were checked against vte, and a huge repetitive CSS (which was
> generated by a small program), so don't let the diffstats scare you. :)
> 
> Since the sample database does not have ANSI control codes, I took a
> Travis log and placed a sample conversion here:
> - text source: http://people.redhat.com/pbonzini/log.txt
> - HTML output: http://people.redhat.com/pbonzini/log.html
> 
> Note that the HTML output is 20% the size of the source, because carriage
> returns are optimized out!

Interesting!

A general question: what about plain text support? For example when we want to
include log snippets to email notifications; and also we probably want to keep
it possible and simple for users to view the full log in their $EDITOR.

Fam

On 27/02/2018 07:48, Fam Zheng wrote:
> On Mon, 02/26 12:27, Paolo Bonzini wrote:
>> Unlike the ansi2html module from PyPI, this implementation focuses on
>> tracking the formatting within a line.  As a result, it provides nice
>> HTML output for e.g. carriage returns or tabs.  There are many test
>> cases that were checked against vte, and a huge repetitive CSS (which was
>> generated by a small program), so don't let the diffstats scare you. :)
>>
>> Since the sample database does not have ANSI control codes, I took a
>> Travis log and placed a sample conversion here:
>> - text source: http://people.redhat.com/pbonzini/log.txt
>> - HTML output: http://people.redhat.com/pbonzini/log.html
>>
>> Note that the HTML output is 20% the size of the source, because carriage
>> returns are optimized out!
> 
> Interesting!
> 
> A general question: what about plain text support? For example when we want to
> include log snippets to email notifications; and also we probably want to keep
> it possible and simple for users to view the full log in their $EDITOR.

Yes, it is easy to split the converter into an abstract class with
text-output and HTML-output subclasses.  When emitting the TestingReport
event, the testing module can pass the logs through the text-output
converter.  While at it, it could only print the last 50-ish lines of
the output (per https://github.com/patchew-project/patchew/issues/31).

Thanks,

Paolo

On Tue, 02/27 10:08, Paolo Bonzini wrote:
> On 27/02/2018 07:48, Fam Zheng wrote:
> > On Mon, 02/26 12:27, Paolo Bonzini wrote:
> >> Unlike the ansi2html module from PyPI, this implementation focuses on
> >> tracking the formatting within a line.  As a result, it provides nice
> >> HTML output for e.g. carriage returns or tabs.  There are many test
> >> cases that were checked against vte, and a huge repetitive CSS (which was
> >> generated by a small program), so don't let the diffstats scare you. :)
> >>
> >> Since the sample database does not have ANSI control codes, I took a
> >> Travis log and placed a sample conversion here:
> >> - text source: http://people.redhat.com/pbonzini/log.txt
> >> - HTML output: http://people.redhat.com/pbonzini/log.html
> >>
> >> Note that the HTML output is 20% the size of the source, because carriage
> >> returns are optimized out!
> > 
> > Interesting!
> > 
> > A general question: what about plain text support? For example when we want to
> > include log snippets to email notifications; and also we probably want to keep
> > it possible and simple for users to view the full log in their $EDITOR.
> 
> Yes, it is easy to split the converter into an abstract class with
> text-output and HTML-output subclasses.  When emitting the TestingReport
> event, the testing module can pass the logs through the text-output
> converter.  While at it, it could only print the last 50-ish lines of
> the output (per https://github.com/patchew-project/patchew/issues/31).

Sometimes high concurrent build yields many non-error lines after the error:

https://patchew.org/QEMU/20180206095603.26500.5959.stgit@pasha-VirtualBox/

So some filter like "grep -C 50 error" would be more robust.

But it seems some people also want the env info in the beginning including env
info and the configure/make commands.  So I never know how to do this right. :(

Fam

On 27/02/2018 14:10, Fam Zheng wrote:
> Sometimes high concurrent build yields many non-error lines after the error:
> 
> https://patchew.org/QEMU/20180206095603.26500.5959.stgit@pasha-VirtualBox/
> 
> So some filter like "grep -C 50 error" would be more robust.

Hmm, good point.  (grep -C 20 probably would be enough).

> But it seems some people also want the env info in the beginning including env
> info and the configure/make commands.  So I never know how to do this right. :(

They can access it in one click if we add a link to the log in the email.

Paolo