QPlainTextEdit with setMaximumBlockCount performance regression

Recently while working with ipython qtconsole, i realized that it is easy to freeze the qtconsole or any app embedding the it by simply doing a "for i in xrange(10000000):print i" loop, then all arguments about separate kernel process safety etc. are voided. Since this is not something i liked, i set about to fix it in true nature of open-source development (scratching my own back). In this post i'll describe some Qt issues, benchmarks and the approaches i took and how you too can deal with similar problem of overwhelming text input to your text buffer.
Note: The ipython pull request https://github.com/ipython/ipython/pull/3409 is part of this experiment.

The Problem:

The qtconsole widget is a QPlainTextEdit, with the maximumBlockCount set to 500 by default, which is a very small buffer size by any standard. However, despite this, in case too much text is written to the QPlainTextEdit, it takes too much time drawing the text and the application appears to be frozen This is because the Qt event loop is blocked by too much time consumed by drawing the QPlainTextEdit and incessant stream of new text to draw at a faster rate than QPlainTextEdit can render.

The Solution Approaches:

My first though at the problem was to use a timer to throttle the stream rendering rate, and append text to the area only every 100ms. That wouldn't cause any perceptible usability loss, but make the program more responsive. Also, another essential idea was to only send the maximumBlockCount number of lines of text ot the QPlainTextEdit. It seems that QPlainTextEdit is very bad at render performance if text is clipped by limiting maximumBlockCount, contrary to its single major use as a way to improve performance and memory usage.

An initial look into the ipython code made it clear that the test code was giving about 8000 lines per stream receive, which i was glad to clip to maximumBlockCount and coalesce multiple text streams into a single appendPlainText call every 100 ms. The qtconsole seemed very responsive, terminating the test print loop without any perceptible delay. All was well, i could go to sleep peacefully now, except for a small glitch which i realized soon enough. Due to a bug, my timer loop wasn't really doing anything. Every stream from ipython kernel was being written to the widget. How then was the widget so responsive, an attentive reader might ask. This post attempts to provide an answer to that very question.

The following are plots of time taken to append plain text to QPlainTextEdit using the code linked here. The x axis is the number of lines appended per call and the different lines are for different maximumBlockCount of the QPlainTextEdit

Appending to an already full buffer

 Clearing and then appending text

 Appending text to empty widget

In all the above cases, the final result is same because the number of lines appended is equal to maximumBlockCount so that all previous content is gone.
As you can see yourself, simply appending text to a full buffer is *very* expensive, so much so that it is almost an order of magnitude larger than clearing the text and then appending for ipython's default case of maxumumBlockCount = 500. All appends are fast until any line overflows the maximumBockCount, when onwards it becomes very expensive to append any more content. I intend to modify the ipython pull request in view of this seemingly bizzare result and attempt to improve the performance further. Hopefully, this would obliterate the need to have a timer based stream output loop and the related complexity. Ideally, someone should fix this right at Qt level, but i do not yet feel confident to do it. Until that happens, this simple workaround should be good enough.


PS: Comments, feedback and further insights welcome