Qt 5.12.2. Python/PyQt (though that does not help, it should not be the issue). Tested under Linux, known from user to happen under Windows too. I am in trouble, and I need help from someone who knows their QtWebEngine!
Briefly: I have to create and delete QtWebEngines (for non-interactive use, read below) a large number of times from a loop which cannot call the main event loop. Every instance holds onto its memory allocation --- which is "large" --- until code finally returns to main event loop. I cannot find any way of getting Qt to release the memory being use by QtWebEngine as it proceeds, only when return to main event loop. Result is whole machine runs out of memory + swap space, until it dies/freezes machine requiring reboot!
-
In large body of code,
QWebEngineViewis employed in aQDialog. -
Sometimes that dialog is used interactively by user.
-
But it is also used *non-*interactively in order to use its ability to print from HTML to PDF file.
-
Code will do a non-interactive "batch run" of hundreds/thousands of pieces of HTML, exporting to PDF file(s).
-
During this large amounts of memory will be gobbled. So much so that machine can even run out of all memory and die!
-
Only a return to top-level, main Qt event loop allows that memory to be recouped. I need something better than that!
I paste below about as minimal an example of code I am using in a test to prove behaviour.
import sys
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWebEngineWidgets import QWebEngineView
class MyDialog(QtWidgets.QDialog):
def __init__(self, parent=None):
super(MyDialog, self).__init__(parent)
self.wev = QWebEngineView(self)
self.renderLoop = QtCore.QEventLoop(self)
self.rendered = False
self.wev.loadFinished.connect(self.synchronousWebViewLoaded)
# set some HTML
html = "<html><body>Hello World</body></html>"
self.wev.setHtml(html)
# wait for the HTML to finish rendering asynchronously
# see synchronousWebViewLoaded() below
if not self.rendered:
self.renderLoop.exec()
# here I would do the printing in real code
# but it's not necessary to include this to show the memory problem
def synchronousWebViewLoaded(self, ok: bool):
self.rendered = True
# cause the self.renderLoop.exec() in synchronousRenderHtml() above to exit now
self.renderLoop.quit()
class MyMainWindow(QtWidgets.QMainWindow):
def __init__(self, parent=None):
super(MyMainWindow, self).__init__(parent)
self.btn = QtWidgets.QPushButton("Do Test", self)
self.btn.clicked.connect(self.doTest)
def doTest(self):
print("Started\n")
# create & delete 500 non-interactive dialog instances
for i in range(500):
# create the dialog, it loads the HTML and waits till it has finished rendering
dlg = MyDialog(self)
# try desperately to get to delete the dialog/webengine to reclaim memory...
dlg.deleteLater()
# next lines do not help from Python
# del dlg
# dlg = None
QtWidgets.QMessageBox.information(self, "Dismiss to return to main event loop", "At this point memory is still in use :(")
if __name__ == '__main__':
# -*- coding: utf-8 -*-
app = QtWidgets.QApplication(sys.argv)
mainWin = MyMainWindow()
mainWin.show()
sys.exit(app.exec())
I have tried various flavours of processEvents() in the loop after deleteLater() but none releases the memory in use. Only, only when the code returns to the top-level, main Qt event loop does it get released. Which is too late.
To monitor what is going on, under Linux I used
watch -n 1 free mem
watch -n 1 ps -C QtWebEngineProc
top -o %MEM
There are two areas of memory hogging:
- The process itself uses up considerable memory per QtWebEngine
- It will run 26 (yes, precisely 26)
QtWebEngineProcprocesses to service the requests, each also taking memory.
Both of these disappear as & when return to Qt top-level event loop, so we know the memory can & should be released. I do not know if this behaviour is QtWebEngine specific.
Anyone kind enough to answer will need to be specific about what to put where to resolve or try out, as I say I have tried a lot of fiddling! Unfortunately, advising to do the whole thing "a different way" (e.g. "do not use QtWebEngineView", "rewrite code so it does not have to do hundreds at a time", etc.) is really not what I am looking for, I need to understand why I can't get it to release its memory as it is now?