Krita Fall 2018 Sprint Results: HDR support for Krita and Qt!

In October we held a Krita developers' sprint in Deventer. One of my goals for the sprint was to start implementing High Dynamic Range (HDR) display support for Krita. Now almost a month have passed and I am finally ready to publish some preliminary results of what I started during the sprint.

The funny thing is, before the sprint I had never seen what HDR picture even looks like! People around talked about that, shops listed displays with HDR support, documentation mentioned that, but what all this buzz was about? My original understanding was like "Krita passes 16-bit color to OpenGL, so we should already be ready for that". In Deventer I managed to play with Boud's display, which is basically one of few certified HDR displays with support of 1000 nits brightness, and found out that my original understanding was entirely wrong :)

Last couple of years the computer display industry has changed significantly. For almost 30 years people expected their monitors to look like normal paper. Displays were calibrated to look like a sheet of white paper illuminated by a strictly defined source of light (D65).

Now, with appearance of HDR technology, the things has changed. Displays don't try to emulate paper, they try to resemble real sources of light! For example, modern displays can show a sun beam piercing through a window not just like "a paper photo of a beam", but just as a real source of light getting directly into you eye! Basically, now the displays have LEDs that can shine as bright as a real sun, so why not use it? :) (Well, the display is only 1000 nits, and the sun 1.6 billion nits, but it's still very bright).

If you look at the original original EXR file you will see how the window "shines" from your screen, as if it were real

By itself, the idea of having a display that can send a sun-strength beam into your eye might be not a lot of fun, but the side effects might of the technology are quite neat.

In the first place, the displays supporting HDR do not work in the standard sRGB color space! Instead they use Rec. 2020, a color space widely used in cinematography. It has different primary colors for "green" and “red” channels, which means it can encode much more variations of greenish and reddish colors.

In the second place, instead of using traditional exponential gamma correction, they use Perceptual Quantizer (PQ), which not just extends the dynamic range to sun-bright values, but also allows to encode very dark areas, not available in usual sRGB.

Finally, all HDR displays transfer data in 10-bit mode! Even if one doesn't need real HDR features, having a 10-bit pipeline can improve both painters' and designers' workflow a lot!

Technical details

From the developer's point of view, the current state of HDR technology is a bit of a mess. It's really early days. The only platform where HDR is supported at the moment is Windows 10 (via DirectX).

Neither Linux nor OSX can handle the hardware in HDR mode currently. Therefore all the text below will be related to Windows-only case.

When the user switches the display into HDR mode, the OS automatically starts to communicate with it in p2020-pq mode. Obviously, all the colors that normal applications render in sRGB will be automatically converted. That is, if an application wants to render an image directly in p2020-pq, it should create a special "framebuffer" object (swap chain), set its colorspace to p2020-pq and ensure that all the internal textures have correct color space and bit depth.

In general, to ensure that the window is rendered in HDR mode, one should do the following:

1. Create a DXGI swap chain with 10-bit or 16-bit pixel format
2. Set the color space of that swap chain to either p2020-pq (for 10-bit mode) or scRGB (for 16-bit mode).
3. Make sure all the intermediate textures/surfaces are rendered in 10/16-bit mode (to avoid loss of precision)
4. Since the GUI is usually rendered on the same swap chain, one should also ensure that the GUI is converted from sRGB into the destination color space (either p2020-pq or scRGB)

In Krita we use Qt to render everything, including our OpenGL canvas widget. I had to go deep into Qt's code to find out that Qt unconditionally uses 8-bit color space for rendering windows. Therefore, even though Krita passes 16-bit textures to the system, the data is still converted into 8-bits somewhere in Qt/OpenGL. So I had to hack Qt significantly...

Making Qt and Angle support HDR

Here comes the most interesting part. We use Qt to access the system's OpenGL implementation. Traditionally, Qt would forward all our requests to the OpenGL implementation of the GPU driver, but... The problem is that quite a lot of OpenGL drivers on Windows are of "suboptimal quality". The quality of the drivers is so "questionable", that people from the Google Chromium project even wrote a special library that converts OpenGL calls into DirectX API calls and use it instead of calling OpenGL directly. The library is called Angle. And, yes, Qt also uses Angle.

Below is a sketch, showing the relation between all the libraries and APIs. As you can see, Angle provides two interfaces: EGL for creating windows, initializing surfaces configuring displays and traditional OpenGL for the rendering itself.

To allow switching of the surface's colorspace I had to hack Angle's EGL interface and, basically, implement three extensions for it:

• EGL_KHR_gl_colorspace — allows switching the surface into sRGB color space (also implements auto gamma correction of texture and framebuffer samplers)
• EGL_EXT_gl_colorspace_scrgb_linear — allows switching the surface into scRGB color space
• EGL_EXT_gl_colorspace_bt2020_pq — allows switching the surface into p2020-pq color space

After that I had to patch QTextureFormat a to support all these color spaces (previously, it supported sRGB only). So now, if you configure the default format before creating a QGuiApplication object, the frame buffer object (swap chain) will have it! :)

// set main frame buffer color space to scRGB
QSurfaceFormat fmt;
// ... skipped ...
fmt.setColorSpace(QSurfaceFormat::scRGBColorSpace);
QSurfaceFormat::setDefaultFormat(fmt);

// create the app (also initializes OpenGL implementation
// if compiled dynamically)
QApplication app(argc, argv);
return app.exec();

I have implemented a preliminary demo app that uses a patched Qt and shows an EXR image in HDR mode. Please check out the source code here:

Demo application itself:
https://github.com/dimula73/hdrtest/tree/test-hacked-hdr

Patched version of QtBase (based on Qt 5.11.2):
https://github.com/dimula73/qtbase/tree/krita-hdr

The application and Qt's patch are still work-in-progress, but I would really love to hear some feedback and ideas from KDE and Qt community about it. I would really love to push this code upstream to Qt/Angle when it is finished! :)

List of things to be done

1. Color-convert Qt's internal GUI textures. Qt renders the GUI (like buttons, windows and text boxes) in a CPU memory (in sRGB color space), then uploads the stuff into an OpenGL texture and renders that into a frame buffer. Obviously, when the frame buffer is tagged with a non-sRGB color space, the result is incorrect --- the GUI becomes much brighter or darker than expected. I need to somehow mark all Qt's internal textures (surfaces?) with a color space tag, but I don't yet know how... Does anybody know how?
2. Qt should also check if the extensions are actually supported by the implementation, e.g. when Qt uses a driver-provided implementation of OpenGL. Right now it tries to use the extensions without any asking :)
3. The most difficult problem: OpenGL does not provide any means of finding out if a combination of frame buffer format and color space is actually supported by the hardware! A conventional way to check if the format/color space is supported: create an OpenGL frame buffer and set the desired color space. If the calls do not fail with error, then the mode is supported. But it doesn't fit the way how Qt selects it! Qt expects one to call a singleton QSurfaceFormat::setDefaultFormat() once and then proceed to creation of the application. If the creation fails, there is no way to recover! Does anybody have an idea how that could be done in the least hackish way?

I would really love to hear your questions and feedback on this topic! :)