Displaying NDI sources on Stream Decks

In the course of my work on our local church A/V system, I've spent quite a lot of time playing with Elgato Stream Decks and NDI cameras. It only occurred to me a week or so ago that it would be fun to combine them.

The Stream Deck screens are remarkably capable - they're 96*96 or 72*72 pixels (depending on the model) and appear to have a decent refresh rate. I tend to think of them just in terms of displaying simple icons or text, but I'd already seen a video of a Stream Deck displaying a video so it wasn't too much of a leap to display live camera outputs instead.

I should make it clear that I had absolutely no practical reason for doing this - although as it happens, the local tweaks I've made to the C# NDI SDK have proved useful in At Your Service for enabling a preview camera view without opening a second NDI network stream.

Tweaking the NDI SDK

The NDI SDK comes with a demo C# sample library. It's not clear how much this is intended to be for production use, but we've been using it in At Your Service for quite a while with very few issues. (We're pinned at a slightly old version of the runtime due to some smoothness issues with later versions, which I'm hoping the NDI folks will sort out at some point - I've given them a diagnostic program to show the issues.)

The SDK comes with a WPF receive view" which displays a camera in a WPF control, as you'd expect. Now I don't know enough about WPF to try to use the Stream Deck as an output device for WPF directly. It may be feasible, but it's beyond my current skillset. However, I was able to reuse parts of the receive view to create a new display-agnostic" class, effectively just dealing with the fiddly bits of the NDI interop layer and providing video frames (already marshalled to the dispatcher thread) as raw data via an event handler. Multiple consumers can then subscribe to that event and process the frames as they wish - in this case, drawing them to the Stream Deck, but potentially delivering them to multiple displays at the same time.

The simplest image processing imaginable

So, the frames are being delivered to my app in a raw 4-bytes-per-pixel format. How do we draw them on the Stream Deck? I've been using StreamDeckSharp which is has been pretty simple and reliable. It has multiple ways of drawing on buttons, but the one we're interested in here is passing in raw byte arrays in a 3-bytes-per-pixel format. (As it happens, the Stream Deck SDK then needs to encode each frame on each button as a JPEG. It's a pity that the data can't be transferred to the Stream Deck in raw form, but hey.) All we need to do is scale the image appropriately.

Again, there may be better ways of doing this, such as asking WPF to write the full frame to a Graphics and performing appropriate resizing along the way. But I figured I'd start off with something simpler - and it turned out to be perfectly adequate. I'm not massively concerned with obtaining the absolute best quality image possible, so rather than using all the pixels and blending them appropriately, I'm just taking a sampling approach: if I'm displaying the original 1920*1080 image on a 96*96 button, I'll just take 96*96 pixel values with as simple code as I could work out.

In the end, I decided to stick to integer scaling factors and just center the image. So to take the above example, 1080/96 is 11.25, so I take samples that are 11 pixels apart in the original image, trimming the top/bottom borders a little bit (because of the truncation from 11.25 to 11) and the left/right borders a lot (because the buttons are square, and it's better to crop the image than to letter-box it). What happens if you've got an image which is just white dots that are 11 pixels apart, and black everywhere else? You end up with a white image (assuming the dots are aligned with the sampling) - but in a camera image, that doesn't really happen.

The maths ends up being slightly more fiddly when displaying a frame across several buttons. There are are gaps between the buttons, and while we could just ignore them, it looks very weird if you do - if a straight line crosses multiple buttons, it ends up looking broken" due to the gap, for example. Instead, if we take the gaps into account as if we were drawing them, it looks fine. The arithmetic here isn't actually hard in any sense of the word - but it still took me a few goes to get right.

Putting it together

Of course, Stream Deck buttons aren't just screens: they're pressable buttons as well. I haven't gone to down on the user interface here: the left-most column just buttons just displays the NDI sources, and pressing one of those buttons displays that source on the rest of the buttons. That part was really simple - as you'd probably expect it to be.

The initial prototype took about two or three hours to write (bearing in mind I already had experience with both the NDI code and the Stream Deck SDK), and then it took another couple of hours to polish it up a bit and get it ready for publication. The application code is all available on my DemoCode GitHub repository; unfortunately even if you happen to have an NDI camera and a Stream Deck, you won't be able to use it without my tweaks to the NDI C# code. (If the C# part of the NDI SDK is ever made open source, I'll happily fork it and include my tweaks there.)

I've put a YouTube video of the results. Hope you enjoy them. I've had a lot of fun with this project, despite the lack of practical applications for it.