V-Drum Explorer: Memory and 7-bit addressing

So, just to recap, I'm writing an explorer for my Roland V-Drums set (currently a TD-17, but with a TD-27 upgrade on the way, excitingly). This involves copying configuration data from the module (the main bit of electronics involved) into the application, displaying with it, editing it, then copying it back again so that I can use the changes I've made. I use MIDI System Exclusive (SysEx) messages to request data from the module.

(Since I last wrote about all of this, MIDI 2.0 has been ratified, which has more sensible 2-way communication. However, I don't think the V-Drums line supports MIDI 2.0 yet, and even if it does I expect we'll need to wait a while for drivers, then managed APIs exposing them.)

In fact, most of the time that I'm working on the V-Drum Explorer I don't have it connected to the drum kit: it's much easier (and quicker) to load the data from a file. Once it's in memory, it really doesn't matter where the data came from. I'll go into the file formats I'm using in another post.

This post is about how that configuration data is organized, and particularly about the 7-bit addressing it uses.

Download the docs!

If you'd like to know about all of this in more detail, it's well worth downloading some of the reference documentation Roland very helpfully provides. The TD-17 is the simplest of the modules we're talking about, so I'd suggest downloading the TD-17 MIDI implementation so you can go at least one level deeper than this blog post, if you're interested. If you think you're likely to want to do that, I'd suggest doing so before reading any further. The important bit starts on page 5 - the Parameter Address Map" which is the bulk of the document.

Configuration as memory

The configuration data in the module isn't stored in any kind of file system, or with separate SysEx messages for different kinds of data. Instead, it's modeled as if the module contains a big chunk of memory, and different areas of that memory have different effects on the module. I don't know what the implementation is like within the module itself, of course; this is just the interface presented over MIDI.

As the simplest possible example, address 0 in the memory (on all three of the modules I have documentation for) represents the currently selected kit". It accepts values between 0 and 99 inclusive, to represent kits 1 to 100 inclusive. (As a reminder, a kit is a configuration of all the pads, allowing you to switch the whole module between (say) a rock feel or something more electro-funky.) So as an example, if my module is currently on kit 11 (Studio / Live room"), and I asked the module to give me the content of address 0, it would return 10. If instead I set the value of address 0 to 30, the module would display kit 31 (More cowbell / pop") and all the sounds would change accordingly.

The documentation describes a number of containers - that's my own term for it, but it seems reasonable. Each container is named, and has its own description in terms of its content at different offsets. For example, address 0 belongs in the [Current] container, which is documented very simply:

[Current]+-----------------------------------------------------+| Offset Address | Description ||----------------+------------------------------------+| 00 00 | 0aaa aaaa | Drum Kit Number (0-99) || | 1-100 |+----------------+------------------------------------+| 00 00 00 01 | Total Size |+----------------+------------------------------------+

The 0aaa aaaa shows that 7 bits of the value are used. Due to MIDI's inherent 7-bit nature, each address can only store 7 bits. Whenever a larger number is required, it's stored across multiple addresses, typically using only the bottom four bits of each 7 bit value.

The content of each container is broadly broken into three types of data - again, all the terminology is mine:

• Primitive fields: strings, numbers etc
• Fields which are other containers (often repeated, e.g. Kit 1 to Kit 100)
• Overlaid containers (fields directly in this container, interpreted according to a different container)

I'll talk about overlaid containers at length another time, as they're tricky.

So basically you end up with a natural tree structure. So far, so good... except for 7-bit addressing.

7-bit addressing

I entirely understand why the values in memory are 7-bit values. That's inherent in MIDI. But Roland also chose to use a 7-bit address space, which makes the code much more complex than it needs to be.

All addresses and offsets are documented using hex as if it were entirely normal - but the top bit of every byte of an address is always clear. So address 00 7F is followed directly by address 01 00 - even if they're within the same container. Now this does mean that the values in the MIDI request messages are exactly the documented addresses: the top bit of each byte in the request message has to be clear, and that drops out naturally from this. But it makes everything else hard to reason about. I've been bitten multiple times by code which looks like it should be okay, but it's either skipping some data when it shouldn't, or it's not skipping addresses when it should. By contrast, it would have been really simple (IMO) to document everything with a contiguous address space, and just specify that when requesting data, the address is specified in seven-bit chunks (so bits 27-21 in the first request byte, then 20-14, then 13-7, then 6-0).

I've tried to isolate this into a ModuleAddress struct, but the details have still leaked out into a few more places. Over the past few days I've tried to be more rigorous about this with a DisplayValue (7-bit) and a separate LogicalValue (contiguous), but it's still leaking more than I want it to. I don't think I can fix it without a more serious rewrite - which I probably want to attempt reasonably soon anyway.

You might wonder why I don't just model everything using the logical contiguous address space, removing all the gaps entirely. The problem is that all the schema information is basically copied from the documentation - and that refers to the 7-bit addressing scheme. I really want the schema to match the documentation, so I can't move away from that entirely. Another thing that makes it tricky is that a lot of the time I deal in offsets rather than addresses. For example, the Kit Unit Common 1" part of a Kit" container is always at offset 00 20 00 relative to the start of the container. That's not too bad on its own, but I also need to express the gap between offsets" which is a sort of offset in its own right (maybe). For example, Kit Unit Common 2" is at offset 00 21 00 within a kit, so in the schema when I describe the Kit Unit Common" repeated field, I describe it as having an initial offset of 00 20 00, with a gap of 00 01 00. That sounds fine - until you've got a repeated field which is large enough to have a gap in the middle, so you need to model that by making the offset have a gap as well. (I'm reminded of calendar arithmetic, which has similar weirdnesses.)

The lesson I've learned from this is that when there's hairiness like this, it's worth abstracting it away really thoroughly. I wish I'd stopped at the first abstraction leak and thought, maybe I need to redesign rather than live with this".

Conclusion

Even without 7-bit addressing, there would have been plenty of challenging choices in the design of the V-Drum Explorer, particularly in field and container representation. More details of those choices will come in future posts - but at least they feel inherently tricky... the kind of thing software engineers are expected to have to think about.

7-bit addressing feels like it was a choice made to make one small use case (MIDI request/response messages) simple, but made everything else trickier. I'd be fascinated to know the module code is a lot more complicated because of this as well, or whether so much is effectively hard-coded (because it needs to actually do stuff rather than just display data on a screen) that it doesn't make much difference.

Next time: using data vs code to represent differences between modules.