Let’s Talk About Latency

This post is all about music production using computers and it’s going to get quite technical, so if this is not for you, thanks for reading this far, but catch you another time. If you care about music production at all, then you probably need to read this, though.

Modern music production relies on computers. There is no realistic alternative. Windows or Mac, you need one to make a record, these days. They’re mission-critical tools. Multi-track tape is all but obsolete (and becoming expensive and hard to find or service). Dedicated hardware devices for recording digital audio just don’t have enough going for them to do the job. They’re way too limited in their capabilities to meet the quality bar required of modern music production for release. Computers are really the only game in town.

Music production is a valuable creative industry and it’s growing. There is a measurably significant contribution to GDP that results from the activities of music producers. This is not the activity of hobbyists; this is serious business. I have a record company to launch. I’ve invested thousands of pounds in my tools. It’s a real enterprise.

To cover my bases, I have a very powerful desktop machine that runs Windows 10 and a MacBook Pro that’s a few years old. I primarily use Cubase Pro 9.5 on Windows and Logic Pro X on the Mac. These are professional programmes for the creation of digital audio and music. I am running them on (close to) best of breed hardware. My operating systems and the software I use are kept scrupulously up to date. I also have a brand new laptop that should easily run a DAW, but my measurements have revealed something shocking (more on this soon).

What’s the problem?

The process of making music requires shuffling vast amounts of audio data around the place, so that the resultant data stream to your speakers is a new “sample”, say 44100 times per second per speaker. Calculating what that sample value should be is something the computer has to do, given files of audio recorded to the Solid State Disk (SSD) and from sums that are done in the digital audio workstation (DAW) software and its plug ins. The essence of making this bit stream sound like music is for the computer to never miss sending out a sample. It’s one sample every 44,100th of a second per speaker, or you hear a drop out. The music stops. The beat is lost. Silence is heard, until the bit stream starts up again. To a musician, stopping the music is unacceptable. You cannot produce music if it’s riddled with holes, like Swiss cheese. Audio has to play continuously, or it’s game over.

Computers have a lot to do, because they have to draw to the screen, manage their hardware, read and write to hard drives, listen to network ports and so on. All of this activity goes on the whole time you’re trying to make music and you can’t do without it. It more or less has to be done. Because computers need to do all this work and make it look like everything was going smoothly the whole time, they slice up time and do different things in different time slots. If they get everything done quickly enough, without variance, then the design of the hardware masks the gaps in each separate task with a buffer and the user is none the wiser that some of what they were doing didn’t happen when they thought it did. Computers jump from task to task, getting tons of stuff done, so that you think the audio is smooth and uninterrupted (as are the graphics on your UI).

When software doesn’t play nicely, it will hog more time than its time slot allows, robbing time from other tasks, so those get lumpy. It’s what we call latency. Latency means you have to wait a lot longer for the computer to catch up with the tasks you want it to do (like making sound). When the latency is long, or varies a lot, the audio playback hardware literally runs out of data temporarily, while it waits for the CPU. That’s what makes the audio glitches.

In Windows machines, there is competition for the CPU from the operating system itself (i.e. Windows). It has work it must do too. If, however, the software in your operating system takes too long, then you cannot use your computer to make smooth, uninterrupted audio at all. You can’t make recordings, because they will lose synchronisation with any music you’re playing against. It’s a music production disaster.

Update Armageddon

Since Windows 10 was introduced, music producers have been reporting that a component of the Windows 10 operating system, called wdf01000.sys, which controls kernel mode drivers (i.e. the very guts of the interface between your operating system and the hardware it runs on), is producing massive latencies, sufficient to stop audio work on the computer. That’s it. Game over. All audio is contaminated, because this component hogs the processor too long, doing it’s (necessary) stuff. Microsoft, in their arrogance, installed a piece of software in one of their “free” updates which has, in effect, put music producers out of business. Since the Sceptre and Meltdown security issues were discovered, vendors like Intel and Microsoft have crippled the processors even more. The net result is that Windows computers no longer have the capability to play or record audio without destroying it.

Nobody appears to be fixing it, either. While they are absorbed with the security issues, DAW vendors, hardware vendors, Microsoft and anybody else you call on to help you are collectively shrugging and saying, “it’s not my problem”. Music producers are stuck with “updated” operating system software which has rendered their music production tools pretty much inoperable.

I have been a maker of digital audio workstations. We got audio to work flawlessly, unconditionally, on late 1980s 80286 processors. An up-to-date Intel i7 is orders of magnitude faster and much more capable than those old computers were. Indeed, a humble Intel Celeron CPU, which is fitted to my ageing backup laptop, bought new circa 2009, has measured latencies that are one fifth of what I am getting on Intel processors running Windows 10 today. I wisely left that backup laptop on Windows 7. It’s not great for audio production, because the Celeron is woefully under-powered, but it works better with audio than all of my up-to-date machines. How can that be an acceptable situation?

It appears that nobody in my software supply chain bothered to measure and test the latency for music production, with any professional music production tools, before releasing their mandatory operating system updates. They simply imposed these “updates” on everybody, threatening that failure to keep up to date would render you helpless in the face of hackers and indemnify them from liability for consequent damage and loss. There was no opt out. Nobody appeared to care about audio. Nobody appears to care still.

Just buy a new machine

Well, that’s what I did. I bought a brand new i7 laptop a few months ago. This is the Windows 10 machine that, despite kicking the old laptop’s specifications to the curb, has deferred procedure call latencies (the thing that causes audio drop outs) that are five times worse than a machine that is nearly a decade old. It doesn’t matter which computer you buy, how new it is, which digital audio tools you install and what other things you tweak, optimise or turn off. This problem is down to an operating system component that isn’t playing nicely. You can’t fix it either, because the source code is proprietary to Microsoft. They assert their monopoly on wisdom, in this matter, even though they have demonstrably dropped the ball.

Just use your MacBook Pro

If you know how to use Logic Pro X on your MacBook Pro, why not just use that? Yes, you’ll have to move a ton of third party plug-in software licenses, but at least it will work, right? Not so fast. MacBook Pros with retina displays are susceptible to an issue known as the “GPU panic”. In this scenario, the whole machine crashes, unpredictably, whenever the graphics processing unit (GPU) feels the urge. It can happen at any time, but tends to be related to the user interface performing animations (which you can’t turn off). If you are in the middle of a recording, a crash results in the loss of a take. Forever. If you are in full music production flow, you lose all your work since your last save. It’s for this reason I’ve started to save like I have a Tourette’s syndrome nervous tick. Clearly, this obsessive compulsive behaviour is not conducive to creating beautiful melodies.

Apple tried to resolve this GPU crashing problem with a logic board swap. Unfortunately, that doesn’t fix it. Some people, having missed the free recall programme (like me), which ended in December 2016, are being asked to pay £400 or more for a replacement board. The point is, the problem comes back. Apple evidently don’t know why. Even if you buy a £2700 top-of-the-range, brand-new MacBook Pro, you’ll find that the same problem could and probably will recur.

There are workarounds that seem to help, but each operating system update that you install overwrites the settings and you have to do them again. These hacks work by switching off the suspect chip. Consequently, you also lose graphics performance, if you rely on the integrated graphics, because you are turning the second, high-powered graphics processing unit off, treating it like useless, dead weight. In effect, you’ve got a graphics processing unit you can’t use and you can see that in a degraded user experience.

The independent repair community, which loves to pour scorn on Apple’s “Geniuses”, appear to have found the root cause of the issue. A single 330 microfarad tantalum capacitor, worth pennies, which regulates the slightly more than one volt power supply to much of the graphics subsystem, becomes intermittent. When it’s warm; it (mostly) works – until it fails completely. If it gets cold; the capacitor doesn’t charge properly, so the voltage drops to around 0.3 volts, and the graphics subsystem unexpectedly shuts down. There are videos on YouTube where independent technicians replace this rogue capacitor with an electrolytic film capacitor of the same value. Unsurprisingly, the machine springs to life and works flawlessly indefinitely.

Here’s the point: Apple don’t recognise the issue or the fix. They won’t perform it. The new logic boards that they put into your machine have the same prone-to-failure tantalum capacitor in circuit. They haven’t addressed or solved the root cause of the issue.

In Summary

So, the current situation in my studio is that I have north of £8000 pounds worth of computing hardware that cannot make music.

On Windows, Microsoft made the problem and won’t acknowledge or fix it. Evidently, they didn’t test their software against audio latency.

On my Apple machine, I have a logic board hardware fault that Apple don’t acknowledge or correctly fix.

In both cases, the software and hardware is proprietary and closed off from other people making better diagnoses, performing better fault finding and eliminating the root causes. We’re locked out. Any attempt to tamper gives them all the excuse they need to stop listening to you at all.

Not that either Microsoft or Apple are listening. They evidently don’t care about music production. In Apple’s case, there is an added irony in this, since they make enormous profits on the back of their distribution of musical artists’ work through iTunes. How is that music going to get made?

Trying to fight these gargantuan corporations, to get them to take your issues seriously, is all but futile. There are forums, but you wind up talking to other powerless people, outside the privileged circle of engineers that can actually effect change to their products. These sacred engineers are protected from the public, engineering busily in their ivory towers, blissfully unaware of the loss, pain and suffering that their engineering blunders are causing. If it were nothing else, it’s atrocious product management (something else I know quite a lot about, through decades of professional experience in the field).

I know all this to be true, because as well as being a music producer, I have over 30 years’ experience as an electrical, electronics and computer engineer, making digital audio products. I was making these things long before most of the current crop of engineers in charge were actually born. I know what I’m looking at, but I am disempowered. I can’t fix the problems because of proprietary control over access to information (not that I should have to – I’d rather be making music). It’s a scandalous state of affairs in which these companies continue to take your money, but provide software and hardware that is not fit for its advertised purposes. Meanwhile, your (my) creative work grinds to a halt and your (my) creative enterprise is stopped in its business. Paralysis, distraction and frustration. Exactly the inspiration you need to write and record a happy tune, isn’t it?

What’s a music producer to do?

About tropicaltheartist

You can find out more about me here: https://michaeltopic.wordpress.com/. There aren’t many people that exist in that conjunction of art, design, science and engineering, but this is where I live. I am an artist, a musician, a designer, a creator, a scientist, a technologist, an innovator and an engineer and I have a genuine, deep passion for each field. Most importantly, I am able to see the connections and similarities between each field of intellectual endeavour and apply the lessons I learn in one discipline to my other disciplines. To me, they are all part of the same continuum of creativity. I write about what I know, through my blogs, in the hope that something I write will resonate with a reader and help them enjoy their own creative life more fully. I am, in summary, a highly creative individual, but with the ability to get things done efficiently. Not all of these skills are valued by the world at large, but I am who I am and this is me. The opinions stated here are my own and not necessarily the opinion or position of my employer.
This entry was posted in Uncategorized and tagged , , , , , , , , , , , , , , , , , , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s