Tuesday, October 26, 2010

Linear Interpolation for Audio in C, C++, Java, etc.

Linear Interpolation in digital audio came up recently, so I'm posting it here. I hope it's useful for other folks.

Technically, Linear interpolation is the act of fitting a line through existing points and computing new data from that line. This might sound complex, but it turns out to be pretty easy, and we can do it with a few lines of code.

Visually, we can think about drawing a line between two points, and then being able to find the y value for any given x. However, I actually think it's easier to think of it non-graphically because linear interpolation is really just a kind of weighted average.

For audio, we frequently want to use linear interpolation because it's easy to implement, computationally efficient, and it's "smooth" in some sense that I won't get into, but I will say that it generally does not create clicks and pops when you don't want it to. Linear interpolation is useful for handling fader changes and button-push "de-bouncing" and so on, and it's often great for simple cross-fades and the like.

The formula for linear interpolation is derived from the formula for the line between two points. You can see wikipedia for the details. I am omitting it here and jumping straight to an example. To perform a linear interpolation of 100 samples where y[0] = 7, and y[100] = 20, our code would look something like this:

double start = 7;
double end = 20;

for( int i=0; i<100; ++i ) {
   double ratio = i/100.0;
   y[i] = start * (1-ratio) + end * i;

You can think of ratio as the weight given to the start variable, and 1-ratio as the weight given to the end variable. As we slide through the samples, we slowly transition from the start value to the end value.

Notice I've been very careful to make sure y[0] is actually 7, and y[99] is not quite 20, so that y[100] will smoothly transition to 20 as required. Off-by-one errors can screw this up and while you might not hear the difference, you want to get that right or you could end up with pops, weird overs, or other subtle problems.

Now you might say that the above code is not very efficient. You can improve on it somewhat using the code below, but be aware that if you interpolating over a large number of samples, especially if you are using single precision floats, you might not quite end up where you expect. The performance gain for this more complex code is likely to be minimal on modern computer hardware, but may be substantial on DSP hardware, where operations like floating point adds take much less time than floating point divides. A clever compiler could theoretically make the same object code out of these two code snippets if it can determine that precision won't be an issue.

double start = 7;
double end = 20;
int length = 100;

double interval = ( end - start ) / length;
y[0] = start;

for( int i=1; i
   y[i] = y[i-1] + interval ;

By the time we get to the end,  y[length-1] should be ( end - start ) / length * length = ( end - start ) larger than the start, which is exactly right.

That's all there is to Linear Interpolation, so let's go to an audio example: Say we want to go from off, or mutted to on, or unmutted, without a click. Instead of 7 and 20, we'd use 0.0 for off, and 1.0 for on. Also, instead of setting the values in the array, we are going to be multiplying the values in the array, because that's how we do gain changes. Now, let's say we don't know what a good length of time is for unmutting, so lets just make that a variable. Below is a function that takes an array of mono samples, transitions them from off to on at a given time with the given interval. I haven't tested this exact code, but it should be good enough for illustrative purposes:

void unmute( float data[],
             int totalSamples, //how many samples in our array
             int startUnmute,  //when do we start unmutting?
             int transitionLength )  //how long is our transition?
   //basic sanity check:
   if( startUnmute + transition > totalSamples )
      exit( -1 ); //or throw an exception if this were java
   //process the muted samples:
   for( int i=0; i
      data[i] = 0; //effectively multiplied by zero

   //process the transition samples.
   // this is where the linear interpolation
   // happens. We are interpolating between 0 and 1,
   // and multiplying the samples by that value:
   for( int i=0; i
      double ratio = i/(double)transitionLength;
      data[i+startUnmute] *= ratio; //multiply by the ratio, which is transitioning from 0 to 1
   // the rest of the samples don't need to be processed:
   //  they are effectively multiplied by 1 already.

Thursday, July 15, 2010

I signed a deal with the devil and now he wants my soul! Tales of Record Industry Woe...

Yesterday I visited Liz Phair's web site hoping to hear this new song where she raps. Instead I was subjected to 90's quality web-design and a short note about how rebellious her new songs are because they cost her her management and record label. "You weren't supposed to hear them," it read. It reminded me of one of those conspiracy theory web sites.

Today I read an article about business practices at major label (Mis-titled "RIAA Accounting" as if RIAA does the accounting at major labels).

Every few weeks or so, I hear a story about how evil record labels are for not paying musicians or not understanding them, or something along those lines. Search the internet and you will find lots of stories about how record labels do the accounting in order to avoid paying bands what they rightfully "deserve" or would have made under different conditions, like with a smaller label, or if they had done it themselves, or if the label had done the accounting "fairly" or something like that.

Well, most of it's true, more or less. Record labels are out to make money. Plain and simple. The record business is, er, a business (newsflash!). What's most surprising to me about this is how surprising it seems to be to so many people.

Let me see if I understand correctly. You say a deal with the devil didn't work out the way you expected?

Musicians have many goals, one of which, like the labels, may be to make money. Many of the goals of musicians are at odds with the goals of the labels. When it comes to signing the contract, though, the labels have the upper hand: musicians are desperate to get signed, and the labels have been doing this for years and they know how to make bands sign a contract that works better for them than for the band.

But let's get something straight: Investing in bands is hard. Seriously, you may think you can pick the next hit band when you hear them at the bar, but there's a lot more to it than writing a catchy song or two. Can the band deal with management? Can they work in the studio? How do test audiences respond to them (this is more important than how real audiences respond to them)? Have they sought legal council? There's a million questions many of which have nothing to do with the quality of their music and it's still a crap-shoot.

And yet, record labels are doing it: they are investing in bands. In order for it to work at all, some really ugly stuff needs to happen. Crappy pop bands need to get signed. Lousy deals need to be made, and, here comes the horrible truth you don't want to hear, most bands fail by any measure. In my opinion, major labels could probably make some more money off smaller artists if they invested more, but instead they focus on the artists they think are going to be huge, because they are big organizations structured for big payoffs. The mid-sized or potentially mid-sized artist is best served by signing to a smaller label or hiring a private publicist, and either paying out of pocket, or getting money for that in record contract. Of course, no one thinks about being a mid-sized artist when they sign a record contract.

It's not pretty, but that's music industry sausage. But, contrary to the complaining, the label doesn't want the band it's invested in to starve, either: a savvy band with good management has plenty of opportunities besides record sales to make money, and the record label has no interest in cutting into that. Usually, these sorts of proceeds are not included in these "record labels are evil" calculations, but that's probably fair, because the point of those calculations is usually to show that labels aren't paying musicians, which, by and large, they aren't. A savvy band should realize that album sales are promotion for they other items, and the label knows that other items such as touring (which, as I understand it, is increasingly becoming part of the label income, too) are promotion for the records.

A smart band will negotiate their record contract rather than just signing it. As I mentioned earlier, it's still pretty one-side, but small things can be huge wins for the band. To use an old example I happen to remember, Primus kept the rights to their demos and made a lot of money off the sales of them when their album sold well. The fact is, contracts wouldn't suck so bad for bands if bands didn't want to be signed so badly. But bands are desperate. They want contracts, so they sign them. Bands can and sometimes do walk away from the table, but that's rare, and the record contract is the only deal on the planet where someone is going to give you significant amounts of money to record and play your music. Plus, it's cool to bitch about record labels being "evil" after you are signed. Like you had nothing to do with it. "I made a deal with the devil and now he wants my soul! Can you believe how evil he is?"

Maybe that's unfair of me, but it's worth emphasizing the flip-side of the argument, when the internet is full of "record labels are evil and screw artists over" talk. I just read one article where the author compares a record deal to a loan. It's not a loan. Loans come with a promissory note, which is a promise to pay the money back. Banks only give you a loan when they think they are going to get the money back, and if they can't get you to pay, they take your collateral, which is usually something like your house. The record label knows that there's a good chance that they are not going to get the money back. Would a bank give a band a loan to make a CD? Maybe before the housing market collapse, but even then you'd need some collateral. Like the afore mentioned house. But the label isn't taking collateral. While there are many crappy things about all this, we have to appreciate that one fact. It's kinda magical.

So if it's not a loan, what is it? It's an investment contract, like venture capital. And as with venture capital, when the original investment goes big, the venture capitalist, ie the record label, gets a big cut. They also want a fair bit of control, ownership, and so on. And yea, that sucks. Talk to anybody who's started a small company with venture capital about how their investors "don't get it". The best position to be in is to have a good contract from the get-go, and to do that you need as much negotiating power as you can get. That might mean turning down the first record contract that comes along, or spending some cash on a real shark to read and help you understand the contract, or maybe you need shop your demos to the labels who are a better match for your needs, rather than just every label in town. At the end of the day it means research, hard work and compromises.

But if that still seems unfair -- and I'm not arguing it's totally fair -- I ask again where else are you going to get money to make music and tour with your band? The fact is that labels are not signing the contract to be nice to you because they think your music is awesome and they believe in awesome music. They are signing you because they want to make money. As much money as possible. I wonder why bands forget they are making a business deal when sign these contracts and that the other party wants something, too. It's not like it's free money: it's an investment in them as a business, and if they can't think of it that way, they shouldn't sign the contract.

Friday, April 9, 2010

Comments on Conversions

It didn't surprise me how impassioned a response I got to my post about converting audio data from integer to floating point. I posted a link to it on one mailing list and got some heated responses. Despite the extremely geeky nature of it, the fact is that I've seen this discussion on mailing lists before and it always seems to turn into a flame war. People put a lot of thought into implementing the simple conversion of audio from float to int and back and no matter what choice they make, they are invariably criticized for it, so it's only natural to be on the defensive.

While I contest that my post represents more thought and analysis (and better thought and analysis) than is available anywhere else publicly (certainly than I know of), I did not intend for it to be the be-all or end-all to the discussion, even if I implied otherwise. Some of the criticisms I received bordered on the absurd (it's true that my blog entry is not peer reviewed), while other criticisms were face-valid, but irrelevant (whether one solution is more pleasing mathematically is irrelevant if it is going to produce worse sounding results). However, digging though the criticisms it's apparent that some things from my analysis can be improved.

To that end, I'm going to use this entry to accumulate comments and thoughts that need to be made on the subject as they come up and/or need to be made. So this is a living blog post that will be updated and revised from time to time.

April 9th 2010

- I claimed that looking at the no DSP case was a "best case" situation, and that any DSP would only make whatever distortion occurred worse. Therefore, I argued, this was the only case that needed to be considered. Not everyone agrees with this, but it's also hard to generalize DSP. I might be worth analyzing some simple DSP like volume ramping.

- I contrasted the distortion produced by using the wrong conversion method to the distortion created by not using dither. However, error produced from truncation is most objectionable with low-level signals while only high-level signals were tested, so this is not a fair comparison.

- It would be worthwhile to test conversions from 24-bit to 16-bit of several different audio source types to determine if the harmonic distortion of the (2^n) model is relevant in that case.

Friday, March 26, 2010

Java (No)FX - why one project dropped JavaFX for Java

There's a lot of FUD out there. For some reason, it seems like Java and JavaFX take a hard hit. I've heard nonsense like "no one uses Java anymore," and other such stuff. Unfortunately, some of the JavaFX FUD might be true: I recently completed a project which was little more than replacing a buggy, slow JavaFX UI for a Java applet with a fast, pure Java UI of the same applet. Thank God we did, because the applet is way better now.

The JavaFX portion was, at one point, the largest JavaFX codebase in existance, and, indeed, the first serious large JavaFX project not funded by Sun. And the company I was working with dropped JavaFX for plain old Java.

Although I can't say what the project was, I can say that this is a project that most companies would have tried to use Flash for, but, in the end, Java provided features that competitors using Flash simply can not offer. As far as I know, this company is now the only company with these features in a browser because they went with a Java foundation.

Now keep in mind that a large part of the problem, here, is that this was really the first major JavaFX project, so it was bound to have difficulties. We had support from Sun, but I think Sun did not realize how far they had to go and how many bugs they still had in their runtime. In my opinion, nobody, neither us nor Sun, was aware of the problems of JavaFX. Despite it's version number (1.1 when we started and 1.2 by the time we abandoned), we found it to be buggy. Had the version number been 0.7, I would right now be saying JavaFX is the coolest thing ever, but the truth is that it's still a nascent technology that has yet to prove itself.

Lack of Competent Developers

Symptomatic of a new technology, we had a hard time finding quality developers. Sun gave us some leads but these developers were simply not up to the task of building our complex app. JavaFX is not a difficult programming language, so having someone in-house learn it would have been preferable, but at the time we did not have the resources.


The performance of the JavaFX portion of our app was extremely poor. The extremely performance critical portions of the app were written in Java and performed well, but almost all the graphics were written in JavaFX. Basic graphics like buttons were okay, but complex drawing was very slow. I don't know for sure if this was a result of poor design or JavaFX itself, but from the code I perused it looked like a little of both. We know JavaFX was at least partly to blame because we had performance issues even when we commented out the complex calculations.

Unstable API

Transitioning from JavaFX 1.1 to JavaFX 1.2 turned out to be very difficult because of backwards compatibility issues. Unfortunately, at a certain point in our project, our app was performing so badly with even rudimentary graphics tasks that Sun and our JavaFX developers insisted on upgrading.

Lack of Quality Control

At one point, after we released a baseline product with minimal features, Sun released a new upgrade to JavaFX. Unfortunately, this meant that JavaFX libraries would be updated on all machines that ran our app. Since there was a bug in the library, our app stopped working.

Poor Developer Communication

In an effort to work around the bug, we asked Sun for the developer version of the new library. Unfortunately, they were not forthcoming. At that point, the decision was made to stop development on the JavaFX solution and seek an alternative. After I convinced the team that Swing was capable of looking great, we developed a pure-Java alternative and have released the new product. The graphics performance is orders of magnitude better, and the look is similar to Flash. People who have seen it so far have gone out of their way to complement the appearance of the app. The appearance matches the design spec almost perfectly except for a few things we have not yet had a chance to attend to. (I am belaboring this point because Swing has a reputation for being ugly, simply because most of the included Look and Feels are ugly)

Future Seems to Be Too Business Oriented

When asking Sun about the future of JavaFX, they said they plan to offer certification levels. It's bad enough that Java skills are based on memorization rather than problem solving abilities now, but I can sort of understand that given the business orientation of Java, and maybe businesses look for that sort of thing.

If hip kids don't learn Java, that's okay. There's more than enough Java programmers to keep Amazon, ebay, oracle and all those other Java giants going. But JavaFX is not like that. JavaFX is a creative tool. It's not a direct competitor to flash, but it's in the same vein, and those people aren't going to take a certification exam. Sun needs to think about what is going to make people think JavaFX is awesome, and certification is not it. Neither is having Gosling slinging T-Shirts at them. JavaFX really really has the potential to be awesome, and I really really want it to be, so here's what I think you need to do (Sun, I'm talking to you):

  • Show people that you are serious about making it awesome. Hire some awesome programmers and have them blog and tweet or whatever the cool kids are doing these days about what they are doing. Really. Hire the best. Hire some young folks. Hire some experienced old guns. Mix and match. Make sure you let them be honest on those blogs.
  • Talk to some cool startups that do mobile stuff like venmo about what they are doing in the mobile sphere and work with them proactively.
  • Suck it up: work with the android people. Sure they stabbed you in the back, but you need them.
  • Help out Apple: really, their JVM sucks. It seems like I am filing bug reports every week. Make sure they get it right because cool geeks use Apple.
  • While you're at it, make sure your JavaFX staff has Macs and other cool toys. Remember, you need to treat those folks as creatives, not code monkeys.