FFT in XNA Content Pipeline for Beat Detection For The Win!

February 5, 2011 - 23:55 — jwatte

I worked a bit more on the FFT-based music visualizer program I started the other day, and it's doing great! A this point, I have a pretty solid rhythm extraction thing going, and I can use autocorrelation (through autoconvolution) of the extracted rhythm track to derive tempo. I'm assuming 4/4 beats here, but... that's a pretty safe assumption for any popular music :-) (And a lot of not-so-popular music, too)

The cutting plane in this screen shot is the "play position" of the song being played back, and it flashes when my "rythm event" detector detects a hit; for four-on-the-floor music, it's very correlated with the bass drum, but even for sparser or denser type mixes, it does a great job of finding useful "hit points" for events in sync with music. Once I have tempo, i can detect sudden shifts in adherence to the tempo, and/or density, to trigger "style" changes, too.

I've attached my music processor used as a pre-process here. You may re-use it under the terms of the MIT open source license. Also note that this requires the djbfft library, which is available under a similar license.

Attachment	Size
disco-music-2.jpg	86.48 KB
fft-processor.zip	317.98 KB

Add new comment

Comments

November 6, 2011 - 05:19 — Anonymous

MusicLibrary.csproj

Hello.
The example is not runnable. MusicLibrary.csproj is referenced from MusicProcessors.csproj, however MusicLibrary.csproj is missing in a zip file. Would you please post it?

November 6, 2011 - 05:18 — Anonymous

MusicLibrary.csproj

Hello.
The example is not runnable. MusicLibrary.csproj is referenced from MusicProcessors.csproj, however MusicLibrary.csproj is missing in a zip file. Would you please post it?

July 1, 2011 - 15:23 — Anonymous

MusicLibrary

MusicLibrary seems to be missing from the source files? Thanks for the article!

April 14, 2011 - 15:13 — Anonymous

Question

May I ask you to elaborate on how exactly to get this project up and running? I've tried installing DJBFFT on a unix system, but when running the makefile, it produces an error

"./load auto-str substdio.a error.a str.a
/usr/bin/ld: errno: TLS definition in /lib64/libc.so.6 section .tbss mismatches non-TLS reference in substdio.a(substdo.o)
/lib64/libc.so.6: could not read symbols: Bad value"

I believe this is resulting in my next problem; where the project cannot find the referenced "MusicLibrary"

Is there any way that I can download a compiled dll for DJBFFT?

I'd love to use this for a game I'm working on that uses music to enhance the game.

Thanks!!

April 15, 2011 - 07:45 — jwatte

The source is already included

Perhaps the error comes from you trying to build a .so file on a 64-bit system, when what you need is a .dll file on a 32-bit system?

Also, you will probably have to edit the DJBFFT files slightly, as they forward declare a number of stdlib functions, that may have variances in their declaration on your particular system (looks like TLS attributes in your case).

Anyway, the source, as needed to build for Windows, is actually included in the downloadable ZIP file, so if that's what you want, why not just use that?

February 22, 2011 - 16:40 — Anonymous

adding more bands

So I was experimenting with adding more bands (ended up with a lot of them that don't respond at all), but I have a good list of all the active fft bands. However, even though I change things around to not use '32' all the time, the beat detection seems to mess up, and the bands seem more messed up in general. Could you point out the areas of the code where that should be changed? or have it use mappedBuckets.Length everywhere?

February 23, 2011 - 14:50 — jwatte

Wish I had a magic bullet

Wish I had a magic bullet that would always detect the beat of a given piece of music :-)

I don't quite understand why some of the bands in the FFT are "empty" -- it may be that the algorithm generates the frequency alias of the band instead, wrapped around Nyquist. I haven't had time to look into that just yet.

Suppose we use 48 kHz sampling rate and 1024 bands, then each band is about 27 Hz in width. This means that the first ten bands cover the 0-270 Hz range, which should be quite sufficient for beat detection -- the bands above that are quite likely to just contain overtones and sustained notes that do not easily lend themselves to beat detection. Hence, I haven't found any need to go into a large amount of bands.

One mechanism that's useful for beat detection is autoconvolution, perhaps hinted through a tempo extracted through looking for level triggers in specific bass bands (which is what I'm currently doing). That's well down the road, though, as I'm not trying to make this a commercial quality pro music tool :-)

Let me know if you do anything interesting with this.

May 3, 2011 - 12:55 — Anonymous

Re: bands with wierd data

I had a quick peek at your code and notice you aren't doing any windowing before taking the FFT of the wave data.

If you don't window you will in effect get discontinuities where the two ends of the window don't meet exactly - this will cause frequency bleeding across lots of bins and generally ruin the nice output from the FFT.

I suggest looking up the "Hann" window, and apply it before you perform your FFT.

The latest git version of the Spek project includes a patch to use a windowing function and it really clears up the spectral data.

Cheers,

Dan

May 3, 2011 - 19:58 — jwatte

That's true; there are a

That's true; there are a variety of windows you can apply, and I didn't apply any, which is pretty shoddy :-)

Hann is just one of them -- Blackman, Hamming and other raised-cosine are others that I've used in the (distant) past, and I think there's one window based on the first lobe of the sinc function, too.

Generally when using FFT, though, I use it for the multiply-accumulate method of convolution, where windowing would actually be harmful, rather than analysis; that's the code that I snarfed this particular application from, and hence why it's pretty "ghetto" like you suggest.

Thanks for the feedback!

May 9, 2011 - 18:28 — Anonymous

Everything you never wanted to know about Jon Watte

Google Ads