To a beginning Processing user, the libraries section of the Processing site can cause a "kid in the candy store" effect. I actually recommend going a little easy, working with one library at a time. But, at the same time, working with libraries really is essential to working with Processing, because Processing "core" is kept as conservative as possible. This is especially true if you’re working with multimedia. Here are the libraries I recommend for specific multimedia applications, as a starting point:

Sound and MIDI

genPunk uses the Ess library to build an entire music app inside Processing. But you may be best off starting with Minim.

Minim is currently the most reliable sound library available, because it hooks directly into JavaSound, as well as the easiest to use and best-documented. It’s worth looking at the other libraries here, but this should be your first stop as a beginner.

(Note that JavaSound tends to be more reliable on Linux and Windows because of some implementation issues with Apple’s version of Java.)

proMIDI is currently the only library for interfacing with MIDI support. I recommend it with some trepidation, because there are some odd things about implementation — note off events don’t work quite right, for instance. But for basic MIDI data, it works. If you’re planning to get heavy into MIDI, I actually suggest accessing JavaSound’s MIDI tools directly; it’s a little tricky, but there’s superb documentation at jsresources.org, which is maintained by the lead developers of the modern version of JavaSound. Another tool to consider is jm-Etude; by working with the excellent jMusic API, it makes writing MIDI code much more musical.

Video

Shadow Monsters, an audiovisual piece by Phil Worthington, uses blob tracking to create an unusual puppetry effect. It uses JMyron for analysis.

First: start with the internal video library. There’s actually a lot you can do here. If you get weird error messages (assuming you’ve read the documentation for platform-specific caveats), try reinstalling QuickTime — buggy QT installers have a tendency to randomly wipe out Java support, and reinstalling sometimes just fixes the problem.

Then, libraries…

Short version: Get JMyron and go to town. Make sure you have the Intel build if you have an Intel Mac.

Long version: Remember the "Read Me" file you ignore? This is a time when you need to read the documentation. See the documentation of Processing’s own video library for some important caveats and bugs on all three platforms. The Mac should work, though sometimes can have random crashes or problems with capture. Windows requires the often-unpredictable WinVDIG for input; Linux doesn’t work at all. Don’t blame Processing: blame Apple’s now-defunct QuickTime for Java, and the general lack of any Windows/Mac cross-platform library.

But don’t give up. There are some nice things that can be done with computer vision on Processing.

I especially recommend JMyron for basic computer vision techniques. Note that you’ll need to download an Intel build of JMyron’s jnilib file to make it work on Intel Macs; you’ll get some odd error messages otherwise. That Intel recompile is on the jibberia (aka Kevin Cox) projects page. (Look for the following text):

libJMyron.jnilib compiled for intel macs replace the one in /Applications/Processing/libraries/JMyron/library/ with this one. yay

One other note: if code for JMyron contains the getForcedWidth() or getForcedHeight() calls, you actually need to remove that for the code to execute without throwing an error on Mac. On Windows, you may need to call those somewhere to avoid weird video problems. Don’t ask. Video can be a pain.

The BlobDetection library also has some nice CV features, but no special capture features. I do not recommend LibCV for capture; the odds of you getting the ancient Sun JMF media library on Java to work at all are, optimistically, extremely slim. But like BlobDetection, it also has some interesting features.

Also, don’t overlook the built-in capabilities in Processing:

pixels[] is an array of pixels in the display window. (If you’re using JMyron, you can copy pixels from JMyron’s capture and analysis features into that array.) A lot of computer vision simply involves looking at what’s contained in this array.

3D

OBJ Loader … um, loads OBJ files (example pictured at right). With plug-ins, this includes, for instance, files exported from Google’s SketchUp. (I believe you need the paid version of SketchUp, not the free version, for this feature — check out the super-cheap academic pricing if you qualify.) It’s the easiest way to get 3D models into Processing.

surfaceLib is the library that does a lot of the tricks that allow those gorgeous, pulsating 3D objects you see in a lot of the Processing videos. But even if that’s not your style, it provides key capabilities like the ability to figure out the coordinates on a sphere without having to crack open your old Calculus book.

Dan Shiffman’s Vector3D is useful for other math tricks, covered in his class’ examples for lovely 3D animation effects — again, without a lot of wading through math texts or duplicating code over and over again.

Physics

traer.physics is a very elegant, simple physics library that’s great at making springs and particle systems. It does not do everything it’s possible for a physics library to do — but it does work in three dimensions (take that, Flash!), and its simplicity makes it a fantastic starting point. It’s also very fast. (Take that again, Flash!)

As the site there says, you just get coordinates, so once you’ve got that you could, for instance, use image() to draw a particle or create geometry or whatever you want.

Prototyping and live performance

When the team working on the Muon installation wanted to adjust the installed version of their artwork, they used the controlP5 library to get exact feedback on what they were doing.

How much time do you spend re-entering numbers for variables to try different settings? That’s fine when you’re first working on code — less so when you’re trying to play live or do an installation. Here’s the solution:

controlP5 has two key features. First, it makes controls (sliders and such) so you can easily adjust variables. Second, it hides those controls when you want them out of the way. So, for instance, you get to a site for an installation, show your controls, make your adjustments, hide your controls.

Java has more powerful features for UI, but for art projects you probably don’t want that much complexity. controlP5 should have you covered.

This is not a complete list

It’s a field guide to some of my favorites. As you can see, it still winds up being pretty long — which is the reason Processing uses libraries in the first place. Once you start adding features to core, it gets complicated fast. In this way, you stay close to the applications for which the features are actually being used — and you can pick and choose just what you need.

As always, check the libraries page for a wide variety of libraries — and keep your eyes on the Processing Blogs or forums, where things often show up first.

You can also make your own libraries according to instructions in the libraries folder of the Processing install. On the top of my list to put together:

• IMAP support
• Flickr API
• Native capture on Mac, Windows, and Linux for video
• Video playback on Mac, Windows, and Linux
• A new MIDI implementation

So if I finish any of those, I’ll be sure to share on our CDM Labs (and probably brag about them on CDMotion and CDMusic, too).