Panic Blog

iTunes 11 is a radical departure from previous versions and nothing illustrates this more than the new album display mode. The headlining feature of this display is the new view style that visually matches the track listing to the album’s cover art. The result is an attractive display of textual information that seamlessly integrates with the album’s artwork.

After using iTunes for a day I wondered just how hard it would be to mimic this functionality — use a source image to create a themed image/text display.

The first step in replicating iTunes theming is obvious: getting the background color used for the track listing. This seemed easy enough, just use simple color frequency to determine the most prevalent color along the left hand side of the artwork. Doing a simple color count gives pretty good results, but looking at iTunes it was clear there was more to it than just that. I proceeded to add a bit of logic to add preference for colored backgrounds instead of just using black and white when those were the most prevalent colors. Doing this presents more interesting styles since seeing only black and white backgrounds would be a bit boring. Of course you don’t want to replace black or white if those colors really are dominant, so I made sure that the fallback color was at least 30% as common as the default black or white.

Once I started filtering black and white backgrounds my results started to get a bit closer to iTunes. After doing some more analysis I saw that iTunes also looks for borders around the artwork. So lets say you have a solid white border around the artwork picture, iTunes will remove the border and base its theming colors off the remaining interior content. I didn’t add this functionality as it was outside the scope of my simple demo application.

After the background color was determined, the next step is to find contrasting text colors. Again, the first thing I tried was simple color counting, this provides surprisingly good results but iTunes does better. If we relied only on color frequency you’d get variants of the same color for the different types of text (EG: primary, secondary, detail). So the next thing I did to improve the results were to make sure the text colors were distinct enough from each other to be considered a separate color. At this point things were really starting to look good. But what other aspects would need to be considered to ensure the text always looked good on the chosen background color? To ensure colorful text I also added a bit of code to make sure the color used for the text had a minimum saturation level. This prevents washed out colors or very light pastel colors from being used that might not give the best appearance. Now that the text had unique colors that looked good with the background, the only remaining problem was that the resulting text colors could end up lacking enough contrast with the background to be readable. So the last thing I added was a check to make sure any text color would provide enough contrast with the background to be readable. Unfortunately this requirement does cause a rare “miss” when finding text colors which then cause the default black/white colors to be used.

The end result looks something like this:

It’s not 100% identical to iTunes — sometimes it’s better! Sometimes just different — but it works pretty well overall.

You can see exactly what I did in the following Xcode demo project:

A few notes about this demo. I did very basic frequency filtering to prevent random colors from appearing as text colors. In my case I chose to ignore colors that only appear once. This threshold should be based on your input image size since smaller images won’t have as many pixels to sample from. Another processing technique that iTunes does, that I would also do if this were shipping code, is to look for compression fringing around the edges of the image. I’ve noticed a few cover art images that contain a single pixel edge of white/gray fringe that should be ignored and removed before sampling for the colors.

(Last but not least, this code was written in a few hours, and is very rough. So just in case you have thoughts about speed or optimizations, please note it was more of a thought exercise than a lesson in algorithm design. Engineer disclaimer complete.)

That being said, I hope this is somewhat interesting! It shows that with just a bit of work you too can have fancy themed designs too.

UPDATE: Thanks to Aaron Brethorst, this code is also now on GitHub.

Let’s face it (sorry): face detection is cool. It was a big deal when iPhoto added Faces support — the ability to automatically tag your photos with the names of your friends and family adds a personal touch. And Photo Booth and iChat gained some awesome new effects in OS X Lion that can automatically track faces in the frame to add spinning birds and lovestruck hearts and so on. While not always productively useful, face detection is a fun technique.

I’ve seen attempts at duplicating Apple’s face detection technology. (Apple is far from the first company to do it.) There are libraries on GitHub and various blog posts for doing so. But recently I realized that Apple added support for face detection in OS X Lion and iOS 5. It seemed to slip under my radar of new shiny things. Developers now have a direct link to this powerful technology on both platforms right out of the proverbial box.

Using Face Detection through Core Image

Apple’s face detection is exposed through Core Image, the super-useful image manipulation library. Two classes are important: CIDetector and CIFeature (along with its subclass, CIFaceFeature). With a little experimenting one night, I was able to get a sample app detecting faces within a static image in about 10 lines of code:

// Create the image
CIImage *image = [CIImage imageWithContentsOfURL:[NSURL fileURLWithPath:@"Photo.jpg"]];
 
// Create the face detector
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:CIDetectorAccuracyHigh, CIDetectorAccuracy, nil];
 
CIDetector *faceDetector = [CIDetector detectorOfType:CIDetectorTypeFace context:nil options:options];
 
// Detect the faces
NSArray *faces = [faceDetector featuresInImage:image];
 
NSLog(@"%@", faces);

Note the dictionary of options. There is only one particularly useful key: CIDetectorAccuracy. It has two possible values: CIDetectorAccuracyLow and CIDetectorAccuracyHigh. The only difference: There seems to be additional processing performed on the image in order to detect faces, but at the cost of higher CPU usage and lower performance.

In cases where you are only apply detection to a single static image, high accuracy is best. Low accuracy becomes handy when manipulating many images at once, or applying the detector to a live video stream. You see about a 2-4x improvement in render time with low accuracy, but face tracking might pick up a couple of false-positives in the background once in a while, or be unable to detect a face at an angle away from the camera as well as high accuracy could.

Now that we have an array of faces, we can find out some information about each face within the image. CIFaceFeature exposes several useful properties to determine the bounding rectangle of the face, as well as the position of each eye and the mouth.

Using these metrics, it’s then possible to draw on top of the image to mark each facial feature. What you get is a futuristic sci-fi face tracker ala the Fifth Element. Leeloo Dallas Multipass, anyone?

// Create an NSImage representation of the image
NSImage *drawImage = [[NSImage alloc] initWithSize:NSMakeSize([image extent].size.width, [image extent].size.height)];
[drawImage addRepresentation:[NSCIImageRep imageRepWithCIImage:image]];
 
[drawImage lockFocus];
 
// Iterate the detected faces
for (CIFaceFeature *face in faces) {
    // Get the bounding rectangle of the face
    CGRect bounds = face.bounds;
 
    [[NSColor colorWithCalibratedWhite:1.0 alpha:1.0] set];
    [NSBezierPath strokeRect:NSRectFromCGRect(bounds)];
 
    // Get the position of facial features
    if (face.hasLeftEyePosition) {
        CGPoint leftEyePosition = face.leftEyePosition;
 
        [[NSColor colorWithCalibratedWhite:1.0 alpha:1.0] set];
        [NSBezierPath strokeRect:NSMakeRect(leftEyePosition.x - 10.0, leftEyePosition.y - 10.0, 20.0, 20.0)];
    }
 
    if (face.hasRightEyePosition) {
        CGPoint rightEyePosition = face.rightEyePosition;
 
        [[NSColor colorWithCalibratedWhite:1.0 alpha:1.0] set];
        [NSBezierPath strokeRect:NSMakeRect(rightEyePosition.x - 10.0, rightEyePosition.y - 10.0, 20.0, 20.0)];
    }
 
    if (face.hasMouthPosition) {
        CGPoint mouthPosition = face.mouthPosition;
 
        [[NSColor colorWithCalibratedWhite:1.0 alpha:1.0] set];
        [NSBezierPath strokeRect:NSMakeRect(mouthPosition.x - 10.0, mouthPosition.y - 10.0, 20.0, 20.0)];
    }
}
 
[drawImage unlockFocus];

With a little more work, it’s pretty easy to apply this technique to live video from the device’s camera using AVFoundation. As you get back frames from AVFoundation, you perform face detection and modify the frame before it is displayed. But I’ll leave that as an activity for the reader. :-)

And amazingly, it even works with cats.

With a little more effort, I was able to grab the closest detected face’s region of the image, and do a simple copy-and-paste onto the other detected faces (adjusting for angle and distance, of course). Behold… Panic’s newest, most terrifying cloning technology!

Here’s a little sample app. Have fun!

We get a lot of support questions via Twitter which, believe it or not, we love. It forces people to ask questions succinctly, and it forces us to answer succinctly. We think it’s a pretty great way to answer questions.

But the part we don’t love is trying to use the Twitter website, or a third-party web service like “ExactTarget SocialEngage”, ugh that name, to answer these questions. There’s just something inherently slow and inefficient about using the web for rapid-fire, high-volume tasks like answering support questions.

So, we built Burnside, a Tweet-to-Email Gateway.

Tweets go into a regular mailbox. When we reply, they go back out as tweets. Since multiple people can work out of the same IMAP box, it’s fast and efficient. And as an added bonus, as we archive tweets one by one, we’re building our own searchable index of tweets in Mail. Try searching for anything older than a week on the Twitter website!

(And what about the 140 character limit? We also wrote a plugin for Apple Mail that displays a character count in the Mac OS X Mail window.)

Burnside is notable for another reason: it’s a unique Panic foray into open source. Usually we keep these things to ourselves, but why?

We’ve just put Burnside up on GitHub for anyone to tweak and install.

FAIR WARNING: Burnside is for system administrators only. This is not a consumer app, and it requires significant configuration and understanding of your mail server.

Today’s Mountain Lion announcement introduces an important new security feature, called Gatekeeper, in addition to the “sandboxing” feature that premiered in Lion. I’d like to talk a little bit about it, and why it’s important to all Mac users.

Malware is out of control. Almost every day I read a new article about a major security breach in a well-known organization. There is big money to be made from stolen credit card numbers and identities. End-user applications on individual computers are a prime attack vector because, even with the best tools and the best programmers, vulnerabilities sneak their way in. Trying to make applications free of vulnerabilities (while still an important goal) is to lose the overall cat-and-mouse race.

As Mac users, we’ve mostly enjoyed a life free of the worry that has followed Windows users for years. Mac OS X is pretty damn secure. But it could be more secure. As Macs enjoy increased popularity, they become a more attractive target to identity thieves and other criminals. Sooner or later, bad people ruin every nice thing. It’s an immutable law of humanity.

So, what to do about this? Code-signing, although it can’t single-handedly fix the problem forever, is a vital weapon in the fight against malware. But many folks are unclear on how it works, or how it helps. Let me try to explain in as close to plain English as I can.

An explanation of code-signing for humans

What is code-signing? Let’s start with a slightly higher-level question: what is signing? Signing is based on technology similar to encryption, so let’s discuss them both broadly.

One of the most prevalent and secure methods of encrypting or signing data is to use what’s called a “key-pair”. As the word “pair” suggests, this means there are two keys which can “unlock” the encrypted data in certain ways.

A “key” is literally just a number. But it’s a very big number, and this is important. If I asked you to guess a number between 1 and 100, you’d have a 1% chance of guessing it on your first try, and you’d be guaranteed to guess it correctly if I gave you 100 tries. But what if I asked you to guess a number between 1 and 3 trillion? That’s a bit more of a challenge.

You’ve probably heard at least in passing about encryption keys and that they have different sizes or lengths (such as 40-bit, 128-bit, or 256-bit). Just like in my number guessing example above, longer keys are harder to guess. Each additional bit that is added to a key makes it exponentially harder to guess or figure out by brute-force attempting to decrypt the data with every possible numerical key. (Is 1 the key? No. Is 2 the key? No. Is 3 the key? Is 3,426,989,662 the key? No.)

We want encryption keys to be very long so that brute-force guessing attempts would take literally thousands of years. They become an unreasonable attack vector given the current average human lifespan.

So, why two keys? In key-pair encryption, one key is called the “private key” and the other is called the “public key”.

The keeper of the private key is able to “sign” data; a process which both identifies its origin and provides reasonable proof that it has not been altered. Private keys must be guarded very carefully, so that signatures cannot be forged.

The public key, as its name suggests, may be distributed freely. In encryption, the public key can be used to encrypt data which can only be read by the owner of the corresponding private key. In other words, with my public key, you could send me a secret message that only I could read.

In signing, the public key can be used for another purpose: to verify (with an extremely high degree of mathematical probability) that a “signed” piece of data came from me. Or, to be more specific, could only have come from someone with access to my private key. Which, hopefully, is just me.

In a nutshell, that’s what signing is. Even without actually encrypting it, I can take a chunk of data, run it through a very complex mathematical process to “sign” it with my unique private key, thus generating a second chunk of data called a “signature” that could (statistically speaking) only have come from that specific combination of data chunk and my private key.

Anyone with that signature and my public key can then be almost 100% sure that data came from me, and that it was not modified by any third-party along the way. The data could’t have any virus or vulnerability injected into it, because then the signature would no longer match the data.

So, signing allows us to, with very high confidence, ensure that we are who we say we are, and that the data we produce really came from us. Code-signing, then, is simply applying that signing process to executable code like a Mac app. If I try to start up an app, the operating system can validate that the app’s signature is valid, and perhaps also that it is the signature of a known, trusted developer. If it doesn’t pass muster, the OS can refuse to run the application.

Which brings me to Gatekeeper.

The role of Gatekeeper

The iOS devices (iPhone and iPad) effectively have had a Gatekeeper built into them since their very first release. When we write an iOS app, we sign it, then send it to Apple to review. Apple can validate the signature to ensure that it hasn’t been tampered with — that it really came from us — and then it goes into the app review process.

If the app passes review, it is then signed again by Apple, and posted to the App Store. Since Apple is the only entity able to sign App Store applications, iOS will simply refuse to run any app that doesn’t have Apple’s signature — it obviously didn’t come from the App Store. (If you “jailbreak” an iOS device, this is the security check you are bypassing. You are lobotomizing iOS so that it will merrily run “unsigned” code from any source. As you can hopefully tell by now, this has both benefits in terms of flexibility and very significant risks in terms of security.)

But how to bring this level of security to Mac OS, which has always allowed unsigned code from any source to run more-or-less without restriction?

The simplest thing Apple could have done would have been to make the Mac App Store the sole source for Mac apps, in the same way the App Store is the sole source for iOS apps, shutting off every other app distribution venue in the process. While this would have immediately solved the problem, you would have seen developers’ heads bursting into flame and flying across the room in rage. Why?

Although security is a vital feature for Apple, developers, and users alike, being unable to run unsigned code cuts a lot of really great things off at the knees. You wouldn’t, for example, be able to just download and run an open source project unless it had been submitted to and reviewed by the App Store. Highly disruptive software (think Napster or BitTorrent) may have not been able to exist on the Mac platform since it would have been likely to run afoul of Apple’s App Store guidelines. Major vendors such as Adobe and Microsoft might have withdrawn their support for the platform, being unwilling to cede 30% of their revenue to App Store distribution.

So, for a while, there was a great deal of consternation among Mac developers, including this author, that this might be the route Apple would take. In recent years, Apple has shown a trend of following the most hardline possible stance that will benefit users and Apple, often at the expense of developer freedom, and gradually backing in certain affordances (push notifications, for example) as user-impacting problems became evident. So it seemed feasible that we’d wake up one day and Apple would decree that all Mac apps must be sold through the App Store.

But instead, Apple went to considerable effort and expense to find a middle ground.

Controlling Gatekeeper in Mountain Lion

In Mountain Lion, you, the user, have three options:

1. You can let anything run on your system, whether or not it is signed. This is the Mac OS of today. It’s like having a jailbroken iPhone.

2. You can allow only Mac App Store apps to run on your system. This is the most secure option, but you lose the ability to run non-App Store software, which currently includes such products as Microsoft Office and Adobe CS.

3. You can allow only Mac App Store apps or apps signed by a developer. This is the new default.

It’s this third option that is critical. As a developer, I can register for a unique ID which allows me to sign my app but does not require it be sold through the App Store. Users get the benefit of knowing the app came from a trusted source. But I retain the ability to sell my app directly to end users.

If my app were to do something nefarious, my developer ID would get revoked and that would be the end of that. My app would no longer be allowed to run (unless you specifically allowed unsigned apps). As a matter of fact, if you try to launch an unsigned or unvalidatable app on a Mac with Gatekeeper enabled, the default button is “Move To Trash”. Pretty hardcore. Kind of awesome.

It is really quite a nice compromise.

I have a personal flaw in the form of a small conspiracy theorist who lives in my head. He worried that this may have been created as just a temporary stepping stone — like Rosetta for the Intel transition, or Carbon for the OS 9 to OS X transition — and that one day, the Mac App Store-only option might still be enforced.

But I can’t find it in me to disparage this goodwill effort that Apple has undertaken to not turn every third-party developer upside-down with regard to app distribution. To me it’s a great sign that they’re aware and at some level sympathetic to our concerns, while remaining committed to a high-security experience for users.

Further cementing this feeling is the fact that we were invited to a private briefing at Apple about Gatekeeper a week before today’s announcement. Cabel was told point-blank that Apple has great respect for the third-party app community, and wants to see it continue to grow — they do not want to poison the well. I think their actions here speak even louder than their words, though.

One worrisome rift

There remains one thing that is of concern to me. Despite these great strides forward, Apple is walking a dangerous line with regard to features that are only available to App Store distributed apps. The two most prominent examples are iCloud and Notification Center. Cabel asked Apple if, thanks to Gatekeeper and Developer ID, App Store-only features would be eventually be available to signed apps that were not distributed through the App Store. There was some shuffling of feet and a “we have nothing to announce at this time”. It didn’t sound particularly optimistic.

It would be a shame if this trend continues, as it creates an artificial gulf between App Store and non-App Store apps. For example, as things stand today, we won’t be able to offer iCloud syncing in, say, Coda 2, when you purchase it directly from us. Only App Store purchasers would get that feature. Making matters worse is Apple offers us no real facility to “cross-grade” you from a direct purchase to an App Store purchase, should you change your mind.

There’s no real engineering reason that I can think of for this. It seems marketing or money-driven, and I think it’s un-Apple-like to chase the money at the expense of user experience in that manner. We hope they change their minds about that particular facet.

Moving forward

Other than that though, we think Gatekeeper is a bold new feature that should do wonders for the security of your Mac for years to come. Even though their rapid pace of development is at times difficult for us to keep up with, we are excited that Apple continues to aggressively push the envelope when it comes to keeping Mac OS X safe and secure.

When something new goes live at Panic, the first to know about it are the dedicated followers of our Twitter account.

And when we tell people there’s something new, like a software update, we really like to provide a specially crafted URL that will allow that person to get said update with one click. We built this functionality into Transmit and Unison, and it makes for a great Twitter experience: here’s a new thing, click here to get it, boom, done.

When the Mac App Store was in beta, I was really hoping for something similar — a simple way to notify a user of an App Store update, with a link to take them to the update in one click.

This is where I tell you something amazing: I filed a bug with Apple and made this specific feature request, even proposing the URL format, and Apple added it to the App Store a couple versions later. The system worked!

So now I’ll reveal the secret to you. To send your users to Mac App Store updates, use this URL:

macappstore://showUpdatesPage

That’s it. You can see it at work here:
 

Coda 1.7.5 has also just hit the Mac App Store! For your convenience, update it here: j.mp/masupdate Enjoy. Now, we go back to 2. —C

— Panic Inc (@panic) December 13, 2011

(We built a redirector since URL shorteners don’t like funky URL types.) And that’s all there is to it.