A few nights ago I was updating some not-to-be-named software on my laptop. This piece of software had a few passwords stored in the Keychain. Since said application was recently updated and therefore the code was modified, the system asked me if I wanted to give access to the keychain to this updated application. The dialog that it shown to the user is shown below:
Bad things happened when I clicked the "Change All" button to once again allow this updated application to access all the passwords it was allowed to access. Specifically, the securityd process was using 1.3-1.7GBs of ram (the rprvt value is all that matters). This was really, really bad as it caused my machine to page-out and page-in like crazy. Due to the high memory usage, it also caused my boot volume to run out of space because of all of the swap files in /var/vm/. My point is that very, very, very bad things happened. After I cleared a lot of unused crap (Garage Band loops and old iDVD themes) off my boot volume, I rebooted. I then tried launching the updated application again. I got the same dialog and the same problem. However, since I now had enough hard drive space available, I just waited for about 10 minutes. The passwords were accessed successfully. I then relaunched the application and securityd crashed. Lovely. Rebooting just repeated the cycle. Also lovely.
So I clearly had a problem. I remember seeing this problem before while browsing the internets. I specifically found it on 43 Folders. I started searching The Google for any solutions. The searches brought up some really amusing "solutions" such as a thread from Mac OS X Hints. People are talking about some crazy voodoo solutions there. Examples include updating the firmware or adding a "keychain" extension to the user's keychain. Some voodoo causes are also mentioned. One such example is that it is cause by syncing keychains between an ICBM and a PowerPC-based Mac.
The Troubleshooting
Ok, so obviously there were no known solutions to the problem that I could find. This kind of pissed me off. Especially since I had to wake up at 5am to make it to a Microsoft Vista launch event. It was 3am at the time. Obviously, there wasn't going to be any sleep for me.
I decided to do my own troubleshooting. The crash of securityd clearly pointed to a corrupted file of some sort. So I had to determine which file was hosed like a bear coming into contact with the strikingly handsome Stephen Colbert. First, I restored all my keychains from known-good backups that another Mac was actively using. I rebooted and tried to reproduce the problem. Bloody 'ell! The problem remained. So it obviously wasn't that the keychains were corrupted. I rebooted again, started up the terminal, ran sudo fs_usage -f filesys securityd to see what files securityd was accessing when it was trying to update the keychain reference for the application. /var/db/SystemKey repeatedly showed up. However, I was very, very tired and getting very, very angry so I decided to go with a machete approach to fix the problem. I ran strings on securityd and it turned up four "interesting" files and folders:
/var/db/SystemKey
/var/db/TokenCache (this is a folder).
/var/db/SystemEntropyCache
/var/db/CodeEquivalenceDatabase
Since I was so angry, I just "moved aside" three of these. "Moved aside" meaning that I just renamed everyone of them to have a ".old" extension using the Terminal (via mv). The TokenCache folder did not exist on my ICBM so I couldn't move it aside. I rebooted. The problem was fixed!. I then went to the Microsoft event. I emailed the very sexy Merlin Mann with the possible shoot-everything fix. He reported back that it fixed the problem for him, at least. Hallelujah!
The Actual Fix
I was pissed off at the fact I wasn't able to narrow down the problem to one file. So after a very good 14 hour sleep (after being awake for over 30 hours), I set about to confirm the issue. I started by examining the files and comparing them to the files on my PowerPC Mac. The TokenCache folder didn't exist on the ICBM and the SystemKey didn't exist on the PowerPC Mac. So I could rule those two out. I examined the SystemEntrophyCache file and its contents were boring. So I was able to rule that out. I then opened the CodeEquivalenceDatabase in TextMate (an awesomely awesome overnight text editor) and the text strings matched the applications that currently have Keychain access. So I put the "moved aside" CodeEquivalenceDatabase file back to its original location and rebooted. I then opened Transmit (an awesomely awesome FTP application) which I had just updated and the problem reoccurred. YAY! We (the royal We) had just narrowed down the exact corrupted file that was creating the problem.
In order to fix this problem if you are having it, just open the Terminal (/Applications/Utilities/Terminal) and type:
sudo mv /var/db/CodeEquivalenceDatabase /var/db/CodeEquivalenceDatabase.old
or
open /var/db (and then manually move CodeEquivalenceDatabase to the trash, if you can).
Upon rebooting, God should be in His Heaven and all should be well with the keychain.
Note 1: If CodeEquivalenceDatabase is corrupted, then updating Mac OS X will also cause securityd to eat gobs of memory near the very end of the update cycle when the installer updates Apple applications. This may make it seem like the update stalled.
Note 2: The /var/db folder is interesting. It's not required to boot Mac OS X and all the files in there are created as needed by Mac OS X. It's also explicitly ignored by repairing permissions. However, it does hold important account information, so I would not delete the folder or the items inside unless you are trying to revert your Mac to "factory fresh" and do not wish to have any of the same accounts available without recreating them.
With Love,
The Malicious Retard.
So this year, Rosyna and I are doing the Macworld thing. We're not exhibiting this time around, but we'll be wandering the show floor, soaking up the sites.
If you're there, keep an eye open for us and say hello! Rosyna's the one who looks like a teletubby!
Update: Ok, it's not even related to Apple at all. The recent one is targeting the open source VLC player. Odd, this "exploit" doesn't even crash VLC for me. It just says, "failed to open socket". Other people say they can crash VLC (and just crash). But nothing happens at all to me with VLC 0.8.6. Nonetheless, it's clear from this that these people are just trolling. 1. VLC isn't made by Apple and has nothing to do with Apple other than it runs on Mac OS X. 2. The same problem exists in the Windows version. 3. The default MP3 URL Playlist (m3u) handler is iTunes, which does not have an issue with the generated m3u.
Even more so, VLC is open source. If he truly, truly wanted stuff to be more secure, he could just fix it himself. But he's trying to troll for attention. So he won't fix it, he'll just advertise it.
Remember, just because something crashes, does not mean it is exploitable. For example, the DMG kernel panic isn't exploitable. The system is having its state corrupted, recognizing it, and halting. So it just amounts to a denial of service attack. By that, I mean denying you from downloading porn on the internets until they reboot.
Only one of the 23 people I've talked to actually got this QuickTime streaming exploit to speak anything. It seems to be pretty machine specific. Everyone can get it to crash QT Player, however. In order to get it to work (by work, I mean to run the exploit), it has to do two things. 1. Make the code executable. 2. Get the code to actually run. Number one is usually quite difficult and it's what makes this example a crapshoot.
So anywho, the awesomely awesome Landon Fuller saw this problem and decide to write an APE module to fix it. Many kudos to him. See, this is the exact kind of legit use I like to see of Application Enhancer. Namely, fixing bugs in third-party, closed-source software. Just like the "fixes" released here such as iSWAD (obsolete, Apple fixed this bug in iSync 2.2 released with Mac OS X 10.4.6), SPWW (still relevant for Preview.app), and FontCard (which fixes a bug in Font Book that causes it to be unusable if specific fonts are installed). And lets not forget PD Tweaker 1.0 from the lovely Drew Thaler.
Of course, the source code to Landon Fuller's fix is available. He also says he'll update it as more attention grabbing crashers are released.
If you compile it yourself, please note that it has to be fat in order to load on the ICBMs. APE will not load single architecture plugins on the ICBMs. This is by design. Also, you'll need to install APE in order to use Landon Fuller's fix. You'll need the APE SDK if you want to compile it yourself. His code is pretty dang easy to read and very short. Which is just teh awesome.
Important Note: QuickTime Link (qtl) files will not load in the browser and always open the QuickTime Link file type handler. This is by design. The QuickTime plugin itself never sees it. Thanks to a very wise person that pointed me to this. Sadly, I can't remember who it was. I know so many very wise people.
Read about and Download the Fix by the talented Landon Fuller.
Look's like Landon Fuller has updated it to include the VLC fix. This update is a bit different in that it waits for the components with the bugs to be loaded instead of loading them before hand.
Rather than updating this post every day, I highly, hight suggest you subscribe to his RSS feed. That way you'll know exactly when he updates the APE Module. It's also really neat that he's using the same APE Module to handle all the bugs.