Apple’s security reality is changing right before our eyes and the company’s response will be telling. The toughest challenge will be shutting down hackers while keeping its trademark usability in tact.

Steve Jobs & Co. is known for creating devices that can spur gadget lust with just a mere rumor. Apple customers for years have taken the view—inspired by the company’s commercials—that its software is safer. If you have a Mac there’s no need for anti-virus software. You’re secure.

The reality is Apple enjoyed security by obscurity. Its market share wasn’t worth the attention from hackers. Now Apple is worth the attention. Where’s the glory in taking out a smaller computing player when you can take out the big dog—Microsoft?

As a result of Apple’s lack of hacker interest, the company could talk about being more secure even as it tended to rewrite QuickTime and plug security holes every time it launched a new product or generated buzz. While you were playing with your latest greatest Apple software release the company would patch vulnerabilities.

Here’s Apple’s chain of events over the last month:

• Mac Defender malware attacks Apple users.
• Apple remains mostly silent and tries to thread the customer service needle.
• Apple then announces a fix and that a future update will put Mac Defender to bed with an update.
• Evil doers launch a new renamed version just a few hours later. The new malware is renamed (predictable) and split into two parts, a downloader that delivers a payload similar to Mac Defender (not so predictable).

Does any of this sound familiar? It should. Microsoft went through this same learning process with its security procedures. Microsoft had to button down its security operations and today is able to fend off a lot of attacks.

Ed Bott nailed the importance of these malware attacks against Apple when he said:

Apple appears to be treating this outbreak as if it were a single incident that won’t be repeated. They seriously underestimate the bad guys, who are not idiots. Peter James, an Intego spokeperson, told me his company’s analysts were “impressed by the quality of the original version.” The quick response to Apple’s move suggests they are capable of churning out new releases at Internet speeds, adapting their software and their tactics as their target—Apple—tries to put up new roadblocks.

If Apple plans to play Whack-a-Mole with these guys, they’re in for months of misery. Just ask any Windows security expert who was around in 2003 and 2004 when Microsoft was learning a similar painful lesson. If each reaction from Apple takes two or three weeks, the bad guys will make a small fortune and Mac users can count on significant pain and anguish.

Microsoft eventually got security religion, but there was a cost—usability. Vista’s most hated feature was UAC (user account control). Bott later noted that UAC was enough to drive any level-headed person to PC rage.

In a nutshell, Microsoft added a key security feature—and drove its users nuts. Apple naturally capitalized on Microsoft’s UAC flub.

This usability vs. security line is one Microsoft has been walking for years. If you use all three of the top Web browsers regularly—IE 9, Google Chrome and Mozilla’s Firefox—you notice pretty quickly that IE 9 has more prompts and security features that can be annoying. I don’t doubt that IE 9 is the most secure browser around, but there are times I feel like I’m taking medicine that has a nasty taste to it.

It’s not like Apple hasn’t paid any attention to security. The biggest issue is that Apple seems to be underestimating what it is up against. Apple is just supposed to work. Security sometimes requires some inconvenience to users. If you build security in from the ground up, usability can suffer.

Apple’s trade-off will between security and UI will be its biggest challenge in the years ahead. If I were to guess, Apple’s Mac malware issues are just the warm-up act for bigger things.

• Why not target Apple’s iOS, which is a dominant mobile OS?
• Why not target iTunes and all of those credit card accounts on file?
• Why not go for the glory of bringing Apple down?

In other words, Apple may have to spend some time talking security frameworks. That’s quite a sea change. If Apple can integrate hardware, software and more security into a package where the consumer doesn’t notice then it will have pulled off a great feat.

Final thought: One natural reaction to talking Apple security is to bring up Google’s Android. Android will be just as big of a hacker target and Google will have to respond to the same challenges as Apple. Ironically, Microsoft’s Windows Phone 7 will have a free pass for a while. Why? Security by obscurity. Microsoft in mobile just isn’t big enough to matter.

Related:

• Mac malware authors release a new, more dangerous version
• Apple Mac OS X update to put Mac Defender malware issue to bed
• Apple continues to tell support reps: do not help with Mac malware
• You have malware on your Mac and you call Apple support … what happens next?
• Apple to support reps: “Do not attempt to remove malware”
• Crying wolf? Apple support forums confirm malware explosion
• An AppleCare support rep talks: Mac malware is “getting worse”
• What a Mac malware attack looks like
• Why malware for Macs is on its way
• Coming soon to a Mac near you: serious malware