March 1st, 2009
Posted by Ed Bott
Last week my colleague Mary Jo Foley reported on rumblings of discontent from the invitation-only Windows 7 technical beta test community:
A number of Windows 7 testers have complained recently that Microsoft was not sharing enough information about changes it planned to make in response to their feedback.
Windows SuperSite’s Paul Thurrott questioned in a post yesterday whether Microsoft had already locked down Windows 7’s feature set before the majority of technical and public beta testers ever got to see a first release of the product. I’ve wondered the same.
This was all in response to another epic post on the Engineering Windows 7 blog by Steven Sinofsky, who tried to explain how the feedback process worked. The whole thing is worth reading, although at 4700+ words I’m afraid most people will just skim it.
Frankly, I’m having a hard time working up any level of sympathy for those doing the complaining, partly because I heartily approve of the way Windows 7 development is going right now and partly because I have seen the feedback process up close and personal. Microsoft is getting a bad rap from a group of people who are mourning the reality that they’re no longer being treated as privileged elites.
I was going to ignore this whole brouhaha, until I read a post on the subject by WinPatrol developer and Microsoft MVP Bill Pytlovany that included this provocative proposition:
Most Beta Testers Suck
As a developer I can tell you , beta tests aren’t what they used to be. The number of people who actually report decent bug information is minimal. Most people download the beta just to be an earlier adopter. Developers are lucky if users read the release notes and compatibility list let alone any beta instructions. There are so many different machine configurations that sadly the only way to find some bugs is to have full global adoption of new software.
Bill isn’t going to endear himself to any beta testers with that line of argument, but he does have a point. I’ve read many of the complaints Mary Jo referred to and a few hundred others on the members-only Windows 7 technical beta newsgroups. I think a lot of beta testers need a refresher course in the basics of what it means to be involved in the development of a product as complex as Windows.
In that spirit, here’s my list of five things every Windows 7 beta tester should know:
1. Things have changed. According to Thurrott, “The real problem here is that the feature set of Windows 7 was frozen well before the Beta release. So the feedback [Sinofsky] discusses throughout this post is 99 percent bug testing, really (and 1 percent, we hear your concerns but have a million reasons why we can’t change a thing).”
And this is a problem? I don’t think it’s any accident that the two most troublesome releases in the history of Windows also had the longest beta cycles. I was running pre-beta builds of Windows 95 in 1993, nearly two years before it was released. The first alpha releases of Longhorn, which eventually became Windows Vista, were handed out at the PDC in late 2003, nearly three years before Vista shipped.
By the time a beta is released, the feature set should be pretty much frozen. That’s how you concentrate on things like quality and performance.
As for the complaint that Microsoft hasn’t listened to feedback and ignored its most loyal customers when developing the feature set for Windows 7, I say, “Give me a break.” Since November 2006, Microsoft has gotten an earful about its Windows design decisions. The feedback loop includes:
-
Every blog post, review, newsgroup posting, and rant about Vista ever published
-
Support calls to Microsoft and its partners
-
Requests from PC makers and software developers
-
Telemetry data (all those crash reports really do go somewhere)
-
Field research and usability testing
-
Interviews with opinion leaders, including Paul and me, who have given feedback to Microsoft in person and on the phone many times in recent years. You think they weren’t taking notes?
That’s a helluva lot of feedback to take into account. There comes a point where more doesn’t mean better.
2. Windows design is a series of compromises. A lot of the complaints I’ve heard boil down to “Well, that’s not how I would have designed that feature.” Right. When you’re building a product that is going to be used by hundreds of millions of people, you have to find some common denominators. And as I wrote last year, sometimes there is no right answer: you can bet that for any decision you make, some nontrivial number of people will think you’re a complete idiot, no matter which option you choose.
I also hear lots of feedback suggesting that Microsoft should never remove a feature and should always give its old-time users a way to preserve the procedures they learned five or 10 or even 20 years ago. Seriously, I’ve heard people argue that Windows 7 should include the old File Manager utility from Windows 3.1. Can you imagine how complicated, even bloated, Windows code would be if no feature was ever cut and you could choose from a dozen or so Classic interfaces going back to 1991? But that’s the logical conclusion from that line of argument.
Things evolve. Old features disappear, and new ones are introduced. Deal with it. If you think there’s a better way to implement a new feature than the one Microsoft chose, blog it. But it helps if you can make a rational case – remember, you’re dealing with engineers. Simply saying “XYZ feature sucks” isn’t likely to win hearts or minds.
3. Writing good bug reports is hard work. I sympathize with testers who complain that their bug report was closed as “Non-repro.” buy viagra cheap But that’s reality. If it was easy to reproduce, the bug would most likely have been caught in one of the many, many automated testing cycles that each Windows build goes through. The really tricky bugs are those that are triggered by unusual combinations of hardware and software under specific conditions.
In fact, if you talk to the developers who dig into those incoming bug reports from technical beta testers, as I’ve done, you’ll quickly learn that it’s a pretty low-yield process. Most are duplicates and the vast majority are just requests for new features or changes to existing ones. When they get closed as “won’t fix” or “by design,” it’s because someone already considered that request and decided for any of a thousand reasons (budget, compatibility, risk of regression, or conflicting data) that the feature is going to remain as it is.
4. One more build does not mean a better product. In fact, you could argue as I did last week that the work of building an “official” beta release slows down progress. Pytlovany, who has worked inside Microsoft, agrees:
Every new beta release is a distraction to developers. The time it takes to create a frozen version takes away from a developers imagination and productivity. […] The internal testing required before any public beta is a lot longer than you might think.
If you’ve identified a bug and it’s made it onto the must-fix list, it shouldn’t take multiple passes to fix it. Microsoft’s Charlie Owen, who works on the Media Center team, had some great advice for beta testers in a blog post last week:
When the Windows 7 Release Candidate becomes available immediately download, install, test deeply and quickly provide actionable feedback.
Seriously: As the release candidate is downloading and with tenderness, kiss your spouse on the cheek and tell him or her you’ll be back in a week or so. Then lock yourself in the home office and be relentless and unforgiving in your testing of the Windows 7 Release Candidate and provide feedback.
5. Shipping is a feature. There is no such thing as perfect software. If the developers of any complex software product like an OS waited for every bug report to be “fixed.” the software would literally never ship.
The one thing Microsoft can do going forward that it has not done well in the past is to incorporate feedback from the current test cycle into the next version. The best way for that to happen is for Microsoft to develop a consistent, professional process for planning and shipping new releases on a predictable schedule. In theory, features that don’t make it into this release have a legitimate shot at making it into Windows 8. As Charlie Owen put it, “You should consider the Windows 7 Release Candidate as your first and best opportunity to influence the next version of Windows.”
