Kelly Browning

About Kelly Browning

Kelly is Matrix Group's Director of Strategy — constructing critical links between new clients and their site users. She enjoys learning about new clients and helping them formulate strategies for most effectively connecting with their users on the web. Before coming to Matrix Group, she was a project manager for Threespot Media. Outside of work, she likes to paint and attend antique auctions.

Comparative Usability Testing DIY Style

When there’s more than one viable design option to consider, comparative usability computer screenshottesting can help you evaluate competing alternatives. This technique is especially useful at the early stages of a design project, because it allows you to explore options, rather than getting locked down into a single approach prematurely.

Everything is multiplied with comparative usability testing: The effort to design and create prototypes, the time to recruit and schedule participants, the work to facilitate the tests, and the amount of information to interpret and communicate. Some big user experience firms can do comparative usability testing with tons of screens, but most of us don’t have the resources to carry out the process at that level.

Fortunately, comparative usability testing can be effective even when done DIY-style.  Applied on an appropriate scale, it can be the perfect technique to help you nail a few key features.

Keep it Simple

When working with limited resources, the key to success is to limit the scope of your design and testing.

  • One feature or screen at a time. If this is your first time doing this, consider testing just one feature or screen at a time so you don’t get overwhelmed. After a while, you’ll get a sense of how much your team can handle.
  • Something with a big impact. You might have time to do this only once or twice in the early stage of your project, so make it count. It only makes sense to test an important feature that will make a significant difference to your users.
  • Don’t muddy the waters. For instance, if you’re testing three shopping cart layouts, keep the product details the same so you can focus on what you’re testing: The layout.
  • Just a few variations. Again, to keep things simple, you’ll probably want to test just a few design variations. I’d suggest 3 at most for practical DIY-style comparative usability testing.

Be Consistent

You’ll want to design scenarios and tasks as you normally would in a usability test, with this important guideline in mind:

  • Use the same tasks for every design variation. The tasks for each design option should be identical or as close as possible. If “enter your birthday” causes problems in design A, you will need to know if it’s also problematic in design B.

Avoid Bias

Interpretation is a bit more complicated with comparative usability testing, and it becomes that much more critical to keep your test results pure, so to speak. To this end:

  • Mix the order of the design variations. If design A is always first, couldn’t that affect how people tend to react to design B? Switch the order to level the playing field.

And the following best practices deserve special emphasis:

  • Be objective or appear that way. Do you have a horse in this race? If so, avoid comments or non-verbal cues that would influence the participant. Better yet, recuse yourself and have somebody else facilitate the tests.
  • Have another person observe the tests. This is a best practice for all usability testing but especially when you are comparing multiple alternatives. Those competing options may have some egos or politics around them, so it’s important to build objectivity into your process by having at least one other person observe the test.

Have Realistic Expectations

After a few tests, one of two things can happen:

  • A clear winner emerges. This is less likely the more complicated the features and the testing. It’s not a realistic goal for most tests.
  • There’s no  clear winner. This is the most likely outcome. You may see strengths and weaknesses in multiple design variations. This is an opportunity to think critically about what happened, why, and consider next steps:

Remember:  It’s Worth the Effort

Comparative usability testing can help you discover the advantages of multiple design variations. It can be invaluable at the earlier stages of your project, a way to ensure that you are pursuing the strongest design possible, rather than wasting time “perfecting” something that will never be as good as a superior alternative. Even on a shoestring budget, comparing multiple options up front can help you perfect the features that can make or break your product.

As of now, there isn’t much out there about DIY comparative usability testing (also called “competitive” usability testing). However, there are a few more formal research studies and other resources out there that are relevant and may be interesting to you. 


What kinds of DIY methods have you used/discovered when on a limited design budget?

User Experience: Megamenus

Megamenus are big menus.

I mean, as far as details go, they’re a bit more complicated than that.  But the essence of a megamenu is that it’s a Great. Big. Layered. Menu.

The more precise definition of megamenus is this: Megamenus are panels of links revealed by a user interaction. These panels are bigger, wider, and hold much more information than the menus you’re used to seeing.

Why Use Them?

If you have a content-heavy website, chances are that you have many different categories of information that users will need to be aware of and choose from.  One layer of information may not cut it because there are just too many options of equal importance that a user needs to get to easily.

Megamenus, if designed and programmed properly, can make it easier for your users to make those decisions, and give them a fast-forward path to their destination. Users don’t have the time to sift through your site, discover all of the options, and then carefully choose among them. Giving users an accurate and thorough set of links to evaluate means they don’t have to explore your whole site to find what they want; it also prevents them from having to pull up a model of their options from within their own working memory. I promise that they won’t forget that there is a section called “Scientific Research” because it’s directly in front of them.

Note the words easier and easily. Choosing among many options requires mental effort of users.  They’ll leave your site if you make things hard for them, so make it easy for them. Basic UX. Learn it; love it.

What It Means for You, the Designer

Again, ease of use is going to be what drives you, here. Megamenus provide a design space where multiple choices can be visually grouped. This grouping helps users make sense of the options available to them.

A big, unlabeled, unordered pile of links will just confuse and frustrate your users. 

Vertical menus don’t offer the layout flexibility. When vertical menus contain a lot of items, they can begin to feel overwhelming for many users.

While designing and implementing megamenus, you can include icons, imagery, and even user controls like checkboxes and search fields.   A warning, though: Triggering the menu from a user’s click in your megamenu is preferable to a hover. Activating menus by hover can cause a sense of instability that adds to user stress while attempting to interact with inputs / controls. Have you ever hovered over a menu item and tried to hover over to the sub-menu that pops out of it, only to have everything disappear? Horrible!

Use this opportunity to find the balance between information and infodump.  Make it interesting rather than overwhelming.

An example of a mega menu that uses click, not hover, to reveal is (see the ‘Programs’ and ‘Listen’ items.)

Articles for Further Reading

Mega Menus Work Well (Jakob Nielsen)

Designing Dropdown Menus: Examples and Best Practices (Smashing Magazine)


6 Epic Forces Battling Your Mega Menus (Jared Spool)