Suzanne Hillman (Outreachy): Usability testing and analysis wrapup

Outreachy ended yesterday, so I’m working on cleaning things up for others to use.

I have completed my summaries of the initial interviews for event creation/planning and ambassadors as resources. I did not manage to translate the CSS from table to div, as things were behaving very oddly when I tried. However, I did pass along the CSS/HTML work I had done to Máirín Duffy.

Mo also has access to all the recordings of my interviews and usability sessions, the survey which ended up with 140 responses, and the MyBalsamiq instance for the Fedora Design team in which I put my mockups. I have put the anonymized transcripts and spreadsheet for the usability testing into the User Research and Analysis ticket shortly.

I hope to use the travel expenses for Outreachy to attend FLOCK in Cape Cod this summer.

Usability Testing

Here, I’ll summarize how the usability testing went overall, what sorts of things I found, and some of the things that Mo and I discussed in our analysis. Due to time restrictions, we did not make it through everything I found for analysis purposes, but I will be available for questions and clarification as needed.

As I said in my last post, I ran remote usability sessions on my prototypes with 5 people. The major time sink for this was creating transcripts, although mostly not word-for-word. As I went, I highlighted the things that seemed relevant so that they would be easier to find when I was summarizing the findings. I ended up using a spreadsheet to organize the findings, first by prototype, and then by content to make reviewing it easier. For more on this, please see the attachments to the research and analysis ticket.

I then met with Mo to discuss my findings and to start preliminary analysis of them. We initially focused on the things that multiple people reported, with some side conversations around related problems when necessary. In many cases, the decision on what to do was pretty simple (often because it was effectively a paper prototype and thus not as interactive as it could be). However, some of the problems my participants ran into were not simple fixes and required a lot of discussion.

One of these related to the problem of deciding who to contact first in a filtered list of people. As it was, the prototype did not show anything about your relationship with those people. Even when you clicked on someone, it was not obvious that the hubs and friends listed started with the ones you had in common:

<figure><figcaption>Do I know Jen Smith better than John Holsberg? Who knows?</figcaption></figure>

When I asked people to find someone to contact, whether because they were visiting an area and wanted to find local Fedorans to meet up with, or because they needed more information about an upcoming event, they had trouble deciding who to talk to. In this prototype, only one person was clickable (Jorge), but my participants had no idea if that was who they actually wanted to talk to.

Affinity Mapping?

Some suggestions people made included having something signalling if they have an existing relationship with the people in the list. In talking with Mo, however, that can get very complicated, very quickly.

We can decide that people who are following each other are friends. However, what if you think someone is fabulous, but don’t need to know about their activity within Fedora, or don’t have the time to add that to what you are paying attention to. At the moment, following someone means that you see what they are doing in your stream, so perhaps mutual following is insufficient information.

Maybe you’re on a team with someone so you’re theoretically interacting with them regularly. This is more likely to be useful information about how you are connected to that person, except possibly if you’re on a huge, geographically disperse team.

Maybe you talk to someone on IRC regularly. That probably means you know them, right? Maybe work with them? Sure, but that may not mean you want to meet up with them.

It absolutely would be helpful to Fedora members to know their relationship, if any, with people they are considering contacting. Precisely how to do this remains to be seen.

Supposing we use team affiliation, following, and conversation frequency. How do we signal this kind of information at a glance without making the information too overwhelming and difficult to process? Icons can be good _if_ they are some of the very few that are quickly understood. Text can get unwieldy. This will have to go unanswered for now, but requires more thought.

Online Status

It’s helpful to know if someone is online right now. It’s probably not as helpful to know precisely when someone was last online when you’re looking at a long list of people.

Lots of applications show that someone is online using a simple blue (or green) dot near their picture or name. This may be the best way for us to do the same, and will also free up some space in the search results for affinity signalling. At the same time, we probably want to make it clear who hasn’t logged in recently. Since this will be pulled from a list of people with FAS account, there may be people in the results who have never logged into Hubs, and we need to show that somehow. We can easily show more precise information about when someone was last online in the dropdown, and from the perspective of too much information, that definitely makes the most sense.

Who wants to meet strangers?

Another frequent problem was that my participants didn’t especially want to be contacting random Fedora people out of the blue. Most said that if they knew someone liked meeting strangers, or were interested in helping others out in some way, this would reduce that barrier to contact.

So, instead of having the ability to select only ambassadors — who may or may not actually want to be contacted right now — make it possible to select people who are interested in meeting new people or answering questions or otherwise welcome random contact. Of course, what this needs to be called is an entirely different question. It might be as easy as an ‘Open/Closed’ sign like some businesses have, but that may be too easily misinterpreted or daunting for other reasons. More research is definitely needed here!

What do the search boxes accept?

In looking at the major search and filter areas of the list of people and events, it can be difficult to determine what is valid.

<figure><figcaption>People search and filter</figcaption></figure><figure><figcaption>Events search and filter</figcaption></figure>


When asked to search for people in Berlin, many tried to use the ‘all people’ search to look for Berlin. The actual intent is that they can change ‘you’ to a specific location other than where the system knows they are. The ‘all people’ search box is for searching by people’s names, nicknames, IRC nicks, and email addresses.

Similarly, when asked to find events in Las Angeles, the ‘all events’ search box was very tempting. Again, the ‘you’ box is there to allow you to change the location away from your own location.

So how does one make this clearer?

We had a few thoughts on how to best handle this one. Mo pointed out that a common pattern is that searches are along the top, and filters are along the left side. In that case, why not let people search by location in the search box?

We also considered that this might be a lot clearer if it were possible to actually type into those fields: once you start typing, type-ahead would quickly make clear the sorts of things the search box was expecting.


Another, similar, problem was that the contents of ‘10’ miles and ‘you’ (different for events and people) were somewhat unclear on what they were able to take as inputs. My mockup had dropdowns for these, since we wanted to not only allow people to start typing in those boxes, but also to make clear the kinds of things that were possible in there. However, in no case did people realize that they could simply start typing, rather than only selecting from the dropdown options.

<figure><figcaption>events near [place] or in [place]</figcaption></figure>

You can pick near ‘you’, near a specific location, or within an area. The latter case wouldn’t need the ’10 miles of’ piece, though.

<figure><figcaption>People near [somewhere], including everyone</figcaption></figure>

I wanted to have the option of everywhere because otherwise actually specifying something in ‘all people’ makes no sense. At the same time, perhaps you _do_ care where. This one confused me and I failed to explain my reasoning to Mo.

<figure><figcaption>Events or people within [miles of somewhere] or [a place]</figcaption></figure>

How far from the place you specified? Or, maybe you just want to say within a place?

This is likely an even more complicated problem than I suspected, but hopefully it was largely due to the fidelity of my prototype.

But what do we do?

We did not come to an agreement on the best way to handle either of these cases, in part because we suspected that this was a problem with the fidelity of the prototype.


Some of the problems I ran into were fairly easy to solve. Others need additional investigation and consideration. There are findings from the testing that have not been reviewed, and I hope that Mo will be able to find the time to go through them and contact me as needed.

I have no question that the usability testing was valuable, and hope that the existing team will be able to continue the work that I started. I wish that I were able to continue to work on this project, and shall see if there is any time available while I apply for jobs and hopefully start a full-time job soon.

I found this to be a fabulously useful experience, and want to thank Máirín Duffy and the Fedora Hubs team for being helpful, approachable, and friendly. And of course, to thank Mo again for having mentioned the possibility of Outreachy to me.

Source From:
Original article title: Suzanne Hillman (Outreachy): Usability testing and analysis wrapup.
This full article can be read at: Suzanne Hillman (Outreachy): Usability testing and analysis wrapup.


Random Article You May Like

Leave a Reply

Your email address will not be published. Required fields are marked *