During the planning stage of software development, one of the most important activities is understanding what problem you are trying to solve. While this seems simple, it’s more than just hearing a client or user tell us “I have problem X”. In many cases, it’s up to us to dig deeper and understand a problem from many angles. More than that, it’s often necessary to ask Is this really the problem? Or are there underlying root causes of the problem that need to be addressed? If we don’t do this, we risk solving the wrong problem or developing a solution that won’t work. For example, consider the following:

Now consider the following questions:

• What is the problem that the cities were trying to solve?
• What are the roots of that problem?
• Did the solution solve the problem?
• What problem did the solution actually solve?

Clearly, this particular solution didn’t address the actual problem of homelessness; at least not in a way that truly eliminates it. Instead, the solution was to shift the problem from one that can be seen, to one that can’t be seen. By not “feeding the bears”, the solution was designed to force the homeless to move to another city or jurisdiction where they might be able to find food. The problem of homelessness was just moved from one spot to another.

Of course, this begs the question – how do we identify and solve the real problem?

Asking Good Questions

Identifying the real problem requires us to ask a lot of good questions that will allow us to immerse ourselves in the challenge, and to understand the problem in as much detail as possible.

Further, it requires us to take our time. Complex problems – especially broad social challenges – are interdisciplinary and multifaceted; so it’s more than likely that you won’t identify the real problem immediately. You’ll need to ask a lot of questions, do a lot of research, and critically assess all of your assumptions. You’ll also need to consider which biases might be getting in the way of your understanding the problem as completely as possible.

But what is a good question?

A good question is one that allows us to gather the information we need as efficiently and accurately as possible, while not confusing the person providing the answer. Good questions are free of our own personal biases, judgements, and beliefs. Good questions are also free of language that might lead the respondent to provide an answer they think we might want to hear.

This all might sound trivial and simple, but developing good questions takes time, care, and a lot of thought. It also involves considering the type of question we are asking, when we are asking it, and how we are asking it.

In general, questions can be considered to be open or closed. They provide different types of data, and are useful in different situations, depending on what you are trying to achieve.

Open Questions

An open question is one that allows the respondent to answer in any way they feel necessary. For example, Can you tell me about your childhood? is an open question. A respondent could give you a very short answer, such as “it was great”, to a very long and winding story that describes seminal moments in their life.

Before continuing, consider:

• Why might you want to ask a respondent – such as our client – an open question?
• When might you want to ask the client this type of question?

Open questions allow us to gather a lot of information (often exploratory) about a topic with which we aren’t necessarily familiar, while simultaneously demonstrating our respect for the respondent. In many cases, the information we collect is full of surprises and bits of details that lead us to things we didn’t know and might never have thought to ask about. This is particularly true in situations where we are asking open questions about a topic that is outside our experiences or disciplinary training.

Data that are collected with open questions are qualitative in nature and may be more heavily weighted based on who is providing the information. Data collected from experts in a particular domain will likely carry more weight than data from someone outside of that domain.

Analyzing data that are collected from open questions is more difficult than data collected in other ways. Being qualitative, the data are noisy and subject to the respondent’s current emotional, physical, or mental state, as well as their biases, experiences, and personal contexts. It’s also much more difficult to identify trends in these types of data than in responses to a simple survey.

Closed Questions

A closed question is typically written in such a way as to limit the potential responses that a client might provide. For example, the question Do you currently live in Guelph? likely will have only a yes or no response (although I’m not sure/I don’t know might also work). That’s not to say that a person who is asked this question won’t opt to provide you with more details, however, the question itself isn’t designed to invite more. Closed questions are typically what you see on multiple-choice exams, or on surveys.

Before continuing, consider:

• Why might you want to ask a respondent – such as our client – a closed question?
• When might you want to ask the client this type of question?

Closed questions allow us to collect information that is easily quantifiable or easily placed in bins. These data can be used to test for statistical significance, create groups/categories, or identify trends. However, they don’t allow us to necessarily capture the variation or nuance because they are asked with a specific set of responses provided to the respondent (such as is seen in a survey). This, of course, implies that we have some sort of a priori understanding or knowledge of whatever mechanism is at play. Finally, closed questions don’t necessarily allow us to capture the why behind a response.

Both types of questions are extremely useful, but each has its time and place. In an exploratory setting, it’s probably better to focus your energy on open questions so that you can learn as much as you can about a topic with which you might not be familiar (such as our community partner’s challenge). Later, when you are in the midst of development, you’ll likely shift toward closed questions. That’s not to say that you won’t continue to use open questions later in development, nor is it to say that you won’t use closed questions when you first chat with the community partner, but the proportion of each will likely change as the project proceeds.

Things To Avoid

Once you’ve decided on the type of question you want to use to better understand a problem (or to verify your understanding of a problem), you’ll also want to consider the following:

Other Things To Consider

When you are collecting information, it’s important that you provide the respondent with some preliminary information. This might include a description of why are you asking them questions, how their data answers/data might be used, how long their information will be stored, how it will be stored, who will have access to it, and more. This lets your respondent know that their responses are important and valued, and helps to also provide them with a sense of trust in you and your ability to use their data wisely and appropriately.

After you’ve asked each question, give the respondent some time to think about it, and to tell you their answer (especially for open questions). If you sense that they might not understand what you’ve asked, don’t rephrase the question immediately. Instead, ask if they need any clarification about what was asked.

If you don’t understand something that a respondent has told you (or maybe you missed something while taking notes), feel free to ask for clarification – but let them finish their initial thoughts first. This is particularly helpful whenever our community partner uses jargon that is common to their domain of expertise.

Most importantly, and especially for open questions, actively listen to what the respondent is saying. There will be times when you might wonder why the respondent is telling you a long and winding story that, in your mind might not even seem relevant to
the question that was asked. However, allowing them to tell their story builds trust, and many times little bits of information that are revealed in these stories become extremely useful later on during development.

For example, while listening to responses from one of our community partners, we learned that many of the users of the tool we were developing for Community Living Cambridge enjoyed video games. While this wasn’t within the scope of the initial challenge, many students in CIS3750 opted to integrate that knowledge into their solution.

Take care of yourself as well. Since our community partners are often working with people who are marginalized, traumatized, abused, or misunderstood by society or its systems and structures, you may hear or see things that are shocking or painful, or that negatively affect your mental health. Should you feel uncomfortable, please reach out to the teaching team, or to the resources listed in the course outline. The work we will do in this class is important, but your health is more important.

Last but not least, be sure to thank your respondents for offering you their experience, knowledge, and time.

Activity 1: What’s Wrong With These Questions?

Review Things To Avoid, then identify any issues you see with each of the following questions. How would you rewrite each of these questions so that they are good questions?

• On a scale of 1 to 4, where 1 is “satisfied” and 4 is “extremely satisfied”, please rank your response to the question “How satisfied were you with your service today?”
• How bad is the Federal Carbon Tax, and how do you think it will impact federal employees at the CRA?
• Should concerned parents vaccinate their children?
• How satisfied are you with CIS3750 and your overall experience in the School of Computer Science?
• How amazing is our new and improved double-decker chocolate raspberry-filled donut?
• When you eat donuts, do you prefer them warmed up?
• Are there any perceived benefits to expanding STEM to the more inclusive STEAM?
• Mitacs provides federal and provincial matching dollars to researchers who work with SMEs and other industry partners to offer 4 to 8-month-long internships to graduate students. Do you think taxpayers should be burdened with the cost of this program?

Review Questions