This post is based on the article Design-based research: what we learn when we engage in design of interactive systems, interactions 18, 5 (September 2011), 56-59.

Margaret Hamilton during her time as lead Apollo flight software designer. Wikimedia Commons

Introduction

More than 20 years ago, Fred Brooks asked, “Is interface design itself an area of research, producing generalizable results?” [1]. He elaborated that a major issue that puzzles the human-computer interaction community is the tension between narrow truths proved convincingly by statistically sound experiments, and broad truths, generally applicable, but supported only by possibly unrepresentative observations—that is, results indisputably true but disputably applicable, and results indisputably applicable but perhaps overly generalized.

Brooks’ question is still relevant. In this chapter, I explore the view that the design of complex and novel computing systems can itself be an area of research, complementing other forms of research, and that it is capable of producing useful and trustworthy results. I call this form of research design-based research, a method of inquiry aimed at exploiting the opportunities that designing complex computing systems provides to advance our understanding of the problem we are solving, the process we are following, and the solution we are building. While many designers and researchers already conduct this form of research, and the idea of design-based research is not new (for example, [2]), there have not been many attempts to explicitly define this method and address the following questions:

  • What can we learn when we engage in the design of computing systems?
  • What kind of generalizable knowledge can we get from design?
  • What is the relationship of design-based research to theoretical and experimental methods?
  • Why can design reveal things that other research methods cannot?
  • What makes design-based research trustworthy?

What We Learn When We Engage in Design

Design can be described as a sequence of decisions made to balance design goals and constraints. In any design activity, designers make a number of decisions, trying to answer the following questions [3]:

  • How will the design process advance?
  • What needs and opportunities will the design address?
  • What form will the resulting product take?

These decisions must be made in every design effort, although they may not be explicit, conscious, or formally represented. In routine design, these decisions are straightforward, requiring little learning by designers. In challenging or innovative designs, however, these decisions can be complex and interdependent, requiring extensive investigation, experimentation, and iterative improvement. In such situations, designers may acquire important new understandings. This ability to acquire new knowledge through design provides the basis for doing research, which aims at capturing this new knowledge and making it available to a broader audience. We may group lessons that we learn in design into three categories: design procedures, problem analysis, and design solutions.

  • A design procedure specifies which processes and individuals are involved in a design. Designers often have to develop specialized procedures to respond to a specific design challenge or the context in which the design is being constructed.

  • Problem analysis describes our current understanding of the problem we are facing. One of the characteristics of design is that we never start with a clear understanding of the problem, and one of the chief services of a designer is helping clients to discover what they want designed [4]. Design problems are often full of uncertainties about both the objectives and their priorities, which are likely to change as the solution implications begin to emerge. Problem understanding evolves in parallel with the problem solution, and many components of the design problem cannot be expected to emerge until some attempt has been made at generating solutions [5]. Simon, in what he calls “designing without final goals,” wrote that a goal of design may actually be understanding the problem and generating new goals, elaborating that the idea of final goals and a static problem definition is inconsistent with our limited ability to foretell or determine the future [6].

  • A design solution describes the resulting product, the outcome of designers’ efforts to address challenges, satisfy constraints, exploit opportunities, and balance the trade-offs identified in the problem analysis. The design solution evolves over the design process as designers deepen their understanding about the design context and problem.

Generalizable Knowledge

To be regarded as a research contribution, design activity should go beyond simply refining practice and also address theoretical questions and issues. Design-based research extends ordinary design activity with a goal of developing generalizable knowledge. In a normal design effort, the primary goal is to create a successful product, and lessons learned are restricted to the particular design and the people involved in it. In the process of generalizing, however, a designer-researcher expands his focus beyond the current design situation, viewing the design problem, solutions, and procedures as instances of more general classes. For each of the collections of lessons learned, we may identify the corresponding type of generalization: domain theories, design frameworks, and design methodologies [3].

  • A domain theory is the generalization of a problem analysis. A domain theory might be about users of computing systems and how they learn to use and interact with the systems, or about the context of the system usage and how it influences the user and interaction. A domain theory is a means of understanding the world, not the design solution or procedure.

  • A design framework is a generalization of the design solution. Design frameworks describe the characteristics that a design solution should have to achieve a particular set of goals in a particular context. In other words, a design framework represents a collection of coherent design guidelines for a particular class of design. Design patterns and software architectures are prominent examples of this class of generalization.

  • A design methodology is a generalization of a design procedure. In contrast to design frameworks, a design methodology provides guidelines for the design process rather than the product. In general, a design methodology describes a process for producing a class of design solutions, the types of expertise required, and the roles of people with these types of expertise.

In general, design-based research cannot develop “grand” or universal theories and frameworks. Rather, it develops generalizable knowledge with an intermediate theoretical scope, covering a gap between a narrow explanation of a specific design and a broad, more generic account that does not limit the design to a particular situation.

Design-Based Research and Other Research Methods

Generalized knowledge can also be derived by using other empirical or theoretical research methods. Design-based research, however, can produce knowledge that normally could not be generated by isolated analysis or traditional empirical approaches, and therefore complements existing empirical and theoretical research methods. Design-based research facilitates disciplined, systematic inquiry into a real-world context while simultaneously doing justice to its complexity. It is conducted in messy, but entirely realistic, situations and while it produces claims with less certainty and replicability than other research methods, it can extend our area of inquiry beyond the scope of these methods.

Design-based research facilitates disciplined, systematic inquiry into a real-world context while simultaneously doing justice to its complexity.

Design-Based Research Versus Other Empirical Research Methods

Controlled experiments are one of the most powerful and conclusive forms of empirical research, used to establish the relationship between the cause and the effect by manipulating an independent variable to see how it affects a dependent variable. Although conducting experimental research has enormous benefits, it also has some serious limitations. The controlled experiment may be conducted only if we know the relevant variables involved in research, we can define important relationships among the variables, and we can control all extraneous variables that might affect the outcome. These conditions significantly limit the scope of experimental research, and in many real-world situations we cannot fulfill them, as a researcher usually cannot maintain control over all factors that may influence the result of an experiment. Attempting to simplify a real-world situation so it can be subjected to experimental research often leads to studying unrealistically simple situations. Attempting to establish an experimental control in a real-world setting, on the other hand, may lead to negative phenomena, such as the Hawthorne effect, in which those who perceive themselves as members of the experimental, or otherwise favored, group tend to outperform their controls, often regardless of the intervention.

Design-based research can produce knowledge that normally could not be generated by theoretical analysis or traditional empirical approaches.

Although significantly different, controlled experiments and design-based research are compatible forms of research that can be and often are used together. Controlled experiments, for example, can guide design decisions and test particular elements of a design on a smaller scale and in more controlled conditions. A decision about which input control to use in a user interface, for instance, may be based on the results of a controlled experiment comparing the efficiency of users’ data input with several alternatives. Controlled experiments may provide reliable information that something “worked,” but they often do not provide sufficient information about exactly what it was that worked, or why or how it worked. Design-based research can help us to characterize and identify relevant variables, create an explanatory framework for the results of the experiments, and provide us with more insights about why and how some elements of a design work.

Ethnographic research and field studies attempt to characterize relationships and events that occur in some setting to produce rich descriptions that make it possible to understand what is happening and why. In contrast to design-based research, however, there is no attempt to change this situation. Design-based research complements these methods by enabling us to learn more about the real world by changing it and reflecting on our experiences in understanding problems, design solutions, and procedures.

Design-Based Research Versus Theoretical Research

Design-based research requires an alternative view on the relationship between theory and practice in which neither is taken as primary. Design activity is often driven by existing theories, and at the same time it can provide a constructive environment for theory development. Design process can often reveal theoretical inconsistencies more effectively than analytical processes, while designing a concrete system based on some theory requires that it be fully specified [4]. On the other hand, the development of theoretical constructs and standards without their grounding in a concrete design often leads to a range of problems, as shown in Henning’s discussion about the reasons for the decline of CORBA (Common Object Request Broker Architecture) [7]. Henning concluded that standards consortia must ensure they standardize only existing best practices and that no standard should be approved without a reference implementation and without having been used to implement a few projects of realistic complexity.

Design-based research requires an alternative view on the relationship between theory and practice in which neither is taken as primary.

Why Design Can Reveal Things That Other Methods Cannot

Design-based research complements existing research methods in its ability to employ in a greater amount the tacit, implicit, intuitive knowledge and skills of both designers and users. Schön calls such knowledge knowing-in-action, revealed only in the way in which we carry out tasks and approach problems: “The knowing is in the action. It is revealed by the skillful execution of the performance—we are characteristically unable to make it verbally explicit” [8].

Design-based research complements existing research methods in its ability to employ in a greater amount the tacit, implicit, intuitive knowledge and skills of both designers and users.

In other words, though we cannot explain such knowledge and skills, we can demonstrate them by being engaged in a particular activity. This observation is supported by studies of embodied cognition, which emphasizes the formative role that the environment plays in the development of cognitive processes [9].

A design activity can set in motion our intuitive and tacit knowledge accumulated through years of research and experience. Much of such valuable knowledge is not captured in existing theories and guidelines. Often, we are not aware that we possess it. Glass, for example, noted that actions of designers are often implicit and intuitive, defining intuition as “a function of our mind that allows it to access a rich fund of historically gleaned information we are not necessarily aware we possess, by a method we do not understand” [10].

Glass further elaborated that our unawareness of such knowledge does not mean we cannot use it. Designers, for example, often cannot explain their own creative processes, but, through design, they can apply and materialize these creative skills in solving a range of complex problems.

Our intuition and tacit skills also play an important role in understanding and setting problems from messy and ill-defined situations. By engaging in design, we can better understand real-world, ill-defined, and wicked problems (as discussed earlier). Similarly, through design we can better understand users’ needs, as our users often cannot precisely explain to us what they want unless we present them with some version of a design solution [4]. Moreover, by engaging users in design, we may employ their knowledge about their domains, as well as their creativity.

While design itself adds discipline and professional attitude to tacit, implicit, and intuitive knowledge and skills, design-based research may be viewed as an attempt to increase awareness of such knowledge and to support, capture, generalize, and share this knowledge beyond the design community. Therefore, design-based research can be an especially valuable method of inquiry in domains such as interaction design, which does not have strong theories, models, and laws to conduct extensive theoretical analyses, simulations, or experiments, but does have practitioners and users who have some (often tacit, implicit, intuitive) knowledge and skills related to the domain.

Trustworthiness of Design-Based Research

While design-based research puts trust in designers’ skills, ingenuity, and ability to correctly observe and generalize issues observed in a design process, this trust should not be blind. Results of research must be presented in a way that enables readers to clearly understand the motivation and reasoning behind particular claims. This means that designers must provide sufficient information so that generalized claims can be verified. Trustworthiness of design-based research comes from making the reasoning behind generalized claims explicit, public, and open to critical reflection and discussion.

Trustworthiness of design-based research comes from making the reasoning behind generalized claims explicit, public, and open to critical reflection and discussion.

Conclusion

The study of computing systems requires the selection of appropriate methods from a wide array for each research question asked. For many of these questions, theoretical analyses, controlled experiments, or ethnographical research are the best methods. However, design-based research can produce knowledge that normally could not be generated by theoretical analysis or traditional empirical approaches. It can help us to better understand the problem and ask better research questions, often having a pioneering role in settling a new research territory that can then be “occupied” by other research methods.

Design-based research can produce knowledge that normally could not be generated by theoretical analysis or traditional empirical approaches.

It can help us to better understand the problem and ask better research questions, often having a pioneering role in settling a new research territory that can then be “occupied” by other research methods.

If lessons learned in design are to become accepted as serious scholarly endeavors within and outside our discipline, we need to take responsibility for creating standards that make such research recognizable and accessible to other researchers. In particular, more work is necessary to create a framework that can enable us to combine results of design-based research with results from other forms of research. And we as a community need to better understand the scope and limitations of design-based research to be able to critically review contributions of this kind.

{pagebreak}

References

  1. Brooks, F.P. Grasping reality through illusion-interactive graphics serving science. Proc. CHI ‘88. ACM, New York, 1988, 1–11.
  2. Zimmerman, J., Forlizzi, J., and Evenson, S. Research through design as a method for interaction design research in HCI. Proc. CHI ‘07. ACM, New York, 2007, 493–502.
  3. Edelson, D.C. Design research: What we learn when we engage in design. Journal of the Learning Sciences 11, 1 (2002), 105–121.
  4. Brooks, F.P. The Design of Design: Essays from a Computer Scientist. Addison-Wesley Professional Boston, MA, 2010.
  5. Lawson, B. How Designers Think: The Design Process Demystified, 4th Edition. Architectural press, Oxford, UK, 2005.
  6. Simon, H.A. The Sciences of the Artificial, 3rd Edition. MIT Press, Cambridge, MA,1996.
  7. Henning, M. The rise and fall of CORBA. Comm. of the ACM 51, 8 (Aug. 2008), 52–57
  8. Schön, D. The Reflective Practitioner. Basic Books, New York, 1983.
  9. Dourish, P. Where the Action Is: The Foundations of Embodied Interaction. MIT press, Cambridge, MA, 2001.
  10. Glass, R.L. Software Creativity 2.0, developer.* Books, Atlanta, GA, 2006.