Design

From TheCommandLineWiki

Part of The Inner Chapters Unbook.

Originally part of podcast episode number sixteen.

Dedicated audio available from Podiobooks.

Original Notes

• Design is an aesthetic
  • In the mind of the beholder
  • Technique can be taught, sensibility has to come from within
• Leaving dross on the die
  • Uncertainty, valid ignorance, discovery
  • Morphing the Mold, SD Magazine, August 2003
  • Speak about host trust negotiation for current SSO project
  • Creativity - emphasize Berkun's point
• Process, not goal
  • Design never ends at the specification
  • Sun Tzu - spec never survives first contact with implementation
  • Good design is informed as much by implementation as the reverse
  • Find an example of multiple iterations
• Pure architects are losers
  • At least should implement infrastructure, framework
  • Even then, there should be a close feedback with first user
  • Lapidary effect, use will help reveal rough edges for smoothing
  • Humility
    • No design is ever perfect
    • Evolution, improvement is always a good thing
    • Learn from mistakes
• Over engineering is a response to ignorance
  • Do no harm
  • Otherwise, as XP advises, only design what you need
  • Conversation can help expose over-engineering
• All tools suck
  • Use what works
  • Anything that you cannot justify in terms of concrete gains, today, must be ditched
  • Only limit is do no harm, do nothing that closes off possibilities later

Transcript

From the Podiobooks version of the chapter.

In this installment, I'm going to talk about design. In the previous two installments I talked about respectively functional decomposition and testing. And as far as software practices go, these are a little bit easier to talk about in that while there is some discretion involved in terms of what's appropriate for your particular programming style, what's appropriate for the project that you're working on. There's not a lot of room I would suppose for debate in terms of good versus bad, in terms of guidance, best practices to strive for. Now when it comes to design on the other hand, I would characterize design as a much more creative activity. I think that dovetails nicely with some of the authors I've read on the subject, some of the books and articles that I've read on the subject of over the years. I think that it makes a certain natural sense.

That being the case, that design is more of an aesthetic makes it a little bit harder to talk about it terms of a practice in that much more so than a particular project dictates or your particular programming style dictates, as I mentioned for the other two practices, design really resides in your own view of things. Relies much more on your particular sensibilities as a creative individual rather than some objective measure, some absolute that we can hold up and perhaps agree on to varying degrees. Again, the technique here can be taught and we'll talk a little bit about technique as well as general characteristics of good design. But I think that sensibility has to be developed from within.

And that maybe speaks to the rant that I held forth with a couple of weeks ago in terms of a vocation, a calling. Or maybe some notion of a native talent, a native creativity, that there just may be people that have a spark of design ability and people that just don't. And that's not a judgmental observation, a critical observation. It's just an observation of the quality of different people.

Now the first thing I wanted to talk about in terms of good design is the notion of leaving dross on the die if you will. The reason that I think about this and the reason that I talk about this is that design more so than any of the other practices that I've talked about, that I am planning on talking about, really is about uncertainty. What I would refer to as a valid ignorance, ignorance that has a good reason for being there. And discovery, discovery in the form of learning, of overcoming that valid ignorance. We don't know something about the domain, the problem we are designing the solution for. Discovering, learning is how we overcome that ignorance.

the notion of leaving dross on the die, spare material on the die, goes back to an article from Software Development magazine from August 2003. This is one of my favorite periodicals on the practice of professional programming. The title of the article is morphing the mold and it actually likens design in software development to the design activities in the automotive industry. And the notion is that if design proceeds very naturally from the broadest strokes to the most detailed that the automotive industries, the Japanese automotive industries in particular what the article talked about, the die makers and the dies are big pieces used to press metal and form it into the various pieces that make up the body panels and the various construction components, large construction components of an automobile. Then the die cutters can go off and they can start cutting away larger pieces of the material and they can leave material on the die where there is some fuzziness in the design, where there is still some aspect that has yet to be pinned down by the designers back at the design shop.

This I think, the article makes good parallels on its own and in my experience since I've read this it very much resonates with the design experiences that I've had. Most recently I worked on a project to implement a single sign on component for a set of enterprise servers. All these servers are under the same umbrella. They are deployed as part of our managed hosting offering. But the security components, the authentication components, are siloed to a certain respect. That you may have to maintain multiple logins to get into if you are a power user of some sort to get into the extranet portion of the offering, to get into the administrative interfaces of the offering, to get into customer support, customer service capabilities. That just doesn't make a lot of sense in terms of there's one person with those multiple identities. It would be better to have one logical identity that encapsulates those all.

The uncertainty that I had in putting this together was how do I validate trust between separate physical hosts, that these services are not always deployed on the same physical system. Therefore you can't assumes that a request that comes in over the wire is to be trusted right out of the gate. Thought I had some good ideas at the outset, but I didn't know exactly what the design was going to look like. The design document that I put together after doing the week and a half design work really left that as a large question mark, as extra material on the die if you will to go back to the original metaphor.

The nice thing about that is that as that piece in particular can be treated as a black box. That there weren't fortunately really repercussions on other aspects of the system that caused maybe more open issues to unfold or make it harder to resolve open issues and really kind of go forward with confidence implementing other parts of the system. So in this case, after the design was done, after the minimum part of the implementation had been completed in terms of the data objects that were needed to support this host to host trust negotiation, the remote call interfaces had been set up to the point, and I had all the pieces really I needed, on the primary end, the new SSO primary server, and then the existing data services that exist and were called by the existing service. Then I could sit down and what I did was some sort of prototyping, some rapid iterations, micro-iterations if that makes sense, to figure out exactly what was needed to address kind of the questions that I had going into it still in terms of like I said how do I authenticate a server identity, how do I know that a server making the request is indeed the one that I think is making the request.

So it turned out to be a good experience, in terms of playing around with some cryptographic protocols and APIs to do digital signatures in terms of authenticating requests, and then also to do encryption of those requests, and that the request be made on any number of different protocols. So to be able to do an application of the encryption so that it's independent of the underlying protocol was pretty cool. And that turned out to work very well in my experience.

Now, the next thing I want to talk about in terms of design is that design, contrary to what I've seen a lot of people characterize it as, or more importantly, I guess, approach it as during the process of building a system. It's not an artifact. It's not an end result. And I apologize if this is obvious to some of you, but I've seen enough evidence that it's maybe not as obvious as it should be, that design really is an ongoing process. And to be fair, there are some methodologies that explicitly put forward design as a process. RUPS (Rational Unified Process) is one that I worked with that talks about how the phases kind of taper off and overlap in a more organic fashion. But it still has, at a certain point, design activity stops, and then implementation takes over, and it's 100%. And then that tapers off, and then maintenance, and so and and so forth, and your remaining phases take over, however that cross-over overlap may work.

Now, in my experience, your design doesn't really end at the specification, that there are things that -- again, going back to my point about design being about addressing ignorance and trying to lay out the mechanism for discovery, how to contain and eradicate that ignorance in the process of building that project over time. There's always going to be some uncertainty, and the specification is just not going to be able to capture everything. It just doesn't make sense. The notion that I always like to use in arguing this notion to some people that I have to argue it with is that if you knew enough to write an absolutely perfect specification, the time invested to write that specification would probably be pretty close to the time that it would take to implement the system, or as close to as makes no odds. So I think it's very natural to leave design in the "design period" a little bit open-ended, to my previous point about leaving dross on the die, if you will. I think it's okay if you can compartmentalize the things that you're not as comfortable writing the detailed design about and minimize the risk that might flow over onto other systems where your confidence is higher. I think that's fine.

The point, though, about process is that even for the systems that you have confidence with, when you actually get in and start to do the implementation, you may find that your assumptions are invalidated, or that your recall of some API or how some function works was not entirely correct. And so I think it's very natural that there's a feedback loop, that the design starts then to be informed by the implementation. And I think that if you kind of ignore this, if you just divorce design at that point from the implementation and go forward, you're throwing out all the value that your design work encapsulates, just for some variance that's uncovered through the implementation work. And that just doesn't make sense to me. I think it makes better sense to have your design documents be living documents, if you will, something that you return to on either an ad-hoc basis as makes sense. So in the case of the host-to-host trust negotiation that I talked about, it makes sense that now, as I've kind of wrapped that up and I've completed my minimal functional testing I'm pretty comfortable with the way the implementation works, it makes sense to take an afternoon, go back, diagram some of that, and capture that back into my original design document so that for those coming in after that want to have some understanding, that there is some guide to the code. If they want to have a high-level understanding, they don't have to read the code at all. They can still go to the design document, and at this point I'm not going to have too much out-of-sync with the actual implementation work.

That's got to be balanced, of course, with a number of resources that you have as always, the process that you're using. You may just not have time; you may have to find creative ways to capture those design evolutions and improvements that happen as a consequence of implementation. But I think that's worthwhile. And I guess that's just my point: it's worth thinking about the design ongoing as you work through it. And then moving forward, also I think that if you keep design more as an ongoing process in your mind, when you sit down to do the explicit design work for your next project, the next piece of work, you may be even more comfortable. You don't have this six- or nine-month gap where you're just doing implementation and nothing else and you have to kind of stretch your design muscles. If you've been thinking about design all along, I think you're going to be more comfortable and more confident doing your next bit of dedicated design work.

Now speaking about dedicated design work, I want to make an argumentative, arguably false (I welcome, as always, debate on this) but I'm going to put forward one of my contentions when it comes to roles and design: that peer designers, typically referred to as just architects, are losers. And that's a judgmental and a critical statement, but I'm going to try to back that up, so try to bear me out, try listen to my argument, and like I said, if you disagree with me, send me a comment through any of the channels that I mention at the end of the show. I welcome debate, and if it's interesting enough I'll address it in a future show, and we'll refine our understanding together.

Anyway, so as I said, I think good design is informed by the implementation work as much as the reverse is also true. I think that to get away with having somebody as a titular architect or designer, at the very least I think that people should be on the hook to be kind of the plumbing work, the scut work, do the infrastructure and the framework code. And if you've worked on a larger project, you know the kinds of components that I'm talking about. These are not specific to any kind of user activity or user story; they're the code components that underlie the rest of the system, whether that's a data-mapping system to go from your object or component realm into a database, or it's an input validation or any other kinds of low-level services that this system needs. Just to give some insight as to where they may be introducing pain into the implementation process for the "non-architects" on the team. I think even then there needs to be a feedback loop there, some close feedback with the first user.

And I think there's actually an interesting correlation or touch point with the discussion of testing from a couple of weeks back, that if you're on a larger team, you have the luxury of being able to stage out the infrastructure development and pick maybe just a couple of screens, a couple of user functions, to leverage that infrastructure development and framework as early as possible. You get a nice lapidary effect, I like to refer to it -- think about a rock tumbler and how that polishes stone -- that that feedback actually can help you find where the rough spots are, both in your design and implementation and in the infrastructure and the framework, and identify what's most important to apply a little bit of smoothing effort to.

Now, if you don't have a large team, I actually think that this is one of the cases where judicious, introspective testing can serve a very similar purpose, that if you write unit tests or integration tests that try to emulate the anticipated use driven off of the functional specification and/or the requirements, you can get that same kind of effect: that your test code will drive and exercise your framework code and your design in much the same way that that first client, that that first programmer client if you will: that other team member that is actually starting to use your work right out of the gate, can have in terms of that feedback loop that I just talked about.

I also think, unfortunately, that pure architects or pure designers are somewhat lacking in humility in my experience, that they get into these roles... I'm not sure how or why, organisations tend to put them above regular programmers in some sense, so it goes back to the rant last week about managers, that people look to get into these pure design, pure architectural roles more as a reward, as a status symbol, as some sort of elitism vs. a desire to actually support and help the team. So, like I said, as a consequence, going back to my original point that there is some lack of humility to the people who typically fill these roles, there isn't an understanding that there is no such thing as a perfect design, that it's okay that there are open issues, that there are questions, that there are things that your design is going to get wrong, and you have to be willing and accepting of having conversations about where the design fails and how to improve and overcome it.

Definitely evolution of design is a good thing. If you're closed off to any changes in the design at all, then how can you evolve the design over time? How can you accept input from the programmers that are implementing the design downstream of that, any customers that are using it that are affected by the design choices you made? How can you make those improvements if you're just closed off. I think evolutionary design, evolutionary implementation is one of the things about agile methodologies that I groove very strongly. It makes a lot more sense. And tying in with that humility and that openness and that acceptance is that you have to be willing as a designer to learn from your mistakes. Like I said, no design is perfect. You're going to get it wrong from time to time. You can either get very defensive and very uppity, and you can just try to cram an incorrect design down someone's throat, or you can learn from it, and improve it, and become better. And I would argue that if your team members are programmers of quality and people of quality, they're going to have more respect for you as a designer, whether you're a dedicated designer or you're doing it on top of other responsibilities as a software engineer, that they're going to have more respect for you. They're going to see that you're open to being proven wrong, and you're open to discussion, that that's a good thing.

So that's why I say "pure architects are losers". Yeah, hopefully that makes some sense, and even if you disagree, hopefully it's maybe in some of the finer points as opposed to the idea itself.

Just two more points, I promise, and then I'm done.

The next one is that I think that over-engineering is a response to ignorance. One of the first points that I made was that design is all about dealing with and tackling, compartmentalising and finding ways to eliminate ignorance. So saying that over-engineering is a response to ignorance may seem kind of strange as a consequence. But hear me out.

The notion I'm trying to communicate here, the point I'm trying to make, is that some of the ignorance may be things that are applicable to the work at hand. These are questions that you legitimately have to solve to have a functional system that meets the requirements. Some of it may be future work, it may be to my point about evolving systems and maybe thinking about the next iteration, the next generation of the design of the overall system. And it's very tricky, I think sometimes, to draw the line between those: between things that you have to accomplish today to meet today's goals, vs. things that you may be asked to do tomorrow to accomplish tomorrow's goal. And I think that people give too much weight to tomorrow's problems in the balance and start to over-engineer to try to anticipate questions or problems that they don't know about, that they may have to solve down the road, and then as a consequence they start to over-engineer. I guess, maybe, to back up, my definition of over-engineering is solving problems that you don't really need to solve.

Kent Beck in his introduction to Extreme Programming does an exceptional job about talking about, don't build the things that you don't need. If you think you're going to have to build it tomorrow but you're not sure, just wait until tomorrow, until you have that certainty. I think it's a very simple idea that doesn't require much more explanation than that.

I think that, on the flip side of that, though, what he doesn't talk about is that you do have to be careful in building only what you need today that you don't suffer too much from myopia, from short-sightedness, that you try to make sure that you do no harm in your design and in your implementation, and that you don't close off possibilities such that, you know, when tomorrow does come around and you had some insight that maybe you would need something, that you made a design or implementation choice that makes it impossible to address that problem, impossible to address that need.

Again, this is tricky. I can't give you any hard and fast guidelines. Experience is really going to help you here to understand when you're starting to go from good design and implementation and maybe starting to lean more into over-engineering. I think conversations, in my experience -- find a trusted teammate, find a mentor or a guru and have a conversation with them -- might help you expose that engineering. You know, get somebody to play devil's advocate, to say "do we really need that today? Gee, that really kinda sounds complex..." You know, when you start to hear that feedback, accept that and start to think about it just in the way that it's meant. Go, "do I really need that today? Maybe it is more complex than it needs to be."

Now that kind of dovetails with my next point, and one of the principles that I'm going to get into in a future installment of the Inner Chapters, and that's that struggle for simplicity, and I'm going to talk about that next in terms of my final point, in terms of tools (and again, I'm going to make a very contentious statement here, and again you're welcome to disagree with me, and if you feel strongly enough to actually let me know that you disagree and you maybe try to persuade me otherwise) is that all design tools suck, without exception.

That being the case, however, suckage is not an absolute, and tools will suck to varying degrees, and you really at the end of the day just have to decide what works for you: whether that's a diagramming language like UML or a case tool, or a design tool, or a diagramming tool, or a design methodology... what have you. These all fall into the design purview of different kinds of tools. I would tend to look at it in terms of maybe an economic perspective and say anything you cannot justify in terms of concrete gains or benefit shouldn't be used. You should just ditch it. So then that drives you down to -- this is my point about dovetailing with the last one -- the simplest tools, the simplest approach to solve the problem, and no more. Okay, so hopefully that makes sense.

And then, again, even with tools, even with kind of applying a budgeting approach to which tools you use, how much you use them, the only limit again really is "Do no harm". Do nothing that closes off possibilities later. So you have to careful of that here, as well, that again: you don't want to over-engineer, but you want to make sure that you leave enough room open for tomorrow's needs and tomorrow's problems.

So hopefully all of that made sense. Design is, I think, a very near and dear topic to all professional developers out there, so hopefully I've given you some food for thought. I doubt I've been able to instill any kind of specific aesthetic when it comes to design in your mind, but hopefully I've given you some points to think about as you approach that and try to develop your own design aesthetic.