Magic Ink by Bret Victor

• I suggest that the root of the software crisis is an identity crises--an unclear understanding of what the medium actually is, and what it's for.
• Graphic design is the art of conveying a message on a two-dimensional surface.
• When the software designer defines the visual representation of her program, when she describes the pictures that the user will interpret, she is doing graphic design, whether she realizes this or not.
• Industrial design is the art of arranging and shaping a physical product so ti can be manipulated by a person.
• A good industrial designer understands the capabilities and limitations of the human body in manipulating physical objects, and of the human mind in comprehending mechanical models.
• Although software is the archetypal non-physical product, modern software interfaces have evolved overtly mechanical metaphors.
• Manipulation of machines is the domain of industrial design.
• When the software designer defines the interactive aspects of her program, when she places these pseudo-mechanical affordances and describes their behavior, she is doing a virtual form of industrial design. Whether she realizes it or not.
• The software designer can thus approach her art as a fusion of graphic design and industrial design.
• Software is for people. To derive what software should do, we have to start with what people do.
• At the present, software can't do much for physical needs. But people are increasingly shifting their intellectual activities to the virtual world of the computer.
• This suggests three general reasons why a person will turn to software: to learn, to create, to communicate. I propose that software can be classified according to which of these needs it serves. I will call these categories information software, manipulation software, and communication software.
• Information software serves the human urge to learn. A person uses information software to construct and manipulate a model that is internal to the mind--a mental representation of information. Good information software encourages the user to ask and answer questions, make comparisons, and draw conclusions.
• Manipulation software serves the human urge to create. A person uses manipulation software to construct and manipulate a model external to herself--a virtual object represented within the computer, or a remote physical object.
• Communication software serves the human urge to communicate. A person uses communication software to construct and manipulate an internal model that is shared with others--an understanding synchronized across multiple minds.
• In terms of raw mechanics, communication can be thought of as creating a response to information learned--that is, the external model manipulated by the speaker is the internal model learned by the listener.
• Manipulation software generally displays a representation of an object--the model--which the user directly manipulates with pseudo-mechanical affordances. Because manipulation is the domain of industrial design, manipulation software emphasizes industrial design aspects.
• The primary design challenge, just as with any industrial design, is to provide affordances that make these mechanical operations available, understandable, and comfortable.
• Although manipulation is the focus, good manipulation software must provide superb visualization as well. This establishes the feedback loop that is critical for all creative activity--the manipulator must see the effects of her manipulation. Thus, manipulation software design is also a significant graphic design challenge.
• Even more graphically challenging is manipulation of abstract objects, such as music or financial data, where the graphical representation must show not only what can be done with it, but what it is in the first place.
• Because of these intertwined design challenges, the design of excellent manipulation software is unbelievably difficult, and mustn't be underestimated. Fortunately, for an enormous class of software, manipulation is not only largely unnecessary, but best avoided.
• Most of the time, a person sits down at her personal computer not to create, but to read, observe, study, explore, make cognitive connections, and ultimately come to an understanding. This person is not seeking to make her mark upon the world, but to rearrange her own neurons. The computer becomes a medium for asking questions, making comparisons, and drawing conclusions--that is, for learning.
• Even consider reading email. Most current designs revolve around the manipulation of individual messages--reading them one-by-one, searching them, sorting them, filing them, deleting them. But the purpose of reading email has nothing to do with the messages themselves. I read email to keep a complex set of mental understandings up-to-date--the statuses of personal conversations, or projects at work, of invitations and appointments and business transactions and packages in the mail. That this information happens to be parceled out in timestamped chunks of text is an implementation detail of the communication process.
• Ignore the structure of current designs, and ask only, "Why is a person using this?" Abstracted, the answer almost always is, "To learn".
• A well-designed information graphic can almost compel the viewer to ask and answer questions, make comparisons, and draw conclusions. It does so by exploiting the capabilities of the human eye: instantaneous and effortless movement, high bandwidth and capacity for parallel processing, intrinsic pattern recognition and correlation, a macro/micro duality that can skim a whole page or focus on the tiniest detail. Meanwhile, a graphic sidesteps human shortcomings: the one-dimensional, uncontrollable auditory system, the relatively sluggish motor system, the mind's limited capacity to comprehend hidden mechanisms. A graphic presents no mechanisms to comprehend or manipulate--it plugs directly into the mind's spatial reasoning centers.
• Compared to excellent ink-and-paper designs, most current software communicates deplorably. This is a problem of surface, but is not a superficial problem. The main cause, I believe, is that many software designers feel they are designing a machine. Their foremost concern is behavior--what the software does. [...] These designers start by specifying functionality, but the essence of information software is the presentation.
• I suggest that the design of information software should be approached initially and primarily as a graphic design project. The foremost concern should be appearance--what and how information is presented.
• The designer must start by considering what the software looks like, because the user is using it to learn, and she learns by looking at it.
• Instead of dismissing ink-and-paper design as a relic of a previous century, the software designer should consider it a baseline. If information software can't present its data at least as well as a piece of paper, how have we progressed?
• Edward Tufte's first rule of statistical graphical design is, "Show the data". All information graphics, statistical or not, must present the viewer with enough information to answer her questions. It seems that many software designers, in their focus on functionality, forget to actually present the data.
• [...] the most egregious problem is simply that there is not enough information to make any sort of decision.
•  The standard 5-star rating system is information-weak--it gives only an average. It can be enhanced with whiskers underneath that indicate the distribution of ratings.
• [...] ideas in the head are nebulous things, and may not translate perfectly to a concrete search term.
• A world of possibilities opens up with a simple change of mindset.
• Just as important as what data is shown is where it is shown.
• Unfortunately, most software graphics are arranged to maximize aesthetics, not to bring out useful relationships in the data.
• Print has one supreme flaw: ink is indelible. An ink-and-paper design is static--it must display all its data, all the time. However, a reader typically only cares about a subset relevant to her current situation. The designer is faced with the challenge of organizing the data so that hopefully mutually relevant subsets are grouped together, and the reader has the challenge of visually or physically navigating through the entire data space to find the group of interest.
• The modern computer system provides the first visual medium in history to overcome this [paper] restriction. Software can:
• infer the context in which its data is needed,
• winnow the data to exclude the irrelevant, and
• generate a graphic which directly addresses the present needs.
• Liberating us from the permanence of publication is the under sung crux of the computer--the dynamic display screen. Its pixels are magic ink--capable of absorbing their context and reflecting a unique story for every reader. And the components surrounding the display--CPU, storage, network, input devices--are its peripherals for inferring context.
• Information software design, then, is the design of context-sensitive information graphics. Unlike conventional graphics, which must be suitable for any reader in any situation, a context-sensitive graphic incorporates who the user is and what exactly the user wants to learn at the moment. Context allows software to winnow its data space to the subset of information that the user cares about, and present the data in such a way that the user's current questions can best be answered.
• All information software consists of context-sensitive graphics, whether the designer realizes it or not.
• There are, in fact, three sources from which software can infer context:
• Environment involves sensing the current state of the world.
• History involves remembering the past.
• Interaction involves soliciting input from the user.
• A person determines her surroundings though the five human senses. Software doesn't operate in a vacuum, either; through connections to hardware and other software, it can sense much about the user's situation.
• Time is one of the fundamental dimensions along which we organize our lives, and in any data space with a temporal dimension, "now" is almost always the prime landmark. Because users often seek information on demand, information related to "now" or "soon" is often the most relevant.
• Similarly, the most interesting spatial landmark is usually "here".
• Creating some information indicates an even stronger topic of interest.
• All software lives within an environment, rich with evidence of context. Using software that doesn't look outside itself is like conversing with a blind person--constantly describing what is plainly visible. On the other hand, the power of the environment is multiplied when it is correlated with the past--that is, when the software makes use of history.
• A human doesn't just use her senses to recognize her situation; she also uses memories of past situations. Software, too, can use its memory to understand the present. The current context, or a good approximation, can often be predicated from a history of past environments and interactions.
• Last-value predictors represent the simplest form of prediction. They simply predict the current context to be the same as the previous one. This is reasonable in many situations where the user's context is fairly static, changing slowly over the short term.
• Learning predictors attempt a deeper understanding of the user. They construct a model to explain past contexts, and use the inferred relationships to predict the current context.
• Large histories can enable some very sophisticated predictors, especially if it is possible to reach into the environment and correlate with other user's histories.
• Most software wakes up each day with a fresh case of amnesia.
• Chris Crawford defines interaction as a three-phase reciprocal process, isomorphic to a conversation: an interactant listens to her partner, thinks about what was said, and speaks a response. Her partner then does the same.
• For manipulation software, interaction is perfectly suitable: the user views a visual representation of the model, considers what to manipulate next, and performs a manipulation. The software, in turn, inputs the user's manipulation request, updates the model, and displays the updated representation.
• Information software, by contrast, mimics the experience of reading, not working. It is used for achieving an understanding--constructing a model within the mind. Thus, the user must listen to the software and think about what it says..but any manipulation happens mentally.
• For information software, all interaction is essentially navigation around a data space.
• Navigation implies state. Software that can be navigated is software in which the user can get lost.
• State is the primary reason people fear computers--stateful things can be broken.
• Interaction should be used judiciously and sparingly, only when the environment and history provide insufficient context to construct an acceptable graphic.
• I believe that with the invention of new context-sensitive graphical forms and research into obtaining and using environment and history, the clicking and dragging that characterizes modern information retrieval will be made to seem laughably archaic.
• Command-line systems are criticized for forcing the user to learn the computer's language. Modern GUIs may be easier to use, but they are no much different in that respect. The GUI language consists of a grammar of menus, buttons, and check boxes, each labeled with a vocabulary of generally decontextualized short phrases. The user "speaks" by selecting from a tiny, discrete vocabulary within an entirely fixed grammatical structure--a bizarre pidgin unlike any human language, unexpressive and unnatural.
• A user can specify context by pointing somewhere on an information graphic and saying, "There!".
• Two of the most fundamental context dimensions are where and when.
• Geographical locations belong on maps, and dates belong on calendars.
• If you do not receive feedback that indicates you are on the right track, the action you took will have very little meaning.
• [...] each interaction can and should result in a discernible change to a context-sensitive information graphic. Providing immediate feedback reduces the amount of manipulation the user must do before either reaching an adequate view or recognizing a wrong turn and backtracking.
• Much current software is interaction-heavy and information-weak. I can think of a few reasons for this.
• First, Our current UI paradigm was invented in a different technological era.
• A second reason why modern software is dominated by mechanical metaphors is that, for the people who create software, the computer is a machine.
• Information software allows the user to ask and answer questions, make comparisons, and draw conclusions.
• Audio works well for infrequent, asynchronous notifications.
• For presenting abstract, non-comparative information [...], an excellent graphical element is simply a concise sentence. [...] There is no "OK" or confirmation button--the sentence always represents the current configuration.
• Sentence-based configuration scales so well because parameters are given meaning by the surround textual context, which itself consist of other parameters.
• The pull down menu, the check box, and the bureaucracy-inspired text entry form were invented 25 years ago, desperation devices to counter inadequate technology. They were created for a world that no longer exists.
• Twenty-five years from now, no one will be clicking on drop-down menus, but everyone will still be pointing at maps and correcting each other's sentences. It's fundamental. Good information software reflects how humans, not computers, deal with information.
•  The problem [pervasive bad design] is primarily cultural.
• The first step toward the information software revolution is widespread recognition of the need to design. It must be universally understood that information software is not a machine, but a medium for visual communication, and both publishers and public must hold it to the same standards that they hold print.
• Today, software consumers demand technological features because software marketing presents features. Consumers ignore design because marketing ignores design. The cycle is vicious, but perhaps vulnerable too--some brilliant new software with engineering, design, and marketing all in sync may raise the bar for everyone.
• The second step toward the information software revolution is finding people with talent for visual communication.
• I believe that ideal candidates for software design are those who have achieved mastery of information graphics in other mediums.
• The third step is complementing the designer's talent with skill. Skill is achieved through education and practice, but dearth of the former has given aspiring designers no entry point--they are expected to learn the art through osmosis and guesswork.
• The paucity of literature on information graphic design is bewildering.
• The fourth step is supplementing the designer's talent and skill with tools and platforms. These two terms are vague in common usage. I will define a tool as a communication device that a designer has control over, and a platform as a communication device that a recipient is expected to provide.
• A tool encodes mental information into physical data, which can travel in a physical medium. A platform decodes the physical data into the mind of the recipient. Because all information transfer short of telepathy requires some medium, this model is universal.
• In general, my tools are whatever I use to make the thing I hand off to you. Your platform is whatever I'm counting on you to already have.
• To deliver her message most effectively, the visual designer needs as much control as possible over what the viewer sees. But, by definition, the designer only has direct control over the tool. She is at the mercy of whatever platform implementation the recipient happens to supply. This implies that a good platform must be as simple and as general as possible.
• For a platform to be reliable, it must either have a single implementation, or be so utterly simple that it can be implemented uniformly.
• From a practical (and historical) standpoint, we can assume that no complex specification will be implemented correctly.
• If we assume a practical need for open, freely implementable standards, the only option is simplicity.
• In order for a designer to take full advantage of the medium, a good platform must provide safe access to everything that is technologically possible.
• The proper way to prevent destructive behavior is a well-designed security model, not arbitrarily amputating the computer's capabilities.
• The fifth and final step into the information software revolution is an environment where experimentation, evolution, and interplay of ideas can thrive.
• Drawing is a visual activity, and working with textual abstractions is entirely inappropriate.
• All software today performs an intricate dance of feeding a processor primitive machine-level instructions, polarizing tiny magnets in hard drives, transmitting bits reliably across wires, and lighting up specific pixels on a screen. The algorithms behind these operations are unimaginable complex, requiring years of study. Yet, even novice programmers have no trouble with these operations. The complexity has been hidden behind abstractions.
• Without these abstractions, our modern software landscape simply wouldn't exist.
• As I see it, the primary challenge for the machine learning community is not the generation and tuning of yet more algorithms, but the design of simple abstractions. Learning magic must be packaged. Like a "file", the abstraction must be usable by any engineer working in any domain. It must be so simple that it can be taken for granted.
• Until machine learning is as accessible and effortless as typing the word "learn", it will never become widespread.
• Monolithic systems don't scale.
• Monolithic systems are bad for users.
• Monolithic systems are bad for software providers.
• An ecosystem is a network of individual components which consumer nutrients and translate them to an enriched form consumed by others, autonomously and with no knowledge of the system as a whole.
• Culture changes much more slowly than technological capability.
• The lesson is that, even today, we are designing for tomorrow's technology. Cultural inertia will carry today's design choices to whatever technology comes next. In a world where science can outpace science fiction, predicting future technology can be a Nostradamean challenge, but the responsible designer has no choice. A successful design will outlive the world it was designed for.
• I believe that in order for a personal information device to be viable in the long term, it must satisfy two conflicting criteria: portability and readability.
• As people increasingly expect information on demand, portability will become ever more critical. Today, people can talk to anyone on the planet by reaching into a pocket; tomorrow's information device must be just as accessible. [...] This implies light weight and small volume.
• Although technology miniaturizes, the human eye span remains a fundamental constant. In order to compete with the book, tomorrow's information device must provide a book-sized surface area. Anything less cannot be read and skimmed comfortably, and cannot support spatially-distributed information graphics.
• To resolve these contrasting size constraints, I predict a computer the size and thickness of a sheet of paper. Like paper, its entire surface is a graphical display. When in use, it is rigid; when not in use, it collapses and can be folded or rolled up (or crumpled!) and tucked into a pocket or purse.
• To serve as a book, the device must have a sufficiently large reading area and high pixel resolution. To serve as a computer, the device must produce dynamic color graphics.
• Dynamic graphics with print resolution will open up a world of possibilities for detailed information graphics which are impossible today in either medium.
• Because the user will carry this device everywhere, the device's environment will literally be the user's own. [...] The device will have a far better sense of the user's environment than the user herself.
• Since its inception, electronic storage has exponentially increased in density and decreased in cost. We can fully expect tomorrow's device to have onboard capacities that stagger modern sensibilities. But, perhaps more importantly, ubiquitous network access will make memory effectively unlimited.
• Touch or motion-based manipulation is somewhat more efficient than the mouse. Eye-tracking and speech may be better still, although even these are unlikely to match the order-of-magnitude improvements predicted for the capabilities above. But non of these mechanisms will ever approach the sheer amount of information that can be absorbed by the eye. No matter what new interactive technology comes along, the bandwidth between the device and the user will remain not merely asymmetric, but utterly lopsided.
• Interaction is already a bottleneck. It will get much worse as graphics, environment, and history experience their expected breakthroughs. To me, the implication is clear--the principles of information software and context-sensitive information graphics will become critical as technology improves.
• The future will be context-sensitive. The future will not be interactive.
• Our pioneers are those who transcend interaction--designers whose creations anticipate, not obey.
• Software design consists of graphic design (drawing pictures) and industrial design (allowing for mechanical manipulation).
• Information software is for learning an internal model. Manipulation software is for creating an external model. Communication software is for communicating a shared model.
• Manipulation software design is hard, but most software is information software.
• Information software design is the design of context-sensitive information graphics. Information software is not a machine, but a medium for visual communication.
• Context can be inferred from the environment, which can include physical sensors, other information software, documents created with manipulation software, and data such as email which acts as a user profile.
• Context can be inferred from a history of past environments and interactions. Last-value predictors provide a rudimentary approach. Learning predictors can infer patterns and make dynamic predictions.
• Context can be inferred from user interaction, but only as a last resort. The best way to reduce or eliminate interaction is through information-rich graphic design that uses the environment and history. Remaining interaction can be reduced with graphical manipulation, relative navigation, and tight feedback loops.
• The information software revolution will require public recognition that information software is a medium of visual communication, designers with talent, skill, and tools, simple and general platforms, and an environment that encourages creativity and sharing.
• A design tool for dynamic graphics that infers behavior from mock ups may allow for natural-feeling creative design with no engineering-related distractions.
• Learning predictors exist and are effective. For them to become widespread, simple abstractions must be invented.
• An information ecosystem of views and translators may be able to offer relevant information of all forms with minimal interaction. Key aspects include topic nomination and translation, confidence levels, learning through feedback, and a fine-grained modular structure wherein small software providers can thrive.
• As technology related to graphics, the environment, and history undergoes revolutionary improvements, interaction will become even more of a critical bottleneck. The best approach is to work towards eliminating it.
• Two centuries ago, Playfair invented statistical graphics and changed the world. The time is ripe for another designer to invent the fundamental context-sensitive graphical forms, and change the world again.
• The only purpose of incremental improvement to a status quo is to reveal its flaws. Progress occurs when the status quo is replaced.