# wallabag_entry

title: What's the Blockchain, and Why Does Bitcoin Depend On It?
url: http://gizmodo.com/whats-the-blockchain-and-why-does-bitcoin-depend-on-it-1698025216
created_at: 2017-08-10 23:19:23
updated_at: 2017-08-10 23:19:23
mimetype: text/html
language: en-US
domain_name: gizmodo.com
http_status: 200
content:
  <div class="img-wrapper lazy-image"><div class="img-permalink-sub-wrapper"><img src="https://i.kinja-img.com/gawker-media/image/upload/s--5ObOr3fn--/c_fit,fl_progressive,q_80,w_320/1210124925415016619.jpg" class="lazyload ls-lazy-image-tag" alt="image" /></div></div><p>It’s going to unshackle us from the oppressive dungeon of fiat currency! But also criminals and rogue cops use it to do nefarious drug stuff! Digital currency is often defined by its volatile hype cycle. And yet its most promising feature is incredibly mundane-sounding: a bookkeeping system called “the blockchain.”</p><div class="ad-unit js_ad-unit google-adx"><div class="ad-inner"><div id="ad-container-51497847" class="ad-container js_ad-container js_ad-container-top"></div></div><p>So what the heck is it?</p><h4>The blockchain basics </h4><p>The blockchain is a simple digital platform for recording and verifying transactions so that other people can’t erase them later — and anyone can see them. “You can think of the blockchain of an ‘append-only’ ledger. You can only write to it, you can’t delete it,” Peter van Valkenburg, the director of research at Coin Center, told Gizmodo.</p><div class="ad-container js_ad-video row ad-wide ad-top js_ad-video-desktop"><div class="ad-instream--waypoint"><div class="ad-instream ad-instream--elastic instream-permalink ad-instream__initial"><div class="elastic__wrapper"><div class="ad-unit js_ad-unit ad-video js_ad-video"><div id="ad-container-56702430" class="ad-container js_ad-container"></div></div></div></div><div class="js_ad-mobile-dynamic js_ad-dynamic"><div class="ad-unit js_ad-unit hide-contents ad-mobile js_ad-mobile"><div class="ad-mobile-inner"><p class="ad-label proxima"><small class="proxima">Advertisement</small></p><div id="ad-container-41049234" class="ad-container js_ad-container"><p class="ad-label-bottom"></p></div></div></div><p>But it’s what the blockchain <em>doesn’t</em> have that makes it controversial, and prone to giving venture capitalists greedboners. Bookkeeping tools generally require a bookkeeper, but with the blockchain, there is no head honcho bookie. It’s a decentralized, crowd-powered spreadsheet, relying on cryptography instead of a central authority.</p><p>Fervent crypto-enthusiasts, never ones for ambivalence, say the blockchain technology Satoshi Nakamoto created to fuel Bitcoin has genuine potential to tip power dynamics in banking, politics, the internet, and everywhere authorities, well, authorize things. Critics see it as either hopelessly impractical or a fast-track to <a href="http://krugman.blogs.nytimes.com/2013/12/28/bitcoin-is-evil/" target="_blank" rel="noopener">a merciless, anarchic hellscape</a>.</p><h4>But how does it work exactly? </h4><p>The blockchain is designed to make transactions safe and reliable even if the people doing them don’t trust each other. </p><div class="js_ad-mobile-dynamic js_ad-dynamic"><div class="ad-unit js_ad-unit hide-contents ad-mobile js_ad-mobile"><div class="ad-mobile-inner"><p class="ad-label proxima"><small class="proxima">Advertisement</small></p><div id="ad-container-32230041" class="ad-container js_ad-container"><p class="ad-label-bottom"></p></div></div></div><p>Nakamoto’s blockchain math is a solution to a famous game theory puzzle called <a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/byz.pdf" target="_blank" rel="noopener">the Byzantine General’s Problem</a>, in which old-timey army officers are planning an attack, but they’re worried about traitors lurking in their midst. Trust no one! But, of course, an army needs to communicate to launch an attack. So the General needs to find a way to pass a message along that cannot be sabotaged.</p><p>This is where the blockchain comes in. Every single time you make a transaction on the blockchain, that transaction is sent out to many nodes in the Bitcoin network. Basically, everybody participating in the Bitcoin process also has copy of that ledger and can check it for inconsistencies. It’s a distributed ledger. The order of transactions is also verified by a cryptographic process that relies on the combined computational power of the crowd. So if you try to pass off a fake exchange, you’ll get caught — because your fellow Bitcoin users can trace every alteration and exchange that goes down.</p><p><a href="https://bitcoin.org/en/how-it-works" target="_blank" rel="noopener">Bitcoin.org sums this up nicely</a> in their introduction to Bitcoin:</p><blockquote><p>The blockchain is a shared public ledger on which the entire Bitcoin network relies. All confirmed transactions are included in the block chain. This way, Bitcoin wallets can calculate their spendable balance and new transactions can be verified to be spending bitcoins that are actually owned by the spender. The integrity and the chronological order of the block chain are enforced with <a href="https://bitcoin.org/en/vocabulary#cryptography" target="_blank" rel="noopener">cryptography</a>.</p></blockquote><p>There are different kinds of blockchains, with varying degrees of complexity, so the math underlying the crypto is always different. But the general principle—a decentralized group of machines capable of verifying transactions—is the same. </p><p class="has-video media-large video-embed align--bleed"><iframe allowfullscreen="allowfullscreen" scrolling="no" height="450" id="youtube-Lx9zgZCMqXE" class="core-inset lazyload" width="800" frameborder="0">[embedded content]</iframe></p><h4>The greed of crowds</h4><p>The idea of a distributed ledger is not new, but Satoshi put a spin on it — community members are incentivized to verify transactions because they earn Bitcoins for their efforts. People who are already “mining” Bitcoin are selected to have their machines join the blockchain and verify that nothing fishy is going down.</p><div class="ad-container js_ad-video row ad-wide ad-top js_ad-video-mobile"><div class="ad-instream--waypoint"><div class="ad-instream ad-instream--elastic instream-permalink ad-instream__initial"><div class="elastic__wrapper"><div class="ad-unit js_ad-unit ad-video js_ad-video"><div id="ad-container-76176343" class="ad-container js_ad-container"></div></div></div></div><p>Bitcoin “miners” use computer power to solve puzzles; if they correctly solve a puzzle, they get Bitcoin as a prize. To make blockchain technology work, the Bitcoin community needs Nakamoto’s clever incentive program. If you want to mine Bitcoin, you<em> have</em> to use a sliver of computational power to verify the blockchain when your turn comes up. The rules of the blockchain are baked into the mining software to make sure miners will check the ledger when it’s their turn. </p><p>It’s the perfect kind of crowdsourcing for libertarian fans of crypto-currency, because it’s entirely based on self-interest.</p><p>Transactions get verified frequently, and once they’ve been verified they’re set in a sort of digital stone. People have found all sorts of different ways to use Bitcoin for devious schemes, or hack cryptocurrency platforms. But none of them have involved destroying its foundation, the blockchain.</p><div class="js_ad-mobile-dynamic js_ad-dynamic"><div class="ad-unit js_ad-unit hide-contents ad-mobile js_ad-mobile"><div class="ad-mobile-inner"><p class="ad-label proxima"><small class="proxima">Advertisement</small></p><div id="ad-container-55868274" class="ad-container js_ad-container"><p class="ad-label-bottom"></p></div></div></div><p>That doesn’t mean it’s foolproof. Interpol and Kaspersky Labs <a href="http://insidebitcoins.com/news/kaspersky-and-interpol-say-blockchain-is-vulnerable/31578" target="_blank" rel="noopener">recently presented research</a> about how people could theoretically insert malware into blockchain transactions. That’s brutal news for blockchain supporters, since the whole point is that the cryptography is solid enough that you don’t need to trust anyone.</p><h4>From Satoshi to Samsung</h4><p>With Bitcoin, blockchain technology is used to transfer money. But it has the potential to do a lot more: it will work for any process of verification, really, and can even be used as a communication tool. This is why companies, investors, and crypto diehards are starting to see potential uses for blockchains all over the place. </p><div class="js_ad-mobile-dynamic js_ad-dynamic"><div class="ad-unit js_ad-unit hide-contents ad-mobile js_ad-mobile"><div class="ad-mobile-inner"><p class="ad-label proxima"><small class="proxima">Advertisement</small></p><div id="ad-container-47447387" class="ad-container js_ad-container"><p class="ad-label-bottom"></p></div></div></div><p>True believers say blockchain technology could revolutionize voting systems and make <a href="http://www.proofofexistence.com/about" target="_blank" rel="noopener">notaries</a> and banks redundant, although they’re still in the theoretical stages of spinning their ideas into reality. The tech is nowhere near ready for such widespread and intense application. But companies are working to steering this technology away from the crypto fringe. </p><p>IBM and Samsung are already <a href="http://www.coindesk.com/ibm-reveals-proof-concept-blockchain-powered-internet-things/" target="_blank" rel="noopener">developing a blockchain-powered backbone</a> for Internet of Things products called ADEPT. These companies are taking the core ledger concept and spinning it out into a way to help devices communicate cheaply. Other projects, like <a href="https://www.ethereum.org/" target="_blank" rel="noopener">Ethereum</a> and <a href="https://www.ripplelabs.com/" target="_blank" rel="noopener">Ripple</a>, take the idea of the blockchain and also spin out the concept for a wider array of applications. </p><p>Enthusiasts <a href="http://ledracapital.com/blog/2014/3/11/bitcoin-series-24-the-mega-master-blockchain-list" target="_blank" rel="noopener">rattle off a vertiginous list of spin-off uses</a>: microfinance, vehicle registration, gun permits, wills, genome data, report cards, DRM—all bandied about as stuff that’d be more secure if blockchains were used to verify them. </p><div class="js_ad-mobile-dynamic js_ad-dynamic"><div class="ad-unit js_ad-unit hide-contents ad-mobile js_ad-mobile"><div class="ad-mobile-inner"><p class="ad-label proxima"><small class="proxima">Advertisement</small></p><div id="ad-container-11479374" class="ad-container js_ad-container"><p class="ad-label-bottom"></p></div></div></div><p>ADEPT is still in very early stages, and it uses a more complicated cryptography than the OG blockchain, but it’s an example of how established corporations are attempting to capitalize on the concept. </p><p>And, as with all new technologies with grand intentions, there are plenty of startups hovering. Namecoin is basically a blockchain-powered alternative to ICANN’s (crappy) domain registry system, and it plans to provide notary services in the future. Van Valkenberg especially likes <a href="https://www.bitmesh.network/" target="_blank" rel="noopener">BitMesh</a>, a blockchain startup that wants to help people sell their extra bandwidth by <a href="http://www.wired.com/2015/03/opinion-bitcoin-may-gets-us-real-net-neutrality/" target="_blank" rel="noopener">establishing mesh networks</a> that’d serve as local marketplaces and alternatives to only using big ISPs, and it’s easy to see why: If BitMesh can make its concept a reality, it’d give people a better, cheaper way to share internet costs.</p><h4>Chain of fools or chain of gold? </h4><p>There is a reason why people use banks, hire lawyers, and get notaries to make sure their documents are legit. Society is organized around the idea that there are authorities that can be trusted. And yeah, people are fallible and companies regularly have huge security breaches using current models. But evacuating authority from central figures and relying totally on a mathematic solution developed by a pseudonym who is more a symbol than a person is also rather terrifying. </p><div class="js_ad-mobile-dynamic js_ad-dynamic"><div class="ad-unit js_ad-unit hide-contents ad-mobile js_ad-mobile"><div class="ad-mobile-inner"><p class="ad-label proxima"><small class="proxima">Advertisement</small></p><div id="ad-container-16814345" class="ad-container js_ad-container"><p class="ad-label-bottom"></p></div></div></div><p>That isn’t to say that the projects blockchain supporters are working on can’t succeed, or shouldn’t succeed. They are looking for better ways to make things known to be true; in the idyllic blockchain-managed world, we would inhabit a world where fraud was impossible. But it’s just important to note how much faith that world places in machines over humanity. </p><p><em>Art by Tara Jacoby </em></p></div></div></div></div></div></div></div></div></div></div></div>

preview_picture:
  https://i.kinja-img.com/gawker-media/image/upload/s--S5bf1wCf--/c_fill,fl_progressive,g_center,h_358,q_80,w_636/1210124925415016619.jpg


title: Institute for Blockchain Studies Website
url: http://www.blockchainstudies.org/
created_at: 2017-08-10 23:19:25
updated_at: 2017-08-10 23:19:25
mimetype: text/html
domain_name: www.blockchainstudies.org
http_status: 200
content:
  				<a href="http://www.blockchainstudies.org/index.html"><img src="http://www.blockchainstudies.org/images/one_210x171.png" alt="" /></a>
  				<b>Blockchain Technology</b>
  				<p>
  					Provide general research and overview regarding the overall direction, progress, magnitude, and impact of blockchain technologies
  				</p>
  			
  				<a href="http://www.melanieswan.com/TFI.html"><img src="http://www.blockchainstudies.org/images/two_210x171.png" alt="" /></a>
  				<b>Blockchain Theory</b>
  				<p>
  					Articulate the theoretical, philosophical, and societal impact of the blockchain as one category of
  					<a href="http://futurememes.blogspot.com/2014/10/ischools-contemporary-information.html">Contemporary Information Technology Theory Studies</a>
  				</p>
  			
  			
  				<a href="http://futurememes.blogspot.com/2014/09/blockchain-health-remunerative-health.html"><img src="http://www.blockchainstudies.org/images/three_210x171.png" alt="" /></a>
  				<b>Blockchain Health</b>
  				<p>
  					Envision blockchain healthtech apps like Personal Health Record Storage, Health Research Commons, Health Document Notary Services, and Doctor Vendor RFP Services
  				</p>
  			
  			
  				<a href="http://www.blockchainstudies.org/index.html"><img src="http://www.blockchainstudies.org/images/four_210x171.png" alt="" /></a>
  				<b>Blockchain Apps</b>
  				<p>
  					Design sample apps like Quantified Self data commons on Ethereum (daily auto-upload of heart rate data and steps-walked), and
  					Cognitive Enhancement with MemoryCoin
  				</p>
  			


title: Blockchain France
url: https://blockchainfrance.net/
created_at: 2017-08-10 23:19:26
updated_at: 2017-08-10 23:19:26
mimetype: text/html
language: fr-FR
domain_name: blockchainfrance.net
preview_picture: https://blockchainfrance.files.wordpress.com/2016/03/970x404-blockchain.jpg
http_status: 200
content:
  <div class="entry-content">
  					<p>Blockchain France fusionne avec le Labo Blockchain, startup spécialisée dans le développement technique, pour créer Blockchain Partner.</p>
  <p><a class="button" href="http://blockchainpartner.fr">Découvrir Blockchain Partner</a></p>
  				</div><div class="entry-content">
  		<p><b>Définition et explication<br /></b></p>
  <p>La blockchain est une technologie de stockage et de transmission d’informations, transparente, sécurisée, et fonctionnant sans organe central de contrôle (<em>définition de Blockchain France</em>). </p>
  <div class="more-link-container"> <a href="https://blockchainfrance.net/decouvrir-la-blockchain/c-est-quoi-la-blockchain/" class="more-link">Lire la suite « Qu’est-ce que la blockchain ? »</a></div>
  	</div><div class="entry-content">
  		<p class="graf--p graf-after--h3"><strong>La France ne doit pas rater la révolution block</strong><strong>chain</strong></p>
  <p class="graf--p graf-after--h3">Nous sommes aux prémices d’une révolution dont la portée est encore difficile à mesurer mais aux champs d’applications infinis.</p>
  <p class="graf--p graf-after--h3">
  </p><div class="more-link-container"> <a href="https://blockchainfrance.net/decouvrir-la-blockchain/notre-tribune/" class="more-link">Lire la suite « Notre manifeste »</a></div>
  	</div><div class="entry-content">
  		<p><strong>« La Blockchain décryptée » : le premier livre en français sur la blockchain</strong></p>
  <p>Nous avons rencontré les acteurs et les penseurs de la blockchain pour vous raconter cette technologie dans toute sa richesse.</p>
  <div class="more-link-container"> <a href="https://blockchainfrance.net/decouvrir-la-blockchain/la-blockchain-decryptee-les-clefs-dune-revolution/" class="more-link">Lire la suite « Notre livre »</a></div>
  	</div><div class="entry-content">
  		<p>-Faites monter en compétence <strong>vos collaborateurs. </strong></p>
  <div class="more-link-container"> <a href="https://blockchainfrance.net/nos-offres/formation-blockchain-acculturation/" class="more-link">Lire la suite « Formation blockchain »</a></div>
  	</div><div class="entry-content">
  		<p>-Identifiez les <strong>cas d’usages</strong> pertinents pour votre organisation grâce à une approche transversale.</p>
  <div class="more-link-container"> <a href="https://blockchainfrance.net/nos-offres/conseil-entreprise-exploration/" class="more-link">Lire la suite « Ateliers exploratoires »</a></div>
  	</div><div class="entry-content">
  		<p>-Partez d’un <strong>cas d’usage concret</strong> et entamez une expérimentation en interne sur un<strong> Proof-Of-Concept.</strong> </p>
  <div class="more-link-container"> <a href="https://blockchainfrance.net/nos-offres/poc-blockchain-experimentation/" class="more-link">Lire la suite « Expérimentation »</a></div>
  	</div><div class="entry-content">
  		<p>La journée d’ateliers exploratoires organisée par Blockchain France, très bien animée, a entièrement répondu à nos objectifs et a permis de faire émerger plusieurs cas d’usage.</p>
  	</div><div class="entry-content">
  		<p>L’accompagnement que nous avons de la part de Blockchain France depuis plusieurs mois répond pleinement à nos attentes, avec une prise de recul très pertinente sur cette technologie.</p>
  	</div>


title: On Public and Private Blockchains - Ethereum Blog
url: https://blog.ethereum.org/2015/08/07/on-public-and-private-blockchains/
created_at: 2017-08-10 23:19:29
updated_at: 2017-08-10 23:19:29
mimetype: text/html
language: en-US
domain_name: blog.ethereum.org
http_status: 200
content:
  <p>Over the last year the concept of “private blockchains” has become very popular in the broader blockchain technology discussion. Essentially, instead of having a <i>fully</i> public and uncontrolled network and state machine secured by cryptoeconomics (eg. proof of work, proof of stake), it is also possible to create a system where access permissions are more tightly controlled, with rights to modify or even read the blockchain state restricted to a few users, while still maintaining many kinds of partial guarantees of authenticity and decentralization that blockchains provide. Such systems have been a primary focus of interest from financial institutions, and have in part led to a backlash from those who see such developments as either compromising the whole point of decentralization or being a desperate act of dinosaurish middlemen trying to stay relevant (or simply committing the crime of using <a href="https://www.reddit.com/r/Bitcoin/comments/3bg1af/i_knew_it_would_never_work_the_moment_she_told_me/">a blockchain other than Bitcoin</a>). However, for those who are in this fight simply because they want to figure out how to best serve humanity, or even pursue the more modest goal of serving their customers, what are the practical differences between the two styles?</p>
  <p>First, what exactly are the options at hand? To summarize, there are generally three categories of blockchain-like database applications:</p>
  <ul><li><b>Public blockchains</b>: a public blockchain is a blockchain that anyone in the world can read, anyone in the world can send transactions to and expect to see them included if they are valid, and anyone in the world can participate in the <em>consensus process</em> – the process for determining what blocks get added to the chain and what the current state is. As a substitute for centralized or quasi-centralized trust, public blockchains are secured by cryptoeconomics – the combination of economic incentives and cryptographic verification using mechanisms such as proof of work or proof of stake, following a general principle that the degree to which someone can have an influence in the consensus process is proportional to the quantity of economic resources that they can bring to bear. These blockchains are generally considered to be “fully decentralized”.</li>
  <li><b>Consortium blockchains</b>: a consortium blockchain is a blockchain where the consensus process is controlled by a pre-selected set of nodes; for example, one might imagine a consortium of 15 financial institutions, each of which operates a node and of which 10 must sign every block in order for the block to be valid. The right to read the blockchain may be public, or restricted to the participants, and there are also hybrid routes such as the root hashes of the blocks being public together with an API that allows members of the public to make a limited number of queries and get back cryptographic proofs of some parts of the blockchain state. These blockchains may be considered “partially decentralized”.</li>
  <li><b>Fully private blockchains</b>: a fully private blockchain is a blockchain where write permissions are kept centralized to one organization. Read permissions may be public or restricted to an arbitrary extent. Likely applications include database management, auditing, etc internal to a single company, and so public readability may not be necessary in many cases at all, though in other cases public auditability is desired.</li>
  </ul><p>In general, so far there has been little emphasis on the distinction between consortium blockchains and fully private blockchains, although it is important: the former provides a hybrid between the “low-trust” provided by public blockchains and the “single highly-trusted entity” model of private blockchains, whereas the latter can be more accurately described as a traditional centralized system with a degree of cryptographic auditability attached. However, to some degree there is good reason for the focus on consortium over private: the fundamental value of blockchains in a fully private context, aside from the replicated state machine functionality, is cryptographic authentication, and there is no reason to believe that the optimal format of such authentication provision should consist of a series of hash-linked data packets containing Merkle tree roots; <a href="https://github.com/scipr-lab/libsnark">generalized zero knowledge proof technology</a> provides a much broader array of exciting possibilities about the kinds of cryptographic assurances that applications can provide their users. In general, I would even argue that generalized zero-knowledge-proofs are, in the corporate financial world, greatly <i>underhyped</i> compared to private blockchains.</p>
  <p>For now, I will thus focus on the simpler “private versus public” blockchain discussion. In general, the idea that there is “one true way” to be blockchaining is completely wrong headed, and both categories have their own advantages and disadvantages.</p>
  <p>First, private blockchains. Compared to public blockchains, they have a number of advantages:</p>
  <ol><li>The consortium or company running a private blockchain can easily, if desired, change the rules of a blockchain, revert transactions, modify balances, etc. In some cases, eg. national land registries, this functionality is necessary; there is no way a system would be allowed to exist where Dread Pirate Roberts can have legal ownership rights over a plainly visible piece of land, and so an attempt to create a government-uncontrollable land registry would in practice quickly devolve into one that is not recognized by the government itself. Of course, one can argue that one can do this on a public blockchain by giving the government a backdoor key to a contract; the counter-argument to that is that such an approach is essentially a Rube Goldbergian alternative to the more efficient route of having a private blockchain, although there is in turn a partial counter-argument to that that I will describe later.</li>
  <li>The validators are known, so any risk of a 51% attack arising from some miner collusion in China does not apply.</li>
  <li>Transactions are cheaper, since they only need to be verified by a few nodes that can be trusted to have very high processing power, and do not need to be verified by ten thousand laptops. This is a hugely important concern right now, as public blockchains tend to have transaction fees exceeding $0.01 per tx, but it is important to note that it may change in the long term with <a href="https://github.com/vbuterin/scalability_paper/raw/master/scalability.pdf">scalable blockchain technology</a> that promises to bring public-blockchain costs down to within one or two orders of magnitude of an optimally efficient private blockchain system</li>
  <li>Nodes can be trusted to be very well-connected, and faults can quickly be fixed by manual intervention, allowing the use of consensus algorithms which offer finality after much shorter block times. Improvements in public blockchain technology, such as Ethereum 1.0’s uncle concept and later proof of stake, can bring public blockchains much closer to the “instant confirmation” ideal (eg. offering total finality after 15 seconds, rather than 99.9999% finality after two hours as does Bitcoin), but even still private blockchains will always be faster and the latency difference will never disappear as unfortunately the speed of light does not increase by 2x every two years by Moore’s law.</li>
  <li>If read permissions are restricted, private blockchains can provide a greater level of, well, privacy.</li>
  </ol><p>Given all of this, it may seem like private blockchains are unquestionably a better choice for institutions. However, even in an institutional context, public blockchains still have a lot of value, and in fact this value lies to a substantial degree in the philosophical virtues that advocates of public blockchains have been promoting all along, among the chief of which are freedom, neutrality and openness. The advantages of public blockchains generally fall into two major categories:</p>
  <ol><li>Public blockchains provide a way to protect the users of an application from the developers, establishing that there are certain things that even the developers of an application have no authority to do. From a naive standpoint, it may be hard to understand why an application developer would want to voluntarily give up power and hamstring themselves. However, more advanced economic analysis provides two reasons why, in Thomas Schelling’s words, weakness can be a strength. First, if you explicitly make it harder or impossible for yourself to do certain things, then others will be more likely to trust you and engage in interactions with you, as they are confident that those things are less likely to happen to them. Second, if you personally are being coerced or pressured by another entity, then saying “I have no power to do this even if I wanted to” is an important bargaining chip, as it discourages that entity from trying to compel you to do it. A major category of pressure or coercion that application developers are at risk of is that by governments, so “censorship resistance” ties strongly into this kind of argument.</li>
  <li>Public blockchains are open, and therefore are likely to be used by very many entities and gain some network effects. To give a particular example, consider the case of domain name escrow. Currently, if A wants to sell a domain to B, there is the standard counterparty risk problem that needs to be resolved: if A sends first, B may not send the money, and if B sends first then A might not send the domain. To solve this problem, we have <a href="https://escrow.com/services/domain-name-holding-escrow.aspx">centralized escrow intermediaries</a>, but these charge <a href="https://escrow.com/support/fee-calculator.aspx">fees of three to six percent</a>. However, if we have a domain name system on a blockchain, and a currency on <em>the same blockchain</em>, then we can cut costs to near-zero with a smart contract: A can send the domain to a program which immediately sends it to the first person to send the program money, and the program is trusted because it runs on a public blockchain. Note that in order for this to work efficiently, two completely heterogeneous asset classes from completely different industries must be on the same database – not a situation which can easily happen with private ledgers. Another similar example in this category is land registries and title insurance, although it is important to note that another route to interoperability is to have a private chain that the public chain can verify, <a href="https://github.com/ethereum/btcrelay">btcrelay-style</a>, and perform transactions cross-chain.</li>
  </ol><p>In some cases, these advantages are unneeded, but in others they are quite powerful – powerful enough to be worth 3x longer confirmation times and paying $0.03 for a transaction (or, once scalability technology comes into play, $0.0003 for a transaction). Note that by creating privately administered smart contracts on public blockchains, or cross-chain exchange layers between public and private blockchains, one can achieve many kinds of hybrid combinations of these properties. The solution that is optimal for a particular industry depends very heavily on what your exact industry is. In some cases, public is clearly better; in others, some degree of private control is simply necessary. As is often the case in the real world, it depends.</p>
  															


title: Visa Incubator to Test Blockchain for Interbank Payments - CoinDesk
url: https://www.coindesk.com/visa-incubator-test-blockchain-interbank-payments/
created_at: 2017-08-10 23:19:33
updated_at: 2017-08-10 23:19:33
mimetype: text/html
language: en-US
domain_name: www.coindesk.com
preview_picture: https://media.coindesk.com/uploads/2016/09/Screen-Shot-2016-09-01-at-8.31.09-AM-e1472733137875.png
http_status: 200
content:
  <p>Visa’s London-based innovation lab has announced its latest blockchain trial will focus on interbank payments.</p>
  <p><a href="https://www.coindesk.com/tag/visa-europe/">Visa Europe Collab</a> revealed today it has partnered with Toronto-based BTL Group to explore how the startup’s interbank settlement system can leverage blockchain and smart contracts to streamline domestic and cross-border payments between banks.</p>
  <p>In statements, Hendrik Kleinsmiede, co-founder of Visa Europe Collab, said the two partners are now inviting European banks to participate in trials that would see them using the emerging technology for live payments to better explore its potential.</p>
  <p>Kleinsmiede said:</p>
  <blockquote><p>"We’ll work closely together on the development and implementation of the PoC, ensuring that all participants come away with new knowledge and insight into the role that the blockchain could play in interbank settlements in the future.”</p></blockquote>
  <p>According to <a href="http://www.ft.com/cms/s/0/483e315e-6fa6-11e6-a0c9-1365ce54b926.html#axzz4J0F7MT6Z"><em>Financial Times</em></a>, the partners will seek to conclude the project within 100 days.</p>
  <p>The trial is the latest that finds Visa Europe Collab exploring blockchain. In July, the firm announced a <a href="https://www.coindesk.com/visa-europe-bitcoin-micropayments/">micropayments trial</a> with bitcoin startup SatoshiPay, a move that followed its initial work on <a href="https://www.coindesk.com/tag/remittance/">remittances</a> with blockchain firm Epiphyte in 2015.</p>
  <p>One of the more novel proofs-of-concept to yet emerge, Visa Europe Collab’s work on this project found it exploring how blockchains could be used to extend the reach of traditional payment networks in developing countries.</p>
  <p>For more on that 2015 trial, read <a href="https://www.coindesk.com/visa-europe-remittances-bitcoin-blockchain/">our full report</a>.</p>
  <p><em>Image via Consensus 2016</em></p>

  			                            <p>
                      <em>The leader in blockchain news, CoinDesk is an independent media outlet that strives for the highest journalistic standards and abides by a <a href="https://www.coindesk.com/editorial-policy/">strict set of editorial policies</a>. Have breaking news or a story tip to send to our journalists? Contact us at <a href="https://www.coindesk.com/cdn-cgi/l/email-protection#355b50424675565a5c5b5150465e1b565a58" target="_top">[email protected]</a>.
                         </em>
                  </p>


title:
  Tweet from


  BlockChainTop

url: https://mobile.twitter.com/BlockChainTop/status/771837700882038784
created_at: 2017-08-10 23:19:34
updated_at: 2017-08-10 23:19:34
mimetype: text/html
language: en
domain_name: mobile.twitter.com
http_status: 200
content:
  <div class="dir-ltr" dir="ltr">  Visa will adopt blockchain – but what exactly is it? | Tec... <a href="https://t.co/SmqYSjKWOL" rel="nofollow noopener" dir="ltr" class="twitter_external_link dir-ltr tco-link" target="_blank" title="http://gettopical.com/blockchain/1a04d944e65bba81bd548c16fe5945a6?src=twitter">gettopical.com/blockchain/1a0…</a> via <a href="https://mobile.twitter.com/coindesk" class="twitter-atreply dir-ltr" dir="ltr">@coindesk</a> <a href="https://t.co/n4JSuIzMkM" rel="nofollow" dir="ltr" class="twitter_external_link dir-ltr tco-link has-expanded-path" target="_top">pic.twitter.com/n4JSuIzMkM</a>
  </div>


title:
  Tweet from


  BlockChainTop

url: https://mobile.twitter.com/BlockChainTop/status/771860479157297154
created_at: 2017-08-10 23:19:36
updated_at: 2017-08-10 23:19:36
mimetype: text/html
language: en
domain_name: mobile.twitter.com
http_status: 200
content:
  <div class="dir-ltr" dir="ltr">  What is blockchain technology, where is it used and how does it work? <a href="https://t.co/h8wKd7m54X" rel="nofollow noopener" dir="ltr" class="twitter_external_link dir-ltr tco-link" target="_blank" title="http://gettopical.com/blockchain/d7c38c460dfbb15398c4b27b63a681f?src=twitter">gettopical.com/blockchain/d7c…</a> via <a href="https://mobile.twitter.com/bitcoinagile" class="twitter-atreply dir-ltr" dir="ltr">@bitcoinagile</a>
  </div>


title:
  Tweet from


  BlockChainTop

url: https://mobile.twitter.com/BlockChainTop/status/772201551305650177
created_at: 2017-08-10 23:19:37
updated_at: 2017-08-10 23:19:37
mimetype: text/html
language: en
domain_name: mobile.twitter.com
http_status: 200
content:
  <div class="dir-ltr" dir="ltr">  What is Blockchain? – Innovation That Matters <a href="https://t.co/q2nOgMXo7b" rel="nofollow noopener" dir="ltr" class="twitter_external_link dir-ltr tco-link" target="_blank" title="http://gettopical.com/blockchain/c1bb750485db36c392921815b5c8fefa?src=twitter">gettopical.com/blockchain/c1b…</a> via <a href="https://mobile.twitter.com/danielselman" class="twitter-atreply dir-ltr" dir="ltr">@danielselman</a> <a href="https://t.co/SQskIxYgSb" rel="nofollow" dir="ltr" class="twitter_external_link dir-ltr tco-link has-expanded-path" target="_top">pic.twitter.com/SQskIxYgSb</a>
  </div>


title: Moving from Docker to rkt – Adriaan de Jonge – Medium
url: https://medium.com/@adriaandejonge/moving-from-docker-to-rkt-310dc9aec938
created_at: 2017-08-10 23:19:40
updated_at: 2017-08-10 23:19:40
mimetype: text/html
domain_name: medium.com
preview_picture: https://cdn-images-1.medium.com/max/1200/1*2J_pK3ZKJmePxCJsb1wSCQ.png
http_status: 200
content:
  <div class="section-divider"><hr class="section-divider" /></div><div class="section-content"><div class="section-inner sectionLayout--insetColumn"><h1 class="graf graf--h3 graf--leading graf--title">Moving from Docker to rkt</h1><p id="c6d2" class="graf graf--p graf--hasDropCapModel graf--hasDropCap graf-after--h3">Even the coolest products and services come with vendor lock-in. And no matter how enthusiastic I have been about Docker in the last three years, at some point this vendor lock-in starts to hurt. The good news is that competition is well on its way to becoming a viable alternative. Perhaps even a better alternative in some regards. This article takes a look at <a href="https://coreos.com/" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">CoreOS</a>’s <a href="https://github.com/coreos/rkt" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">rkt</a> <em class="markup--em markup--p-em">(pronounced: “rock-it”)</em> and why you need to start investigating rkt <em class="markup--em markup--p-em">now</em>…</p><h3 class="graf graf--h3 graf-after--p">What is wrong with Docker?</h3><p class="graf graf--p graf-after--h3">If you are anything like me, you may still be so enthusiastic about Docker that you’re blindsided to some of its flaws. Don’t get me wrong, I am not saying Docker is bad. But it is not perfect either. So let’s take a look at some of these flaws.</p><h4 class="graf graf--h4 graf-after--p">Docker starts behaving like The Old Microsoft</h4><p id="b434" class="graf graf--p graf-after--h4">Once companies realize they have a monopoly, they start behaving like monopolists. Just like Microsoft once practically killed Netscape by including Internet Explorer with Windows, now Docker is trying to defeat Kubernetes, Mesos/Marathon and Nomad by including Swarm into the Docker Core.</p><p id="fe05" class="graf graf--p graf-after--p">Docker deserves credit for simplifying Swarm and making it easily accessible to the average Docker user. Just as much as Microsoft deserves credit for the improvements they introduced in Internet Explorer 6.0 (True story!). The biggest issue in MSIE 6 is that it made Microsoft so dominant that they stopped innovating for five years.</p><p class="graf graf--p graf-after--p">Please realize the consequence of including Swarm in the core of Docker… Imagine your next assignment for a client that wants to move to Docker: When it comes to selecting the best scheduler, you first have to argue why Swarm is insufficient, before you can even start talking about Kubernetes, Mesos/Marathon or Nomad. Swarm is already there anyway so why would you need to install another scheduler on top of it? Guilty until proven innocent — a reversed burden of proof.</p><p id="d636" class="graf graf--p graf-after--p">I hope Docker will do a better job improving Swarm than Microsoft did with MSIE. But even now, Kubernetes has a head start on Swarm in many ways. And as Kubernetes keeps developing, Swarm is falling further behind. It is good Docker created an easy to use scheduler for Docker but in my opinion, they should have kept it separate from the Docker core.</p><h4 class="graf graf--h4 graf--startsWithDoubleQuote graf-after--p">“Docker’s architecture is fundamentally flawed”</h4><p class="graf graf--p graf-after--h4">At the heart of Docker is a daemon process that is the starting point of everything Docker does. The docker executable is merely a REST client that requests the Docker daemon to do its work. Critics of Docker say this is not very Linux-like.</p><p class="graf graf--p graf-after--p">Where it starts hurting is if you use an init system like systemd. Since systemd was not designed for Docker specifically, when you try to start a Docker process, you actually start a Docker client process that in turn requests the Docker daemon to start the actual Docker container. There is a risk that the Docker client fails while the actual Docker container keeps running. In such a situation, systemd concludes that the process has stopped and restarts the Docker client — in turn possibly creating a second container <em class="markup--em markup--p-em">(yes, you can work around this but that is besides the point)</em>.</p><p id="ec50" class="graf graf--p graf-after--p">About a year ago, I was investigating systemd as a simple Docker scheduler. I ran into the same risks of combining Docker with systemd. Back then, I considered these as downsides of systemd rather than downsides of Docker. In my view back then, systemd was not designed to be “Docker-native”.</p><p class="graf graf--p graf-after--p">Now, I learned that perhaps Docker is the issue rather than systemd. As mentioned, Docker is not very Linux-like and because of this generic Linux tools do not play nice with Docker. Who is at fault, relative newcomer Docker or the Linux tools that have been around for years? Some say that “Docker’s architecture is fundamentally flawed” — this is a statement of Alex Polvi, CEO of CoreOS.</p><h4 id="dcda" class="graf graf--h4 graf-after--p">The Docker build process is stuck in second gear</h4><p class="graf graf--p graf-after--h4">One of the nice features of Docker was the introduction of Dockerfiles that you can maintain in version control to reproduce Docker images. The only issue here is that the Dockerfile syntax has been frozen in Docker’s roadmap for a long time now. This means that the Dockerfile format has not evolved with the insights in the Docker community for at least a year and a half.</p><p id="e40b" class="graf graf--p graf-after--p">Of course, at some point we need a stable format. But only when the format has fully matured. Just to give you an example where Dockerfiles are lacking, consider Kelsey Hightower’s statement in his blog <a href="https://medium.com/@kelseyhightower/12-fractured-apps-1080c73d481c#.smga9216i" class="markup--anchor markup--p-anchor" target="_blank">12 Fractured Apps</a>:</p><blockquote class="graf graf--blockquote graf--startsWithDoubleQuote graf-after--p"><div>“Remember, ship artifacts not build environments.”</div></blockquote><p class="graf graf--p graf-after--blockquote">Fact is that the Dockerfile format does not support this separation very well. Consider for example building a Golang executable: the build environment is several hundreds of megabytes while the resulting executable may be as small as a couple of megabytes only. Sure, you can work around this by building the image on your local system. But you cannot elegantly build a small Golang image from a single Dockerfile using an Automated Build in Docker Hub. <a href="https://github.com/docker/docker/issues/7115" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">Issues proposing to extend the Dockerfile format</a> to better support this principle have been on hold for years because of the freeze.</p><p class="graf graf--p graf-after--p"><strong class="markup--strong markup--p-strong"><em class="markup--em markup--p-em">Update (11 May 2017):</em></strong><em class="markup--em markup--p-em"> The above is FIXED with Multi-Stage Builds. See my new blog “</em><a href="https://medium.com/@adriaandejonge/simplify-the-smallest-possible-docker-image-62c0e0d342ef" class="markup--anchor markup--p-anchor" target="_blank"><em class="markup--em markup--p-em">Simplify the Smallest Possible Docker Image</em></a><em class="markup--em markup--p-em">”</em></p><h3 id="f38c" class="graf graf--h3 graf-after--p">How does rkt improve the situation?</h3><p class="graf graf--p graf-after--h3">The short answer is that rkt now provides a viable alternative to Docker. It has a more Linux-like architecture. And a strong competitor will keep the monopolist sharp.</p><p class="graf graf--p graf-after--p">The long answer:</p><p class="graf graf--p graf-after--p">Late 2014, CoreOS announced Rocket — later abbreviated and renamed as rkt — as a competing container platform. While rkt got a lot of attention in the first weeks after the announcement, it became silent for some time. CoreOS continued developing rkt into a viable alternative to Docker and only came back in the spotlight with the release of rkt 1.0 in February 2016.</p><p class="graf graf--p graf-after--p">Now that Docker announced its inclusion of Swarm into Docker Engine 1.12, it is time to start looking seriously at rkt as an alternative to Docker. Can it replace Docker now or in future? Is it difficult to switch between Docker and rkt?</p><p id="f267" class="graf graf--p graf-after--p">Let’s take a look at rkt and find out…</p><h4 id="b598" class="graf graf--h4 graf-after--p">rkt can run Docker images</h4><p id="ce6f" class="graf graf--p graf-after--h4">Consider you want to replace your staging and production systems with rkt while keeping all your development systems as is… In this case you can replace Docker with rkt on your runtime systems only. It is not precisely a <em class="markup--em markup--p-em">drop-in</em> replacement. After all, the architecture is different — but we’ll get to that. On the other hand, it is not too difficult to learn the new command line instructions for running a Docker container on rkt.</p><p class="graf graf--p graf-after--p">Open a CoreOS instance on your favorite cloud provider and type:</p><pre class="graf graf--pre graf-after--p">rkt run --insecure-options=image --port=80-tcp:80 docker://nginx</pre><p class="graf graf--p graf-after--pre">or replace nginx with your favorite Docker image. Under the hood, rkt converts the Docker image to Application Container (appc) format.</p><p id="d269" class="graf graf--p graf-after--p">This means that you don’t need Docker to run Docker images! And it means that you can reuse anything you created with Docker without the least bit of migration pain — other than learning rkt’s command line syntax.</p><p id="b5ad" class="graf graf--p graf-after--p">Let’s take a look at the individual parts of this command:</p><pre class="graf graf--pre graf-after--p">--insecure-options=image</pre><p class="graf graf--p graf-after--pre">If you leave this part out, rkt will refuse to start your image as it cannot find a signature (.asc file) to check the integrity of the Docker image. rkt is built secure by default. In this case, it means that you can save yourself some typing on the command line by providing the proper signatures.</p><pre id="eb7a" class="graf graf--pre graf-after--p">--port=80-tcp:80</pre><p class="graf graf--p graf-after--pre">Just like in Docker, you need to explicitly expose a port to the outside world. Unlike Docker, rkt ports are <em class="markup--em markup--p-em">named</em> rather than numbered. If this were a native rkt image (or appc image to be precise) it would have probably read:</p><pre class="graf graf--pre graf-after--p">--port=http:80</pre><p class="graf graf--p graf-after--pre">However, since Docker does not have a concept of named ports, the exposed ports are automatically named as <em class="markup--em markup--p-em">&lt;number&gt;-&lt;protocol&gt;</em>. This also means that if you can only expose ports that have been explicitly specified using the EXPOSE keyword in the Dockerfile.</p><pre id="ecdf" class="graf graf--pre graf-after--p">docker://nginx</pre><p class="graf graf--p graf-after--pre">The image is expected in the default Docker repository (Docker Hub). Instead of this example, there can be longer URLs pointing to either alternative Docker repositories (docker://…) or to generic websites containing appc images (https://…) but then you are not running Docker images anymore.</p><h4 id="cfa4" class="graf graf--h4 graf-after--p">rkt has a simpler architecture</h4><p class="graf graf--p graf-after--h4">In rkt, there is no daemon process. The rkt command line tool does all the work. See the below picture <a href="https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">borrowed from the CoreOS website</a>:</p><div class="aspectRatioPlaceholder is-locked"><div class="aspectRatioPlaceholder-fill"><img class="graf-image" src="https://cdn-images-1.medium.com/max/1600/1*2J_pK3ZKJmePxCJsb1wSCQ.png" alt="image" /></div>Process model for rkt vs Docker<p id="b1a6" class="graf graf--p graf-after--figure">Unlike Docker, this means that if you use systemd to start rkt containers, you are actually monitoring the container process rather than monitoring a client process that connects to a daemon that in turn (directly or indirectly — depending on your Docker version) starts the container.</p><p class="graf graf--p graf-after--p">On the flip side of this, you cannot type</p><pre class="graf graf--pre graf-after--p">rkt run -d    ...etc</pre><p id="ddc9" class="graf graf--p graf-after--pre">and daemonize the process like the Docker client. Instead, you’d have to run an init system to daemonize the process. For example, you can run this:</p><pre id="c796" class="graf graf--pre graf-after--p">systemd-run --slice=machine rkt run   ...etc</pre><p class="graf graf--p graf-after--pre">Also, you cannot run the rkt command from a remote machine like the docker client. On the plus side, you can consider this as a security feature as well.</p><h4 id="d53e" class="graf graf--h4 graf-after--p">rkt follows an open standard for images</h4><p class="graf graf--p graf-after--h4">Right now, rkt allows you to use Application Container (appc) or Docker images. In the near future, the Open Container Initiative (OCI) format will be added to this, but we’ll come to that.</p><p class="graf graf--p graf-after--p">The advantage of having an open standard for container images is that it allows the open source community to provide multiple ways of building images. So you are not tied to the Dockerfile format only.</p><p id="f20d" class="graf graf--p graf-after--p">The default way to build an appc image is using a command line tool called <a href="https://github.com/appc/acbuild" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">acbuild</a>. Honestly, it is a matter of taste whether you’d prefer acbuild over the Dockerfile format. The advantage is that it stays closer to Linux principles by nature.</p><p id="c530" class="graf graf--p graf-after--p">The bigger advantage is that the open format allows for alternative build mechanisms. For example, consider the all-round build tool <a href="https://github.com/blablacar/dgr" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">dgr</a> or the Golang specific build tool <a href="https://github.com/appc/goaci" class="markup--anchor markup--p-anchor" rel="nofollow noopener" target="_blank">goaci</a>.</p><p class="graf graf--p graf-after--p">Once the OCI format comes available, there will be even more possibilities but at the time of writing this article, we’ll have to wait for this.</p><h3 class="graf graf--h3 graf-after--p">rkt: Are we there yet?</h3><p id="d141" class="graf graf--p graf-after--h3">If you want to start using rkt for real <em class="markup--em markup--p-em">right now</em>, there are still a few bumps in the road. Although you can navigate around these, it must be said that at the time of writing this article it may still be early to go all-in with rkt. But that should be a temporary problem.</p><h4 class="graf graf--h4 graf-after--p">The OCI image format is not ready yet</h4><p class="graf graf--p graf-after--h4">As mentioned, rkt supports the Docker image format and can interact with Docker repositories. If you are okay to stick with the Docker image format, it’ll work fine.</p><p class="graf graf--p graf-after--p">However, if you are a bit of a purist — like me — you may not appreciate the fact that you cannot use named ports yet. So you’ll want to use appc containers. And you <em class="markup--em markup--p-em">can</em> use appc containers.</p><p id="df2a" class="graf graf--p graf-after--p">But how do you upload your appc container to the CoreOS alternative to Docker Hub — quay.io? Personally I haven’t found a way to do so…</p><p id="f1ad" class="graf graf--p graf-after--p">The nice thing about the appc format is that you are not tied to a particular repository like Docker. Instead, you have the possibility to host the image on a regular http-server or local file system. You can enrich the meta data in HTML files containing meta-tags.</p><p id="d972" class="graf graf--p graf-after--p">Still, rkt can only really take off once the OCI image format becomes sufficiently mature and well-adopted to be accepted as standard by everyone — including Docker.</p><h4 class="graf graf--h4 graf-after--p">Nomad &amp; K8S support not fully mature</h4><p id="f096" class="graf graf--p graf-after--h4">In order to run containers in production, at some point you need a scheduler to control what runs where. Both Kubernetes and Nomad have support for rkt already. The catch for now is that the support is not yet as mature as you might hope.</p><p class="graf graf--p graf-after--p">The Kubernetes support for rkt is labeled <em class="markup--em markup--p-em">under active development</em>. It has minimal documentation and a list of known issues. The Nomad support is labeled <em class="markup--em markup--p-em">experimental </em>and does not support dynamic ports.</p><h4 id="f458" class="graf graf--h4 graf-after--p">A bit less portable to other platforms</h4><p class="graf graf--p graf-after--h4">The good news is that rkt is not only for CoreOS. You can install rkt on multiple well-known Linux distributions, including Debian, Ubuntu and Fedora.</p><p id="d43c" class="graf graf--p graf-after--p">If you want to run rkt on your Apple/Windows development machine, you can of course do so running on top of virtualization layers like VirtualBox. However, the facilities are not yet as nice as Docker’s toolbox — especially the latest beta versions of Docker for Mac/Windows which gives you a native-like feel. Admitted, this may be a slight downside to rkt’s architecture where the rkt executable does all the work rather than being a thin REST client.</p><p id="a141" class="graf graf--p graf-after--p">If you want to do your development on non-Linux machines, the best way is still to work with Docker images locally and convert these to appc images when you go to staging and production.</p><h3 id="a225" class="graf graf--h3 graf-after--p">Conclusion</h3><p id="deb6" class="graf graf--p graf-after--h3">Although it is still early, rkt has now become a viable alternative to Docker. If you don’t need all the dynamic features of Kubernetes and Nomad and the more static options like systemd and fleet sufficiently meet your scheduling requirements then you can already move your staging and production servers to rkt right now.</p><p class="graf graf--p graf-after--p">Give it a little more time and there will be true interoperability between Docker and other container platforms in the form of OCI images. At the same time, allow Kubernetes and Nomad support for rkt to evolve a bit and then the Docker and rkt container platforms will be as good as interchangeable.</p><p id="c829" class="graf graf--p graf-after--p graf--trailing">Is there a strong need to abandon Docker as fast as we can? No, not really. Despite the flaws mentioned in this article, Docker is still an innovative platform with a large ecosystem of development tools, schedulers and orchestrators. The important thing is to have more than one option to choose from. And now Docker has a serious competitor to keep them sharp!</p></div></div></div>


title: How blockchain will disrupt your business
url: http://www.cio.com/article/3115776/internet/how-blockchain-will-disrupt-your-business.html
created_at: 2017-08-10 23:19:43
updated_at: 2017-08-10 23:19:43
mimetype: text/html
language: en
domain_name: www.cio.com
http_status: 200
content:
  <p>
  Like mobile and cloud, blockchain — first implemented in the original source code of bitcoin in 2009 — stands poised to profoundly disrupt business. If it lives up to its promise, it won't just be financial institutions that are disrupted.
  </p><p>
  "If you can transfer money or something of value through the internet just like another form of data, what else can you do with it? It provides a way to establish trust in the digital world," says Angus Champion de Crespigny, Financial Services Blockchain and Distributed Infrastructure Strategy Leader, Ernst &amp; Young. "How do you ensure something is the original copy of something on the internet? Prior to blockchain technology, you couldn't."
  </p><p>
  "If you want to prove something happened in the digital world, there is no more secure place to do that," he adds. "Once information is recorded on there, it is, for all intents and purposes, impossible to go back and retroactively change that. When there are such drastic new technologies that emerge, it isn't just a matter of looking at your business and thinking how this technology is going to make your business more effective. What you should be doing is considering that maybe your business isn't structured correctly for this new world."
  </p><h2>History repeats itself with blockchain?</h2><p>
  In the early days of the Italian Renaissance, Hindu-Arabic numerals from the east and a system of accounting used by the Italian mercantile houses of the various northern Italian republics came together to disrupt the very foundation of business. With the birth of modern double-entry bookkeeping, merchants were able to ensure the integrity of the financial values recorded in their ledgers. Today the system is used in every business.
  </p><p>
  In some ways, blockchain is a modern form of digital double-entry bookkeeping, but on steroids. Don Tapscott and son Alex Tapscott, authors of <em>Blockchain Revolution: How the Technology Behind Bitcoin Is Changing Money, Business, And the World</em>, say blockchain represents the second generation of the internet, with "the potential to transform money, business, government and society.
  </p><p>
  In <a href="http://time.com/4320254/blockchain-tech-behind-bitcoin/" target="_blank" title="How the Tech Behind Bitcoin Will Change Your Life">an article for <em>Time</em></a> they describe blockchain as "the first native digital medium for peer-to-peer value exchange. Its protocol establishes the rules — in the form of globally distributed computations and heavy duty encryption — that ensure the integrity of the data traded among billions of devices without going through a trusted third party. Trust is hard-coded into the platform. That's why we call it the Trust Protocol. It acts as a ledger of accounts, a database, a notary, a sentry and clearing house, all by consensus."
  </p><h2>Understanding blockchain</h2><p>
  In essence, blockchain is a distributed database, or "timestamp server," as it was called by the mysterious Satoshi Nakamoto in the <a href="https://bitcoin.org/bitcoin.pdf" target="_blank" title="Bitcoin: A Peer-to-Peer Electronic Cash System">paper that proposed bitcoin</a>. The blockchain consists of blocks of items — each block is a timestamped batch of valid individual transactions and the hash of the previous block, creating a link between the two. Because each timestamp includes the previous timestamp in its hash, it forms a chain. Each new transaction must be authenticated across the distributed network of computers that form the blockchain before it can form the next block in the chain.
  </p><p>
  "Consensus in a distributed system is determined by entities checking each other's work and providing a stamp of approval as to transactions and activities allowed," Champion de Crespigny explains.
  </p><p>
  Blockchain also leverages a technology called "smart contracts," which are bits of executable code that only act when specific conditions within the blockchain are met. This allows a blockchain to automate activity like payment transfers when a task is completed, or even a partial payment when a milestone is achieved.
  </p><p>
  By providing a way to record transactions as automated trusted activity among digitally networked peers, audit and professional services firm <a href="http://www.ey.com/GL/en/Industries/Technology/ey-blockchain-reaction-tech-plans-for-critical-mass" target="_blank" title="Blockchain Reaction: Tech Plans for Critical Mass">Ernst &amp; Young believes</a> "blockchain technology has the potential to streamline and accelerate business processes, increase cybersecurity and reduce or eliminate the roles of trusted intermediaries (or centralized authorities) in industry after industry."
  </p><p>
  To be sure, there are challenges: technical challenges, the shadow of the public's perception of bitcoin, tax and regulatory questions, not to mention resistance from the very businesses that stand to be disintermediated. But Champion de Crespigny warns that a "wait-and-see" approach, could leave your business in the same position as the record executives who spent their time thinking about how to use the internet to sell more CDs.
  </p><p>
  "The technical innovation will be extremely powerful and will change how business models operate," he says.
  </p><p>
  Ernst &amp; Young cites one early experiment: A New York City neighborhood that has set up a private blockchain that helps homeowners share solar power generated on their rooftops without the local utility's involvement. There are other pilots, too: Internet of things (IoT)-connected washing machines that order their own detergent when supplies run low and autonomous agriculture sensors that control water flow in fields.
  </p><p>
  If the challenges can be overcome, Champion de Crespigny says blockchain adoption will be extremely rapid and extremely disruptive. For instance, Ernst &amp; Young notes that if blockchains can prove they reduce cost and increase trust in financial transactions, we can expect financial services firms to abandon existing transaction-processing technologies in favor of blockchain technologies. Software and services incumbents that aren't prepared may be left in the dust. And the end result of the disruption could be even more significant.
  </p><p>
  "What is very interesting to me is the idea of value just becoming another form of data and allowing finance or payments to move from what is currently an industry vertical to a horizontal that goes across other industries," Champion de Crespigny says.
  </p><p>
  Blockchain also promises to be an especially good mechanism for distributing computing workloads, which would affect infrastructure as a service (IaaS) providers and providers of other cloud services. The technology could also rewrite the book on cybersecurity when it comes to authentication, and could form the trusted basis for all sorts of machine-to-machine interactions in IoT.
  </p><p>
  "This is not an area where you can just offload to technology," Champion de Crespigny says. "You need to understand conceptually what this is and what impact it could have on your business. You need to challenge your current business models and processes. Are you doing this this way because you've always done it this way, or are there better ways to do it?"
  </p><h2>The next step in consumption-based pricing</h2><p>
  The cloud has already led many companies to start migrating from unit-based pricing models to consumption-based models, and IoT looks to be building on that trend. The addition of blockchain technologies to the mix may further accelerate the migration, according to Greg Cudahy, Global Leader for Telecommunications, Media and Technology at Ernst &amp; Young.
  </p><p>
  "Consumption-based pricing will clearly be a business process disruptor," Cudahy says. "Eventually, maybe even semiconductor companies won't be paid on the basis of the cost of a chip, but will bill customers differently based on how often and how much the customer uses that capacity."
  </p><p>
  At the same time, the technology has the potential to completely upend the way companies approach accounting and regulatory compliance, reducing the cost of tax compliance, reducing or eliminating tax evasion and exposing corporate information to more public scrutiny.
  </p><p>
  "Today, the processes of validating that the revenue was earned and then paying the appropriate tax on that revenue are done retroactively," says Channing Flynn, Ernst &amp; Young Global Technology Sector Leader, Tax Services. "But blockchain technology may be able to do it in real time, openly and transparently."
  </p><p>
  Other potential scenarios identified by Ernst &amp; Young include the following:
  </p><ul><li><strong>Embedded health.</strong> An entire healthcare information ecosystem could exist as a blockchain that connects insurers, providers and patients.</li>
  <li><strong>Eliminating digital rights theft.</strong> The use of a music file, for instance, could be recorded by a public blockchain. Artists could release their music in a blockchain-based music ecosystem and control the data and terms of use, with royalties distributed in real time via smart contracts.</li>
  <li><strong>New credit markets for low-cost assets.</strong> Ernst &amp; Young notes that transaction costs put a practical lower limit on the use of assets as collateral to secure a loan. By taking much of the cost out of transactions, blockchain could make lending based on new classes of lower-value assets practical.</li>
  <li><strong>Pay-for-performance.</strong> Leveraging smart contracts, blockchain can automatically enforce pay-for-performance agreements. The days of getting a pizza in 30 minutes or less or it's free may be long gone, but blockchain could enable its return. It could even enable a sliding scale.</li>
  <li><strong>Government tax enforcement.</strong> In a world of digital transactions recorded in real time, illegal transactions become extremely difficult to conceal.</li>
  <li><strong>Industrial mash-ups.</strong> Blockchain could enable a whole new world of fluid partnerships between businesses. Ernst &amp; Young calls it an "industrial mash-up," by which it means alliances in which one or more parties make use of assets or capabilities of another party to create new business value, without affecting the other party's ongoing use of the assets or capabilities for their original business purpose.</li>
  <li><strong>Industrial IoT.</strong> Bringing together industrial mash-ups and IoT, Ernst &amp; Young believes blockchain technology could help companies better utilize their high-value industrial assets by connecting everything from shipping containers to MRI machines to construction equipment into real-time digital marketplaces that would allow businesses to sell and buy idle time.</li>
  </ul><p>
  "It's time to think through, in a logical, strategic way, how to effectively disrupt yourself before others do it to you," Champion de Crespigny says. "Companies should identify new blockchain-enabled opportunities, assess the risks and time their entries to capture and build competitive positions."
  </p>

preview_picture:
  https://images.techhive.com/images/article/2016/09/understanding_blockchain-100680915-primary.idge.jpg


title: Phpseclib: Securely Communicating with Remote Servers via PHP — SitePoint
url: https://www.sitepoint.com/phpseclib-securely-communicating-with-remote-servers-via-php/
created_at: 2017-08-10 23:19:46
updated_at: 2017-08-10 23:19:46
mimetype: text/html
language: en
domain_name: www.sitepoint.com
preview_picture: https://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2016/10/1475494409272063.jpg
http_status: 200
content:
  <p>PHP has an <a href="http://php.net/manual/en/book.ssh2.php">SSH2</a> library which provides access to resources (shell, remote exec, tunneling, file transfer) on a remote machine using a secure cryptographic transport. Objectively, it is a tedious and highly frustrating task for a developer to implement it due to its overwhelming configuration options and complex API with little documentation. </p>
  <p><img alt="Connection between client and server" src="https://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2016/10/1475494409272063.jpg" width="450" /></p>
  <p>The <a href="http://phpseclib.sourceforge.net/ssh/intro.html">phpseclib</a> (<strong>PHP</strong> <strong>S</strong>ecure <strong>C</strong>ommunications <strong>Lib</strong>rary) package has a developer friendly API. It uses some optional PHP extensions if they’re available and falls back on an internal PHP implementation otherwise. To use this package, you don’t need any non-default PHP extensions installed.</p>
  <h2 id="installation">Installation</h2>
  <pre class="bash language-bash">composer require phpseclib/phpseclib&#13;
  </pre>
  <p>This will install the most recent stable version of the library via <a href="https://www.sitepoint.com/php-dependency-management-with-composer/">Composer</a>.</p>
  <h2 id="usecases">Use-cases</h2>
  <p>Before diving in blindly, I’d like to list some use-cases appropriate for using this library:</p>
  <ol><li>Executing deployment scripts on a remote server</li>
  <li>Downloading and uploading files via SFTP</li>
  <li>Generating SSH keys dynamically in an application</li>
  <li>Displaying live output for remote commands executed on a server</li>
  <li>Testing an SSH or SFTP connection</li>
  </ol><h2 id="connectingtotheremoteserver">Connecting to the Remote Server</h2>
  <p>Using <a href="http://phpseclib.sourceforge.net/ssh/intro.html">phpseclib</a>, you can connect to your remote server with any of the following authentication methods:</p>
  <ol><li>RSA key</li>
  <li>Password Protected RSA key</li>
  <li>Username and Password (<strong><em>Not recommended</em></strong>)</li>
  </ol><h3 id="rsakey">RSA Key</h3>
  <p>We will assume that you have a secure <strong>RSA</strong> key already generated. If you are not familiar with generating a secure RSA key pair, you can go through <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2">this article</a>. For a video explanation, you can refer to <a href="https://serversforhackers.com/video/creating-and-using-ssh-keys">Creating and Using SSH Keys</a> from Servers For Hackers.</p>
  <p>To log in to a remote server using <strong>RSA</strong> key authentication:</p>
  <pre class="php language-php">namespace App;&#13;
  &#13;
  use phpseclib\Crypt\RSA;&#13;
  use phpseclib\Net\SSH2;&#13;
  &#13;
  $key = new RSA();&#13;
  $key-&gt;loadKey(file_get_contents('privatekey'));&#13;
  &#13;
  //Remote server's ip address or hostname&#13;
  $ssh = new SSH2('192.168.0.1');&#13;
  &#13;
  if (!$ssh-&gt;login('username', $key)) {&#13;
      exit('Login Failed');&#13;
  }&#13;
  </pre>
  <h3 id="passwordprotectedrsakey">Password Protected RSA Key</h3>
  <p>If your RSA keys are password protected, do not worry. <a href="http://phpseclib.sourceforge.net/ssh/intro.html">Phpseclib</a> takes care of this particular use case:</p>
  <pre class="php language-php">namespace App;&#13;
  &#13;
  use phpseclib\Crypt\RSA;&#13;
  use phpseclib\Net\SSH2;&#13;
  &#13;
  $key = new RSA();&#13;
  $key-&gt;setPassword('your-secure-password');&#13;
  $key-&gt;loadKey(file_get_contents('privatekey'));&#13;
  &#13;
  //Remote server's ip address or hostname&#13;
  $ssh = new SSH2('192.168.0.1');&#13;
  &#13;
  if (!$ssh-&gt;login('username', $key)) {&#13;
      exit('Login Failed');&#13;
  }&#13;
  </pre>
  <h3 id="usernameandpassword">Username and Password</h3>
  <p>Alternatively, to log in to your remote server using a username and password (we don’t recommend this practice):</p>
  <pre class="php language-php">namespace App;&#13;
  &#13;
  use phpseclib\Net\SSH2;&#13;
  &#13;
  //Remote server's ip address or hostname&#13;
  $ssh = new SSH2('192.168.0.1');&#13;
  &#13;
  if (!$ssh-&gt;login('username', 'password')) {&#13;
      exit('Login Failed');&#13;
  }&#13;
  </pre>
  <p>For other options such as No Authentication or Multi-Factor authentication please <a href="http://phpseclib.sourceforge.net/ssh/auth.html#rsakey">refer to the documentation</a></p>
  <h2 id="executingcommandsontheremoteserver">Executing Commands on the Remote Server</h2>
  <p>The code to execute commands on a remote server is pretty simple. You call the <code>$ssh-&gt;exec($cmd)</code> method with the command to execute as the parameter.</p>
  <pre class="php language-php">namespace App;&#13;
  &#13;
  use phpseclib\Crypt\RSA;&#13;
  use phpseclib\Net\SSH2;&#13;
  &#13;
  $key = new RSA();&#13;
  $key-&gt;loadKey(file_get_contents('privatekey'));&#13;
  &#13;
  //Remote server's ip address or hostname&#13;
  $ssh = new SSH2('192.168.0.1');&#13;
  &#13;
  if (!$ssh-&gt;login('username', $key)) {&#13;
      exit('Login Failed');&#13;
  }&#13;
  &#13;
  $ssh-&gt;exec('ls -la');&#13;
  </pre>
  <h3 id="executingmultiplecommandsonremoteserver">Executing Multiple Commands on Remote Server</h3>
  <p>In real life applications, we rarely execute a single command. We often need to traverse around the server using <code>cd</code> and execute many other commands. If you try to execute multiple commands on the remote server as below, it won’t give you the desired output:</p>
  <pre class="php language-php">$ssh-&gt;exec('pwd'); //outputs /home/username&#13;
  &#13;
  $ssh-&gt;exec('cd mysite.com');&#13;
  &#13;
  $ssh-&gt;exec('pwd'); //outputs /home/username&#13;
  </pre>
  <p>The reason for above is that a call to the <code>exec</code> method does not carry state forward to the next <code>exec</code> call. To execute multiple commands without losing state:</p>
  <pre class="php language-php">$ssh-&gt;exec('cd /home/username; ls -la'); //Lists all files at /home/username&#13;
  </pre>
  <p>You can append as many commands as you wish with a semicolon or new line character <code>PHP_EOL</code>.</p>
  <p>For example, if you want to run a full deployment script for Laravel:</p>
  <pre class="php language-php">$ssh-&gt;exec(&#13;
        "git pull origin master" . PHP_EOL&#13;
          . "composer install --no-interaction --no-dev --prefer-dist" . PHP_EOL&#13;
          . "composer dump-autoload -o" . PHP_EOL&#13;
          . "php artisan optimize" . PHP_EOL&#13;
          . "php artisan migrate --force"&#13;
  );&#13;
  </pre>
  <h3 id="exitingonfirsterror">Exiting on First Error</h3>
  <p>In the above script, the whole set of commands is executed as a single shell script. Every command will be executed, even if some of them fail, just like in a regular shell script. As a default, this is fine, but if we need to exit on the first error, we have to alter our bash script. This is not something specific to <a href="http://phpseclib.sourceforge.net/ssh/intro.html">phpseclib</a>, it is related to bash scripting. </p>
  <p>If you put a <code>set -e</code> option at the beginning of the script, the script will terminate as soon as any command in the chain returns a non-zero value.</p>
  <p>For example, the modified version of the above deployment script would be</p>
  <pre class="php language-php">$ssh-&gt;exec(&#13;
      "set -e" . PHP_EOL&#13;
          . "git pull origin master" . PHP_EOL &#13;
          . "composer install --no-interaction --no-dev --prefer-dist" . PHP_EOL&#13;
          . "composer dump-autoload -o" . PHP_EOL&#13;
          . "php artisan optimize" . PHP_EOL&#13;
          . "php artisan migrate --force"&#13;
  );&#13;
  </pre>
  <p>The above script will terminate if any of the commands results in an error.</p>
  <div class="widget maestro maestro-content-type-html hide-for-tablet-SP hide-for-mobile-SP" id="maestro-636"><p>--ADVERTISEMENT--</p>
  </div><h2 id="gatheringoutput">Gathering Output</h2>
  <p>The <code>exec</code> method returns the output of your remote script:</p>
  <pre class="php language-php">$output = $ssh-&gt;exec('ls -la');&#13;
  echo $output;&#13;
  </pre>
  <p>Sometimes, however, it does not return the whole output. You can overcome this by passing a closure as a second argument to the <code>exec</code> method to make sure that any uncaught output will also be returned.</p>
  <pre class="php language-php">$ssh-&gt;exec(&#13;
      $deployment_script, function ($str) {&#13;
          $output .= $str;&#13;
  });&#13;
  &#13;
  echo $output;&#13;
  </pre>
  <p><em>Note: Error output, if any, will also be returned by the <code>exec</code> method or the underlying closure.</em></p>
  <h3 id="displayingliveoutput">Displaying Live Output</h3>
  <p>If you want to execute the script via console commands and display live output, you can achieve this by echoing the output in the underlying closure.</p>
  <pre class="php language-php">$ssh-&gt;exec($deployment_script, function ($str) {&#13;
      echo $str;&#13;
  });&#13;
  </pre>
  <p>If you want to display it in a web browser, you need to flush (send) the output buffer with <code>ob_flush()</code>.</p>
  <pre class="php language-php">$ssh-&gt;exec(&#13;
      $deployment_script, function ($str) {&#13;
          echo $str;&#13;
          flush();&#13;
          ob_flush();&#13;
  });&#13;
  </pre>
  <h2 id="otherconfigurationoptions">Other Configuration Options</h2>
  <p>It’s also possible to set many other available configuration options. You can call them as <code>$ssh-&gt;{option}</code>. </p>
  <p>For example: <code>$ssh-&gt;setTimeout(100)</code>.</p>
  <p>All the options we haven’t covered yet are in the table below:</p>
  <table><thead><tr><th>Option</th>
  <th>Use case</th>
  </tr></thead><tbody><tr><td><code>setTimeout($seconds)</code></td>
  <td><code>$ssh-&gt;exec('ping 127.0.0.1');</code> on a Linux host will never return and will run indefinitely. <code>setTimeout()</code> makes it so it’ll timeout. Setting <code>$timeout</code> to false or 0 will mean there is no timeout.</td>
  </tr><tr><td><code>write($cmd)</code></td>
  <td>Inputs a command into an interactive shell.</td>
  </tr><tr><td><code>read()</code></td>
  <td>Returns the output of an interactive shell</td>
  </tr><tr><td><code>isTimeout()</code></td>
  <td>Return true if the result of the last <code>$ssh-&gt;read()</code> or <code>$ssh-&gt;exec()</code> was due to a timeout. Otherwise it will return false.</td>
  </tr><tr><td><code>isConnected()</code></td>
  <td>Returns true if the connection is still active</td>
  </tr><tr><td><code>enableQuietMode()</code></td>
  <td>Suppresses stderr so no errors are returned</td>
  </tr><tr><td><code>disableQuietMode()</code></td>
  <td>Includes stderr in output</td>
  </tr><tr><td><code>isQuiteModeEnabled()</code></td>
  <td>Returns true if quiet mode is enabled</td>
  </tr><tr><td><code>enablePTY()</code></td>
  <td>Enable request-pty when using <code>exec()</code></td>
  </tr><tr><td><code>disablePty()</code></td>
  <td>Disable request-pty when using <code>exec()</code></td>
  </tr><tr><td><code>isPTYEnabled()</code></td>
  <td>Returns true if request-pty is enabled</td>
  </tr><tr><td><code>getLog()</code></td>
  <td>Returns a log of the packets that have been sent and received.</td>
  </tr><tr><td><code>getServerPublicHostKey()</code></td>
  <td>Returns the server public host key. Returns false if the server signature is not signed correctly with the public host key.</td>
  </tr><tr><td><code>getExitStatus()</code></td>
  <td>Returns the exit status of an SSH command or false.</td>
  </tr><tr><td><code>getLanguagesClient2Server()</code></td>
  <td>Return a list of the languages the server supports, when receiving stuff from the client.</td>
  </tr><tr><td><code>getLanguagesServer2Client()</code></td>
  <td>Return a list of the languages the server supports, when sending stuff to the client.</td>
  </tr><tr><td><code>getCompressionAlgorithmsServer2Client()</code></td>
  <td>Return a list of the compression algorithms the server supports, when sending stuff to the client.</td>
  </tr><tr><td><code>getCompressionAlgorithmsClient2Server()</code></td>
  <td>Return a list of the compression algorithms the server supports, when receiving stuff from the client.</td>
  </tr><tr><td><code>getMACAlgorithmsServer2Client()</code></td>
  <td>Return a list of the MAC algorithms the server supports, when sending stuff to the client.</td>
  </tr><tr><td><code>getMACAlgorithmsClient2Server()</code></td>
  <td>Return a list of the MAC algorithms the server supports, when receiving stuff from the client.</td>
  </tr><tr><td><code>getEncryptionAlgorithmsServer2Client()</code></td>
  <td>Return a list of the (symmetric key) encryption algorithms the server supports, when sending stuff to the client.</td>
  </tr><tr><td><code>getEncryptionAlgorithmsClient2Server()</code></td>
  <td>Return a list of the (symmetric key) encryption algorithms the server supports, when receiving stuff from the client.</td>
  </tr><tr><td><code>getServerHostKeyAlgorithms()</code></td>
  <td>Return a list of the host key (public key) algorithms the server supports.</td>
  </tr><tr><td><code>getKexAlgorithms()</code></td>
  <td>Return a list of the key exchange algorithms the server supports.</td>
  </tr></tbody></table><h2 id="alternatives">Alternatives</h2>
  <ol><li><a href="https://www.libssh2.org/">LIBSSH2 – The SSH library</a> – The library provides similar functionality, but is a little less intuitive to use, and it requires you to have <a href="https://pecl.php.net/package/ssh2">libssh2</a> installed on the server, which most shared hosts don’t have. </li>
  <li><a href="http://symfony.com/doc/current/components/process.html">Process component</a> – Symfony’s component for writing your own script for connecting and executing commands – as you would do in a normal terminal. This limits us in the configuration options that we might need to set. Achieving the same functionality the above configuration methods provide us with with Process would require in-depth bash knowledge. If your use-case involves only running a local script, however, this might prove to be a useful component.</li>
  </ol><h2 id="summary">Summary</h2>
  <p>In this article, we introduced <a href="http://phpseclib.sourceforge.net/ssh/intro.html">phpseclib</a>, a package which provides an alternative for <a href="http://php.net/manual/en/book.ssh2.php">SSH2</a>. We have covered the configuration options necessary to get started, and the table above should help you fill in the gaps and give an overview of other configuration options available to you.</p>
  <p>For an in-depth implementation of key-based communication, see our <a href="https://www.sitepoint.com/encrypt-large-messages-asymmetric-keys-phpseclib/">previous tutorial</a>.</p>
  <p>How do you execute remote commands? Can you think of any advanced use cases for this library? What are they? Let us know in the comments!</p>


title: Gitless
url: http://gitless.com/
created_at: 2017-08-10 23:19:48
updated_at: 2017-08-10 23:19:48
mimetype: text/html
domain_name: gitless.com
http_status: 200
content:
  <div id="main" class="twothird">
        
        <p>
        Gitless is a version control system built on top of Git.
        Many people complain that Git is hard to use. We think the problem lies
        deeper than the user interface, in the concepts underlying Git. Gitless
        is an experiment to see what happens if you put a simple veneer on an app
        that changes the underlying concepts. Because Gitless is implemented on
        top of Git (could be considered what Git pros call a "porcelain" of Git),
        you can always fall back on Git. And of course your coworkers you share a
        repo with need never know that you're not a Git aficionado.
        </p>
        <p>
        Check out the <a href="http://gitless.com/#documentation">documentation</a> to get started.
        If you are new to version control, the
        documentation should be enough to get you started. If you are a Git pro
        looking to see what's different from your beloved Git you'll be able to
        spot the differences by glancing through the
        <a href="http://gitless.com/#vs">Gitless vs. Git</a> section.
        If you are
        interested in software design and want to learn more about the research
        behind Gitless see the 
        <a href="http://gitless.com/#research">research</a> section.
        </p>
      </div><div class="twothird">
        
        
        <h2>Interface</h2>
          <ul><li>
              <code><a href="http://gitless.com/#gl-init">gl init</a></code>
              - create an empty repo or create one from an existing remote
                repo
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-status">gl status</a></code>
              - show status of the repo
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-track">gl track</a></code>
              - start tracking changes to files
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-untrack">gl untrack</a></code>
              - stop tracking changes to files
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-diff">gl diff</a></code>
              - show changes to files
            </li><li>
              <code><a href="http://gitless.com/#gl-commit">gl commit</a></code>
              - record changes in the local repo
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-checkout">gl checkout</a></code>
              - checkout committed versions of files
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-history">gl history</a></code>
              - show commit history
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-branch">gl branch</a></code>
              - list, create, edit or delete branches
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-switch">gl switch</a></code>
              - switch branches
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-tag">gl tag</a></code>
              - list, create, or delete tags
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-merge">gl merge</a></code>
              - merge the divergent changes of one branch onto another
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-fuse">gl fuse</a></code>
              - fuse the divergent changes of one branch onto another
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-resolve">gl resolve</a></code>
              -  mark files with conflicts as resolved
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-publish">gl publish</a></code>
              - publish commits upstream
            </li>
            <li>
              <code><a href="http://gitless.com/#gl-remote">gl remote</a></code>
              - list, create, edit or delete remotes
            </li>
          </ul><h2>Guide</h2>
        <p>
        Contents:
          <a href="http://gitless.com/#repository">Creating a Repository</a>,
          <a href="http://gitless.com/#changes">Saving Changes</a>,
          <a href="http://gitless.com/#branching">Branching</a>,
          <a href="http://gitless.com/#tagging">Tagging</a>,
          <a href="http://gitless.com/#remote">Working with Remote Repositories</a>.
        </p>
        <a id="repository">Creating a Repository</a>
        <p>
        Say you are in directory <code>foo</code> and you want turn it into a
        repository. You do this with the
        <code><a id="gl-init">gl init</a></code> command. This transforms
        the current working directory into an empty repository and you
        are now ready to start saving changes to files in <code>foo</code>:
        </p>
        <div class="code">
        <pre>$ mkdir foo
  $ cd foo/
  $ gl init
  ✔ Local repo created in /MyFiles/foo</pre>
        </div>
        <p>
        In most cases there's already some existing repository you
        want to work on instead of starting with an empty repository. To make a
        local clone of a remote repository you can give the URL of the repository
        as input to the same <code>gl init</code> command:
        </p>
        <div class="code">
        <pre>$ mkdir experiment
  $ cd experiment/
  $ gl init https://github.com/spderosso/experiment
  ✔ Local repo created in /MyFiles/experiment
  ✔ Initialized from remote https://github.com/spderosso/experiment</pre>
        </div>
        <a id="changes">Saving Changes</a>
        <p>
        Now that you have your local repository, it's time to start saving
        changes to files. A file in Gitless can be <em>tracked</em>,
        <em>untracked</em> or <em>ignored</em>.
        A <em>tracked file</em> is a file whose changes Gitless will detect.
        Tracked files are automatically considered for commit if they are
        modified and appear listed under the "Tracked files with modifications"
        section of <code>status</code>.
        Conversely, an <em>untracked file</em> is a file whose changes Gitless
        will not detect. These are not automatically considered for commit and
        appear listed under the "Untracked files" section of
        <code>status</code>.
        Finally, an <em>ignored file</em> is a file that is completely ignored by
        Gitless and it won't even appear in the output of <code>status</code>.
        </p>
        <p>
        An example output of the <code><a id="gl-status">gl status</a></code>
        command
        (<code>foo.py</code> and <code>bar.py</code> are tracked files
        with modifications, <code>.gitignore</code> is an unmodified tracked file,
        <code>baz.py</code> is an untracked file and <code>foo.pyc</code> is an
        ignored file):
        </p>
        <div class="code">
        <pre>$ ls
  bar.py  baz.py  foo.py  foo.pyc .gitignore
  $ gl status
  On branch master, repo-directory //
  Tracked files with modifications:
    ➜ these will be automatically considered for commit
    ➜ use gl untrack &lt;f&gt; if you don't want to track changes to file f
    ➜ if file f was committed before, use gl checkout &lt;f&gt; to discard local changes
      foo.py
      bar.py
  Untracked files:
    ➜ these won't be considered for commit)
    ➜ use gl track &lt;f&gt; if you want to track changes to file f
      baz.py</pre>
        </div>
        <p>
        Now, how do files move between these three different <em>disjoint</em>
        states?
        </p>
        <p>
        A file is ignored if it's matched by the ignore specification described in
        a <code>.gitignore</code> file.
        In the example above,
        there is a <code>.gitignore</code> file whose content is '*.pyc'; since
        <code>foo.pyc</code> is matched by that pattern it's therefore an
        ignored file.
        </p>
        <p>
        A new file that is not matched by the ignore spec is initially an untracked
        file. If you want to track it you can do so with the
        <code><a id="gl-track">gl track</a></code> command. You can stop tracking
        changes to a tracked file
        with the <code><a id="gl-untrack">gl untrack</a></code> command.
        You can always revert a file back to some previous version with the
        <code><a id="gl-checkout">gl checkout</a></code> command.
        </p>
        <div class="code">
        <pre>$ gl track baz.py
  ✔ File baz.py is now a tracked file
  $ gl track baz.py
  ✘ File baz.py is already a tracked file
  $ gl untrack baz.py
  ✔ File baz.py is now an untracked file
  $ gl checkout foo.py
  You have uncommitted changes in foo.py that would be
  overwritten by checkout. Do you wish to continue? (y/N)
  &gt; y
  ✔ File foo.py checked out successfully to its state at HEAD</pre>
        </div>
        <p>
        To save changes to files you use
        <code><a id="gl-commit">gl commit</a></code>.
        By default, all tracked modified files are considered
        for commit, but the set of files to commit can be customized with the
        <code>o/only</code>, <code>e/exclude</code>, and <code>i/include</code>
        flags:
        </p>
        <div class="code">
        <pre>$ gl commit -m "foo and bar"
  $ gl commit -m "only foo" -o foo.py
  $ gl commit -m "only foo and baz" -o foo.py baz.py
  $ gl commit -m "only foo" -e bar.py
  $ gl commit -m "only foo and baz" -e bar.py -i baz.py
  $ gl commit -m "foo, bar and baz" -i baz.py</pre></div>
        <p>
        There's also a <code>p/partial</code> flag that allows you to
        interactively select segments of files to commit.
        </p><p>
        The <code><a id="gl-diff">gl diff</a></code> command can be used to see the
        difference between the working and committed versions of files. Like
        <code>commit</code>, the default set of files to diff is
        the set of all tracked modified files but it can be customized with the
        <code>o/only</code>, <code>e/exclude</code>, and <code>i/include</code>
        flags.
        </p>
        <p>
          To remove files simply do it like you would in
          your operating system (e.g., using Unix's <code>rm</code> command). Gitless
          will detect the change if the file was tracked, and it will appear as removed
          in <code>status</code>. Gitless currently doesn't detect renames. If you rename a file,
          Gitless will interpret this as a file with the old name being removed and a new one
          (with the new name and content) being created. If you want the renamed file to be tracked
          again you need to track it with <code>gl track</code>.
        </p>
   
        <a id="branching">Branching</a>
        <p>
        A branch is an independent line of development. You are always working
        on some branch. Each branch has its own history (which you can look at
        with the <a id="gl-history">gl history</a> command). Any changes to
        existing files or new files you create on a branch will not be present on
        the other branch when you switch branches.
        </p>
        <p>To create a new branch you use the
        <a id="gl-branch">gl branch</a>
        command. To switch to another branch you use the
        <a id="gl-switch">gl switch</a> command:
        </p>
        <div class="code">
        <pre>$ gl branch -c develop
  ✔ Created new branch develop
  $ gl switch develop
  ✔ Switched to branch develop
  </pre>
        </div>
        <p>
        To list all branches:
        </p>
        <div class="code">
        <pre>$ gl branch
  List of branches:
    ➜ do gl branch &lt;b&gt; to create branch b
    ➜ do gl branch -d &lt;b&gt; to delete branch b
    ➜ do gl switch &lt;b&gt; to switch to branch b
    ➜ * = current branch
      * master
        develop
  </pre>
        </div>
        <p>
        Each branch has a head, which is the last commit done on the branch.
        By default, the head of a new branch is going to be equal to the
        head of the current branch. If you want a different commit to be the
        head of the new branch you can provide one with the
        <code>dp/divergent-point</code> flag.
        </p>
        <p>
        To specify a commit you can use its id, or you can specify it via an ancestry
        reference with <code>~</code>: <code>HEAD~n</code> refers to
        the nth commit before head.
        </p>
        <p>
        To change the head of the current branch you use the <code>sh/set-head</code> flag.
        The <code>sh</code> flag is useful to, for example, amend the last commit:
        to do so, run <code>gl branch -sh HEAD~1</code>.
        Changing the head of the current branch won't touch your working directory, if you
        additionally want to reset your working directory to match the new head
        you use <code>gl checkout</code>.
        </p>
        <p>
        Eventually branches will end up having divergent
        changes. There are two ways to bring changes from one branch onto the
        current branch: <em>merge</em> and <em>fuse</em>.
        </p>
        <p>
        <em>Merging branches.</em>
        For merging the changes in <code>develop</code> onto the current branch
        you do <code><a id="gl-merge">gl merge develop</a></code>. This creates
        a new merge commit that includes the changes in <code>develop</code> in
        addition to those changes in the current branch:
        </p>
        <div class="fig">
          <img src="http://gitless.com/merge.png" alt="merge" /></div>
        <p>
        <em>Fusing branches.</em> Fusing branches gives you more control than
        merging. When you fuse changes from some branch onto the
        current branch you can specify the commits to fuse and the insertion
        point. By default, all divergent commits are fused and the insertion
        point is the divergent point (the point where the source branch diverged
        from the current).
        For example, the following figure depicts a situation in which there are
        two branches: <code>master</code> (the current branch) and
        <code>develop</code>. The last commit these two branches have in common is
        <code>A</code>. This commit is the "divergent point" (because
        it is the point at which <code>master</code> and <code>develop</code>
        diverged).
        After doing <code><a id="gl-fuse">gl fuse develop</a></code>, the commits
        in <code>develop</code> are inserted in <code>master</code> after the
        divergent point:
        </p>
        <div class="fig">
          <img src="http://gitless.com/fuse.png" alt="fuse" class="fig" /></div>
        <p>
        To choose other insertion points you use the
        <code>ip/insertion-point</code> flag. You can give a commit id as input,
        <code>head</code> or <code>dp/divergent-point</code>:
        </p>
        <div class="fig">
          <img src="http://gitless.com/fuse_ip.png" alt="fuse with insertion point specified" /></div>
        <p>
        The <code>o/only</code> and <code>e/exclude</code> flags can be used to
        customize the set of commits to be fused:
        </p>
        <div class="fig">
          <img src="http://gitless.com/fuse_ip_o.png" alt="fuse with insertion point and commit specified" /></div>
        <p>
        During this process conflicts could occur. If so, the <code>gl status</code> command will change
        accordingly to indicate the files in conflict. Once you edit those files in
        conflict you mark them as resolved with
        <code><a id="gl-resolve">gl resolve</a></code> (passing the files to mark as input). Once all
        conflicts have been resolved
        you do <code>gl commit</code> to commit to continue with the fuse or
        merge.
        </p>
        <p>
        A branch can have an "upstream branch." If a branch has an upstream
        associated with it, then <code>gl fuse</code> or <code>gl merge</code> can
        be used as shorthands for <code>gl {fuse, merge} upstream_branch</code>.
        To set an upstream branch for the current branch use
        <code>gl branch -su upstream_branch</code>.
        </p>
        <a id="tagging">Tagging</a>
        <p>
        You use tags to signify a commit is special in some way. For example, you
        can use <code><a id="gl-tag">gl tag</a></code> to create a tag with name
        "v1.0" and make it point to the commit that represents release v1.0:
        </p>
        <div class="code">
        <pre>$ gl tag -c v1.0
  ✔ Created new tag v1.0
  </pre>
        </div>
        <p>In this case, the tag will point to the head of the current branch, but
        you can tag other commits with the <code>ci/commit</code> flag.
        </p>
        <p>
        To list all tags:
        </p>
        <div class="code">
        <pre>$ gl tag
  List of tags:
    ➜ do gl tag &lt;t&gt; to create tag t
    ➜ do gl tag -d &lt;t&gt; to delete tag t
      v1.0 ➜  tags 311bf7c Ready to release
  </pre>
        </div>
        <a id="remote">Working with Remote Repositories</a>
        <p>
        To refer to a remote repository you could always use its URL, but an
        easier alternative is to add the repository as a "remote" with the
        <code><a id="gl-remote">gl remote</a></code> command:
        </p>
        <div class="code"><pre>$ gl remote -c experiment https://github.com/spderosso/experiment
  ✔ Remote experiment mapping to https://github.com/spderosso/experiment created
  successfully
    ➜ to list existing remotes do gl remote
    ➜ to remove experiment do gl remote -d experiment
  </pre></div>
        <p>
        Now you can use <code>experiment</code> to refer to this remote repository
        and use <code>experiment/some-branch</code> to refer to the branch of name
        <code>some-branch</code> that lives in <code>experiment</code>.
        </p>
        <p>
        <em>Downloading Changes.</em>
        It is also possible to fuse or merge changes from remote
        branches. For example, doing <code>gl merge experiment/master</code>
        would merge changes in that remote branch that are not present in your
        local current branch. You can also use <code>gl fuse</code>.
        </p>
        <p>
        <em>Uploading Changes.</em>
        To send changes upstream you use
        <code><a id="gl-publish">gl publish</a></code>. The publish command will
        default to updating the upstream branch of the current branch if none is
        given as input.
        </p>
        <p>
        <em>Creating, Deleting, or Listing Remote Branches.</em>
        To create, delete, or list remote branches you use the same <code>gl branch</code>
        command that you use for local branches. If you do
        <code>gl branch -c experiment/develop</code> this will create a branch
        <code>develop</code> that lives in the remote <code>experiment</code>.
        Recall that, by default, the head of this new branch will be equal to the head of
        the current branch, so all commits that are not present in the remote
        will be uploaded. To list remote branches use the
        <code>r/remote</code> flag of <code>gl branch</code>.
        </p>
        <p>
        <em>Creating, Deleting, or Listing Remote Tags.</em>
        To create or delete remote tags you use the <code>gl tag</code> command.
        If you do
        <code>gl tag -c experiment/v1.0</code> this will create a new tag
        <code>v1.0</code> that lives in the remote <code>experiment</code>. You can also list
        remote tags with the <code>r/remote</code> flag of <code>gl tag</code>.
        </p>
        <p>
        When you create a local repository from a remote (by passing a URL as
        input to the <code>gl init</code> command), a local branch is created
        for each remote branch, and each local branch is automatically configured
        to have as upstream its remote counterpart.
        </p>
    
        
      </div><div class="twothird">
        
        
        <h2>Saving Changes</h2>
        <p>There's no staging area in Gitless. This, coupled with a flexible
        commit command makes saving changes to the repository very
        straightforward:
        </p>
        <div class="code vs">
          <pre>Commit all modified tracked files
  $ gl commit
  The default set of files to be committed are all
  modified tracked files
  Leave some modified tracked files (`foo`, `bar`)
  out of the commit
  $ gl commit -e foo bar
  e/exclude excludes files from the default set
  of files to be committed
  Include some untracked files in the commit
  $ gl commit -i foo2 bar2
  i/include includes files to the default set
  of files to be committed
  Commit only some of the modified tracked files
  $ gl commit -o foo3 bar3
  o/only restricts the set of files to be committed
  to only the specified ones
  Commit only some of the modified tracked or
  untracked files
  $ gl commit -o foo3 bar3 foo4
  </pre>
  <pre>Commit all modified tracked files
  $ git commit -a
  -a/--all automatically stages files that have been
  modified or deleted
  Leave some modified tracked files (`foo`, `bar`)
  out of the commit
  $ git add all files you want to commit minus foo bar
  $ git commit
  Include some untracked files in the commit
  $ git add foo2 bar2 + other files you want to commit
  $ git commit
  Commit only some of the modified tracked files
  $ git commit foo3 bar3
  Commit only some of the modified tracked or
  untracked files
  $ git commit foo3 bar3 foo4
  error: pathspec 'foo4' did not match any file(s)
  known to git.
  ...hmm ok... 
  $ git add foo4
  $ git commit foo3 bar3 foo4</pre>
        
        </div>
        <p>Also, you can change the classification of any file to tracked,
        untracked or ignored, it doesn't matter whether the file exists at head
        or not:
        </p>
        <div class="code vs">
          <pre>$ gl status
  ...
  Tracked files with modifications:
  ...
    foo
  ...
  Stop tracking changes to `foo`
  $ gl untrack foo
  ✔ File foo is now an untracked file
  Now `foo` won't be automatically considered for
  commit
  $ gl status
  ...
  Untracked files:
  ...
    foo (exists at head)
  ...
  Start tracking changes to `foo` again
  $ gl track foo
  ✔ File foo is now a tracked file
  Now `foo` will be automatically considered for commit
  $ gl status
  ...
  Tracked files with modifications:
  ...
    foo
  ...
  </pre>
  <pre>$ git status
  ...
  Changes not staged for commit:
  ...
    modified: foo
  ...
  Stop tracking changes to `foo`
  No way to do this, if you want to prevent the
  accidental commit of `foo` you can do `git add` of
  all files but `foo` everytime you do a commit.
  Alternatively, you can ignore the file by marking it
  as assumed unchanged
  $ git update-index --assume-unchanged foo
  Now the file behaves as if it were ignored
  $ git status
  ...
  `foo` won't appear listed in status
  Start tracking changes to `foo` again
  $ git update-index --no-assume-unchanged foo
  $ git status
  ...
  Changes not staged for commit:
  ...
    modified: foo
  ...
  </pre>
          
        </div>
        
        
        <h2>Branching</h2>
        <p>
        The main thing to understand is that in Gitless a branch is a completely
        independent line of development. Each branch keeps its working version
        of files separate from each other. Whenever you switch to a different
        branch, the contents of your working directory are saved, and the ones
        corresponding to the branch you are switching to are retrieved. The
        classifications of files are also saved (i.e., a file can
        be untracked on some branch but tracked on another and Gitless will
        remember this):
        </p>
        <div class="code vs">
          <pre>$ gl status
  ...
  Tracked files with modifications:
  ...
    foo
  ...
  Create new branch `develop`
  $ gl branch -c develop
  ✔ Created new branch develop
  Switch to `develop`
  $ gl switch develop
  ✔ Switched to branch develop
  $ gl status
  ... no changes to foo here
  Switch back to `master`
  $ gl switch master
  ✔ Switched to branch master
  $ gl status
  ...
  Tracked files with modifications:
  ...
    foo
  ...
  If you want the uncommitted changes to follow you
  into the new branch you can use the mo/move-over
  flag to move over the changes in the current branch
  to the destination branch
  </pre>
  <pre>$ git status
  ...
  Changes not staged for commit:
  ...
    modified: foo
  ...
  Create new branch `develop`
  $ git branch develop
  Switch to `develop`
  $ git checkout develop
  $ git status
  ...
  Changes not staged for commit:
  ...
    modified: foo
  ...
  Switch back to `master`
  $ git checkout master
  ...
  Changes not staged for commit:
  ...
    modified: foo
  ...
  </pre>
        </div>
        <p>
        This means that in Gitless you don't have to worry about uncommitted
        changes conflicting with the changes in the destination branch:
        </p>
        <div class="code vs">
          <pre>Say we have uncommitted changes to `foo` that
  conflict with the state of `foo` in branch `develop`
  Switch to `develop`
  $ gl switch develop
  ✔ Switched to branch develop
  </pre>
  <pre>Say we have uncommitted changes to `foo` that
  conflict with the state of `foo` in branch `develop`
  Switch to `develop`
  $ git checkout develop
  error: Your local changes to the following files
  would be overwritten by checkout:
       foo
  Please, commit your changes or stash them before
  you can switch branches.
  Aborting
  $ git stash
  Saved working directory and index state WIP on
  master: fbe3b8c ...
  HEAD is now at fbe3b8c ...
  $ git checkout develop
  Switched to branch 'develop'
  </pre>
        </div>
        <p>
        And if you are in the middle of a fuse/merge and you want to put aside
        the conflict resolution for later, you can. The conflict will be there
        when you switch back:
        </p>
        <div class="code vs">
          <pre>$ gl fuse develop
  ...
  ✘  There are conflicts you need to resolve
  $ gl status
  On branch master, repo-directory //
  You are in the middle of a fuse; all
  conflicts must be resolved before
  committing
  ...
  Tracked files with modifications:
  ...
    foo (with conflicts)
  ...
  Switch to `bugfix`
  $ gl switch bugfix
  ✔ Switched to branch bugfix
  No conflicts here
  Switch back to `develop`
  $ gl switch develop
  ✔ Switched to branch develop
  $ gl status
  On branch master, repo-directory //
  You are in the middle of a fuse; all
  conflicts must be resolved before
  committing
  ...
  Tracked files with modifications:
  ...
    foo (with conflicts)
  ...
  </pre>
  <pre>$ git rebase develop
  First, rewinding head to replay your work on top of
  it...
  Applying: this commit should trigger a conflict
  Using index info to reconstruct a base tree...
  M foo
  Falling back to patching base and 3-way merge...
  Auto-merging foo
  CONFLICT (content): Merge conflict in foo
  Failed to merge in the changes.
  Patch failed at 0001 foo conflict
  The copy of the patch that failed is found in:
     ...
  $ git status
  rebase in progress; onto 989269e
  You are currently rebasing branch 'master' on
  '989269e'.
    ...
  Unmerged paths:
    ...
    both modified:   foo
  ...
  Switch to `bugfix`
  $ git checkout bugfix
  foo: needs merge
  error: you need to resolve your current index first
  Maybe stash works?
  $ git stash
  foo: needs merge
  foo: needs merge
  foo: unmerged (0b3c542edb2e9e8ff801c669d7a5f2d78...)
  foo: unmerged (94421333de34e32405f632d0f7b63b39c...)
  foo: unmerged (eb97dba229aab53fe5f231e60491dd2a7...)
  fatal: git-write-tree: error building trees
  Cannot save the current index state
   ...hmm..ok...
  $ save all the changes to files somewhere else
  $ git rebase --abort
  $ git checkout bugfix
  Switched to branch 'bugfix'
  Switch back to `develop`
  $ git checkout develop
  $ git status
  ...
  We aborted the rebase to be able to switch branches,
  so there's nothing here now
  $ git rebase develop
  ... same conflict as before
  $ Copy back the changes we saved
  </pre>
          
        </div>
        
        
        <h2>Working with Remote Repositories</h2>
        <p>
        Syncing with other repositories in Gitless works quite similar to
        Git:
        </p>
        <div class="code vs">
          <pre>Configure a new remote `experiment`
  $ gl remote -c experiment
  https://github.com/spderosso/experiment
  ✔ Remote experiment mapping to
  https://github.com/spderosso/experiment
  created successfully
    ➜ to list existing remotes do gl remote
    ➜ to remove experiment do gl remote -d experiment
  Download and apply changes from a branch in
  `experiment`
  $ gl fuse experiment/master
  ...
  ✔ Fuse succeeded
  If you instead want to merge the changes, you
  can do `gl merge experiment/master`
  You can also set an upstream for the current branch
  $ gl branch -su experiment/master
  ✔ Current branch master set to track
  experiment/master
  $ gl fuse 
  ! No src branch specified, getting changes from
  upstream branch experiment/master
  ...
  ✔ Fuse succeeded
  Send changes to the remote
  $ gl publish
  ! No src branch specified, sending changes to
  upstream branch experiment/master
  ✔ Publish succeeded
  Only the changes in the current branch are uploaded</pre>
  <pre>Configure a new remote `experiment`
  $ git remote add experiment
  https://github.com/spderosso/experiment
  Download and apply changes from a branch in
  `experiment`
  $ git pull experiment/master
  ...
  If you instead want to rebase the changes, you can
  do `git pull --rebase experiment/master`
  You can also set an upstream for the current branch
  $ git branch --set-upstream master experiment/master
  $ git pull
  ...
  Send changes to the remote
  $ git push
  What happens when you do a push depends on the value
  of the `push.default` config variable
  </pre>
          
        </div>
        
      </div><div class="twothird">
        
        <p>
        Here's a list of papers related to Gitless:
        </p>
        <ul><li>
          <a href="https://spderosso.github.io/oopsla16.pdf" title="pdf - Purposes, Concepts, Misfits, and a Redesign of Git">
            Purposes, Concepts, Misfits, and a Redesign of Git</a><br />S. P. De Rosso and D. Jackson.
          In <em>Proceedings of the 2016 ACM SIGPLAN International
          Conference on Object-Oriented Programming, Systems, Languages, and
          Applications (OOPSLA 2016)</em>
          </li>
          <li>
          <a href="https://spderosso.github.io/onward13.pdf" title="pdf - What's Wrong with Git? A Conceptual Design Analysis">
            What's Wrong with Git? A Conceptual Design Analysis</a><br />S. P. De Rosso and D. Jackson.
          In <em>Proceedings of the 2013 ACM International Symposium on New
          Ideas, New Paradigms, and Reflections on Programming &amp; Software
          (Onward! 2013)</em>
          </li>
        </ul></div>


title: Information Software and the Graphical Interface
url: http://worrydream.com/MagicInk/
created_at: 2017-08-10 23:19:50
updated_at: 2017-08-10 23:19:50
mimetype: text/html
domain_name: worrydream.com
http_status: 200
content:
  Magic Ink:  Information Software and the Graphical Interface
  <div id="draft">
  <p>This draft was released March 15, 2006.  Please email comments to <strong>bret<img width="10" height="11" src="http://worrydream.com/MagicInk/p/a.png" alt="image" />worrydream.com</strong>.  You can also <a href="http://worrydream.com/MagicInk/MagicInk.pdf">download the PDF</a>. </p>
  </div>  
  <div id="toc"><p><strong>Contents:</strong></p>
  <p>What is software?
  </p>
  <ul><li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#of_software_and_sorcery">Of software and sorcery</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#what_is_software_design">What is software design?</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#what_is_software_for">What is software for?</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#manipulation_software_design_is_hard">Manipulation software design is hard</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#most_software_is_information_software">Most software is information software</a></li>
  </ul><p>Graphic design
  </p>
  <ul><li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#information_software_design_is_graphic_design">Information software design is graphic design</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#demonstration_showing_the_data">Demonstration: Showing the data</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#demonstration_arranging_the_data">Demonstration: Arranging the data</a></li>
  </ul><p>Context-sensitivity
  </p>
  <ul><li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#contextsensitive_information_graphics">Context-sensitive information graphics</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#inferring_context_from_the_environment">Inferring context from the environment</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#inferring_context_from_history">Inferring context from history</a></li>
  </ul><p>Interactivity
  </p>
  <ul><li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#interactivity_considered_harmful">Interactivity considered harmful</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#reducing_interaction">Reducing interaction</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#how_did_we_get_here">How did we get here?</a></li>
  </ul><p>Intermission
  </p>
  <ul><li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#case_study_train_schedules">Case study:  Train schedules</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#demonstration_trip_planning_redux">Demonstration:  Trip planning redux</a></li>
  </ul><p>Changing the world
  </p>
  <ul><li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#designing_the_information_software_revolution">Designing the information software revolution</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#designing_a_design_tool">Designing a design tool</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#engineering_inference_from_history">Engineering inference from history</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#engineering_inference_from_the_environment">Engineering inference from the environment</a></li>
  <li><a class="toc_a" target="_self" href="http://worrydream.com/MagicInk/#information_and_the_world_of_tomorrow">Information and the world of tomorrow</a></li>
  </ul></div> 
  <h3>Information Software and the Graphical Interface</h3>
  <h4>by Bret Victor</h4>
  <div id="frontmatter">
  <h2 id="abstract">Abstract</h2>
  <p id="p54">The ubiquity of frustrating, unhelpful software interfaces has motivated decades of research into “Human-Computer Interaction.”  In this paper, I suggest that the long-standing focus on “interaction” may be misguided.  For a majority subset of software, called “information software,” I argue that interactivity is actually a curse for users and a crutch for designers, and users’ goals can be better satisfied through other means.</p>
  <p id="p55">Information software design can be seen as the design of <em>context-sensitive information graphics</em>.  I demonstrate the crucial role of information graphic design, and present three approaches to context-sensitivity, of which interactivity is the last resort.  After discussing the cultural changes necessary for these design ideas to take root, I address their implementation.  I outline a tool which may allow designers to create data-dependent graphics with no engineering assistance, and also outline a platform which may allow an unprecedented level of implicit context-sharing between independent programs.  I conclude by asserting that the principles of information software design will become critical as technology improves.</p>
  <p id="p56">Although this paper presents a number of concrete design and engineering ideas, the larger intent is to introduce a “unified theory” of information software design, and provide inspiration and direction for progressive designers who suspect that the world of software isn’t as flat as they’ve been told.</p>
  <h2 id="scope_and_terminology">Scope and terminology</h2>
  <p id="p57">“<b>Software</b>,” as used here, refers to user-facing personal desktop software, whether on a native or web platform.  “<b>Software design</b>” describes all appearance and behaviors visible to a user; it approaches software as a <em>product</em>.  “<b>Software engineering</b>” implements the design on a computer; it approaches software as a <em>technology</em>.  These are contentious definitions; hopefully, this paper itself will prove far more contentious.</p>
  <div id="front_toc">
  <h2 id="contents">Contents</h2>
  <p>What is software?
  </p>
  <ul><li><a target="_self" href="http://worrydream.com/MagicInk/#of_software_and_sorcery">Of software and sorcery.</a> Is “interaction design” the cure for frustrating software, or the disease itself?
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#what_is_software_design">What is software design?</a> Software is not a new and mysterious medium, but a fusion of two old ones.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#what_is_software_for">What is software for?</a> People turn to software to learn, to create, and to communicate.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#manipulation_software_design_is_hard">Manipulation software design is hard.</a> Creating software for creating is tricky business.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#most_software_is_information_software">Most software is information software.</a> People spend more time learning than creating.
  </li>
  </ul><p>Graphic design
  </p>
  <ul><li><a target="_self" href="http://worrydream.com/MagicInk/#information_software_design_is_graphic_design">Information software design is graphic design.</a> People learn by looking.  Looks are all that matters.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#demonstration_showing_the_data">Demonstration: Showing the data.</a> Redesigning Amazon as an information graphic.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#demonstration_arranging_the_data">Demonstration: Arranging the data.</a> Redesigning Yahoo! Movies as an information graphic.
  </li>
  </ul><p>Context-sensitivity
  </p>
  <ul><li><a target="_self" href="http://worrydream.com/MagicInk/#contextsensitive_information_graphics">Context-sensitive information graphics.</a> Software trumps print by showing only what’s relevant.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#inferring_context_from_the_environment">Inferring context from the environment.</a> The outside world can suggest what’s relevant.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#inferring_context_from_history">Inferring context from history.</a> Memories of the past can suggest what’s relevant.
  </li>
  </ul><p>Interactivity
  </p>
  <ul><li><a target="_self" href="http://worrydream.com/MagicInk/#interactivity_considered_harmful">Interactivity considered harmful.</a> The user can suggest what’s relevant, but only as a last resort.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#reducing_interaction">Reducing interaction.</a> Approaches to easing the pain.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#how_did_we_get_here">How did we get here?</a> The popular focus on interactivity is a vestige of another era.
  </li>
  </ul><p>Intermission
  </p>
  <ul><li><a target="_self" href="http://worrydream.com/MagicInk/#case_study_train_schedules">Case study:  Train schedules.</a> Designing a trip planner as an information graphic.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#demonstration_trip_planning_redux">Demonstration:  Trip planning redux.</a> Redesigning Southwest Airlines as an information graphic.
  </li>
  </ul><p>Changing the world
  </p>
  <ul><li><a target="_self" href="http://worrydream.com/MagicInk/#designing_the_information_software_revolution">Designing the information software revolution.</a> Five steps from artifice to art form.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#designing_a_design_tool">Designing a design tool.</a> Dynamic graphics without the programming.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#engineering_inference_from_history">Engineering inference from history.</a> How software can learn from the past.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#engineering_inference_from_the_environment">Engineering inference from the environment.</a> A platform for implicit communication between software.
  </li>
  <li><a target="_self" href="http://worrydream.com/MagicInk/#information_and_the_world_of_tomorrow">Information and the world of tomorrow.</a> Why all this matters.
  </li>
  </ul></div>  
  </div>  
  <p class="logo"><img width="435" height="83" src="http://worrydream.com/MagicInk/p/logo_what_is_software.png" alt="image" /></p>
  <h2 id="of_software_and_sorcery">Of software and sorcery</h2>
  <blockquote>
    <p id="p44">A computational process is indeed much like a sorcerer’s idea of a spirit. It cannot be seen or touched. It is not composed of matter at all. However, it is very real. It can perform intellectual work. It can <b>answer questions</b>. It can <b>affect the world</b> by disbursing money at a bank or by controlling a robot arm in a factory. The programs we use to conjure processes are like a sorcerer’s spells.<br />—Abelson and Sussman, <a href="http://mitpress.mit.edu/sicp/">Structure and Interpretation of Computer Programs</a> (1984)</p>
  </blockquote>
  <p id="p59">Merlin had it easy—raising Stonehenge was a mere engineering challenge.  He slung some weighty stones, to be sure, but their placement had only to please a subterranean audience whose interest in the matter was rapidly decomposing.  The dead are notoriously unpicky.</p>
  <p id="p60">Today’s software magicians carry a burden heavier than 13-foot monoliths—communication with the living.  They often approach this challenge like Geppetto’s fairy—attempting to instill the spark of life into a mechanical contraption, to create a Real Boy.  Instead, their vivified creations often resemble those of Frankenstein—helpless, unhelpful, maddeningly stupid, and prone to accidental destruction.</p>
  <p id="p61">This is a software crisis, and it isn’t news.  For decades, the usability pundits have devoted vim and vitriol to a crusade against frustrating interfaces.  Reasoning that the cure for unfriendly software is to make software friendlier, they have rallied under the banner of “interaction design,” spreading the gospel of friendly, usable interactivity to all who would listen.</p>
  <p id="p62">Yet, software has remained frustrating, and as the importance of software to society has grown, so too has the crisis.  The crusade marches on, with believers rarely questioning the sacred premise—that software must be interactive in the first place.  That software is meant to be “used.”</p>
  <p id="p63">I suggest that the root of the software crisis is an identity crisis—an unclear understanding of what the medium actually <em>is</em>, and what it’s <em>for</em>.  Perhaps the spark of life is misdirected magic.
  A good introduction to the crisis is Alan Cooper’s <a href="http://www.amazon.com/gp/product/0672326140">The Inmates Are Running The Asylum</a> (1999).  Essential concepts of interactive design are presented in Don Norman’s <a href="http://www.amazon.com/gp/product/0465067107">Design Of Everyday Things</a> (2002), Jef Raskin’s <a href="http://www.amazon.com/gp/product/0201379376">The Humane Interface</a> (2000), and Cooper’s <a href="http://www.amazon.com/gp/product/0764526413">About Face</a> (2003).</p>
  <p id="p64">A person experiences modern software almost exclusively through two channels:</p>
  <ul><li>She reads and interprets pictures on a screen.</li>
  <li>She points and pushes at things represented on the screen, using a mouse as a proxy finger.</li>
  </ul><p id="p65">Thus, software design involves the design of two types of artifact:</p>
  <ul><li>Pictures.</li>
  <li>Things to push.</li>
  </ul><p id="p66">These are not brave new realms of human endeavor.  We share the blood of cavemen who pushed spears into mammoths and drew pictures of them in the living room.  By now, these two activities have evolved into well-established design disciplines:  graphic design and industrial design.</p>
  <p id="p67"><strong>Graphic design</strong> is the art of conveying a message on a two-dimensional surface.  This is a broad field, because people have such a variety of messages to convey—identity, social status, emotion, persuasion, and so on.  Most relevant to software is a branch that Edward Tufte calls <em>information design</em>—the use of pictures to express knowledge of interest to the reader.** Edward Tufte, <a href="http://www.amazon.com/gp/product/0961392142">The Visual Display of Quantitative Information</a> (2001).      Some products of conventional information graphic design include bus schedules, telephone books, newspapers, maps, and shopping catalogs.  A good graphic designer understands how to arrange information on the page so the reader can ask and answer questions, make comparisons, and draw conclusions.</p>
  <p id="p68">When the software designer defines the visual representation of her program, when she describes the pictures that the user will interpret, she is doing graphic design, whether she realizes this or not.</p>
  <p id="p69"><strong>Industrial design</strong> is the art of arranging and shaping a physical product so it can be manipulated by a person.  This too is a broad field, because people work with such a variety of objects—cutlery to chairs, cell phones to cars.  A good industrial designer understands the capabilities and limitations of the human body in manipulating physical objects, and of the human mind in comprehending mechanical models.  A camera designer, for example, shapes her product to fit the human hand.  She places buttons such that they can be manipulated with index fingers while the camera rests on the thumbs, and weights the buttons so they can be easily pressed in this position, but won’t trigger on accident.  Just as importantly, she designs an understandable <em>mapping</em> from physical features to functions—pressing a button snaps a picture, pulling a lever advances the film, opening a door reveals the film, opening another door reveals the battery.</p>
  <p id="p70">Although software is the archetypical non-physical product, modern software interfaces have evolved overtly mechanical metaphors.  Buttons are pushed, sliders are slid, windows are dragged, icons are dropped, panels extend and retract.  People are encouraged to consider software a machine—when a button is pressed, invisible gears grind and whir, and some internal or external state is changed.  Manipulation of machines is the domain of industrial design.</p>
  <p id="p71">When the software designer defines the interactive aspects of her program, when she places these pseudo-mechanical affordances and describes their behavior, she is doing a virtual form of industrial design.  Whether she realizes it or not.</p>
  <p id="p72">The software designer can thus approach her art as a fusion of graphic design and industrial design.  Now, let’s consider how a user approaches software, and more importantly, <em>why</em>.</p>
  <h2 id="what_is_software_for">What is software for?</h2>
  <p id="p73">Software is for people.  To derive what software should do, we have to start with what <em>people</em> do.  Consider the following taxonomy of human activity:** There are any number of ways of breaking down the spectrum of human activity.  I don’t claim that the subdivision given here is <em>definitive</em>, but that it’s <em>useful</em>.  Consider it a set of basis vectors into the space of human activity.  Different basis sets are helpful for reasoning about different problems, but they all describe the same space.    </p>
  <p><img width="345" height="103" src="http://worrydream.com/MagicInk/p/taxonomy.png" alt="image" /></p>
  <p id="p74">At the present, software can’t do much for physical needs—if your avatar eats a sandwich, you remain hungry.  But people are increasingly shifting their intellectual activities to the virtual world of the computer.  This suggests three general reasons why a person will turn to software:</p>
  <ul><li>To learn.</li>
  <li>To create.</li>
  <li>To communicate.</li>
  </ul><p id="p75">I propose that software can be classified according to which of these needs it serves.  I will call these categories <em>information</em> software, <em>manipulation</em> software, and <em>communication</em> software.</p>
  <p id="p76"><strong>Information software</strong> serves the human urge to learn.
  <img width="200" height="107" src="http://worrydream.com/MagicInk/p/information_model.png" alt="image" />
  A person uses information software to <b>construct and manipulate a model that is internal to the mind</b>—a mental representation of information.  Good information software encourages the user to ask and answer questions, make comparisons, and draw conclusions.  A person would use recipe software, for example, to decide what to cook for dinner.  She would learn about various dishes (where “learning” could be as informal as a quick skim for something tasty that contains ingredients on hand), compare her options, and make her decision.  In effect, she is constructing an internal understanding of culinary possibilities, and mentally prodding this model to reveal the optimal choice.  It’s the same effect she would hope to achieve by consulting a recipe <em>book</em>.</p>
  <p id="p77"><strong>Manipulation software</strong> serves the human urge to create.
  <img width="200" height="127" src="http://worrydream.com/MagicInk/p/manipulation_model.png" alt="image" />
  A person uses manipulation software to <b>construct and manipulate a model external to herself</b>—a virtual object represented within the computer, or a remote physical object.  Some examples include software for drawing, writing, music composition, architectural design, engineering design, and robot control.  Manipulation software can be considered a virtual <em>tool</em>—like a paintbrush or typewriter or bandsaw, it is used as an interface between creator and artifact.</p>
  <p id="p78"><strong>Communication software</strong> serves the human urge to communicate.
  <img width="200" height="121" src="http://worrydream.com/MagicInk/p/communication_model.png" alt="image" />
  A person uses communication software to <b>construct and manipulate an internal model that is shared with others</b>—an understanding synchronized across multiple minds.  Examples include software for email, group discussions (whether voice, video, or text), and collaborative working.  In terms of raw mechanics, communication can be thought of as <em>creating</em> a response to information <em>learned</em>—that is, the external model manipulated by the speaker is the internal model learned by the listener.  Thus, this paper will simply treat communication software as manipulation software and information software glued together, and mention it no further.** This dismissal is rather disingenuous—communication software is fundamentally unlike the other two because its user is a <em>group</em>, and a group as a whole can have different goals than any of its constituents individually.  The considerations of social software design are well beyond the scope of this paper, but see <a href="http://www.shirky.com/">Clay Shirky</a>’s essays, particularly <a href="http://shirky.com/writings/group_politics.html">Social Software and the Politics of Groups</a> (2003).      This design approach is widespread—email software typically has separate reading and writing modes; messageboards similarly segregate browsing and posting.</p>
  <p id="p79">Manipulation software generally displays a representation of an object—the model—which the user directly manipulates with pseudo-mechanical affordances.  Because manipulation is the domain of industrial design, manipulation software emphasizes industrial design aspects.</p>
  <p id="p80">Consider a tool for laying out a small newspaper.  The user will spend most of her time performing a number of pseudo-physical operations—writing, drawing, cutting, moving, rotating, stretching, cropping, layering—within a virtual space.  The primary design challenge, just as with any industrial design, is to provide affordances that make these mechanical operations <em>available</em>, <em>understandable</em>, and <em>comfortable</em>.  However, in a physical space, each operation would use a specialized tool.   Designing a “mega-tool” that cleanly incorporates all operations (and flattens them into two dimensions, and uses only the gestures “click” and “drag”) is a significant challenge indeed.</p>
  <p id="p81">Although manipulation is the focus, good manipulation software must provide superb visualization as well.  This establishes the feedback loop that is critical for all creative activity—the manipulator must see the effects of her manipulation.  Thus, manipulation software design is also a significant graphic design challenge.</p>
  <p id="p82">For example, the newspaper editor needs to see what a page looks like—close-up, from a distance, and in relation to other pages—and how it <em>would</em> look in a variety of other configurations.  She wants to see misspelled words, lines that are poorly justified or hyphenated, and widows and orphans.  She wants to see columns that are short or overlong, and how they can be corrected by changing column width or leading.  She wants to know what stories and ads are still on the table, their sizes, and how they can be fit in.  She wants to know how recently and how often stories about a given topic have run, and how readers have responded.  She wants to know past response to a given ad, as a function of the topics or authors of the stories it was coupled with.  Finally, the presentation of all this information must not distract the editor from the primary task of manipulating the layout.</p>
  <p id="p83">Furthermore, the industrial and graphic designs in manipulation software must be in intimate synergy, since it is the graphic design which describes how the object can be manipulated—the mechanical affordances are graphical constructs.  Even more graphically challenging is manipulation of abstract objects, such as music or financial data, where the graphical representation must show not only what can be done with it, but <em>what it is</em> in the first place.** As opposed to painting software, for instance, where the graphical representation can be the artifact itself.  This is not a pipe, but it’s close enough.    </p>
  <p id="p84">Because of these intertwined design challenges, the design of excellent manipulation software is <em>unbelievably</em> difficult, and mustn’t be underestimated.  Fortunately, for an enormous class of software, manipulation is not only largely unnecessary, but best avoided.</p>
  <p id="p85">J.C.R. Licklider once examined how he spent his research time:</p>
  <blockquote>
    <p id="p45">In the spring and summer of 1957… I tried to keep track of what one moderately technical person [myself] actually did during the hours he regarded as devoted to work…  About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. Much more time went into finding or obtaining information than into digesting it. Hours went into the plotting of graphs, and other hours into instructing an assistant how to plot. When the graphs were finished, the relations were obvious at once, but the plotting had to be done in order to make them so…  Throughout the period I examined, in short, my “thinking” time was devoted mainly to activities that were essentially clerical or mechanical: searching, calculating, plotting, transforming, determining the logical or dynamic consequences of a set of assumptions or hypotheses, preparing the way for a decision or an insight.** J.C.R. Licklider, “<a href="http://memex.org/licklider.pdf">Man-Computer Symbiosis</a>” (1960).   </p>
  </blockquote>
  <p id="p86">For Licklider and other early visionaries such as Vanevar Bush and Doug Engelbart,** See Bush’s paper “<a href="http://sloan.stanford.edu/mousesite/Secondary/Bush.html">As We May Think</a>” (1945) and Engelbart’s paper “<a href="http://www.dougengelbart.org/pubs/augment-3906.html">Augmenting Human Intellect</a>” (1962).    the ideal of the then-hypothetical personal computer was a brain supplement, enhancing human memory and amplifying human reasoning through data visualization and automated analysis.  Their primary concern was how a machine could help a person <em>find</em> and <em>understand</em> relevant knowledge.  Although they were generally discussing scientific and professional work, their prescience fully applies in the modern home.</p>
  <p id="p87">Most of the time, a person sits down at her personal computer not to create, but to <em>read, observe, study, explore, make cognitive connections, and ultimately come to an understanding</em>.  This person is not seeking to make her mark upon the world, but to rearrange her own neurons.  The computer becomes a medium for asking questions, making comparisons, and drawing conclusions—that is, for <em>learning</em>.</p>
  <p id="p88"><b>People turn to software to learn the meaning of words, learn which countries were bombed today, and learn to cook a paella.  They decide which music to play, which photos to print, and what to do tonight, tomorrow, and Tuesday at 2:00.  They keep track of a dozen simultaneous conversations in private correspondence, and maybe hundreds in public arenas.  They browse for a book for Mom, a coat for Dad, and a car for Junior.  They look for an apartment to live in, and a bed for that apartment, and perhaps a companion for the bed.  They ask when the movie is playing, and how to drive to the theater, and where to eat before the movie, and where to get cash before they eat.  They ask for numbers, from simple sums to financial projections.  They ask about money, from stock quote histories to bank account balances.  They ask why their car isn’t working and how to fix it, why their child is sick and how to fix her.  They no longer sit on the porch speculating about the weather—<em>they ask software</em>.</b></p>
  <p id="p89">Much current software fulfilling these needs presents mechanical metaphors and objects to manipulate, but this is deceiving.  People using this software do not <em>care</em> about these artificial objects; they care about seeing information and understanding choices—manipulating a model in their heads.</p>
  <p id="p90">For example, consider calendar or datebook software.  Many current designs center around manipulating a database of “appointments,” but is this really what a calendar is for?  To me, it is about combining, correlating, and visualizing a vast collection of information.  I want to understand what I have planned for tonight, what my friends have planned, what’s going on downtown, what’s showing when at the movie theater, how late the pizza place is open, and which days they are closed.  I want to see my pattern of working late before milestones, and how that extrapolates to future milestones.  I want to see how all of this information interrelates, make connections, and ultimately make a decision about what to do when.  Entering a dentist appointment is just a tedious minor detail, and would even be <em>unnecessary</em> if the software could figure it out from my dentist’s confirmation email.  My goal in using calendar software to ask and answer questions about what to do when, compare my options, and come to a decision.</p>
  <p id="p91">Consider personal finance software.  Entering and classifying my expenses is, again, tedious and unnecessary manipulation—my credit card already tracks these details.  I use the software to <em>understand</em> my financial situation and my spending habits.  How much of my paycheck goes to rent?  How much to Burrito Shack?  If I give up extra guacamole on my daily burrito, will I be able to buy a new laptop?  What is my pattern of Christmas spending, and will I have to cut back if I don’t take any jobs for a month?  If I buy a hybrid car, how much will I save on gas?  I want to ask and answer questions, compare my options, and let it guide my spending decisions.</p>
  <p id="p92">Consider an online retailer, such as Amazon or Netflix.  The entire purpose of the website—the pictures, ratings, reviews, and suggestions—is to let me find, understand, and compare their offerings.  The experience is about building a decision inside my head.  In the end, I manipulate a shopping cart, but that is merely to put my mental process to effect, to reify the decision.  At the best retailers, this manipulation is made as brief as possible.</p>
  <p id="p93">Even consider reading email.  Most current designs revolve around the manipulation of individual messages—reading them one-by-one, searching them, sorting them, filing them, deleting them.  But the <em>purpose</em> of reading email has nothing to do with the messages themselves.  I read email to keep a complex set of mental understandings up-to-date—the statuses of personal conversations, of projects at work, of invitations and appointments and business transactions and packages in the mail.  That this information happens to be parceled out in timestamped chunks of text is an implementation detail of the communication process.  It is not necessarily a good way to <em>present</em> the information to a learner.</p>
  <p id="p94">Similar arguments can be made for most software.  Ignore the structure of current designs, and ask only, “Why is a person using this?”  Abstracted, the answer almost always is, “To learn.”</p>
  <p id="p95">So far, this categorization has just been an exercise in philosophy.  But this philosophy suggests a very practical approach to software design.</p>
  <p class="logo"><img width="435" height="83" src="http://worrydream.com/MagicInk/p/logo_graphic_design.png" alt="image" /></p>
  <p id="p96">It might seem like I’m demanding a lot from my software.  But it’s not deep magic—no simulations of complex phenomena, no effects on the external world, certainly no sentience or spark of life.  I’m asking for software to display a complex set of data in a way that I can understand it and reason about it.  This is a well-established problem; it’s the raison d’etre of information graphic design.  My demands are perfect examples of graphic design challenges.</p>
  <p id="p97">A well-designed information graphic can almost <em>compel</em> the viewer to ask and answer questions, make comparisons, and draw conclusions.  It does so by exploiting the capabilities of the human eye:  instantaneous and effortless movement, high bandwidth and capacity for parallel processing, intrinsic pattern recognition and correlation, a macro/micro duality that can skim a whole page or focus on the tiniest detail.  Meanwhile, a graphic sidesteps human shortcomings:  the one-dimensional, uncontrollable auditory system, the relatively sluggish motor system, the mind’s limited capacity to comprehend hidden mechanisms.  A graphic presents no mechanisms to comprehend or manipulate—it plugs directly into the mind’s spatial reasoning centers.</p>
  <p id="p98">For example, consider this train timetable:</p>
  <p id="p99"><img width="597" height="163" src="http://worrydream.com/MagicInk/p/train_table.png" alt="image" /></p>
  <p id="p100">This design may be adequate for commuters, whose questions mostly concern when trains arrive at stations.  But train system operators have a different set of questions:  Where exactly are the trains at any given time?  How fast are they moving?  Where do two trains cross?  (They better not be on the same track at that point!)  Where are the trains at the start of the day, and where do they end up at night?  If a train is delayed, how do all these answers change?  Like some of the software questions above, these questions seem very difficult to answer.  But consider this revised timetable design:</p>
  <p id="p101"><img width="558" height="285" src="http://worrydream.com/MagicInk/p/train_graph.png" alt="image" /></p>
  <p id="p102">Each train is represented by a distinctly-colored line, with distance along the track plotted vertically and time horizontally.  The slope of the line represents the train’s direction and speed; horizontal sections are stops.  This graphic incorporates no more <em>data</em> than the previous one, yet all of the operators’ questions are answered at a glance.  Important features such as crossings are <em>emphasized</em> simply because the eye is naturally drawn toward line intersections.  Footnotes are unnecessary; the exceptions are no longer exceptional when seen in context.  Should a train be delayed, all revised stops and crossings can be “calculated” simply by drawing a new line.** Graphical train timetables date from the late 1800s.  For the origin of this and other classic graphical forms, see Howard Wainer’s book <a href="http://www.amazon.com/gp/product/0691103011">Graphic Discovery</a> (2005). 
   </p>
  <p id="p103">Compared to excellent ink-and-paper designs, most current software communicates deplorably.  This is a problem of surface, but not a superficial problem.  The main cause, I believe, is that many software designers feel they are designing a machine.  Their foremost concern is behavior—what the software <em>does</em>.  They start by asking:  What functions must the software perform?  What commands must it accept?  What parameters can be adjusted?  (In the case of websites:  What pages must there be?  How are they linked together?  What are the dynamic features?)  These designers start by specifying <em>functionality</em>, but the essence of information software is the <em>presentation</em>.</p>
  <p id="p104">It must be mentioned that there is a radically alternative approach for information software—<em>games</em>.  Playing is essentially learning <em>through</em> structured manipulation—exploration and practice instead of pedagogic presentation.  Despite the enormous potential for mainstream software, accidents of history and fashion have relegated games to the entertainment bin, and the stigma of immaturity is tough to overcome.  (The situation is similar for graphic novels.)  Raph Koster’s <a href="http://www.amazon.com/gp/product/1932111972">Theory of Fun for Game Design</a> (2004) and James Paul Gee’s <a href="http://www.amazon.com/gp/product/1403965382">What Video Games Have To Teach Us About Learning and Literacy</a> (2003) deal directly with games as learning tools.  Salen and Zimmerman’s <a href="http://www.amazon.com/gp/product/0262240459">Rules of Play</a> (2003) and Chris Crawford’s <a href="http://www.amazon.com/gp/product/1886411840">Art of Interactive Design</a> (2003) and <a href="http://www.amazon.com/gp/product/0131460994">Chris Crawford on Game Design</a> (2003) discuss learning through play in a broader context.
  <b>I suggest that the design of information software should be approached initially and primarily as a graphic design project.</b>  The foremost concern should be appearance—what and how information is presented.  The designer should ask:  What <em>is</em> relevant information?  What questions will the viewer ask?  What situations will she want to compare?  What decision is she trying to make?  How can the data be presented most effectively?  How can the visual vocabulary and techniques of graphic design be employed to direct the user’s eyes to the solution?  The designer must start by considering what the software <em>looks like</em>, because the user is using it to learn, and she learns by looking at it.</p>
  <p id="p105">Instead of dismissing ink-and-paper design as a relic of a previous century, the software designer should consider it a <em>baseline</em>.  If information software can’t present its data at least as well as a piece of paper, how have we progressed?</p>
  <h2 id="demonstration_showing_the_data">Demonstration: Showing the data</h2>
  <p id="p106">Edward Tufte’s first rule of statistical graphic design is, “<em>Show the data.</em>”  All information graphics, statistical or not, must present the viewer with enough information to answer her questions.  It seems that many software designers, in their focus on functionality, forget to actually present the data.</p>
  <p id="p107">Consider the information presented when searching a popular online bookstore.** Based on <a href="http://www.amazon.com">amazon.com</a> as of January 2006.   </p>
  <p><img width="712" height="424" src="http://worrydream.com/MagicInk/p/books_original.png" alt="image" /></p>
  <p id="p108">There are a number of graphic design criticisms one could make—the uniform text size and weight results in a solid, oppressive mass; the abundance of saturated primary colors gives a distracting, carnival-like appearance; the text is spread all over the page, giving the eye no well-defined path to follow.  However, the most egregious problem is simply that there is <em>not enough information to make any sort of decision</em>.</p>
  <p id="p109">The user’s goal is to find the best book about some particular topic.  Given that the books shown are presumably related to this topic, what questions does the user have?</p>
  <ul><li>Is the book <em>appropriate</em>?  That is, what is it about, and do I care?</li>
  <li>Is the book <em>good</em>?  That is, what did other people think of it, and do I trust them?</li>
  </ul><p id="p110">The answers will be used to compare the available books, and decide upon one to follow up on and possibly buy.</p>
  <p id="p111">Unfortunately, these questions are completely unaddressed by the information provided.  To see relevant information, the user must click on each listing individually.  That is, she must navigate by <em>hand</em> instead of by <em>eye</em>, and must use her memory to compare information across <em>time</em> instead of <em>space</em>.</p>
  <p id="p112">The problem is that this graphic was designed as an <em>index</em> into a set of webpages, but is used as a <em>catalog</em> for comparing a set of books.  The purpose of this graphic should not be to return a list of query matches, but to help the user <em>learn</em> about books related to her topic of interest.</p>
  <p id="p113">Consider this redesign:</p>
  <p><img width="800" height="559" src="http://worrydream.com/MagicInk/p/books_redesign.png" alt="image" /></p>
  <p id="p114">Is a book appropriate?  A synopsis and table of contents give an overview of the book’s contents.  Is a book good?  A rating and reviews indicate popular opinion.  Because all of this information is on a single page, it can be compared by eye, with no need for memory.</p>
  <p id="p115">The standard 5-star rating system is information-weak—it gives only an <em>average</em>.  It can be enhanced with whiskers underneath that indicate the <em>distribution</em> of ratings.  This allows the viewer to differentiate between a book that was unanimously judged middling <img src="http://worrydream.com/MagicInk/p/books_stars_1.png" alt="image" /> and one that was loved and hated <img src="http://worrydream.com/MagicInk/p/books_stars_2.png" alt="image" />—these are both 3-star ratings, but have very different meanings.  The viewer can also see whether a highly-rated book got <em>any</em> bad reviews; in a sea of praise, criticism often makes enlightening reading.  As a whole, the whiskers give a visual indication of the number of ratings, which reflects the trustworthiness of the average.  The whiskers are unobtrusive, and can easily be ignored by viewers who don’t care about distribution.</p>
  <p id="p116"><img width="200" height="107" src="http://worrydream.com/MagicInk/p/books_eyespan.png" alt="image" />
  Text weight and color is used to emphasize important information and call it out when skimming.  Text in grey can be read when focused upon, but disappears as background texture when skimming.  All critical information is contained in a column with the <b>width of an eyespan</b>, with a picture to the left and supplementary information to the right.
  The viewer can thus run her eye vertically down this column; when she spots something interesting, she will slow down and explore horizontally.</p>
  <p id="p117">The user wants to see books related to a topic in her <em>head</em>.  But ideas in the head are nebulous things, and may not translate perfectly to a concrete search term.  For this reason, a mini-list of related books is provided for each book.  <img width="200" height="52" src="http://worrydream.com/MagicInk/p/books_related.png" alt="image" />
  This is similar to a “related words” section in a thesaurus listing—it allows the user to correct a near miss, or veer off in a tangential but intriguing direction.</p>
  <p id="p118">Conventional software designers will worry about functionality—how does the user interact with this graphic?  Clearly, other than the “related books” listing, a click <em>anywhere</em> in a book’s section should reveal details and purchasing options.  What else could the user mean by clicking?  It’s analogous to pulling the book off a physical shelf.</p>
  <p id="p119">This is a significant redesign over the original; yet, I consider it a conservative one.  A more ambitious design could surely show even <em>more</em> data, perhaps allowing the user to browse within the book or fully explore the space of related books.  A world of possibilities opens up with a simple change of mindset.  This is not a list of search results—it is an information graphic.  It is for <em>learning</em>.</p>
  <h2 id="demonstration_arranging_the_data">Demonstration: Arranging the data</h2>
  <p id="p120">Just as important as <em>what</em> data is shown is <em>where</em> it is shown.  Unlike the words in a paragraph, the elements in a graphic can be deliberately <em>placed</em> to encourage spatial reasoning.  Unfortunately, most software graphics are arranged to maximize aesthetics, not to bring out useful relationships in the data.  (That is, when any skilled thought is given to appearance at all.)</p>
  <p id="p121">Consider this excerpt of a graphic for browsing nearby movie showings:** Based on <a href="http://movies.yahoo.com">movies.yahoo.com</a> as of January 2006.    </p>
  <p><img width="581" height="363" src="http://worrydream.com/MagicInk/p/movies_original.png" alt="image" /></p>
  <p id="p122">If a person is in the mood for a movie, what questions might she have?</p>
  <ul><li>What movies are showing today, at which times?</li>
  <li>What movies are showing around a <em>particular</em> time?</li>
  <li>Where are they showing?</li>
  <li>What are they about?</li>
  <li>Are they good?</li>
  </ul><p id="p123">The user will use the answers to compare the available movie showings and decide upon one to go see.</p>
  <p id="p124">Although the above graphic clearly has an information deficiency (What are these movies about?  Are they good?), the worst problem is that the data is not arranged in any useful manner.  Understanding which movies are playing when involves scanning a pageful of theaters, extracting movies of interest and mentally merging their showtimes.  A viewer’s eye might leap erratically around the screen as she compares showtimes of a given movie at six theaters, trying to find the one that best fits her dinner plans.</p>
  <p id="p125">The primary question is, “What <em>movies</em> are showing today, at which <em>times</em>?”  Given the two spatial dimensions available to us, this should suggest a graphic with <em>movies</em> along one axis and <em>times</em> along the other.</p>
  <p id="p126">Consider this redesign:</p>
  <p><img width="804" height="365" src="http://worrydream.com/MagicInk/p/movies_redesign.png" alt="image" /></p>
  <p id="p127">As with the bookstore redesign, enough information is given about each movie to determine its content and quality, although films have enough external marketing that the intent is more to <em>remind</em> than <em>introduce</em>.  Text weight is again employed to make critical information stand out and supplementary information disappear until focused upon.</p>
  <p id="p128">More interesting is the chart on the right, which plots movie showings by time.  To find all movie showings around a particular time, the viewer simply scans her eye vertically down the page.  If she is only interested in a particular movie, she looks only within that movie’s range.  The current time is indicated by shading the past, providing a springboard for the viewer’s eye; in this example, it is about 4:45.</p>
  <p id="p129">The original design grouped movies by theater; this redesign groups theaters by movie.** I assume that Yahoo! simply mimicked the newspapers, and newspapers arrange by theater for business reasons.      The assumption is that the viewer would rather see a particular movie at any theater than any movie at a particular theater.  However, to ease correlation of the various movies offered at a given theater, each theater is color-coded.  If the viewer prefers to avoid the Gulliver Theater because of sticky floors, the consistent yellow background may help her filter out its showtimes.</p>
  <p id="p130">No theater addresses are shown.  The viewer is likely to be familiar with the theaters in her area, and if she isn’t, a simple address would be useless without a map or directions.  Presumably, a mouse click or hover over a theater’s name would reveal this information, or perhaps it could be displayed elsewhere on the page.</p>
  <p id="p131">This demonstration and the previous one have attempted to illustrate the power of approaching information software as <em>graphic design</em>, instead of as styling the regurgitation of a database.  To design excellent software, however, this mindset is necessary but insufficient.  Something major is missing.</p>
  <p id="p132">Very little in the above designs is software-specific.  For the most part, the designs would work almost as well on <em>paper</em>.  Modern magic shouldn’t merely match our ancient technology—it should surpass it.  We’ve seen how graphic design can improve software, but <em>how can software improve graphic design?</em></p>
  <p id="p133">The answer lies with <em>context</em>.</p>
  <p class="logo"><img width="435" height="83" src="http://worrydream.com/MagicInk/p/logo_context_sensitivity.png" alt="image" /></p>
  <h2 id="contextsensitive_information_graphics">Context-sensitive information graphics</h2>
  <p id="p134">Print has one supreme flaw: ink is indelible.  An ink-and-paper design is static—it must display all its data, all the time.  However, a reader typically only <em>cares</em> about a subset relevant to her current situation.  The designer is faced with the challenge of organizing the data so that hopefully mutually-relevant subsets are grouped together, and the reader has the challenge of visually or physically navigating through the entire data space to find the group of interest.</p>
  <p id="p135">For example, a rider consulting a bus schedule must comb through a matrix of times and stations to find the single relevant data point—the time of the next bus.** And then, she must consult her watch and do some arithmetic to calculate the information she actually cares about—how long she will be waiting.      Any driver who’s been lost in an unfamiliar city knows the frustration of locating the immediate vicinity on a roadmap.  And a reader consulting an encyclopedia must not only find the right entry on the page and the right page in the book, but even the right book on the shelf!  These are consequences of static graphics.  Because ink is permanent, the reader must navigate through lots of paper.</p>
  <p id="p136">The modern computer system provides the first visual medium in history to overcome this restriction.  Software can:</p>
  <ul><li><b>infer the context</b> in which its data is needed,</li>
  <li><b>winnow the data</b> to exclude the irrelevant, and</li>
  <li><b>generate a graphic</b> which directly addresses the present needs.</li>
  </ul><p id="p137">Liberating us from the permanence of publication is the undersung crux of the computer—the dynamic display screen.  Its pixels are magic ink—capable of absorbing their context and reflecting a unique story for every reader.   And the components surrounding the display—CPU, storage, network, input devices—are its <em>peripherals for inferring context</em>.</p>
  <p id="p138">Information software design, then, is the design of <em>context-sensitive information graphics</em>.  Unlike conventional graphics, which must be suitable for any reader in any situation, a context-sensitive graphic incorporates <em>who</em> the user is and <em>what</em> exactly the user wants to learn at the moment.  Context allows software to winnow its data space to the subset of information that the user cares about, and present the data in such a way that the user’s current questions can best be answered.</p>
  <p id="p139">All information software consists of context-sensitive graphics, whether the designer realizes it or not.  For example,
  the list of query results from an internet search engine is a context-sensitive information graphic.  The software’s data space consists of all the websites in the world.  This is winnowed down to a dozen, using context that is inferred entirely from the search term contributed by the user.** Clicking a “next” button contributes further context—dissatisfaction with the first set of results.      Despite its enormous data space, this software restricts itself to a meager scrap of context, impersonal and imprecise.</p>
  <p id="p140">There are, in fact, three sources from which software can infer context:</p>
  <ul><li><b>Environment</b> involves sensing the current state of the world.</li>
  <li><b>History</b> involves remembering the past.</li>
  <li><b>Interaction</b> involves soliciting input from the user.</li>
  </ul><h2 id="inferring_context_from_the_environment">Inferring context from the environment</h2>
  <p id="p141">A person determines her surroundings through the five human senses.  Software doesn’t operate in a vacuum, either; through connections to hardware and other software, it can sense much about the user’s situation.  Some examples of context clues in the software’s environment include:</p>
  <ul><li><p id="p1"><strong>Date and time.</strong>  Time is one of the fundamental dimensions along which we organize our lives, and in any data space with a temporal dimension, “now” is almost always the prime landmark.  Because users often seek information on demand, information related to “now” or “soon” is often the most relevant.  Fortunately, every general-purpose computer knows when “now” is. A person using a software bus schedule, for example, should never have to hunt for the next bus.</p></li>
  <li><p id="p2"><strong>Geographical location.</strong>  Similarly, the most interesting spatial landmark is usually “here.”  Unfortunately, this currently can be harder to determine automatically, but the payoff is enormous.** I believe that location is such vital context, Powerbooks should come with <acronym title="Global Positioning System">GPS</acronym> receivers pre-installed, with an easy software <acronym title="Application Programming Interface">API</acronym>.  Developers would then write software to take advantage of it, and other computer makers would follow suit.  Someday, a computer without <acronym title="Global Positioning System">GPS</acronym> might seem as silly as a computer without a clock.      Obviously, a software roadmap needs to know the user’s location, but so does the bus schedule, as well as business listings, transportation planners, travel guides, and much other information software.</p></li>
  <li><p id="p3"><strong>Physical environment.</strong>  Given a time and location, many details of the physical environment, such as the weather, are just a network connection away.  Consider a travel guide that suggests parks when sunny and museums when rainy.</p></li>
  <li><p id="p4"><strong>Other information software</strong>, such as open websites.  By reading some information, the user is indicating a topic of interest.  All other information software should take heed.  Consider a person reading the website of an upcoming stage play.  When she opens her calendar, the available showings should be marked.  When she opens a map, she should see directions to the playhouse.  When she opens a restaurant guide, she should see listings nearby, and unless the play offers matinees, they shouldn’t be lunch joints.</p></li>
  <li><p id="p5"><strong>Documents created with manipulation software.</strong>  <em>Creating</em> some information indicates an even stronger topic of interest. Consider a person who requests information about “cats” while writing a paper.  If the paper’s title is “Types and Treatment of Animal Cancer,” the information should skew toward feline medical data.  The title “History of Egypt” indicates interest in ancient feline worship instead.  And if the paper contains terms related to building construction, “cats” probably refers to the decidedly non-feline Caterpillar heavy machinery.** This example is from Budzik and Hammond’s paper <a href="http://web.media.mit.edu/~lieber/IUI/Budzik/Budzik.pdf">User Interactions with Everyday Applications as Context for Just-in-time Information Access</a> (2000).    </p></li>
  <li><p id="p6"><strong>Email.</strong>  Names, addresses, and phone numbers in recent email clearly constitute valuable hints.  A recipient who opens a calendar should find the sender’s schedule juxtaposed with her own.  When she opens a map, addresses in the email should be marked.  But beyond that, recent correspondence can indicate current activities, and an email archive as a whole can describe the user’s characteristics and interests.  Consider a person who requests information about “racing.”  The fields of running, bicycles, and cars have distinct sets of terminology; if one set regularly shows up in the person’s conversations, “racing” isn’t so ambiguous.</p></li>
  </ul><p id="p142">All software lives within an environment, rich with evidence of context.  Using software that doesn’t look outside itself is like conversing with a blind person—constantly describing what is plainly visible.** Some of the suggestions given here may seem daunting (or infeasible) to an engineer.  Implementation will be discussed later in the paper.   </p>
  <p id="p143">On the other hand, the power of the environment is multiplied when it is correlated with the past—that is, when the software makes use of <em>history</em>.</p>
  <h2 id="inferring_context_from_history">Inferring context from history</h2>
  <p id="p144">A human doesn’t just use her senses to recognize her situation; she also uses memories of past situations.  Software, too, can use its memory to understand the present.  The current context, or a good approximation, can often be <em>predicted</em> from a history of past environments and interactions.</p>
  <p id="p145"><strong>Last-value predictors</strong> represent the simplest form of prediction.  They simply predict the current context to be the same as the previous one.  This is reasonable in many situations where the user’s context is fairly static, changing slowly over the short term.  For example, if yesterday, the user looked for one-bedroom apartments in North Berkeley, she is is probably still interested in one-bedroom apartments in North Berkeley today.  If nothing else, the software should present this information immediately, without asking for details.</p>
  <p id="p146">Last-value prediction is frequently thought of and implemented as manipulation of explicit state—that is, the context is a persistent object that remains as is unless changed by the user, so the software always appears as the user left it.** The engineering challenge then becomes merely persisting across invocations of the program.  Often, not even this is bothered with.      This stateful conceptual model mimics physical reality, and can be comfortable if the user cares enough about the software’s state to keep her own mental state in sync.  However, this is often not the case with information software, especially software that is consulted intermittently.  (If you put down a newspaper for a few hours, you won’t be distressed to find it on a different page when you return.  You probably won’t even notice.  On the other hand, you would be delighted if you often came back to find it on exactly the page you wanted to read.)  By thinking of this as context prediction instead of state maintenance, the door is opened to more sophisticated predictors.</p>
  <p id="p147"><strong>Learning predictors</strong> attempt a deeper understanding of the user.  They construct a model to explain past contexts, and use the inferred relationships to predict the current context.</p>
  <p id="p148">One simple approach to learning is to discover a common attribute of recent contexts, and narrow the current context along that attribute’s dimension.  For example, in a music library, as the user chooses several bluegrass songs in a row, the software can graphically emphasize other songs in this genre.  With further confidence, it might consider de-emphasizing or omitting songs outside of the genre.  As another example, consider a user who requests information about “Lightwave,” then about “Bryce,” then “Blender.”  These terms have many meanings individually, but as a group they are clearly names of 3D rendering software packages.  A subsequent search for “Maya,” another 3D package, should not display information about the ancient civilization.  In fact, information about Maya could be presented automatically.</p>
  <p id="p149">Another simple approach is to establish the user’s velocity through the data space.  If a person asks a travel guide about the Grand Canyon on one day, and Las Vegas the next day, the following day the software might suggest attractions around Los Angeles.** A better travel guide would suggest skipping Los Angeles.   </p>
  <p id="p150">In general, the problem is one of inferring a <em>pattern</em> that explains the user’s interests as a function of the environment, and extrapolating along the pattern to classify the current environment.  As an example of general pattern modeling, consider a person who, as a byproduct of traveling to work, always checks the train schedule from Berkeley to San Francisco in the morning, and San Francisco to Berkeley in the evening.  If the software can discover and model this pattern, it can present the appropriate information at each time without the user having to request it.  When she looks in the morning, she sees by default the San Francisco-bound schedule; in the evening, the Berkeley-bound schedule.** Again, this may sound daunting to an engineer.  Implementation will be discussed later in the paper.    </p>
  <p id="p151">Large histories can enable some very sophisticated predictors, especially if it is possible to reach into the environment and correlate with other users’ histories.  For example, by asking their users to rate each movie they return, Netflix is able to infer some enormously valuable context—each user’s <em>taste</em>.  This allows them to winnow an enormous dataset (their catalog of movies) down to a dozen data points (movies the user hasn’t seen, which were enjoyed by people with similar taste), which can be presented in a single, navigation-free graphic.  The winnowing is impressively on-target—two-thirds of users’ selections come from recommendations.** Laurie J. Flynn, <a href="http://www.nytimes.com/2006/01/23/technology/23recommend.html">Like This? You’ll Hate That.</a>  <em>New York Times</em>, Jan. 23, 2006.     TiVo similarly uses a collaborative predictor to infer which television programs the user would be interested in.  These are presented on a “suggestions” page, and recorded automatically when possible.** For technical details, see Ali and van Stam’s paper <a href="http://portal.acm.org/citation.cfm?id=1014052.1014097">TiVo: Making Show Recommendations Using a Distributed Collaborative Filtering Architecture</a> (2004).    </p>
  <p id="p152">Amazon, iTunes, and an increasing number of other online retailers are currently incorporating similar schemes.  However, with the exception of the lowly junk-mail filter, non-retail information software that learns from history is still rare.  Typically, users can only hope for last-value prediction, if that.  Most software wakes up each day with a fresh case of amnesia.</p>
  <p id="p153">Unfortunately, software that doesn’t learn from history dooms users to repeat it.  And repeat it they will—tediously explaining their context, mouse click by mouse click, keystroke by keystroke, wasted hour by wasted hour.  This is called <em>interactivity</em>.</p>
  <p class="logo"><img width="435" height="83" src="http://worrydream.com/MagicInk/p/logo_interactivity.png" alt="image" /></p>
  <h2 id="interactivity_considered_harmful">Interactivity considered harmful</h2>
  <p id="p154">Chris Crawford defines interaction as a three-phase reciprocal process, isomorphic to a conversation:  an interactant <b>listens</b> to her partner, <b>thinks</b> about what was said, and <b>speaks</b> a response.  Her partner then does the same.** See Crawford’s book <a href="http://www.amazon.com/gp/product/1886411840">The Art of Interactive Design</a> (2003), or his essay <a href="http://www.erasmatazz.com/library/the-journal-of-computer/jcgd-volume-7/fundamentals-of-interactivi.html">Fundamentals of Interactivity</a> (1993).    For <em>manipulation</em> software, interaction is perfectly suitable: the user views a visual representation of the model, considers what to manipulate next, and performs a manipulation.  The software, in turn, inputs the user’s manipulation request, updates the model, and displays the updated representation.  With good feedback and an effective means of “speaking” to the software, this process can cycle smoothly and rapidly.  It mimics the experience of working with a physical tool.</p>
  <p id="p155">Information software, by contrast, mimics the experience of <em>reading</em>, not working.  It is used for achieving an understanding—constructing a model within the mind.  Thus, the user must <em>listen</em> to the software and <em>think</em> about what it says… but any manipulation happens mentally.** Except possibly for signaling a decision, such as clicking a “buy” button, but that <em>concludes</em>, not <em>constitutes</em>, a session.     The only reason to complete the full interaction cycle and <em>speak</em> is to explicitly provide some context that the software can’t otherwise infer—that is, to indicate a relevant subset of information.  <b>For information software, all interaction is essentially navigation around a data space.</b></p>
  <p id="p156">For example, Amazon’s data space consists of their catalog of items.  For a yellow pages directory, the data space contains all business listings; for a movie guide, all showtimes and movie information; for a flight planner, trips to and from all airports.  In all of these cases, every interaction, every click and keystroke, search term and menu selection, simply serves to adjust the user’s view into the data space.  This is simply <em>navigation</em>.</p>
  <p id="p157">Alan Cooper defines <em>excise</em> in this context as a cognitive or physical penalty for using a tool—effort demanded by the tool that is not directly in pursuit of a goal.  For example, filling a gas tank is done to support the car, not the goal of arriving at a destination.  Cooper goes on to assert that software navigation is nothing but excise:</p>
  <blockquote>
    <p id="p46">…the most important thing to realize about navigation is that, in almost all cases, it represents pure excise, or something close to it.  Except in games where the <em>goal</em> is to navigate successfully through a maze of obstacles, navigation through software does not meet user goals, needs, or desires.  Unnecessary or difficult navigation thus becomes a major frustration to users.  In fact, it is the authors’ opinion that poorly designed navigation presents the <strong>number-one problem</strong> in the design of any software application or system…** Alan Cooper and Robert Reimann, <a href="http://www.amazon.com/gp/product/0764526413">About Face</a> (2003), p143. 
     </p>
  </blockquote>
  <p id="p158">If all interaction is navigation, and navigation is the number-one software problem, interactivity is looking pretty bad already.  However, when compared with the other two sources of context, interactivity has even <em>worse</em> problems than simply being a frustrating waste of time:</p>
  <ul><li><p id="p7">The user has to already know <em>what</em> she wants in order to ask for it.  Software that infers from history and the environment can proactively offer potentially relevant information that the user wouldn’t otherwise know to ask for.  Purely interactive software forces the user to make the first move.</p></li>
  <li><p id="p8">The user has to know <em>how</em> to ask.  That is, she must learn to manipulate a machine.  Donald Norman’s concept of determining a user’s “mental model” has become widespread in the software usability community, and is now considered a core design challenge.** See Donald Norman’s book <a href="http://www.amazon.com/gp/product/0465067107">The Design of Everyday Things</a> (2002), p9.      However, Norman described this concept in the context of <em>mechanical</em> devices.  It only applies to software if the software actually contains hidden mechanisms that the user must model.  A low-interaction, non-mechanical information graphic relieves both user and designer from struggling with mental models.</p></li>
  <li><p id="p9">Navigation implies state.  Software that can be navigated is software in which the user can get lost.  The more navigation, the more corners to get stuck in.  The more manipulable state, the more ways to wander into a “bad mode.”  State is the primary reason people fear computers—stateful things can be <em>broken</em>.** The only state kept by a book is which page it is open to, which is why “getting lost in a book” describes a <em>pleasurable</em> experience!   </p></li>
  </ul><p id="p159">Beyond these cognitive problems are <em>physical</em> disadvantages of interaction.  The hand is much slower than the eye.  Licklider described spending hours plotting graphs and seconds understanding them.  A user who must manually request information is in a similar situation—given the mismatch between mousing and reading speeds, most of her time may be spent navigating, not learning.  Further, the user might prefer to learn information while using her hands for other purposes, such as writing or eating or stroking a cat.  Each time software demands the user’s hands, this activity must be interrupted.  Finally, the growing prevalence of computer-related repetitive stress injuries suggests that indiscriminate interactivity may be considerably harmful in a literal, physical sense.</p>
  <p id="p160">Unless it is enjoyable or educational in and of itself, interaction is an essentially <em>negative</em> aspect of information software.  There is a net positive benefit if it significantly expands the range of questions the user can ask, or improves the ease of locating answers, but there may be other roads to that benefit.  As suggested by the above redesigns of the train timetable, bookstore, and movie listings, many questions can be answered simply through clever, information-rich graphic design.  Interaction should be used <em>judiciously</em> and <em>sparingly</em>, only when the environment and history provide insufficient context to construct an acceptable graphic.</p>
  <p id="p161">It is unfortunate that the communities concerned with human factors of electronic artifacts have latched onto the term “interaction.”** Most professional communities and academic programs use the term Human-Computer Interaction, or <acronym title="Human-Computer Interaction">HCI</acronym>; the <acronym title="Association for Computing Machinery">ACM</acronym> special-interest group is <a href="http://www.sigchi.org/">CHI</a>, the converse.  Many practitioners, following Cooper and Bill Moggridge, refer to their profession as “interaction design.”      <b>For information software, the real issue is <em>context-sensitivity</em>.</b>  Interaction is merely one means of achieving that.  And as long as “speaking” is constrained to awkwardly pushing metaphors with a mouse, interaction should be the last resort.</p>
  <p id="p162">The working designer might protest that interaction is unavoidable in practice, and may even consider my ideal of interaction-free software to be a scoff-worthy fantasy.  This is only because the alternatives have been unrecognized and underdeveloped.  I believe that with the invention of new context-sensitive graphical forms and research into obtaining and using environment and history, the clicking and dragging that characterizes modern information retrieval will be made to seem laughably archaic.  But every condonation of “interactivity,” from the annals of academia to the corporate buzzvocabulary, postpones this future.</p>
  <h2 id="reducing_interaction">Reducing interaction</h2>
  <p id="p163">When the user is forced to interact, the software assumes the form of manipulation software.  The external model, manipulated through navigation, is the software’s model of the context.
  <img width="200" height="128" src="http://worrydream.com/MagicInk/p/interaction_model.png" alt="image" />
  However, unlike genuine manipulation software, the user does not <em>care</em> about this model—it is merely a means to the end of seeing relevant information.</p>
  <p id="p164">The designer’s goal is to let the user adequately shape the context model with as little manipulation as possible.  Assuming that graphic design, history, and the environment have been taken as far as they will go, there are a few techniques that can lessen the impact of the remaining interaction:</p>
  <ul><li><b>Graphical manipulation</b> domains present the context model in an appropriate, informative setting.</li>
  <li><b>Relative navigation</b> lets the user <em>correct</em> the model, not <em>construct</em> it.</li>
  <li><b>Tight feedback loops</b> let the user stop manipulating when she’s close enough.</li>
  </ul><p id="p165"><strong>Graphical manipulation.</strong>  Command-line systems are criticized for forcing the user to learn the computer’s language.  Modern GUIs may be easier to use, but they are not much different in that respect.  The <acronym title="Graphical User Interface">GUI</acronym> language consists of a grammar of menus, buttons, and checkboxes, each labeled with a vocabulary of generally decontextualized short phrases.  The user “speaks” by selecting from a tiny, discrete vocabulary within an entirely fixed grammatical structure—a bizarre pidgin unlike any human language, unexpressive and unnatural.** One might wonder what Sapir and Whorf would conclude.   </p>
  <p id="p166">As an alternative, consider a child describing his toy at “Show and Tell”:** From Scott McCloud’s book <a href="http://www.amazon.com/gp/product/006097625X">Understanding Comics</a> (1994), p138.   </p>
  <p><img width="800" height="393" src="http://worrydream.com/MagicInk/p/mccloud.png" alt="image" /></p>
  <p id="p167">Because the child’s “telling” skills are underdeveloped, he communicates complex concepts through <em>showing</em>.  Similarly, a <acronym title="Graphical User Interface">GUI</acronym>’s stunted grammar makes telling tedious, but software’s dynamic display is ideal for showing.  A user can specify context by pointing somewhere on an information graphic and saying, “There!”</p>
  <p id="p168">Two of the most fundamental context dimensions are <em>where</em> and <em>when</em>.  For millennia, people have described these concepts with specialized information graphics.  But much modern software abandons this tradition, as seen on the website of a popular moving company:** Based on <a href="http://www.uhaul.com">uhaul.com</a> as of January 2006.    </p>
  <p><img width="531" height="264" src="http://worrydream.com/MagicInk/p/uhaul_original.png" alt="image" /></p>
  <p id="p169">These drop-down menus are awkward and uninformative.  Geographical locations belong on maps, and dates belong on calendars.  Consider this redesign:</p>
  <p id="p170">Even this is not ideal.  Locations and dates should be chosen from the user’s <em>own</em> map and calendar.  But until platforms that enable such a thing are widespread, software can at least provide temporary ones.
  <img width="597" height="401" src="http://worrydream.com/MagicInk/p/uhaul_redesign.png" alt="image" /></p>
  <p id="p171">As an example of more application-specific context, a prominent online flower shop lets the user narrow the view via a set of drop-down menus.** Based on <a href="http://www.teleflora.com/searchrequest.asp">teleflora.com</a> as of January 2006.     Compare it with a simple visually-oriented redesign:</p>
  <p><img width="540" height="357" src="http://worrydream.com/MagicInk/p/flowers.png" alt="image" /></p>
  <p id="p172">Many types of context can be naturally expressed in some informative graphical domain, relieving the user from manipulating information-free general-purpose controls.  Several more examples will be given in the case study below.</p>
  <p id="p173"><strong>Relative navigation.</strong>  If the software properly infers as much as possible from history and the environment, it should be able to produce at least a reasonable <em>starting point</em> for the context model.  Most of the user’s interaction will then consist of <em>correcting</em> (or confirming) the software’s predictions.  This is generally less stressful than constructing the entire context from scratch.</p>
  <p id="p174">For example, Google Maps offers both absolute navigation (typing in an address) and relative navigation (panning and zooming the current map).  However, it initially displays by default the entire continent; this effectively demands that the user type in an absolute location to get started.** The user can manually specify an initial location, but she will presumably set this to her home.  Ironically, her own neighborhood is the <em>least</em> likely place she’ll need mapped.     A better design might start at the last place the user looked (last-value prediction), with a nearby list of locations predicted by history (recently visited or manually bookmarked) and the environment (addresses extracted from email, open websites, and calendar software).  A reasonable starting point would almost always be a click away, and from there, the user could use relative navigation (dragging and zooming) or simply “navigate” by eye if the graphic is detailed enough.  Conceptually, a prediction list might itself be considered relative navigation, as a set of “shortcuts” through the data space.</p>
  <p id="p175">An even better design would recognize the prediction list as information software in its own right, and would take steps to <em>show the data</em> (such as annotating the predictions with driving times to and from common locations, taking current traffic conditions into account) and <em>arrange the data</em> (perhaps spatially arranging the predictions on their own map).  This might answer most of the user’s questions without any interaction at all.</p>
  <p id="p176"><strong>Tight feedback loops.</strong>  Salen and Zimmerman offer a game design perspective on a principle that is crucial for all interactive software:</p>
  <blockquote>
    <p id="p47">If you shoot an asteroid while playing a computer game and the asteroid does not change in any way, you are not going to know if you actually hit it or not.  If you do not receive feedback that indicates you are on the right track, the action you took will have very little meaning.  On the other hand, if you shoot an asteroid and you hear the sound of impact, or the asteroid shudders violently, or it explodes (or all three!) then the game has effectively communicated the outcome of your action.** Katie Salen and Eric Zimmerman, <a href="http://www.amazon.com/gp/product/0262240459">Rules of Play</a> (2003), p35.    </p>
  </blockquote>
  <p id="p177">This principle is universal.  If the user clicks a checkbox and <em>nothing happens</em>, her action is rendered ambiguous or even meaningless.  She cannot evaluate a response and let it guide her next action.  In terms of Crawford’s conversation metaphor, the software is failing to speak back—she is shouting into the wind.</p>
  <p id="p178">For information software in particular, all interaction specifies context.  Thus, <b>each interaction can and should result in a discernible change to a context-sensitive information graphic.</b>  Providing immediate feedback reduces the amount of manipulation the user must do before either reaching an adequate view or recognizing a wrong turn and backtracking.</p>
  <p id="p179">Any web form with a “submit” button or dialog box with an “accept” button fails this point.  Google Maps offers reasonable feedback during relative navigation, but none during absolute navigation, such as typing in an address.  Even a simple predictive auto-complete would be helpful, but consider the possibilities suggested by Ben Fry’s <a href="http://benfry.com/zipdecode/">zipdecode</a> applet.  (First click “zoom” in the lower-right, then type in numbers.)  Imagine honing in on familiar areas simply by typing the first few digits of a zip code—type “9” to immediately zoom into the US west coast, followed by “4” to zoom into the SF bay area and then “5” for the east bay.  Because of the immediate feedback, the user can stop typing when she gets close enough, and use relative navigation from there.</p>
  <h2 id="how_did_we_get_here">How did we get here?</h2>
  <p id="p180">Much current software is interaction-heavy and information-weak.  I can think of a few reasons for this.</p>
  <p id="p181">First, our current UI paradigm was invented in a different technological era.  The initial Macintosh, for example, had no network, no mass storage, and little inter-program communication.  Thus, it knew little of its <em>environment</em> beyond the date and time, and memory was too precious to record significant <em>history</em>.  Interaction was <em>all it had</em>, so that’s what its designers used.  And because the computer didn’t have much to inform anyone of, most of the software at the time was manipulation software—magic versions of the typewriter, easel, and ledger-book.  Twenty years and an internet explosion later, software has much more to say, but an inadequate language with which to say it.** Make no mistake, I revere <acronym title="Graphical User Interface">GUI</acronym> pioneers such as Alan Kay and Bill Atkinson, but they were inventing rules for a different game.  Today, their windows and menus are like buggy whips on a car.  (Although Alan Kay clearly <em>foresaw</em> today’s technological environment, even in the mid-’70s.  See “A Simple Vision of the Future” in his fascinating <a href="http://www.smalltalk.org/smalltalk/TheEarlyHistoryOfSmalltalk_V.html">Early History of Smalltalk</a> (1993).) 
   </p>
  <p id="p182">A second reason why modern software is dominated by mechanical metaphors is that, for the people who create software, the computer <em>is</em> a machine.  The programmer lives in manipulation mode; she drives her computer as if it were a car.  Thus, she inadvertently produces software that must be operated like a machine, even if it is <em>used</em> as a newspaper or book.  Worse, the people who design platforms and <acronym title="Graphical User Interface">GUI</acronym> toolkits are even more prone to this perspective, since they work at a lower level.  The application software designer is then almost forced into a mechanical model by the design environment.** Apple’s Interface Builder, for example, makes it simple to place buttons, sliders, and blocks of text.  Dynamic graphics, the cornerstone of information software, must be tediously programmed with low-level constructs.    </p>
  <p id="p183">Even software that starts out information-rich and interaction-simple tends to accumulate wasteful manipulation as features are added over successive versions.  It’s easier on both the designer and the programmer to plug in another menu item and dialog box than to redesign a dynamic graphic, and sometimes it’s justified as a less jarring change for the user.  After ten versions, the software can grow into a monstrosity, with the user spending more time pulling down menus than studying and learning information.</p>
  <p id="p184">Software doesn’t have to be this way, but the solution will require a significant re-thinking of both the design process and the engineering platforms.  After a detailed case study of one recent design, I will discuss what’s needed to usher in the information software revolution.</p>
  <p class="logo"><img width="435" height="83" src="http://worrydream.com/MagicInk/p/logo_intermission.png" alt="image" /></p>
  <div id="case_study">
  <h2 id="case_study_train_schedules">Case study:  Train schedules</h2>
  <p id="p185">I recently created a program for planning trips on <acronym title="Bay Area Rapid Transit">BART</acronym>, the San Francisco bay area subway system, in the form of a “Dashboard widget” (mini-application) for the Apple Macintosh.</p>
  <p class="image"><img width="350" height="215" src="http://worrydream.com/MagicInk/p/widget_first.png" alt="image" /></p>
  <p id="p186">The design has clearly been successful.  Even though the target audience is fairly small (SF bay area public transportation riders with the latest Mac OS and knowledge of how to customize it), user feedback has been <a href="http://worrydream.com/bartwidget/praise.html">wildly enthusiastic</a>, and the widget received the 2007 Apple Design Award, as well as Macworld magazine's rare perfect <img src="http://worrydream.com/MagicInk/p/macworld_mice.gif" alt="image" /> rating.  If you are unfamiliar with the widget, you can watch a one-minute demo movie:</p>
  <p class="image"><a href="http://worrydream.com/MagicInk/bart_widget_demo.html"><img width="220" height="135" src="http://worrydream.com/MagicInk/p/widget_movie.png" alt="image" /></a></p>
  <p id="p187">As information software, the widget was approached primarily as a graphic design project.  I will discuss how its design exemplifies the viewpoints in this paper, and also point out where it falls short and could be improved.** The widget originally inspired this paper, not vice-versa.  Thus, the widget does not reflect new ideas conceived while writing this.  (Yet!)      I will also compare it to the trip planner on the official <acronym title="Bay Area Rapid Transit">BART</acronym> website, which follows the typical mechanical paradigm of drop-down menus, “submit” button, and table of results.</p>
  <p id="p188">The <acronym title="Bay Area Rapid Transit">BART</acronym> widget was designed around three classical forms of graphical communication:  the timeline, the map, and the sentence.</p>
  <h2 id="showing_the_data">Showing the data</h2>
  <p id="p189">Information software allows the user to ask and answer questions, make comparisons, and draw conclusions.  In the case of trip planning, some questions are:</p>
  <ul><li>When is the next train leaving?  How long is that from now?</li>
  <li>When is that train arriving?  How long is that from now?</li>
  <li>Which line is that train on?</li>
  <li>Does that trip have a transfer?  If so, when, where, and for how long?</li>
  <li>What about the train after that?  And after that?</li>
  <li>How frequently do the trains come?</li>
  <li>What about trains around 7:00 am on Tuesday?</li>
  </ul><p id="p190">Users use the answers to compare the available trips, and draw a conclusion about which to take.  Naturally, it must be possible for that conclusion to take the form of a plan: “Which train will I take?  I will take the 7:32 train.”  However, the plan then becomes a mental burden on the user.  A good design would also allow for a series of quick boolean conclusions over time: “Should I start walking to the station <em>now</em>?  No… no… no… okay, let’s go.”</p>
  <p id="p191">The choice of graphical representation depends on what sort of data space is left after context-based winnowing.  What context can be inferred?</p>
  <p id="p192">The user is expecting to leave around a particular time; thus, the graphic can exclude trips outside of some narrow time window.  Furthermore, the most common time is “soon”; thus, the software can initially assume that the time window is “the near future.”  Also, notice that all of the questions implicitly refer to a single route—a particular origin and destination pair.  That is, the user wants to compare trips along the time dimension, but not the space dimensions.  Thus, the graphic need only concern itself with a single route, which we last-value predict to be “the same as last time.”** A learning predictor for the route is presented later in the paper.   </p>
  <p id="p193">After winnowing the data, we are left with a handful of trips—ordered, overlapping spans of time.  We need a graphical construct that allows the viewer to compare the start, end, and length of each span.  A natural choice is a <em>time bar graph</em>, which allows for important qualitative comparisons at a glance:  When does each span start and end?  How long is each span?  How close together are they?
  The time bar graph may have been invented by proto-chemist Joseph Priestly in 1765 to compare the <a href="http://www.math.yorku.ca/SCS/Gallery/images/priestley.gif">lifespans</a> of various historical figures.  Priestly’s chart inspired William Playfair to invent the modern statistical bar graph.  Howard Wainer claims to have uncovered a bar graph from 3000 years earlier, plotting population changes in the tribes of Isreal after the exodus.  See <a href="http://www.amazon.com/gp/product/0691103011">Graphic Discovery</a> (2005), p18.</p>
  <p class="image"><img width="353" height="173" src="http://worrydream.com/MagicInk/p/widget_unmarked_1.png" alt="image" /></p>
  <p id="p194">The most important context, the current time, can be emphasized by shading the past.  The most important data point, the next train, can be emphasized by keeping it in a constant location, the second row.  This answers the most important qualitative questions:  Is the next train coming soon?  Did I just miss a train?  For an experienced viewer, the conclusive question, “Should I start walking to the station <em>now</em>?”, can be answered literally at a glance.</p>
  <p class="image"><img width="353" height="173" src="http://worrydream.com/MagicInk/p/widget_unmarked_2.png" alt="image" /></p>
  <p id="p195">The graphic can then be unobtrusively annotated with quantitative information, so closer inspection answers all of the questions precisely:</p>
  <p class="image"><img width="350" height="215" src="http://worrydream.com/MagicInk/p/widget_marked.png" alt="image" /></p>
  <p id="p196">Transfers can be regarded as segmentation of the overall trip.  The question that must be answered exactly is <em>where</em> to transfer.  The questions of when and how long should be answered qualitatively; the exact times would be irrelevant clutter.** A better design would probably place the transfer station name closer to the graphical representation of the transfer, instead of over to the side.   </p>
  <p class="image"><img width="350" height="215" src="http://worrydream.com/MagicInk/p/widget_transfer.png" alt="image" /></p>
  <p id="p197"><strong>And that’s about it.</strong>  Although there clearly is more to the widget than this, most of the “user experience” is represented by the picture above.  That is, this software is normally “used” by simply <em>looking</em> at it, with no interaction whatsoever.  In contradiction to the premise of interaction design, this software is at its best when acting <em>non-interactively</em>.</p>
  <p id="p198">Accordingly, all interactive mechanisms—the buttons and bookmarks list—are hidden when the mouse pointer is outside the widget.  Unless the user deliberately wants to interact with it, the widget appears as a pure information graphic with no manipulative clutter.** Tufte uses the term “administrative debris.”    </p>
  <p id="p199">Of course, if the predicted context is wrong, the user must interact to correct it.  This involves navigation in the usual two dimensions, <em>time</em> and <em>space</em>.</p>
  <p id="p200">The widget initially assumes a time window of “the near future.”** This window changes over time, of course.  The widget naturally stays in sync, always displaying relevant information.  A button to manually “refresh” the display would be almost obscenely mechanical.      There are two cases in which this context is incorrect:</p>
  <ul><li>The user wants to see even later trips.</li>
  <li>The user wants to plan for some other time entirely.</li>
  </ul><p id="p201"><strong>Relative navigation.</strong>  To see earlier or later trips, the user can simply drag the graphic around.  A cursor change suggests this, as well as a brief message when the widget is first started.** <img src="http://worrydream.com/MagicInk/p/widget_drag_message.png" alt="image" /> 
   
  The mouse scrollwheel and keyboard arrow keys also serve to navigate through time.  The “underlying” graphic is infinite—the user can scroll forever.  Thus, a <acronym title="Graphical User Interface">GUI</acronym> scrollbar would be inappropriate.</p>
  <p id="p202"><strong>Absolute navigation.</strong>  To plan around an arbitrary time, the user clicks a button to reveal the hours of the day, from morning to night, laid out linearly.  The user can then click anywhere on the mechanism to jump to that time.</p>
  <p class="image"><img width="354" height="31" src="http://worrydream.com/MagicInk/p/widget_timebar.png" alt="image" /></p>
  <p id="p203">The mechanism’s labeling is intentionally vague, so the user will click approximately in the right area, and then continue to drag left or right until the correct information is displayed on the chart of train schedules.  This forces the user to keep her eyes on the information graphic, instead of wasting effort precisely manipulating the navigation mechanism.** This is the same concept suggested by the Google Maps prediction list above.  Instead of precise, tedious absolute navigation, offer quick ballpark navigation, followed by relative navigation in a tight feedback loop.      Unlike the time of day, the predicted date (today) is probably close—few people plan subway trips weeks in advance.  Thus, the date control is relative.</p>
  <p id="p204">The assumed context includes where the user is coming from and where she is going.  The assumption is “the same as last time”; that is, this appears as explicit state.  There are three cases for which the context is incorrect.</p>
  <p id="p205">The most common case is that the user is making a round trip, and wants to come home.  The “reverse route” button serves this case.** A better design could probably eliminate this interaction with a predictor as described above (and implemented below), or a graphic that somehow incorporates both directions at once.   </p>
  <p class="image"><img width="353" height="39" src="http://worrydream.com/MagicInk/p/widget_reverse.png" alt="image" /></p>
  <p id="p206">The second case is that the user is making a common trip, and knows exactly where she wants to go.  The bookmarks feature serves this case.  When the user clicks the heart button
  <img src="http://worrydream.com/MagicInk/p/widget_heart.png" alt="image" />,
  the trip is added to a bookmarks list.  From then on, that trip and its reverse can be selected with a click.  No manipulation is needed to bring up the bookmarks list—it slides out when the mouse is over the widget.** A better design might further reduce interaction by annotating each bookmarked trip with its next depart time.  In many cases, that would eliminate the need to even click on the bookmark.  Another improvement would be to automatically infer “bookmarks” from recent trips or environmental clues. 
   </p>
  <p class="image"><img width="348" height="101" src="http://worrydream.com/MagicInk/p/widget_bookmarks.png" alt="image" /></p>
  <p id="p207">The most interesting case is the least common, but the most stressful for the user—selection of an unfamiliar station.  The user needs information to decide which station to travel to; thus, this can be approached as an information software problem in itself.  Some questions the user might have:</p>
  <ul><li>Where are the stations?</li>
  <li>What are the lines?</li>
  <li>What order are the stations on a particular line?</li>
  <li>Which stations are near a particular area?</li>
  </ul><p id="p208">These questions involve orientation and navigation in a physical two-dimensional space.  The standard graphical device for this situation is the map.  The map allows the user to ask and answer the above questions, make comparisons among the available stations, and decide which station she’s looking for.This map courtesy of <a href="http://www.newmediasoup.com/">newmediasoup</a>.</p>
  <p class="image"><img width="460" height="402" src="http://worrydream.com/MagicInk/p/widget_map_full.png" alt="image" /></p>
  <p id="p209">Once the user has decided, she must indicate her selection to the software.  This manipulation can be done in the same graphical domain as the information.  “From” and “To” markers appear directly on the map; these are dragged to the desired station.  Instead of having to name the station, the user effectively points at the map and says, “There!”  Although it is less important in this case, the feedback loop remains tight; the train chart updates as the markers are moved.** Widgets are expected to have a small screen footprint, which is why the map can be hidden.  Ideally, the map would always be visible.  A better design might then overlay dynamic information on the map, such as the positions of the trains and arrival times at stations. 
   </p>
  <p class="image"><img width="350" height="513" src="http://worrydream.com/MagicInk/p/widget_map_open.png" alt="image" /></p>
  <h2 id="configuring_notifications">Configuring notifications</h2>
  <p id="p210">Instead of the user continually asking “Should I start walking to the station now?”, she might prefer the software to notify her directly:  “Start walking to the station now!”  Audio works well for infrequent, asynchronous notifications.  The widget can speak announcements of upcoming trains. (Hear a <a href="http://worrydream.com/MagicInk/bart_widget_voice.mp3">sample</a>.) Vocal announcements were originally a semi-hidden Easter Egg, but they got enough of a user response that they were moved up to first-class feature.</p>
  <p id="p211">The design challenge is allowing the user to express if and when she wants announcements.  For example, if the user is about ready to go home and it’s a twelve-minute walk to the <acronym title="Bay Area Rapid Transit">BART</acronym> station, she would want the software to announce trains departing in twelve minutes.  But if she’s meeting a friend at the station, she would want to hear about trains <em>arriving</em> in twelve minutes.  Normally, of course, she doesn’t want to hear anything at all.</p>
  <p id="p212">A typical design would use a preference dialog or form that the user would manipulate to tell the software what to do.  However, an information design approach starts with the converse—the <em>software</em> must explain to the <em>user</em> what it will do.  It must graphically express the current configuration.</p>
  <p id="p213">For presenting abstract, non-comparative information such as this, an excellent graphical element is simply a concise sentence.** Chris Crawford discusses the relative merits of pictorial and textual representation in his essay <a href="http://www.erasmatazz.com/library/the-journal-of-computer/jcgd-volume-8/representation-versus-depic.html">Representation Versus Depiction</a> (1994).   </p>
  <p class="image"><img width="310" height="55" src="http://worrydream.com/MagicInk/p/widget_demo_sentence.png" alt="image" /></p>
  <p id="p214">As with the map, once the information graphic is established, manipulation can be incorporated.  In this case, some words are colored red, and the user can click on these words to change them.** Numerical and time parameters transform into edit controls when clicked—the idea for this was inspired by Jeremy Ruston’s wonderful <a href="http://www.tiddlywiki.com/">TiddlyWiki</a>.   </p>
  <p class="image"><img width="352" height="546" src="http://worrydream.com/MagicInk/p/widget_sentences.png" alt="image" /></p>
  <p id="p215">The user always sees the software presenting information, instead of herself instructing the software.  If the information presented is wrong, the user corrects it in place.  There is no “OK” or confirmation button—the sentence <em>always</em> represents the current configuration.  The graphic fades out when the mouse is clicked outside of it or the mouse leaves the widget.</p>
  <p id="p216">This approach scales well to more complex configuration.  The widget allows spoken announcements to be associated with a bookmark and a particular time.  This is useful for daily trips, such as to and from work.  The user thinks, “It takes me 15 minutes to drive to <acronym title="Bay Area Rapid Transit">BART</acronym>, it takes ten minutes to walk from the station to work, and I have to be at work by 8.”  This graphic represents her thought precisely, as well the trip home:</p>
  <p class="image"><img width="352" height="272" src="http://worrydream.com/MagicInk/p/widget_big_sentences.png" alt="image" /></p>
  <p id="p217">Sentence-based configuration scales so well because parameters are given meaning by the surrounding textual context, which can itself consist of other parameters.  A typical configuration dialog box attempts to express each parameter in isolation, resulting in intimidating (or bewildering) verbosity:** Some people claim that no interface can be fundamentally more “intuitive” than another, because intuition is simply a result of familiarity.  But surely these people were parsing and producing complete sentences long before they could manage a dialog box.  The human brain actually does have some hard-wiring. 
   </p>
  <p class="image"><img width="510" height="440" src="http://worrydream.com/MagicInk/p/widget_strawman.png" alt="image" /></p>
  <p id="p218"><img width="200" height="135" src="http://worrydream.com/MagicInk/p/widget_tail.png" alt="image" />
  Some additional graphical touches help bring the design together.  The sentence is contained within a cartoon speech bubble which, beyond simply looking cute, implies that the activity pertains to speech, and points via the tail to the button which spawned it and the trip to which it refers.  More importantly, if a voice announcement is activated, the button’s icon changes to an active speaker.
  <img src="http://worrydream.com/MagicInk/p/widget_active_speakers.png" alt="image" />
  This avoids a “hidden mode” problem by providing a clear visual indication of where the voice is coming from and how to turn it off.</p>
  <h2 id="comparison">Comparison</h2>
  <p id="p219">The trip planner on the official <acronym title="Bay Area Rapid Transit">BART</acronym> website refuses to divulge any information whatsoever without a sequence of menu selections and a button-push.** Based on <a href="http://www.bart.gov">bart.gov</a> as of January 2006. 
   </p>
  <p class="image"><img width="510" height="209" src="http://worrydream.com/MagicInk/p/widget_bart_planner.png" alt="image" /></p>
  <p id="p220">Because the <acronym title="Bay Area Rapid Transit">BART</acronym> system is two-dimensional, no linear arrangement of the stations can convey useful information.  Instead, the stations are listed alphabetically, but because many stations go by several names (“Berkeley” or “Downtown Berkeley”?  “Oakland City Center / 12th St.,” “City Center / 12 St.,” or “12th St”?) the selection is difficult even for those familiar with the system.  The user can click a link to see a map, but the map graphic is static; the selection must be made through drop-down menus.  Information and navigation are completely segregated, and the feedback loop is enormous.</p>
  <p id="p221">The results screen shows <em>no</em> useful information at a glance:</p>
  <p><img width="530" height="543" src="http://worrydream.com/MagicInk/p/widget_bart_results.png" alt="image" /></p>
  <p id="p222">The starting and ending stations, always the same, clutter the results.  Transfers are treated as two separate trips, and the relevant times (the start and end of the entire trip) are in opposite corners, with distracting clutter in between.  Not only does the information not stay in sync with the current time, there is no relative time information at all.  Other than a “later times” link (which leads to a page with only an “earlier times” link!) navigation through time or space requires hitting the back button and working a drop-down menu.  </p>
  <p id="p223">For all its interactivity, the information here is sparse, poorly presented, and hard to get to.  Yet, this sort of design is so typical of software on all platforms, it has almost become an accepted norm.  For many people, this is “how computers work.”</p>
  <h2 id="conclusion">Conclusion</h2>
  <p id="p224">Ironically, the <acronym title="Bay Area Rapid Transit">BART</acronym> widget appears so fresh because its underlying ideas are so <em>old</em>.  The time bar graph was invented about 250 years ago.  The map and the written sentence are both about 5000 years old.  They are beautiful, venerable forms of visual communication.  The bugs have been worked out.  They are universally, intuitively understood.</p>
  <p id="p225">The pulldown menu, the checkbox, and the bureaucracy-inspired text entry form were invented 25 years ago, desperation devices to counter inadequate technology.  They were created for a world that no longer exists.</p>
  <p id="p226">Twenty-five years from now, no one will be clicking on drop-down menus, but everyone will still be pointing at maps and correcting each others’ sentences.  It’s fundamental.  Good information software reflects how humans, not computers, deal with information.</p>
  </div>
  <h2 id="demonstration_trip_planning_redux">Demonstration:  Trip planning redux</h2>
  <p id="p227"><acronym title="Bay Area Rapid Transit">BART</acronym>’s official planner is somewhat of a straw man, since <acronym title="Bay Area Rapid Transit">BART</acronym> has little competitive pressure to provide a quality website.  The airline industry, on the other hand, has every incentive to give customers a smooth decision-making experience.  However, planning a trip through the sky is almost identical to planning one underground.** This example is based on <a href="http://www.southwest.com">southwest.com</a> as of January 2006, but I checked ten other airline websites and found them (almost eerily) similar. 
   </p>
  <p id="p228">First, a mechanical, information-free configuration screen:</p>
  <p><img width="726" height="314" src="http://worrydream.com/MagicInk/p/flight_original_1.png" alt="image" /></p>
  <p id="p229">Followed by a table of textual results:</p>
  <p><img width="813" height="505" src="http://worrydream.com/MagicInk/p/flight_original_2.png" alt="image" /></p>
  <p id="p230">The actual information is squeezed into a few columns on the left, with most of the screen a monument to Southwest’s intricately stratified pricing structure.  (Additional columns to the right are not shown.)</p>
  <p id="p231">What questions might a user have?</p>
  <ul><li>What cities does this airline fly out of?  Where are they?</li>
  <li>What flights are available on the days I want to travel?</li>
  <li>When do they depart and arrive?</li>
  <li>How long are they?  (This can get confusing across time zones.)</li>
  <li>How many stops are there?  How many transfers?</li>
  </ul><p id="p232">Consider this redesign:</p>
  <p><img width="808" height="991" src="http://worrydream.com/MagicInk/p/flight_redesign.png" alt="image" /></p>
  <p id="p233">The times and lengths of the flights, and the count, times, and lengths of stops and transfers, can be compared visually.  Trips without transfers stand out because they are entirely blue; non-stop flights would appear unbroken.  Anomalies, such as the 6:50 from Hartford which arrives <em>later</em> than the 7:20, stand out literally.  Times can be converted into either time zone simply by referencing the appropriate header bar.</p>
  <p id="p234">There is some attempt to use color symbolically.  On the map, the calendar, and the flight chart, green represents “home,” and yellow the destination.  However, it is not critical that the user notice this.</p>
  <p id="p235">Interaction is simplified to the point where a short, instructive sentence can describe each and every click.  At the most, the user will click twice on the map, drag across the calendar, and click twice on the ticket prices, possibly with some page scrolling.  Last-value prediction (automatically selecting the last route purchased, and displaying a list of recent trips) may eliminate or reduce the map clicks for many travelers.  A learning predictor, capable of inferring that the user always spends the first Monday through Friday of the month in Baltimore and selecting that range on the calendar automatically, could eliminate all context-establishing interaction, leaving only the decision-conveying interaction of clicking ticket prices.  Of course, since everything is on the same page and feedback loops are tight, the user can explore different dates and cities, and see the available flights immediately.</p>
  <p id="p236">With air travel in a slump for the past few years, airlines have been desperate for any passengers they can get.  Unsuccessful ones have even faced bankruptcy.  With so much at stake, why hasn’t <em>any</em> airline attempted to improve the ticket-buying experience through better software design?</p>
  <p id="p237">The problem is primarily <em>cultural</em>.  Asking “Why doesn’t Southwest design better software?” is challenging the symptom, not the disease.  The real question is, “<b>Does software design <em>exist</em> yet?</b>”  Before we can expect better airline websites, we may need to change a worldview.</p>
  <p class="logo"><img width="435" height="83" src="http://worrydream.com/MagicInk/p/logo_changing_the_world.png" alt="image" /></p>
  <p id="p238">Mass production of machines emerged at the start of the 20th century.  Henry Ford’s assembly line methods spread throughout the manufacturing world, dramatically lowering production costs and making a variety of machines affordable for the average person.  But many of these products were unpleasant to interact with.  Between the businessman’s specifications and the engineer’s implementation, there was no <em>design</em>.  Within a few decades, a new profession arose to fill the gap—industrial design.</p>
  <p id="p239">The next revolution in the mass production of machines was software.  The late 1970s saw the rise of the personal computer, a device capable of behaving as <em>any</em> machine—typewriter, adding machine, filing cabinet, arcade game—when given the right instructions.  Manufacturing a “machine” was now just a matter of printing its instructions onto a disk, and production costs plummeted.  But much of this software was unpleasant to interact with.  Between the businessman’s specifications and the engineer’s implementation, there was no <em>design</em>.  Within a couple decades, a new profession arose to fill the gap—interaction design.</p>
  <p id="p240">The mass production of <em>information</em> has a very different history than the mass production of machines.  Industrial design brought art to existing mass-produced technology, but printing brought mass-producing technology to an existing art.</p>
  <p id="p241">Before the 15th century, books were precious and extremely rare, for each had to be copied by hand.  A single book might cost as much as a farm.  Books were also exquisite works of art, carefully lettered in calligraphy, lavishly illustrated and decorated.  In the 1440s, Johann Gutenberg’s movable type press boosted book production over a <em>thousand-fold</em>, making books affordable (and literacy worthwhile, and political awareness possible) for the average person for the first time.  Fortunately, Gutenberg and contemporary printers were exceptionally devoted to the art form, and took great pains to preserve the quality of the hand-lettered page.** Gutenberg’s emulation of calligraphy was so accurate, his bibles were sold as handmade manuscripts in Paris.  When people noticed the quantity and similarity of the books, they did not suspect printing, but witchcraft!  See Philip Meggs’s superb <a href="http://www.amazon.com/gp/product/0471699020">History of Graphic Design</a> (2005).      The explosion of new books of all kinds, as well as the rise of the broadside (precursor to the poster and the newspaper), created a great demand for artists in the new medium, many of whom transitioned from the old medium.  The art of laying out a page eventually became known as graphic design.</p>
  <p id="p242">The next revolution in the mass production of information was the web.  Unlike early printers, unfortunately, early web technologists cared little for the artistic qualities of their predecessor, but the capabilities eventually evolved to approximate the printed page on the computer screen.  Publishing was now just a matter of sending bits through a wire.  The explosion of websites created a great demand for artists in the new medium, many of whom transitioned from the old medium.  The art of laying out a webpage became known as web design. For related historical allegories, see Peter Drucker’s fascinating essay <a href="http://www.versaggi.net/ecommerce/articles/drucker-inforevolt.htm">The Next Information Revolution</a> (1998).</p>
  <p id="p243">These parallel evolutions have produced designers for interactive machines (conventional software) and designers for static page layouts (conventional websites).  From this viewpoint, the chimeric effects of convergence are almost to be expected.  The emerging “interactive web” embraces a ludicrously mixed metaphor of <em>machines on pages</em>, a monstrous hybrid of virtual mechanical affordances printed on virtual paper.  Information is trapped behind interactive mechanisms <em>and</em> presented in static layouts—it is the worst of both worlds.</p>
  <p id="p244">Good context-sensitive information graphics are neither interactive nor static, neither machines nor page layouts.  Design has not evolved to produce them.  The culture is blind to the possibilities.</p>
  <p id="p245">Who will draw information software?  And how?</p>
  <p id="p246"><strong>The first step</strong> toward the information software revolution is widespread <b>recognition</b> of the need for design.  It must be universally understood that information software is not a machine, but a <em>medium for visual communication</em>, and both publishers and public must hold it to the same standards that they hold print.  People constantly settle for ugly, clunky software, but demand informative, professionally-designed books, newspapers, magazines, and—ironically—brochures, ads, and manuals for that very software.** As brochures have become websites, this duality has veered into absurdity:  “Let’s design beautiful software to sell our ugly software!”  The wrapper tastes better than the candy.      Though once justified by technological limitations, this double standard is now dangerously obsolete.  It is the first and largest obstacle to revolution.  Without consumer demand, design appears to give no return on investment.</p>
  <p id="p247">Prominent usability pundits have claimed that the public is becoming more discriminating, but since this claim underlies their consultancies’ sales pitch, it is far from an unbiased observation.  I see the opposite—as technology races ahead, people are tolerating increasingly worse design just to use it.  The most beautifully-designed DVD player will go unsold if the competition costs the same and has S-Video output, or plays MP3s from memory sticks.  Good design makes people happy, but feature count makes people pay.</p>
  <p id="p248">I don’t know the solution to cultivating a culture of good taste, but I believe lessons can be learned from the emergence of industrial design, about seventy years ago.** See the chapter “Through the Back Door” in Henry Dreyfuss’s recently rereleased autobiography <a href="http://www.amazon.com/gp/product/1581153120">Designing for People</a> (1955).      At a time when many products competed on ornamentation, the simplified, functional creations of industrial designers were too untraditional to sell on looks alone.  The salesman made inroads by directly touting the tangible benefits of good design, such as comfort and safety.  He would demonstrate to a homemaker how his vacuum cleaner or iron was designed to reduce fatigue and cramping.  He would demonstrate to a farmer how his machinery was designed to eliminate the finger-severing accidents that were, to that point, distressingly common.  Explicitly informed of the benefits, people gradually came to demand, then expect, such conscientious design in their everyday products.</p>
  <p id="p249"> Other factors that boosted industrial design were fashion (top designers were promoted as celebrities) and price (good design often <em>lowered</em> manufacturing and materials costs).  See Raymond Loewy’s autobiography <a href="http://www.amazon.com/gp/product/1581153120">Industrial Design</a> (1979).  Both factors can be applied to software.
  Today, software consumers demand technological features because software marketing <em>presents</em> features.  Consumers ignore design because marketing ignores design.  The cycle is vicious, but perhaps vulnerable too—some brilliant new software with engineering, design, and marketing all in sync may raise the bar for everyone.</p>
  <p id="p250"><strong>The second step</strong> toward the information software revolution is finding people with <b>talent</b> for visual communication.  Currently, almost all software is designed by people who are very comfortable with computers; their interest in technology motivated them to enter the field.  This suggests an enormous exclusion of potential talent—imagine if all graphic designers had to be comfortable running a print shop!** One might argue that the entire next generation will be comfortable with computers.  But comfort with today’s “computers” may prove irrelevant—who can say what a “computer” will be in twenty years?  It is better to look for interest and talent in communicating with people, not with technology, since people don’t change nearly as fast.      I believe that ideal candidates for software design are those who have achieved mastery of information graphics in other mediums.  There may be multitudes of artists, currently drawing business graphics or maps or comics, who could excel at information software design if they had any idea that it was a legitimate artistic field.  Recent years have brought a wealth of beautiful amateur websites, created by visually-oriented people dabbling in the only sort of software design accessible to them.  But because full-fledged software is seen as an artifact of arcane technology, a product of “programmers,” these people lack the confidence to consider life beyond HTML.</p>
  <p id="p251"><strong>The third step</strong> is complementing the designer’s talent with <b>skill</b>.  Skill is achieved through <em>education</em> and <em>practice</em>, but dearth of the former has given aspiring designers no entry point—they are expected to learn the art through osmosis and guesswork.  Effective education can entail any, but ideally all, of:  classes, books, and examples.</p>
  <ul><li><p id="p10"><strong>Classes.</strong>  The renowned <a href="http://www.artcenter.edu">Art Center College of Design</a> in Pasadena offers forty courses in industrial design.  Students learn art theory, draftsmanship, and visual communication theory.  They learn about form, and the visual and tactile properties and constraints of materials.  They learn about cognitive and behavioral psychology, and explore how users experience products.  They follow the entire production process:  researching the needs of the target markets; sketching ideas and proposals; drawing detailed renderings; designing virtual 3D models; constructing physical models out of clay, plastic, and fiberglass; constructing a functional mechanical solution; designing logos and retail packaging.  They learn to devise artistic solutions to problems, to think creatively and think critically, to invent concepts and critique those of others.  They interact with industry representatives and do team projects under corporate sponsorship.</p>
  <p id="p11">Art Center offers only five courses that could be somewhat related to information software.** All five use “interactive” as a synonym for “software”:  <a href="http://www.artcenter.edu/datalog/CourseDescriptionDetail.jsp?id=410&amp;dept=1">Interactive Structures</a>, <a href="http://www.artcenter.edu/datalog/CourseDescriptionDetail.jsp?id=2617&amp;dept=1">Information and Interactivity</a>, <a href="http://www.artcenter.edu/datalog/CourseDescriptionDetail.jsp?id=346&amp;dept=1">Branding and Interactivity</a>, <a href="http://www.artcenter.edu/datalog/CourseDescriptionDetail.jsp?id=2836&amp;dept=3">Interactive Design 1</a>, and <a href="http://www.artcenter.edu/datalog/CourseDescriptionDetail.jsp?id=2867&amp;dept=3">Interactive Design 2</a>.  The intent of this example is not to malign Art Center’s curriculum, but to demonstrate the lack of resources for the aspiring software designer.    For the most part, students learn to make websites.  There is nowhere near the breadth or depth offered to designers of physical products.  Art Center clearly knows how to put together an applied arts curriculum.  What’s missing is the understanding of software as an applied art.</p>
  <p id="p12">There are other schools that offer specializations in “information architecture,” “usability,” and other recently-coined areas, but these subjects approach software design from a scientific perspective, neglecting the essentially artistic aspect of visual communication and the creative and critical techniques used by art schools for centuries.  Experimental analysis can be valuable, but only if an artist has created a design worth analyzing.</p></li>
  <li><p id="p13"><strong>Books.</strong>  Information software design will need a body of pedagogical literature, once enough theory is developed to make pedagogy possible.  Until that point, the student has little recourse—the closest established areas, information graphic design and “user interface design,” are both severely underserved.</p>
  <p id="p14">The paucity of literature on information graphic design is bewildering.  Edward Tufte’s books are highly acclaimed, and deservedly so, but they almost win their titles by default.  In a typical bookstore, they are lost amidst a sea of fashion rags masquerading as graphic design guides, or perhaps submerged in a “computer” section overflowing with the latest engineering fads.  They have too little company to define a category.** Their best company is probably William Cleveland’s <a href="http://www.amazon.com/gp/product/0534037291">The Elements of Graphing Data</a> and <a href="http://www.amazon.com/gp/product/0963488406">Visualizing Data</a>.  I doubt you will find either in a bookstore.    </p>
  <p id="p15">The shortage of <em>good</em> books on user interface design is more understandable, since pedagogy requires a working paradigm—the status quo must be at least acceptable.  Accordingly, I haven’t yet found a <em>textbook</em> that is at all helpful for software design; the only books I’ve found worthwhile are the few that <em>challenge</em> the status quo and present fresh, progressive ideas.  For the field to progress, we need less recycled platitudes and more cutting-edge research.
  The industrial design literature, incidentally, seems to consist primarily of photographs of chairs.  I don’t know what this means.  </p></li>
  <li><p id="p16"><strong>Examples.</strong>  In all artistic fields, from painting to writing to music to architecture, students study the works of the masters.  (In fact, artists who claim to be “self-taught” usually mean that they learned purely from examples.)  Much has been written about the failure of software engineering schools to provide examples of great works, expecting students to somehow derive style from first principles.** This is true in most other engineering disciplines as well, though less discussed.  I can think of only one exception from my own schooling in electrical engineering—<a href="http://www.its.caltech.edu/~mmic/people2/Rutledge.html">David Rutledge</a>’s innovative <a href="http://www.amazon.com/gp/product/0521646456">introduction</a> to analog electronics, taught through gradual construction and analysis of a commercial radio transceiver.  Engineering study typically focuses on how something <em>should be</em> done, not how it <em>has been</em> done, to the detriment of the culture.      Since software <em>design</em> isn’t yet recognized as an artistic field in the first place, its situation is even worse—the very concept of a gallery of software designs will seem absurd to most people.  But a corpus is crucial for the development of any artistic field.  Outstanding designs must be <em>recognized</em>, <em>collected</em>, and <em>explicated</em>.  Furthermore, outstanding <em>designers</em> should be recognized and encouraged to teach, instead of hidden behind a corporate label.</p></li>
  </ul><p id="p252"><strong>The fourth step</strong> is supplementing the designer’s talent and skill with <b>tools and platforms</b>.  These two terms are vague in common usage.  I will define a <em>tool</em> as a communication device that a designer has control over, and a <em>platform</em> as a communication device that a recipient is expected to provide.  This is best demonstrated with Claude Shannon’s communication model:</p>
  <p>* Adapted from Claude Shannon, <a href="http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf">A Mathematical Theory of Communication</a> (1948), p2.<img width="540" height="306" src="http://worrydream.com/MagicInk/p/platform_model.png" alt="image" /></p>
  <p id="p253">A tool encodes mental <em>information</em> into physical <em>data</em>, which can travel in a physical <em>medium</em>.  A platform decodes the physical data into the mind of the recipient.  Because all information transfer short of telepathy requires some medium, this model is universal.  If I write you a letter, my tools are pen and paper, and your platform is knowledge of my written language.  If I broadcast a radio signal, my tools are a microphone and transmitter, and your platform is a radio receiver.  In general, my tools are whatever I use to make the thing I hand off to you.  Your platform is whatever I’m counting on you to already have.</p>
  <p id="p254">To deliver her message most effectively, the visual designer needs <em>as much control as possible</em> over what the viewer sees.  But, by definition, the designer only has direct control over the <em>tool</em>.  She is at the mercy of whatever platform implementation the recipient happens to supply.  This implies that a good platform must be as <em>simple</em> and as <em>general</em> as possible.</p>
  <ul><li><p id="p17"><strong>Simplicity.</strong>   “I conclude that there are two ways of constructing a software design:  One way is to make it so simple that there are <em>obviously</em> no deficiencies, and the other way is make it so complicated that there are no <em>obvious</em> deficiencies.”  C.A.R. Hoare, <a href="http://www.braithwaite-lee.com/opinions/p75-hoare.pdf">The Emperor’s Old Clothes</a> Turing Award lecture (1980), p81.
  From a practical (and historical) standpoint, we can assume that <em>no complex specification will be implemented exactly</em>.  This, in itself, is not a problem.  However, multiple, decentralized implementations of a complex specification will be incorrect in different ways.  A platform consisting of the union of all possible implementations is thus arbitrarily <em>unreliable</em>—the designer can have no assurance of what a recipient actually receives.  <b>For a platform to be reliable, it must either have a single implementation, or be so utterly simple that it can be implemented uniformly.</b>  If we assume a practical need for open, freely implementable standards, the only option is simplicity.** <acronym title="Portable Operating System Interface">POSIX</acronym>, Java, and newer web standards (<acronym title="Document Object Model">DOM</acronym>, <acronym title="Cascading Style Sheets">CSS</acronym>) are some attempts at universal platforms (for various domains) that have proven too complex to implement uniformly.  In each case, the power of the platform is effectively constricted to some simple, reliable <em>subset</em>, and enormous time is wasted designing around incompatibilities.  By contrast, JPEG, MP3, and modern CPU instruction sets are universally dependable, because much of the complexity is placed at the encoding tool, not the decoding platform.  (Almost a century ago, a similar justification was used to reject single-sideband public radio.)  The complex Perl and Flash platforms are dependable only because they have centralized implementations. 
   </p></li>
  <li><p id="p18"><strong>Generality.</strong>  If we think of a computer as a machine that runs software, then in some sense, all data handled by a computer platform must be “software.”  The data making up a JPEG image, for example, can be thought of as the encoding of a <em>program</em> that describes a picture.  (This is sometimes called the “data is code” equivalence.)  But the limitations of the JPEG platform result in severely lobotomized “programs”—they cannot animate, respond to context, incorporate new compression techniques, or otherwise take any advantage of the <em>computer</em> beyond what JPEG explicitly allows.  A crippled platform cripples a designer’s means of expression.</p>
  <p id="p19">In order for a designer to take full advantage of the medium, a good platform must provide safe access to <em>everything</em> that is technologically possible.  A platform for information software must offer: inputs from the environment (that is, communication with other software and physical sensors), from history (that is, storage), and from the user (that is, interaction); computational resources with which to respond to inputs; and unrestricted graphical output.  Anything less robs information software of its full potential.  The proper way to prevent destructive behavior is a well-designed security model, not arbitrarily amputating the computer’s capabilities.</p></li>
  </ul><p id="p255">Alarmingly, the latest platforms forgo both of these virtues.  <acronym title="Cascading Style Sheets">CSS</acronym>, a language for specifying visual appearance on the web, is a particularly egregious example.  It is so complex that it has <em>never</em> been implemented correctly; yet, successive versions specify even more complexity.  At the same time, it is so underpowered that many elementary graphic designs are impossible or prohibitively difficult, and context-sensitivity (or <em>anything</em> computational) must be addressed externally.  Most <acronym title="Cascading Style Sheets">CSS</acronym> lore is dedicated to describing the tangles of brittle hacks needed to circumvent incompatibilities or approximate a desired appearance.</p>
  <p id="p256">One cause of the <acronym title="Cascading Style Sheets">CSS</acronym> mess is the eschewing of elegant, flexible abstractions for “1000 special cases,” a detrimental approach which precludes simplicity and generality in any domain.  However, the larger and more germane fault is the language’s attempt to serve as <em>both tool and platform</em>, thereby succeeding as neither.</p>
  <p id="p257">For universal reliability, the ideal platform must be optimized for ease of implementation.  For artistic expressiveness and exploration, a <em>tool</em> must be optimized for the designer’s manipulation.  Thus, the tool and platform cannot be the same—we must expect a layer of translation between what the designer works with and what the platform interprets.** There is a direct analog with RISC computer processors, whose simplified instruction sets are targeted at compilers, not programmers.  This considerably eases implementation of the processor, although the motive in this case is more performance than reliability.    </p>
  <p id="p258">A simple and general platform shifts complexity to this translation layer—the tool’s “back end”—where the designer has control over it.  If a particular tool is implemented incorrectly, the designer can work around its particular idiosyncrasies, or switch to a different tool.  (It is much easier for a designer to switch or upgrade tools than for a sea of users to switch or upgrade platforms.) Meanwhile, the tool’s “front end”—that which the designer interacts with—can be simple or complex, general or domain-specific, according to the designer’s needs and skill level.</p>
  <p id="p259">The platform must make it <em>possible</em> to create information software.  The tool must make it <em>easy</em>.  A specific look at some tools and platforms for information software will be offered in the next few sections.</p>
  <p id="p260"><strong>The fifth and final step</strong> into the information software revolution is an <b>environment</b> where experimentation, evolution, and interplay of ideas can thrive.  Much like our geological environment, a creative environment can become fatally polluted by short-sighted business interests.</p>
  <p id="p261">Before 1786, authors invariably presented quantitative data as tables of numbers.  In this year, an economist named William Playfair published a book called <em>The Commercial and Political Atlas</em>.** Remarkable recent efforts have brought this classic back into print, as <a href="http://www.amazon.com/gp/product/0521855543">Playfair’s Commercial and Political Atlas and Statistical Breviary</a> (2005).        In order to illustrate his economic arguments, Playfair <em>single-handedly</em> invented the line graph, the bar graph, and the pie chart, and thereby the entire field of statistical graphics.  Within years, his inventions had spread across Europe, transforming the landscape of visual communications and heralding an age of discoveries in data made visible.  Today, children take these graphical forms for granted; they seem as obvious and fundamental as written language.</p>
  <p id="p262">Imagine if Playfair had patented his invention and prosecuted his imitators, suppressing the crucial period of initial excitement and growth.  Would we today be staring at tables of numbers, unable to apply our visual cortex to unlocking their patterns?</p>
  <p id="p263"><strong>This path is inevitable</strong>, for it is the path of all artistic media.  Books, newspapers, and the static visual arts have already completed it, or almost so.  Movies, television, and published music are struggling at step five, but completion is only a matter of time.  For information software as well, it is only a matter of time.  But a decade or a century?</p>
  <p id="p264">Of course, design is nothing without implementation.  If information software is to consist of dynamic graphics that infer from history and the environment, it must be possible and easy to create such things.  The following sections will discuss a design tool for dynamic graphics, and engineering approaches to inferring from history and the environment.</p>
  <p id="p265">Software tools for drawing static graphics or composing static animations have long been commonplace.  But the designer who wants to create <em>dynamic</em> graphics—graphics whose properties are data-dependent—currently has two undesirable options:</p>
  <ul><li><p id="p20">She can learn some sort of programming language.  Many designers are intimidated by engineering and may lack the talent or desire to program.  They are completely justified—<b>drawing is a visual activity, and working with textual abstractions is entirely inappropriate</b>.  Painters, illustrators, and sculptors manipulate the artifact directly—there is no abstraction, and visual feedback is immediate.  Would we have any of our great works of art if the creators had to work with “rectangle.width = 17” instead of visible brushstrokes?** Early music composers typically worked in silence, with pen and paper, and did not actually hear their compositions until they were presented to musicians.  Composers who couldn’t handle this abstraction were belittled.  With the growing popularity of the clavier and harpsichord, and then the piano, it became acceptable for composers to hear their creations as they composed.  Most of our classical masterpieces were composed in this way.  Today, not only is every composer expected to work at an instrument, illiteracy is even becoming acceptable! 
   </p></li>
  <li><p id="p21">Alternately, a designer can draw a series of <em>mockups</em>, snapshots of how the graphic should look for various data sets, and present these to an engineer along with a verbal description of what they mean.  The engineer, who is skilled in manipulating textual abstractions, then implements the behavior with a programming language.  This results in ridiculously large feedback loops—seeing the effect of a change might take a day instead of a second.  It involves coordination and communication between at least two people, and requires that the designer <em>justify</em> herself—she must convince the engineer and possibly layers of management that each change is worth the engineer’s time.  This is no environment for creative exploration.</p></li>
  </ul><p id="p266">There is nothing wrong with the concept of drawing mockups.  It is a natural, visual way to work, and is ubiquitous across many artistic disciplines, from architecture to industrial design.  The problem lies with engineering the behavior the mockups describe.  But, consider what exactly the engineer does.  From a set of mockups, the engineer <em>infers the pattern</em> they conform to—how the graphic changes as a function of the data—and codifies this inferred pattern in a computer program.</p>
  <p id="p267">Is a human really necessary?  Couldn’t this pattern be inferred by a software tool instead?</p>
  <p id="p268">Going down this path leads to a computer science discipline known as “<b>programming by demonstration</b>” (<acronym title="Programming By Demonstration">PBD</acronym>) or “programming by example.”** See Allen Cypher (ed.), <a href="http://www.acypher.com/wwid/">Watch What I Do</a> (1993, available online) and Henry Lieberman (ed.), <a href="http://www.amazon.com/gp/product/1558606882">Your Wish Is My Command</a> (2001).  Both are compendia of research projects, not textbooks.      This field is concerned with teaching behavior to a computer implicitly, through a series of examples, rather than with explicit instructions.  Researchers have created systems (with varying degrees of success) for constructing interactive <acronym title="Graphical User Interface">GUI</acronym> widgets, defining parameterized graphical shapes, moving and renaming files, performing regular expression-like text transformation, and other domain-specific tasks.  With these systems, the user typically performs a few iterations of a repetitive task manually, and the system then performs the rest according to an inferred generalization, perhaps asking for clarification or confirmation.</p>
  <p id="p269">This section outlines a hypothetical but plausible tool to allow designers to create dynamic data-dependent graphics with <em>no conventional programming</em>.  These dynamic graphics would serve as the user-facing visible representation of information software.  In a sense, this is a tool for “drawing information software.”</p>
  <p id="p270">The tool can be considered an extension of a conventional vector-oriented drawing program.** Popular examples of drawing tools are Adobe Illustrator and Macromedia Flash.  The necessary feature is the representation of graphical elements as objects with variable properties, rather than as arrays of pixels.      Using the same drawing process as with a conventional tool, the designer draws a mockup of the graphic—how the graphic should look for some particular set of data.  She then takes a <em>snapshot</em> of this graphic, and indicates the data set that it corresponds to.  She then modifies the graphic to correspond to a slightly different data set, takes another snapshot, and so on.  Each snapshot serves as an example.  With well-chosen examples, the tool will infer how to generate a graphic for arbitrary data.
  The concept of snapshots may have been introduced by David Kurlander’s <a href="http://www.acypher.com/wwid/Chapters/12Chimera.html">Chimera</a> (1991), which used common features in a set of snapshots to infer constraints while drawing a static graphic.</p>
  <p id="p271">This tool is significantly less ambitious than many in the literature, for several reasons:</p>
  <ul><li>Many systems use “programming by demonstration” as a means toward <em>end-user programming</em>, typically to allow novices to automate repetitive tasks.  This tool is intended for professional designers with specialized skills and training, and thus can assume a higher level of user sophistication.</li>
  <li>Many systems attempt to infer a full computational procedure, and have the most difficulty with computational concepts such as conditionals and iteration.  As we will see, this tool mostly has to infer <em>mappings</em> from some set or numerical range to another—functions in the <em>mathematical</em> sense rather than the (imperative) <em>computational</em> sense.  This may be significantly easier.</li>
  <li>Similarly, many systems attempt to infer stateful programs.  This introduces enormous complexity, because the user must teach an output that depends on both the input and a potentially large hidden state.  A graphic is stateless; its appearance is only a function of the input data.  Examples require no context.</li>
  </ul><div id="tool_demo">
  <p id="p272"><strong>Demonstration.</strong>  I will demonstrate how we might use this tool to design the <acronym title="Bay Area Rapid Transit">BART</acronym> widget described above.</p>
  <p class="image"><img width="350" height="215" src="http://worrydream.com/MagicInk/p/widget_marked.png" alt="image" /></p>
  <p id="p273"><strong>Train component.</strong>  We start by modeling a single train bar.  <img src="http://worrydream.com/MagicInk/p/tool_single_bar.png" alt="image" />  This graphic has a number of dynamic aspects: position, length, color, and label.  For now, we will just handle the color and label.  We draw a picture, take a snapshot, and indicate the data properties that it corresponds to:</p>
  <p class="snapshot"><img width="244" height="44" src="http://worrydream.com/MagicInk/p/tool_train_1.png" alt="image" /></p>
  <p id="p274">This is what a <strong>Train</strong> should look like if the “Line” property is “Orange” and the “Destination” property is “Fremont.”  With only a single example, of course, the tool cannot infer anything dynamic.  Let’s teach it how to change the label.</p>
  <p class="snapshot"><img width="253" height="72" src="http://worrydream.com/MagicInk/p/tool_train_2.png" alt="image" /></p>
  <p id="p275">Compare these two snapshots.  The graphics are exactly the same, except a label has changed from “Fremont” to “Richmond.”  The data is exactly the same, except the “Destination” property has changed from “Fremont” to “Richmond.”  The simplest inference is that the “Destination” property should be used as the label text.  The tool will learn and use this rule, provided no other example contradicts it.</p>
  <p id="p276">We now teach the tool how to respond to the “Line” property.</p>
  <p class="snapshot"><img width="526" height="106" src="http://worrydream.com/MagicInk/p/tool_train_3.png" alt="image" /></p>
  <p id="p277">The graphics in the new snapshots are exactly the same as the orange-line Richmond-bound example, except for hue adjustments.  The data is exactly the same, except for the “Line” property.  Thus, the tool infers that each given value of the “Line” property corresponds to a particular hue adjustment.</p>
  <p id="p278">At this point, the tool should understand how to draw a <strong>Train</strong> for any arbitrary data set, as long as “Line” is within the provided domain.  (How we know that it has learned correctly will be discussed below.)</p>
  <p id="p279">If we want to clarify the model for posterity, we can add <em>visual comments</em> simply by drawing outside the snapshots:</p>
  <p><img width="710" height="105" src="http://worrydream.com/MagicInk/p/tool_train_comments.png" alt="image" /></p>
  <p id="p280"><strong>When component.</strong>  Now, we’ll model the time-related text that appears to the left and right of a <strong>Train</strong>.
  <img width="174" height="29" src="http://worrydream.com/MagicInk/p/tool_when_sidenote.png" alt="image" />
  We will use two data properties.  “Now” refers to the current time, and “Time” refers to the start or end of the trip.  Here are our first two snapshots:</p>
  <p class="snapshot"><img width="139" height="79" src="http://worrydream.com/MagicInk/p/tool_when_1.png" alt="image" /></p>
  <p id="p281">In these examples, “Now” stays constant while “Time” varies.  The tool will easily infer that the first row corresponds to “Time” (again, as long it doesn’t contradict further examples).  The second row is more problematic.  The tool infers linear relations when given two points, so our examples indicate this relation:</p>
  <p class="snapshot"><img width="192" height="139" src="http://worrydream.com/MagicInk/p/tool_when_graph_1.png" alt="image" /></p>
  <p id="p282">The correct relation actually depends on “Now,” but we haven’t yet demonstrated variance with respect to “Now.”  Our third snapshot does so:</p>
  <p class="snapshot"><img width="139" height="112" src="http://worrydream.com/MagicInk/p/tool_when_2.png" alt="image" /></p>
  <p id="p283">The simplest non-trivial relation now depends on the difference between “Now” and “Time”, which is correct:** To understand how the tool might figure this out, let us take f(x,y) to be the number in the text label, and x and y to be our two data properties (here expressed as minutes since 3:00, although any units and origin will work).  The three snapshots give us these constraints:</p><p>f(0,1) = 1 <br />f(0,119) = 119 <br />f(-60,-59) = 1 </p><p>One of the simplest and most naturally-occurring functions of two variables is linear combination:</p><p>f(x,y) = ax + by + c </p><p>Solving for the coefficients gives us </p><p>a = -1,  b = 1,  c = 0 </p><p>Because linear combination results in such simple coefficients, the tool will have high confidence in this inference, and will use it unless contradicted by some other example. 
   </p>
  <p class="snapshot"><img width="192" height="139" src="http://worrydream.com/MagicInk/p/tool_when_graph_2.png" alt="image" /></p>
  <p id="p284">Now, let’s teach the tool how to present larger time differences:</p>
  <p class="snapshot"><img width="290" height="143" src="http://worrydream.com/MagicInk/p/tool_when_3.png" alt="image" /></p>
  <p id="p285">This gives us the following relation, with interpolation in black and two possible extrapolations in red and blue.</p>
  <p class="snapshot"><img width="363" height="86" src="http://worrydream.com/MagicInk/p/tool_when_graph_3.png" alt="image" /></p>
  <p id="p286">The blue extrapolation is desired.  The tool can probably infer it, since it results in an arguably simpler relation.  (The red interpretation makes “in 2 hours” a special case, whereas the blue interpretation understands it as part of a general rule.)  But if the tool infers incorrectly, the designer can easily correct it.  (How so will be discussed below.)</p>
  <p id="p287">We now provide snapshots to cover earlier times.</p>
  <p class="snapshot"><img width="602" height="140" src="http://worrydream.com/MagicInk/p/tool_when_4.png" alt="image" /></p>
  <p id="p288"><strong>Timeline component.</strong>  Time extends infinitely; thus, the timeline is conceptually an infinitely-wide bar.  Of course, only a portion of this bar is actually visible at any given instant.  Dealing directly with infinite graphics will be discussed below.  Here, I will demonstrate how this can be easily simulated with a normal graphic.</p>
  <p class="snapshot"><img width="557" height="149" src="http://worrydream.com/MagicInk/p/tool_timeline.png" alt="image" /></p>
  <p id="p289">The red box indicates the clipping region of the graphic.  The section within the box is the portion that will actually be visible.  These snapshots differ from each other in only two aspects:  the position of the clipping region and the text labels.  The inferred position of the clipping region is shown below as a function of “Now,” with the interpolation in black and extrapolation in blue:</p>
  <p class="snapshot"><img width="380" height="112" src="http://worrydream.com/MagicInk/p/tool_timeline_graph_1.png" alt="image" /></p>
  <p id="p290">Because the graph above is somewhat abstract, it may easier to view images of the <strong>Timeline</strong> itself (rotated sideways) as a function of “Now”:</p>
  <p class="snapshot"><img width="380" height="222" src="http://worrydream.com/MagicInk/p/tool_timeline_graph_1_eg.png" alt="image" /></p>
  <p id="p291">We can see that the clipping region slides rightward with time, snapping back to the left on the half hour.  The cyclic extrapolation can either be inferred by the tool or specified by the designer, as will be explained below.</p>
  <p id="p292">The first text label’s value as a function of “Now”:</p>
  <p class="snapshot"><img width="333" height="125" src="http://worrydream.com/MagicInk/p/tool_timeline_graph_2.png" alt="image" /></p>
  <p id="p293">That is, the text label is “rounded down” to a multiple of fifteen minutes.  The rest of the labels will be inferred similarly.  With a little thought, it is clear that this graphic, when cropped to the red rectangle, appears to scroll boundlessly with respect to “Now.”</p>
  <p id="p294"><strong>Row component.</strong>  Next, we combine some of the components created above to form a compound component:</p>
  <p class="snapshot"><img width="398" height="56" src="http://worrydream.com/MagicInk/p/tool_row_layout.png" alt="image" /></p>
  <p id="p295">Adjacent pairs of snapshots describe how to adjust, respectively, the end point of the <strong>Train</strong>, the start point of the <strong>Train</strong>, and the clipping region:</p>
  <p class="snapshot"><img width="516" height="186" src="http://worrydream.com/MagicInk/p/tool_row.png" alt="image" /></p>
  <p id="p296">Notice that adjustments were made within individual components.  The length of the <strong>Train</strong> was changed, and the second <strong>When</strong> was right-justified.** Thus, modeled components are not black boxes, only adjustable through data properties, but can be modified at any level in the hierarchy. 
   
  The tool allowed “Depart Time” to be explicitly linked to the first <strong>When</strong>’s “Time” property, and “Arrive Time” to the second, so these relationships did not need to be inferred.  (These links are not shown here.)</p>
  <p id="p297"><strong>Title component.</strong> We are almost done.  We have to put the title together:</p>
  <p class="snapshot"><img width="401" height="55" src="http://worrydream.com/MagicInk/p/tool_title.png" alt="image" /></p>
  <p id="p298">No inference is used here; we explicitly link the properties to the appropriate labels.  (Again, not shown.)</p>
  <p id="p299"><strong>Trip Planner component.</strong> Finally, we are ready to lay out the top-level component.  We draw the background picture and place the components created above.</p>
  <p class="snapshot"><img width="345" height="230" src="http://worrydream.com/MagicInk/p/tool_planner_layout.png" alt="image" /></p>
  <p id="p300">Our final graphic and its data properties look like so:</p>
  <p class="snapshot"><img width="582" height="239" src="http://worrydream.com/MagicInk/p/tool_planner.png" alt="image" /></p>
  <p id="p301">No inference is used here.  We explicitly link the top-level properties to the appropriate component properties.</p>
  <p id="p302">Our dynamic graphic is complete.  The final program would consist of this graphic and a data source that fills in the properties.  Of course, this small example does not entirely emulate the actual <acronym title="Bay Area Rapid Transit">BART</acronym> widget, but it is easy to see how additional features can added, simply with models and snapshots.** Except for animation, interaction, and state.      It is also easy to see how a completely different design, such as the tables on the official <acronym title="Bay Area Rapid Transit">BART</acronym> website, could be composed on top of the exact same data source.</p>
  </div>
  <p id="p303"><strong>Editing snapshots.</strong>  The essence of this process is elimination of abstraction.  The designer works with concrete, visible examples.</p>
  <p id="p304">However, this raises a concern about editing.  An advantage of abstraction is that it localizes common properties, so widespread changes can be made with a single edit.  What if the designer decides that a <strong>Train</strong> should have square corners instead of rounded?  Having to individually edit each of the snapshots is unacceptable—such a burden would squelch experimentation.</p>
  <p id="p305">Instead, the designer simply <em>selects</em> the snapshots she wants changed, and proceeds to edit <em>one</em> of them.  The changes propagate to all selected snapshots.  This is possible because the tool treats the snapshots as variations on a single graphic, rather than independent graphics.</p>
  <p id="p306"><strong>Editing curves.</strong>  A more quantitatively-oriented designer may prefer to manipulate inferred relations directly.  Mapping curves can be shown graphically, and the designer can move anchor points around, add new anchor points, and introduce curvature by stretching the interpolation curves.  This allows for non-linear or nuanced behavior that would be difficult to specify purely with examples.</p>
  <p class="snapshot"><img width="478" height="124" src="http://worrydream.com/MagicInk/p/tool_curve_1.png" alt="image" /></p>
  <p id="p307">The curves are an abstraction, but because it is purely visual, designers may find it comfortable.  To lessen the abstraction, abundant concrete examples from along the curve are shown, and a designer can point anywhere in the plane to see an example that corresponds to that point.</p>
  <p class="snapshot"><img width="318" height="181" src="http://worrydream.com/MagicInk/p/tool_curve_2.png" alt="image" /></p>
  <p id="p308">Curve editing may also allow for better control over extrapolation:</p>
  <p class="snapshot"><img width="342" height="306" src="http://worrydream.com/MagicInk/p/tool_curve_extrapolation.png" alt="image" /></p>
  <p id="p309">In the above, we are essentially using “drawing by example” to specify the extrapolation curve.</p>
  <p id="p310"><strong>Feedback through examples.</strong>  Conventional software engineers will be worried by the rampant ambiguity in this design process.  In the demonstration above, the snapshots are visible but the inferred relations tying them together are not.  How does a designer know if the tool’s understanding matches her own?</p>
  <p id="p311">Unlike a programmer typing into a text editor, the designer does not create these snapshots in isolation.  The tool provides a design environment that <em>actively</em> communicates the dynamics of the graphic.</p>
  <p id="p312">One approach is for the tool to directly ask the designer about ambiguous cases.  The tool can present the designer with a data set that would disambiguate an unclear relation, and the designer would then draw a snapshot for that particular data set.  We might imagine the design process becoming inverted, driven by the tool—the designer would create a few representative examples, and then let the tool explicitly ask for all of the examples necessary to fill out the model.** There are a number of Programming By Demonstration research systems that take a similar interactive approach to disambiguation. 
   </p>
  <p id="p313">However, visual art has traditionally been composed actively, not reactively, and this approach may feel unnatural and stifling.  A more natural and information-oriented approach uses <em>continuous peripheral feedback</em>.  The tool adorns the screen with an array of its own examples that represent the inference it currently understands.  As the designer works, she can visually verify that the inferred relations are correct:</p>
  <p class="snapshot"><img width="422" height="210" src="http://worrydream.com/MagicInk/p/tool_feedback_examples.png" alt="image" /></p>
  <p id="p314">The tool can graphically emphasize feedback examples according to how little confidence it has in them.  If one is incorrect, the designer creates a new snapshot simply by clicking on it and correcting it.</p>
  <p id="p315"><strong>Feedback through curves.</strong> <img width="199" height="76" src="http://worrydream.com/MagicInk/p/tool_curve_sidenote.png" alt="image" />  In addition to feedback through examples, the mapping curves described above also provide feedback.  As the designer creates snapshots, she can see the inferred curves.  If an inference is incorrect, she can either create more snapshots, or directly edit the curve (as long as the tool has correctly inferred which <em>variables</em> are involved in the mapping).  </p>
  <p id="p316">If the tool feels an extrapolation is ambiguous, it can display all of the candidate extrapolations on the curve, and the designer can select one with a click:</p>
  <p class="snapshot"><img width="312" height="167" src="http://worrydream.com/MagicInk/p/tool_feedback_extrapolation.png" alt="image" /></p>
  <p id="p317"><strong>Hints.</strong>  If necessary, the designer can add hints to encourage the tool to prefer certain inferences.  There are two types of hints:  dependencies and constraints.</p>
  <p id="p318">A <em>dependency</em> hint suggests that a particular graphical aspect is related to a particular data property.  The specific mapping between the two must be specified through examples or curves, but this hint tells the tool which variables are involved.</p>
  <p class="snapshot"><img width="306" height="34" src="http://worrydream.com/MagicInk/p/tool_hint.png" alt="image" /></p>
  <p id="p319">A <em>constraint</em> hint suggests that a particular graphical aspect is related to some other graphical aspect.** Drawing with constraints is as old as drawing on computers—both originated with Ivan Sutherland’s seminal <a href="http://www.cl.cam.ac.uk/TechReports/UCAM-CL-TR-574.pdf">Sketchpad</a> (1963).  Constrained drawing was further explored in a number of research projects (of note are David Kurlander’s <a href="http://www.acypher.com/wwid/Chapters/12Chimera.html">Chimera</a> (1991) and Michael Gleicher’s <a href="http://www.cs.wisc.edu/graphics/Papers/Gleicher/CMU/briar.pdf">Briar</a> (1993)) but has rarely appeared in general-purpose commercial tools.       A “hard” constraint specifies an invariant relationship, such as two components that must always be aligned or parallel.  In the example below, we ensure that the train cannot shrink smaller than the text label, by constraining the right edge of the train to lie to the right of the label:</p>
  <p class="snapshot"><img width="190" height="43" src="http://worrydream.com/MagicInk/p/tool_hint_constraint_hard.png" alt="image" /></p>
  <p id="p320">A “soft” constraint specifies a <em>goal</em> that should be fulfilled as well as possible, given the other constraints.  The example below models a character whose eyes will watch the train, wherever it goes.  The eyeball is hard-constrained to lie within the eye, and soft-constrained to move as close as possible to the train.</p>
  <p class="snapshot"><img width="495" height="173" src="http://worrydream.com/MagicInk/p/tool_hint_constraint_soft.png" alt="image" /></p>
  <p id="p321">Hints may allow a designer to specify complex relationships that would be difficult to describe purely through snapshots.</p>
  <p id="p322"><strong>Recursion.</strong>  In the demonstration above, an infinitely-wide timeline bar was modeled by cyclicly panning over a finite graphic whose labels change on each cycle.  This corresponds to the computational concept of iteration.  An alternative for modeling infinite graphics is recursion.  Consider this layout:** The approach shown here is precisely how infinite data structures are represented in lazy programming languages such as Haskell.</p><p>In fact, if we think of a component model as a function definition, and the placement of a component as a function call, this tool can be seen as an editor for an underlying functional “language.”  Intriguing features include purity (evaluating a component has no side effects), laziness (components placed outside a clipping region need not be evaluated), and a combination of applicative evaluation (via mapping curves, whether explicit or inferred by the tool) and constraint-solving (via constraint hints).  If we allow a component’s parameters (the function’s “arguments”) to themselves be dynamic graphics instead of merely text, and provide a means of graphically extracting part of a parameter and recursing on the rest, this language should be as expressive and powerful as any textual functional language.   </p>
  <p class="snapshot"><img width="488" height="44" src="http://worrydream.com/MagicInk/p/tool_recursion_layout.png" alt="image" /></p>
  <p id="p323">This <strong>Timeline</strong> component <em>contains</em> a <strong>Timeline</strong> component, shifted right by 100 pixels.  The graphic now must be infinitely wide, because it is 100 pixels wider than itself.  The tool draws this graphic by continuously “copying” the entire <strong>Timeline</strong> graphic, and “pasting” it into the yellow box:</p>
  <p class="snapshot"><img width="489" height="256" src="http://worrydream.com/MagicInk/p/tool_recursion_expanded.png" alt="image" /></p>
  <p id="p324">With each paste, the yellow box shifts over by 100 pixels, and the pasting could go on forever.  Now, we edit the text labels in the first paste (indicated by red arrows):</p>
  <p class="snapshot"><img width="488" height="39" src="http://worrydream.com/MagicInk/p/tool_recursion_corrected.png" alt="image" /></p>
  <p id="p325">As we do so, the tool infers a linear relation between the top-level label (12:00) and the pasted one (12:30), and that relation is used to generate the labels in subsequent pastes (1:00, 1:30, etc.).  That is, the tool learns to add a half hour each time it pastes.  The result is a timeline whose labels increment forever.</p>
  <p id="p326">For the final component, we need just two snapshots, to show how “Now” should pan across the graphic:</p>
  <p class="snapshot"><img width="565" height="77" src="http://worrydream.com/MagicInk/p/tool_recursion_snapshots.png" alt="image" /></p>
  <p id="p327">Some may claim that recursion is inappropriate for graphic designers.  I would argue that recursion’s reputation for abstruseness is due to the textual abstractions used in mathematics and programming, and especially because expansion is rarely shown explicitly.  I believe that, with training, any designer who appreciates MC Escher can learn to make powerful use of visual recursion.</p>
  <p id="p328"><strong>Insulation from engineering.</strong>  One of the primary benefits of this tool is the freedom it gives designers in composing the appearance of information software.  The engineer’s job is to create a data source, and possibly spot-optimize the tool’s inferences if any are prohibitively inefficient.  Unless complex behavior is necessary, the engineer is completely uninvolved with graphical presentation, to the relief of both designer and engineer.</p>
  <p id="p329">For example, in the actual <acronym title="Bay Area Rapid Transit">BART</acronym> widget, the ending times become left-justified if the trip is too short:</p>
  <p class="snapshot"><img width="486" height="37" src="http://worrydream.com/MagicInk/p/tool_left_justify.png" alt="image" /></p>
  <p id="p330">Excellent software is characterized by this sort of attention to detail.  However, if a designer were to <em>request</em> that this minor feature be implemented, she would probably be rebuffed by both engineer and management.  This tool allows a perfectionist designer to add this feature on her own, just by taking a couple additional snapshots.** Implementation of the inferencing described here may involve algorithms that are unfamiliar to many engineers.  Interested (or skeptical) engineers are encouraged to read the two machine learning books cited immediately below, as well as the two Programming By Demonstration books cited above. 
   </p>
  <h2 id="engineering_inference_from_history">Engineering inference from history</h2>
  <p id="p331">The section “Inferring context from history” presented the need for software to learn from the past.  Good information software will attempt to predict current context by discovering patterns in past contexts.  Although such application software is rare, there is nothing novel or exotic about the algorithms required.  The computer science discipline devoted to this subject is called “<b>machine learning</b>” or “learning systems,” and several decades of research have produced a variety of algorithms for modeling and predicting behavior.** Tom Mitchell’s book <a href="http://www.amazon.com/gp/product/0070428077">Machine Learning</a> (1997) gives a good introductory overview to the basic algorithms.  Russell and Norvig’s book <a href="http://www.amazon.com/gp/product/0137903952">Artificial Intelligence: A Modern Approach</a> (2003) covers learning within a much broader context, but is less focused and concise.   </p>
  <p id="p332">Consider the example presented earlier of a train trip planner that predicts the route that the user wants to see.  There are typically daily or weekly patterns to a person’s local travel schedule.  A planner that models these patterns could automatically present the user with appropriate information, eliminating most interaction.</p>
  <p id="p333">As a demonstration, I implemented this behavior with the very simple probabilistic algorithm described below:** You can see the Lua <a href="http://worrydream.com/MagicInk/predictor.lua">source code</a> for the algorithm and test simulator. 
   </p>
  <p id="p334"><strong>History collection.</strong>  Each time the user indicates interest in a particular route, it is recorded in a history with the date and time.
  <img width="191" height="106" src="http://worrydream.com/MagicInk/p/predictor_history.png" alt="image" />
  The user indicates interest either by explicitly switching the planner to display a route, or by looking at the planner and then looking away, indicating that the shown route is still interesting.</p>
  <p id="p335"><strong>Prediction.</strong>  When the user looks at the planner, each history entry “votes” for its route with a certain weight, and the route with the largest total weight is displayed.  Each entry’s weight is a product of three factors, which depend on the time, the day of the week, and the age of the history entry.</p>
  <ul><li><p id="p22"><img width="221" height="189" src="http://worrydream.com/MagicInk/p/predictor_time_both.png" alt="image" /><strong>Time.</strong>  If the time is 9:00, the user’s route at 9:10 yesterday is very relevant.  What the user did at 10:00 is not quite so relevant, and her 12:00 activity is probably unrelated.  Thus, each vote is weighted by a window around the time of the history entry.</p>
  <p id="p23">Typically, if the user is preparing to catch a train, she won’t just look once at the planner and memorize the time.  She will glance at the planner frequently over a span of time.  Each of these looks should not count as an individual vote.  Instead, the entire span of checking is coalesced into a single history entry, with a widened time window.  (Also, throughout these frequent check-ups, the user sees only last-value prediction.  Learning prediction is only used if some time has passed since the previous look.)</p></li>
  <li><p id="p24"><img width="200" height="106" src="http://worrydream.com/MagicInk/p/predictor_day.png" alt="image" /><strong>Day of the week.</strong>  A user will typically exhibit a superposition of daily patterns, such as going to and from work, and weekly patterns, such as cello practice every Tuesday.  To allow for both, history entries from a different weekday are allowed to vote, but have a smaller weight.  The bleed across days allows the algorithm to learn daily patterns faster, but because other days are penalized, weekly patterns can be learned as well.  Saturday and Sunday are independent from weekdays and from each other.</p></li>
  <li><p id="p25"><img width="189" height="93" src="http://worrydream.com/MagicInk/p/predictor_age.png" alt="image" /><strong>Age.</strong>  Older history entries are given less weight, and eventually are forgotten.  This makes the algorithm <em>adaptive</em>.  If the user adopts a new pattern, such as switching jobs or joining the Thursday-night knitting circle, the algorithm is able to keep up, instead of having to be manually reset.</p></li>
  </ul><p id="p336">Finally, the most recent route is given a bonus vote.  This causes the algorithm to default to last-value prediction if there is no compelling reason to do otherwise.</p>
  <p id="p337"><strong>Results.</strong>  I tested this algorithm with user models that simulate a variety of schedules.  Various trade-offs are possible through choices of weights and window widths; the results below are intended to convey a qualitative idea of the algorithm’s performance.</p>
  <p id="p338">For a user who simply uses the planner to go to and from work, the algorithm learns the pattern flawlessly within a week.  When the user switches schedules, the algorithm adapts within a couple weeks.** Of course, humans won’t check the planner at exactly the scheduled time, and neither does the model.  The simulated times are normally distributed around the base time shown in the schedule, with a standard deviation of half an hour.    </p>
  <p class="image"><img width="498" height="95" src="http://worrydream.com/MagicInk/p/predictor_results_1.png" alt="image" /></p>
  <p id="p339">More complicated schedules are also learned quickly and almost flawlessly.</p>
  <p class="image"><img width="498" height="95" src="http://worrydream.com/MagicInk/p/predictor_results_2.png" alt="image" /></p>
  <p id="p340">Up to a certain level, random (unscheduled) behavior can be added without the algorithm losing the pattern.** This graph plots mispredicted <em>scheduled</em> views.  Obviously, the random views are always mispredicted. 
   </p>
  <p class="image"><img width="498" height="101" src="http://worrydream.com/MagicInk/p/predictor_results_3.png" alt="image" /></p>
  <p id="p341">In conclusion, it appears that this algorithm would successfully be able to infer the context of a regular user, allowing relevant information to be presented with little or no interaction.</p>
  <p id="p342"><strong>So what?</strong>  As an ad-hoc solution to a particular problem, this algorithm seems to work quite well.  <b>As a general solution, it is no solution at all.</b></p>
  <p id="p343">This simple, understandable example was intended to demonstrate that learning prediction is not science fiction—it is a viable and powerful approach to context inference, one that every software designer must keep on her palette.  However, the best learning algorithms are considerably more complex than this one.  Currently, machine learning is considered an experts-only area, where the fruits of research are restricted to specialists.  Implementing learning behavior typically involves calling in an expert, not assigning it to the application programmer.</p>
  <p id="p344">Unfortunately, an algorithm that can only be wielded by a master is almost worthless.  There are far more applications than experts; if application programmers cannot make use of learning, learning applications will remain rare and exotic.</p>
  <p id="p345">This predicament has been overcome many times before.  All software today performs an intricate dance of feeding a processor primitive machine-level instructions, repolarizing tiny magnets in hard drives, transmitting bits reliably across wires, and lighting up specific pixels on a screen.  The algorithms behind these operations are unimaginably complex, requiring years of study.  Yet, even novice programmers have no trouble with these operations.  The complexity has been hidden behind <em>abstractions</em>.</p>
  <p id="p346">Programmers write to “files,” read from “sockets,” draw with “fonts” and “images,” and think in “programming languages.”  Behind each abstraction are experts who devote their entire careers to their particular niche, following cutting-edge research and participating in the community.  In front of the abstractions are armies of application programmers, blessedly able to take all this for granted.  <em>Without these abstractions, our modern software landscape simply wouldn’t exist.</em></p>
  <p id="p347">As I see it, the primary challenge for the machine learning community is not the generation and tuning of yet more algorithms, but the design of simple abstractions.  Learning magic must be <em>packaged</em>.  Like a “file,” the abstraction must be usable by <em>any</em> engineer working in <em>any</em> domain.  It must be so simple that it can be taken for granted.</p>
  <p id="p348">Today, a Perl programmer needs just four letters to invoke decades of research into filesystems and physical media:  “open.”  A finely-tuned mergesort is available with the word “sort,” and even more finely-tuned hashing algorithms require just a pair of brackets.  <b>Until machine learning is as accessible and effortless as typing the word “learn,” it will never become widespread.</b> </p>
  <p id="p349">One small step for trip planners gets mankind nowhere.</p>
  <h2 id="engineering_inference_from_the_environment">Engineering inference from the environment</h2>
  <p id="p350"><strong>Other information software</strong>, such as open websites.  By reading some information, the user is indicating a topic of interest.  All other information software should take heed.  Consider a person reading the website of an upcoming stage play.  When she opens her calendar, the available showings should be marked.  When she opens a map, she should see directions to the playhouse.  When she opens a restaurant guide, she should see listings nearby, and unless the play offers matinees, they shouldn’t be lunch joints.</p><p><strong>Documents created with manipulation software.</strong>  <em>Creating</em> some information indicates an even stronger topic of interest. Consider a person who requests information about “cats” while writing a paper.  If the paper’s title is “Types and Treatment of Animal Cancer,” the information should skew toward feline medical data.  The title “History of Egypt” indicates interest in ancient feline worship instead.  And if the paper contains terms related to building construction, “cats” probably refers to the decidedly non-feline Caterpillar heavy machinery.
  The section “Inferring context from the environment” presented a number of environmental sources from which information software could infer context.  The hardware-related sources, such as clocks and location sensors, might have seemed obvious.  The software-related sources, such as other information software and documents created with manipulation software, might have seemed so far-fetched as to be implausible.  This section will present the <b>information ecosystem</b>, a software architecture which might allow for such behavior.</p>
  <p id="p351">Consider this scenario:</p>
  <ul><li><p id="p26">I receive an email from a friend. <br /><img width="183" height="92" src="http://worrydream.com/MagicInk/p/ecosystem_email.png" alt="image" /></p></li>
  <li><p id="p27">After reading the email, I open my map software to find that nearby pizza restaurants are prominently marked.</p></li>
  </ul><p id="p352">How might such behavior be implemented?</p>
  <p id="p353">One approach is to build a <em>system</em> that directly performs the desired behavior.  In this case, perhaps one would design an email program with a built-in map.  If the current email contains the word “pizza,” the program would perform an internet search for pizza places and display them on its map.</p>
  <p id="p354">There are several reasons why the system-based approach is unappealing:</p>
  <ul><li><p id="p28">Monolithic systems don’t scale.  The system described is a trivial solution to a general problem.  What about information from a website showing up on my calendar?  What about seeing encyclopedia entries related to the paper I’m writing?  The possibilities grow combinatorially—it is impossible to deliberately handle them all.</p></li>
  <li><p id="p29">Monolithic systems are bad for users.  Email and maps are distinct concepts.  There is no reason why a user should turn to the same software package for two unrelated purposes.** For that matter, email and calendars are distinct concepts as well.     Also, the components of integrated systems tend to be of lower quality than their dedicated counterparts.  You could chop your vegetables and assemble your furniture with a Swiss Army knife, but you probably don’t.</p></li>
  <li><p id="p30">Monolithic systems are bad for software providers.  In a healthy marketplace, whether of groceries or auto parts, individual providers offer components which combine with others for a complete solution.  A small software provider could provide an excellent email program, or an excellent map.  But only a large corporation has the resources to develop an integrated package.  Once small companies can’t compete, progress stagnates.</p></li>
  </ul><p id="p355">What we need, then, is not a <em>system</em> that implements this behavior, but a <em>platform</em> that enables such a system to grow organically, via small contributions from diverse providers.</p>
  <p id="p356">In forsaking integration, however, we forsake designed coordination between components.  The email program and the map will be designed by two different software providers, oblivious to one another.  The programs must somehow exchange information <em>without knowing anything about each other</em>—without even knowing the other exists.</p>
  <p id="p357">As it happens, such a mechanism has long existed for <em>manipulation</em> software—copy-and-paste.  This mechanism uses the platform as an intermediary.  When the user “copies” a picture in a drawing program, the program hands data off to the platform.  When the user then “pastes” the picture into a word processing document, the program requests data from the platform, and handles it according to its type.  The drawing and word processing programs know nothing of each other—they know only of the platform and standard data exchange formats.</p>
  <p id="p358">Extending this concept to information software involves two additional concerns:</p>
  <ul><li><p id="p31">Autonomy.  As befitting manipulation software, copy-and-paste requires explicit manipulation by the user.  Information software must be able to share information implicitly and autonomously, with no user interaction.</p></li>
  <li><p id="p32">Translation.  An email is not a map location.  Nor is a website a calendar event, nor a word processing document an encyclopedia entry.  The information must be <em>translated</em> from one form to another.</p></li>
  </ul><p id="p359">Given that this platform exists to promote inference from the environment, let us take some inspiration from a <em>biological</em> environment.  The very essence of a biological environment is autonomous translation.  Plants translate sunlight into fruit, large animals translate fruit into dung, small animals translate dung into soil, plants translate soil into fruit.  <b>An ecosystem is a network of individual components which consume nutrients and translate them to an enriched form consumed by others, autonomously and with no knowledge of the system as a whole.</b></p>
  <p id="p360">If we adopt this process in software, considering our “nutrients” to be information, we have an <em>information ecosystem</em>.  Consider this system:</p>
  <p><img width="805" height="403" src="http://worrydream.com/MagicInk/p/ecosystem_platform.png" alt="image" /></p>
  <p id="p361">The components above the platform are <b>views</b>.  This is the software that the user sees and interacts with.  Views interact with the platform in two ways:</p>
  <ul><li><p id="p33">Views <em>nominate</em> a <b>topic of interest</b>.  For example, if the user is reading an email, she is probably interested in information related to the contents of the email.  The email program would give the email to the platform as a topic.  This is analogous to “copying,” but happens implicitly.</p></li>
  <li><p id="p34">Views <em>request</em> topics of interest, of some particular type.  The map, for example, would request topics that have a geographical location associated with them.  If a restaurant were a topic, the platform would give it to the map, and the map would display it.  This is analogous to “pasting,” but again is implicit.</p></li>
  </ul><p id="p362">The components below the platform are <b>translators</b>.  The platform gives them information objects, which they convert from one type to another and return to the platform.  Sometimes this involves decomposing an object into constituent parts (“digesting” it); other times, it involves enriching the object with additional information.</p>
  <p id="p363">The platform itself acts as an intermediary between components, attempting to fulfill requests by constructing a chain of translators to convert topics into the requested types.</p>
  <p id="p364">This is how the behavior in the original example might come about:</p>
  <p><img width="487" height="1041" src="http://worrydream.com/MagicInk/p/ecosystem_chain.png" alt="image" /></p>

  <p id="p365">Although it is clearer to visualize the process as described, an actual implementation would have to be <em>lazy</em>, driven by demand.  That is, the process would start at the end with the map’s request, and the platform would then construct the chain <em>back</em> toward the email program, according to the types and properties of the objects each component claims to consume and produce.  This is necessary for efficiency reasons, but also explains how EpicurioCity knows the area to look for restaurants—the map actually requests objects with locations <em>around a particular area</em>, and EpicurioCity then attempts to produce objects that match this type.  It also explains how EpicurioCity knows how many objects to produce from its almost infinite collection—it produces objects as long as the map consumes them.</p>
  <p id="p366">Assuming the proper components are in place, we now have a system that maps pizza places in response to Joe’s email.  Now, consider what would happen if, instead of receiving an email, I were to type the word “pizza” into a document.  Surely the last word typed would be nominated as a topic.  It would then get picked up by EpicurioCity and translated into restaurants, and these would show up on the map.  Thus, we have the remarkable emergent behavior of being able to look up pizza places simply by <em>typing the word “pizza” anywhere on the computer</em>.</p>
  <p id="p367">This sort of emergent, non-designed behavior is the overall goal of the platform.  Through topic nomination, the system models the user’s immediate interests, and through translation, every view can attempt to serve these interests in any way possible.</p>
  <p id="p368"><strong>Confidence.</strong> At this point, the biggest problem concerns the question, “Just <em>what</em> should be a topic?”  If every component  nominated everything that could possibly be relevant, the map would become so cluttered as to be useless.  The problem is addressed by recasting the question more fuzzily:  “How <em>much</em> is something a topic?”</p>
  <ul><li><p id="p35">Every topic is nominated with a <b>level of confidence</b>.** I will consider confidence levels to lie between 0 and 1, so that multiplication makes sense.      An email that I’m reading <em>right now</em> would be nominated with high confidence.  When I finish reading it and move on to something else, its confidence diminishes.  The title of a paper I’m currently typing in would have high confidence; the title of a paper I haven’t touched for a few minutes is lower.  The paragraph that I’m working on has higher confidence than surrounding paragraphs.</p></li>
  <li><p id="p36">Translators produce <b>dilution of confidence</b>.  As they translate, they multiply the object’s confidence level by their own confidence in the translation.  The text digester will have more confidence in unusual, prominent words, and words that seem related to other topics or the context in which topics were found.  Partial or tenuous matches can be translated with low confidence instead of omitted.  For example, a movie translator from CelluDroid.com might translate “pizza” into the film “Pizza Cato: The String Cheese Connection,” but because the name is only partially matched and the context of the consumed object contains no references to movies, the confidence would be low.</p></li>
  <li><p id="p37">Views, such as the map, use confidence to determine the <b>graphical emphasis</b> of displayed objects.  This is a critical part of the graphic design.  Beyond simply adjusting size, emphasis can involve all sorts of standard graphical techniques—contrast, color saturation, shading, shadowing, grouping, or placement on a different graphic entirely.  Objects with confidence below some threshold will probably be discarded.</p></li>
  </ul><p id="p369">As events cause confidence levels to change, the changes propagate through the chain of translators, adjusting the emphasis of displayed objects.</p>
  <p id="p370"><strong>Feedback.</strong>  The next problem with this system is inappropriate translations.
  <img width="183" height="92" src="http://worrydream.com/MagicInk/p/ecosystem_email.png" alt="image" /> Consider again my friend’s email. The text digester might pick out the word “dude,” which would go through the business listings at AgoraBiblia.com, resulting in the neighborhood dude ranch showing up on my map.  This would be a nuisance if it occurred every time I received an email from my friend.</p>
  <p id="p371">The problem is addressed through <b>backpropagation of feedback</b>.  Feedback can be either explicit or implicit.  Explicitly, I can indicate to the map that I am uninterested in dude ranching.  This negative feedback is returned to the AgoraBiblia.com translator, resulting in low confidence in future dude ranch matches.  The feedback may even propagate back to the text digester, slightly lowering the confidence that the word “dude” indicates a topic of interest.  Implicitly, simply looking at the map without indicating interest in the dude ranch will cause a slight negative feedback, resulting in its de-emphasis over time.  On the other hand, if I frequently click on pizza places, positive feedback will backpropagate through the chain of translators, increasing confidence in all things pizza-related and resulting in their emphasis on the map.</p>
  <p id="p372">In effect, the entire environment becomes a learning system, tailoring itself to the individual user.  While topics model the user’s immediate interests, the history acquired through feedback allows the system to model the user’s long-term characteristics.</p>
  <p id="p373"><strong>Protocol.</strong>  The last problem I will consider here is the political issue of protocol creation.  Just what is a Restaurant object, and who decides that?  Standards, especially premature ones, stifle invention and progress, but anarchy results in incompatibility.  It may be possible to address this problem through namespacing and published proprietary protocols.</p>
  <p id="p374">To answer the above question, there <em>is</em> no Restaurant object.  Instead, EpicurioCity produces a <tt>com.EpicurioCity.Restaurant</tt> object,** Or however namespacing is spelled in the implementation language.    whose protocol is defined and managed by EpicurioCity.com.  This proprietary object can be composed of other proprietary objects, as well as some standard objects defined by the platform, such as <tt>Text</tt>, <tt>Keyword</tt>, and <tt>Location</tt>.  Note that this proprietary Restaurant is not hindered from showing up on the map, since the map will accept anything with a Location (and presumably some other standard properties such as a name and description).** In object-oriented terminology, <tt>com.EpicurioCity.Restaurant</tt> conforms to the Mappable interface, and the map requests Mappable objects.  However, this “interface” can be very informal, and even unknown to the Restaurant.  If the Restaurant happens to define enough standard properties, it can be mapped.      A restaurant guide view, on the other hand, would be written to take advantage of the extra information that <tt>com.EpicurioCity.Restaurant</tt> offers—ratings, reviews, and such.</p>
  <p id="p375">When another provider, CuisineCousins.com, develops a competing restaurant translator, it can follow EpicurioCity’s published protocol and produce <tt>com.EpicurioCity.Restaurant</tt> objects.  This makes their new translator immediately compatible with existing views.  Meanwhile, the translator can <em>simultaneously</em> offer their own objects, such as a <tt>com.CuisineCousins.Eatery</tt>, with whatever advantages over EpicurioCity’s protocol.  View providers can then update their software to also accept CuisineCousins’s protocol, if CuisineCousins offers a compelling enough advantage.</p>
  <p id="p376">If a de facto standard emerges and stabilizes, it might eventually get canonized as the official <tt>Restaurant</tt> object.  Even then, though, providers will be able to add proprietary namespaced extensions to it.</p>
  <p id="p377"><strong>Modularity.</strong>  An obvious benefit to this platform is that it enforces modularity between data and views.  Unlike current systems, in which almost all data and functionality is locked up behind a user interface, every service on this system is available to every view.  More subtly but just as importantly, the fact that translators have no end-user interface means they can be created by engineers.  Only the views must be designed for users.  Meanwhile, a designer who is dissatisfied with a view can simply create and release a replacement, with no engineering worries about data acquisition.  Because the system can be easily improved without cross-disciplinary concerns, creativity and invention should flourish.</p>
  <p id="p378">Today’s ubiquitous <acronym title="Graphical User Interface">GUI</acronym> has its roots in Doug Engelbart’s groundshattering research in the mid-’60s.  The concepts he invented were further developed at Xerox <acronym title="Palo Alto Research Center">PARC</acronym> in the ’70s, and successfully commercialized in the Apple Macintosh in the early ’80s, whereupon they essentially <em>froze</em>.  Twenty years later, despite thousand-fold improvements along every technological dimension, the concepts behind today’s interfaces are almost <em>identical</em> to those in the initial Mac.  Similar stories abound.  For example, a telephone that could be “dialed” with a string of digits was the hot new thing ninety years ago.  Today, the “phone number” is ubiquitous and entrenched, despite countless revolutions in underlying technology.  Culture changes much more slowly than technological capability.** Other obsolete but entrenched designs: the QWERTY key layout (intentionally sub-optimal to reduce typewriter jams), the von Neumann architecture (see John Backus, <a href="http://www.stanford.edu/class/cs242/readings/backus.pdf">Can Programming Be Liberated from the von Neumann Style?</a>, 1978); C and UNIX (see Richard Gabriel, <a href="http://www.jwz.org/doc/worse-is-better.html">The Rise of “Worse is Better”</a>, 1991). 
   </p>
  <p id="p379">The lesson is that, even today, we are designing for tomorrow’s technology.  Cultural inertia will carry today’s design choices to whatever technology comes next.  In a world where science can outpace science fiction, predicting future technology can be a Nostradamean challenge, but the responsible designer has no choice.  A successful design will outlive the world it was designed for.</p>
  <p id="p380">With what artifact will the people of tomorrow learn information?  I believe that in order for a personal information device to be viable in the long term, it must satisfy two conflicting criteria:  portability and readability.</p>
  <ul><li><p id="p38"><strong>Portability.</strong>  Consider today’s ubiquitous information device—the book.  We have the technology to manufacture 5000-page desk-sized tomes, but despite the high information content, such books are rare.  The reason is simply that they can’t be carried around.  As people increasingly expect information on demand, portability will become ever more critical.  Today, people can talk to anyone on the planet by reaching into a pocket; tomorrow’s information device must be just as accessible.  Like a wallet and keys, the computer will be dropped into the pocket or purse before leaving the house.** Ideally, it will even supplant both wallet and keys.     This implies light weight and small volume.</p></li>
  <li><p id="p39"><strong>Readability.</strong>  Consider again the book.  We have the technology to produce books smaller than a business card, but despite the improved portability, such books are also rare.  The supremely-portable postage-stamp-sized book is non-existent.  The catch:  <b>Although technology miniaturizes, the human eyespan remains a fundamental constant.</b>  In order to compete with the book, tomorrow’s information device must provide a book-sized surface area.  Anything less cannot be read and skimmed comfortably, and cannot support spatially-distributed information graphics.</p></li>
  </ul><p id="p381">To resolve these contrasting size constraints, I predict a computer the size and thickness of a sheet of paper.  Like paper, its entire surface is a graphical display.  When in use, it is rigid; when not in use, it collapses and can be folded or rolled up (or crumpled!) and tucked into a pocket or purse.</p>
  <p id="p382">Regardless of whether I’ve guessed its form accurately, we can predict the device’s expected characteristics by extrapolating technological trends.  Consider the capabilities relevant to context-sensitive information graphics:  graphical output, history, environment, and user interaction.</p>
  <ul><li><p id="p40"><strong>Graphical output.</strong>  To serve as a book, the device must have a sufficiently large reading area and high pixel resolution.  To serve as a computer, the device must produce dynamic color graphics.  In matching each of today’s devices, tomorrow’s device will overcome the shortcomings of the other.  Dynamic graphics with print resolution will open up a world of possibilities for detailed information graphics which are impossible today in either medium.</p></li>
  <li><p id="p41"><strong>Environment.</strong>  Because the user will carry this device everywhere, the device’s environment will literally be the user’s own.  Assuming a sufficient networking model, the device will be able to sense an enormous amount of information from the environment—geographical location, physical surroundings (streets, stores, transportation options, entertainment options), social surroundings (friends, strangers with interests in common, strangers who can serve a need), and more.  The device will have a far better sense of the user’s environment than the user herself.</p></li>
  <li><p id="p42"><strong>History.</strong>  Since its inception, electronic storage has exponentially increased in density and decreased in cost.  We can fully expect tomorrow’s device to have onboard capacities that stagger modern sensibilities.  But, perhaps more importantly, ubiquitous network access will make memory effectively <em>unlimited</em>.  The device will have the means to remember everything the user has ever done and every environment in which she did it.  With such a tremendous history and sense of the environment, software will have an unprecedented potential to predict the user’s current context.</p></li>
  <li><p id="p43"><strong>Interaction.</strong>  Touch or motion-based manipulation is somewhat more efficient than the mouse.  Eye-tracking and speech may be better still, although even these are unlikely to match the order-of-magnitude improvements predicted for the capabilities above.  But none of these mechanisms will ever approach the sheer amount of information that can be absorbed by the eye.  No matter what new interactive technology comes along, the bandwidth between the device and the user will remain not merely asymmetric, but utterly lopsided.</p></li>
  </ul><p id="p383">Interaction is already a bottleneck.  It will get much worse as graphics, environment, and history experience their expected breakthroughs.  To me, the implication is clear—the principles of information software and context-sensitive information graphics will become <em>critical</em> as technology improves.</p>
  <p id="p384">The future will be context-sensitive.  The future will not be interactive.  </p>
  <p id="p385">Are we preparing for this future?  I look around, and see a generation of bright, inventive designers wasting their lives shoehorning obsolete interaction models onto crippled, impotent platforms.  I see a generation of engineers wasting their lives mastering the carelessly-designed nuances of these dead-end platforms, and carelessly adding more.  I see a generation of users wasting their lives pointing, clicking, dragging, typing, as gigahertz processors spin idly and gigabyte memories remember nothing.  I see machines, machines, <em>machines</em>.</p>
  <p id="p386">I expect that designers who cling to these models will appear to the next generation like classical physicists as the world turned quantum, like epicycle-plotters as Kepler drew ellipses, like Aristotelians as Galileo stood atop the tower at Pisa.  No matter how hard they work or how much they invent, these designers will not be revered as pioneers.  They are blazing trails through a parking lot.</p>
  <p id="p387">Our pioneers are those who <em>transcend</em> interaction—designers whose creations <em>anticipate</em>, not <em>obey</em>.  The hero of tomorrow is not the next Steve Wozniak, but the next William Playfair.  An artist who redefines how people learn.  An artist who paints with magic ink.</p>
  <p class="logo"><img width="435" height="83" src="http://worrydream.com/MagicInk/p/logo.png" alt="image" /></p>
  <div id="backmatter">
  <h2 id="summary">Summary</h2>
  <p id="p388">Software design consists of graphic design (drawing pictures) and industrial design (allowing for mechanical manipulation).</p>
  <p id="p389">Information software is for learning an internal model.  Manipulation software is for creating an external model.  Communication software is for communicating a shared model.</p>
  <p id="p390">Manipulation software design is hard, but most software is information software.</p>
  <p id="p391">Information software design is the design of context-sensitive information graphics.  Information software is not a machine, but a medium for visual communication.</p>
  <p id="p392">Context can be inferred from the environment, which can include physical sensors, other information software, documents created with manipulation software, and data such as email which acts as a user profile.</p>
  <p id="p393">Context can be inferred from a history of past environments and interactions.  Last-value predictors provide a rudimentary approach.  Learning predictors can infer patterns and make dynamic predictions.</p>
  <p id="p394">Context can be inferred from user interaction, but only as a last resort.  The best way to reduce or eliminate interaction is through information-rich graphic design that uses the environment and history.  Remaining interaction can be reduced with graphical manipulation, relative navigation, and tight feedback loops.</p>
  <p id="p395">The information software revolution will require public recognition that information software is a medium of visual communication, designers with talent, skill, and tools, simple and general platforms, and an environment that encourages creativity and sharing.</p>
  <p id="p396">A design tool for dynamic graphics that infers behavior from mockups may allow for natural-feeling creative design with no engineering-related distractions.</p>
  <p id="p397">Learning predictors exist and are effective.  For them to become widespread, simple abstractions must be invented.</p>
  <p id="p398">An information ecosystem of views and translators may be able to offer relevant information of all forms with minimal interaction.  Key aspects include topic nomination and translation, confidence levels, learning through feedback, and a fine-grained modular structure wherein small software providers can thrive.</p>
  <p id="p399">As technology related to graphics, the environment, and history undergoes revolutionary improvements, interaction will become even more of a critical bottleneck.  The best approach is to work towards eliminating it.</p>
  <p id="p400">Two centuries ago, Playfair invented statistical graphics and changed the world.  The time is ripe for another designer to invent the fundamental context-sensitive graphical forms, and change the world again.</p>
  <h2 id="recommended_reading">Recommended reading</h2>
  <p id="p401">Most of the works cited in this paper are recommended—just skim up the sidenotes.  The following landmark books deserve special mention:</p>
  <p id="p402">Edward Tufte. <a href="http://www.amazon.com/gp/product/0961392142">The Visual Display of Quantitative Information</a> (2001), <a href="http://www.amazon.com/gp/product/0961392118">Envisioning Information</a> (1990), <a href="http://www.amazon.com/gp/product/0961392126">Visual Explanations</a> (1997).  The three testaments of the information design bible.  (A <a href="http://www.edwardtufte.com/tufte/books_be">fourth</a> is on the way.)  If you’ve already read them, read them again.</p>
  <p id="p403">Scott McCloud. <a href="http://www.amazon.com/gp/product/006097625X">Understanding Comics</a> (1994).  Like information graphics, comics convey information through arrangements of words and pictures—they are Tufte’s “small multiples” applied to storytelling.  McCloud’s analysis of how people read and understand the visual language of comics is essential reading for all information graphic designers.</p>
  <p id="p404">Thomas Kuhn. <a href="http://www.amazon.com/gp/product/0226458083/">The Structure of Scientific Revolutions</a> (1962).  The only purpose of incremental improvement to a status quo is to reveal its flaws.  Progress occurs when the status quo is <em>replaced</em>.</p>
  <h2 id="acknowledgments">Acknowledgments</h2>
  <p id="p405">I am grateful for the detailed and helpful feedback I received on earlier drafts from Andy Likuski, Jonathan Harel, Justin McCarthy, Jon Nakasone, and <a href="http://lostgarden.com/">Daniel Cook</a>.</p>
  <p id="p406">I also appreciate everyone who sent in feedback on the <acronym title="Bay Area Rapid Transit">BART</acronym> widget, and particularly <a href="http://www.newmediasoup.com/">Joel Dreisbach-Penner</a>, <a href="http://www.wjdesign.com/">Walter Jew</a>, and <a href="http://www.tavallai.com/">Omid Tavallai</a> for their contributions.  This paper sprung from that little widget.</p>
  <p id="p407">This paper was formatted with a modified version of <a href="http://daringfireball.net/">John Gruber</a>’s <a href="http://daringfireball.net/projects/markdown/">Markdown</a> utility.  If you like, you can see the <a href="http://worrydream.com/MagicInk/markdown.pl">modified Markdown</a> and the paper’s <a href="http://worrydream.com/MagicInk/draft.txt">source code</a>.</p>
  <p id="p408">Half of this paper was written in the San Leandro and Dublin public libraries.** Have you been to your public library?  It’s like Starbucks, but free of charge, noise, and corporate branding.    The other half was written in Cordonices Park and Live Oak Park in Berkeley.</p>
  <p id="p409">Please see my <a href="http://worrydream.com/">website</a> for a biographical avalanche.</p>
  <h2 id="epilogue">Epilogue</h2>
  <blockquote>
    <p id="p48">In the early days, I was solving one problem after another after another; a fair number were successful and there were a few failures. I went home one Friday after finishing a problem, and curiously enough I wasn’t happy; I was depressed. I could see life being a long sequence of one problem after another after another. After quite a while of thinking I decided, “No, I should be in the mass production of a variable product. I should be concerned with all of next year’s problems, not just the one in front of my face.” By changing the question I still got the same kind of results or better, but I changed things and did important work. I attacked the major problem—How do I conquer machines and do all of next year’s problems when I don’t know what they are going to be? How do I prepare for it? How do I do this one so I’ll be on top of it? How do I obey Newton’s rule? He said, “If I have seen further than others, it is because I’ve stood on the shoulders of giants.” These days we stand on each other’s feet!</p>
    
    <p id="p49">You should do your job in such a fashion that others can build on top of it, so they will indeed say, “Yes, I’ve stood on so and so’s shoulders and I saw further.” The essence of science is cumulative. By changing a problem slightly you can often do great work rather than merely good work. Instead of attacking isolated problems, <b>I made the resolution that I would never again solve an isolated problem except as characteristic of a class.</b><br />—Richard Hamming, <a href="http://www.cs.virginia.edu/~robins/YouAndYourResearch.html">You and Your Research</a> (1986)</p>
    
    <p id="p50">Before I release v1.0 of the <acronym title="Bay Area Rapid Transit">BART</acronym> widget, I’d like to write a little paper about its design…<br />—Bret Victor (2005)</p>
  </blockquote>
  </div>  


title: a-t-elle vraiment été en production ?
url: https://blog.imirhil.fr/2016/10/09/docker-container-hell.html
created_at: 2017-08-10 23:19:51
updated_at: 2017-08-10 23:19:51
mimetype: text/html
domain_name: blog.imirhil.fr
http_status: 200
content:
  <p><a href="https://www.docker.com/">Docker</a>. Une technologie si jeune (2013) et pourtant dorénavant vendue un peu partout comme la technologie miracle qui va résoudre tous les problèmes de déploiement, vous rendre riche et vous ramener l’être aimé.</p><p>Et pourtant, en pratique cette technologie me pose pas mal de problèmes et de cas de conscience.
  Revue de détails.</p>
  <p>Le but initial est un peu dans le prolongement de ce qu’a fait Java : construire une fois, déployer partout (« build once, deploy everywhere »).
  On ne peut nier que l’idée initiale soit bonne tellement le problème du déploiement est un véritable parcours du combattant en pratique.</p>
  <p>Un logiciel est rarement un morceau totalement isolé.
  Il vient avec son interpréteur, ses bibliothèques, nécessite un moteur de base de données, des fichiers de configuration, un serveur web et tout un tas d’autres choses.
  Cet écosystème va être à gérer tout au long de la chaîne de fabrication, que ça soit en développement, en phase de validation/qualification, en recette ou encore en production.</p>
  <p>Avant Docker, chaque étape nécessitait de remonter l’ensemble de l’environnement, d’y installer le logiciel (parfois à partir des sources via compilation), de le lancer…
  Et c’était un gros bordel, puisque la moindre déviation entre deux environnements pouvait conduire à des bugs non reproductibles en fonction de l’endroit où l’on lançait le logiciel, d’où l’apparition du trop fameux « Ça marche chez moi ™ ».</p>
  <p>Avec Docker, point de tout cela.
  Les développeurs construisent une image du logiciel qui va contenir tout ce qui lui faut pour fonctionner, depuis l’OS jusqu’au logiciel lui-même en passant par toutes les autres briques.
  S’ils ont satisfaction de ce qu’ils ont obtenu, ils livreront à l’étape suivante cette image, qui n’aura pas bougé d’un iota et pourra être lancée à l’identique, et ainsi de suite jusqu’à la très attendue mise en production.</p>
  <p>Ça c’est la théorie. Maintenant, la pratique :D</p>
  <p>Le premier rempart à l’usage de Docker est que les besoins entre la production et le développement sont assez différents.<br />Un développeur va vouloir accéder facilement aux journaux, si possible en mode <code class="highlighter-rouge">debug</code>, alors que la prod préférera les envoyer à un <a href="https://www.elastic.co/fr/products/logstash">Logstash</a> et en mode <code class="highlighter-rouge">info</code> voire <code class="highlighter-rouge">warn</code> ou <code class="highlighter-rouge">error</code>.
  Un développeur préférera utiliser directement le serveur d’application léger de son choix comme <a href="https://eclipse.org/jetty/">Jetty</a>, <a href="http://code.macournoyer.com/thin/">Thin</a> ou <a href="http://gunicorn.org/">Gunicorn</a>, alors que la production configurera un backend <a href="https://nginx.org/">Nginx</a> devant ou utilisera des serveurs d’application plus puissant tel que <a href="https://tomcat.apache.org/">Tomcat</a> ou <a href="https://www.phusionpassenger.com/">Passenger</a>.
  Un développeur préférera sûrement compiler en mode <code class="highlighter-rouge">debug</code> pour avoir des retours utilisables en cas de problème alors que la production insistera pour le faire en mode <code class="highlighter-rouge">release</code>.
  La production voudra mettre en place un pare-feu, ou ses outils de supervision de parc, dont le développement n’a même aucune idée de l’existence puisque ça ne fait pas partie de ses compétences !
  J’évite même de parler d’intégrer ses outils de développement à un environnement Docker, par exemple lancer son projet Java présent sur son Eclipse local sur le Tomcat présent sur l’image Docker, ça ferait trop mal.<br />Bref, en pratique, c’est compliqué…
  On peut quand même s’en sortir pour certains morceaux, surtout grâce à certaines fonctionnalités de Docker, mais l’intérêt en devient limité.</p>
  <p>Pour les environnements différents, il « suffit » que les devs travaillent hors Docker à l’ancienne, puis une fois parvenus à quelque chose de satisfaisant, s’attaquent à la construction d’une image Docker. 
  Ça implique une espèce de mini-chaîne complète (dev/test/validation/qualification/production) faite uniquement par les développeurs, afin de s’assurer que l’image finale est à peu près conforme à ce qui est attendu (quel logiciel fournir, quelles dépendances disponibles…).
  Parce que du coup ça n’implique plus que si ça tourne sur leur environnement de dev, ça tourne sur l’environnement Docker.
  Pas forcément très folichon niveau processus sinon avoir ramené toutes les considérations des autres étapes sur celle de développement.</p>
  <p>Pour les besoins propres à la production (pare-feu, monitoring…), Docker permet de créer une image à partir d’une autre.
  La production repartira donc de l’image fournie par les développeurs pour refaire sa propre image incluant tout le nécessaire.
  Ça casse aussi l’intérêt de Docker qui garantit que ça tournera en production, étant donné que la production peut elle aussi introduire des bugs.
  Par exemple l’installation d’un pare-feu va peut-être installer une bibliothèque utilisée par l’application mais dans une version différente.
  La production devra donc aussi repasser une bonne partie des étapes précédentes (test/validation/qualification) avant la mise en production réelle.
  Pas folichon non plus.</p>
  <p>Dans les deux cas il aurait été plus intelligent que le dev et la prod travaillent ensemble dès le départ (qui a dit <a href="https://fr.wikipedia.org/wiki/Devops">Devops</a> ?) pour fournir une image Docker à la QA qui partira en production telle quelle une fois approuvée.
  Mais alors qu’on utilise Docker ou n’importe quelle autre technologie (virtualisation classique, automatisation d’installation via un outil comme <a href="https://www.chef.io/">Chef</a>, <a href="https://puppet.com/">Puppet</a>, <a href="https://www.ansible.com/">Ansible</a> ou <a href="https://saltstack.com/">Salt</a>) on aurait obtenu le même résultat.</p>
  <p><strong>LE</strong> gros point noir selon moi.</p>
  <p>Docker repose sur le principe d’<strong>immuabilité</strong> des images.
  Une fois une image livrée, on n’y touche plus et au prochain changement nécessaire, on refait une image depuis zéro.</p>
  <p>Ça pose un problème assez énorme en termes de sécurité.
  Le jour où vous avez une faille dans le logiciel livré, on comprend bien qu’on n’échappera de toute façon pas à un correctif, une regénération, un passage intégral de toute la chaîne de qualification et une nouvelle mise en production.
  Mais si c’est une bibliothèque utilisée par le logiciel, par exemple <a href="https://www.openssl.org/">OpenSSL</a> qui connaît au moins une faille critique de sécurité par jour ?</p>
  <p>Dans une infrastructure sans Docker, les gentils administrateurs systèmes se seraient sagement connectés en SSH aux machines impactées, auraient simplement fait un <code class="highlighter-rouge">apt update &amp;&amp; apt upgrade</code> et basta.<br />Un patch de sécurité n’introduisant pas de changement de fonctionnalités et les développeurs de OpenSSL faisant bien leur travail, ils livrent des versions patchées assurant la rétro-compatibilité avec les versions précédentes.
  Les admins peuvent donc appliquer le patch assez rapidement sans avoir besoin de consulter les développeurs.
  Et la faille est rapidement corrigée avec un risque de régression négligeable (qu’on a parfaitement su accepter et maîtriser pendant des décennies, et même Docker ne pourra jamais garantir le 0 bug). En prime, le logiciel final, lui, n’a pas changé de version.</p>
  <p>Dans une infrastructure dockerisée, c’est une autre histoire…<br />Les conteneurs devant être immuables, il est interdit aux administrateurs de se connecter aux machines pour les mettre à jour.
  Un changement de sécurité sur une bibliothèque réclame donc de redérouler l’intégralité de la chaîne : on met à jour la bibliothèque, on construit une nouvelle image Docker (qui change donc de version), on repasse toute la QA, on met en production.
  La mise en production nécessite un arrêt de l’ancienne image, la migration des données sur la nouvelle image et le redémarrage du système.
  Bref, vous allez tourner un long moment avec votre image trouée avant d’avoir pu fixer le problème…</p>
  <p>Quand on évoque ce problème avec la communauté Docker, ils avancent alors leur recette miracle : l’<strong><a href="https://fr.wikipedia.org/wiki/Intégration_continue">intégration continue</a></strong>.<br />Certes, si vous réussissez à intégrer l’intégralité de la QA (tests unitaires, tests fonctionnels, tests d’intégration, tests de validation, tests de qualification, tests de recette) dans une suite de tests automatiques, c’est peut-être envisageable de faire une modification dans le code, de cliquer sur un bouton et d’avoir automagiquement la nouvelle version en production.<br />En pratique, on a déjà du mal à atteindre 100% de couverture sur les tests unitaires, et plus on monte dans les étages plus c’est compliqué. Les tests fonctionnels doivent simuler des humains presse-boutons, les tests d’intégration sont un enfer à réaliser vu le nombre de composants en jeu, les tests de qualification sont souvent longs (tests de performance, de tenue de charge…), etc.<br />Sauf à investir un budget colossal en automatisation de tests, l’intégration continue revient finalement à se restreindre à un sous-ensemble des tests (on se limite aux cas principaux et on laisse tomber les cas dégradés par exemple) et donc à potentiellement laisser passer des bugs lorsque les administrateurs pousseront le bouton « Déployer » après une modification rapide pour fixer une faille de sécurité.
  Seuls les grands comme Google ou Amazon peuvent se permettre l’outillage de détection d’un problème en production (du <a href="https://fr.wikipedia.org/wiki/Test_A/B">test A/B</a> par exemple) et donc d’alléger ou de supprimer toutes les phases de test : on déploie plus ou moins à l’arrache, si ça merde, on revient en arrière immédiatement !</p>
  <p>Les plus perspicaces d’entre vous auront noté que tout ce qui précède repose sur une hypothèse très forte : on doit être mainteneur de l’image Docker utilisée !<br />En pratique, beaucoup utilisent des images pré-construites qu’ils assemblent selon leurs besoins.
  Il existe des dépôts d’images, dont le plus connu est le <a href="https://hub.docker.com/">Hub Docker</a>.
  Du coup, en cas de faille, il va vous falloir attendre une mise-à-jour.
  L’unique mainteneur est <a href="http://savoiragile.com/2015/03/19/mefiez-vous-des-bus/">passé sous un bus</a> ? Vous êtes mal…<br />En termes de sécurité, c’est même encore plus gore dans ce cas, puisque vous n’avez pas beaucoup de moyens de vous assurer que l’image de 600Mo (n’oublions pas que ça intègre un OS complet) que vous allez utiliser n’intègre pas une porte dérobée ou une version obsolète d’une bibliothèque, surtout quand l’image <a href="https://github.com/discourse/discourse_docker/blob/master/image/base/install-nginx#L18-L34">est réalisée avec les pieds</a> et ne permettent aucune vérification a posteriori.
  Les paquets Debian sont par exemple construits en <a href="https://wiki.debian.org/ReproducibleBuilds">compilation reproductible</a> et vous pouvez facilement vous assurer que le <code class="highlighter-rouge">nginx</code> que vous utilisez est bien le même que celui du paquet Debian officiel alors que sous Docker, je vous souhaite bien du courage ne serait-ce que pour connaître le numéro de version utilisé.
  Sur chaque image Docker, il faudrait faire une revue assez poussée de ce qui se trouve sur l’image, au moins pour lister les différents composants et bibliothèques intégrés et ainsi connaître les impacts réels sur votre parc d’un correctif de sécurité.</p>
  <p>L’humanité s’est battue pendant des décennies pour mettre en place le multi-processus, puis le multi-thread…
  Mais ça c’était avant ! Avec Docker, vous ne pouvez lancer qu’un seul et unique processus dans une image. Pas plus.<br />Ça n’a l’air de rien, mais c’est très handicapant en pratique.
  Vous ne pouvez pas avoir <a href="https://fr.wikipedia.org/wiki/Cron"><code class="highlighter-rouge">cron</code></a> à tourner régulièrement par exemple. Donc pas de <a href="https://linux.die.net/man/8/logrotate"><code class="highlighter-rouge">logrotate</code></a>.
  Vous ne pouvez pas avoir de serveur <a href="http://www.openssh.com/">SSH</a> pour vous connecter à distance sur l’image.
  Pas de serveur de mail non plus, par exemple pour les rapports journaliers de <a href="https://wiki.debian-fr.xyz/Logwatch"><code class="highlighter-rouge">logwatch</code></a> (de toute façon on n’a pas <code class="highlighter-rouge">cron</code> pour les lancer…). Rien.
  La seule et unique chose que vous allez pouvoir lancer sera donc votre application. Et c’est tout.</p>
  <p>Le problème est que comme on l’a vu précédemment, une application se suffit généralement assez peu à elle-même.
  Elle nécessite par exemple une base de données, un frontal web, de pouvoir envoyer des courriels…
  Et donc de lancer plusieurs processus !</p>
  <p>En méthode <em>quick &amp; dirty</em>, vous pouvez contourner le problème en lançant un bête script bash qui lancera tout en arrière-plan, ou plus malin un gestionnaire de processus comme <a href="http://supervisord.org/">supervisord</a> ou <a href="https://github.com/SamSaffron/pups">pups</a> qui se chargera lui-même de lancer tout le reste de ce que vous avez besoin.
  C’est tout de même assez galère à faire puisque votre distribution adorée vous fournira des scripts de démarrage pour le système d’init habituel (sysv, systemd, upstart…) et non pour supervisord ou pups, il vous faudra donc faire un travail de portage pour chaque composant nécessaire.</p>
  <p>La méthode recommandée par Docker pour gérer vos environnements est l’utilisation de <a href="https://docs.docker.com/compose/">Docker Compose</a>.
  Vous allez créer autant de conteneurs que de composants de votre écosystème (un pour l’application, un pour la base de données, un pour le serveur de courriel…) et les assembler entre-eux pour qu’ils communiquent correctement.<br />Pour certains composants comme la base de données, je trouve ça intéressant de séparer du reste, exactement comme on l’aurait fait dans une infrastructure non virtualisée.
  Pour d’autres, comme un serveur de courriel dédié pour envoyer 3 courriels au mois, c’est du gaspillage de ressources flagrant.
  Et pour la majorité, c’est d’une prise de tête sans nom…
  Par exemple dans une application <a href="http://rubyonrails.org/">Ruby-on-Rails</a> utilisant <a href="http://sidekiq.org/">Sidekiq</a> comme ordonnanceur, on va se retrouver à avoir 4 conteneurs :</p>
  <ul><li>1 pour l’application Ruby-on-Rails</li>
    <li>1 pour le backend web <a href="https://nginx.org/">nginx</a></li>
    <li>1 pour Sidekiq</li>
    <li>1 pour le serveur <a href="http://redis.io/">Redis</a> qui sert à la communication entre RoR et Sidekiq</li>
  </ul><p>Alors que tout mettre sur la même machine se justifie largement tant qu’on n’a pas une volumétrie délirante (je tiens les 200.000 vues quotidiennes sur <a href="https://tls.imirhil.fr/">Cryptcheck</a> sans soucis avec 1 seul conteneur), on se retrouve avec 4 machines à gérer et à devoir mettre à jour (les 4 utilisent OpenSSL par exemple) et une duplication de l’environnement Ruby (RoR &amp; Sidekiq).
  On risque aussi de rencontrer des dégradations de performances, puisqu’on passe de communications sur la <a href="https://fr.wikipedia.org/wiki/Localhost">boucle locale</a> voire des <a href="https://fr.wikipedia.org/wiki/Berkeley_sockets">sockets UNIX</a> à une communication TCP/IP externe.
  Et la sécurité devient tout autant un enfer, avec de la gestion de pare-feu à mettre en œuvre.</p>
  <p>Pour gérer autant de conteneurs, vous allez aussi devoir passer à des outils de gestion de déploiement et d’orchestration, comme <a href="http://kubernetes.io/">Kubernetes</a>.
  Toujours pareil, quand vous vous appelez Google, Wikimedia, SAP ou Ebay, c’est peut-être gérable. Si vous êtes une petite boîte, ça risque d’avoir un surcoût non négligeable et pas forcément rentable.</p>
  <p>On a vu juste avant que Docker incite à créer plein de conteneurs pour les assembler entre eux.
  En interne pour la fabrication de ses propres images, Docker se base aussi sur des images pré-construites qu’il empile les unes sur les autres au travers de <a href="https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/">OverlayFS</a>.
  Pour l’<a href="https://hub.docker.com/r/opensnp/rails-sidekiq/">image Sidekiq</a>, c’est pas moins de 39 couches empilées.</p>
  <p>Déjà, ça n’aide pas à la sécurité non plus, puisqu’une faille corrigée sur une couche réclame la reconstruction de toutes les images situées sur les couches supérieures. Ou alors vous corrigez violemment sur votre couche terminale, mais vous dupliquez alors vos bibliothèques (installées par une couche N mais masquées via overlayfs par la couche N+X) et vous avez le travail à faire sur toutes vos images utilisant la couche faillible.</p>
  <p>Ensuite, si on utilisait des images standard, on se retrouverait à consommer plusieurs giga-octets pour pas grand-chose.
  Du coup, la communauté Docker a commencé à faire la chasse aux méga-octets, et s’est prise de passion pour une distribution présentée comme très légère : <a href="https://www.alpinelinux.org/">Alpine Linux</a>.</p>
  <p>Cette distribution vient avec un gros piège… Elle est conçue à la base pour aider à débugger des kernel ! Parce que quand vous développez ce genre de logiciel, vous êtes bien content d’avoir une image de 5Mo qui démarre en 2s, vu le nombre de redémarrages que vous allez faire.
  Et comme vous ne ferez presque rien côté utilisateur, on peut y mettre des trucs très légers comme <a href="http://www.musl-libc.org/">musl</a> et <a href="https://busybox.net/">busybox</a> au lieu des mastodontes que sont la <a href="https://www.gnu.org/software/libc/">glibc</a> et les <a href="https://www.gnu.org/software/coreutils/">coreutils</a>.</p>
  <p>Sauf que musl n’est pas compatible avec la glibc disponible un peu partout.<br />Ni au niveau binaire, ce qui signifie que vous devez compiler explicitement vos logiciels avec cette libc et donc maintenir à la fois un livrable <em>-glibc</em> pour les gens hors de Alpine et un <em>-musl</em> pour Alpine.<br />Ni au niveau fonctionnalités, ce qui fait que vous pouvez rencontrer des bugs incompréhensibles et non reproductibles sur d’autres plate-formes plus standard. Ça peut aller jusqu’à l’impossibilité totale de compiler votre logiciel, comme c’est le cas actuellement avec <a href="https://github.com/gliderlabs/docker-alpine/issues/11">OpenJDK 8</a> ou <a href="https://github.com/phusion/passenger/issues/1870">Phusion Passenger</a>.</p>
  <p>Bref, vous allez vous retrouver à soit utiliser des images du Hub Docker avec une chance <a href="https://github.com/search?l=dockerfile&amp;q=&quot;FROM+alpine&quot;&amp;ref=searchresults&amp;type=Code&amp;utf8=✓">non négligeable</a> d’utiliser un conteneur Alpine dans votre chaîne et faire la chasse aux bugs vraiment chiants à comprendre, soit à devoir faire votre propre image personnelle sans Alpine…
  Le tout en croisant les doigts à chaque construction d’image pour ne pas tomber en plus sur une image contenant une faille de sécurité…</p>
  <p>Au final, Docker passe en plus complètement à côté de la plaque en termes de consommation de ressources.
  À titre d’exemple, la stack précédente RoR/Redis/Sidekiq/Nginx ramène pour 60 overlays Docker et 3.1 Go d’espace disque, quand je m’en tire pour 1.8 Go pour Cryptcheck avec une stack dev/RoR/Redis/Sidekiq/Nginx/Elasticsearch/CryptCheck/données.
  <a href="https://www.youtube.com/watch?v=tXFYxBdKiNY">Un beau gâchis d’espace</a>…</p>
  <p>Cette tendance du « je package tout dans un seul truc » est devenu à la mode et on la retrouve vraiment partout.
  Même si la complexité induite par ce type de systèmes peut être problématique, c’est vraiment le problème de la gestion de la sécurité qui est très dangereuse en pratique.
  On a déjà du mal à maintenir nos parcs plus ou moins à jour avec une infrastructure pas trop complexe, ça risque de devenir un véritable carnage une fois des outils comme Docker (mal) utilisé un peu partout…</p>
  <p><a href="https://golang.org/">Go</a>, langage d’ailleurs utilisé par Docker lui-même, compile vos projets sous forme d’un <a href="https://fr.wikipedia.org/wiki/Édition_de_liens">exécutable statique</a> qui embarque donc toutes vos bibliothèques.
  Ça a l’avantage de ne pas nécessiter leur installation, mais ça pose tous les problèmes de sécurité vus auparavant avec la recompilation nécessaire de tous vos binaires au moindre changement d’une bibliothèque.<br />Sachant en plus que la gestion des dépendances y est très mauvaise puisque se base par défaut sur les branches <code class="highlighter-rouge">master</code> de dépôts GitHub et non sur des tags, c’est une bombe à retardement dans vos systèmes.
  Par exemple vous êtes actuellement incapable de recompiler d’anciennes versions de pas mal de logiciels puisque des dépendances ont fait <a href="https://www.google.fr/search?q=undefined:+os.Unsetenv">des modifications</a> <a href="https://www.google.fr/search?q=undefined:+os.Unsetenv#q=+unknown+tls.Config+field+'GetCertificate'+in+struct+literal">non rétro-compatibles</a> avec les anciennes versions de Go et que les versions des dépendances utilisées à l’époque ne sont mentionnées nulle part.
  La situation devrait cependant s’améliorer avec l’introduction du <em>vendoring</em> depuis <a href="https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment">Go 1.5</a>.</p>
  <p><a href="http://snapcraft.io/docs/snaps/">Snap</a>, <del>la nouvelle idée à la con</del> le nouveau format de paquets d’Ubuntu/Canonical embarque aussi dans une image statique votre logiciel et toutes ses bibliothèques.
  La problématique de sécurité devient encore pire puisqu’ici, on parle d’une utilisation en tant qu’environnement de bureau.<br />Par exemple sur mon PC, je me retrouverais avec 60 versions de libssl, utilisée par postfix, openssl, bind9, gstreamer, virtualbox, isync, ntp, postgresql, nmap, tor, mumble, irssi, xca, openssh, apache2, telnet ou encore socat…
  Le jour où il faudra mettre à jour tout ça, ça va être une belle tranche de rigolade et on n’aura sûrement pas la réactivité qu’a pu avoir le projet Debian sur Heartbleed par exemple, corrigé en <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=743883">quelques heures</a> et disponible tout aussi rapidement sur l’ensemble des dépôts.
  <a href="https://kyrofa.com/posts/snapping-nextcloud-mysql#comment_39">Quand je leur ai posé la question</a>, ils ont uniquement réfléchi à la mise-à-jour d’un point de vue « binaires » et n’ont même pas pensé à la problématique des migrations de données.<br />Encore une fois, transformer les mainteneurs d’une application en mainteneurs d’un écosystème complet est loin d’être anodin ici, et le travail à abattre fera que la sécurité ne pourra plus être assurée.</p>
  <p>Utiliser des conteneurs, ça peut quand même avoir du bon.</p>
  <p>Je m’en sers beaucoup en développement pour monter un environnement stable et surtout pouvoir gérer des environnements difficiles à mixer proprement dans un même système (gcc 4.9 &amp; 6.1 par exemple) ou avec des dépendances pouvant entrer en conflit avec celles du système (l’enfer ffmpeg/avidemux/audacity…).<br />Ça permet aussi de revenir rapidement à un état propre et maîtrisé. En phase de développement on a généralement tendance à installer des tonnes de choses à la truelle et à modifier violemment son système pour faciliter le debug ou pour chercher la cause d’un problème. Mais c’est toujours intéressant de recompiler sur une machine vierge pour vérifier qu’on n’a pas oublié un bout.<br />Et enfin ça peut faciliter la reproductibilité d’un bug si l’utilisateur qui le détecte parvient à vous fournir une image démontrant le problème plutôt que de faire face à des « mais ça ne marche pas chez moi ™ ».</p>
  <p>Mais par pitié, arrêtez de vouloir mettre en production des blobs galères à gérer, en particulier en termes de sécurité, qui compliquent le moindre audit pour savoir ce qui tourne là-dedans et qui transforment les infrastructures en plat de spaghetti…</p>
  <p>Un petit conteneur <a href="https://linuxcontainers.org/lxc/">LXC</a> tout simple, construit à partir de <a href="https://wiki.debian.org/fr/Debootstrap">debootstrap</a> générant une image de maximum 200Mo toute mouillée et dans laquelle vous allez installer vos logiciels au mieux avec du apt/dpkg standard et au pire quelques scripts automatisant l’installation (du bon vieux bash over chroot ou du ansible/salt), ça fonctionne simplement et c’est <a href="https://wiki.debian.org/UnattendedUpgrades">facile à gérer</a> côté sécurité.
  Et vous découpez vos conteneurs non plus par logiciel comme sous Docker, mais par besoin : 1 conteneur pour toute la stack Discourse, 1 pour votre serveur de courriel entrant, 1 pour votre serveur DNS faisant autorité, etc.</p>


title: BlaBlaCar généralise les conteneurs et préfère Rocket à Docker
url: http://www.silicon.fr/blablacar-generalise-conteneurs-rocket-159564.html
created_at: 2017-08-10 23:19:53
updated_at: 2017-08-10 23:19:53
mimetype: text/html
language: fr-FR
domain_name: www.silicon.fr
preview_picture: http://www.silicon.fr/wp-content/uploads/2015/01/BlaBlaCar.jpg
http_status: 200
content:
  <p>Une production informatisée généralisée sur les conteneurs et le choix de Rocket plutôt que celui de Docker. Cette seule phrase suffit à résumer le caractère novateur du projet mené par BlaBlaCar sur toute la seconde partie de 2015. « <em>Quand on a lancé la société, aucune offre de Cloud sérieuse n’était disponible</em>, explique Simon Lallemand, ingénieur système au sein de BlaBlaCar. <em>On a donc pris l’habitude de gérer nous-mêmes le matériel informatique et le réseau.</em> » Une culture que la start-up entend bien conserver, même si, voici environ 18 mois, la société s’est lancée dans une rationalisation de ses infrastructures, avec aujourd’hui seulement « <em>deux modèles de serveurs achetés par racks entiers</em> ».</p>
  <p>Restait à y répartir les différents services opérés afin d’optimiser la charge de ces machines. Même si elles goûtent à VMware et KVM, les équipes de production de la start-up décident de sauter l’étape de la virtualisation : « <em>Plutôt que de construire un Cloud privé, on a fait le choix de partir directement sur les conteneurs qui sont davantage centrés sur les applications qu’on fait tourner et qui ont un impact moindre sur les performances de la machine </em>», résume Simon Lallemand.</p>
  <h2>Plus de 3 000 conteneurs</h2>
  <p>La réflexion sur le sujet est entamée dès le début de l’année 2015. Une petite équipe de 4 à 5 personnes s’empare du dossier au printemps de la même année. Contrairement à d’autres organisations, le projet vient bien ici des ingénieurs système, et non des développeurs. « <em>On savait que, de toute façon, on n’y couperait pas. Alors on a préféré prendre les devants car nous avions vu ailleurs quelques expériences malheureuses avec des codes pas très stables arrivant en production</em> », explique l’ingénieur.</p>
  <a href="http://www.silicon.fr/wp-content/uploads/2016/10/lalleman.jpg"><img class="wp-image-159503 size-medium" src="http://www.silicon.fr/wp-content/uploads/2016/10/lalleman-370x248.jpg" alt="lalleman" width="370" height="248" /></a>Silon Lallemand, lors de sa présentation au TIAD (The Incredible Automation Day), le 5 octobre.<p>Et BlaBlaCar ne chôme pas : dès septembre-octobre 2015, les premières applications sur conteneurs passent en production. « <em>On a commencé par le plus difficile – les bases de données, les composants stateful – en se disant que si nous y parvenions, le reste serait plus simple</em> », résume Simon Lallemand. Fin novembre 2015, 80 % des services de BlaBlaCar, dont tous les services critiques, fonctionnent au sein de conteneurs. Depuis, les équipes de la start-up se sont attaquées aux applications restantes, des services secondaires pour l’activité du spécialiste du covoiturage. Au total, entre 3 000 et 4 000 conteneurs sont aujourd’hui en production.</p>
  <h2>La ‘prod’ de BlaBlaCar vote Rocket</h2>
  <p>Plus surprenant encore, BlaBlaCar n’a pas fait le choix de la solution Docker aujourd’hui dominante dans le monde des conteneurs. « <em>Nous avons mené nos premiers tests sur Docker car, effectivement, c’est cette technologie qui était soutenue par la communauté la plus importante</em> », se remémore Simon Lallemand. Une déception : les équipes d’exploitation se trouvent confrontées alors à des problèmes de stabilité et à des difficultés dans la gestion du réseau. « <em>A l’époque, côté réseau, Docker était un enfer</em> », résume Nicolas Blanc, le responsable de l’équipe architecture de BlaBlaCar, croisé lors de TIAD (The Incredible Automation Day), un événement organisé par la société de service D2SI où la start-up spécialiste de covoiturage présentait son projet.</p>
  <p>Témoins de l’émergence de Rocket (renommé rkt mi-2015), la solution alternative conçue par CoreOS, les équipes de BlaBlaCar s’intéressent dès lors à ce projet. Très tôt. Les premiers tests sont menés sur la version… 0.2. « <em>Rocket s’attaquait aux limitations que nous avions identifiées avec Docker. En particulier, la solution éliminait le recours à un daemon (processus s’exécutant en arrière-plan, NDLR) et abordait le volet réseau de façon très modulaire</em>, raconte Simon Lallemand. <em>Et dès les premières versions que nous avons testées – pourtant des moutures très alpha -, nous avons pu travailler sur une technologie déjà stable</em>. » BlaBlaCar trouve également face à lui des équipes de CoreOS très attentives à ses attentes. Logique : la start-up française serait à ce jour le plus gros déploiement de la technologie rkt dans le monde. « <em>En définitive, la plus grosse difficulté que nous avons rencontrée concernait les changements des API de Rocket, qu’il nous fallait répercuter, et pas du tout la stabilité du runtime</em> », reprend l’ingénieur. Convaincues, les équipes d’exploitation de BlaBlaCar passent Rocket en production dès la version… 0.7.</p>
  <h2>Bientôt l’allocation automatique</h2>
  <p>D’abord testée sur les nouveaux serveurs que déploie la start-up – sur lesquels est installée la distribution Linux CoreOS, optimisée pour les conteneurs -, la solution est aujourd’hui généralisée. Pour Simon Lallemand, ce passage aux conteneurs génère de gros gains de productivité : « <em>Dans notre équipe, seulement deux ou trois personnes se soucient encore du matériel, plus les autres. Les gens sont devenus plus autonomes.</em> » Ajouter un service, comme un serveur Web frontal pour encaisser un pic de charge, se traduit par une modification d’un fichier de configuration. « <em>Pour créer de nouveaux services, on est passé d’un ordre de grandeur de la semaine à une durée de l’ordre de l’heure ou de quelques minutes </em>», résume l’ingénieur. Prochaine étape : mettre en place un système automatique d’allocation des ressources, via l’orchestrateur Kubernetes, dont le déploiement est attendu fin 2016 ou début 2017. Depuis sa version 1.3, Kubernetes, issu de Google, supporte Rocket. Pour l’instant, les équipes de BlaBlaCar se contentent de deux outils maison, Dgr et Ggn, dédiés respectivement à la construction et au déploiement de conteneurs. Tous deux sont disponibles sur Github (<a href="https://github.com/blablacar/dgr" target="_blank">ici</a> et <a href="https://github.com/blablacar/ggn" target="_blank">là</a>).</p>
  <a href="http://www.silicon.fr/wp-content/uploads/2016/10/lallemand2.jpg"><img class="wp-image-159504" src="http://www.silicon.fr/wp-content/uploads/2016/10/lallemand2.jpg" alt="lallemand2" width="291" height="291" /></a>Simon Lallemand : « la topologie des services change très vite. »<p>S’il fluidifie le travail des équipes d’exploitation, le passage aux conteneurs implique des changements dans l’outillage. Premier chantier : celui de la communication entre services. « <em>C’est un des points les plus délicats</em>, avertit Simon Lallemand. <em>Car, avec les conteneurs, la topologie des services change beaucoup plus vite qu’avec une infrastructure classique.</em> » Impossible donc de se contenter des mécanismes qu’utilisaient jusqu’alors BlaBlaCar, des mécanismes basés sur Chef. La start-up a donc mis en œuvre un processus de découverte de services (Service Discovery), inspiré de l’outillage développé par AirBnB et réécrit en langage Go (<a href="https://github.com/blablacar/go-synapse" target="_blank">Go-Synapse</a>, le routeur, et <a href="https://github.com/blablacar/go-nerve" target="_blank">Go-Nerve</a>, pour le suivi des statuts). « <em>C’est tout un nouveau mécanisme qu’on a dû mettre en place, sans modification de l’application BlaBlaCar</em> », précise l’ingénieur système.</p>
  <h2>Fournir les conteneurs aux développeurs</h2>
  <p>Autre chantier : le monitoring. Si la start-up a, dans un premier temps, conservé ses outils traditionnels, elle a commencé à déployer, courant 2016, Prometheus, un outil Open Source capable d’agréger toutes les métriques d’un même service. « <em>Avec le passage aux conteneurs, on a besoin d’un monitoring orienté vers les services et non plus vers les hôtes</em> », plaide Simon Lallemand.</p>
  <p>Si, chez BlaBlaCar, la révolution des conteneurs vient de la production – ce n’est pas si courant -, elle n’ignore pas pour autant les développeurs. Ces derniers bénéficient déjà, sur leurs postes de travail, des modèles de conteneurs exploités en production. « <em>Nous allons leur fournir de plus en plus d’outils pour qu’ils puissent créer directement leurs services en conteneurs</em> », résume Simon Lallemand.</p>
  <p><strong>A lire aussi :</strong></p>
  <p><a href="http://www.silicon.fr/blablacar-migre-bases-donnees-mariadb-141900.html" target="_blank">Pourquoi BlaBlaCar migre ses bases de données sous MariaDB</a></p>
  <p><a href="http://www.silicon.fr/blablacar-automatise-production-it-112053.html" target="_blank">Comment BlaBlaCar a automatisé sa production IT</a></p>
  <p><a href="http://www.silicon.fr/blablacar-plebiscite-salaries-glassdoor-122179.html" target="_blank">BlaBlacar est plébiscité par ses salariés, selon Glassdoor</a></p>


title: Busting the Myths Around Blockchains | Ripple
url: https://ripple.com/insights/busting-myths-around-blockchains/
created_at: 2017-08-10 23:19:55
updated_at: 2017-08-10 23:19:55
mimetype: text/html
language: en-US
domain_name: ripple.com
preview_picture: https://cdn.ripple.com/wp-content/uploads/2016/09/shutterstock_235007398.jpg
http_status: 200
content:
  <p><a href="https://cdn.ripple.com/wp-content/uploads/2016/09/shutterstock_235007398.jpg"><img class="alignnone wp-image-12942" src="https://cdn.ripple.com/wp-content/uploads/2016/09/shutterstock_235007398-1024x608.jpg" alt="shutterstock_235007398" width="800" height="475" /></a></p>
  <h6>We’re here to take a wrecking ball to a few prevailing myths about blockchains. Image: Shutterstock</h6>
  <p>Much has changed with blockchain technology since the pseudonymous Satoshi Nakamoto created the first Bitcoin software in 2009, and even since Ripple was founded four years later. What financial institutions once considered as the wild west for crypto-enthusiasts has now taken the world stage as blockchains are heralded as the industry’s most significant innovation. </p>
  <p>This evolution has led to misconceptions about what blockchains are and where they are headed in the future.</p>
  <p>To clarify the conversation around blockchains, we’d like to dispel a few myths.</p>
  <p><b>Myth: Blockchain technology is all hype.</b></p>
  <p><b>Reality: Blockchains are being adopted to solve real problems.</b></p>
  <p>Considering that <a href="http://prwire.com.au/pr/62010/gartner-blockchain-and-connected-home-are-almost-at-the-peak-of-the-hype-cycle">Gartner</a> recently reported that blockchain is nearing its peak of inflated expectations, it’s easy for skeptics to believe that the financial technology trend is all hype and no substance.</p>
  <p>However, growing market adoption demonstrates a true customer need: institutions are expected to invest <a href="http://www.whitecase.com/publications/insight/beyond-bitcoin-blockchain-revolution-financial-services">$400 million</a> in blockchain technology by 2019. Although blockchains come in many forms, they offer the following key benefits to financial settlement:</p>
  <ul><li>Distributed validation with a cryptographic guarantee of settlement.</li>
  <li>Increased settlement speed and reduced settlement risk.</li>
  <li>The immutability and auditability of transactions.</li>
  </ul><p>As enthusiasm for the blockchain revolution reaches its climax, banks are already working toward exploring the opportunities that blockchains hold for their businesses.</p>
  <p>The danger in this fervor, however, exists when interoperability is not at the forefront of industry innovation. Many blockchains are permission-based, creating further network fragmentation. If modern infrastructure does not support the interconnectivity of all ledgers, then the full potential of blockchain technology will never be reached.</p>
  <p><b>Myth: Blockchains are well-suited for bank-to-bank settlement.</b></p>
  <p><b>Reality: Blockchains aren’t designed for the diversity of institutional use cases on a global scale.</b></p>
  <p>Blockchains are well-suited to improve liquidity management by leveraging their native digital assets but are simply not designed to support the diversity of institutional use cases and the scale that global settlements require for these reasons:</p>
  <ul><li><b>No transaction privacy:</b> Blockchains record all transactions on a global public ledger, which could put banks in a compromising position if identified.</li>
  <li><b>Lack of scalability:</b> As a single database, public ledgers cannot scale to process trillions of transactions.</li>
  <li><b>Settlement lag:</b> Settlement using the Bitcoin blockchain takes up to sixty minutes, creating exposure and risk.</li>
  </ul><p>While many banks understand the value of using digital assets to <a href="https://ripple.com/insights/ripple-and-xrp-can-cut-banks-global-settlement-costs-up-to-60-percent/">reduce liquidity costs</a>, most prefer that this blockchain feature remains an option rather than a requirement. The inflexibility of blockchains limits their institutional applicability, preventing the technology from serving a broader range of financial use cases.</p>
  <p><b>Myth: One blockchain will eventually reign supreme, so banks should wait and see which blockchain technology to adopt.</b></p>
  <p><b>Reality: The world’s transactional needs cannot be served by one blockchain.</b></p>
  <p>Different types of ledgers exist to serve different customers and specific needs, so it’s misguided to believe that one blockchain, which is a ledger, will effectively support all the various audiences and use cases for financial institutions.</p>
  <p>Moreover, with cross-border payments alone increasing at a rate almost three times faster than the global GDP, putting the world’s transactions on one blockchain seems as reasonable as putting the entire Internet on one database.</p>
  <p>To support the world’s diverse and growing transaction needs, the ideal design for a global settlement network is much like how the Internet is designed — networks of databases communicating.</p>
  <p>By using an IP infrastructure, ledgers or networks can then specialize in features and functionality to best serve bank customers. This IP-based design is simpler and more scalable than creating a new ledger system, like a blockchain, wherein all existing ledgers must be copied to it.</p>
  <p>All ledgers should be connected with a simple, common protocol so that money can move around the world just as frictionlessly as information does online today. Enabling this Internet of Value is precisely why Ripple advocates for the Interledger Protocol (ILP).</p>
  <p><b>The Case for ILP</b></p>
  <p>The Interledger Protocol is an emerging global technical standard that provides the same benefits as other blockchain systems, including the certainty and auditability of transactions, but adds enterprise-grade features, such as unbounded scalability, complete transaction privacy and the optional use of digital assets.</p>
  <p>ILP can work with any blockchain or payment network, regardless of its underlying technology. Much like HTTP is the foundation of data communication for the World Wide Web, Interledger can act as the foundation of the Internet of Value, making sending money just as easy as sending emails.</p>
  <p>In essence, we believe that ILP is the key for a global settlement network.</p>
  <p>To learn more about the Interledger Protocol, <a href="http://interledger.org">click here</a>.</p>


title: How to build a wireless motions sensor for your Raspberry Pi @Raspberry_Pi #PiDay #RaspberryPi
created_at: 2017-08-10 23:19:58
updated_at: 2017-08-10 23:19:58
mimetype: text/html
language: en-US
domain_name: blog.adafruit.com
preview_picture: https://cdn-blog.adafruit.com/uploads/2014/02/NewImage163.png
http_status: 200
url:
  https://blog.adafruit.com/2014/02/21/how-to-build-a-wireless-motions-sensor-for-your-raspberry-pi-raspberry_pi-piday-raspberrypi/

content:
  <div class="breadcrumbs hidden">
  <div class="container">
  <h2><a href="https://blog.adafruit.com">BLOG</a></h2>
  </div>
  </div>
  <div class="container post-container">
  <div class="row post-row">
  <div class="col-md-8 post-96949 post type-post status-publish format-standard has-post-thumbnail hentry category-raspberry-pi category-sensorsparts category-wireless" id="post-96949">
  <div class="post-caption">
  <h3>February 21, 2014 AT 7:00 am</h3>
  <h2 class="storytitle">How to build a wireless motions sensor for your Raspberry Pi @Raspberry_Pi #PiDay #RaspberryPi</h2>
  </div> 
  <p><a href="http://connecteddigitalworld.com/2014/02/15/build-wireless-motion-sensor-raspberry-pi/"><img src="https://cdn-blog.adafruit.com/uploads/2014/02/NewImage163.png" alt="NewImage" title="NewImage.png" width="600" height="351" class="img-responsive" style="border: 0px;" /></a></p>
  <p>Jonathan Evans wrote this useful tutorial for <a href="http://connecteddigitalworld.com/2014/02/15/build-wireless-motion-sensor-raspberry-pi/">connected digital world.</a></p>
  <blockquote><p>In this tutorial I’ll show you how to construct a wireless motion sensor and interface it to a Raspberry Pi minicomputer. I’ve detailed two options that both work very well and have long lasting battery power consumption. The first option involves hacking a key fob remote that has four buttons. We will use the remote to send a signal every time the motion detector senses motion. The four buttons of the remote gives you the ability to run 4 wireless motion sensors.</p>
  <p>The second option uses a much more sophisticated, but low cost, RF transmitter and receiver made by Ciseco. Ciseco have developed firmware that can be loaded onto the RF modules to give them different personalities (temperature sensor, button, hall affect, relay etc…). The XRF is more expensive than the key fob but it offers much better transmission distances that will penetrate walls a lot better than the key fob remote. The XRF also has the ability to send battery levels and operate different networks that give you an almost limitless amount of sensors that can be connected to the Raspberry Pi.</p></blockquote>
  <p><a href="http://connecteddigitalworld.com/2014/02/15/build-wireless-motion-sensor-raspberry-pi/">Read more.</a></p>
  <p><a href="http://connecteddigitalworld.com/2014/02/15/build-wireless-motion-sensor-raspberry-pi/"><img src="https://cdn-blog.adafruit.com/uploads/2014/02/NewImage164.png" alt="NewImage" title="NewImage.png" width="522" height="273" class="img-responsive" style="border: 0px;" /></a></p>
  <div class="ctx-subscribe-container ctx-personalization-container ctx_default_placement ctx-clearfix"><div class="ctx-social-container ctx_default_placement ctx-clearfix"><div class="ctx-module-container ctx_default_placement ctx-clearfix">
   <hr /><div class="af-blog-adv">
  <p><a href="https://www.youtube.com/playlist?list=PLjF7R1fz_OOXWHQhEVEI5Jqf18TQRr5Hu">Check out all the Circuit Playground Episodes!</a> Our new kid’s show and <a href="http://adafru.it/subscribe">subscribe!</a></p>
  <p>Have an amazing project to share? Join the SHOW-AND-TELL every <a href="https://plus.google.com/u/0/b/112526208786662512291/112526208786662512291/posts" title="SHOW AND TELL" target="_blank">Wednesday night at 7:30pm ET on Google+ Hangouts.</a></p>
  <p>Join us every Wednesday night at 8pm ET for <a href="http://www.adafruit.com/ask" title="ASK AN ENGINEER" target="_blank">Ask an Engineer!</a></p>
  <p>Learn resistor values with <a href="https://itunes.apple.com/us/app/mhos-resistance/id785181044?mt=8" title="Mho's Resistance" target="_blank">Mho’s Resistance</a> or get <a href="https://itunes.apple.com/us/app/circuit-playground/id492487671?mt=8" title="Circuit Playground" target="_blank">the best electronics calculator for engineers “Circuit Playground”</a> – <a href="https://itunes.apple.com/us/artist/adafruit-industries/id492487674" title="Adafruit's Apps!" target="_blank">Adafruit’s Apps!</a></p>
  </div><hr /><div class="af-blog-adv"><p><a href="https://www.adafruitdaily.com/">Maker Business</a> — How Authority and Decision-Making Differ Across Cultures</p><p><a href="https://www.adafruitdaily.com/">Wearables</a> — Surface shine and prep</p><p><a href="https://www.adafruitdaily.com/">Electronics</a> — Straw Diffusers</p><p><a href="https://www.adafruitdaily.com/">Biohacking</a> — Using the VuFine+ Wearable Display to Become a Cyborg</p></div><div class="af-blog-adv">Get the only spam-free daily newsletter about wearables, running a "maker business", electronic tips and more! Subscribe at <a href="https://www.adafruitdaily.com/">AdafruitDaily.com</a> !</div>
  <hr /><div class="ctx-module-container ctx_shortcode_placement ctx-clearfix">
  <hr />
  <p>No comments yet.</p>
  <p>Sorry, the comment form is closed at this time.</p>
  </div>
  <div class="col-md-4 post-meta">
  <div class="meta">
  Filed under: <a href="https://blog.adafruit.com/category/raspberry-pi/" rel="category tag">Raspberry Pi</a>, <a href="https://blog.adafruit.com/category/sensorsparts/" rel="category tag">sensorsparts</a>, <a href="https://blog.adafruit.com/category/wireless/" rel="category tag">wireless</a> —
  by Jessica <p>Comments Off on How to build a wireless motions sensor for your Raspberry Pi @Raspberry_Pi #PiDay #RaspberryPi </p></div>
  </div>
  </div>
  </div>
  <noscript>
  <div class="pagination-arrow pagination-previous"><a href="https://blog.adafruit.com/2014/02/21/raspberry-pi-powered-bots-raspberry_pi-piday-raspberrypi/" rel="prev"><i class="fa fa-angle-left"></i></a></div>
  <div class="pagination-arrow pagination-next"><a href="https://blog.adafruit.com/2014/02/21/tool-chess-set/" rel="next"><i class="fa fa-angle-right"></i></a></div></noscript></div></div></div></div>


title: Input Masking | CSS-Tricks
url: https://css-tricks.com/input-masking/
created_at: 2017-08-10 23:20:00
updated_at: 2017-08-10 23:20:00
mimetype: text/html
language: en
domain_name: css-tricks.com
preview_picture: https://cdn.css-tricks.com/wp-content/uploads/2016/11/mask.gif
http_status: 200
content:
      <p class="author-byline">
        
                  By
          <a href="https://css-tricks.com/author/chriscoyier/">
                      
              Chris Coyier          
          </a>
                  On
          
            November 30, 2016        <a href="https://css-tricks.com/tag/a11y/" rel="tag">a11y</a>, <a href="https://css-tricks.com/tag/input-masking/" rel="tag">input masking</a>, <a href="https://css-tricks.com/tag/ux/" rel="tag">ux</a>        
        
      </p><div class="article-content">
                        
        
        <p>I don't have any UX research to cite, but anecdotally, I like it when inputs that expect data in a specific format use an <strong>input mask</strong>. I thought I'd just line up a few demos for really easy reference.</p>
  <img src="https://cdn.css-tricks.com/wp-content/uploads/2016/11/mask.gif" alt="" />
  <h3>Robin Herbots's (jQuery) Inputmask</h3>
  <p><a href="https://github.com/RobinHerbots/Inputmask">This is an actively maintained plugin.</a></p>
  <p class="codepen">See the Pen <a href="http://codepen.io/chriscoyier/pen/PbOEqL/">jQuery Input Masks</a> by Chris Coyier  (<a href="http://codepen.io/chriscoyier">@chriscoyier</a>) on <a href="http://codepen.io">CodePen</a>.</p>
  <p>It requires the jQuery dependency and the bundled plugin is 180 KB (raw), so it's fairly hefty.</p>
  <h3>Estelle Weyl's Input Masking</h3>
  <p>Estelle <a href="https://github.com/estelle/input-masking">created a vanilla JS version</a>:</p>
  <p class="codepen">See the Pen <a href="http://codepen.io/chriscoyier/pen/ZBavRJ/">Input Masks</a> by Chris Coyier  (<a href="http://codepen.io/chriscoyier">@chriscoyier</a>) on <a href="http://codepen.io">CodePen</a>.</p>
  <p>No dependencies and 5 KB (raw). There is a React component version as well.</p>
  <h3>Beware</h3>
  <blockquote class="twitter-tweet">
  <p lang="en" dir="ltr">A quick screen/audio recording of tabbing around a form using JS input masks with VoiceOver enabled. <a href="https://t.co/o0ChutJ2XN">https://t.co/o0ChutJ2XN</a></p>
  <p>— Scott Jehl (@scottjehl) <a href="https://twitter.com/scottjehl/status/317313054503211010">March 28, 2013</a></p></blockquote>
  <p>I tested the current version of Inputmask, and this is still an issue. Estelle's version doesn't do this as the mask is shown via <code>placeholder</code>, not the actual value of the input.</p>
  <h3>Filament Group's Politespace</h3>
  <p>Another <a href="https://github.com/filamentgroup/politespace">crack at an accessible version of masking</a> comes from Filament Group. </p>
  <p class="codepen">See the Pen <a href="http://codepen.io/chriscoyier/pen/ObOzeM/">Politespace</a> by Chris Coyier  (<a href="http://codepen.io/chriscoyier">@chriscoyier</a>) on <a href="http://codepen.io">CodePen</a>.</p>
  <p>It's different from the others in that you don't get you use the mask as you are editing the input, it applies formatting after you've left it. </p>
        
      </div>


title: Installing web apps on phones (for real)
url: https://joreteg.com/blog/installing-web-apps-for-real
created_at: 2017-08-10 23:20:02
updated_at: 2017-08-10 23:20:02
mimetype: text/html
language: en
domain_name: joreteg.com
http_status: 200
content:
  <hr />Last updated: 1 Dec 2016 | tagged: pwa, mobile, web, android, chrome<p>If you're building for the web, you've likely heard the term "Progressive Web App" by now.</p><p>Its definition is a bit vague and somewhat contentious perhaps, but generally the idea is that it's a web page that has "taken all the right vitamins" so it can behave more like an app you installed from the app store. It starts as a normal tab in your browser and if it has all the <em>right stuff</em> the browser will prompt visitors if they'd like to "Add to Homescreen", which up to this point has largely equated to being a glorified bookmark.</p><p>Once opened from the Homescreen it hides the browser UI and appears as its own app when switching apps, etc.</p><p><strong>This is huge!</strong></p><p>If you're not familiar with the challenges currently facing native app developers, here are a few:</p><ol><li>It's nearly impossible to break into the small group of 5 - 8 apps users use regularly.</li><li>On average, in the US, smartphone owners install <a href="http://qz.com/253618/most-smartphone-users-download-zero-apps-per-month/"><em>zero</em> new apps per month</a>.</li><li>You have no control over the approval process.</li><li>Paid apps obviously require paying fees to the app store.</li></ol><p>PWAs are an attempt to hit the sweet spot balance between web and native. Fast, native-app-like experiences without the list of problems above. The simple act of opening one of these <em>Webpages With Special Sauce™</em> means you've already downloaded "the app" so there's no need for an App Store run-around where you ask users to download a 50mb binary over their flaky mobile connections just so they can read your web content. Arguably, it's really more like "saving experiences" you're currently having in your browser for easy access later, even if you're offline. This ability for a web page to graduate to becoming a full app-like experience appears, by all measures and case studies that have come out so far to be pure magic juice for businesses who care about you know, those silly little useless metrics like conversions or retention or whatnot. My professional interpretation of the data I've seen on this so far is: OMG.</p><p>But don't take my word for it, <a href="https://youtu.be/eI3B6x0fw9s?t=14m3s">watch this instead</a>.</p><p>PWAs are the entire reason I switched to Android after being on iOS for 7-some years. The <a href="https://joreteg.com/blog/why-i-switched-to-android">blog post I wrote about why I switched</a> apparently struck a nerve. It had 80k+ uniques in the first two weeks and led to several response posts by folks like <a href="https://adactio.com/journal/10410">Jeremy Keith</a>, et. al. I'm not saying this to brag (well, maybe just a little), but the point is, this stuff is a big friggin' deal and we have good reason to be excited about it!</p><p>For me personally, building web apps that feel like native apps is something I've been trying to do for ages. Exhibit A: this screen cap <a href="https://vimeo.com/8254856">I made about something I built for this</a> <em>seven</em> years ago!</p><p>Let's just say the concept has just always made a lot of sense to me. However...</p><a name="the-implementations-have-historically-fallen-short" class="anchor" href="https://joreteg.com/blog/installing-web-apps-for-real#the-implementations-have-historically-fallen-short" id="the-implementations-have-historically-fallen-short">The implementations have historically fallen short</a><p>Of course, building app-like websites are hardly a new idea. This is how Steve Jobs originally proposed folks build for the iPhone! But as I explained in more detail in my post. I don't believe the incentives made sense for platforms to do this kind of thing at the time.</p><p>I'll spare you repeating my other post, you can <a href="https://joreteg.com/blog/why-i-switched-to-android">just go read it if that sounds interesting</a>.</p><a name="the-time-has-come" class="anchor" href="https://joreteg.com/blog/installing-web-apps-for-real#the-time-has-come" id="the-time-has-come">The time has come</a><p>PWAs have already come <em>a long</em> way even without the changes I'm discussing here. They're already solidly in the "very likely a much better investment than building native apps" category for most companies considering building mobile apps.</p><p>But, as awesome as PWAs are on Android, they still have had a few annoyances that keep them from being truly "first class citizens" on the platform.</p><p>For example:</p><ol><li>PWAs added to Homescreen are not listed in the "all apps" panel on Android. This also means it isn't possible to have PWAs that are "installed" but not on a Homescreen somewhere like you can with other Android apps.</li><li>Once it's added to your Homescreen, the app developer can't later decide to change the name or the icon. Changing the published <a href="https://developer.mozilla.org/en-US/docs/Web/Manifest">Web App Manifest</a> has no impact on previous "Add to Homescreen"-ed apps.</li><li>When you migrate from one Android phone to another, as I recently did when I got a Pixel, my "installed" PWAs lost their full-screen-ness and didn't run in standalone mode until I removed and re-added them.</li><li>Even though App Store discovery has proven very challenging certainly users have been trained to look for apps in app stores. So if you tell your users you built an app the reaction is probably going to be to go search for it in the app store. But, as of right now, they're not listed in app stores (though <a href="https://blogs.windows.com/msedgedev/2016/07/08/the-progress-of-web-apps/#clsLLk1RvqTVvAyK.97">Microsoft is changing that</a>).</li><li>The phrasing of "Add to Homescreen" is pretty weak. It doesn't imply that you're doing more than adding a bookmark. For better or worse we've taught users that apps are to be "installed". As a developer of PWAs, I can tell you that the phrase "Add to Homescreen" feels like it's selling what I built short. I know users won't quite get this, whereas they certainly know what "installing an app" means.</li><li>An installed native app can "own" a "url scope", which means its possible to create a link that opens an installed application. This may not seem like a big deal, but say you're building a password-less login system where a magic login link is emailed to you that logs you into the application. As it stands right now, there's no way for me to tell the OS to open that link in the installed PWA, instead they'll always open in the default browser.</li><li>Web push notifications as implemented right now, come from Chrome, not the PWA. This also means that if you block notifications, it's kind of all or nothing. Because the OS is blocking <em>Chrome</em> from notifying users.</li><li>Stuff like storage usage and battery consumption get bundled in with Chrome rather than being presented as isolated apps to the user.</li></ol><p>So here's the interesting thing about all these problems: <strong>most, if not all of them, stem from the fact that "installed PWAs" are <em>not truly apps</em> in the eyes of the operating system!</strong></p><a name="that39s-what39s-changing" class="anchor" href="https://joreteg.com/blog/installing-web-apps-for-real#that39s-what39s-changing" id="that39s-what39s-changing">That's what's changing!</a><p>Soon "installed PWAs" will actually be APKs, as in <em>real</em> Android apps will be <strong>created and side-loaded</strong> into the OS by Chrome. This means they won't be websites that have been "Add to Homescreen"-ed they'll be <strong>actually generated and installed</strong> on the fly, and with the language to match.</p><p>It was rather subtly announced by <a href="https://twitter.com/paul_kinlan">Paul Kinlan</a>, please watch his talk from Chrome Dev Summit the specific part I'm referring to <a href="https://youtu.be/YJwrBbze_Ec?t=21m33s">starts right here</a>. It seemed like not a lot of people even caught it. Personally, I erupted into spontaneous applause (in fact, you can hear me on the video right <a href="https://youtu.be/YJwrBbze_Ec?t=24m25s">here</a>), but I think, given that this was the last talk of the conference and a lot of folks were kinda zoning out I felt like perhaps the implications of this were a bit lost.</p><p>So, let me be clear: I think this is a <strong>BIG FRIGGIN' DEAL</strong> for the web and worth being very excited about!</p><a name="what-happens" class="anchor" href="https://joreteg.com/blog/installing-web-apps-for-real#what-happens" id="what-happens">What happens?</a><p>Beware, I'm told the way it's implemented as I will describe below is not its final state. Apparently the whole "side-loading" bit will be smoothed over and be more seamless. There's a <a href="https://bugs.chromium.org/p/chromium/issues/list?can=2&amp;q=component%3AMobile">list of bugs</a> related to all of this stuff, you're curious.</p><p>It's not landed yet, it's currently behind a flag in Canary so details are sure to change, but you can try it today if you've got an up-to-date copy of Android and Chrome Canary installed.</p><a name="1-turn-on-the-flag" class="anchor" href="https://joreteg.com/blog/installing-web-apps-for-real#1-turn-on-the-flag">1. Turn on the flag</a><p>Go to <code>chrome://flags</code> and enable "Improved add to Home screen":</p><p><img src="https://joreteg.com/images/1/chrome-flag.png" width="270" height="480" alt="image" /></p><a name="2-open-a-sweet-pwa-and-install-it" class="anchor" href="https://joreteg.com/blog/installing-web-apps-for-real#2-open-a-sweet-pwa-and-install-it">2. Open a sweet PWA and install it...</a><p>I'd recommend something like <a href="https://paperplanes.world/">paperplanes.world</a> and select "add to Homescreen" from the menu. You'll see this:</p><p><img src="https://joreteg.com/images/1/prompt.png" width="270" height="480" alt="image" /></p><p><strong>Note:</strong> The language is all about "installing this application" because you're actually side-loading an Android APK here!</p><p>Next you get this "app installed" language:</p><p><img src="https://joreteg.com/images/1/installed.png" width="270" height="480" alt="image" /></p><p>And now, you've actually got an APK installed for this web app!</p><p>Which means it's also listed in my "all apps" view:</p><p><img src="https://joreteg.com/images/1/all-apps.png" width="270" height="480" alt="image" /></p><p>You can even view app info like other apps:</p><p><img src="https://joreteg.com/images/1/app-info.png" width="270" height="480" alt="image" /></p><a name="3-why-does-any-of-this-matter" class="anchor" href="https://joreteg.com/blog/installing-web-apps-for-real#3-why-does-any-of-this-matter">3. Why does any of this matter?</a><p>A platform with over 1 billion monthly active mobile is working to make web apps first class citizens on that platform in every way they can. This is a <em>huge</em> deal. But, to be clear this isn't about Google and Chrome, sure the term Progressive Web App may have been <a href="https://infrequently.org/2015/06/progressive-apps-escaping-tabs-without-losing-our-soul/">coined</a> and promoted by Googlers. But, don't confuse this with just being a Google Thing™.</p><p>I'm pretty sure the folks at Microsoft, Mozilla, and Opera would take that personally. Friggin' Steve Jobs promoted the basic idea. Microsoft has <a href="https://blogs.windows.com/msedgedev/2016/07/08/the-progress-of-web-apps/#clsLLk1RvqTVvAyK.97">committed to listing PWAs in their store</a>! Which is actually more than Google has committed to doing though, if I were a betting man, I'd say this little APK tweak is a step they're taking in that direction. But regardless, the point is this isn't about Google, it's a Web Thing™.</p><p>Right now the Google folks are definitely among the group pushing the envelope with this concept the hardest. But none of this would get me excited if it were not based on extending the reach of standards-based web applications onto their host systems.</p><p>As someone who's been working in this space for a long time I'm super excited to see metrics support and economic forces all finally seem to be aligning behind this idea, its time seems to finally have come.</p><p>It's a good time to go all in on the web. I can't wait to see what the next few years bring. Personally, I feel like the web is well poised to replace the majority of apps we now get from app stores.</p><p>Oh, and for those of you worried about Apple, don't be. They don't like to be the laggard and there are positive signs that they're coming around to the idea. I think the best way to make that happen is to building amazing web experiences that make choosing to invest in building native apps for App Stores seem arduous and unnecessary. Apple certainly don't want Safari to be <a href="https://nolanlawson.com/2015/06/30/safari-is-the-new-ie/">the new IE</a>, so watch them closely and help speak out where you can.</p><p>If you want to (nicely) disagree with me or if you just think there may be something to this whole Web Thing™ I'm <a href="https://twitter.com/henrikjoreteg">@HenrikJoreteg</a> on twitter you can hit me up there.</p><hr /><h4>Thanks for reading, you may want to:</h4>


title: Design Better Forms – uxdesign.cc
url: https://uxdesign.cc/design-better-forms-96fadca0f49c?gi=841f526e597c
created_at: 2017-08-10 23:20:04
updated_at: 2017-08-10 23:20:04
mimetype: text/html
domain_name: uxdesign.cc
preview_picture: https://cdn-images-1.medium.com/max/1200/1*VvQeOFsY57NJxtZmKyRnHA.jpeg
http_status: 200
content:
  <div class="section-content"><div class="section-inner sectionLayout--fullWidth"><div class="aspectRatioPlaceholder is-locked"><img class="graf-image" src="https://cdn-images-1.medium.com/max/2000/1*VvQeOFsY57NJxtZmKyRnHA.jpeg" alt="image" /></div></div><div class="section-inner sectionLayout--insetColumn"><p id="bcc1" class="graf graf--p graf-after--figure graf--trailing">Whether it is a signup flow, a multi-view stepper, or a monotonous data entry interface, forms are one of the most important components of digital product design. This article focuses on the common dos and don’ts of form design. Keep in mind that these are general guideline and there are exceptions to every rule.</p></div></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Forms should be one column</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Top align labels</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Group labels with their inputs</h3></div><div class="section-content"><p></p><h3 id="b3ee" class="graf graf--h3 graf--leading">Avoid all caps</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Show all selection options if under 6</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Resist using placeholder text as labels</h3></div><div class="section-content"><p></p><h3 id="f278" class="graf graf--h3 graf--leading">Place checkboxes (and radios) underneath each other for scannability</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Make CTAs descriptive</h3></div><div class="section-content"><p></p><h3 id="bc56" class="graf graf--h3 graf--leading">Specify errors inline</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Use inline validation after the user fills out the field (unless it helps them while in the process)</h3></div><div class="section-content"><p></p><h3 id="cb9b" class="graf graf--h3 graf--leading">Don’t hide basic helper text</h3></div><div class="section-content"><p></p><h3 id="aad9" class="graf graf--h3 graf--leading">Differentiate primary from secondary actions</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Use field length as an affordance</h3></div><div class="section-content"><p></p><h3 class="graf graf--h3 graf--leading">Ditch the * and denote optional fields</h3></div><div class="section-content"><div class="section-inner sectionLayout--insetColumn"><h3 class="graf graf--h3 graf--leading">Group related information</h3><div class="aspectRatioPlaceholder is-locked"><img class="graf-image" src="https://cdn-images-1.medium.com/max/1600/1*nmeMIuW7csU9uVTB9BIBTg.jpeg" alt="image" /></div>Users think in batches, and long forms can feel overwhelming. By creating logical groups the user will make sense of the form much faster.</div></div><div class="section-content"><div class="section-inner sectionLayout--insetColumn"><h3 class="graf graf--h3 graf--leading">Why ask?</h3><p id="d822" class="graf graf--p graf-after--h3">Omit optional fields and think of other ways to collect data. Always ask yourself if the question can be inferred, postponed, or completely excluded.</p><p class="graf graf--p graf-after--p graf--trailing">Data entry is increasingly automated. For example, mobile and wearable devices collect large amounts of data without the user’s conscious awareness. Think of ways you can leverage social, conversational UI, SMS, email, voice, OCR, location, fingerprint, biometric, etc.</p></div></div><div class="section-content"><div class="section-inner sectionLayout--insetColumn"><h3 class="graf graf--h3 graf--leading">Make it fun</h3><p class="graf graf--p graf-after--h3 graf--trailing">Life is short. No one wants to fill out a form. Be conversational. Be funny. Gradually engage. Do the unexpected. It is the role of the designer to express their company’s brand to elicit an emotional reaction. If done correctly, it will increase completion rates. Just make sure you don’t violate the rules listed above.</p></div></div><div class="section-content"><div class="section-inner sectionLayout--insetColumn"><p id="dad6" class="graf graf--p graf--leading">This article is part of a larger initiative to build a user interface pattern library focused on the crossroads of usability and aesthetics. If this sounds interesting, <a href="https://ohapollo.com/resources.html" class="markup--anchor markup--p-anchor" rel="noopener" target="_blank">Subscribe to receive updates</a>.</p><blockquote class="graf graf--pullquote graf-after--p"><div>If you found this article of value please click the ♡ below so others can find this post.</div></blockquote><h3 id="e5c0" class="graf graf--h3 graf-after--pullquote graf--trailing"><a href="https://andrewcoyle.com/" class="markup--anchor markup--h3-anchor" rel="noopener" target="_blank">Find me</a> on <a href="https://twitter.com/CoyleAndrew" class="markup--anchor markup--h3-anchor" rel="nofollow noopener noopener noopener noopener" target="_blank">Twitter</a>, and <a href="https://dribbble.com/AndrewCoyleDesign" class="markup--anchor markup--h3-anchor" rel="nofollow noopener noopener noopener noopener" target="_blank">Dribbble</a>.</h3></div></div>


title: Carb loading: what is new? | Jeukendrup - Trusted sports nutrition advice & exercise science news
url: http://www.mysportscience.com/single-post/2016/05/12/Carb-loading-what-is-new?_escaped_fragment_=
created_at: 2017-08-10 23:20:06
updated_at: 2017-08-10 23:20:06
mimetype: text/html
language: en
domain_name: www.mysportscience.com
preview_picture: http://static.wixstatic.com/media/a04440_f44a5d292c784a9f98714e42b510bc37.png
http_status: 200
content:
  <img width="649" title="Carbohydrate loading" src="http://static.wixstatic.com/media/a04440_f44a5d292c784a9f98714e42b510bc37.png/v1/fill/w_649,h_484/a04440_f44a5d292c784a9f98714e42b510bc37.png" height="484" alt="Carbohydrate loading" /><p>
                      
                          Carb loading has been a popular practice for athletes in the days leading up to an event. There is a a lot of discussion about the best ways to do this. Starting with a full tank is generally believed to be a good idea. But early protocols to achieve high glycogen concentrations in the muscle were extreme and had a lot of side effects. So the methods of carb loading have evolved! Here we will discuss the current thinking and I will give my personal interpretation and practical recommendation.
                      
                  </p>
                  
                  <p>
                      
                          How it all began
                      
                  </p>
                  <p>
                      
                          In the 1960s the biopsy needle was redeveloped and this allowed researchers to collect a small amount of muscle tissue and measure muscle glycogen (1), the storage form of carbohydrate in the muscle. A number of discoveries were made:
                      
                  </p>
                  <ul class="font_8"><li>
                          <p>
                              Glycogen concentration in the muscle is dependent on diet. The more carbohydrate in the diet the higher the glycogen stores
                          </p>
                      </li>
                      <li>
                          <p>
                              Glycogen concentration declines during exercise, especially higher intensity exercise
                          </p>
                      </li>
                      <li>
                          <p>
                              Higher glycogen concentrations in the muscle resulted in less fatigue and better performance
                          </p>
                      </li>
                  </ul><p>
                      These findings have only been confirmed since then.
                  </p>
                  
                  <p>
                      
                          Early observations
                      
                  </p>
                  <p>
                      
                          It was also observed that if you deplete muscle glycogen first, then reduce carbohydrate intake for 3 days followed by a very high carbohydrate intake for 3 days, that muscle glycogen bounced back much more than just eating carbohydrate every day. Supercompensation referred to the observation that glycogen did not just return to normal after depletion, it actually “supercompensated” and achieved “supra-normal” concentrations. This observation resulted in the development of the classical supercompensation diet which was then successfully used by runners like European Marathon Champion Ron Hill in the 1970s.
                      
                  </p>
                  
                  <p>
                      
                          Side effects 
                      
                  </p>
                  <p>
                      
                          This protocol involved an extremely hard workout 7 days before the race, followed by carbohydrate restriction for 3 days. It may not be ideal to have such a hard workout 7 days before. Without carbohydrate recovery in the days after is likely to be very poor. Athletes were also recommended not to exercise the week before the race. For many athletes this is a greater punishment than the extreme diet itself. The high fat, no carb diet in the 3 days after the glycogen depleting exercise also caused a lot of gastro-intestinal problems in many runners. So, overall, although this protocol was highly effective, the side effects may have outweighed the potential benefits (2).
                      
                  </p>
                  
                  <p>
                      
                          Moderation
                      
                  </p>
                  <p>
                      
                          Therefore, a more moderate approach was proposed in the 1980s. The glycogen depleting exercise was removed and as training was reduced towards the race, the carbohydrate intake was gradually increased. Glycogen concentrations appeared to be very high as well after 6-7 days, even though they were not quite as with the traditional protocol.
                      
                  </p>
                  
                  <p>
                      
                          2 days may be enough
                      
                  </p>
                  <p>
                      
                          Then studies in the 1990s demonstrated that very well trained athletes could achieve similar muscle glycogen concentrations with just 1 or 2 days of carbohydrate loading and reduced training on those days. In less trained individuals this appeared to take a little longer.
                      
                  </p>
                  <div>
                      <img width="649" title="Muscle glycogenolysis is dependent on glycogen stores" src="http://static.wixstatic.com/media/a04440_9159572e8ee64ae9b45b5d15eebc93af.png/v1/fill/w_649,h_402/a04440_9159572e8ee64ae9b45b5d15eebc93af.png" height="402.2353760445682" alt="Muscle glycogenolysis is dependent on glycogen stores" /></div>
                  
                  <p>
                      
                          Studies also demonstrated that the rate of glycogen breakdown during exercise was directly proportional to the amount of glycogen present in the muscle. In other words, if you have extremely high muscle glycogen stores you will break them down faster than when you have normal or high glycogen stores. An hour or 2 hours into the exercise, glycogen concentrations may therefore be comparable whether you started with extremely high or just high glycogen stores.
                      
                  </p>
                  
                  <p>
                      
                          Practical guidelines
                      
                  </p>
                  <p>
                      So what we can take away from these studies are the following practical guidelines:
                  </p>
                  
                  <ul class="font_8"><li>
                          <p>
                              Make sure you start exercise with sufficient muscle glycogen
                          </p>
                      </li>
                      <li>
                          <p>
                              Sufficient means high stores but these stores don’t need to be extremely high. Extremely high glycogen stores are also broken down faster
                          </p>
                      </li>
                      <li>
                          <p>
                              For trained individuals this can be achieved by eating carbohydrate rich for 2 days prior to a race, whilst reducing glycogen use (reducing training)
                          </p>
                      </li>
                      <li>
                          <p>
                              
                                  Because training is reduced and therefore energy expenditure is reduced, a higher carbohydrate intake should not be the result of just eating more. It should be the result of emphasising carbohydrate sources and reducing fat intake. Very often carb loading and overeating seem to be confused by athletes
                              
                          </p>
                      </li>
                      <li>
                          <p>
                              
                                  There are many different ways to achieve high glycogen stores. The type of carbohydrate seems to have little or no effect, both solid and liquid carbohydrate sources seem to have the same effects
                              
                          </p>
                      </li>
                      <li>
                          <p>
                              A carbohydrate intake of 5-7 g/kg per day seems to be sufficient in the majority of cases (with low energy expenditure)
                          </p>
                      </li>
                      <li>
                          <p>
                              
                                  Athletes who often experience gastro-intestinal issues should select their carbohydrate sources more carefully and could benefit from a lower fiber intake.
                                  
                                      <a target="_blank" href="http://bit.ly/GIprob03" shape="rect">http://bit.ly/GIprob03</a>
                                  
                              
                          </p>
                      </li>
                  </ul>
                  <p>
                      
                          References
                      
                  </p>
                  <p>
                      Bergstrom, J., Hermansen, L., Hultman, E., and Saltin, B. Diet, muscle glycogen and physical performance. Acta Physiol Scand. 71: 140-150, 1967.
                  </p>
                  <p>
                      Jeukendrup and Gleeson Sports Nutrition Human Kinetoics Champaign IL, 2010
                  </p>
                  
                  
                  <p>
                      
                          Read more on
                      
                  </p>
                  <p>
                      
                          Prevention of gastrointestinal problems
                          
                              <a target="_blank" href="http://bit.ly/GIprob03" shape="rect">http://bit.ly/GIprob03</a>
                          
                      
                  </p>
                  <p>
                      
                          Causes of gastrointestinal problems
                          
                              <a target="_blank" href="http://bit.ly/GIprob02" shape="rect">http://bit.ly/GIprob02</a>
                          
                      
                  </p>
                  <p>
                      
                          Gastro-intestinal symptoms in athletes
                          
                              <a target="_blank" href="http://bit.ly/1O3nolY" shape="rect">http://bit.ly/1O3nolY</a>
                          
                      
                  </p>


