THIS is how you do posterity

I just posted this to Facebook, but they deliberately design their service as a memory hole, so….

Facebook is designed as a memory hole. Its search tools are crippled and external search engines are not allowed a look in. The manner in which posts and comments are displayed is predicated on their ephemerality. We -they- are deliberately repudiating posterity.

It galls me to no end that I have to bitch about this on Facebook in order for it to be seen. Nobody will even remember I said it….

End of an age….

In an online discussion concerning an ‘old-school’ geek’s fear of change, there was a lot of back and forth concerning luddism on the one hand and change for change’s sake on the other. At one point, someone chastised the poster for his trepidation.

“I’m going to guess he’s going to look back on his life and realize that he was dumb to think he’d seen it all at age 24. He talks as though the Third Age of Middle Earth is ending…”

In some important ways, it is. The process isn’t complete, but there is a fundamental change happening, and it will discomfit some of us.

The days of ‘Homesteading the Noosphere‘ (as ESR put it), are coming to a close. Scale, network topologies, business models and legal encroachment on the principles of individual online freedom are all conspiring to make the technological world we live in substantially more constrained than it’s been since the internet became part of our lives.

The land rush is over, the cowboys are gone (either buried or rich) and the homesteaders are being bought out by the speculators and tycoons. Community-based governance is under siege by national and international interests.

And this is being reflected in the tech world. The craftsman’s approach to software (always greater in repute than in reality) is decidedly more difficult to practice as a trade than it was. Toolkits are giving way to frameworks and apps replace applications. Backyard-mechanic roadsters and dirt-track races are swallowed up by Nascar – VCs get us excited by the prospect of building only big enough to sell out to someone bigger.

The physical networks themselves are being taken back by the telcos and proffered to governments for surveillance in exchange for ever more egregious rent-seeking behaviour. What we used to call sharing is now piracy. The word ‘copyright’ now means ‘don’t copy at all, ever.’

And in the midst of it all, we’re grateful to lockin-vendors who make Free software difficult, if not impossible, to use. We rent what we used to own. Even our identities are no longer our own.

I grieve to say it, but unless there’s a sudden and immense resurgence of the DIY spirit, especially in peer networking and distributed data, we’re going to fall back into the bad old days of the dumb terminal and the smart network. And that network’s smarts will not exist for our benefit.

I’m pushing 50 now, and do I fear change? Not really. I just regret the lost freedom, the creative anarchy of the ’90s, the ability to hack something cool and new, the chance to achieve things never before possible. It’s not gone yet. We could still turn things around. But every day we don’t brings us a day closer to the day when we can’t any longer.


WordPress galleries – slideshow hack

WordPress 3.5 has a very nice photo new gallery UI, and because I’m the kind of guy who just can’t be happy with nice, I had to go and hack it to make it run a slide show by default.

If you’re a user, you already have the slideshow feature – and I’ll bet money that it’s a damn sight more refined than mine – but for those of us running our own sites, here’s all you need to create simple, straighforward slideshows using any size of image:
Continue reading

Pinch Me…

Steve Wildstrom over at tech.pinions argues that the furore over the Apple v Samsung case has been driven in part by reductivist and simply incorrect reporting. He singles out the Washington Post, which published an article asserting that one of the patents at stake was Apple’s so-called Pinch to Zoom behaviour:

There’s one serious problem with the first sentence, which was repeated dozens of times in stories in print and on the Web. Apple only has a limited patent (US 7,812,826) on the pinch to shrink, stretch to zoom gesture that is a core element of touch interfaces. And the ’826 patent wasn’t in dispute in the Samsung case because Apple never asserted it.

That’s true enough. I too have grown a little tired of people pointing to 19th Century school slates as prior art. The whole ‘rectangle with rounded corners’ thing is a gross oversimplification of a much more subtle and complex claim. Apple isn’t contending that nobody else gets to make phones and tablets with rounded corners; they’re saying that the overall shape and dimensions of the devices in question is one of many factors that distinguishes their products from others – or should, anyway.

You can’t take a single claim from a patent filing and argue that its lack of originality negates the entire patent. That’s not how these things work. Quite the opposite, in fact. You have to take all of the claims together, and then decide whether they look like a unique creation. Patent claims use painfully obtuse language – often deliberately so. They attempt to perform the almost impossible task of using words to so clearly describing a physical object (or process, but more on that in a bit) that there’s no ambiguity when the questions of copying or originality come up in court.

The kind of patent that Apple’s been most intransigent about are what are called ‘trade dress’ patents. These describe the distinctive features of the device that Apple considers uniquely theirs; they’re used to protect the company from knock-offs and copycats. And copycatting is what Samsung is being accused of.

But when comparing a Samsung device to, say, an iPhone, they’re not just looking at the rectangular shape with rounded corners. They’re looking at that, plus the dimensions, the black front, the ratio of the screen to the border, the placement and colour of the icons on the screen… all of it at once.

Thanks to Steve Jobs, Apple as a company is notoriously painstaking and obsessive about the design and detail of every single thing they make. This hugely intensive creative process is one of the reasons there’s so little variety in Apple’s offerings. They are far more concerned with making one thing perfectly than making a bunch of things well. Apple’s corporate culture is more artisanal in philosophy than any other manufacturer I’ve ever seen. Taste doesn’t just matter; it’s the essence of what they do.

Contrast that with Google. They’re engineers all the way down. The thing that genuinely leaves me breathless when I think about them is their ability to simplify the stupefyingly massive scale of the World Wide Web and filter the entire thing through a single text box. Ask anyone who wrangles data for a living: this really is an astounding accomplishment, and Google are (so far) at the head of their class.

They do have taste, though, and occasionally they’re capable of creating things that are genuinely pleasing. The latest versions of the Android operating system are good examples. But they’re not particularly original. There’s not really anything in that operating system that provides any particular sense of novelty. It’s telling that their biggest effort in recent iterations of the system is something they call ‘Project Butter‘. It’s a system-wide effort to make everything smooth as… well, you know, butter.

If the rumour mill is to be believed – and I think it’s credible in this case – it was precisely this lack of inventiveness that sent Steve Jobs around the bend. He felt robbed because, after showing the world how to make an operating system, Google used it as inspiration for their own. To add insult to injury, Samsung used the OS to bootstrap their own efforts to make tablets and phones that so closely aped Apple’s iconic creations that customers were failing to distinguish between the two offerings. And this is what caused him to launch an all-out war to stop Samsung and to sully Android in the process.

It used to be that companies used their patent portfolios for profit rather than prohibition. IBM is famous for having a small army of lawyers who would appear in tech boardrooms armed with sheaves of patents and make corporate officers an offer they couldn’t refuse: Accept a cross-licensing agreement that allowed the company to continue its work, but also left IBM clear to take advantage of any truly interesting patents the company had in hand. Those companies whose portfolios were lacking in depth were typically required to pay a tithe to continue doing business. It added cost to the products, but it didn’t stop the companies from selling them.

Apple chose to think differently. As they made abundantly clear in their numerous suits brought against against Samsung across the globe, Apple had no interest whatsoever in licensing their ‘trade dress’ patents. They simply wanted Samsung to stop. They were deploying their patents as weapons, not as levers.

The resulting acrimony has so far had decidedly mixed results. The US suit, whose price tag has left most observers goggling, has largely been decided in their favour. In the UK, a judge acerbically observed that whether Samsung had ‘slavishly copied’ Apple was immaterial, because Samsung’s products were easily distinguishable from Apples. They simply weren’t as cool.

If you ask me, he was right. And the reason he was right is what prompted me to write this column.

Wildstrom is right in stating that most tech writers have lost the plot where this story is concerned. But in dithering over the details he too has missed something essential: Patents are designed to give inventors a time-limited monopoly over a particular creation as an incentive for them to share their inner workings. These in turn would inspire others to riff on these inventions, but the law would stop others from claiming the invention as their own.

In the physical world, this works pretty well. When an auto maker invents a new internal combustion engine, other manufacturers can lift the hood and use their understanding of these innovations to drive their own engineering efforts. But they can’t just throw together an identical assembly and call it their own. Take Mazda’s famous ‘wankel’ engine, for example. Patent law doesn’t prohibit me from making a rotary engine of my own; it just stops me from making the same one that Mazda makes.

But this principle doesn’t translate well into the world of software. One of the patents at issue was the behaviour of the user interface when the user scrolls to the end of a long list. Apple made the list ‘bounce’ a bit at the end. It’s a behaviour known as over-scrolling, and it’s a very pleasing, intuitive way, to let someone know they can’t scroll any further. It’s an elegant, more innately human, interaction that lets users know that the system knows what they’re trying to do, rather than simply becoming unresponsive (which is how Microsoft, for example, behaved).

But to assert, as Apple has, that nobody else is allowed to use this behaviour without their permission… well, that’s not what patents were designed for. If we consider how they were originally envisioned, almost the opposite is true. The patent system exists to encourage others to create their own, potentially better way to do it.

There’s a lot of validity to Apple’s desire to discourage copycats. But in their suits against Samsung and others they’ve lost sight of the fact that imitation is an integral part of invention. Edison didn’t invent the first filament lightbulb, he just invented the one with the best materials. Bell riffed on telegraphy and existing voice transmission technology to make voice over the wire possible. Then Marconi came along and effectively developed telegraphs – and later, telephones – without cables. The progression was so clear that for decades radio was known simply as The Wireless.

Yes, Samsung did do point-by-point comparisons between Apple’s interface and their own. But then they went ahead and, taking iOS as their inspiration, created their own. Apple’s objection to this ‘slavish’ copying is arguably an indictment of Samsung’s creativity, but their courtroom tactics are analogous to Alexander Bell taking Marconi to court for using diaphragms in his speakers and microphones.

Samsung’s devices may behave similarly to Apple’s. After all, a smart phone is a smart phone is a smart phone. But just because Apple made the first commercially successful one, or even because others have indulged in the sincerest form of flattery, that doesn’t give Apple any moral right over the idea. If Samsung were trying to pass off their products as iPhones, or if they had stolen trade secrets, source code or materials that were uniquely Apple’s, then the Company That Steve Build might have had a case.

But that’s not the issue here. Samsung’s products are indisputably their own. And yes, yer honour, they’re not nearly as cool.

The cuckoo’s egg

[Originally published with a slightly longer title in the Vanuatu Daily Post]

It’s fairly natural in business to want a return on one’s investment. That’s pretty much the point of capitalism, after all. But investment implies risk, too; if you put your money into play, you are accepting at least a small likelihood that it will be lost. Normally, the riskier the venture, the higher the expected return if things go right.

In some endeavours, however, the risk is unavoidable and the reward is slow in coming – if it comes at all. Telecommunications infrastructure is one such area. Especially here in the Pacific, capitalisation can require investment levels that –rightly– make the average investor blanch. It’s no accident, therefore, that private sector players often seek institutional backing before embarking on large-scale projects.

The plain fact is that in this global marketplace, some sort of inducement is needed in order to make investment in the comparatively tiny Pacific market attractive. Pretend for a moment you’re an investor. Given the choice between backing a fibre-optic cable landing in Port Vila or one that lands in Jakarta, which would you choose?  All other things being equal, you’d be a fool to choose the former.

Competitors have cried foul in the past when concessionary financing was used to induce increased private sector participation in various Pacific telecoms markets, but experience shows that healthy competition has raised revenue levels across the board. Competition, even with its attendant risks, is better for all concerned.

The challenge, then, is how to make the capitalisation phase, with its necessary reduction in risk, lead seamlessly into a competition phase, with a level of risk characteristic of a healthy marketplace?

Continue reading

Dr Strangepost, or How I Learned to Accept the Facebook API and Love Social Media

I’ve been spending a good part of the last week or so building social media tools for my employer. We’ve come to the conclusion that services like Facebook provide some of the most immediate and detailed sources of information on important events in the Pacific. I’ll go into this in more detail in an upcoming discussion paper, but for the moment, suffice it to say that Facebook’s ubiquity and its accessibility via mobile devices makes it one of the most compelling means of transmitting information quickly and easily to large numbers of people.

We track a number of public groups, and they’ve provided us with invaluable information. Probably the most vivid example of this is PNG’s Sharp Talk group, which has more closely tracked a number of events in PNG politics than even the most resourceful of the traditional media.

Our work is predicated on the idea that dialogue leads to better understanding. So, naturally, we want to understand how people think and feel about events in the world around them. We also want to be able to record these voices for posterity, and to use the most compelling amongst them to educate others.

So with that in mind, I set out to use Facebook’s programming interfaces to allow staff to flag posts and comments, and to track and store important threads in a local database. This way, we could have a record of commentary as it happened, and we’d be able to perform key word searches, statistical analysis and a few other kinds of research that would (we hope) lead to a better understanding of current issues and how they were perceived.

I struggled. I struggled a lot. It’s kind of an unfamiliar feeling, after 20 or so years in the data-mangling game. There were a number of all-too-familiar issues: Documented approaches that were no longer supported, new approaches that required undocumented steps. These are, for better or for worse, the stock in trade of a lot of web services. Things move quickly, so stability is more of an aspirational goal than anything else.

But there were a few things that, taken together, brought me to a complete stop.

First, it’s nearly impossible to obtain access to the site without an interactive login. I get why Facebook do this – they want to make it hard for spyware and other malicious programs to fake the approval process. And that’s okay, as far as it goes, but it assumes that apps using their programming interfaces are interactive – that is, that someone is sitting in front of the thing, driving it. But that’s precisely the opposite of what I’m trying to do. I want our staff to be able to flag something as interesting and NOT have to sit around reading the bloody page all day long.

Second, the thing just doesn’t work half the time. About 50% of the queries I post to Facebook’s graph service either time out or return an empty data set. Still, with a bit of persistence, I suppose I could manage. Just program the software to retry the query until it gets useful results. This has a few nasty implications – for example: how will I know when I’m supposed to get an empty result set? But I’ve dealt with worse in my time.

The third problem is much bigger. Facebook’s results work the same way Facebook own interface works. In other words, rather than getting the complete set of comments on a given post, they give you the most recent few, and a ‘next’ link to get a few more. Again, they’re assuming that you’re using the data interactively.

I spent a while pondering how to make my proposed service work, puzzling out scenarios and trying to find computer logic that would cope with the shortcomings. And then I had my eureka moment. Facebook doesn’t want people to write apps like mine.

The more I dug into other people’s coding projects and programming interfaces to Facebook’s service, the more I came to realise that it’s actually possible to reliably push things onto Facebook, but it’s quite hard to get them back out. Posting photos, links, likes and comments is pretty straightforward, but trying to make sense of them once they’re on the site is another task entirely. If Facebook were to make it easy to perform automated analysis on their content, they’d be undermining the stickiness of their own service. By allowing it to become commoditised, they’d face losing their identity. In short, they’d become a platform, not a brand.

For Facebook, the idea of someone seeing Facebook data wrapped in someone else’s interface is worst thing in the world.

This made me very sad.

Easily interchangeable data is the very essence of wealth in the knowledge economy. It’s taken me a while to realise it, but some people don’t understand that sharing that wealth creates more for everyone. As far as information is concerned, apparently, most of us have yet to see the merits of free trade. We’re still back in a world of monopolies, hoarding and guilds.

But that’s okay. This will change. The Internet is a porous medium. Generally speaking, information will leak out, and the commodification of social data will eventually happen.

In the mean time, I’ll get to work on building a different set of tools to make sure that, in our little patch of the world at least, the leaking happens sooner than later.

Selling Democracy (Slight Return)

Writing about the influence of the Internet on pro-democracy movements earlier this year, I observed:

As individual control over the flow of information rises, central control wanes. And this, obviously, is the crux of the dilemma facing businesses and governments across North Africa and throughout the world. They are belatedly coming to realise that they are fighting a many-headed hydra. As they cut off one avenue of communication, another rears its head.

But that hydra has a body, and the body is the network itself.

The US Congress’ latest attack on the integrity of the Internet demonstrates that at least some American businesses have heard this message loud and clear.

Their intent now is simply to cordon off what they consider to be the American part of the Internet, and to beat into submission any site that refuses to play by American rules. The rules, needless to say, are expressly designed to impose an economy of hoarding and scarcity on a technological landscape premised on bounty and sharing.

To what end? What do so-called content-producers stand to gain from all of this? Not much, really. Except that they can continue using the same business model that has stood them in such good stead for the last few decades.

Essentially, the SOPA legislative package doesn’t create a Great Firewall of America; it encases the giants of the media industry in amber.

The starkest evidence of the fossilising effect of this legislation was provided by a recent House Judiciary Committee hearing, which consisted of little more than a gaudy carousel of facile pronouncements:

How low was the level of debate? The hearing actually descended to statements like “the First Amendment does not protect stealing goods off trucks” (courtesy of the AFL-CIO’s Paul Almeida).

Criticisms were brushed aside with the blithe assertion that if these rules don’t fit the Internet as it is today, then it’s the Internet that should change, not the rules.

The one detractor allowed a place on the stage was Google, which objected strenuously to to the vague, often blatantly prejudicial language of the Bill.

It was up to Google alone to make the argument that SOPA’s definition of “rogue sites” is poor, that its remedies are extreme, and that plenty of legitimate sites could be targeted. One has only to think of YouTube, which even without SOPA is being sued by Viacom for $1 billion and would certainly have been hammered years ago under SOPA’s crazy language (sites can be dismantled under SOPA if they take “deliberate actions to avoid confirming a high probability of the use of the US-directed site to carry out acts” of infringement. What does that even mean? And how does it fit with existing robust safe harbors for user-uploaded content sites?)

Perhaps the most shocking aspect of this hearing was that Google, the sole opponent to the legislation allowed to present at the hearing, was castigated by most of the people present, impugned for purportedly profiting from piracy and cast as the villain in this whole affair.

Seeing one of the few growing and dynamic drivers of the information economy not only cast out of the fold but actively opposed, one can only conclude that the captains of the US media industry are perfectly content to cut their nose off to spite their face. They will burn the bridge represented by Google rather than cross it.

I see two immediate dangers if this regime is actually allowed to take the shape proposed for it:

  1. Innovation in content re-use and sharing will move outside of the US. Some will move into the shadows (kind of like offshore pirate radio in days of yore, except the ships and radios are available for the cost of a laptop). Some will move into the less governed – or governable – areas.
  2. US influence on innovation and invention will decline significantly. This legislative package will serve as a clear signal that Silicon Valley is no longer the influence it used to be. (Indeed, the Valley’s lack of standing in DC was evidenced by committee members’ contempt for Google throughout the hearing.)

The latter outcome is the more dangerous of the two. Losing influence in the direction the Internet’s development takes also means losing the uniquely American ethos of freedom and individualism.

There are numerous new media and technological players poised in the wings right now. But few of them (with the possible exception of Al Jazeera) have any moral stake in human rights or even individual expression.

Warnings that SOPA’s passage will mark the death knell of the Internet as we know it are, therefore, not exaggerations.

Like a bug in amber, the US media and content industries may be preserved for a while, but the life that they breathed into world culture – the American ethos of individual freedom – that will diminish and die.

When I talk about the ‘American ethos of individual freedom’, I’m using ‘American’ to modify ‘ethos’ not ‘individual freedom’. Tons of cultures have extremely strong traditions of freedom and individual rights. Few have that particularly American flavour of it. And it’s that particular flavour which, not coincidentally, fits so well with the Internet today.

Mystery & Wonder

plongeeAccording to Andrew Sullivan, Alexis Madrigal claims that flocking behaviour is “… a beautiful phenomenon to behold. And neither biologists nor anyone else can yet explain how starlings seem to process information and act on it so quickly.”

That second sentence is just false, as even a quick visit to wikipedia is sufficient to discover: Current research shows that this vastly complex behaviour requires no interaction between all points, and no orchestration by some unseen hand.

Flocking behaviour can be simulated in computers by creating groups of simple bots, each of which responds independently to three simple rules:

1) Separation – avoid crowding neighbors (short range repulsion)

2) Alignment – steer towards average heading of neighbors

3) Cohesion – steer towards average position of neighbors (long range attraction)

Some researchers have even gone so far as to create real, flying drones that exhibit this behaviour.

The miracle is not that this grand ballet is so complex, but that it’s so damn simple in its essence.

Look, I marvel just as much as the next person when watching vast flocks of starlings. And there are few things more graceful and poignant than an entire school of sardines arcing over the waves in consecutive leaps as they flee from predators. There is little in life so exhilarating as being engulfed in a pocket of azure space as a school of reef fish flow soundlessly around you.

These are all examples of of simple creatures following simple rules, collectively iterating and permuting in patterns whose complexity the human mind finds attractive, even enthralling. Because it cannot follow the linear progression of individual acts in such a vastly parallel pattern, the brain hits the overload switch, which results in our sense of wonder.

It is, almost literally, mind candy. But that does NOT make it a mystery.

I’m not asking that we put aside our wonder, but can we please accept that many of these so-called mysteries are NOT mysterious. (Well, not any longer, anyway.) I’m as big a fan of exaltation as the next person, but I cringe when we allow it to curb our perceptions and our ability to learn.

Remembering Steve Jobs

Okay, look: Gallows humour aside (for the moment), Steve Jobs doesn’t deserve our reverence. He deserves our respect, yes, for being one of the only people in the industry to actually think about how people used hardware. He was a great hardware designer in part because of his obsession with detail and his absolute inability to compromise on a principle.

I admire him for that. And I’m more than a little disgusted to hear about Jobs’ ‘visionary’ genius from the likes of Ballmer and Gates – who, not to put too fine a point on it, wouldn’t know a good design if it slapped them in the face with a dead salmon.

Who the fuck are they to judge? And who the fuck are we to listen?

No, the thing we need to admire about Jobs – the thing we need to LEARN about Steve Jobs – is how he thought, how he never stopped trying to make things simpler, how he utterly refused to compromise, how he refused to accept ‘improvement’ as the criterion for success. It was necessary, of course, and relentlessly pursued, but it was the means to another end….

And that was good design. Something the technological world knows far too little about. And with his passing, most of its collective knowledge and ability pass with him.

If you really want to show respect and admiration for Steve Jobs, understand him.

Emulate him. Let them call you arrogant and impolite if they must, but be a perfectionist. Be unforgiving, cruel even, to yourself and others. But be simple and clear, too. If you do that, then one day you might – just might – do one perfect thing.

Find Duplicate File Names in CouchDB

I was stumped for a bit, trying to figure out how to help my editorial staff avoid uploading the same file twice. In a repository spanning tens of thousands of titles in over a hundred different collections, our staff can’t easily tell whether a document is already in a collection or not.

Turns out that finding duplicate attachments is fairly easy. First create the view:

function(doc) {
  if (doc._attachments){
    for (var i in doc._attachments){
      emit([doc.collection, i], doc._id);

Which returns JSON output that looks like this:

[“collection name”, “filename.rtf”]

So all I have to do to find the duplicates is query that view using the composite key and see if it returns any rows:

http://my.couchdb.server:5984/database-name/_design/my-listings/_view/attachment-exists?key=[“collection name”,”filename.rtf”]

I could do the same with MD5 checksums, too, but I won’t. The problem is that even a single character change is enough to make two documents different. So if someone opens their copy of a file and Word changes the metadata in it, it’s no longer byte-for-byte identical, even though the text has not changed. This means that the number of false negatives (i.e. duplicate files that are NOT found) would be too high for people to rely on.

What I’d really like to find is an algorithm that determines whether the textual content of two documents is significantly similar….