The arrest of Vanuatu PM Sato Kilman’s personal secretary at Sydney airport and the subsequent expulsion of all AFP staff from Vanuatu should offer players on both sides of the diplomatic fence the opportunity to reflect and, with luck, learn a little from each other. But that’s probably going to take a while.

Tempers in Vanuatu remain quite heated, and it appears that Australia has yet to figure out how to climb down from the limb it’s on. Indeed, some suggest that Australia has yet to realise it’s on a limb at all.

Obviously, no one involved thought that the diversion of PM Kilman’s party into Australian territory in order to arrest Clarence Marae would have significant repercussions. They wouldn’t have done it if they had. But judging from Foreign Affairs Minister Bob Carr’s statements so far, his government has yet to realise just how important deference and respect are in Melanesia.

It’s tempting to conflate the PM’s employment of a convicted criminal with this remarkable breach of diplomatic protocol. Some online commenters were quick to say that if the PM kept better company he wouldn’t have found himself in this situation in the first place. That may provide a handy little sound bite, but it does nothing to add to our understanding of the issue.

Clarence Marae’s alleged involvement in criminal tax evasion is a legal matter. The public shaming of a head of government is entirely political. Not to put too fine a point on it, one would hope that those in charge of international relations would be able to draw a distinction between the two.

Australia protests constantly that it treats its Pacific neighbours as peers, but its actions don’t seem to bear this out. It has misread events and conditions on the ground in Fiji, PNG and now Vanuatu, and in each case made itself look a great deal clumsier than it should be.

Nearly all of these mistakes can be explained by an implicit assumption that its role as benefactor somehow makes it smarter or better than its island peers. Neither of these is always true, but more importantly, neither of these matters. Equality and respect don’t stem from equal capability or even equal intelligence or skill. In sports, I don’t respect only those teammates who are stronger or better than me.

The same should be true in diplomacy. Would Australian police have used the same tactics to arrest a member of an American, Chinese or Russian delegation, or for that matter, even a Burmese or Albanian diplomatic group? So what, then, could inspire Foreign Affairs to remain silent while just such a plan was hatched for the Vanuatu delegation? It’s unfortunate to have to say this, but it’s hard to imagine any other reason than a misplaced and parochial sense of superiority.

Worse still, Bob Carr’s first reaction was effectively to say, ‘Nice aid programme you have there. Be a shame if something were to happen to it.’ This statement can only lead us to wonder whether AUSAid’s constant protestations that its programme is non-political are entirely true. There should be more to this relationship than aid.

There was a time when this kind of chastisement might have worked, but it’s behind us now. There are any number of countries who would quite happily step into any gap left by a withdrawal of Australian project and budget support. In strategic terms, Australia has as much to lose as Vanuatu if their relationship breaks down.

Still, Bob Carr did leave himself an opening. He admitted that he had not been briefed on the tactics used by the AFP to draw Clarence Marae into Australian jurisdiction. In an interview, he seemed to spontaneously suggest that a distinction could be made between the legality of Marae’s arrest and the treatment of the Prime Minister. He should pursue this further; it might just be the way out.

Kilman has made no effort whatsoever to defend Marae. So if Australia can find its way to see past the arrest, it can perhaps find a way to reconcile with Vanuatu by expressing regret over the PM’s treatment.

Nobody on the Vanuatu side is in a hurry to mend fences. We’re only months away from a general election, and while people here are not at all comfortable with seeing their head of government associating with convicted criminals, there is still a significant portion of society whose sense of national pride has been pricked by these events. Respect and deference are not abstract notions in Melanesia. It may be hard to grasp this if all you know is the law, but Melanesian kastom is predicated on peace first, justice second. In a nutshell, living on an island sometimes requires that we tolerate behaviour that we consider beyond the pale. Surrounded as we are by ocean, we don’t have a pale to go beyond.

It all comes down to this: Australia should take this opportunity to learn a little humility and respect and to try its hand at peace-making, Melanesian-style. If it does find a way to re-engage, it’s just possible that its law enforcement personnel might get the chance to help us – not punish, help – to be a little wiser in choosing the company we keep.

We track a number of public groups, and they’ve provided us with invaluable information. Probably the most vivid example of this is PNG’s Sharp Talk group, which has more closely tracked a number of events in PNG politics than even the most resourceful of the traditional media.

Our work is predicated on the idea that dialogue leads to better understanding. So, naturally, we want to understand how people think and feel about events in the world around them. We also want to be able to record these voices for posterity, and to use the most compelling amongst them to educate others.

So with that in mind, I set out to use Facebook’s programming interfaces to allow staff to flag posts and comments, and to track and store important threads in a local database. This way, we could have a record of commentary as it happened, and we’d be able to perform key word searches, statistical analysis and a few other kinds of research that would (we hope) lead to a better understanding of current issues and how they were perceived.

I struggled. I struggled a lot. It’s kind of an unfamiliar feeling, after 20 or so years in the data-mangling game. There were a number of all-too-familiar issues: Documented approaches that were no longer supported, new approaches that required undocumented steps. These are, for better or for worse, the stock in trade of a lot of web services. Things move quickly, so stability is more of an aspirational goal than anything else.

But there were a few things that, taken together, brought me to a complete stop.

First, it’s nearly impossible to obtain access to the site without an interactive login. I get why Facebook do this – they want to make it hard for spyware and other malicious programs to fake the approval process. And that’s okay, as far as it goes, but it assumes that apps using their programming interfaces are interactive – that is, that someone is sitting in front of the thing, driving it. But that’s precisely the opposite of what I’m trying to do. I want our staff to be able to flag something as interesting and NOT have to sit around reading the bloody page all day long.

Second, the thing just doesn’t work half the time. About 50% of the queries I post to Facebook’s graph service either time out or return an empty data set. Still, with a bit of persistence, I suppose I could manage. Just program the software to retry the query until it gets useful results. This has a few nasty implications – for example: how will I know when I’m supposed to get an empty result set? But I’ve dealt with worse in my time.

The third problem is much bigger. Facebook’s results work the same way Facebook own interface works. In other words, rather than getting the complete set of comments on a given post, they give you the most recent few, and a ‘next’ link to get a few more. Again, they’re assuming that you’re using the data interactively.

I spent a while pondering how to make my proposed service work, puzzling out scenarios and trying to find computer logic that would cope with the shortcomings. And then I had my eureka moment. Facebook doesn’t want people to write apps like mine.

The more I dug into other people’s coding projects and programming interfaces to Facebook’s service, the more I came to realise that it’s actually possible to reliably push things onto Facebook, but it’s quite hard to get them back out. Posting photos, links, likes and comments is pretty straightforward, but trying to make sense of them once they’re on the site is another task entirely. If Facebook were to make it easy to perform automated analysis on their content, they’d be undermining the stickiness of their own service. By allowing it to become commoditised, they’d face losing their identity. In short, they’d become a platform, not a brand.

For Facebook, the idea of someone seeing Facebook data wrapped in someone else’s interface is worst thing in the world.

This made me very sad.

Easily interchangeable data is the very essence of wealth in the knowledge economy. It’s taken me a while to realise it, but some people don’t understand that sharing that wealth creates more for everyone. As far as information is concerned, apparently, most of us have yet to see the merits of free trade. We’re still back in a world of monopolies, hoarding and guilds.

But that’s okay. This will change. The Internet is a porous medium. Generally speaking, information will leak out, and the commodification of social data will eventually happen.

In the mean time, I’ll get to work on building a different set of tools to make sure that, in our little patch of the world at least, the leaking happens sooner than later.

As individual control over the flow of information rises, central control wanes. And this, obviously, is the crux of the dilemma facing businesses and governments across North Africa and throughout the world. They are belatedly coming to realise that they are fighting a many-headed hydra. As they cut off one avenue of communication, another rears its head.

But that hydra has a body, and the body is the network itself.

The US Congress’ latest attack on the integrity of the Internet demonstrates that at least some American businesses have heard this message loud and clear.

Their intent now is simply to cordon off what they consider to be the American part of the Internet, and to beat into submission any site that refuses to play by American rules. The rules, needless to say, are expressly designed to impose an economy of hoarding and scarcity on a technological landscape premised on bounty and sharing.

To what end? What do so-called content-producers stand to gain from all of this? Not much, really. Except that they can continue using the same business model that has stood them in such good stead for the last few decades.

Essentially, the SOPA legislative package doesn’t create a Great Firewall of America; it encases the giants of the media industry in amber.

The starkest evidence of the fossilising effect of this legislation was provided by a recent House Judiciary Committee hearing, which consisted of little more than a gaudy carousel of facile pronouncements:

How low was the level of debate? The hearing actually descended to statements like “the First Amendment does not protect stealing goods off trucks” (courtesy of the AFL-CIO’s Paul Almeida).

Criticisms were brushed aside with the blithe assertion that if these rules don’t fit the Internet as it is today, then it’s the Internet that should change, not the rules.

The one detractor allowed a place on the stage was Google, which objected strenuously to to the vague, often blatantly prejudicial language of the Bill.

It was up to Google alone to make the argument that SOPA’s definition of “rogue sites” is poor, that its remedies are extreme, and that plenty of legitimate sites could be targeted. One has only to think of YouTube, which even without SOPA is being sued by Viacom for $1 billion and would certainly have been hammered years ago under SOPA’s crazy language (sites can be dismantled under SOPA if they take “deliberate actions to avoid confirming a high probability of the use of the US-directed site to carry out acts” of infringement. What does that even mean? And how does it fit with existing robust safe harbors for user-uploaded content sites?)

Perhaps the most shocking aspect of this hearing was that Google, the sole opponent to the legislation allowed to present at the hearing, was castigated by most of the people present, impugned for purportedly profiting from piracy and cast as the villain in this whole affair.

Seeing one of the few growing and dynamic drivers of the information economy not only cast out of the fold but actively opposed, one can only conclude that the captains of the US media industry are perfectly content to cut their nose off to spite their face. They will burn the bridge represented by Google rather than cross it.

I see two immediate dangers if this regime is actually allowed to take the shape proposed for it:

1. Innovation in content re-use and sharing will move outside of the US. Some will move into the shadows (kind of like offshore pirate radio in days of yore, except the ships and radios are available for the cost of a laptop). Some will move into the less governed – or governable – areas.
2. US influence on innovation and invention will decline significantly. This legislative package will serve as a clear signal that Silicon Valley is no longer the influence it used to be. (Indeed, the Valley’s lack of standing in DC was evidenced by committee members’ contempt for Google throughout the hearing.)

The latter outcome is the more dangerous of the two. Losing influence in the direction the Internet’s development takes also means losing the uniquely American ethos of freedom and individualism.

There are numerous new media and technological players poised in the wings right now. But few of them (with the possible exception of Al Jazeera) have any moral stake in human rights or even individual expression.

Warnings that SOPA’s passage will mark the death knell of the Internet as we know it are, therefore, not exaggerations.

Like a bug in amber, the US media and content industries may be preserved for a while, but the life that they breathed into world culture – the American ethos of individual freedom – that will diminish and die.

UPDATE:
When I talk about the ‘American ethos of individual freedom’, I’m using ‘American’ to modify ‘ethos’ not ‘individual freedom’. Tons of cultures have extremely strong traditions of freedom and individual rights. Few have that particularly American flavour of it. And it’s that particular flavour which, not coincidentally, fits so well with the Internet today.

That second sentence is just false, as even a quick visit to wikipedia is sufficient to discover: Current research shows that this vastly complex behaviour requires no interaction between all points, and no orchestration by some unseen hand.

Flocking behaviour can be simulated in computers by creating groups of simple bots, each of which responds independently to three simple rules:

1) Separation – avoid crowding neighbors (short range repulsion)

2) Alignment – steer towards average heading of neighbors

3) Cohesion – steer towards average position of neighbors (long range attraction)

Some researchers have even gone so far as to create real, flying drones that exhibit this behaviour.

The miracle is not that this grand ballet is so complex, but that it’s so damn simple in its essence.

Look, I marvel just as much as the next person when watching vast flocks of starlings. And there are few things more graceful and poignant than an entire school of sardines arcing over the waves in consecutive leaps as they flee from predators. There is little in life so exhilarating as being engulfed in a pocket of azure space as a school of reef fish flow soundlessly around you.

These are all examples of of simple creatures following simple rules, collectively iterating and permuting in patterns whose complexity the human mind finds attractive, even enthralling. Because it cannot follow the linear progression of individual acts in such a vastly parallel pattern, the brain hits the overload switch, which results in our sense of wonder.

It is, almost literally, mind candy. But that does NOT make it a mystery.

I’m not asking that we put aside our wonder, but can we please accept that many of these so-called mysteries are NOT mysterious. (Well, not any longer, anyway.) I’m as big a fan of exaltation as the next person, but I cringe when we allow it to curb our perceptions and our ability to learn.

I admire him for that. And I’m more than a little disgusted to hear about Jobs’ ‘visionary’ genius from the likes of Ballmer and Gates – who, not to put too fine a point on it, wouldn’t know a good design if it slapped them in the face with a dead salmon.

Who the fuck are they to judge? And who the fuck are we to listen?

No, the thing we need to admire about Jobs – the thing we need to LEARN about Steve Jobs – is how he thought, how he never stopped trying to make things simpler, how he utterly refused to compromise, how he refused to accept ‘improvement’ as the criterion for success. It was necessary, of course, and relentlessly pursued, but it was the means to another end….

And that was good design. Something the technological world knows far too little about. And with his passing, most of its collective knowledge and ability pass with him.

If you really want to show respect and admiration for Steve Jobs, understand him.

Emulate him. Let them call you arrogant and impolite if they must, but be a perfectionist. Be unforgiving, cruel even, to yourself and others. But be simple and clear, too. If you do that, then one day you might – just might – do one perfect thing.

Turns out that finding duplicate attachments is fairly easy. First create the view:

function(doc) {
  if (doc._attachments){
    for (var i in doc._attachments){
      emit([doc.collection, i], doc._id);
    }
  }
}

Which returns JSON output that looks like this:

["collection name", "filename.rtf"]

So all I have to do to find the duplicates is query that view using the composite key and see if it returns any rows:

http://my.couchdb.server:5984/database-name/_design/my-listings/_view/attachment-exists?key=["collection name","filename.rtf"]

I could do the same with MD5 checksums, too, but I won’t. The problem is that even a single character change is enough to make two documents different. So if someone opens their copy of a file and Word changes the metadata in it, it’s no longer byte-for-byte identical, even though the text has not changed. This means that the number of false negatives (i.e. duplicate files that are NOT found) would be too high for people to rely on.

What I’d really like to find is an algorithm that determines whether the textual content of two documents is significantly similar….

There seems to be a nearly universal preference for narrative over fact in most (if not all) of the US debate over economic policy. People invest the issue with their own biases (a common propensity) then construct or defend the most closely aligned story.

In short, people have been led to believe that the whole situation:

a) makes sense;
b) can be simply expressed; and
c) has a straightforward solution, if only the rest of the world can be made to see it.

This explains not only the refusal even to grant that a debt policy must of necessity consider revenue generation AND reduced spending, but also the tendency to draw the Hayek/Keynes/Friedman debate as a zero-sum argument.

Government, at the best of times, is more a clusterfuck than anything else. It requires a level of opportunism and ideological/ethical/moral compromise that few of us can stomach. Tragically, it breeds people who can stomach it far too easily.

Human society requires narrative in order to make sense of this otherwise senseless situation. (We can’t all be Sartre or Clauswitz – and really, who wants to be?) But its desire for narrative has been cynically abused so consistently and for so long by propaganda that the possibility for civic (not to say civil) discourse has been reduced nearly to zero.

The increasingly (irretrievably?) fictional rhetoric driven by the various camps within the anarchic village that is Washington has made mutual understanding (and therefore compromise) impossible. We can, in other words, no longer talk usefully amongst ourselves.

To help fight spam and prevent fake profiles, use the name your friends, family or co-workers usually call you. For example, if your full legal name is Charles Jones Jr. but you normally use Chuck Jones or Junior Jones, either of those would be acceptable.

Audrey Watters at ReadWriteWeb got a little further clarification from a Google spokesperson concerning Google Profiles and the use of real names:

“We are not requiring people to use their ‘real name’, but rather they need their Google profile to include the name they commonly go by in daily life. I know that sounds like the same thing, but there are some differences. For a hypothetical example, Samuel Clemens could choose to be known as ‘Mark Twain,’ although we wouldn’t allow him to go by Authordude88. And for a real life example, 50 Cent is using Google+, after we verified that this is the name he is commonly referred to. More details can be found here.

That page goes on to say that your name should use your first and last names, avoid ‘unusual’ characters (more about this below) and that your profile should represent only one person.

There are numerous problems with this policy which, taken together, make it impossible to implement it consistently or, indeed, objectively. Arguably, this policy would have disallowed some or all of the following:

Jesus Christ

‘Christ’ is an title, not an actual name

Buddha

It’s really a title, and it’s only one word

Pol Pot, Lenin & Stalin

All noms de guerre, associated with illegal and subversive activities at some point in history.

The Apostle Paul

He was ‘really’ Saul

Socrates

What, no last name?

Ellery Queen

‘He’ is actually a ‘they‘.

Acton, Currer and Ellis Bell

The Bronte sisters, who hid their identities (and location) to avoid scandal in their community

George Eliot and George Sand

Just a couple of the most notable women who could only be taken seriously after assuming a male identity

I could go on at great length, but suffice it to say that there are problems. You’ll note, by the way, that many of the names listed above refer to individuals who were guilty of subversive and often illegal activities. In many cases, too, there was a point in time where these names were not commonly known, or were disputed (even proscribed) by large segments of society, or by the powers that be.

Let me try to make these apparently silly examples clearer. It’s easy, with the benefit of hindsight to say, “Dude, that’s JESUS. Everybody knows he’s the Christ.” Well, that may be true now, but what about when he was some misfit wandering from town to town, pissing off a lot of Pharisees in the process? And yes, knowing what we know now, maybe we wouldn’t want to give a voice to Pol Pot, Lenin or Stalin. But how would we have felt about them in the early years of the 20th Century?

My question is: Are we on the side of the Pharisees, the Tsars and the Cambodian despots? Because that’s who we’re helping here, metaphorically speaking.

I’m not advocating taking a particular side. I’m suggesting exactly the opposite – not taking sides. That’s why I deliberately included some decidedly contentious figures in the list. (I could just as easily have included the authors of the Federalist Papers.) I just want to know that there’s room in our society for gadflies like Socrates, that it’s okay for some as-yet-unknown literary genius to speak freely and loud.

(And that, yes, even the soon-to-be villains can be captured in the public dialogue. There’s actually an argument to be made for listening to nuts like bin Laden and Breivik, in order that we better understand – and engage – our enemy.)

There are technical problems with any set of rules applying to names. As Patrick McKenzie eloquently demonstrates, just about any rule you think might apply to names actually doesn’t. Furthermore, the rationale that disallowing pseudonyms would have any effect whatsoever on spam and/or civility in public discourse, let alone that it will ‘help people know who they’re talking to,’ is entirely unproven.

But the issue is bigger than just technical. Skud writes that disallowing pseudonymity can be discriminatory and downright dangerous. The fact that her argument isn’t comprehensive makes it all the more compelling.

Throughout history, and for countless reasons, the use of pseudonyms and the appropriation of unofficial names are common, reputable and widely accepted practices,

One of the most common responses to these (and other) objections can be stated succinctly enough: Google’s Service – Google’s Rules. Fair enough, but let’s consider the implications of this. If we as a society allow ourselves to be utterly circumscribed by corporate policies over which we have no control (and which, as here, are pretty much arbitrary in nature), we’re in effect voting ourselves back into feudalism, where the rule of law becomes meaningless – or rather, indistinguishable from fiat.

I know some of you are writhing in your chairs right now, waiting to shout, “Oh come on, Crumb! Lighten up. This is a bloody social network we’re talking about, not some proletarian revolutionary struggle.” Well, no. This is a social network, and if it wants to reflect society then it needs to bloody well reflect it. In many parts of the world, just hanging out with your buddies on a service like this can get you into a lot of trouble.

Identity matters, for political, economical, social and philosophical reasons. The ability to define one’s identity freely is a fundamental human right. Google’s aim is to reduce bad behaviour, and that’s laudable. But if they want to do it right, they should focus on behaviour, not practices that are only tangentially linked to the problem.

If Google really wants their network to reflect society rather than deform it, they need to back off the name issue and look at fostering a culture of respect and civility instead.

Police Commissioner Joshua Bong’s call for improved government intelligence was roundly supported by all sectors of Vanuatu Society. The announcement, made at the closing of a recent security conference, met with enthusiastic responses from everyone this writer interviewed.

A survey of 100 people asking the question ‘Do you support intelligence in government?’ resulted in a 97% response for the ‘yes’ side. Two respondents, both MPs, had not finished reading the question when the poll closed. The third, a prominent minister, replied that he has campaigned for intelligence and that he supported the idea of intelligence in principle, but he could not condone its use in government at this time, as it might undermine the balance of power.

There were a few mixed responses. The reaction of one group of youths was difficult to gauge, as their sustained laughter made it impossible for them to speak. A chief from Kivimani village on the island of Futua Lava seemed to call for part-time intelligence, observing, “Ol minista oli waes finis, be waes ia i kasem olgeta long aftanun nomo.”

Approached for comment, a police spokesman said, “That’s not the kind of intelligence we meant. We meant analysis and data gathering and…. Oh. Right. Yeah, I think I see what you mean. Yes, I think intelligence in government would be a great idea.”

More on this breaking story as it appears. Assuming more intelligence actually does appear.

If you must recommend Ubuntu Linux to others, recommend nothing later than 10.04, the last LTS release.

10.10 saw a number of minor but irritating bugs creep in that show a significant shortage of testing and forethought. There were countless small things like context menus no longer working after returning from a suspended state or new window positioning that’s completely counter-intuitive. Some of them, like changing sides for window buttons or listing indecipherable package descriptions above package names in Update Manager, were deliberate (and conceivably, in some universe, necessary), but most of the changes were clearly mistakes. When these are combined with long-standing bugs (like Network Manager arbitrarily deciding to disable the Save button) and inconsistencies, they begin to weigh against Ubuntu’s many virtues.

In 11.04, Unity, combined with an increase in the number of stupid bugs (that spiffy state-of-the-machine motd message is FUBAR’ed now on console login) clearly indicates that Ubuntu is more interested in new and shiny than they are in quality. A quick scan of Launchpad (itself a new product designed to simplify bug maintenance and supplant the competition, but which has done neither) shows that there are, on average, 100 open bugs per project.

Ubuntu is slipping out of control. Canonical have stopped listening and – more importantly – working with the community. The number of defects is growing, but Canonical’s response is to make it harder for mere mortals to submit bugs. They seem to think that strong guidance is needed for their product to grow in new and interesting ways. Fair enough, but they’re confusing leadership with control. They’re simply imposing their views because they don’t value the discussion. They’re treating criticism as opposition and shutting themselves off from valid feedback.

Worse, they simply don’t have the number of skilled developers they need to achieve their goals. When I look at the bug queues on some packages, I shudder in sympathy with the poor souls who are expected to wrangle them. Canonical is clearly embarked on an impossible task, but nobody’s either got the guts or the vision to spell this out to Shuttleworth and co.

Getting buy-in and active participation from the community is a pain in the arse at the best of times, but the alternative is far worse. Heaven knows that the GNOME dev camp are… special, to be nice. But it’s clear that, given the choice between getting a partial but workable success through compromise or taking their ball and going home, Canonical has consistently chosen the latter.

This cannot end well. It will, however, end sooner than later.