New Adventures in Software by Dan Dyer

Some time ago I wrote about my issues with Google and its anti-privacy agenda. It was my intention to avoid using Google services where possible because I don’t like Larry Page’s vision and Google’s (and Facebook’s and others’) tracking of individuals’ activity across the web. I have very little to hide but that’s not the point.

Two of the biggest enablers of Google’s surveillance machinery are Adsense and Google Analytics. These services give Mountain View a foothold on millions of properties across the web, from which they can observe users activities and track them from site to site. Today I finally got around to removing these from my own sites. I never paid much attention to the analytics anyway and there are more privacy-friendly alternatives if required. Adsense never generated enough income to compensate for the cheapening of the message.

Other unrelated changes I’ve made to this blog include changing the canonical domain to blog.dandyer.co.uk instead of blog.uncommons.org (the old URLs will redirect). In addition, some time ago I disabled comments. There were occasionally a few worthwhile responses but over 99.6% of all comments submitted were spam. The bottom half of the web is generally a cesspit. If you feel the need to respond to something I’ve written, you can contact me on Twitter or e-mail dan at this domain.

Google policy is to get right up to the creepy line and not cross it.

These are the words of former Google CEO Eric Schmidt, speaking to The Atlantic in October 2010.

We don’t need you to type at all. We know where you are. We know where you’ve been. We can more or less know what you’re thinking about.

This was two and a half years ago. Today Google is so far over the creepy line it can’t even see the line any more. The problem is not so much individual Google products but the way in which the vast data from all these disparate services is combined to build a very detailed profile of you. Not just what you do on Google sites but also any other site you visit that includes Google’s +1, Adsense or Analytics JavaScript (including this one). Google can read every e-mail you send and receive, it knows everything you search for online and knows pretty much every website you ever visit. It knows where you are, where you’ve been and probably with whom. Google knows more about you than your mother does and with Google+ it’s all neatly connected to your real identity.

I’ve so far avoided signing up for Google+ but it’s increasingly difficult to ignore, particularly as an Android developer, as it’s becoming more tightly integrated into everything that Google does. It’s the keystone of Google’s anti-privacy agenda.

The privacy implications are not the only issue. Most users perceive Google’s search results to represent some sort of objective truth, with links ranked only by their relevance to the search query, but in fact each user is served personalised results. If you and I both search for information on some contentious political topic, Google won’t necessarily give us the same response; it will show us each what it thinks we want to see based on its profiling of us.

In his Google I/O Q&A yesterday, Google CEO Larry Page dismissed concerns over the implications of this profiling:

[Audience member]: Most of my opinion, I can trace back to a Google search. As search becomes more and more personalized, and predictive, I worry that it informs my world view and rules out the possibility of some other serendipitous discovery. Any comment on that?

[Page]: People have a lot of concern about that – I’m totally not worried about that at all.

Personally, I don’t like Google’s all-encompassing vision. It’s not the only company that employs such methods but I can easily ignore the likes of Facebook and Bing as I don’t use them. Google is everywhere.

Spurred more by the closure of Google Reader than anything else, a couple of months ago I began to consider alternatives to relying on the benevolence of one omniscient company for the services I use every day.

The first thing to go was GMail. E-mail is important enough to be worth paying for so I signed-up with FastMail (now owned by Opera), which offers an ad-free service from $4.95 per year. It has a refined web interface, IMAP access, and a simple way to import your existing messages from GMail.

I eventually replaced Google Reader with The Old Reader and for search I’m now using DuckDuckGo, which promises not to track its users or filter search results. DuckDuckGo doesn’t quite match Google in terms of the freshness or depth of its results but it’s fast and has some useful features. On DuckDuckGo’s recommendation I also installed Ghostery for Opera to block Google and others’ attempts to track me on third-party sites. I’ve removed the Google +1 buttons from this blog and will be looking for more privacy-friendly alternatives to Google Analytics and Adsense (both of which would be useless anyway if everybody is driven to use the likes of Ghostery).

I’m not giving up on Google entirely, I just prefer to keep it at arm’s length. I’ll still be developing for Android (for which there were many very welcome developer announcements at I/O yesterday) and no doubt I’ll continue to use some of its other services.

About 18 months ago when I was looking for contract work, I signed up for Stack Overflow Careers. This was back when you had to pay to use the service (it’s now free but registration is by invitation only – more on that in a minute). The asking price had quickly dropped from an ambitious $99 a year to a merely illegal $9 by the time I signed up.

The Careers site has evolved considerably since it launched and now allows you to create quite a sophisticated online CV that aggregates your Stack Exchange activity, open source projects (GitHub, Google Code, etc.) and more. The idea of the service is that employers can search for candidates who appear to be good matches for their vacancies. All the enquiries I had were from companies developing financial software in London, which would have been ideal except for the work and the location.

Anyway, it seems that Stack Overflow is desperate for more people to sign up for Careers. When it went free-but-invite-only I was awarded 5 invites to distribute to suitable individuals. Last week I was given 20 more invites. Today I have been given two further batches of 20 invites. I currently have 63 invites available. If you want to try out the service for yourself, all you have to do is follow this link to claim one of my invites.

A while ago I created a new website for my main Open Source project, the Watchmaker Framework for Evolutionary Computation. While the new website was a definite improvement over the previous effort, it was still lacking something. It wasn’t distinctive. What I really needed was a logo, something that visually identified the project. But how do you represent evolution and/or a watchmaker in the form of a simple, distinct picture?

It was beyond my modest artistic skills so I headed to Reddit and floated the question of how to find a graphic designer who would be willing to contribute to an Open Source project. I wasn’t expecting much, after all I was asking somebody to work for free on a fairly obscure project.

The usual way to get somebody to make you a logo is to find a professional graphic designer and pay them, or to stump up some cash to fund a contest on worth1000.com or 99designs.com. The prices start at $204 at the latter. I got a few suggestions from the Reddit crowd that I should try this monetary compensation idea.

I also got several people offering to produce a logo for me free-of-charge, and even a few who spontaneously decided to create something and submit it as a suggestion (see some of the links in the Reddit thread).  I really wasn’t sure if there would be many graphic designers who were interested in helping out Open Source software projects but it seems there are plenty. What is lacking is somewhere on the web to connect these willing designers with needy projects.

One of the people who got in touch to offer his services was Charles Burdett. He sent me a link to his impressive portfolio. I quickly accepted Charles’ offer before he had a chance to change his mind. This meant turning down a number of other offers that I received. I’m extremely grateful to all of the people who were willing to help me out, and some of the concepts that people suggested would have made very good logos.

Charles came up with the concept that now adorns the Watchmaker project website – the clockwork Ichthyostega. To be honest, I don’t even know how to pronounce that but Wikipedia tells me that Ichthyostega was a creature from the Upper Devonian Period. It was an intermediate form between fish and amphibian, so a significant step in the evolution of life on this planet.

This logo fits in very nicely with the existing site design and, because it’s a simple outline drawing, it should be very versatile for use in different contexts. I’m very happy with the result and I’d like to thank Charles for all of his work on this. If you like this logo, or any of Charles’ other work, he is available for hire.

The end of the year is here and, as is traditional among bloggers and mainstream media alike, I’ve lazily compiled a top 10 list to mark the occasion without having to exert myself. So here it is, according to Google Analytics, the top 10 articles of 2009 from this sporadically updated blog.

When I checked the stats, four of the top five most read articles this year were actually from 2008, with Why are you still not using Hudson? claiming first place. This list includes only those articles that were first posted in 2009.

1. 5 Ways to Become a Famous Programmer
2. Practical Evolutionary Computation: An Introduction
3. Random Number Generators: There Should be Only One
4. Practical Evolutionary Computation: Implementation
5. Debugging Java Web Start Applications
6. Using PHPUnit with Hudson
7. Understanding PHP – A journey into the darkness…
8. Programming the Semantic Web and Beautiful Data
9. Uncommons Maths 1.2
10. The Java Language Features that Nobody Uses

Happy New Year.

Opera Software would have you believe that yesterday they reinvented the web.  The launch of their new Opera Unite service has received a decent amount of publicity. By now you’ve probably heard all about it, but if not you can read the details here.

The 10 second summary is that version 10 of Opera’s web browser contains a web server that allows users to serve web content directly from their desktop machines or laptops. However, this description doesn’t really capture the potential of the platform.

Some commentators have dismissed the announcement with a “so what?”. Opera Unite content is only going to be available while the user’s computer is switched on and running Opera and will be constrained by their available upload bandwidth (which often isn’t much thanks to the ‘A’ in ADSL). That doesn’t really cut it when compared to low-cost web hosting packages capable of serving thousands of users, but then the comparison isn’t particularly helpful.

I don’t need Opera Unite to host my personal website from my desktop. I can install and configure Apache, tweak my firewall/router settings and find a solution to dynamic IP address issues. The point is that with Opera Unite, you don’t have to do any of that.  Opera have completely eliminated all of that hassle and in doing so have made web serving accessible to even non-technical users.  But that’s only half of the story. Serving your personal home page via Opera Unite is still sub-optimal. If you want (semi-)permanent web hosting, pay for some cheap PHP hosting or get a WordPress.com account.

If somebody gives you an Opera Unite URL, you shouldn’t expect that resource to be still around tomorrow or next year like you would with a link to Wikipedia. The real value in Opera Unite is in ad hoc sharing and transient collaboration. Things that were possible but bothersome previously are now trivial because you don’t have to worry about server configuration and networking issues.

For example, say I wanted to invite every reader of this blog to join a chat session. I could try to find out which IM clients you all use and try to arrange something via MSN Messenger, Skype or Google Talk. Or I could install and configure my own IRC server. Or I could try to find a third-party server to host the chat room. With Opera Unite I can simply open up my lounge and give you all the URL (regardless of which browser you happen to be using). It just takes a few clicks. The service is transient.  When we’re done, I kick you all out.

In our chat session I might decide to share some photos or other files with you.  I could send them via e-mail or upload them to an FTP server or a service like Flickr, but again it’s simpler with Unite. I just enable the appropriate service and share the URL. You can browse my shared directory and grab what you want directly from my machine. The link probably won’t work tomorrow, but you won’t need it tomorrow. Temporary is fine when it’s this easy.

The other service that I’m already finding useful is the media player, which enables me to remotely play my home MP3 collection from the office. The Unite platform is based on open standards, so it will be interesting to see what other ideas for services people come up with.

I’ve said it before, I don’t get Twitter.  For me, the hysteria generated by the flat-lined signal-to-noise ratio of this limited medium is deeply confusing. The last time I felt like this was back in the early 90s as I stared cross-eyed and frustrated at one of those Magic Eye pictures, trying to find the dolphin. How come everybody else can see something in it while to me it’s just nonsense?

Since I last wrote on the subject of Twitter, the hype has increased still further. Somebody used TwitPic to post a picture of the AirBus that landed in the Hudson River in New York. As a result, this “citizen journalist” earned himself 15 minutes of international fame, including featuring on the BBC’s television news at least twice. I somehow suspect that had he chosen to upload his snap to Flickr instead, the BBC would not have been nearly as interested. Somebody at BBC News is a Twitter lover. The corporation’s online coverage of the recent attacks in Mumbai prominently featured information sourced from Twitter, whether accurate or not.

If further evidence were needed that with Twitter the medium really is more important than the message, it arrived last week. What was this seismic event that many are heralding as the tipping point for the microblogging revolution?  Stephen Fry got stuck in a lift. No, really, that’s it. Had General Melchett used his Blackberry to phone a journalist instead of to tweet, he could not have bribed them to write about this unfortunate but utterly banal occurrence.

I am not alone in my dismissal of Twitter as an irrelevance. Terence Blacker writing in the Independent aptly described it as “self-stalking” and summed it up as follows:

Twitter may have novelty value but it is more than mere surface silliness. It is anti-thought, the deadening white noise of modern life with all its pointless business. As for the dotty idea that short computer messages are full of wit, insight or observation – that is, to quote the master twitter himself, “arse, poo and widdle”.

I discovered the Independent article via Graham Linehan’s blog. Linehan (co-writer of the peerless Father Ted) has a different take on the value of Twitter:

…the manifold possibilities of Twitter are enough to make you giddy. This is a new world, people! We are officially in the future, not with jetpacks, but with something much cooler – the hive mind. Ignore those grumpy luddites in the broadsheets and elsewhere, who don’t understand it, can’t be bothered to learn how it works and are frightened at the prospect that people are entertaining themselves in a way that doesn’t involve accepted media forms.

By now, I think that I’ve firmly established that I don’t really see the point of Twitter, but I don’t want to be dismissed as a “luddite” who “can’t be bothered to learn how it works”. So it’s time to take the red pill and see for myself how deep this rabbit hole goes. Today I’m embarking on an experiment: a two-week trial to see whether there is any substance to the Twitter hype. I’m deeply sceptical but also approaching it with an open mind. Will I experience a higher level of conciousness or will I endure and proliferate a fortnight of pointless anti-thought?

I’ve just signed up for a Twitter account. If you’re already a Twitter user, I need some followers (sounds like I’m starting my own religion here). I also need interesting people to follow. Please also send me tips on how to get the most out of Twitter. If you are one of the many people who responded to my previous post agreeing that Twitter was pointless, I will report back here at regular intervals in the next couple of weeks and let you know how it’s going on the other side.

Stand back, I’m going in…

WordPress is great. In some ways. Sometimes. It has all these plugins that add all kinds of neat features. It even has a really easy-to-use auto-update feature for plugins. So when I saw that a new version of Ryan McGeary’s very useful WP-Syntax plugin was available, I let it do the upgrade for me.

Unfortunately, this made a mess of my previous post (thanks to David for pointing this out to me – I wasn’t actually diligent enough to check for myself that the upgrade hadn’t broken anything). All the < and > characters in my Java code had been replaced by HTML entities (> and <). Rolling back to WP-Syntax 0.9.1 didn’t fix the problem, which was odd. Then a few neurons flickered into life and I had a vague recollection of having solved this problem previously. It turns out that I had modified the PHP source for the WP-Syntax plugin (as described here). I simply had to make the same change for the new version.  I’m ignorant as to why WP-Syntax does not include this modification by default. Maybe it breaks something else, but it works for me.

Since my previous post on the subject, Stackoverflow.com has moved from private beta to public beta. I’ve had more time to use the site and have some more thoughts.  The criticisms here are meant to be constructive. Hopefully the feedback from users will help the Stackoverflow team to make a good site even better.

Performance

First the good news. The site has transitioned from private to public very well. Jeff and his team seem to have got it right in terms of architecture and infrastructure because, even with the increased load, it remains blindingly fast.

Front Page

In terms of usability, I think there’s more that could be done to help me find the content that I’m interested in. The default front page is, to be honest, not very useful. New questions are coming in so fast and on so many topics that displaying the most recent questions is just noise.

I would prefer to have a personalised home page that shows me relevant questions based on my previous answering/voting history.  I realise that this is major new functionality and I’m not criticising the Stackoverflow team for not having this in the initial version, it makes sense to get the site up and running first. However, it would be great if this could be implemented at some point. I’m not alone on this one, it’s the second most popular requested feature at the moment.

Presently I’m finding stuff that I want to look at by going to the tags page and clicking on interesting topics. But I’m sure I’m missing out on questions that would be of interest if only I could find them.

Tag Cloud

The tag cloud on the right of the front page isn’t very helpful either. It’s ordered with the most recent first. If I just wanted to view questions tagged “html”, I’m going to struggle to find the tag in the cloud. An alphabetical ordering would be more usable. Unfortunately, this has already been suggested and rejected.

Voting and Reputation

I outlined my concerns on the voting mechanism previously. In the interests of being constructive, rather than just a whiny blogger, I’ve opened new issues on the Stackoverflow Uservoice page. If you agree with me, please vote on these issues:

• Don’t allow users to answer and vote on the same question (and remove the -1 penalty for down-voting)
• Make down votes equivalent to up votes

Addressing each of these will help in resolving The Fastest Gun in the West Problem (currently the number one voted-on issue). The problem is that early answers get the votes and later, better answers are largely ignored. Removing the penalty for down-voting will encourage more down votes where they are deserved (so an early answer that is later shown to be wrong is less likely to retain a high score). Also, if a down vote was as powerful as an up vote, people might be more careful in crafting good answers as opposed to quick answers.

Over the last few days I’ve been playing with the beta of Stack Overflow. In case you are unaware, Stack Overflow is a joint venture between Jeff Atwood of Coding Horror and Joel Spolsky of Joel on Software fame. It’s basically a question and answers site for software developers. A mixture of Experts Exchange, Proggit and Wikipedia. The site is scheduled to come out of beta on Monday when it will open its doors to everyone.

From initial impressions I think it’s fair to say that the site will be a success, initially at least. Being A-list bloggers (and now podcasters too), Jeff and Joel have been able to generate a lot of exposure for their project.

Like Jeff’s blog, the minimalist site design is clean and bold and so far the whole system is very responsive (we’ll see if that’s still the case when the traffic spikes on Monday). The beta audience are already posting thousands of questions, almost all of which generate extremely prompt answers (of varying quality).

However, I think the site suffers a little from the ambitions of trying to be too many different things; is it a programming forum, or is it a Wiki?. There are a lot of different ideas in the implementation that interact via a quite complicated set of rules that have evolved over the course of the beta.

Reputation & Badges

Stack Overflow has two mechanisms for measuring a user’s standing within the community. Firstly, each user has a reputation score. This starts at 1 and increases as you make positive contributions (posting questions and answers that get voted up).  As you reach various milestones you get more privileges within the community, such as being able to vote on answers or tag other people’s questions.

Your reputation can be diminished if you get voted down or reported for abuse, but it can’t go below 1 and on the whole it’s heavily biased in favour of upward movement.

The second incentive for users to contribute is the ability to collect “badges”. This works exactly like the Cub Scouts. Some badges are easy to achieve (just fill in your profile or post your first question), and others are much harder to obtain (get 100 up votes for one of your answers).

Voting

Voting is one area of the site that I think could do with an overhaul. It’s unbalanced and not transparent enough. If your answer gets voted up you gain 10 points of reputation. But if your answer gets voted down you only lose 2 points. So if you post something that sounds plausible to the uninformed masses but is actually wrong, you could get 5 up votes and 6 down votes for a net score of -1 yet still gain a 38-point reputation boost. An up vote should have equal weight to a down vote, just like on DZone or Slashdot. It also might be better to show both the number of up votes and the number of down votes (as on DZone) rather than just the net total. This would make it easier to identify controversial content (something with 10 down votes and 12 up votes is not quite the same as something with no down votes and 2 up votes).

Another problem with the voting is that down votes penalise the voter as well as the user whose answer is being voted on. So if you post something wrong like “Java passes objects by reference”, I can either ignore it or lose 1 point of reputation for giving you the down vote that you deserve (even then it will take five of us to fully cancel out the one up vote that you got from someone who didn’t know better).

When I queried the justification for penalising down-voters I was told that it was to combat attempts to game the system. Apparently, earlier in the beta, users were posting answers to questions and then voting down everybody else’s answers so that their answer would appear at the top. The idea was that by making users pay to vote down this behaviour would be discouraged. A better solution to this problem would have been to remove the conflict of interest by not allowing users to answer and vote on the same question (which is how Slashdot’s mod points work), rather than punishing all down votes across the whole site.

The net effect of this voting system is that everybody’s reputation increases pretty quickly. Beyond the minimum score required to get full privileges the numbers can become meaningless. To avoid having to rename the site Integer Overflow there are a couple of artifical limits that restrict the number of votes you can cast and the number of reputation points you can earn each day.

Other Thoughts

Aside from my reservations about the voting, my impressions of Stackoverflow are mostly positive. The fact that it has already attracted hundreds of enthusiastic participants suggests that it has genuinely found a niche. However, I do feel that it is probably more elaborate than it needs to be (I don’t really get the need for the Wiki functionality).

Further Reading

Denton Gentry and Sara Chipps have also written about their impressions of Stack Overflow. Or, on a less positive note, you could try Crap Overflow.