1, Infinite Loop.

I have been working on the iOS platform for two years now. I really enjoy it. I like the focus on fluidity and good user experience championed by Apple but I also like the engineering challenges that programming apps offers. Here is a whirlwind tour of what difficulties you could expect when beginning to program on iOS.

Fluidity

The first constraint that you have to take into account is this famous focus on fluidity. As I once saw on Twitter, the rule is "60fps or GTFO" (I think this was from @flyosity). Apple has set a standard of smoothness through all their apps (OK, almost all). The scrolling speed and responsiveness of the iOS platform is still globally a notch above everything else. This is seen as a given and you will see most of your users tap impatiently on their device if your app freezes more than half a second. And freezing while scrolling will most probably decrease your average number of stars on the app store.

Concurrency

To get fluidity, the recipe is well known: there is one main thread in charge of updating the UI of your apps. You just need to avoid doing input/output (disk or network) and expensive computations on this thread that would cause UI freezes.

Put differently, you have to make your program concurrent: launching the expensive tasks in background and waiting for them to callback. Problem: concurrency is hard, PhD thesis level hard. It is very difficult to think about a concurrent program because you lose the strict sequentiality, and you can easily get in troubles when modifying the same memory space through different threads.

The main problem here is that if you forget to go back to the main thread to update your UI at some point, you will experience crashes in your app (because UIKit, the graphical framework of Apple, is not thread-safe), but as always with concurrency, these crashes might not always happen, and be hard to reproduce in the debugging environment, because setting a breakpoint for example will change the sequence of events.

The matter is further complicated by the fact that if you use Core Data (and if you need structured data storage, this is the way), it brings its own constraints. Basically, you cannot manipulate a ManagedObject (the base data class of Core Data) on another thread than the one where it was created without risks of crashes. Happily, iOS offers some nice ways of distributing your work over multiple threads, but this brings me to the next difficulty of the iOS platform.

More than one way to do it

The Perl motto applies very well to iOS: "There's more than one way to do it". The technologies used in iOS date back to NeXTSTEP in 1989 and it shows, because you still find in the libraries the traces of this long history, with lots of variations in the way to do things over time. You often find very similar implementations of the same concepts, one in Objective-C and the other in a pure C API (UIFont vs CTFont, for example), with naturally some discrepancies. To get back to threads for example, you have at least 4 different ways to launch a task on a background thread:

1. using NSObject's performSelectorInBackground:withObject:
2. using NSThread
3. using the Grand Central Dispatch C API
4. using NSOperation introduced in iOS4

How do you know which one you should use? As a rule of thumb, you should use the highest level of abstraction that suits your needs, but this brings us to the next difficulty of iOS.

The doc

You will have to navigate the docs to find out which way is the right one in your case. Unfortunately, Apple's documentation is not easy for the beginners. As I read somewhere that I cannot find back now: the documentation is quite clear once you have understood the subject matter, meaning that everything is explained, but not always in the most straightforward manner, and sometimes, information is hidden in suprising spots. Notably, when lost, you should hunt first example Xcode projects provided by Apple to illustrate the use of some APIs, but beware, some of them are seriously outdated, and then you should look at WWWDC videos. Honestly, it was quite infurating for me to discover that some information is only available in video format (about Core Data contexts merging for example) which is obviously not very googlable (especially since this material is in fact behind a login wall).

Network

Another thing that makes iOS programming difficult is that you cannot count on stable network connections (but this applies to all mobile platforms, obviously). Every network request can fail due to the user movement. This is quite different from what happens when you do front end development or server development, where most connections are quite stable. You have to provide graceful behaviours for so much more types of problems.

What makes this even more important is that, compared to web apps, where users have been long taught to do a refresh when something seems to freeze, on mobile, users are not used to relaunch their apps in case of problems (and happily so). If your app is stuck in a bad state with no way for the user to get back in a better situation, there is a good chance that he will simply uninstall the app.

Other difficulties arise when you need to offer some functionalities offline (think about people using your app on the plane). This brings synchronization problems, that as a general rule, are as hard as concurrency problems. Nothing strictly related to iOS, but still a major cause of headaches.

Resources

Constrained resources must be cited too as difficulties. Over 70MB of RAM, your app will probably be killed by the system (but not for sure, it depends on other processes and the memory available on the device). Lots of javascript web apps consume more than that these days! Aside of memory, CPU is not that powerful, a lot less than your laptop computer obviously. This has to be taken into account too.

Memory management used to be a major pain point in objective-c, but it has gotten a lot better since the introduction of automatic reference counting in 2011. You still have to understand what will be kept in memory (everything for which your keep a strong pointer), and you will probably have to understand at some point how ARC behaves when using blocks, but over is the time where you spent hours chasing the superfluous release in your app.

Fighting the system

The degree of liberty offered in iOS to developers is way smaller than what is offered on Android. For example, you are not allowed to keep a daemon running in background when your app is minimized (to download files for example). If you have something like that to do, you have at most 10 minutes to execute them after the user minimized your app. Similarly, you are not allowed to keep a connection open to a server passively either, unless you are a VOIP app. There are also constraints to access the address book of users, keeping the GPS active, and so on, and so forth.

Most of these constraints stems from Apple's intent to keep the user experience pristine from bad developers experimentations, but that does not mean that these constraints cannot be frustrating and that finding workarounds is easy.

App Layout and Text manipulation

There is nothing similar to the flowing layout of HTML in iOS. If you want for example a UI with a paragraph of text and a button just underneath, you will have to compute the height of the text, and put the button at the right place. This can seem particulary brutal for web developers, and be hard to explain to customers and managers accustomed to the relative plasticity of html and css.

Furthermore, this may not apply to all types of apps, but if you need pixel perfect text manipulation, like I did for checkthis, you are in for lots of troubles. Either you have text editing using UITextview but without too much control on things like line height and stylings used for links, either, you have the full control offered by CoreText but without any editing capacities. You have to program all editing by yourself. I must say that there is hope on this front, with this (paying) component, but I did not test it. If you do not need text edition, but just more control over the layout and styling, you can use TTTAttributedLabel.

For layouting text, lots of people resort to HTML and CSS in UIWebView but this can get a bit unholy and offers challenge on its own due to the asynchronous rendering of webviews. I will not dive too much into the controversy of Native vs Web Apps, but let me just state that most popular apps on the App Store are mainly native.

Closed source

This might be obvious to some, but the fact that you cannot access the sources of the UIKit framework, or other important parts of the iOS platform can turn into a real problem when you hit a hard to understand bug. I lost count of the number of time I read on Twitter something along the lines of : "Apple's component x does not do expected behaviour Y, please give me back the days lost trying to understand." and I have been a victim numerous times too. The platform is not especially buggy, but not having access to the source complexifies debugging when needed.

As a side note, the quality of iOS open source varies wildly and there are big surprising holes that are not covered, like for example having a good Twitter connection library that would take into account the latest additions of the Social Framework, while fallbacking to classic in-browser-oauth when on iOS4-5 or when the user did not setup his accounts. I am not complaining about anyone's work here, just observing.

Conclusion

In conclusion, iOS is at the same time one of the most rewarding and challenging platform that I had to work with. There are lots of constraints to take into account but happily, I find the tooling extremely good. Instruments and Xcode are first rate tools. They obviously have their own quirks, but compared to Eclipse or Visual Studio, they fare very well.

On the rewarding side, Apple's customers are using tons of apps that they will take wherever they go, and this is the most personnal computing that I know of. Furthermore, there is a lot of demand for competent mobile developers these days and mobile is where the double digit growth is. So, if you are on the fence, I would encourage you to take the plunge into iOS development.

A gift for my sister in law. She’s a shrink. This thing makes me giddy.

Réservoir de Central Park

Near the betacowork

This one was of my best concert experience. An attendance around 40 people, in an old Brussels house. A super tight set by the BNRS band

The last 11 months have been quite hectic. Mainly, I bought a house and moved into it with my family, I got a new job as the senior programmer in a small web agency and finally, I sneaked in some side projects that were important to me.

This blog post is the first part, from January to July, of an almost exhaustive recap of what I have been programming this year.

First, in January and already detailed on this site, there has been populationpyramid.net, a website allowing you to browse the population pyramids of all the countries in the world, based on the United States data. This was my first incursion in the open data world, and a first attempt at making data visualization in HTML5.

Then, at the beginning of February, I became the IT manager of a small web agency in Brussels: "my media is rich". My duties include system administration, technical architecture, lots of development and globally, technical lead in all software matters. The first thing they had me doing was an iPad application aimed at the Brussels car show. A 6 days deadline for a commercial project on a platform I had never programmed on before, that is the kind of challenge I like to take on (at a bit of expenses to my sleep and family, I must admit). I finally got that one off the ground and learned quite a lot in the process about how web agencies are working (in a very industrialized way, with well defined position: the designer, the motion designer, the project manager, the programmer, the commercial). It was quite interesting. You can see the app here.

The one point that has been obvious from the start there, was that I enjoyed the change to small, well defined projects that those companies churn out, by opposition to the never ending projects I have been working on before, either at the university for my research or the projects I had in my previous company. I had the feeling of getting things done, without too much meetings or email exchanges.

Next up, I had to realize some websites for big clients of ours like http://www.bfoskodarally.be (my first use of Django in a team, with a customized admin aimed to be used by a client) and http://www.generousdays.be, which is now offline. I learnt quite a lot about javascript with those projects.

Then came a very nice challenge. I had to realize one of the fist viral marketing campaign on LinkedIn for Audi. This gave the Audi A6 Challenge. The concept was quite simple: we had to create a "Hot or Not" on LinkedIn, meaning that any user would have to vote repeatedly on two of his/her LinkedIn connection to decide which one they thought as the best professional.
There is a lot to be said about this project. We got quite a bit of attention, even from the people at LinkedIn (if only to inform us about some rule trespassing that we had unwillingly been doing). I liked this project because it was the first time that I got to really work with an API of a social network (the kind your keep hearing about all the time). I was happy that we managed to create a system where the contest was still winnable by lots of people by the end. There is indeed a risk in such contests that a few people take the lead in the beginning and then get unreachable scores, preventing other people to take part. I used a variation of the ELO score used in chess, where people lose more points when losing against somebody with a lower score. This prevented people to get away too fast. Furthermore, we managed quite efficiently to shut down cheating, which is a tough problem for every contest on the web as I have discovered. There are lots of sites for exchanging cheats in such contests (like this one). Finally, the user interface and design were quite polished, which again was a big change of focus from my previous jobs.

Following that, I had to implement a small iOS application for stands on the Carrefour Running Tour Except for the tight deadline again and the fact that I learned to hate iTunes Connect, not much to say about this.

During those months, from february to july, there were three side projects

• In April, I helped a bit to enhance the Django development dashboard, by proposing to add some sparklines (the yellow lines on the site), and implementing them using raphael.js. This forced me to delve into Jacob Kaplan-Moss's code (one of the creators of Django), where I could pick a few interesting tricks. In the end, I submitted a pull request that got accepted, which was important to me, as my first participation to a preexisting open source project, even if in the end, Jacob did not keep much of my code anyway (for good reasons). I have to participate in such projects again. I was very enthusiastic about this.

• At end of May, I was contacted by Jonathan Van Parijs, who organizes the Hack Democracy meetups in Brussels to take part, on the computer front, to a participatory democracy experiment, the g1000. I will not delve into the ideas of the initiative, which are well explained on the site. They asked me to help for the creation of their website using a yet to be defined Photoshop design… In two weeks… In my spare time…
  Let just say that I did not get enough sleep again during those weeks, but I fortunately had help from Jonathan and my awesome colleague, Christophe Gérard (he's freelance now, hire him). This project, not excessively complex, was executed using Django again, but in the process, I have been forced to enhance my Photoshop slicing and css skill a lot.
  From this experience, I keep lots of good memories.
  I will especially remember the long hacking nights with a sense of purpose and in the end the launch of the site at midnight on the 10th of june, from my nephew's room (we were invited to his mother's birthday), with loud basses coming from the party downstair, using my android phone tethering connection because nothing else was available. The opening of the site at midnight had been announced in newspapers and through a press conference and so, we had an expecting audience that brought us some hundreds of visits in the first two hours. A nice, memorable launch.
  Later on, the site has been updated a lot, and in the end, after the main event, it has already got around 150 000 visits. Not too shabby for such a subject, but I must say that the event got a lot of press coverage, even internationally. Technically, the site has been hosted on a 10€/month cloud server at Rackspace, using nginx, mysql and memcached and we never had any performance hiccups at anytime.
  

• The third one is not a programming project. At the end of june, I took two evenings to write down what happened to my significant other when her gmail account had been hacked. This gave the previous blog post If I wrote it, it is obviously because I thought this could have some interest, but I was definitely overwhelmed by the reaction. I submitted it to Hacker News before going to sleep and it raised to the top spot on the front page, where it remained for almost a day bringing around 35000 visits in one day. There were lots of interesting comments (notably, the top one, from the well-known Matt Cutts). On the technical side, this blog is hosted on a shared server at djangohosting.ch and it never got unresponsive. This leads me to wonder which kind of traffic you need on a rather static site to have performance issues. I still do not have answer for this…

In part two of this recap, I will talk about the rather big projects that happened in the second part of the year, and about what's next (with a surprise!).

(To Be Continued...)

On may 17th, in the evening, I received an email from the Gmail account of Charlotte, my significant other. It was written in french (which is normal for her) and looked like this :

How are you ? Would you have time to spend by email on a peculiar situation about me ? I am in deep problems and couldn’t cope with your support.

Hoping to hear from you really soon.

Best, Charlotte

You’ll find the french original text here under (so that people can find it on Google).

I was quite busy and so immediately dismissed this as spam, and did not bother to check where this email had been sent from. Faking email addresses is way to easy to bother for each suspect email. As many people with a public email address, I often receive fake emails from myself.

But this time, the problem was deeper, as I learnt when Charlotte, the real one, called me to warn me that she could not access her Gmail account anymore and that her phone was constantly ringing because of people worried about her. She also told me about a popup that she had seen in the morning about suspect access to her account from the Ivory Coast. At the time, she was quite busy, clicked on some option that looked reassuring and went on with her day. Damn, that was bad.

I immediately tried to access the account (I know her password), to discover that her password had been changed and much worse, that the recovery email address had been changed too (it read something like xxxx1@live.fr where it should have read something like xxxx@ucl.ac.be). I also soon realized that the security question had been changed too. Really, really bad.

Without much hope, I began to fill the “last chance form” of Google. Let it be clear that if you use a free Google account (and even a paying one, in my opinion), you have very few chances to get real people at Google looking at your problem. It is quite logical if you think of the very high ratio of users versus google members : to keep things manageable, they automate administration tasks as much as possible and so, you are in fact only interacting with their programs, never with them directly.

So, when you fill the “last chance form” of Google, you know that you have to fill it as precisely as possible, so that you request pass their automated test. And to examine your request, Google is asking as much information as possible, notably : - the date of your google account registration and the verification code you received at this moment (happily, both information were available in my Gmail account) - the name of the people to whom you write emails the most - the name of the tags you employ and so on and so forth.

With one phone call to Charlotte, I was able to get almost all these information and finally submitted the form, hoping to get an answer for the next day, if I ever got one.

In fact, I got an answer under 15 minutes. I was really happy about this. I immediately logged into Charlotte’s account and went at the bottom of Gmail home page to view the little snippet of text showing if the account was open anywhere else (more info here). By clicking on details on the right of this snippet, you can also sign out all other sessions. There was indeed a session opened in the Ivory Coast (while we are living in Belgium….) so I did close the sessions and changed the password, security question and backup email address. I finally examined all settings of the Gmail account to find out that a forwarding of all emails had been put in place to the following address : tatianalabelle1@live.fr (I don’t see any reason to keep this secret….). Now I did feel almost safe, but still a bit nervous.

Time now for some damage evaluation. I immediately saw that all contacts had been deleted (annoying but not too bad) and that apparently all emails had been put in the trash (but I immediately saw that some were missing too, from the last few days) except for the responses to the fake “emergency” email. Already around 10 responses. I immediately responded to people not to take this into account. This was a bit embarassing for Charlotte, since lots of his colleagues and her whole familly and friends had been contacted. This took me around 10 minutes.

That’s when I lost the connection to the account again.

I immediately tried to login again, to find out that passwords, security questions and backup email addresses had been changed again. It was time for heavy swearing on my part, with some punching of the table.

I then did everything again : filling the form, waiting 15 minutes to get an answer, signing out sessions, changing passwords, security questions and checking again if no other session had been opened in the mean time. According to Google, it was not the case. As a side not, I was very glad that the “last chance form” did work twice. I was really thinking that it could be blocked after a first attempt, to allow further investigation. But I guess that since a human investigation is not really an option, Google chose to let the system run as much as needed. I will probably never know… So I began responding to emails again.

That’s when I lost the connection again…. Password, security question and backup email changed again.

At this stage, I was getting a bit desperate. I called Charlotte, asking her if any of our computers were open with a session (this may not sound very rational, but you never know). It turns out that the windows XP machine that I keep for gaming was on. I told her to turn it off (I did not had tim to inspect it since). Charlotte also told me to delete all her emails if I could get into the account again, to decrease privacy intrusions (even if at this stage, it could have been too late). In despair, I wrote an email to tatianalabelle1@live.fr asking to stop pirating my girlfriend email account.

I tried the “last chance form” again and it did work a third time. I did everything to try to secure the account and began to respond to emails again. It turns out that this time, I did not lose the connection again. I have been checking this regularly for than a month now, and there was no more suspect activity.

This whole story did left a sour taste though. Charlotte lost all her contacts and past emails. This is not too bad for her, she is not an heavy Gmail user (she has a professional email address too) but I definitely am. I use it for all my emails, my calendar, and lots of other google tools (Analytics for websites for example). If I ever lose my Google account, I could lose a great deal of time and value. This is accentuated by the fact that I have used my Gmail address for registering in lots of other systems too (Ebay, PayPal, Facebook, Twitter, Apple Dev Center, Itunes, some server hosting and so on). For almost all those services, if you have access to my email, you can get access to the service (by filling the “lost password” form). They will send a link to allow you to change the password and login again.

In the precise case of Charlotte, using this kind of accounts did not seem to be the plan. Some people who responded to the fake email got responses where they were asked to send money using Western Union to help Charlotte in Africa, where she had allegedly almost been raped and had not access to a phone. Not very subtle but I still felt the bullet passing much too close…

To mitigate the risk, Google recently launched two-factor authentication, a mechanism that requires you to input, on top of your password, a code generated by an application installed on your phone (iPhone, Android and maybe some others). I have activated this today. You can find more information about this here : advanced sign in security for your google account This indeed increases security, but tends to be a bit cumbersome (I often have a depleted battery, for example, which could prevent access to my emails from a computer) and does not solve other case (like somebody stealing my laptop and using an already opened session).

In the end, I am still feeling a tad insecure about using Gmail as may main account. I think it’s way too difficult to get somebody on the line to help you in case of problem. I am nevertheless quite addicted to their interface, and consequently does not want to give it up, but I will probably switch my main account to a domain I own, so that at least I could shut down the email address in case of need, but still use the Gmail tools through Google apps.

I decided to write this all down so that it could serve as a cautionary tale. You should keep in mind the limitation of your email provider and if you decide to use Gmail, you should keep as much information about your Gmail registration as possible (try to find back your first registration confirmation), and if the option is offered to you, which seems to be the case only for heavy users (probably the people using some paying services from Google), you should activate two-factor authentication. As a side note, I would be definitely interested in buying from Google a gadget similar to the Blizzard Battle net authenticators to use two-factor authentication. I would then be able to keep my phone as a backup option only.

Finally, I am still wondering how those guys in the Ivory Coast got access to Charlotte’s email account. My main hypothesis is that she must have accessed her account on a pc infected by a key logger (To excuse her, she’s working in a big structure, she not in IT and could not possibly control every machine). That said, I found the action quite thoroughly organized (notably, forwarding emails to an external email address is a step that could easily be missed during the recovery of an account), and the most distressing to me is that I am still unable to explain how those guys were able to get access to the account twice after I changed the password, security questions and backup email address from my Mac that does not seem to be compromised.

If you have plausible explanations, I am definitely interested in hearing them.

Original French Text of the fake email :

Comment vas tu ? Aurais tu du temps à consacrer à une situation particulière me concernant discrètement et par mail ? Je suis dans des difficultés telles que je ne saurai que faire sans ton soutien et apport.

Je reste dans l’attente urgente de te lire.

Cordialement,

Charlotte

For some months, I have been interested in data visualization. I have been playing with Processing, and Raphaël and read one of Edward Tufte's books.

Besides, as any motivated web developer, I am interested by the new crop of features appearing in our browsers under the banner HTML5, especially SVG and canvas, which allows to draw vectorial graphics in your browser more easily and to manipulate them using Javascript.

Finally, I am fascinated by the idea of open data, that is, the public availability of data in an easily processable format. For an example of projects made possible by open data, you could check www.wheresmyvillo.be which was made possible by the availability (not official in this case ...) of raw data about the bike sharing scheme of Brussels on the web. My interest about this subject was initially sparked by Adrian Holovaty, one of the creator of django, who created www.everyblock.com, a site allowing you to be informed about every public bit of information in you neighborood (from meetups about programming to burglaries).

With all this in mind, in November last year, I went to the first Hacks/Hackers event in Brussels. It is an event aimed at bringing programmers and journalists together to find out interesting ways to exploit open data. It was a very nice meeting, very inspiring, and it crystalized my interests. I had to do something combining it all.

So, in December, I began to look for interesting public data sets. I found out, thanks to a Wikipedia article reference, that the United Nations have made available the content of their World Population Prospects at http://esa.un.org/unpp/. After some thinking and googling, I came to the conclusion that something as basic as the set of population pyramids of all countries of the world were not directly available online. I settled to publish it, using Raphaël.js to obtain a beautiful visualization. Although not very original, I think those graphs are highly thought-provoking.

Here are a few interesting points:

• One can easily spot the problem for future public retirement plans problem in Europe (with more and more elderly people ). See here for example : population pyramid for Belgium in 2050
• I am still wondering what is happening in countries like Qatar in 2010 (population pyramid for Qatar in 2010) How come there are so many more men than women ? According to a Hacker News comment, this could be caused by migrant workers
• Near 1950 in Europe, you can spot the "dent" made by WW1 in the population, plus the baby boom. Example : population pyramid for France in 1950
• It is already interesting to compare the shape of the pyramid for countries like Afghanistan in 2010 (population pyramid for Afghanistan in 2010) VS the US in 2010 ( population pyramid for the U.S.A. in 2010). One can easily see the difference in development, to say the least.

It took me approximatively two months to finish the project (in fact approximatively five long evenings, but it took me two months to find those five evenings....). From the start, I had envisioned a project with no backend, only Javascript goodness. So, yes, no Django this time. This made possible to host the project on GitHub pages (from the homepage : "The GitHub Pages feature allows you to publish content to the web by simply pushing content to one of your GitHub hosted repositories.") which meant free hosting, and furthermore was a good fit, since I wanted to publish all code under an open source license.

About the code

All code is available on GitHub at : https://github.com/madewulf/populationpyramid.net. I do not think this code would be very good for reuse, but it could be used as an inspiration. The resulting html page is unfortunately completely dependent on the availability of JavaScript to render anything useful and will consequently probably not be referenced very efficiently by Google. I hope to fix that later on (if I can find one or two more evenings ;-) )

In the process, I learnt to use a few things:

• Raphaël.js support for SVG paths. This allows you to draw complex shapes in a quite straightforward way using the SVG path syntax and there are nice animation effects available (the "morphing" effect from one population pyramid to another is only one line of code). As a side note, Raphaël is really a great tool because it fulfills today lots of the promises of HTML5 about vector graphics, even in Internet Explorer (from version 6) by using Mircrosoft VML where SVG is not available. The only downside is that it does not work on Android machines (but works very well on your iThings).
• the 960 grid system a CSS framework that allowed me to finally fix some of my gripes about CSS (the other ones should be fixed by the use of something like SASS).
• the use of cURL to scrape the content of web sites. I shoud definitely have known this before.
• the use of Google Web Fonts. I think a fair part of the reasonable visual appeal of the site is due to the typography;
• the set up of a custom domain name on top of GitHub pages. Super duper easy. By the way, thanks to the GitHub people for offering such awesome tools!

Publication

Once I got a sufficiently good version of the site, I published it at http://populationpyramid.net. I wanted some people to see it and maybe give me some feedback. After one Facebook wall post and a tweet that did not bring much traffic, I posted the link to Hacker News. I am a daily reader of this site about tech news and the startup world, and I thought that by insisting on the technological side of my endeavour, people there would be interested. I consequently chose the title "Population Pyramids of the World in SVG using Raphaël.js" and it worked quite well. Two hours after publication, PopulationPyramid.net was featured on the front page of Hacker News (here is the comment page). It reached around the 15th spot, which was much more than what I hoped for. Over two days, Hacker News brought around 1200 direct referred visits and around 1000 more through tweets of the different Hacker News feeds.

Next, although I am not a reader of this site, I posted the link to the programming section of Reddit thinking that it could interest people there (I realized later on that I maybe did not pick the right section, which could explain some downvoting there, although it seems to be a common practice on Reddit). There again, PopulationPyramid.net reached the front page, where it remained for around 24 hours, which brought around 1800 direct visits (here is the comment page ). I am quite happy with those numbers, even if honestly, I thought that those sites were able to bring more visits (but I only have a single data point here). That said, even if the traffic spike was nice, I mainly hope that the site will be used as a reference later on. My dream is to have a school using it as material for some homework for teenagers. If you know somebody that could be interested...

Conclusion

I loved to do this side project and I definitely recommend to any programmer wanting to improve to do this kind of one shot project to pick up a bunch of new technologies. Furthermore, I hope that this project can bring some value to people around the world to better understand the social problems we are currently facing. I chose not to add any comments to the figures because I lacked expertise and time, but also because I hope that people will take time to think about those images and express their own interpretations.

New Feature Ideas

As a final note, here is a list of features I would like to add someday:

• offering the ability to compare two pyramids;
• allowing to embed a pyramid in a distant site (so that people could blog about some pyramids);
• adding a curve showing the population size evolution;
• exploiting the wealth of other data available on the UN website. There is for sure something interesting to do with the available migration rates for example;
• add coordinate information when hovering over a data point;
• allow to download the different images as separate files (currently, they are generated in your browser on the fly). This would probably involve using a Raphaël serializer and would allow to create a site allowing progressive enhancements, solving the dependency on Javascript.

One point of the Django documentation that I did not get right at first is : how to get the ORM to generate a bit complex join queries. Yeah, sorry, I learnt SQL before ORMs, as many people, and I definitely think in terms of relational models and SQL joins.

For example, given the following models involving deliveries of products in a driver schedule, how do you get the list of products appearing in a schedule ? (I know that this sounds a lot like a Data Base 101 course).

class Product(models.Model):
name = models.CharField(max_length=150)
class Schedule(models.Model):
name = models.CharField(max_length=150)
class Delivery(models.Model):
.... some data fields here
product=models.ForeignKey(Product)
schedule=models.ForeignKey(Schedule)

The answer is deceptively simple, just write :

s = Schedule.objects.get(....)
Product.objects.filter(delivery__schedule = s).distinct()

which roughly will translate into the following SQL :

select distinct(product.*) from product
inner join delivery on product.id = delivery.product_id
inner join schedule on schedule.id = delivery.schedule_id
where schedule.id = some_id

The important thing to notice is that in the line

Product.objects.filter(delivery__schedule = s).distinct()

you can use delivery as the beginning of the lookup parameters, while there is no delivery field in the Product class. Django will understand.

What is maybe a bit misleading (even if it is quite logical if you think about it), is that there actually is a field named delivery_set created on the Product class (it is a RelatedManager), but you can not use it in the lookup parameters. It took me a long time to figure this out, so it might be the case for other people too....

That said, everything is explained in the Django doc here even if for once, I find the explanation a bit perfunctory

Disclaimer

This blog post is not very original, it is essentially a mix of two previous blog posts, one from Andrew Turner and one from Derek Willis.

And now, the real content

For my pet project, ZoFa, I wanted to allow my users to publish their schedule, computed on the site, to their favourite calendar application, particulary to Google Calendar, which is the calendar application I am using myself.

To do this, there are two steps to accomplish : First,create a view generating an ical file (with extension .ics) I used the vobject library to do this as showed here.

By the way, vobject is a part of the doomed Chandler project described in excellent book. Highly recommended.

Having battled quite some time with this, I think that the following cod snippet could be of interest to some people. So, here is the code of my view :

def ical(request, worker_id):
worker=Worker.objects.get(pk=int(worker_id))
cal = vobject.iCalendar()
cal.add(‘method’).value = ‘PUBLISH’ # IE/Outlook needs this
for shift in worker.shift_set.filter(published=True):
vevent = cal.add(‘vevent’)
vevent.add(‘dtstart’).value=shift.begin #begin and end are python datetime objects
vevent.add(‘dtend’).value=shift.end
vevent.add(‘summary’).value=shift.name
vevent.add(‘uid’).value=str(shift.id)
icalstream = cal.serialize()
response = HttpResponse(icalstream, mimetype=‘text/calendar’)
response[‘Filename’] = ‘shifts.ics’ # IE needs this
response[‘Content-Disposition’] = ‘attachment; filename=shifts.ics’
return response

Finally, just add the url to your urls.py :

url(r’^ical/(?P<worker_id>\d+)/shifts.ics$’,‘ical’,name=“shifts_ical”)

It seemed necessary for Google to pick it up to add a file name at the end of the url, but I am not sure about this.

The second step needs that your user import the url of your calendar into Google Calendar. To do this in the current interface :

1. open the “other calendars” little frame
2. click on “add” (at the bottom)
3. click “add by url” and paste the url.

Some concluding remarks :

• It can take some time before Google really requests your calendar and displays it
• The .ics file generated can also be used in Outlook, ICal or Thunderbird.
• Your url needs to be publicly available : there is no way to provide to Google Calendar a login and a password to your site.

The next step for me would be to automatize the addition to google calendar (with a nice “Add your schedule to Google Calendar” button) but I still have no idea about the feasability of this.

Last week, after two years of using and being delighted at the Django web development framework, I travelled to Berlin to meet the Django community at Djangocon Eu in Berlin.

Here are my impressions:

The attendance

People were very friendly, open and quite talkative, at the opposite of what I feared for such a technical conference. From my previous life in research, I am used to people brought together to present their research material, while not that much interested about others work. This was definitely not the case here.

I did not make any survey, but I think you could put most people in 3 big categories: web freelancers, web agency employees, and students . I did not meet any other person working like me in a company using Django for internal projects, although I think this is quite common. Most original occupation I encountered : a Swiss army lieutenant. And only men except for one exception...

The talks

The level of the talks was not necessarily always excellent, but again, coming from my experience of boring scientific talks beginning with an equation, I was quite enthusiastic. That said, this type of conference is suffering from the widely different levels of skills in the attendance that makes possible to get beginner-intermediate level talks (like Honza Král talk about testing) next to quite advanced talks (like Armin Ronacher talk about wsgi). Everything for everybody I guess, but also something inadequate for each one too.

My highlights are not very original : Jacob Kaplan-Moss Keynote, and Russel Keith-Magee talk about the Django development process. I liked those two presentations because they allowed me to get a hint of the opinions of key personalities in the Django community, while enjoying their very good presentation skills. They definitely made me want to contribue and gave me the feeling that the core developers of Django are constituting a well balanced team, where no one is the real star attracting all the attention (on the opposite of the Drupal or Ruby on Rails communities for example). The egos seem to be well managed. Furthermore, their attention to the human side of the problems is really interesting. I heard at leat thrice the motto "We must avoid technical solutions to people problems" coming out of their mouth, which I found quite significant. In my opinion, this makes for a really attractive community.

One final note on the talks topic : I know that NoSQL is the big buzzword currently and I can not decide if this is the case for good reasons but I am not sure that the attendance did enjoy so much time spent talking about it.

The organization

The organization, venue and food were all really great. Kudos to the organizers. As I have read on Twitter, those little tables with electrical plugs and excellent quality wireless network were really a plus.

Fun facts

• Like in rock concerts, it seems cool to wear t-shirts of previous editions, the older the better.
• There is a trend to make slides with no more than 5 words per slide, where almost all the space is devoted to huge pictures, possibly unrelated to the current topic and usually taken from Flickr
• Most people were multitasking to death : at the same time talking to their neighbours, tweeting, reading web sites (generally the ones cited by the speaker, but not always), posting pictures to Flickr. It enforced to me the image of the web developper who needs to be able to swallow tons of information daily, and to switch context instantly.
• Twitter is the second venue of the conference, where questions and answers are exchanged during and after the talks.
• I think there were more smartphones than people.

Conclusion

I am very happy that I did the effort to go to Berlin to meet like-minded people in real life. I came back with lots of subjects I want to dig in (Hudson, Class based views, South, and so much more)

In summary, even if I definitely think that those words are hugely overused in this community : Djangocon was AWESOME!. Djangocon For The Win !

P.S.: Remember that English is not my mother tongue. I will gratefully commit all grammar and spelling patches...