Finding an optimised keyboard layout for Malagasy [ongoing]

The current keyboard layout in use by most Malagasy language speakers puts whoever who wants to write in Malagasy at a huge disadvantage. It is impossible to write quickly in their language without stressing out their hand muscles. A typical malagasy sentence is quite often longer than a French one due to word length. Depending on the text sample, It may vary from 7% longer (compare the first 10 verses of the Chapter 1 of the Gospel of John) to 20% longer for more complex texts. A text that had required 10 hours to be written in French will take 11 up to 14 hours for Malagasy. At the scale of a company, or even a country, that is a huge waste of time, mostly due to a legacy that has lost all its relevance as keyboards do not have the same constraints as typewriters.
To tell you my story: since I’ve got my Samsung tablet, I’ve almost never used the default Samsung keyboard. So what did write my text messages with? I’m using my own keyboard layout; I’ll show you why and how.

A quick review on Malagasy uses

Before I get to the point, let’s see on what my fellow Malagasy citizens type their Malagasy language text with:

azerty.jpg
Fig. 1: AZERTY keyboard, made by French as an imitation of the American QWERTY

This, ladies and gentlemen, is the layout that is currently being used and known by most of the 24 million people in Madagascar. No need to say that their fellow citizen swho have emigrated to France also use it.
The problem is that layout is not suitable for Malagasy. At all.
heatkey.jpg
Fig.2: Heat map on an AZERTY keyboard used to type in Malagasy.

The heat map above has been generated using the Malagasy version of the Rainilaiarivony Wikipedia article. As a Wikimedia contributor, I’ve had the pleasure to type it… using the AZERTY keyboard. It was really a pain, and it looked like you did a lot of effort only to get less than the English version from which I had been translating.
azerty1.jpg
Fig. 3: In an AZERTY keyboard, when typing in Malagasy, your left pinky travels A LOT

That is also felt by my fellow citizens, a lot of whom have taken bad writing habits like writing SMS. That habit is sometimes taken to a new level, so that an unexperimented reader may find difficult or even impossible to read a text written in that SMS-style writing.
Even though most people browse the Web in French or English far more often than in Malagasy, using the QWERTY/AZERTY layouts is a pain, even if this is all we have, and even if this is what most people will know. Even if it’ll never have the success of the traditional layouts, I’ll give my two cents for a layout optimised for Malagasy language

Solutions

To palliate this strong disadvantage given to Malagasy regarding keyboard typing speed. I’d been using the German Neo keyboard layout. This was an already good alternative to the QWERTY which I’d been using for 4 years, but it was still sub-optimal, as my left little pinky is above a letter that is never used in Malagasy, my mother tongue.

neo
Fig. 4: German Neo Layout (see: neo-layout.org)

While looking for a solution to my problem I’ve discovered patorjk.com. From a given text, this website basically calculates which keys are most hit while the text is typed. From those keys’ position, a rating will be given. That rating takes into account for 1/3 the distance your finger had moved, how you use your fingers for 1/3 and how you often you have to switch fingers and hands while typing for 1/3. The higher the rating, the lower your hands will have to travel to type the text; so mechanically you’d be less tired typing the text in an optimal keyboard than in a standardised one.
So for our Rainilaiarivony text, there are the rating for the keyboards:
rating
Fig. 5: Layout ratings

The loser here is clearly the AZERTY, used by most of my fellow citizens. The standardised  Dvoraks are good candidates for typing Malagasy, and maybe we should consider those keyboards since they are widely supported in modern operating systems.
Here is what the programmer Dvorak looks like:
dvorak.jpg
Fig. 6: Programmer Dvorak Keyboard

Setting the Malagasy Optimised Layout

First version (7 November 2016)

The Dvorak score was impressive at the first sight, but the Dvorak was not the optimal layout for Malagasy. The one which the algorithm had found optimal was the following one:

malagasy1.jpg
Fig.7: Algorithmically generated Layout from patorjk.com (some keys’ positions have been frozen for more practicality)

That layout looks pretty decent but the keys are put in a little bit messy way. On the basis of that keyboard, the German Neo and the arrangement of a bunch of standard ergonomic keyboards I’ve come out to the following layout:
malagasy
Fig. 8: Own-made keyboard (the Malagasy Keyboard)

I’ve rerun the analysis on the same Rainilaiarivony article on that keyboard and a couple others. Here are the ratings:
ratings2
Fig. 9: Ratings of the Malagasy keyboard layout on the basis of the Malagasy version of the Rainilaiarivony

Well, to say the least, it looks like I’ve done way more than what the algorithm had succeeded to find. I’m pretty sure the layout I’ve designed is not very far from the perfect Malagasy-optimised Dvorak. Let’s go further into the report and see the row usage comparison.
row-usage.jpg
Fig. 10: Row usage comparison.

Yes, the AZERTY is an absolute typist horror when it comes to Malagasy.
The use rate of the home row for the our Malagasy keyboard is not very far from the optimal/personnalized layout generated by the algorithm.

version 2 (13 November 2016)

ergo1
Fig. 11: Hot keys on the second attempt.

Well, after a few day testing the keyboard layout I’ve got on the first attempt, I’ve felt some mandatory re-tuning of the optimised keyboard. That implied moving some keys to get the hot ones (the ones I have to hit most to type down my text) right under my index and my right middle finger. Since the left finger almost always type vowels, I’ve made them stay as most as possible at the home row unless you want to type some foreign words – in which case you’ll have some gymnastic to do.
ergo3.jpg
Fig.12: Finger usage of various keyboards.

As shown in fig. 12, the total number of hits in the Rainilaiarivony article is distributed as such: ~53% for the left hand and ~47% for the right hand. This excludes the thumb hitting the spacebar.
ergo4.jpg
Fig.13: Second attempt’s rating.

We’re getting better. Though the article is the same, I’ve switched to selecting the article from its HTML form. Since working on the article over and over again may constitute some bias, I’ve tried using some text samples from the Sarasara Tsy Ambaka.
I took quite a huge text sample (containing ~260,000 characters). It took a while to process but it takes out much of the bias related to the Rainilaiarivony article. The results still makes our Malagasy optimised keyboard the best layout ever to exist for the Malagasy language (cf. figure 14)
ergo5.jpg
Fig. 14: Layout ratings comparison.

I have to note that the calculated optimised layout gets closer and closer to the one I’ve designed, at least for the home row. Have a look:
ergo6.jpg
Fig. 15: The calculated layout. Looks a bit familiar, right?

As of this second version, we have an fairly optimised layout for Malagasy language, i.e. you’ll gradually type faster as your hand muscles get used to the new layout. Even for typing other languages such as French, this layout surpasses the AZERTY as the latter keyboard layout had been initially made to avoid the jamming of typewriters.

My conclusions

I may never say it much enough: the AZERTY keyboard is the absolute worst keyboard to type Malagasy with. Even the QWERTY does better. The Dvorak is a pretty good candidate for a widespread “more ergonomic” layout due to its presence in all modern widespread operating systems, but there is better.
Even if the French have designed the BÉPO layout for their language, it has failed to replace the omnipresent and inherited AZERTY slow layout. There is only one person I know who uses it on a daily basis. We also have to add to the fact that BÉPO has been around since 2008 and the Klavie Malagasy (“Malagasy Keyboard”) has only been written about just now, in 7th November 2016. As heavy as it is, the legacy left by AZERTY is highly likely to continue to be used in Madagascar probably for decades as long as keyboard typing exists, even if we relevantly know that the AZERTY layout is totally unsuitable to write French let alone Malagasy.
Right now I’m typing this article in English on a QWERTY keyboard. I’m planning to translate it to Malagasy as it gets more complete in order to reach more of the target audience.
I’ve already implemented that layout on my tablet so I’ve got all the time I need to adapt my fingers from the old Neo layout to the new Klavie Malagasy.

Updates

v2.1 as of 19 December 2017

Attached a PDF file containing the test corpus. A slightly better version has been proposed in the comments (thanks Ian!); and even though it has lower score than the v2.0, it has a really awesome idea of putting the T on the home row.
To better track all the changes, the project now has its own repository on Github. Long live open source!

Resources

Switching to Linux: good or bad choice?

Last updated on July 13, 2014
Do you want to switch to Linux? Before doing so, I invite you to reconsider all implied consequences of a switching to another operating system.
 
Linux? What is that?
But in the first place, what is Linux? It is the kernel of the GNU/Linux operating system. To be frank with you, «Linux» is a generic name for a few dozens of distributions having one thing in common: the Linux kernel. What is a kernel? It is a software that manages your hardware (motherboard, CPU, hard disk, networking, etc.) to make it work with applications you use. Current Microsoft Windows’ kernel is NT. By the past it also had MS-DOS which was the kernel used for Windows 1 up to Windows ME. I can write about this longer, but then we’d be off-topic.
So, Linux is an operating system, competing with Windows. It has to be known that Desktop computer market is the «final frontier» for Linux. All desktop computers nowadays come with Microsoft Windows pre-installed.
Because they use different kernels, Windows’ software will not work on Linux. There’s still a (poor) workaround for this problem, but I’ll talk about it later. This is also a blessing because Windows’ viruses can’t run on Linux whatsoever.
I’m not saying Linux is totally clean of viruses – because people have already created viruses that have successfully infected a Linux system – but still, with right reflexes, you’ll avoid most of problems. The most basic tip is to never run a Linux-based system as a root user, unless you know exactly what you’re doing. You can still run tasks requiring root privileges by using your own user password, but it will mostly happen when you install programmes.
 
Linux is Free
Primarily, Linux distributions can be used legally free of charge, by anyone. This means you don’t need to install an «anti-product activation » thing picked from a weird site, to use your operating system at will. The latter action, often performed by Windows users, is not only illegal, but can also compromise your security by letting that weird software from a weird site dig «holes» (backdoors) in your firewall. For people who like doing computer DIY, Linux is also open-source, developped by a community counting thousands of programmers an code reviewers. Have you found a bug in the software? You have the freedom to patch it and share your patch to other people. Yes, Linux licence allows this.
You also have a vast array of choices regarding distributions (commonly known as «distros»).
Linux distros are all built to do things in a certain way, so you have to think about what you’ll be doing with the OS, and then you download the distro that fits your needs. It is not like Windows, where you first install your OS, and then figure out what you need.
All distros (eleven) have their own software repository and desktop environment (DE) but they all have something in common: the Linux kernel, hence the generic name. By May 2014, the most recent version is 3.14 issued two months ago.
Something that discriminates each distro is at first sight their desktop environment, then the default software. Ubuntu itself has six desktop environments (Edubuntu, Kubuntu, Mythbuntu, Ubuntu Studio, Xubuntu, Lubuntu). Depending on your taste, you choose your DE: Unity has a very «modern» appearance; KDE is a very flexible desktop making it look almost like what you want it to (you can even rotate icons on the desktop!); LXDE offers a lightweight DE as well as XFCE. About updates, they are done through an update manager. Also, most of distro issue a new version every year.
 
The switch
So you’ve finally decided to switch. Your CD is burnt (or your USB key is configured), and you are going to shut down your PC. Please don’t do it yet, there are some matters to be thought about : do you use specific software for your videos? Do you play games? Have you some specific hardware for which installation requires a driver burnt on a CD?
To answer these questions, you’ll have to do some research on the Web. If you use frequently used software, then it is likely to find a free and/or open-source equivalent on some distro. If you use something like AutoCAD or Photoshop, then you’ll still find «free» equivalent of these on Linux, but they won’t always be as powerful. Furthermore, chances are that Photoshop format will not be compatible with their free equivalents.
About games, forget about playing Call of Duty, Battlefield or League of Legends on Ubuntu. The Steam Machine is on its way, so gaming will soon be possible and be more and more common on Linux.
If you cannot separate of your Windows software, there’s still a workaround: Wine. This piece of software allows you to run simple programmes on Linux. It is not guaranteed that everything will work on it, but still, it’s better than nothing. If you depend on a Windows-OS-only software to do your business, I advise you to dual-boot your computer. Then you’ll have and a Windows OS to run your software and a Linux distro to do your things as well. Note that Windows files can be accessed easily from Linux, when the opposite requires you to download software, and mount manually Linux partitions from that software. It is the way most people do when switching to Linux, avoiding all the inconvenience of having data requiring to be backed-up on another HDD.
Your hardware has come with an installation CD? The best way to proceed in this case is to check if the distro you’re going to install will support it.
Still, the best way to know if the distro you’ve chosen fits to the hardware is to boot using the CD which is most of the time a Live CD. Live CDs allow you to test the operating system on your computer without changing anything a single byte to the hard disk drive, as every required data is charged into memory. You can then choose to install the OS on your hard drive once you’re satisfied by the OS behaviour on your computer.
If you decide to switch, take the time to check if you’ve successfully backed-up all your data. We never know if something is going to fail, and to have twice the same data is always better than not having the data. If you can’t somehow migrate your data because you don’t have an external HDD, you can still choose to dual-boot your computer, so you’ll still have access to your data stored on the Windows NTFS (or FAT32, or FAT) partition. You can even choose to install your OS in an external HDD, if you need all the space on your computer HDD for your data. But to boot, do not forget to plug-in the USB key !
Usually, installation won’t take a long time. To install Kubuntu 12.04, I only needed 50 minutes to format the entire disk (500 GB) and get the PC ready for work.
 
My personal story
Because I got fed up by the inefficiency of my (free) anti-virus programme and by Trojans, key-loggers and root-kits compromising personal data security (my credit card number somehow leaked when I made an online purchase on a well-known financial transaction platform), I decided to make the big switch by changing the OS of my 4 year-old laptop computer to some Linux distribution.
Because I do care a lot about hardware support and user-friendliness, I’ve taken the decision to choose Kubuntu 12.04, first because it is a long-term support version (i.e. updates will be done on this OS for 5 years), and secondly because I am familiar and have positive experience with Ubuntu distros in terms of hardware support.
I made the switch a month ago by changing my laptop OS from Win7 to Kubuntu 12.04. The most annoying thing I’ve had to face since the switch is (still) hardware support. If your hardware is a little complex, crap happens quite a lot. Before definitely switching to Kubuntu, I tried Ubuntu (unity desktop), Mandriva (now OpenMandriva), Mint, Mageia and Debian. The latter three were unable to support my networking hardware, and (perhaps I have deficient research skills, but…) I found no workaround for it. Same problem for my printer. My connected printer refuses to do its job when I order it, which is quite frustrating to the average user.
When the switch has been complete, I noticed that Kubuntu – or at least the 12.04 version – has a serious memory leakage problem: kded4 process occupies more and more memory as time passes, and after a week of activity it ‘eats’ up to two gigabytes of memory. The PC then gets slower and slower, making it totally unusable so I’ve had to find a workaround to make the inflation cease. The price of this has been the inability to make the PC sleep, which reveals to be quite impractical, especially when you are working outside without an accessible plug to help your laptop keep the charge.
Even if Ubuntu support fairly well all the laptop’s hardware, some hardware problems still arise when you don’t expect them: I wanted to make an Ad-hoc connection to a friend’s laptop, but Kubuntu prevented me to do it because of kernel bugs. Also, a friend of mine had a Ubuntu 12.10 version and I was really astonished to see a so unstable Ubuntu version: random errors pop up every 10 minutes! I finally advised him to install another version.
Despite the lack of hardware support, switching to some Linux distribution is something great, especially when your hardware can’t support the latest Windows version. Also, for people who don’t want to invest tens of euros (or dollars) in an anti-virus solution, it is also a good choice.
Useful links