Finding an optimised keyboard layout for Malagasy [ongoing]

The current keyboard layout in use by most Malagasy language speakers puts whoever who wants to write in Malagasy at a huge disadvantage. It is impossible to write quickly in their language without stressing out their hand muscles. A typical malagasy sentence is quite often longer than a French one due to word length. Depending on the text sample, It may vary from 7% longer (compare the first 10 verses of the Chapter 1 of the Gospel of John) to 20% longer for more complex texts. A text that had required 10 hours to be written in French will take 11 up to 14 hours for Malagasy. At the scale of a company, or even a country, that is a huge waste of time, mostly due to a legacy that has lost all its relevance as keyboards do not have the same constraints as typewriters.
To tell you my story: since I’ve got my Samsung tablet, I’ve almost never used the default Samsung keyboard. So what did write my text messages with? I’m using my own keyboard layout; I’ll show you why and how.

A quick review on Malagasy uses

Before I get to the point, let’s see on what my fellow Malagasy citizens type their Malagasy language text with:

azerty.jpg
Fig. 1: AZERTY keyboard, made by French as an imitation of the American QWERTY

This, ladies and gentlemen, is the layout that is currently being used and known by most of the 24 million people in Madagascar. No need to say that their fellow citizen swho have emigrated to France also use it.
The problem is that layout is not suitable for Malagasy. At all.
heatkey.jpg
Fig.2: Heat map on an AZERTY keyboard used to type in Malagasy.

The heat map above has been generated using the Malagasy version of the Rainilaiarivony Wikipedia article. As a Wikimedia contributor, I’ve had the pleasure to type it… using the AZERTY keyboard. It was really a pain, and it looked like you did a lot of effort only to get less than the English version from which I had been translating.
azerty1.jpg
Fig. 3: In an AZERTY keyboard, when typing in Malagasy, your left pinky travels A LOT

That is also felt by my fellow citizens, a lot of whom have taken bad writing habits like writing SMS. That habit is sometimes taken to a new level, so that an unexperimented reader may find difficult or even impossible to read a text written in that SMS-style writing.
Even though most people browse the Web in French or English far more often than in Malagasy, using the QWERTY/AZERTY layouts is a pain, even if this is all we have, and even if this is what most people will know. Even if it’ll never have the success of the traditional layouts, I’ll give my two cents for a layout optimised for Malagasy language

Solutions

To palliate this strong disadvantage given to Malagasy regarding keyboard typing speed. I’d been using the German Neo keyboard layout. This was an already good alternative to the QWERTY which I’d been using for 4 years, but it was still sub-optimal, as my left little pinky is above a letter that is never used in Malagasy, my mother tongue.

neo
Fig. 4: German Neo Layout (see: neo-layout.org)

While looking for a solution to my problem I’ve discovered patorjk.com. From a given text, this website basically calculates which keys are most hit while the text is typed. From those keys’ position, a rating will be given. That rating takes into account for 1/3 the distance your finger had moved, how you use your fingers for 1/3 and how you often you have to switch fingers and hands while typing for 1/3. The higher the rating, the lower your hands will have to travel to type the text; so mechanically you’d be less tired typing the text in an optimal keyboard than in a standardised one.
So for our Rainilaiarivony text, there are the rating for the keyboards:
rating
Fig. 5: Layout ratings

The loser here is clearly the AZERTY, used by most of my fellow citizens. The standardised  Dvoraks are good candidates for typing Malagasy, and maybe we should consider those keyboards since they are widely supported in modern operating systems.
Here is what the programmer Dvorak looks like:
dvorak.jpg
Fig. 6: Programmer Dvorak Keyboard

Setting the Malagasy Optimised Layout

First version (7 November 2016)

The Dvorak score was impressive at the first sight, but the Dvorak was not the optimal layout for Malagasy. The one which the algorithm had found optimal was the following one:

malagasy1.jpg
Fig.7: Algorithmically generated Layout from patorjk.com (some keys’ positions have been frozen for more practicality)

That layout looks pretty decent but the keys are put in a little bit messy way. On the basis of that keyboard, the German Neo and the arrangement of a bunch of standard ergonomic keyboards I’ve come out to the following layout:
malagasy
Fig. 8: Own-made keyboard (the Malagasy Keyboard)

I’ve rerun the analysis on the same Rainilaiarivony article on that keyboard and a couple others. Here are the ratings:
ratings2
Fig. 9: Ratings of the Malagasy keyboard layout on the basis of the Malagasy version of the Rainilaiarivony

Well, to say the least, it looks like I’ve done way more than what the algorithm had succeeded to find. I’m pretty sure the layout I’ve designed is not very far from the perfect Malagasy-optimised Dvorak. Let’s go further into the report and see the row usage comparison.
row-usage.jpg
Fig. 10: Row usage comparison.

Yes, the AZERTY is an absolute typist horror when it comes to Malagasy.
The use rate of the home row for the our Malagasy keyboard is not very far from the optimal/personnalized layout generated by the algorithm.

version 2 (13 November 2016)

ergo1
Fig. 11: Hot keys on the second attempt.

Well, after a few day testing the keyboard layout I’ve got on the first attempt, I’ve felt some mandatory re-tuning of the optimised keyboard. That implied moving some keys to get the hot ones (the ones I have to hit most to type down my text) right under my index and my right middle finger. Since the left finger almost always type vowels, I’ve made them stay as most as possible at the home row unless you want to type some foreign words – in which case you’ll have some gymnastic to do.
ergo3.jpg
Fig.12: Finger usage of various keyboards.

As shown in fig. 12, the total number of hits in the Rainilaiarivony article is distributed as such: ~53% for the left hand and ~47% for the right hand. This excludes the thumb hitting the spacebar.
ergo4.jpg
Fig.13: Second attempt’s rating.

We’re getting better. Though the article is the same, I’ve switched to selecting the article from its HTML form. Since working on the article over and over again may constitute some bias, I’ve tried using some text samples from the Sarasara Tsy Ambaka.
I took quite a huge text sample (containing ~260,000 characters). It took a while to process but it takes out much of the bias related to the Rainilaiarivony article. The results still makes our Malagasy optimised keyboard the best layout ever to exist for the Malagasy language (cf. figure 14)
ergo5.jpg
Fig. 14: Layout ratings comparison.

I have to note that the calculated optimised layout gets closer and closer to the one I’ve designed, at least for the home row. Have a look:
ergo6.jpg
Fig. 15: The calculated layout. Looks a bit familiar, right?

As of this second version, we have an fairly optimised layout for Malagasy language, i.e. you’ll gradually type faster as your hand muscles get used to the new layout. Even for typing other languages such as French, this layout surpasses the AZERTY as the latter keyboard layout had been initially made to avoid the jamming of typewriters.

My conclusions

I may never say it much enough: the AZERTY keyboard is the absolute worst keyboard to type Malagasy with. Even the QWERTY does better. The Dvorak is a pretty good candidate for a widespread “more ergonomic” layout due to its presence in all modern widespread operating systems, but there is better.
Even if the French have designed the BÉPO layout for their language, it has failed to replace the omnipresent and inherited AZERTY slow layout. There is only one person I know who uses it on a daily basis. We also have to add to the fact that BÉPO has been around since 2008 and the Klavie Malagasy (“Malagasy Keyboard”) has only been written about just now, in 7th November 2016. As heavy as it is, the legacy left by AZERTY is highly likely to continue to be used in Madagascar probably for decades as long as keyboard typing exists, even if we relevantly know that the AZERTY layout is totally unsuitable to write French let alone Malagasy.
Right now I’m typing this article in English on a QWERTY keyboard. I’m planning to translate it to Malagasy as it gets more complete in order to reach more of the target audience.
I’ve already implemented that layout on my tablet so I’ve got all the time I need to adapt my fingers from the old Neo layout to the new Klavie Malagasy.

Updates

v2.1 as of 19 December 2017

Attached a PDF file containing the test corpus. A slightly better version has been proposed in the comments (thanks Ian!); and even though it has lower score than the v2.0, it has a really awesome idea of putting the T on the home row.
To better track all the changes, the project now has its own repository on Github. Long live open source!

Resources

Fomba fandraisan’anjara dimy amin’i Wikibolana

Ity lahatsoratra ity dia dikanteny amin’ny teny malagasin’ny lahatsoratra nosoratako tamin’ny teny anglisy vao andro vitsivitsy izay.
Mandray anjara tamin’i Wikibolana aho nanomboka tamin’ny taona 2010. Lasa fahazarana ilay izy: isam-bolana,isan-kerinandro, isan’andro, ary isa-maraina na isa-kariva, dia alefako ny mpitety tranonkala ijerena izay zavatra nitranga teo amin’i Wikibolana, ary mijery ny zavatra izay azoko atao mba hanampiana votoatiny be kokoa.
Tsindraindray dia tena manam-piniavana ny hanampy fampahalalana iray amin’ny pejy maro aho ka mandany ora maromaro manoratra fandaharana hanampiana izany amin’ny fomba faran’izay haingana.
Ary tsindraindray aho dia tena tsy tia handray anjara, ka mijery ny fiovana farany sy mijery ireo pejy izay mety nosimbaina aho, na mijery ireo pejy novain’ny mpikambana hafa.
Na dia izany aza, dia betsaka ireo fomba fandraisana anjara ao amin’i Wikibolana. Dimy amin’ireo no ho atolotra eto:
(1) Manoratra pejy amin’ny tanana. Zavatra mora indrindra atao na dia zavatra mahavizaka indrindra amin’ny voalaza aza. Manomboka nanoratra pejy amin’ny tanana (amin’ny alalan’ny fitendry) avokoa ny mpandray anjara rehetra, ary mety ho toa izay hatrany mandritra ny telopolo taona. Amin’ny 2045, dia ho lasa tola ny Wikipedia na i Wiktionary amin’ny endriny ankehitriny raha tsy efa manova ny votoatiny ho azy.
Alohan’ny hitrangan’izany dia ho betsaka dia betsaka ny asa atao. Na dia izany aza, dia afaka mampitombo ny habetsaky ny asa vitanao ianao amin’ny alalan’ny fianarana manoratra fandaharana. Rehefa hay tokoa izay dia:
(2) Manoratra fandaharana manoratra pejy izay mety ilaina ahitsy rehefa aty aoriana. Mora izany, ka niezaka tamin’izany aho nandritry ny telo taona. Rehefa mandeha ny fotoana dia betsaka ireo pejy voaforona, ka na dia kely aza ny taham-kadisoana dia lasa betsaka ireo pejy misy hadisoana. Ekena izany, fa betsaka noho izany ireo pejy tsy misy hadisoana. Rehefa ampiarahana amin’ny rakibolan-teny mitovy hevitra sy fahaiza-manodina teny voajanahary (Natural language processing) dia afaka mampamaritra teny tsy afaka dikaina amin’ny fiteny tanjona ianao.
(3) Manoratra fandaharana mamaky gazety ahitana ireo teny sy pejy tsy misy. Rehefa feno ny rakibolana dia mihasarotra hatrany ny fahitana teny vaovao hoforonina. Mety tsy hanam-piniavana ny hamaky lahatsora-gazety am-polony ianao, ka manorata fandaharana hamaky azy ireo ho anao ary maka ireo teny tsy ampy ho anao. Rehefa vita izany dia manorata fandaharana mitady ireo teny nakambana rehefa ary hanampy azy ireo ao amin’i Wikibolana. Ny lenta eo ambonin’izany karazam-pandaharana izany dia mpitady teny mivantana mamaky fahan-tsoratra avy ao amin’i Twitter ohatra, ary mametraka ireo teny rehetra tsy mbola voafaritra ao amin’i Wikibolana amin’ny faran’ny andro.
Zavatra iray ny mianartra manoratra fandaharana, fa zavatra roa samihafa ny fanampiana fampahalalana ary ny mahafantatra hoe fampahalalana inona no tsara ampiana. Rehefa mitsiry ny hevitra, na misy angona mikasika ny teny mahaliana eo am-pelantanana dia manorata fandaharana hametrahana ireo singam-pampahalalana ireo amin’i Wikibolana. Ataovy am-panajana ireo zom-pamorona izany.
(4) Mitety rakibolana ary manampy teny tsy fahita matetika. Mahaliana anao ve ny etimôlôjia? Am-pianarana teny vaovao ve ianao? Misy ao amin’i Wikibolana ve ireo teny ireo? Aza misalasala fa ampio ireo teny ireo. Atao am-panajana zom-pamorona foana izany. Azo heverina hoe asan’ny tena ny fakana teny maro avy amin’ny rakibolana maromaro, fa aza mandikadika foana ny famaritan-teny. Nanao izany aho ary saika voatory noho ny fitarainan’ny tompon’asa. Raha havanana amin’ny haranitan-tsaina voatrolombelona (AI) sy fahaiza-manodina teny voajanahary ianao dia manorata fandaharana mandika fehezanteny.
Mahery ny fandaharana. Betsaka ny fotoana ilaina amin’ny fanoratana fandaharana tsara, ka tsy vonona ny hianatra izany ny ankamaroan’ireo mpandray anjara, ka inna ny atao?
(5) Manorata amin’ny Wikibolana amin’ny teny nibeazanao. Raha atokantsika ny teny angisy dia soratana anaty fiteny 170 ny Wikibolana. Betsaka amin’ireo no manam-pejy latsaky ny iray hetsy. Vokatry ny finiavako ny hamorona ny rakibolana lehibe indrindra amin’ny teny malagasy ny haben’i Wikibolana malagasy amin’izao fotoana. Raha tsy teny anglisy no eny nibeazanao, dia mianara teny vahiny ary ampio ny teny ampiasainy, na ao amin’ny Wikibolana na aiza na aiza. Raha tsy mahaliana ana ny fianarana teny vahiny dia ampio ireo tenin-jatovo tsy mbola hita amin’ny teny nibeazanao.

Google translate now available in Malagasy

Good news, if it can be said, for my fellow Malagasy citizens: Since 6th of December 2014, Google Translate has been allowing them to see almost any web page in their mother tongue in addition to 89 others. Many people, myself included, have been waiting for this moment that would have come sooner or later. First of all, I would like to address a big thanks to all people that have made this possible. Thanks to you, the Malagasy language is getting further integrated into the polyglot Web world. You’ve also given a chance to the 15 million monolinguals to have an approximate understanding of what other people have written using other languages are writing.

Before Google Translate

Before we’ve got Google translate to translate almost anything in our language, including curse words, several websites have helped us Malagasy and other language enthusiasts to write corpora in a proper way in our mother tongue: many of us have already heard about Freelang, tenymalagasy.org and so on. The only drawback of these website is that they do not work in a collaborative way: they are not «crowdsourced». Wikibolana is a Malagasy language crowdsourced dictionary, but I have been so far the one that has generated most of its content.

Is it really that good?

Well, let’s be honest: absolute accuracy has been the motto for no machine translation system ever. But for a brand new language on Google Translate, Malagasy is… quite good. Daring to translate a language with such an unusual syntax like Malagasy is already a huge challenge, a challenge worth to be accepted. At first sight, idiomatic sentences and expressions are fairly well handled. Still when it comes to very complex sentences, it is a  mess: verbs are at the wrong place, which either gives the sentence a completely different meaning, or makes it look like an incomplete sentence. There are also some fails as the one in the screen shot below.

GTfail
“ahave” does not mean anything in Malagasy. But this is not the opinion of Google Translate

Let’s see an example of a translation of a paragraph of the article Madagascar in the English Wikipedia:

Original in English In 2012, the population of Madagascar was estimated at just over 22 million, 90 percent of whom live on less than two dollars per day. Malagasy and French are both official languages of the state. […] The island’s elephant birds, a family of endemic giant ratites, went extinct in 17th century or earlier, most probably due to human hunting of adult birds and poaching of their large eggs for food. Google-translated in Malagasy (as of December 2014) Tamin’ny 2012, ny mponina ao Madagasikara dia tombanana ho 22 tapitrisa mahery kely, 90 isan-jaton’ny izay [no] miaina amin’ny  [vola] latsaky ny roa dolara isan’andro. Malagasy sy Frantsay dia samy fiteny ofisialy ao amin’ny fanjakana. […] Ny nosy vorona ny elefanta, ny fianakaviana ny fizahantany ratites goavana, dia efa lany tamingana tamin’ny taonjato faha-17, na teo aloha, indrindra noho ny olona angamba ny olon-dehibe ny fihazana sy ny vorona lehibe Fihazana ny atodiny ho sakafo.  

The green-coloured sentences are syntactically correct without correction. The first one has required the red words in square brackets to sound correct. The third one hurt my brain: “The elephants are a bird island, the family of big tourists, have gone extinct in 17th century, or before, perhaps because of people, adults, hunting and adult birds who have their eggs hunted for food.” It hurt to understand, and also hurt to back-translate. Astonishingly making a round-trip translation has given a correct sentence in English, so please always have your translations checked human translators.

Efforts to be continued

One can take part to increase translation accuracy by translating articles by using the Google translator toolkit, or by using and correcting translations provided by Google translate itself.

African language Wikimedia projects summary

A few months ago I wrote an article which summarises my history on the Malagasy Wiktionary, and more generally my history on Malagasy language Wikimedia projects.
I am back here to write a short summary recapitulating the current progression of African language WMF projects. In this article you’ll learn about the current stage of African language projects and their trend.
In terms of community size, the biggest African-language community is the Afrikaans language Wikipedia community; followed by Egyptian Arabic speaking community and Swahili speaking community.
If we look closer to the statistics. The award goes to the Afrikaans language Wikipedia community which has 7 to 8 very active contributors (performing more than 100 edits per month).
The Egyptian Arabic Wikipedia community counts 2-3 very active contributors, which is big for an African language but very small comparing to Standard Arabic community counting more than twenty times more active users (83 very active users in June 2013), most of them being Egyptian contributors.
About Swahili, the number of very active users is one to two. On a 2-year term, this number can be averaged to 1. But the number of active users (i.e. making more than 5 edits per month) is 9 in average, which is a fine thing for a language that is spoken in countries where internet access is quite hard.
These numbers were obviously averaged from July 2011 to June 2013, so it smoothes short-term variations.
In terms of raw article size, the biggest African language Wikimedia project is the Malagasy Wiktionary – which currently counts 2.5 million articles, only smaller than English and bigger than French! – , the Malagasy Wikipedia (40,000+ articles) and the Yoruba Wikipedia (30,000+ articles), followed by the Afrikaans and the Swahili language Wikipedias (respectively 27,000+ and 25,000+ articles).
The Malagasy Wiktionary balecame very big for reasons you can read here, the Malagasy Wikipedia is big thanks to geography articles (~20,000 articles) and celestial objects (~8,000 articles); the Yoruba Wikipedia is made big by articles about people and also celestial objects (~15,000 objects).
Many Wikimedians who consult the statistics should know that the number of content pages does not determine the quality or the comprehensiveness of an encyclopedia. Judging wikis by article count is like judging a book by the appearance of its cover. And many book readers and critics know that looking at the cover is not enough to judge a novel. Here, by its raw size, the Malagasy language dominate in the two biggest projects (Wikipedia and Wiktionary) but that doesn’t mean it has a very active community.
To judge about the quality, comprehensiveness and completeness of the articles of such wikis, it is better to dive into this kind of statistics where scores are given by the absence/presence of vital articles and the size (number of characters) of such articles (if they exist). That kind of statistics are better than article count and page depth which can be inflated by the use of bot and the generation of tons of non-article pages (talk pages, subpages, redirects…).
According to the List of Wikipedias by sample of articles, the best scored African language Wikipedia is the Afrikaans Wikipedia, which ranks 58th and the Swahili Wikipedia (79th) followed by Egyptian Arabic, Yoruba and Somali Wikipedias. Malagasy Wikipedia is quite far behind and ranks 155th which is only higher than Lingala (161st), Wolof (175th) and Shona (187th) Wikipedias having less than 5,000 articles. Which means article count is only the cover of the book and thus some efforts have to be done there to make Malagasy Wikipedia more comprehensive.
What about the trend?
Less than a year ago, some Wikipedias found a way to grow in number of article thanks to species databases. The first ones I saw to grow this way are Winaray and Cebuano Wikipedias. Winaray Wikipedia gained 100,000 articles primarily thanks to low quality geography stubs (consisting in one or two sentences), and secondarily thanks to articles about species, animal and vegetal ones, making it to have 510,000 articles. Cebuano has more than decupled in article count within the last 50 weeks, from 40,000 to more than 500,000 articles. This mania of creating article about species has propagated to Swedish and Dutch Wikipedia which has recently surpassed the German Wikipedia, and in response to that, the latter Wikipedia seemed to have boycotted the Dutch Wikipedia, by deleting the link to the Dutch Wikipedia in the German language Wikipedia main page.
Now let’s write about the growth trend of African language Wikimedia projects. First off, let’s talk about Wikipedias, then Wiktionaries and finally other «minor» Wikimedia projects.

Wikipedia language edition

Current article count

Growth (in 300 days) (1)

Malagasy

40,619

+2,415

Yoruba

30,624

+582

Afrikaans

27,801

+3,928

Swahili

25,368

+1,232

Amharic

12,722

+1,015

Egyptian Arabic

10,764

+1,939

Somali

2,830

+383

Lingala

2,035

+118

Kinyarwanda

1,816

+7

Kabyle

1,517

+778

Wolof

1,172

+49

Kongo

826

+135

Northern Sotho

688

Igbo

739

+44

Zulu

586

+22

Setswana

496

–1

Bambara

392

+6

Siswati

368

+6

Ewe

302

+12

Hausa

291

+17

Oromo

276

+36

Tigrinya

259

+2

Tsonga

250

+7

Sango

204

+17

Kirundi

192

+8

Sesotho

189

+44

Akan

179

+17

Fulfude

166

+12

Luganda

166

–2

Twi

157

+12

Chamorro

157

+6

Xhosa

151

+10

(1) Calculated following this site, data retrieved in July 26th 2013.
On Wikipedia, the growth is slow comparing to other languages spoken in developped countries, where Internet access is easy and unexpensive to the normal citizen. The African language with the biggest community grows at approximately 5,000 articles per year, which is fairly high comparing to Swahili which growth is almost twice lower. If the current trend continues, the Afrikaans Wikipedia will surpass the Yoruba language Wikipedia next year, and the Malagasy Wikipedia in the next 2 years, as the two current biggest Wikipedias are stagnating in article growth.
On smaller Wikipedias, the trend is positive, though slow. All open Wikipedias have more than 100 articles.
The biggest of them is the Malagasy Wiktionary which has its growth kept by the use of Bot-Jagwar. Owned by myself, Bot-Jagwar runs from the Cloud, so it works regardless my computer and my internet connection’s healths. Thanks to it, the Malagasy Wiktionary gains 300 to 500 content pages daily. Automations eases many things in many ways, but automated processes can fail. So I have to keep an eye not only on the source code but also to entries generated thanks to that source code.
African language Wikipedias are slowly but surely gaining articles as time passes. There seems to be a moratorium in closing African language Wikipedias, and this is fine because languages mainly spoken in developping countries need time to develop a community. Furthermore, the official language in these countries, especially African ones, are very often not the local language.

Kurzweil Curve showing growth of computing power. It shows that all human brains can be simulated by 2050.
Kurzweil Curve showing growth of computing power. It shows that all human brains can be simulated by 2050. What about having billions of “virtual” contributors on Wikipedia in 2050? Source (kraxinglogic.com)

An increase of bot-made articles (which constitute nowadays 20% of articles created in Wikipedia) can indicate that in a near future, perhaps in 25 or 30 years, a bot will be able to write article like humans do. This is because Ray Kurzweil predicts the ability to simulate the human brain to be possible in twelve years and that current computers’ calculation power were supercomputers’ in the 1990s.
What about me? Well, it’s been a while since my last big article on the Malagasy Wikipedia. And according to the list of Wikipedias by sample of article, several hundreds of article needed in all Wikipedias are missing, so my first goal for Wikipedia is to fill these gaps, slowly but yet surely. I prefer contributing about geography, but as I am the only contributor of the Wiki, I have to fill gaps a bit everywhere : Biography, Chemistry, Sports, etc. At that pace, I can barely create three or four articles per day. At that pace, I can fill the 1,000 articles that every Wikipedia should have list whithin the year.
It’s been a while since the last time I blogged in Malagasy, So this article will be followed by a Malagasy language article. Perhaps a translation of this one, perhaps a new one.
Useful resources
To read further about what’s mentioned here.

  1. The law of Accelerating Returns by Kurzweil
  2. http://www.wikistatistics.net for all statistics about Wikimedia projects

[polldaddy poll=7298306]