Happy @threadapalooza! 100 (tweet-sized chunks of) stories from trying to build mobile money in Ethiopia, Ghana, Nigeria, Senegal and Cote d’Ivoire.

(Minus the ones I’m not supposed to talk about :P)

It all starts in ~Sep 2015 with me pretending to be an accountant.

“Huh?” It turns out doing the accounting for an international money transfer business is hard. There is a normal way of doing this, but instead of trying to find out what that was, we were like, “this seems like it should be 100% automated! We’ll just do that.”

LOL

(This was part of our general take that hiring was for chumps, instead we’d scale ourselves by building internal tools)

So that’s how I ended up closing Wave’s 2015 books by spelunking through a homegrown database for 3mo with handwritten SQL and a half-baked Flask-Admin UI.

That wasn’t the worst part—that was spending ~50% of waking hours in Jan / Feb on the phone, explaining our poor life choices to our auditors

We ultimately passed the audit fine! Although IIRC we did get a footnote being like “you guys are idiots, do it the normal way next year”

Anyway, for reasons that I’m sure were completely unrelated, around the same time I broke up with my then-girlfriend, who happened to be engineer #1 (I was #2)

So I did what any reasonable person would and moved 10 timezones away, to work on our mobile money pilot in Ethiopia

IIRC the actual reasoning was that if I lived in Ethiopia, I’d have enough comparative advantage at product work that it couldn’t possibly make sense for me to work on accounting in 2016

(LOL)

Anyway, at that point we were mostly doing international money transfer (US -> Kenya)

Mobile money was a tiny pilot project the founders were running because they realized that international money transfer was limited by market size, because Kenya was the only country with a really good mobile money system.

OK, challenge accepted—we’ll just build the mobile money system ourselves so that international transfers will work better!

(LOL, “just”)

Anyway, I moved into the house the founders were renting in Addis Ababa and started working on that.

[ok, meetings starting, more later!]

Tbh I don’t remember much about my first trip to Addis. Our product ROYALLY sucked because:
• it was an Android app (penetration was low)
• built assuming good cell network (Ethiopia had ~the worst mobile data in the world)
So we mostly just flailed around trying to get users.

Mostly I remember the founders deciding they needed me to manage the eng team so they could focus on finding product market fit with mobile money.

Timezones made that hard (also, my incompetence at management) so I moved back to the US.

By the next time I moved back we’d completely overhauled the product to use USSD, an old-school protocol where you dial a bunch of digits and symbols, and get a pop-up message in response. This was awesome because it didn’t need mobile data, or a smartphone:

It was non-awesome in that it required a server in Addis that was VPN peered with the govt monopoly telecom, which went down ALL. THE. TIME.

We ended up building a “backup data center” in Jijiga, the capital of the Somali region, where we lived bc most of our users were there:

The “data center” is a closet just outside the frame :)

As you’d guess, it went down a lot too. “@.channel USSD is down” became the call of doom.

Our users would angrily call support demanding their $, so we built Slack tools for the support team to enter “manual transactions.”

Obviously we’d get a lot more support calls than normal so we’d effectively stop the entire company and get everyone on support. Here’s a manual transaction party during (+ caused by) a power outage.

(phone flashlight + water bottle = lantern—great power outage lifehack!)

The other fun thing we started doing at this time was letting people buy phone airtime with their Wave balance. This got tons of users bc it was the only way of topping up electronically.

You might think the telco would give us some kind of API for this.

HA HA HA

Actually, it was done by another computer in our “data center” (read: closet)...

...plugged into a bunch of USB cellular modems...

...imitating a person (or 10) sending airtime to their friends with their phone.

“How did you get the airtime onto the USB modems though?”

Glad you asked! There was no way to buy it in bulk electronically—only via scratch cards. So we’d buy huge batches of scratch cards and have a team of people manually scratching them off and topping up our airtime SIMs.

Every couple days, a SIM would trip the telco fraud rules (understandably) and get blocked with $X000 of airtime on it, so iirc it also ended up being a ~full time job navigating telco’s call center phone tree to get them unblocked.

[break for more meetings!]

Jigjiga, where the founders + local team lived, was a city of ~300k in the middle of a desert.

The best part was the people. Somali culture had a strong tradition of money transfer (that’s how we ended up there) so people REALLY loved Wave. Like, “sacrifice a goat” level loved.

Our local team was incredibly smart and dedicated. The image I always remember was our head of support, Osob, dual-wielding phones to do manual transactions faster. Or our “super agent”, Ahmed Sharif, taking his cash-laden motorcycle across the river to settle a faraway agent.

The first time I met Ahmed, he told me our customer support didn’t use enough data and suggested a bunch of (very smart) dashboard improvements.

He and the others we hired were amazing coworkers—which made it sobering to think about how underutilized many were in previous jobs.

This turned out to be a common theme with other local teams—lots of smart, energetic people with crappy jobs because their environment sucked.

There was not a lot of opportunity in Jigjiga. IIRC we guesstimated regional GDP/capita at $600. The regional president was an autocrat.

Outside the cities there was an early curfew because the region had problems with separatist rebels. We were told not to go hiking because there were paramilitary police camps atop many hills.

(Jigjiga proper was very safe—female colleagues said they felt safe alone at night)

Anyway, aside from the people, it wasn’t exactly a fun place. Food was particularly challenging, at least if you wanted vegetables. I remember coworkers ecstatic about getting ahold of some lettuce, then drying it by swinging a pillowcase because they couldn’t buy a salad spinner

Finally I discovered Meskel, the one Amharic restaurant in town, which had UNBELIEVABLY DELICIOUS shiro wot (ground chickpea stew). I’d go there once every few days with some massive tupperware, ask them to fill ‘er up, and reheat for 3 meals a day. Never got tired of it.

Anyway, we were growing really fast in Ethiopia, so we decided to try launching the same thing in Ghana. Exciting! Of course we were too cool for Accra, instead we went to where we thought people had the biggest money transfer problem: Tamale, a desert town in the north.

I decided to visit to help do the local integrations we needed for launch.

Temale sucked in exciting, different ways from Jigjiga. It was a *hot* desert climate and I visited in the hot season: average high ~100f / 37c. It also had THE WORST electrical grid.

Our house had ACs, but the grid usually ran at 170 volts. We got voltage regulators, which dropped like flies. Our household was 3 people:
• CEO, talking to users / running the company
• Me, full time programming in my underwear
• Ops coworker, full time fixing the AC/plumbing

Unfortunately we realized that we weren’t prepared to compete with MTN, Ghana’s biggest telco (they had their own mobile money and could effectively shut us down if we ever got big). So we cancelled our Ghana plans. Shortly after, we also had to stop in Ethiopia (long story).

That really, really sucked—things were going incredibly well and suddenly we had to rebuild from ~zero.

The first place we tried was Nigeria. As in Ghana, we skipped the capital and went for the northern region with more unbanked people—in this case Kano.

[break, more later!]

Kano (through no fault of its own) was definitely the low point of my mobile money adventure

1. We were all depressed from having to leave Ethiopia
2. Getting traction with mobile money was hard

It turned out that banks worked better in Nigeria than in most other places—

the central bank had mandated the implementation of instant inter-bank payments, so people used bank accounts kind of like they’d use mobile money elsewhere.

It still sucked because your closest branch might be hours away, but Wave wasn’t the same 10x improvement as in Ethiopia

But the real reason Kano sucked was the hotel.

We were so pumped about it! It was going to be so much better than Jigjiga! It had an Indian restaurant! With vegetables!

After a few months of 3 meals a day of Indian food and product not working though, it was... soul crushing.

My partner was on summer break staying with her parents in Warsaw, and I ended up bailing out of Nigeria earlier than planned to join her. I think her parents thought I was some kind of criminal at this point based on how I couldn’t predict which country I’d be in in 2 weeks

Anyway, soon after that Wave gave up on Nigeria and started working on Senegal instead. And things worked a lot better!

We ended up being popular with fish traders, since that’s a big industry in Senegal. The only disadvantage was it turns out cash carries scents really well...

and we were storing all the fish cash in a safe in our house. And that was also the cash we used for expenses (since banks sucked). Soon all our cash, and ourselves, smelled like fish.

It was worth it though because we finally had users again!!

[pausing again for a bit!]

OK, I realized I missed some silly things that happened in Ethiopia so I’m going to break chronology for a bit. Let’s see...

One was our antics trying to make our app work on bad Internet.

Even after we switched to USSD for customers, our agents used a smartphone app.

Unfortunately the cell network in Ethiopia was ludicrously bad, largely because the autocratic govt owned the single monopoly telco and they did. not. care.

When we first came to Jigjiga, internet only worked between ~11pm-7am. Too congested otherwise.

So during the day, we had meetings by using Skype to dial the founders’ cells long distance, paying ~$0.30/min.

It got better over time... in Jigjiga. In rural areas our app was near-unusable—think multi-minute loading spinners. Agents ended up with massive queues in their shops

You know what still worked though? SMS!!

Challenge accepted.

Some coworkers built a custom network layer that took our requests, jammed them into an itty-bitty SMS (~100 bytes), encrypted them, and sent them to our local SMS shortcode.

This was so ludicrously janky I still can’t believe it actually worked.

I remember we cut a corner and didn’t implement this protocol for login because it was more complex and “people don’t do it that often, it’ll be fine!”

HA HA HA

The next month I heard a story about a new agent who accidentally hit the logout button and had to drive 2h to the nearest town with mobile data so he could log back in

We’d immediately compensated for the better product by opening agents in places with even worse internet

SMS fallback fixed our worst-case latency, but it was still really slow, because nobody foresaw that some unhinged software engineers would one day use it as a replacement for the internet. We needed more.

Then we hit national i̶n̶t̶e̶r̶n̶e̶t̶ ̶s̶h̶u̶t̶d̶o̶w̶n̶ college exam week

“What?” Yeah so about the govt-owned telco...

• Every year Ethiopia runs a national college entrance exam
• ~Every year the answer key leaks and people start sharing it
• What’s a proportionate and reasonable way to prevent that?

SHUT DOWN THE WHOLE INTERNET!

(every year!)

Not the whole internet though, it seemed like Ethiopian sites still worked. Which meant DNS was still working, which meant... we checked...

All UDP port 53 traffic was allowed

Coworkers quickly hacked up another custom UDP transport layer, disguising our traffic as DNS.

(To be clear, this “disguise” was “three dogs stacked up in a trenchcoat” quality. iirc the internet filtering got more effective in future years & this no longer works.)

The new transport was also better because it cut out 2 packet roundtrips by optimizing the cryptography...

Basically, it was an incredibly shitty half-baked homegrown QUIC ( http://en.wikipedia.org/wiki/QUIC ).

The shittiest part was that we decided requests would never be over 1kb, and implementing sequencing seemed hard, so we just didn’t do it Fit your junk into one packet or die trying.

Second shittiest part was retry behavior. Theoretically, UDP packets can be lost, so we should have done something with acknowledgements and retransmissions.

In practice, that sounded hard so instead we just re-sent every request and response four times

We’ve kept using that UDP protocol ~unchanged since then! We’re just now killing it (it got hard to keep responses under 1kb, and data in SN is way better than ET).

[Wow long digression on networking! It’s near bedtime so I’ll have to write up the other fun bits tomorrow.]

[PS we’re far from done doing stuff like this—if you want stories like these of your own, we’re hiring all kinds of engineers + many other roles! DMs open ]