In August 2020, and for the first time in my career, I made the decision to leave a job without having another one lined up. This was a highly unusual decision for me: all my previous moves had been planned and I always waited to have an offer in hand before giving notice. I’d previously seen friends and coworkers make the same decision and I felt uneasy about it, primarily for material reasons. I tend to be conservative when it comes to managing my personal finances and I don’t like the idea of suddenly cutting my main source of income without knowing when –or how– it would come back.

But for the most part, I’d seen this approach work, occasionally leading to great results. From experience, I knew interviewing with other companies while still on the job can quickly lead to feeling like you’re living a double life (and that the second life feels like no less work than the first one). The idea of having an open schedule to research, prep and to interview was alluring. Over time, I had warmed up to the idea that this might be something I could do myself.

Considering the uncertainty caused by the pandemic and by the upcoming US election, having to look for a new gig in the second half of 2020 wasn’t really a position I wanted to find myself in. Along with a few others, two factors principally contributed to my decision to quit before figuring out what would be next:

1. I was simply not happy with my position at Lyft and the environment that the company fostered: the pandemic hit the ride-sharing industry very hard, morale was low and my team suffered from it
2. My team was on the verge of embarking in a large long-term project and interviewing in parallel of that would mean not being able to properly honor these commitments

Once I was out of a job, I first took a bit of time off and then worked to get myself back into interviewing shape. This post unpacks my experience going through the interviewing wringer with about a dozen companies over the course of two months. I can’t say I was too surprised by what I found, but being able to compare a handful of companies over a relatively short time span painted an informative picture of what interviewing is like these days.

Acknowledgments

I have to acknowledge that the decision to quit my job bore less risk for me than for most. I have enough savings to stop working for a few months without being hit too hard, the employment market for software engineers looked strong despite the pandemic, and my spouse’s benefits allowed me to retain medical coverage. I don’t think I would have been in the position to do so had any of these conditions not been met.

I would like to credit Carlin Eng for inspiring me to write such a post. He went through a short halt in his career after leaving Strava and wrote an excellent article that unpacks how the experience went for him. I hope my post can serve as a slightly different take on the same topic: I wanted to go a little deeper than Carlin did on the topics of preparing for interviewing and what the final stages of the process can look like.

I also want to clarify that in spite of explaining my approach for engaging with interviews, this post is not an endorsement of the interviewing process itself. Our industry has, in my opinion, consistently failed at developing a method to vet candidates which is both effective and respectful of people. What I mean by “effective” here is: scalable in the moment and able to accurately predict the actual performance of engineers on the job. Google itself acknowledged that, in spite of spending tens of millions of dollars every year on moving candidates through the pipeline, stellar candidates can turn into very poor engineers. And we don’t hear of poor candidates that would have made great engineers—they were by definition left behind.

What I’m getting to here is that companies cynically focus on the process piece ("Does this candidate meet $BAR?") while candidates care about outcomes ("I’m a good engineer, I should be given an offer"). So much frustration stems from that disconnect. It took me about a decade to wrap my head around it and this post contains some of what I learned along the way. But I want to be clear: I’m playing this game in spite of its core premise being broken to some degree.

As a final disclaimer: what is written below is my personal take and experience on the interviewing and hiring process. One size doesn’t fit all and your mileage may vary. There is no single way to prepare for coding interviews, or to navigate a design a question, or to consider an offer. The approaches and points of view I’m sharing here are the ones I hold true, either because they stem from core beliefs or because I’ve seen them work as a candidate or interviewer. I believe there’s value in making the process less opaque, but please keep a healthy skepticism of what is written below.

On Hiring

By the time it is actively hiring, a company has entered a failure mode: there are needs in the face of which the current staff is unable to scale. In that light, I reason about the hiring process as a method to fail as gracefully as possible: just like a normal incident response, the process needs to be tight, coherent, snappy, and transparent. I began realizing the importance of building a good hiring process since I worked in a startup whose throughput depended on getting the right people. I carried it with me in my following gigs, and that’s something I pay a good amount of attention to, both as an interviewer and as a candidate.

For software engineering candidates, interviewing is vastly different task than what they are expected to do on their day job. You’d never be expected to have to code yourself out of a never-seen-before problem and ship it without writing any tests in under 20 minutes. You also wouldn’t crack out a full-fledged design for a scalable distributed system without doing research or experimentation to inform your proposal. Yet somehow, these are the skills you are expected to demonstrate during interviews.

What I’m getting to here is that neither side of the hiring process is effortless: each party should put some work into preparing for it. Part of it is theater, part of it is contrived, but being good at it doesn’t come naturally—at least not to me.

Preparation

Coding

Language

An important aspect of going into code interviews is to have some proficiency in a programming language. From my experience conducting interviews, I would actively discourage anyone from using a compiled language. In coding interviews, time is of the essence: running your code is something you will want to do very early and very often.

Compiled languages not only tend to be more verbose, time is also lost with the various type checking and assembly steps that the compiler needs to do every time you want to run the code. Scripting languages are definitely the way to go. A usual favorite is Python, but I myself have developed a strong personal preference for Ruby.

Ruby is full of one-liners and idioms that, once mastered, become very useful shortcuts to use during an interview:

• Keywords such as unless and common boolean functions such as .nil?, .empty? make the code easier to read and can allow an almost direct translation of the problem statement into code.
• attr_accessor and friends are great to define value classes in just a few lines, which is something you occasionally have to do.
• The built-in Hash class allows for default values: Hash.new { |hash, key| hash[key] = Array.new }, which is a very common task in interviews. Not having to handle this explicitly saves time.
• The entire Enumerable module is phenomenal when it comes to dealing with collections.
• The lack of return in methods make the code more succinct, albeit possibly less readable for non-Ruby programmers
• The chaining of functional calls can be really powerful to do semi-complex transforms and filtering in a bind:

['fig', 'pear', 'apple', 'flea', 'kea', 'rhea'].reject do |word|
    word.length <= 3
end.sort do |a, b| 
    a.length != b.length ? a.length <=> b.length : a <=> b
end.first

I wouldn’t recommend learning a new language just for the sake of interviewing; my own proficiency in Ruby stems from 4+ years writing production-grade code at Strava. But if you are on the fence, I do believe Ruby to be a great language for coding interviews.

On brainteasers

As Carlin pointed out in his post, Cracking the Coding Interview(CtCI) is a useful book to refer yourself to ahead of the actual interviews. But I also do not believe there is a ton of value trying to implement or memorize the solution to the hundreds of problems contained in the book. Some trivial problems found in the early pages can be useful to get yourself back into a mode where you have to deliver a working solution quickly and then fiddle with it with an extra constraint.

Other more complex ones may be getting frankly into the brainteaser territory, which, as far as I’m concerned, are not worth preparing for. What I qualify as a brainteaser is a problem that meets the following criteria:

1. there is an optimal solution that doesn’t lend itself to practical trade-offs
2. getting to the solution requires an insight or directed hints from the interviewer

The fewer hints you get, the better engineer you’re supposed to be. But if you Googled the question, you would likely find the one true solution in the first page of results.

It’s a bad sign if an interviewer is assessing you using problems that can be solved through Google — maybe it means candidates that passed are brilliant problem solvers who can think on their feet, or they might have memorized the solutions when they inevitably ran into the problem on Leetcode. What I’m looking for in a good coding problem is one of two things:

• a rough match with the specific domain that the candidate is expected to work in: build tooling stuff for build tools teams, infra stuff for infra teams, web stuff for web teams, etc…
• an open-ended question where there are trade-offs to make and therefore hopefully more than a single valid approach

Brainteasers just don’t make the cut.

What I’m getting to is that if you pass a brainteaser interview, you’re going to end up working with other people who have met the same bar. It might work out in the end — and historically, you might argue that it has worked for companies the likes of FAANGs. My point here is that the very questions you are asked during your interview offer an insight into how the company selects candidates, and what is valued in terms of skills.

Good programming interview questions are few and far apart — companies that spend time crafting a good bank of questions have an edge over those that don’t. Going off of stuff you find on Leetcode is lazy and provides a weak signal. For candidates, this should be a flag that company isn’t taking hiring seriously.

Algorithms and data structures

As far as preparing to answer coding problems, I forced myself to re-implement a number of commonly used data structures: hash tables, binary search trees, tries, linked lists, heaps, etc… This is the kind of stuff that you would normally always pull out of a standard library but going through those basics provided a playground to brush up my Ruby programming and a refresher on complexities. Once you have these, you can then work on higher-level structures and algorithms: expiring caches, traversal algorithms, bloom filters, tree re-balancing.

I then spent time working through abstractions such as iterators, comparators, streams, and, lastly, sorting algorithms. I did go through some Leetcode-like problems —brainteasers, to call them what they are— but I felt like they were less important to my preparation than most other things I decided to do on my own.

Non-technical skills

One value of CtCI is the preparation it forces you to do for non-technical questions. I’m not going to disclose the exact contents of the book but it fundamentally parallels the STAR framework:

• Situation: Describe the context of a project you contributed to or led.
• Task: What were you expected to do and what was the scope?
• Action: What did you actually do? Did you go beyond what was expected? Were there trade-offs along the way?
• Result: What was your impact? Don’t limit yourself to technical achievements

I came ready to discuss these things for 3 to 4 distinct projects or programs I personally worked on. To this, I also added a few follow-up questions that I’d be asking myself if I was the interviewer:

• With the gift of hindsight, what would I do differently?
• Some examples of situations where I was a responsible for communicating or coordinating across functions.
• Times where personal or technical conflicts arose and how they were handled.
• A recent example where I changed my mind about something.

Design

100% of the design questions I’ve worked on during my month of interviewing were distributed systems problems, which naturally lend themselves to trade-offs between performance, reliability and correctness. The way I went about preparing for this was two-fold:

1. Going over my own previous designs documents and working through the architecture and the decisions I made at the time. It’s helpful to look at those with a critical eye and attempt to find faults with certain approaches I took.
2. Reading up articles / explainers on the implementation of key pieces of software:
   • MySQL indexing and replication
   • Kafka (1, 2)
   • Envoy’s life of a request and support for load balancing
   • Redis
   • Cassandra
   • TCP — which might seem like a peculiar recommendation, but this is the base protocol that powers the Internet. It is not only fundamental but also packs a lot of good features that are often overlooked and re-implemented at a higher level of the stack (flow control! congestion control!).

Keep in mind that, ultimately, very, very few engineers in the world actually design and develop scalable distributed systems from scratch. Part of an engineer’s job is to know how to piece things together in order to meet the stated goals of the system being built. Knowing what those pieces are, and knowing how they work and what their limits are, that’s the thing you want to be able to demonstrate during a design interview.

Basically, it’s unlikely that an interviewer will grill a candidate on their full understanding of Paxos, but it’s good for the candidate to know how a leader election happens. A design interview is just as much an assessment of communication skills as it is of technical skills.

Process

My personal approach to answering design questions is to let my stream of consciousness take over: I openly share the thoughts that come to me, or the struggles I have about an approach. This gives opportunities for the interviewer to intervene, object, assist, or clarify a requirement. I’ve generally followed a 5-or-so step process:

1. Ask clarifying questions, or, alternatively, make statements that clarify the problem at hand: “it seems to be a read-heavy system, so I’ll make sure to to center my design around that”. This is especially important for parts that appear trivial to you, since it’ll help the interviewer know that you are thinking this through and not glossing over things.
2. Top-down first: decompose the problem in large chunks that each have a coherent role, e.g. “I’ll need a queue, a load balancer and some storage”. Establish the scale for each of those, the interfaces and the invariants that need to remain true between them. If no obvious architectural choices come to you, just talk out loud about your thinking process.
3. Bottom-up second: run through the set of components you have identified and make implementation proposals for each of them. Explain out loud what your thinking is and the rationale for those decisions.
4. Do a run through of the entire system once and see if the goals are met. Identify bottlenecks or failure modes. Discuss how you would counter them or solve for them. Sometimes, the interviewer will challenge the design or change a parameter that feels like the rug is being pulled from under your feet. This can make you feel like you design suddenly has to cover more ground but it is generally an invitation to discuss trade-offs: no real-world system is perfect, and your design cannot be perfect either.
5. Open things up: if you were writing a true technical specification, there would be a ton of other considerations you’d need to cover: ease of operations, observability, configurability, security, resilience, deployment, etc… If you have the time, talking about those can yield a solid conversation to wrap things up and convey that you care about more than the base design.

Company Selection

I didn’t really have a positive filter for the profile of companies I considered. I did however prune out companies and domains early on: this is ultimately a fairly personal choice but it might explain how I ended up with the final slate of companies.

• Facebook was a no-go from the beginning: their product and leadership team have enabled fascism to take root in the country I live in. They thrive in supplying polarizing content and spreading lies to their users.
• I avoided the crypto industry as a whole, mostly because I’m not interested in it but also because of the kind of toxicity that festers in circles seeking to profit from sole disruption. With the gift of hindsight, the recent madness around NFTs and web3 has put the final nail in that coffin.
• Even though most companies in the Bay Area have called for extended periods where employees are working from home, I wanted to favor a strong presence in San Francisco, which is where I live. This eliminated a number of big targets: Apple, Google, Netflix to name a few.

Other than that, I tend to favor companies whose product I use every day, e.g. the ones whose app has a place on my phone’s home screen. I made myself available on LinkedIn and began looking for postings on the websites of said companies.

Timeline

I overall engaged with a total of 12 companies, across a 10-week period.

• I directly applied to 8 companies, out of which 3 didn’t answer at all (Netlify, GitLab, 1Password), 2 were quick dead-ends (Local Laboratories, Substack), and 3 led to interviews
• I responded to 2 sourcers who reached out on LinkedIn, both of which led to interviews
• I was referred into 2 companies (GitHub and Slack), both of which fizzled out quickly

For the companies I directly applied to, I was intent to write a cover letter — even though I had the intuition they would be dismissed or discarded by screeners, it was a forcing function for me to investigate the company, to dig into job description and write down why I think I’d be a match for the role. This was more for my benefit than for theirs.

In total, I was concurrently pursuing 5 leads throughout the month of October. Since I don’t feel like there is value in naming them, I will from this point on refer to them using greek letters: α, β, γ, δ, ε. And since my initial selection was fairly open, I ended up with a fairly varied set of company profiles: headcounts ranging from a couple dozen of employees to many thousands, valuations going from $100M to many tens of billions, and a healthy mix of private and public companies.

Beyond searching for a job, I spent the rest of my days working on personal projects, playing games, running errands, going on runs or on bike rides. Following Carlin’s example, I tried to set aside a day a week to go outside on a ride and empty my head. In spite of this, I didn’t feel quite enough in control of the whole process to really be at ease.

Some companies I was excited about were entirely non-responsive and I had to make peace with that. The process dragged on with certain companies while others were really snappy. Offers that came with deadlines meant having to rush things — which is good to an extent, but also incurs a higher level of stress.

The end-to-end timeline ran from the tail-end of August to the beginning of November (actually, election week 😬). I kicked it off with a couple of companies which I applied to in the couple of weeks right after leaving Lyft. I resumed applying in mid-September and sent applications or responded to a handful of companies each week.

In addition to sending my resume around, I started going through my interview training program. There wasn’t a ton of responses at first but things warmed up once October came around.

Candidate Experience

Overall, the interviewing process went pretty well across the board, with the notable exception of one company where the day of onsite went poorly: company α.

Interview formats

I was happily surprised to see new interview formats introduced since the last time I went though the process:

• Three companies have adopted a format where two interviewers are present on all or most of the calls, presumably to avoid situations where a candidate feels cornered (or discriminated against) and isolated in the presence of a single interviewer. This mostly worked well, especially when both interviewers made a effort to create a three-way conversation. At other times, the third person on the call remained silent for the entire interview, which instead triggered a feeling of being put under extra scrutiny.
• Ahead of the “onsite”, one company asked me to prepare a 20 minute presentation on a past project of my choosing, which the interviewers then used as a basis for asking follow-up questions. This ended being a great interview, since I felt like I was given a chance to showcase my best work.
• Rather than doing a coding challenge, one company offered to pair me with an engineer for a full day of (compensated) work where I went through the engineering onboarding process, ran the stack locally and worked through a coding problem. I was also given the chance to attend the company all-hands.

Diversity

I identify as the absolute dominant profile in the north american software industry: white, cisgender, straight, and male. I’ve also been on the organizing side of DEI initiatives in companies and products I worked on. I wanted to pay attention to this as I was going through my interview process and it’s no surprise: no company is really doing this well.

As I went through the “onsites”, I tried to keep a running tally of the profiles of people who were interviewing me: their role and seniority, but also their apparent gender and ethnicity. A major caveat here is that I never explicitly asked my interviewers how they identify in terms of gender identity or racial background, which means the data presented below is tainted by my own perception and bias.

I was interviewed by 35 people in total, and the panels were all between 6 and 8 people. I’ve opted against presenting a per-company breakdown in this post because I believe the samples are too small to be meaningful. As a whole, however, the data paints a relevant picture:

1. Bar for one, all my panels had at least one female interviewer. Two thirds of my interviewers were male.
2. All panels had at at least one non-white person, sometimes just a single one. Two of the five companies had people of color in positions of leadership. Only one black person interviewed me.
3. One interviewer shared that they were involved in their LGBT ERG

Two caveats are worth mentioning: in some cases, a single interviewer was wearing multiple of those hats, and sometimes those interviewers were in non-technical roles (e.g. designer or product manager). The numbers would be grimmer if I focused solely on engineering roles.

Similarly to the questions they chose to ask, the makeup of an interview panel can clue candidates as to what is valued at the company. Three of the five companies had me interview with non-engineers, and another one had three of my onsite interviews grill me extensively about cross-functional collaboration and project management. This is great! Companies should absolutely vet an engineer’s experience working with less technical partners.

Notably, company α’s panel was solely comprised of men, all of them engineers. I was only asked technical questions, a few of these being brainteasers. In the moment, I was honestly frustrated that I didn’t do better. In hindsight, and despite the hiring manager’s reassurance that the company fostered a very healthy culture, I didn’t have a ton of regret about flunking.

Logistics

The pandemic threw a wrench in many processes, and the interviewing one is no exception. By the time I started to interview, I had worked from home since the sheltering in place had begun almost six months prior. For companies, onsites are an opportunity to showcase their offices and perks. Meeting some prospective teammates in person and casually chatting over lunch is part of the experience as well.

All of that is gone in 2020. It makes for a less appealing day, but also requires a bit more preparation for candidates — things that coordinators would normally be in charge of. Here’s what was helpful for me:

• A cell phone plan with plenty of minutes, as I expected having to receive and place a lot of calls. Not having to bother about overages is really nice.
• A good Bluetooth headset with a built-in mic is great for both phone calls and VC
• A water bottle refilled in full before each interview, especially during the “onsite”
• Some snacks to refuel between interviews

Companies also need to change their ways — notably, it becomes harder for candidates and interviewers to exchange non-verbal cues. This means communication skills become increasingly important for both. The one notable negative experience I had was a design interview where both interviewers kept their webcams turned off, which obviously made the interview a lot harder for me. Putting candidates at ease by doing proper introduction also becomes more important, and increases the chances that candidates will perform their best. Interviewers that cut the introduction short without expressing the slightest interest in what I had to say were a clear red flag for me.

Offers

Comparison

It’s already hard enough to precisely know what lies within a single offer, and the fit for the role and with the folks also plays a huge part in a decision that is deeply personal. In my case, 4 of the 5 companies I interviewed with ended up extending an offer. The basic components are still the same: base pay, equity, bonus and benefits. There’s no tried and true method for comparing those uniformly, especially as the cost of equity can differ significantly:

• ISOs have a cost but are eligible to be taxed as capital gains once you hold them for over 1 year — but the value might fluctuate in the meantime.
• RSUs are ‘free’ — they’re essentially income given to you as stock — but are taxed as such, at a much higher rate than ISOs.

Equity compensation is a complex, potentially thorny problem which warrants knowing what you get into before you sign a contract. The Holloway guide to equity compensation is a free, comprehensive document about equity compensation that I cannot recommend enough.

Below is the framework I came up with to compare offers:

• Yearly Base is the cash part of the offer, expressed as a fraction of the average of all yearly bases.
• Equity Value is the current presumed value of the full equity grant, as a fraction of the base. Grants across all offers have a 4-year vesting schedule.
• Equity Cost is the yearly cost of acquiring the equity, i.e. the number of options that vest in a year multiplied by the strike price, which is relevant for ISOs.
• Sign-on is a one-time bonus, expressed as a fraction of the base
• Annual Bonus is a yearly target for bonus, expressed as a fraction of the base

Equity

My philosophy in terms of equity is two-fold:

• As far as offers and compensation packages go, equity is almost the least important component of the lot: companies can do well and they can also do terrible, sometimes for reasons entirely out of their control. Equity is the cherry on top of the cake. And even though the cherry can sometimes become its own standalone cake, the stars need to align for that to happen.
• In spite of treating equity as second class income, I’ve consistently chosen to purchase my initial grant of options: the premise is that I wouldn’t join a company if I didn’t believe its capital would eventually turn up. This requires budgeting for the purchase of the options, which makes a dent in the yearly base.

Equipped with this framework, let’s look at company ε: the base pay is the highest of the lot but 18.7% of it needs to go into purchasing equity each year, which is also quite high. In the case of company γ, the equity more than doubles the base pay year after year, which comes at no direct cost except for the associated tax burden, which is high because RSUs are taxed as income.

One offer should clearly stand out in the table above: company β is a small, early-stage company. The quoted base (which was presented as non-negotiable) was lower than my previous one at Lyft, and it significantly drags down the average presented in the table. To make up for it, the cost of acquiring the equity is fairly low (7.1% of the base) and the stock has a lot of room to grow since the company is so young. This company also has no sign-on, no bonus structure and the parental leave is unspecified at this time.

Benefits

Benefits are where comparing offers gets a lot harder, primarily because there’s so much ground to cover and what matters to people is so personal. Some companies offer stipends for gym memberships, internet access reimbursement, lump sums for buying office equipment, etc… Health coverage can be such a personal decision that I’m not even going to begin trying to pitch policies against one another. I focused on three aspects of benefit packages:

• PTO: it’s very common for companies these days to offer untracked PTO, which is then presented as “unlimited”. This is a problem for two reasons:
  1. Studies have shown that people with “unlimited” PTO end up taking less time-off than when PTO was capped.
  2. Untracked PTO is not actually paid. Capped PTO accrues and has value which is owed to you in case you do not take it by the time to you leave.
• Parental leave: While I personally do not have kids, I recognize parental leave as a staple of a good benefits package. The smallest company I received an offer from didn’t have a parental leave policy and claimed this was something they will get to eventually, and that folks have taken time-off ad-hoc at the time they became parents.
• 401K matching: all companies run 401K programs, but the ones that match contributions tend to be few and far apart. I’ve always maxed out my 401K contributions — matching is essentially free pre-tax money going from the company to my future self. Some companies such as Google match 100% of the contribution up to the federal limit, while others set lower caps and rates.

Role fit

While the compensation package is definitely a factor, I’ve primarily made my decision based on how excited I was about the role. As can be seen in the timeline, company δ took well over two weeks to extend an offer after giving me the thumbs up. It was apparent that it took time for the hiring team to come up with a compensation package, which was not a good sign: compensation packages should be an easy thing for a company to figure out and companies should strive to work on the candidate’s schedule.

The other reason for this delay is that I had several follow-up conversations with prospective managers in order to determine team fit. Even though I originally applied for a TLM position, it was clear that the expected people management aspect of that role was going to be too heavy for my taste. I was oriented towards a different team and talked with another manager, then with an engineer on that team, until another switch was made to have me report to a higher-level manager at the very last minute of the process.

It also became evident that the organization and reporting structure was a work in progress, which made it hard to have a consistent conversation about the role, the context of the work and expectations. In contrast, the other companies I interviewed with were clearer as to what they were looking for, and they crafted the process around that goal. From my perspective, the exchanges with those companies were more consistent and productive.

Parting Notes

Upon making the decision to quit, I knew I would need to keep an open mind about the path this job search would lead me on. I’d never been in a situation where I interviewed with so many companies at once, which turned out to be a fairly involved process. It required some level of work pretty much every day over the course of two months. This is about the average workload I originally anticipated, but I don’t think I expected how fragmented this time would be, and how unevenly distributed it would be.

I try not to let it get to me too much, but I was raised with the idea that work —and, by extension, employment— serves not just a social purpose but also a personal one. In other words, labor is expected to fulfill individuals and idleness is good only in small quantities. As such, the decision to go into unemployment voluntarily and for an unknown length of time didn’t only have financial or logistical consequences. To an extent, having a day job sort of alleviates the pressure of having to answer the age-old question: “Am I doing the right thing with my time?”.

I felt varying degrees of guilt from the moment I last closed my Lyft laptop and until my signature was at the bottom of an offer letter. This feeling was only reinforced by the fact that my spouse was in the room next door, working her ass off every day.

There was also a broader context to this: on top of a pandemic that had brutally changed our day-to-day for over half a year, the US election was set to be a very disputed one, with more in the balance than any other in my lifetime. Working through a deeply personal process at a time where so much was at stake on a global scale felt quite odd — a feeling that was sometimes openly shared by my interviewers.

Outcome

🎉 I joined Stripe’s Payments API team in January of 2021 🎉

Based on this outcome, I can’t look back and say that quitting and making room for job searching was a bad decision: Stripe was on the shortlist of companies I was excited about when I began my search.

But it’s also clear that the whole process didn’t feel great and I can’t say precisely how I would change things if I had do to it again. I’m happy about where I landed but I came out of it spent, and I’m wary that this isn’t avoidable. The guilt I experienced is also something that’s too personal for me to just shake off.

The two takeaways I have from this experience are:

1. Working with several companies in parallel made me more empowered to ask questions that got me a better grasp of what the company really needed: knowing what will be expected out of my work made me a lot more at ease when signing an offer
2. Anyone who’s gone through the tech interviewing wringer knows too well that this is a nerve-racking process: multiplying this five-ways makes only makes sense when your time is clear from other distractions

For me, in hindsight, part of the value was in exploring the variety of roles I could see myself in: TLM, staff engineering or founding engineer of an org. It’s one thing to daydream about making such moves and another to have real people trusting you to act in that capacity in real, critical projects they want to ship. I came out of it with a refined sense of what I like to do and a chance to match that with a company’s needs. Which is about as graceful as a failure can be.

Luckily, the language itself is actually fairly simple and well documented and the official tutorials are well-structured. I wholeheartedly recommend A Tour of Go followed by Effective Go — reading through these should take about a day or so and give you a sense of what the language can do for you.

But not everything I needed to do was trivial and I was taken aback more than once spending hours trying to get to something working when I felt like it should have taken minutes. Other times, I was actually thrilled to see lesser-known capabilities of the language shine and allow myself to be as expressive as I needed to be. This post is an ongoing collection of those times, and what I’ve learned from them.

Build Constraints

Build constraints are directives to the Go tooling to either include or exclude a given file. There exists a set of default tags but you may also define your own custom tags:

// +build foo

Tags may be OR’d, AND’d and NOT’d:

// +build foo,!bar baz
// +build qux

This evaluates to ((foo AND NOT bar) OR baz) AND qux — in other words, if someone tries to run any Go tool and sets a -tags=bar,qux, the file in the example above will not be considered.

Tags apply to most of the Go toolchain: the compiler (build), but also the testing and the lint tooling. Files will be included by the tooling if their tag directive(s) match the tags currently active in the build. By default, Go has a set of tags that let you match the host OS, the architecture of the CPU, the current version of Go, etc… Any tool that iterates over Go files will have a way to specify the tags you would like to include.

Tags are useful to define category of files that you want to include conditionally — for instance, you may want have a tag for // +build integration tests — since they are generally slower to run and might require a specific setup. As an example, let’s say you have code accessing S3 that you need to write a test for: clients/s3/s3_test.go

// +build integration

package s3

import (
    "testing"
)

func TestS3(t *testing.T) {
    // Actual testing code here
}

Simply running go test ./... would not include the integration test by default, since the constraint is not set. To run the integration test, you would need to run:

go test -tags=integration ./clients/s3/

Field Tags

When you define a struct type, the fields declared in it may be annotated with tags, which are declared as a string literal that follows the field declaration:

type Foo struct {
    bar int "this is a tag"
    baz string `tag declared with backticks in order to use a literal "`
} 

Those tags become available at runtime, through the reflect package. They are represented by the reflect.StructTag type. This type implements a convention that a tag to specify multiple values, each keyed by a name and enclosed in quotes: `foo:"bar" baz:"qux"` is tag with two keys: foo and baz.

Tagging fields is relevant in the context where a struct is a model that may be serialized in and out of the Go runtime. In such cases, the tag for a given field can contain a pseudo configuration for how that field should be written or read. The list of well-known struct tags is informative, and links to popular serialization formats such as JSON and XML. But certain ORMs and cloud providers have also opted for an approach that uses tags to map Go fields to column names.

Tags allow programmers to define the external shape of a type at once and across a range of formats. The directives present in each tags' value can be as complex as the serialization scheme allows. The DynamoDB client is such an example:

type EmailThread struct {
    Labels []string `dynamodbav:"labels,omitemptyelem,stringset"`
    Title string `dynamodbav:"title"`
    LastEmailSentAt time.Time `dynamodbav:"lastEmailSentAt,unixtime"`
} 

Enumerations

This is a head scratcher: there is no strongly-typed enumerations in Go. Enum-like types should be defined as type aliases for int and their values declared as consts whose first value is iota. iota is a special variable in the Go language:

• its default value is zero
• it is auto-incremented for every line in a const block
• uninitialized constants declared within that const block will default to the current value of iota
• its value is reset to zero when exiting the const block

For a language seeking to reduce confusion and surprises, this is immensely confusing and surprising. Practically, you define an enum in the following way:

type HTTPMethod int

const (
    Get HTTPMethod = iota
    Post
    Put
    Delete
)

Since the Go type-system doesn’t provide first-level support for enumerated types, the names of the constants are lost in the compilation. In other words, you must write custom code to get the name an enumerated value at runtime. This is commonly done by defining a String() function:

func (m HTTPMethod) String() string {
    return []string{"GET", "POST", "PUT", "DELETE"}[m]
}

Note that the order of the strings in the array must be in the same order as the one in which the individual constant declarations is done, and there is no compile-time guarantee that you will not go out of bounds, which would happen if you defined a new value in the enumeration but forget to add it to the array of names.

JSON

The encoding/json package encapsulates the logic for serializing structs into JSON and deserializing JSON into structs. It’s a reflection-based engine that will do a decent job by default and that may be customized using field tags. In case you need full control of how struct is serialized, you need to implement custom interfaces.

A good example of such a need is enumerations: if they are defined used iota values, a value will by default be encoded as an integer. If you prefer to have a string, you can make the type implement the Marshaller interface, which has a single MarshalJSON function:

type HTTPMethod int

const (
    Get HTTPMethod = iota
    Post
    Put
    Delete
)

func (m HTTPMethod) String() string {
    return methods[m]
}

var methods = []string{"GET", "POST", "PUT", "DELETE"}
var methodsByName = map[string]HTTPMethod{
    "GET": Get,
    "POST": Post,
    "PUT": Put,
    "DELETE": Delete,
}

func (m HTTPMethod) MarshalJSON() ([]byte, error) {
    return json.Marshal(m.String())
}

Conversely, you would need to implement the Unmarshaler interface to convert the string back into the enumerated value — you might want to refactor the previously defined String() method to share the string constants between the marshalling and unmarshalling functions

func (m *HTTPMethod) UnmarshalJSON(b []byte) error {
    var s string
    if err := json.Unmarshal(b, &s); err != nil {
        return err
    }
    value, ok := methodsByName[s]
    if !ok {
        return errors.Errorf("Unable to unmarshal HTTPMethod value %s", s)
    }
    *m = value
    return nil
}

Time formatting

This is just a complete head scratcher and there’s no way around it except biting the bullet: Go’s built-in time formatting doesn’t use traditional format strings such as YYYY-MM-DD but instead chose an unambiguous fixed point in time that you have to write your desired format in. That point in time is January 2nd 2006, 3:04pm and 5 seconds, in the GMT -7 timezone. Which, written another way is: 01/02 03:04:05PM '06 -0700 — 1, 2, 3, 4…

So, date formats are actually written as a plain date, e.g. ISO 8601 datetime is: 2006-01-02T15:04:05. Ref

Decorating an interface implementation

Go’s type system allows for declaring structs that wrap an interface implementation and decorate specific methods. There’s no special syntax for it — it works just like interface composition. Let’s say you have a naive in-memory database that stores models that have an identifier and a name:

type Model struct {
    Id uint64
    Name string
}

type Database interface {
    Create(name string) *Model
    Save(model *Model)
    Get(id uint64) *Model
}

type database struct {
    models map[uint64]*Model
}

func (db *database) Create(name string) *Model {
    id := rand.Uint64()
    db.models[id] = &Model{Id: id, Name: name}
    return db.models[id]
}

func (db *database) Save(model *Model) {
    db.models[model.Id] = model
}

func (db *database) Get(id uint64) *Model {
    return db.models[id]
}

func NewDatabase() Database {
  return &database{models: make(map[uint64]*Model)}
}

The Database interface is public, and as such may be relied-upon by external code. Now let’s say you want to track the “popularity” of a certain model, say, by the number of times it is accessed by Get. Your options are:

1. Modify the database implementation to add a counter - but any other implementation of the Database interface will not inherit that behavior.
2. Implement your own Database, which is not great because there are two methods out of three you don’t care about at all

The third option is to decorate the Database interface:

type trackingDecorator struct {
    Database // wrap an arbitrary implementation of Database 
    getCounter map[string]int32
}

// Override the method you care about
func (td *trackingDecorator) Get(id uint64) *Model {
    model := td.Database.Get(id)
    if model != nil {
        td.getCounter[model.Name] += 1
    }
    return model
}

func NewTrackingDecorator(wrapped Database) Database {
    return &popularityTrackingDatabase{
        Database:     wrapped,
        getCounter:    make(map[string]int32),
    }
}

Declaring the Database interface as an anonymous field of the trackingDecorator struct will automatically make the latter implement the interface. Upon construction, you will need to supply an actual implementation you want wrapped:

• By default, all interface method calls are forwarded to the wrapped instance
• If the new struct declares any method of the interface, that method will be called instead

Most importantly: in the second case, the methods on the wrapped instance can be called from the overridden code e.g. td.Database.Get(id) in the example above. This lets you decorate select methods of arbitrary interfaces, even if you can’t access the underlying implementation.

Testing

GoMock .Do()

When you use GoMock, you can use custom matchers in order to assert on the correctness of the arguments received by your mocked functions. Matchers are nice but they need to be reusable to prove their worth. There are certain cases when you don’t know (or don’t care) about the full argument, e.g. you just want to make sure that a dictionary containing headers has the one key-value pair you care about, or that a URL contains just the one param you need.

GoMock’s .Do is designed for cases like this: it lets you capture the parameters passed to the mocked function and run arbitrary assertions on them. Say you have a raw client for a “TaskQueue” and a class wrapping that client to provide convenience functions to enqueue messages of a certain type to certain routes:

// This is the mocked interface - not going into the details of what a Task contains but assume
// that it is a complex object with lots of fields and implementation details.
type TaskScheduler interface {
    ScheduleTask(ctx context.Context, task *Task) (*ScheduleResponse, error)
}

func TestEnqueueMyTask(t *testing.T) {
    t.Parallel()
    ctrl := gomock.NewController(t)
    defer ctrl.Finish()

    ts := taskScheduler.NewMockTaskScheduler(ctrl)
    // Pass `gomock.Any()` in lieu of asserting on a fully built Task
    client.EXPECT().ScheduleTask(gomock.Any(), gomock.Any()).
        Return(&tq.ScheduleResponse{}, nil).
        // The signature of the function passed to `.Do` should be the same as `ScheduleTask`
        // but we don't care about the context
        Do(func(_ context.Context, task *Task) {
            // You can run assertions on just the parts of Task that you care about
            assert.Equal(t, tq.Method_PUT, task.Payload.Method)
            // etc…
    })

    controller := MyController(scheduler: ts)
    err := controller.MethodThatSchedulesATaks(context.Background(), …)
    assert.NoError(t, err)
}

Skipping tests

It’s useful every now and then to disable a test without wanting to delete the code. You can use Skip or Skipf to do that:

func TestProductionRequestEndToEnd(t *testing.T) {
    t.Skipf("Disabled because of the refactoring in FOO-539")
    ctx := context.Background()
    tc := getTestConfig()
    …
}

This is better than renaming the request or commenting the code out because the test runner will print a message every time the test is run, reminding you that it should be re-enabled at some later point in time.

Running a specific test

The go test command has a -run flag which may be used to specify a specific test method to run, but the actual syntax is incredibly subdocumented. You need to pass in the directory in which the test is, not just the test name:

go test -run=TestFoo ./path/to/you/test/file/ -v -count=1

-count=1 will disable test caching, which would otherwise occur when your source files do not require rebuilding. As per the doc: “The idiomatic way to disable test caching explicitly is to use -count=1” 🤷.

Compiling tests without running them

The Go test tooling does not decouple the compilation of tests from their execution. In other words, the philosophy is that you either want to run your tests or… run your tests – and there’s no way to just check that your tests compile, which is the first step before they can be run.

It turns out the -run flag discussed above doesn’t actually just take the name of a test, but a regular expression that matches the names of the test functions to run. If you set that expression to something that cannot match any function, the testing framework will stop just after the compilation step. The most common trick is to use a line beginning (^) immediately followed by a line ending ($)

go test -run=^$ ./...

Here be dragons: the testing framework will not execute any tests but it will execute code that belongs in package initializers: should this code have any side-effects, this command would trigger them.

Subtests

Go’s standard test framework notably lacks the concepts of setup and teardown phases, which in other languages allow for, well, setting up the context of multiple tests and disposing of those resources once the tests have run.

Not all hope is lost though: t.Run() is a method intended to isolate the behavior to test in a dedicated block. Because you can nest them, you can progressively specialize The structure of the code can resemble Ruby’s RSpec

func TestBowling(t *testing.T) {
    bowling := Bowling{}

    t.Run("When there are no strikes or spare", func(t *testing.T) {
        bowling.Frame(3, 2)
        assert.Equal(t, 5, bowling.Score())
    })

    t.Run("When there is a spare", func(t *testing.T) {
        bowling.Frame(7, 3)
        assert.Equal(t, 10, bowling.Score())

        t.Run("Followed by a regular frame", func(t *testing.T) {
            bowling.Frame(5, 4)
            assert.Equal(t, 24, bowling.Score())
        })
    })
}

As you can sort of guess, the problem with this approach is that all tests will share the same instance under test — if that object is stateful, you will end with tests whose sequencing cannot change. This is a pretty big anti-pattern in general, but especially in Go where parallelism is built into the test framework.

t.Run() can still prove useful to break down long test methods into chunks that are easier to digest and help with making common scenarios clearer for the next maintainer.

Precommit

When I get into a new language or stack, I try to write a pre-commit script with the goal of catching problems and correcting them before the code is pushed to a remote branch (I know about pre-commit but personally find it cumbersome). This is the Ruby script I have for my Go projects and here’s what it does:

• Runs go fmt, goimports, go vet and golangci-lint run --fix on all new and modified files
• Checks whether new files have an associated _test.go file
• Check for calls to fmt.Print* – those should likely be logs statements instead
• Generates files that need it
• Runs all the tests
• Builds the integration tests: files with integration or endtoend constraints, which would not be picked up by go test

#!/usr/bin/env ruby

class String
  # colorization
  def colorize(color_code)
    "\e[#{color_code}m#{self}\e[0m"
  end

  def red
    colorize(31)
  end

  def green
    colorize(32)
  end

  def yellow
    colorize(33)
  end
end

require 'pathname'
require 'set'

def generated_file?(file_path)
  !File.open(file_path).read().match(/^\/\/ Code generated .* DO NOT EDIT\.$/).nil?
end

def needs_generation?(file_path)
  !File.open(file_path).read().match(/^\/\/go:generate /).nil?
end

def test_file?(file_path)
  !file_path.match(/_test\.go$/).nil?
end

def abort_unless(command, failure_message='Failure', success_message='Success')
  print command and STDOUT.flush
  output = `#{command} 2>&1`
  if !$?.success?
    puts "  \u{1F44A}  #{failure_message}".red
    STDERR.puts output.red
    abort("\u{1F44A}  run: #{command}")
  else
    puts "  \u{1F44C}  #{success_message}".green
  end
end

class CommandResult
  def initialize(command, output, failure, failure_message='Failure', success_message='Success')
    @command = command
    @output = output
    @failure = failure
    @failure_message = failure_message
    @success_message = success_message
  end

  def result
    print @command and STDOUT.flush
    if @failure
      puts "  \u{1F44A}  #{@failure_message}".red
      STDERR.puts @output.red
      abort("\u{1F44A}  run: #{@command}")
    else
      puts "  \u{1F44C}  #{@success_message}".green
    end
  end
end

def abort_unless_async(command, failure_message='Failure', success_message='Success')
  output = `#{command} 2>&1`
  CommandResult.new(command, output, !$?.success?, failure_message, success_message)
end

def run_go_fmt(files)
  puts "Running go fmt…"
  files.each do |file|
    abort_unless("go fmt #{file}")
  end
  # Eagerly re-add all files, as they may have been modified
  `git add #{files.join(' ')}` unless files.empty?
end

def run_go_imports(files)
  puts "Running goimports…"
  abort_unless("goimports -w #{files.join(' ')}")
  # Eagerly re-add all files, as they may have been modified
  `git add #{files.join(' ')}` unless files.empty?
end

def run_lint(files)
  puts "Running lint"
  abort_unless("golangci-lint run --fix")
  # Eagerly re-add all files, as they may have been modified
  `git add #{files.join(' ')}` unless files.empty?
end

def run_go_generate(go_files)
  puts "Running go generate…"
  # threads = go_files.map { |go_file| Thread.new { abort_unless_async("go generate #{go_file}") } }
  # threads.each do |thread|
  #   thread.value.result
  # end
  go_files.each do |go_file|
    abort_unless("go generate #{go_file}")
  end
  generated_files = `git diff --name-only --diff-filter=AM | grep '\.go$'`.split.select { |file| generated_file?(file) }
  # Re-add all generated files
  `git add #{generated_files.join(' ')}` unless generated_files.empty?
end

def run_go_vet(go_files)
  puts "Running go vet…"
  go_files.each do |file|
    abort_unless("go vet #{file}")
  end
end

def check_tests_for_new_files(new_go_application_files)
  puts "Checking for tests…"
  new_go_application_files.each do |new_go_application_file|
    target_test_file = "#{new_go_application_file.gsub(/\.go$/, '_test.go')}"
    STDERR.puts "#{new_go_application_file} was added in this change but is missing a matching test (#{target_test_file})".yellow unless Pathname.new(target_test_file).exist?
  end
end

def check_fmt_print(go_files)
  puts "Checking for fmt.Print…"
  abort_unless("! grep -Hn 'fmt\.Print' #{go_files.join(' ')}")
end

def run_tests()
  puts 'Running tests…'
  abort_unless("go test ./...")
end

def build_integration_tests()
  puts 'Building integration tests'
  abort_unless("go test -run=^$ -tags 'integration endtoend' ./...")
end

new_go_files = `git diff --cached --name-only --diff-filter=A | grep '\.go$'`.split
new_go_application_files, new_go_test_files = new_go_files.partition { |file| not test_file?(file) }
new_go_application_files, new_go_generated_files = new_go_application_files.partition { |file| not generated_file?(file) }
go_files = `git diff --cached --name-only --diff-filter=ACM | grep '\.go$'`.split
go_application_files, go_test_files = go_files.partition { |file| not test_file?(file) }
go_application_files, go_generated_files = go_application_files.partition { |file| not generated_file?(file) }

def all_go_files()
  # Recursively list all go files in current package and emit lines in the format
  # /absolute/path/to/module/directory:[file.go otherfile.go]
  modules_and_files = `go list -find -f '{{.Dir}}:{{.GoFiles}}' ./...`.split("\n")
  git_root = `git rev-parse --show-toplevel`.strip + '/'
  all_go_files = []
  modules_and_files.each do |module_and_files|
    # Split module path and files
    module_dir, files = module_and_files.split(':')
    # Remove git root directory from absolute path of module
    module_dir = module_dir[git_root.length..-1]
    module_dir = (module_dir.nil? or module_dir.empty?) ? "" : "#{module_dir}/"
    files[1..-2].split.each do |file|
      all_go_files << "#{module_dir}#{file}"
    end
  end
  all_go_files
end

unless go_files.empty?
  run_go_fmt(go_application_files + go_test_files)
  run_go_imports(go_application_files + go_test_files)
  check_fmt_print(go_application_files + go_test_files)
  run_go_vet(go_application_files + go_test_files)
  run_lint(go_application_files + go_test_files)
  check_tests_for_new_files(new_go_application_files)
  run_go_generate(all_go_files().select { |file| needs_generation?(file) })
  run_tests()
  build_integration_tests()
  puts "\u{1F389}  All good!".green
  exit(0)
end

AWS is ubiquitous in the infrastructure world but an application developer like me can mostly get by without understanding its intricacies and subtleties. Ironically, engineers that work at some of the largest Internet companies in the world (e.g. Google, Microsoft, Facebook) may also find themselves lost when having to rely on public cloud offerings, because their experience lies with homegrown technologies.

The AWS reference documentation is well-written and plentiful, but lacks a narrative that takes the reader from basic to simple to slightly more advanced concepts. This post is the result of my own learning experience and is seeking to be a pragmatic, progressive introduction to AWS. In other words, this document is selective as to the technologies it covers and (lightly) opinionated. Feedback is welcome.

Regions

A region is a real-world area in the world where AWS servers are co-located. Regions have a broad geographical location which is reflected in their name, e.g. “ap-south-1” (Asia Pacific South) or “us-west-2” (West Coast of the United States). Regions allow you to deploy your application to a location physically close to where your users are, with the goal of reducing the network roundtrip. The list of AWS regions can be found here.

Availability Zones

An availability zone (abbreviated AZ) is a physical subdivision of a region that has its own power source and network links to the public Internet, each of which are 100% independent from other AZs in the same region. As their name indicates AZs are intended to provide resilience within a region: running your application across multiple AZs in a region should allow for your users to still use it when one of them goes offline.

Even though they are physically independent, you should be able to make network requests across AZs without incurring a performance penalty: AZs are in close proximity and are interconnected by dedicated high-throughput, low-latency network links. Regions provide between 2 and 5 AZs each.

EC2

An Elastic Cloud Computing (EC2) instance is a virtual machine which provides core resources such CPU, RAM, mass storage and networking. It is the base unit for running application code on the AWS platform. Unlike physical servers, EC2 instances are ephemeral: once it is shut down, the same instance cannot be turned back on and, if needed, a distinct instance must be provisioned to replace it.

Types

EC2 instances have a type, which determines the amounts and attributes of each resource. Over time, the number of instance types has grown to cover a wide range of use cases. The nomenclature is fairly confusing and intricate, so you need to reason in terms of what the application code you need to run is going to need:

• For general purpose applications: A, T, and M types
• CPU-intensive applications: C types
• Memory-intensive applications: R, X, and z types
• Applications requiring GPUs: P, G, Inf, and F types
• Storage-intensive applications: I, D, and H types

Comparing EC2 instances should be a science, and it turns out to be more of an art. As is the case when you provision a machine in the real world, there is a ton of flexibility and tradeoffs to consider. At a high-level, you should first determine the primary bottleneck of your application and chose an appropriate letter-type, then benchmark it across a range of actual types to figure out determine which is right.

CPU

Not all EC2 instance classes run on the same CPU, so AWS uses vCPU as a mean to compare them — a vCPU generally corresponds to having exclusive access to a single thread of the virtual CPU. But there are exceptions to this rule: for instance, the T2 type provides a physical core as opposed to a single thread. More variables also need to be considered:

• The brand and the class of the CPU varies across instance types, e.g. AMD EPYC or Intel Xeon
• The model of the CPU is not the same across instances either: Xeon E5 2686 v4, Intel Xeon Platinum,…
• For some instance types, AWS reasons in terms of CPU capabilities, e.g. Intel AVX or Intel Turbo.
• AWS recently introduced Graviton, which uses an entirely different architecture (ARM).

Mass storage

As noted earlier, an instance that was shut down cannot be brought back up. This is specifically a concern if the instance has stored data locally that needs to be persisted past its termination. The top-level dichotomy in terms of mass storage is EBS v. local instance store:

• Certain instance types only support Elastic Block Store (EBS). EBS is AWS' offering for a network store, i.e. it allows persisting data independently of the instance it is attached to. Some instance types are optimized for EBS usage, which means the throughput of the network link will be guaranteed (the specific value of the throughput may be different across instance types).
• Certain instance types will come with built-in local storage, which can be either spinning disks or SSD.

Provisioning and Pricing

The default mode of provisioning EC2 instances is called “on-demand” and is the most intuitive one. The instance will be provisioned on the fly and will run until it is shut down. The cost is based on the amount of the instance was running: the amount of computing resources available is linear with the cost a customer is willing to pay.

Reserved instances provide the same range of options as the on-demand offering, but offer a lower cost in return for a lasting commitment: the customer declares a need for a given number of instances for up to 3 years. By being able to anticipate demand, AWS can deliver the supply more efficiently, and pass down the savings.

Spot is a different mode of provisioning, where the instances that are not reserved and not used on-demand are made available at a discounted price to workloads that are either stateless or highly fault-tolerant.

Spot instances run at a lower priority that on-demand or reserved instances, so that when a request is made for a number of spot instances, the request may be only partially fulfilled if there isn’t enough supply. Conversely, AWS reserves the right to reclaim Spot instances if they are needed to fulfill on-demand requests. This is why Spot instances make sense for workloads that can be interrupted rapidly: in exchange for a decreased availability, Spot instances come at a very large discount compared to either of the other modes.

These provisioning modes are not exclusive to one another: it’s actually a common practice to “pad” a pool of reserved instances with on-demand or spot instances when the application is confronted with a spike that couldn’t be anticipated, e.g. large one-off migrations.

AMI

An Amazon Machine Image (AMI) is a data bundle, which packs the software required to launch an EC2 instance. Several operating system vendors (e.g. Microsoft, RedHat, Ubuntu) publish their operating systems as AMIs, which makes it trivial to use on AWS. Some AMIs may be subject to licensing restrictions or need to be purchased directly from the vendor.

It is common practice to build custom AMIs, which allows for the application to be installed and baked into the image ahead of it running in production. The resulting image may be kept private in case it includes proprietary software.

The typical build process of an AMI consists in provisioning an EC2 instance with a public AMI to use as a base, install any custom software needed at runtime and finally register the resulting volume as an AMI. The process is different depending on whether you intend to use the AMI to image an EBS volume or a local instance store.

Launch Configuration and Templates

Provisioning a single EC2 instance is a tedious task, as you need to pick a wide range of parameters: region, AZ, type, AMI are required parameters, but the number of options and parameters can be tedious to specify every time. Since launching EC2 instances is something that can happen several times a day even for medium-sized setups, there is a clear need for that process to be automated.

Launch configurations capture the set of parameters needed to launch an instance. Launch templates serve the same purpose as launch configurations but are more capable in that they can be versioned. Modifying a template creates a new version where parameters may be changed, added, or removed. You can then launch EC2 instances from any version, but only one version can be the default one at a given time. Templates are the preferred way to launch EC2 instances in Auto-Scaling Groups.

Auto-Scaling Group

An Auto-Scaling Group (ASG) is tasked with managing a set of EC2 instances, with the goal of adjusting the number of instances up and down, as configured by a scaling policy. ASG have the following properties and constraints:

• ASGs require having the full set of parameters needed to launch new EC2 instances at any given time.
• ASGs must also be configured with an upper and lower bound of instances to keep around at a given time.
• ASGs may manage instances across multiple Availability Zones but not across multiple Regions.

The scaling policy of an ASG determines when and how the ASG will grow or shrink the pool of EC2 instances it manages. The simplest policy is a static one, where the the upper and lower bound are equal: in this case, the ASG will simply monitor the health of instances and bring up a new instance only in case a previous one was deemed unhealthy. Beyond this simple use case, scaling policies can be configured in a variety of ways:

• Time-based scheduling allows adjusting the number of instances at a set time, which is useful when the change in demand is predictable
• Built-in metrics such as CPU utilization can directly feed into the scaling policy, or you can specify any metric AWS has access to
• Policies define by how much to adjust the size of the pool, either as an absolute change or relative to the current healthy count
• Cooldown settings allow to avoid scaling down too fast and finding yourself in a situation where the ASG oscillates between scale-ups and scale-downs

Lastly, be aware that multiple scaling policies can be active within a single ASG at once. When they conflict, AWS honors the ones that keeps capacity the highest.

Elastic Load Balancing

An Elastic Load Balancer (ELB) is tasked with fronting and dispatching network traffic. An ELB dispatches traffic to targets in a target group, which in the most common case is an ASG. An ELB can also distribute traffic to a set of IP addresses, or to AWS Lambda functions. ELBs can operate at two levels of the networking model:

• At the transport level (OSI 4), individual TCP or UDP socket connections will be streamed to the backend targets: ELBs of this kind are called Network Load Balancers (NLB)
• At the application level (OSI 7), incoming HTTP requests are parsed and individually distributed to the backend targets: ELBs of this kind are called Application Load Balancers (ALB) and are more featureful than NLBs. They can be configured to route traffic to a specific group based on the data within an HTTP request: path, header value, query parameter, etc…

Because it specializes in handling and routing network traffic, an ELB can augment an ASG in a couple of ways:

• By measuring the inbound traffic and allowing the ASG to use it as a metric to drive a scaling policy
• By performing instance health checks that use the network, e.g. simple ICMP pings or HTTP probes

Lastly, the routing logic of an ELB can be configured in a few ways:

• How long to let a new target warmup for, or how long it should be given to process the requests in flight after being de-registered
• Stickiness parameters between clients and targets: an ALB can be made to consistently connect clients to the same target
• The method for picking a target in a group, e.g. round-robin or least-loaded target in the group

Lastly, and importantly, an ELB may be reachable from the public Internet but its IP shouldn’t be expected to remain static over time. All ELBs will be provisioned with a publicly resolvable DNS name (*.elb.amazonaws.com), which will remain constant throughout the lifespan of the ELB. This record can be used directly or can be CNAME’d to a more user-friendly form (api.example.com → api-1234567890.ap-south-1.elb.amazonaws.com).

Virtual Private Cloud

A Virtual Private Cloud (VPC) is a bit of an umbrella term for a set of related capabilities that are intended to isolate and protect the network traffic to and from EC2 instances. Even though AWS resources are generally shared between AWS customers, a VPC provides a logical separation of your AWS resources from other accounts. A VPC can span multiple availability zones but is limited to a single region.

There is a pre and post-VPC world in AWS, but VPCs have been the default networking stack for EC2 instances since 2013. Concepts and resources introduced prior to the advent of VPCs are generally labelled as “Classic” in the AWS terminology. The rest of this post will focus on the VPC stack and avoid discussing the legacy stack.

Addresses, CIDR Block and Subnets

When it comes to networking, half of the problem is addressing: A VPC is assigned a range of at most 2¹⁶ private IP addresses which can not be routed over the public Internet. This range is broken down in disjoint subnets, each of which maps to one Availability Zone.

The common notation for these ranges is called the CIDR notation, where a range is written as a IP address followed by a slash and a mask, e.g. 192.0.2.0/24. The value of the mask is used to tell how many addresses are in the block. In the case of IPv4, which uses 32 bits, a mask of 24 bits gives you 8 variable bits: 2⁸ = 256 addresses. The lower the value of the mask, the larger the block.

On AWS, it is common to have VPCs with a 172.x.0.0/16 address space, which in turn is divided in multiple /20 subnets – for instance, with 4 availability zones:

Beyond private IPs, a Subnet can also be configured to supply a public IP address to the instances that are launched in it. If so, the network interface of the instance will be given a random publicly routable IP address coming from AWS’s pool. If you prefer for instances to not get a public IP address, this behavior can be overridden in the launch template.

Route Tables and Internet Gateways

After addressing, the other half of networking is routing. VPCs have a routing table which governs the way traffic is routed within the VPC, as well as ingress and egress traffic. Inter-VPC traffic is kept within through a local route in the routing table: this route cannot be changed.

Egress traffic to the public Internet is generally directed at an Internet Gateway (IGW), which interfaces the VPC with the public Internet. Internet Gateways have two purposes:

• routing egress traffic out of the VPC
• routing ingress traffic for instances that have a public IP address

By default, a VPC will have a route to the IGW that applies to all subnets in the VPC - such subnets are called public subnets. When that route is not present for a given subnet, the subnet is fully isolated from the Internet and is deemed private.

Even though directly accepting ingress traffic through an Internet Gateway is feasible, it’s a more common pattern to use ELBs to front the traffic and dispatch it to an ASG where the instances do not need to be provisioned with a public IP address.

Security Groups and Network ACLs

VPCs allow for fine grained control over the traffic that is allowed to flow to and from instances. Network ACLs apply to one or many subnets and act like a standard firewall: they are tasked with filtering packets. They consist of matching rules that are evaluated in order: the first rule that matches the traffic will determine whether the packet is allowed to proceed. ACLs for ingress and egress traffic are defined separately.

Security Groups (SGs) also offer filtering capabilities but they act at the instance level: a given Security Group may only apply to select instances in a subnet, and they may span over instances across multiple subnets. Importantly, SGs also offer the flexibility to match traffic based other SGs. This allows grouping instances by “type” on a network basis and enforcing a given network topology between different groups.

For instance, say you have a Web frontend which relies on two backend services: Backend Service 1 and Backend Service 2. Each backend service uses the same Database server. The Web frontend shouldn’t talk to the database directly. Additionally, let’s assume that Backend Service 2 is of a higher tier of importance that Backend Service 1, and should therefore avoid making calls to it:

Some of these rules are redundant and could therefore be omitted, but the principle remains: Security Groups are powerful in that they can apply at once to multiple instances of the same kind.

Default VPCs

VAs mentioned earlier, VPCs have become the default mode of managing EC2 instances and network setups. When you create a new account, a default VPC and subnets are provisioned by default in every AWS region. In each region, this VPC will consist of a /16 CIDR block (65K private addresses), divided in as many /20 subnets (4K private addresses) as there as AZs.

All subnets will be public ones and a default route will allow them to reach the Internet through the Internet Gateway. There will be a default Network ACL allowing for all traffic to flow, and a default Security Group as well. You can chose to either take the default VPC and tune it to your needs, or to provision a new VPC from scratch.

Elastic IP Address

An Elastic IP Address (EIP) represents a single public IP address that is routable and reachable from the public Internet. An EIP can be provisioned within your account and assigned to an instance, which then becomes accessible on the Internet.

Unlike instances, which are ephemeral, EIPs are static and can be freely attached to a given instance, then detached and moved to a different one. When an EIP is attached to an instance, the public IP of that instance changes to be that specified by the EIP. EIPs will retain their associated public address until they are explicitly released.

The goal of EIPs is to provide resilience: in case of an issue with the underlying instance or the application it runs, the EIP can be reassigned to another instance running the same application. The new instance will immediately begin receiving connections directed at that IP (with the caveat than previous connections will be interrupted and need to be re-established).

Note that EIPs can be created for free, but will incur charges when they are not attached to a running instance (because they consume an address that is left unused).

Elastic Network Interfaces

An Elastic Network Interface (ENI) augments the concept of EIP by adding more settings such as private IP addresses and Security Groups but still remain decoupled from instances. Just like EIPs, ENIs can be freely attached to and detached from instances.

An ENI can only be mapped to one instance, but an instance can have multiple ENIs at once: the limit is dependent on the instance type. All EC2 instances come with a default primary interface which is proper to the instance and cannot be detached.

An ENI possesses the following attributes:

• One private IPv4 address (and, optionally, secondary private addresses)
• Up to one Elastic IP per private IPv4 address
• One optional public IPv4 address
• Security Groups
• A MAC address

An ENI can only be defined within a VPC, and is more specifically defined within a Subnet: it will use private IPs that are in the CIDR block for that Subnet. You can choose to automatically obtain a private IP from the pool of the subnet, or If the Subnet is configured to allocate a public IP for each instance, creating an ENI will also give it a random public IP from the AWS pool of public IP addresses.

Conversely, an ENI can only be associated with Security Groups from the VPC they belong to.

Finally, an ENI can be attached to an Elastic IP address: this allows the same kind of failover scenario as with plain EIPs with the added benefit that the ENI also carries the Security Groups to the new instance.

1. I � Unicode (Intro)
2. The Character Set
3. The Encodings
4. The Algorithms (This article)

As seen in the other two parts of this series, Unicode concerns itself with defining a universal set of characters and representing those characters as sequences of bytes. The Unicode standard also concerns itself with text as a broader domain. In this section, we are going to dive a bit deeper in

• How is text sorted? How do you define an ordering across characters in a set as vast as the one defined by Unicode?
• How is text compared? We saw that Unicode sometimes defined distinct code points for the same entity, e.g. U+FF76 HALFWIDTH KATAKANA LETTER KA ｶ and U+30AB KATAKANA LETTER KA カ. What happens when those two are compared?

All of these are defined by algorithms, which are specified as part of the standard

The collation algorithm

Collation is fancy term for sorting: your goal in this case is to go from an unordered collection of Unicode strings to a state where these are in the correct order. What is critical to understand here is that the “correct order” is neither universal nor invariant: it changes from person to person, and it changes over time.

Different scripts and different languages impose different rules for deciding the ordering of strings. All code points do have a U number but it does not imply any kind of ordering. The Unicode Collation Algorithm is specified as Technical Report 10 in the Unicode standard.

There is generally a notion of basic order across distinct characters in a script, but languages have an incredible variety when it comes to defining the ordering rules:

• Characters that have diacritical marks (e.g. accents) generally follow specific ordering rules, or can occasionally be considered as separate characters than the one they are graphically based on.
• Groups of multiple characters may be contracted as a single token for the purpose of sorting the string, e.g. “ch” in Czech sorts after “h”, whereas it sorts after the “c” character in Spanish.
• Characters may be expanded into clusters of multiple distinct characters, e.g. “ä” in certain German contexts expends to “ae”, which means a word such as “Äquivalent” will be sorted before “Affekt”.
• When more than single words are compared, the rules for handling punctuation such as a dash may vary.
• Text may need to be normalized, which will be covered in details further down in this post.
• When it applies in the script, the casing of letters might be an element for determining the ordering rule.

In other words, trying to sort text without knowing the expectations of the person who will read the result is pointless: you need to know the language and the context in which the sorted content will be used and presented.

The Unicode Collation Algorithm is therefore more of a framework for collation to be defined and implemented for a given context. The goal of the algorithm is to translate strings of Unicode code points into a sort key, which is an array of 16-bit integers. The sort keys can then be compared in binary to establish the order of the source strings they were computed for.

The following snippet of code demonstrates how to build a collator to properly sort a list of strings, in Java:

package co.eiggen.unicode;

import com.ibm.icu.text.CollationKey;
import com.ibm.icu.text.Collator;
import com.ibm.icu.util.ULocale;

import java.math.BigInteger;
import java.util.ArrayList;
import java.util.List;
import java.util.Collections;

public class Collation {
  public static void main(String... args) {
    ULocale germanPhoneBookLocale = ULocale.GERMAN.setKeywordValue("collation", "phonebook");
    Collator germanPhoneBookCollator = Collator.getInstance(germanPhoneBookLocale);
    
    List<String> words = new ArrayList<String>();
    words.add("Affekt");
    words.add("Äquivalent");

    for (String word : words) {
      byte[] collationKey = germanPhoneBookCollator.getCollationKey(word).toByteArray();
      BigInteger collationKeyAsInteger = new BigInteger(1, collationKey);
      System.out.printf("Collation key for %s is %s\n", word,  collationKeyAsInteger.toString(16));
    }

    System.out.printf("Initial order: %s\n", words.toString());
    Collections.sort(words, germanPhoneBookCollator);
    System.out.printf("Sorted order: %s\n", words.toString());
  }
}

The code yields the following output:

Collation key for Affekt is 2a3434323e50010a01dc0900
Collation key for Äquivalent is 2a324a523a542a403244500145480d01dcc60d00
Initial order: [Affekt, Äquivalent]
Sorted order: [Äquivalent, Affekt]

As can be seen in the above snippet, the most popular implementation of the Unicode Collation Algorithm is ICU, which stands for International Components for Unicode. It’s a library developed for C/++ and Java that provides bindings to CLDR, the Common Locale Data Repository.

CLDR, as the name implies, is a repository of data that pertains to regions and languages of the world, and the combination thereof. Specifically for the purpose of sorting, CLDR encodes all the differences mentioned above: diacritics, casing, expansion and contraction, etc… CLDR is also an authority for a range of tasks pertaining to software internationalization: formatting of dates, numbers, currencies, pluralization rules, calendars, segmentation, etc…

Even though CLDR is maintained by the Unicode consortium, its scope goes well beyond the handling of text. In addition to Java and C/C++, bindings of CLDR have been created for Ruby and JavaScript, even though the latter is being progressively replaced by the Intl object.

The normalization algorithm

Normalization is an issue that occurs when you want to compare strings rather than order them. As the first part of this series explained, Unicode defines over a million code points, with some of them being strikingly similar to others.

In other cases, the standard included code points that explicitly represent typographic ligatures: groupings of multiple characters that are rendered on screen by a single font glyph. One of the most common ligature in the Latin script is U+FB01 LATIN SMALL LIGATURE FI (fi). Ligatures do not have any standalone semantic, they are just a presentation artifact.

The normalization forms, defined at the standard level by Technical Report 15, intend to bring a solution to this issue. “Why is normalization important?”, you might ask.

The most common use case is indexing: if you want to build a search engine, or a database engine, a lot of the work will consist in making sense of the content indexed and making it queryable. On either side of it, the user data should be normalized so that a text containing the “fi” ligature would actually match an input such as “fi”.

The same logic applies to a number of other characters beyond ligatures:

• Unicode has dedicated code points for superscript numbers, e.g. U+2078 SUPERSCRIPT EIGHT (⁸)
• Full-width and half-width Katakana characters and Latin characters, which were covered in the first post in this series
• Unicode defines circled alphanumeric code points such as U+2466 CIRCLED DIGIT SEVEN (⑦)
• Old Roman numerals such as U+216F ROMAN NUMERAL ONE THOUSAND (Ⅿ) or even composed forms such as U+2177 SMALL ROMAN NUMERAL EIGHT (ⅷ)

The normalization algorithm defines 4 normal forms which have different characteristics and will yield distinct results:

• A normalization form may be concerned with canonical equivalence (NF) or compatibility (NFK) — canonical normalization mandates strict canonicity of the strings, while compatible normalization is more lenient and considers distinct code points or combination of code points to have the same meaning.
• A normalization form may be concerned with the composed (C) or decomposed form (D) of the underlying characters – some diacritical characters may have a standalone form, e.g. U+F4 LATIN SMALL LETTER WITH CIRCUMFLEX (ô), but may also be written as a sequence of U+6F LATIN SMALL LETTER O (o) and U+302 COMBINING CIRCUMFLEX ACCENT ( ̂). In the latter case, the order in which the code points are combined affect the normalized form of the string.

Picking which form is the best suited for the task depends on the goal you have. The table below shows the result of testing for equality in the normalized forms of two input strings:

As can be seen from these results, the largest difference in normalization results stems from the distinction between canonical and compatibility forms: the canonical normalization is very strict, while the compatibility one matches code points based on semantics.

There are no examples where the decomposition criteria matters when it is used consistently. Which you pick is up to you but be consistent in your choice. If you were to compare the normalized outputs of, e.g. NFKD and NFKC for the same string input string, they would be non-equal for any string that contains composed characters:

The code used to generate these tables is available here.

Miscellaneous other algorithms

Segmentation

As we just saw in the previous section about the Normalization algorithm, characters may actually be represented by several code points — these are called combined characters and are semantically equivalent to their combined form. What this means pragmatically is that the end user doesn’t care how the underlying text is formed, they care about the end result that they can see and interact with. Whether “ñ” is written as U+F1 LATIN SMALL LETTER N WITH TILDE or U+6E LATIN SMALL LETTER N along with U+303 COMBINING TILDE makes no difference.

The problem becomes more complex with more advanced forms of composing characters, e.g. “👨🏿‍🦲” is actually a combination of four (!) code points—but when it is rendered in a UI, it should still be represented as a single character. Segmentation (specified in Unicode Technical Report 29) is intended to allow the programmatic chunking of a Unicode string into clusters of code points that match more closely what an end-user might expect.

Segmentation doesn’t just address the grouping of code points into higher-level symbols, it also defines how to break a text into words, sentences and paragraphs. There are several use cases for this:

• Computing the user-visible length of a string
• Sentence-level breaking is useful for search engines to index content
• Implementing text selection, e.g. double-clicking on a word extends the selection to the full word

ICU ships with a segmentation implementation of compliant with the Unicode standard.

The Bidirectional Algorithm

The bidirectional algorithm (Unicode Technical Report 9) is perhaps the most abstract of one, making it arduous to grasp and to implement correctly. You should almost never need to directly interact with it, and instead rely on the implementation at the OS or browser level to do the right things.

Writing systems of the worlds have a natural directionality, which is proper to each of them. Latin, Greek, and many other scripts are written left to right (LTR), but scripts such as Hebrew and Arabic are written and read from right to left (RTL). The bidirectional algorithm concerns itself with the rendering of text in mixed contexts, e.g. when Arabic and Latin characters belong in the same sentence or paragraph.

It introduces formatting characters that can be used to assign segments of text an explicit directionality, which may be different than the one they would have by default. These characters have no semantics of their own and are only hints to systems tasked with rendering text to the user:

The directional formatting characters are used only to influence the display ordering of text. In all other respects they should be ignored—they have no effect on the comparison of text or on word breaks, parsing, or numeric analysis.

1. I � Unicode (Intro)
2. The Character Set
3. The Encodings (This article)
4. The Algorithms

Perhaps Unicode’s greatest insight is the decoupling of characters from bytes. Before Unicode, most systems used for representing strings equated the bytes read from, e.g., a file, to the characters in memory. It made systems easy to understand but inherently limited.

For instance, the C programming language defines a very minimal character set which contains basic latin characters and the built-in char type as follows:

If a member of the basic execution character set is stored in a char object, its value is guaranteed to be nonnegative. If any other character is stored in a char object, the resulting value is implementation-defined but shall be within the range of values that can be represented in that type.

When it came down to representing those characters in memory and writing them to a file, all C implementations settled on ASCII. 'a' the ASCII character was mapped to 0x61 and that was also the byte value of the character 'a' when declared in C. That is not the case in Unicode: the lowercase latin letter ‘a’ is defined as U+0061. That’s a subtle difference but a critical one: the U+ notation is how you know the value is referring to a Unicode code point and not bytes.

Encodings sit between between bytes and code points. They consist of rules for translating code points into bytes and bytes into code points. What it implies is that you cannot just take bytes and turn them into a meaningful Unicode string of text without knowing what encoding was used to write the bytes.

Unicode defines several encodings and all of them are designed to support the full range of all possible code points across all planes. Encodings are designated by the “UTF” prefix, which stands for “Unicode Transformation Format”

UTF-32

UTF-32 is the simplest and most straightforward of all Unicode encodings. It’s a fixed-length encoding where each code point is encoded over 32 bits. The U+ value of a code point is directly converted into bytes with the same value:

Because there are “only” over one million Unicode code points but that 32 bits can represent up to 4 billion values, 11 bits are always set to 0. This makes UTF-32 an extremely wasteful encoding.

One useful purpose for it is to trivially index a code point in a string of bytes: the code point at index n is simply the n^th 32-bit chunk in your string. But there’s little practical use to this: Unicode code points may combine with adjacent ones to form the actual grapheme, which is the unit of text that is truly meaningful to end users.

Example

Here’s an example of select code points and how they are encoded into bytes using UTF-32:

UTF-16

UTF-16 is a more complex form of encoding, perhaps the most peculiar of all Unicode encodings. It’s a variable-length encoding with 16-bit chunks: a given code point may be encoded using one or two chunks. As with other encodings, it supports all Unicode code points and follows the following structure:

The reason UTF-16 is more complex than other encodings lies in its tight coupling with the definition of the character set itself. At the beginning of this article, I said that Unicode’s greatest insight was the decoupling between characters and bytes: this is true today but was not always the case.

The reason behind the 17 planes

In its very first version, Unicode defined a single plane of 65,536 elements and the UTF-16 encoding was a direct map from U numbers to bytes — this was a very simple architectural choice for implementors of the standard to follow.

When the BMP started running short of space and that the need to grow beyond it became clear, Unicode was confronted with a choice:

• Deprecate UTF-16 and define a new encoding that would cover the new full breadth of the new space. This has the downside asking implementors to roll out and adopt a brand new encoding.
• Define a scheme that would make the encoding more complex but allow backwards compatibility with existing implementations.

The Unicode Consortium chose the latter option: out the already packed BMP, the standard would reserve a range of 2,048 code points that would be guaranteed to never be allocated: U+D800 → U+DFFF. This space would instead be used create an index for pointing at code points outside of the BMP.

UTF-16 needed to remain an encoding where chunks would take 16 bits each. Characters beyond the BMP would necessarily take at least 2 16-bit chunks to be represented: UTF-16 calls those chunk surrogate pairs: this required further dividing the 2,048 block in 2 blocks of 1,024 values each: U+D800 → U+DBFF and U+DC00 → U+DFFF: one for the high surrogate and one for the low surrogate.

1,024 values requires 10 significant bits to represent, which means that a single 1,024 block is able to map to 2¹⁰ values. Since there are 2 blocks, their indexing powers are multiplied and bring the total of supplementary code points to 2¹⁰ × 2¹⁰ = 2²⁰.

In other words, sacrificing just over two thousand code points in the BMP opened up the space for 2²⁰ brand new code points. Since a plane contains 2¹⁶ code points, the total number of planes outside of the BMP is 2²⁰ ÷ 2¹⁶ = 2⁴ = 16, bringing the total count of planes to 17.

Even though the scheme was first published in 1996 in Unicode 2.0, it took until 2004 for implementors such as Sun to surface this new mechanism to developers in Java 1.5.

That’s the secret of UTF-16, and the reason it is tightly coupled to the Unicode standard itself: The U+D800 → U+DFFF range of code points is guaranteed by the standard to never be assigned and its values are instead used for indexing characters outside the BMP, but exclusively for the purpose of allowing UTF-16 do do to (neither UTF-32 nor UTF-8 have that problem). When this is needed, the two resulting 16-bit chunks are called a surrogate pair.

The UTF-16 scheme

To encode a character in non-BMP planes requires some bit fiddling:

1. Take the U-value of the code point and map it to bytes, e.g. U+1F976 → 0x1F976
2. Subtract 0x10000 (65536) — since the maximum code point value is U+10FFFF (1114111), the range of the remaining value is 0x0–0xFFFFF (0–1048575)
   1. Left-pad the value with 0, up to to 20 bits
   2. Take the high ten bits of that value (range 0x000–0x3FF, 0–1023) and add 0xD800 (55296) to form the high surrogate (range 0xD800-0xDBFF, 55296–56319)
   3. Take the low ten bits of that value (also in range 0x000–0x3FF) and add 0xDC00 (56320) to form the low surrogate (range 0xDC00–0xDFFF, 56320–57343)

Properties

UTF-16 has properties that make it a good general purpose encoding. Primarily, any code point in the BMP will be encoded over 16 bits: this is a use case that covers most language on Earth in a decently low number of bits. Second, despite the apparent complexity the encoding rules, decoding a UTF-16 stream is relatively simple to implement in code: if a chunk is in the high surrogate range, consume another chunk and combine the two.

Examples

Below is an example of the same code points being encoded to UTF-16¹:

UTF-8

UTF-8 is by far the most popular of Unicode encodings and the one most developers are familiar with. Like UTF-16, it is a variable-length encoding where code points are encoded using chunks of 1 byte (8 bits). Just like other Unicode encodings, any code point may be encoded and decoded using UTF-8. The scheme for encoding a code point into UTF-8 is as follows. UTF-8 divides the set of code points in four contiguous ranges and maps them to be represented over 1, 2, 3 or 4 bytes:

Encoding a code point is a relatively simple task, especially compared to UTF-16. Let’s take U+801 SAMARITAN LETTER BIT (ࠁ) as an example.

1. From the code point, determine the number of bytes needed to encode it using the ranges in the table above (3 bytes in this case)
2. Take the U number of the code point and turn it into binary U+801 → 0x801 = 0b100000000001
3. Pad left with 0s to reach the target number of bytes, which is 16 in our case: 0b100000000001 → 0b0000100000000001
4. Fill in the blanks starting from the left:
   1. The first byte will take the form 0b11xxxxxx → 0b11100000 (0xE0)
   2. The second byte will take the form 0b10xxxxxx → 0b10100000 (0xA0)
   3. The third byte will take the form 0b10xxxxxx → 0b10000001 (0x81)

Decoding UTF-8 is also relatively simple, as the first bytes tells you how many bytes are expected to participate in the encoding of the code point.

As of 2020, UTF-8 is by far the most popular form of encoding for text. There are a few reasons for this, but the obvious one is the encoding’s backwards compatibility with ASCII. ASCII is a Latin-centric codec which was incredibly popular in the early days of computing and was still in use through the 2010’s To this day, it is still fairly common for programs written today to special case characters that are outside the set of characters defined in ASCII.

The genius of UTF-8 is that any valid ASCII content is naturally valid UTF-8 content. In other words, a file written as ASCII text when the standard was initially published in 1963 (almost 60 years ago!) can be read by a program interpreting it as UTF-8 in 2020.

On the Web, this encoding went from being used by ~5% of pages in 2006 to under just 65% in 2012 and 95% in 2020. UTF-8 benefited from being the recommended encoding for HTML5 documents, JSON, and a variety of other formats and standards. It is also the internal encoding for strings in the Go and Rust programming languages.

The following table shows the encoded values of the same code points used in the previous examples:

A note on byte-order marks

Whenever a data serialization format encodes data over multiple bytes, the question of endianness is one that must be addressed. A UTF-16 implementation that would naively rely on the native of the host machine might produce a different output than a machine using the converse endianness mode. This can be an issue for UTF-32 and UTF-16.

To that effect, the Unicode standard defines a code point that has a special status: U+FEFF ZERO WIDTH NO-BREAK SPACE. This code point, which is called a byte-order mark (BOM) is expected to be present as the first characters of byte strings that encode Unicode strings over multiple bytes:

• Upon encoding a string, encoders would prefix it with the BOM and then run the normal encoding logic in their native endianness
• When reading bytes that are expected to encode code point over multiple bytes, decoders would peek the first chunk of bytes:
  • If the result is 0xFEFF, the rest of the bytes should be decoded as little-endian
  • If the result is 0xFFFE, the rest of the bytes should be decoded as big-endian

This is a step that is generally transparently handled by implementations and that you shouldn’t have to pay attention to:

• UTF-32 would normally require a BOM (encoded on 4 bytes, i.e. 0x0000FFFE or 0x0000FEFF) but since the encoding is practically never used, implementations are inconsistent and may omit it.
• UTF-16 remains a relatively common encoding and most encoders would automatically prefix the BOM to the bytes they produce — this was the case for the Java code used to generate the encoding table above, I chose to omit the BOM from the output but it was actually present.
• UTF-8 operates at the byte level, and therefore doesn’t need the BOM in either encoding or decoding. It occasionally happens that UTF-16 content that is then serialized into UTF-8 embed a BOM, but implementations are expected to ignore it.

1. I � Unicode
2. The Character Set (this article)
3. The Encodings
4. The Algorithms

To understand Unicode’s contributions to the space of character sets, it’s worth taking a look at what the landscape looked like before it came around:

• character sets were numerous: over the decades that saw the explosion of enterprise and personal computing, dozens if not hundreds of character sets were defined. Some were defined as pure standards (ASCII, EBCDIC), while others were tied to platforms (Windows 125x, Mac OS Roman)
• character sets were limited: they generally targeted regions (“ISO 8859-1 Western Europe”) or scripts (“Windows 1254” for Turkish) but none would really be able to capture the full breadth of the world’s languages by itself. At best, they defined a few thousand characters

Space

Unicode set out to solve those issues by primarily giving itself space: rather than collecting the characters and putting them in a set, Unicode first defined the structure in which the characters would exist: in its first version, the space of all characters was specified as an array of 65,536 elements (2¹⁶).

As the standard progressed in its task to inventory the world’s characters, it became clear that the initial space was going to run out. To that effect, the second major version of Unicode allocated 16 more planes, each having 65,536 elements of their own. The total number of possible characters is 1,114,112 (17 × 65,536)

There are way fewer known characters in all of the world’s scripts and languages. From that point on, Unicode had plenty of room to grow.

The very first of the 17 planes is called the Basic Multilingual Plane (BMP) and contains characters that cover most of the written languages on Earth:

Addresses and names

Second, Unicode defined an addressing scheme. In previous character sets, there was an immediate equivalence between the character and the bytes you used for representing that character. In contrast to that approach, the Unicode character set only concerns itself with giving each character an address in the array (its U+ number) and a name.

The number and the name define what is called a code point. All code points are created equal — some have semantics but from an addressing standpoint, there is no fundamental difference between characters such as U+006B LATIN SMALL LETTER K (k), symbols such as U+27A4 BLACK RIGHTWARDS ARROWHEAD (➤) and emojis such as U+1F63B smiling cat face with heart-shaped eyes (😻).

It’s in the Unicode mission statement:

Unicode provides a unique number for every character

Code points bonanza

Unicode defines over a hundred thousand code points that cover over a hundred scripts (a script is a writing system such as Arabic, Cyrillic, ancient Greek, Hebrew, Latin, etc…). Because there is so much room to grow, Unicode also defines characters that are used in music or in mathematics.

Because there is so much space available, Unicode can afford to define code points that resemble one another but that have different semantics. U+03A3 GREEK CAPITAL LETTER SIGMA (Σ) and U+1D6BA MATHEMATICAL BOLD CAPITAL SIGMA (𝚺) are different code points even though the latter stems from the former.

Look-alike characters also get distinct code points: U+0430 CYRILLIC SMALL LETTER A (а) is a different character than U+0061 LATIN SMALL LETTER A (a). Same goes for U+03B2 GREEK SMALL LETTER BETA (β) and U+00DF LATIN SMALL LETTER SHARP S (ß). In other words, Unicode doesn’t concern itself with the shape of a character, it concerns itself with its formal definition and its function. And since there’s plenty of room, Unicode can be extremely specific to define all the variations of what might otherwise pass for the same character.

The minutiae can go quite far. An interesting example is that the Western latin script and its numbering system is fairly prevalent in the world, and it is common that other scripts (Arabic, Cyrillic, Hebrew, etc…) need to use latin characters in contexts where it mixed with local ones.

For mostly aesthetic reasons, the Japanese and Chinese languages have defined a different mode for writing those, where, in order to match the natural width of ideographs, the physical width of the glyph (the space taken by the character on the screen) is bigger than it normally would be.

Unicode defines those as separate code points from the original one they are based on. For instance, U+FF43 FULLWIDTH LATIN SMALL LETTER C (ｃ) is intrinsically the same as U+0063 LATIN SMALL LETTER C (c) but it is defined separately from the standpoint of Unicode.

Conversely, there is a also a set of Katakana characters that are defined as “half-width” (e.g. U+FF76 HALFWIDTH KATAKANA LETTER KA ｶ) compared to their natural full-width base (U+30AB KATAKANA LETTER KA カ).

Versions

Lastly, Unicode took care of defining versions of its character set. As of this writing, the most recent version of Unicode is 13, and it defines 143,859 code points. Unlike other character sets, Unicode planned ahead to allow for the character set to grow and for these changes to roll-out in a way that is backwards-compatible.

When new characters such as emojis came around, all that needed to happen is to assign each of them a U+ number and a name, and that was that.

Fonts and rendering

After new code points are defined in a Unicode release, two more steps need to be taken for the end user to actually be able to benefit from it:

• OS vendors (Apple for macOS, Google for Android, etc…) need to bump up their Unicode version to include support for those characters
• Font vendors (e.g. Twitter, Microsoft, Apple, Google) need to add glyphs (graphical symbols) to represent those characters.

As mentioned before, Unicode doesn’t focus on character shapes but end users need that! Fonts embed the data for converting Unicode code points into glyphs, which are what you see on your screen. They are orthogonal to the Unicode standard but they are critical to the user experience of the Unicode character set.

A “missing character” in a string you see on screen might be because the operating system doesn’t recognize it, or because there’s no available font that knows what to make of that code point (or sequence of code points). No single font can represent all Unicode code points because of size limits with popular font file formats. As a result, vendors segment their font files by script or sometimes regions.

If you were to use an old version of, e.g. Android, newer emojis would not render correctly but would also not crash your system: the Unicode implementation would just see a code point it doesn’t know about and handles it

• If this is an issue with the underlying encoding, the character should be replaced U+FFFD REPLACEMENT CHARACTER (�)
• If there is no available font to render the code point, the system should use a generic replacement glyph, e.g. “󰀀”

If you work in computer science, the sentence “It’s probably just a Unicode character” is probably one that you have heard thrown around more than once in your career. It’s one after which the people in the room generally shrug it off as one of those inevitable quirks of working with computers and move on with their day because, well, what else is there to do today?

In another instance, you might have found yourself trying make sense of a blob of bytes content using various flavors and combinations of ASCII and then UTF-8 and then converting back to binary without always understanding what went on and how you got it working. But that email parsing functionality you were tasked to deliver is working now, at least until you start shipping your software in Bulgaria or Uzbekistan, at which point you’ll hopefully be two jobs further along in your career.

And there is this set of JIRA tickets, all carefully labelled i18n by your predecessor, where your users report being unable to get the right search results when they search for the content they have hosted in the CMS you sell them. Some of them even provide reproducible examples of issues with your software’s indexing. But there’s few enough of them that you brush it off as an edge case: “It’s probably just a Unicode character or something like that”. You decide to further label all tickets with intern and starter-task.

A primer on Unicode

Joking aside, this series is intended as a moderately advanced explainer for people who want to educate themselves about what Unicode is. The reason for it to be is that there are very few places on the Internet that strike the right balance between Joel Spolsky’s Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) (which is an absolute must-read) and the technical portal on the official Unicode website (that homepage got a welcome revamp in 2020, let’s hope it will expand to the documentation sections too)

Unicode is a lot of different things to different people. It’s easier to work backwards from what the goal of Unicode is: enable everyone to use their own language on any digital platform. That’s a lofty goal, especially at the time it was formulated all the way back in the early 1990’s.

However, and unlike most standards and specifications that came in the wake of dozen others, Unicode has actually risen to be the prevailing set of modern standards for managing text on billions of computers and phones. Since there’s a XKCD for everything, there’s also an anti-XKCD for everything:

I will repeat myself: Unicode is a lot of different things to different people. That’s because the Unicode standard is a vast and complex one: it attempts to formalize the complexities and intricacies of text, which is a thorny and culturally touchy domain. It seeks to bridge the gap between computers and programmers, who expect rules and structures, and languages and humans, who are messy and history-rich.

For this series, I will focus on three aspects of the Unicode standard:

• The character set: what is the Unicode character set and how does it differ from ones that came before?
• The encodings: what exactly is an encoding and why it is a requirement for Unicode I/O?
• The algorithms: beyond characters and codepoints, what higher-level problems does the Unicode standard consider and address?

Code and examples

The code used to generate the tables in this series (and more) can be found on GitHub at jsilland/unicode. The examples are in Java but are intended to demonstrate concepts that are applicable and accessible in most languages that have access to a port of CLDR. Contributions and examples in other programming languages are welcome.

Despite its status as the go-to social network for millions of athletes, Strava was missing support for emojis until the second half of 2017. This wasn’t even something that came on our radar as a feature request from users, but the content on Strava looked flat compared to other places where users are free to type in text. Facebook, Twitter, Slack and even GitHub are places where emojis have taken on like wildfire: it’s just part of the world’s vocabulary and Strava users were left behind.

But even before the engineering team looked seriously into supporting emojis, one of my coworkers asked why the caption of a picture taken on another employee’s activity showed signs that we actually did: specifically, the caption simply read as two red hearts: “❤️❤️”

The persistence layer

Most entities at Strava are persisted in MySQL databases. The main instance encoded most string using utf8, which is an incredibly misleading name in the context of MySQL. As you can see in the linked documentation, this encoding cannot actually encode a large portion of the Unicode codepoints:

• “No support for supplementary characters (BMP characters only)”
• “A maximum of three bytes per multibyte character”

Strava’s core database was set up before MySQL introduced the utf8mb4 character set, which extends the support of UTF-8-encoded text to the actual full range of all codepoints defined Unicode.

However, pictures (and their captions) are managed by a separate backend, which has its own database. The captions persisted in that database can contain emojis because the column was explicitly set to support it. As a matter of fact, one could easily INSERT values that contained any codepoint and retrieve them later, as long as this was done from the MySQL CLI.

The serving layer

Now that we know that the data in the database may be an emoji, let’s try to see how it is returned to the client. Using httpie and jq, we can specifically look at that field in an API response and examine the bytes¹:

$ http "https://www.strava.com/api/v3/activities/<redacted>/photos?photo_sources=true" \
  'Authorization: Bearer <redacted>' | jq '.[8].caption' | xxd
00000000: 22e2 9da4 efb8 8fe2 9da4 efb8 8f22 0a    "............".

We see the same sequence of bytes, repeated twice: e29da4efb88f. This is the raw hexadecimal representation of content that is UTF-8 encoded. Decoding that sequence to the Unicode codepoints yields two of them:

• U+2764 HEAVY BLACK HEART: this character is actually not in the Emoji block (range: U+1F600..U+1F64F) but in the Dingbats block (range: U+2700..U+27BF)
• U+FE0F VARIATION SELECTOR-16: this is a variant selector - it specifies a variant of the previous character’s glyph, and, in this case, I assume it changes the color to red.

What’s interesting is that the heart codepoint actually appears in the exhaustive list of Emoji codepoints on the Unicode website. The character, which was originally defined in 1995, was grandfathered into the emoji range despite having a codepoint in a very different range than other emojis.

We’ve asserted that the content is actually not an emoji, and can therefore be rendered properly by Rails v3. Problem solved. However, the premise of my coworker’s question was that emojis simply do not work and it’s worth exploring why that is the case.

❤️ but no 🎉

At the time I worked there, Strava’s main frontend was a relatively vanilla Rails setup running v3.2. Let’s look into how that version of Rails encodes strings in JSON - the code below is a simplified version of activesupport/lib/active_support/json/encoding.rb²:

string = "\u2764\uFE0F"
string = string.encode(::Encoding::UTF_8, :undef => :replace).force_encoding(::Encoding::BINARY)
json = string.gsub(escape_regex) { |s| ESCAPED_CHARS[s] }
json = json.gsub(/([\xC0-\xDF][\x80-\xBF]|
                   [\xE0-\xEF][\x80-\xBF]{2}|
                   [\xF0-\xF7][\x80-\xBF]{3})+/nx) { |s|
            s.unpack("U*").pack("n*").unpack("H*")[0].gsub(/.{4}/n, '\\\\u\&')
          }
json = %("#{json}")
json.force_encoding(::Encoding::UTF_8)

Making sense of this logic is not easy, to say the least:

1. The string is first encoded to UTF-8 (undefined characters are replaced with the replacement character) and then force-encoded to binary. This will enable the rest of the code to dig into the binary representation of the text encoded in UTF-8
2. A first round of escaping takes place:

• gsub is a method on Ruby strings that takes in a regular expression and yields each match in the receiver to a block which is tasked with determining the string to replace the match with.
• escape_regex is essentially [\x00-\x1F"\\], i.e. bytes 0x00 to 0x1F, the double quote (reserved character in JSON) and the backslash.
• ESCAPED_CHARS is a mapping that contains the escaped version of these, e.g. \x17 → \u0017.

3. A second round of replacement takes place, again using gsub

• The code first attempts to chunk the string in groups of two, three or four bytes depending on the range of the leading byte:
  • 0xC0-0xDF → 2 bytes
  • 0xE0-0xEF → 3 bytes
  • 0xF0-0xF7 → 4 bytes
• Each chunk of bytes then goes through another conversion pipeline that uses Ruby’s pack and unpack methods
  • decoded as a UTF-8 character (unpack("U*"))
  • converted into big-endian 16-bit values (pack("n*"))
  • finally converted into a hexadecimal string (unpack("H*"))

The chunking in 2/3/4 bytes is done based on the correct bytes prefixes for UTF-8, where codepoints are encoded on 1-4 bytes

As one might suspect from the complicated logic implemented here, the final conversion pipeline is why Rails doesn’t support emojis out of the box. All responses to XHRs and the entirety of the API downstream traffic go through this JSON encoder. This block of code was once removed and then re-added because the fix was deemed to be changing the behavior of a stable version (later versions of Rails do not have that issue).

Interestingly, rendering an emoji character in an HTML context does work even in v3.2 because the encoding path is completely independent ¯\_(ツ)_/¯. But what actually happens? Knowing the escaping logic is faulty, we’d expect the dingbat character to go through that phase unscathed, while a “real” emoji will come out mangled:

• For “❤️” as an input (string = "\u2764\uFE0F"), the return value is "\"\\u2764\\ufe0f\"".
• For “🎉” as an input (string = "\u{1F389}"), which is a character in the Emoji block, this returns "\"\\uf389\"".

The outputs here seems does not fully allow us to blame the JSON encoder: the two outputs look roughly similar and it’s unclear if either or both of them are good or faulty.

API clients

To understand why the output of the second call is actually an issue, we need to look into what happens on the receiving end of it. JSON stands for “JavaScript Object Notation”: a data representation which is proper to JavaScript and we should therefore continue to investigate in that context. Using the Chrome inspector, it’s trivial to convert our content from JSON into JavaScript strings:

JSON.parse("\"\\u2764\\ufe0f\"")
"❤️"
JSON.parse("\"\\uf389\"")
""

This is the evidence we were looking for all along. It took going all the way to the client but we finally know the issue is real: only one of the two characters is failing to be unescaped correctly by a real-world JSON parser.

To the JSON RFC…

Now that we now that the encoder is faulty, we should look into the root cause of the issue—and specifically understanding why there is such a triple conversion at the last step. That will require a dive in the JSON RFC:

Any character may be escaped. If the character is in the Basic Multilingual Plane (U+0000 through U+FFFF), then it may be represented as a six-character sequence: a reverse solidus, followed by the lowercase letter u, followed by four hexadecimal digits that encode the character’s code point. The hexadecimal letters A though F can be upper or lowercase. So, for example, a string containing only a single reverse solidus character may be represented as “\u005C”.

The heart character is in the BMP, the Basic Multilingual Plane (range: U+0000..U+FFFF), so that section applies and the output of the escaper makes sense.

…And back again

What does the spec have to say about characters that are not in the BMP, however?

To escape an extended character that is not in the Basic Multilingual Plane, the character is represented as a twelve-character sequence, encoding the UTF-16 surrogate pair.

That should be the case for the the party popper character (U+1F389) but the JSON encoder shipping with with Rails 3 doesn’t do that. It did escape the character but actually stripped the leading 1, which is kind of important 😕. As per the spec, characters may be encoded in UTF-16 and represented using the two resulting bytes. Converting 🎉 to UTF-16 yields “D83C DF89”, so let’s try that:

JSON.parse("\"\uD83C\uDF89\"")
"🎉"

We have our first step towards a fix: if we can get the encoder to to spit out the correct UTF-16 bytes representing the “🎉” character, we could mitigate the issue. But it seems like a fairly painful mitigation: do we really need to jump through all those hoops to just represent emojis in JSON? The sad truth is that all that escaping is not even needed to begin with. The spec also contains the following language:

JSON text SHALL be encoded in Unicode. The default encoding is UTF-8

UTF-8 is just an encoding, which supports all Unicode codepoints, including emojis. And the spec didn’t say that such characters must be escaped, it said that they may. So let’s try to be literal for one second:

JSON.parse("\"🎉\"")
"🎉"

This is exactly the path taken by the JSON escaper in Rails 4+: it mostly became a passthrough for text content which is already encoded in UTF-8.

Step by step

The full solution to supporting emojis on Rails 3 and MySQL 5 is as follows:

1. Convert your MySQL VARCHAR and TEXT columns to use utf8mb4 as an encoding. Depending on the volume of data you have, this may take a while.
2. Set utf8mb4 as an encoding in your database connection settings.
3. Monkey-patch the default Rails JSON encoder — you can do that in an initializer, e.g. config/emoji_support.rb. This commit which was proposed in the 3.2 branch to fix this very issue is a good place to start.
4. Test, test, test. Depending on the age of your Rails application, there can be subtle bugs that have never manifested with the lower range of Unicode codepoints. One that wasn’t caught in time for Strava is this one.

The Strava app has a feature where users can mention each other in comment boxes: when you start typing someone’s name, it triggers an autocomplete box to let you pick a friend’s name, and then bolds and links the text to the profile of the user being mentioned. At the time the mentioning feature was launched, the app didn’t support the full range of Unicode codepoints (practically speaking: emojis were excluded from Strava).

A short while after adding support for emojis, we observed an issue when users would use both mentions and emojis in the same comment:

The bolding was off - and the faulty offset was apparently related to the number of emojis preceding the mention:

Digging Deep

The issue occurred on both our official Android and iOS clients, which hinted at a server-side issue. Hours were spent digging into the mentions logic on the server but no obvious flaws were found. And at the API level, the response returned by the server looked legit:

{
  "text": "👏👏👏👏👏 Jason Milliron Julien Silland",
  "mentions_metadata": [{
    "uri": "strava://athletes/6527",
    "start": 6,
    "end": 19
  }, {
    "uri": "strava://athletes/136697",
    "start": 21,
    "end": 34
  }]
}

For backwards compatibility, mentions were added to the comment object as a mentions_metadata field, an array of objects that contain the start and end index of a single mention, and the target of the link to go to when the link is clicked. The indices in that response are correct: the first mention begins at character 6 and ends at character 19, the second one starts at character 21 and ends at 34:

On the wire, this string was encoded in UTF-8 – for future reference, here’s the hex version of the raw UTF-8 bytes:

On the client side, the bytes were decoded into the native string type and decorated using the routines proper to their platform, in Java on Android:

String comment = new String(bytes, StandardCharsets.UTF_8);
SpannableString stylable = new SpannableString(comment);
stylable.setSpan(new StyleSpan(Typeface.BOLD), mention.start, mention.end, …);

And in Objective-C on iOS:

NSString *comment = [[NSString alloc] initWithData:bytes
                                          encoding:NSUTF8StringEncoding];
stylable = [[NSMutableAttributedString alloc] initWithString:comment]; 
[stylable addAttribute:(NSString *)kCTFontAttributeName
                 value:(__bridge id)font
                 range:mention.range];

The code on both platforms is legible and straightforward, and doesn’t look to be flawed with an off-by-x index issue. It’s a wild guess but since the API response looks correct, and the client code looks correct, we should look at a deeper level to understand the discrepancy.

As it turns out, Java and Objective-C have striking similarities in how they represent strings. Both have good Unicode support but both are also relatively ancient: Java’s JDK 1.0 was released in 1995 and Cocoa’s Foundation Kit (which contains NSString) was developed for NeXTSTEP 3.0, which shipped in 1992. Unicode 1.0 was published in 1992 and played a role in how the standard libraries of each language would encode strings.

With fewer than 65K codepoints defined at the time, it made sense to encode strings using UCS-2, a precursor to UTF-16: strings would be backed by an array of fixed 16-bit chunks. This was both forward-looking and practical: although there would be a memory penalty when representing ASCII-compatible text (which would generally fit on 8-bit chunks), opting for 16-bit chunks opened up compatibility with most of the world’s scripts and made character iteration fast and easy.

However, this direct equivalence between characters and items in the backing array breaks for emojis and other characters that take more than a single chunk to represent.

Different Strokes

When a human thinks about string length and indices, they generally reason about characters they can see on-screen. A programming language will generally reasons in the length of the underlying structure that stores the string. What is supposed to be an implementation detail becomes a leaky abstraction. As it turns out, most languages are leaky by default and few of them do the intuitive thing when computing the length of a string.

The following table lists the result of computing the length of the string “👏👏👏👏👏 Jason Milliron Julien Silland” across a few popular languages:

• Java, Objective-C and JavaScript all use UCS-2 for their strings: the first 5 ‘👏’ each occupy 2 elements in the array, and the remaining 30 codepoints each occupy one: (5 × 2) + 30 = 40
• Ruby and Swift operate at the grapheme cluster level and each return 35
• Go and Rust represent strings as an array of UTF-8 bytes: the first 5 ‘👏’ occupy 4 elements each, and the remaining 30 codepoints each occupy 1: (5 × 4) + 30 = 50
• Strings in Python 3 are strongly typed as such, but they were raw byte arrays in Python 2 and lower

The root cause of the rendering issue is that the indices returned from API operate in terms of human characters whereas both clients use them to index an array of 16-bit chunks. When a character such as 👏 takes 32 bits to represent, the indices are off. It’s clear from this example that you should be skeptical of the commonly available methods to obtain the length of a string. There are only a handful of situations where it can be relied on:

• You are positively certain that it measures what you are seeking to measure
• You will never need to assert the equivalence of the same string across multiple programming languages
• You strongly guard against the presence of any character that would throw the calculation off

A more realist stance is that if your code operates on strings (measuring length, indexing, trimming, splitting), it should be ready to handle the complexity that comes with it - this is espcially important for content that is user-submitted.

Digging Deeper

The solution here is to break down what is really a string of bytes into chunks that make sense for humans – these are called “grapheme clusters” in the Unicode specification. However, doing so isn’t a simple matter of accounting for characters that take twice or four times the length of characters in the ASCII range. Unicode defines modifiers and zero width joiners which are used to modify and link the semantics of another codepoint.

For instance, the single emoji 👨🏿‍🦲is actually a combination of three different codepoints and one “joiner” character that serves as an hint to implementors that the codepoints on either side of it participate to the same cluster:

In a language such as Go, this single character would be reported as having a length of 15!

Unicode defines grapheme clusters as part of the standard: these are rules for segmenting strings into chunks that map well to what a human would consider to be characters in a string. Most high-level languages such as Java, Objective-C and Swift expose this higher-level abstraction:

• Java’s documentation on character representations, and especially String.codepointAt(index), Character.isHighSurrogate, isLowSurrogate, isSupplementaryCodePoint
• Objective-C’s NSString and especially rangeOfComposedCharacterSequenceAtIndex
• Swift’s documentation on strings and characters documentation
• Go documentation on strings and normalization