downstack

Writing

What broke, and what it cost to find out. Every number here carries its date and its source.

01
published by downstack
08
published elsewhere

RSS

Published by downstack

What gets published here

The rules this site is written under, and what has to be true before anything goes up.

downstack.devEditorial

Subscribe by RSS

Published elsewhere

Eight posts about Courier, written by members of the team and published on someone else’s site.

These are not downstack publications. They were published by gojek.io on the dates shown, and every entry links to the original there.

2021Four posts

  • Introducing Courier: The Information Superhighway Between Mobile and Server

    Anubhav Guptagojek.ioopens on gojek.io

    About a 50 percent reduction in network consumption by the merchant app after rollout, with merchants working to a three minute window to accept an order.

    gojek.io, 2021-08-10

    downstackThe architectural argument for choosing MQTT over gRPC and WebSocket on mobile, decided on battery and background execution rather than on throughput.

  • The Quest For A Message Broker For Our Courier

    Ajat Prabhagojek.ioopens on gojek.io

    Over 300,000 connected clients at over 200 messages per second on four GCP n1-standard-2 machines, each 2 vCPU and 7.5 GB.

    gojek.io, 2021-08-20

    downstackA broker evaluation across five candidates, measured rather than argued. It is five years old, so read it for the method and not for the conclusion.

  • Courier Library For Gojek's Information Superhighway

    Deepanshugojek.ioopens on gojek.io

    In Doze mode Android will not fire an app's alarm more than once every 9 minutes, which is what the ping sender has to be built around.

    gojek.io, 2021-10-01

    downstackThe Android client, where the operating system sets the keepalive and not the protocol. This is the half of the problem broker vendors do not write about.

  • Adaptive Heartbeats For Our Information Superhighway

    Deepanshugojek.ioopens on gojek.io

    Four search strategies compared for finding the keepalive interval a network will tolerate, binary, exponential, composite and linear, bounded below at 4 minutes.

    gojek.io, 2021-11-16

    downstackA fixed keepalive interval is a guess, and on mobile it is usually the wrong one. The interval a carrier NAT will tolerate is discoverable per network at runtime.

2022Two posts

  • Acing 1 Million Concurrent MQTT Connections

    Dhruv Jaingojek.ioopens on gojek.io

    Ephemeral port exhaustion at 28,200 per worker VM. EMFILE at 40,000. HAProxy port exhaustion at 113,000. A VerneMQ listener ceiling near 362,000. An HAProxy maxconn cap near 710,000. One million connections reached with no connect drops, at 4,680 messages per second.

    gojek.io, 2022-04-15 ยท load test, not a production peak

    downstackThe one that names what broke rather than what worked. Five ceilings, four of them below the broker. If you read one thing on this page, read this.

  • Courier Is Now Open-Source

    Deepanshugojek.ioopens on gojek.io

    Three client libraries open sourced for long-running MQTT connections, courier-android, courier-iOS and courier-go, documenting all three QoS levels (0 at most once, 1 at least once, 2 exactly once) and the events for tracking end-to-end delivery and connection lifecycle.

    gojek.io, 2022-07-08

    downstackThe post that put the client libraries in public. It is the reason the rest of this list is checkable rather than only assertable: the code the other posts describe is code you can read.

2023Two posts

  • Courier: Reimagining How We Send Push Notifications

    Deepanshugojek.ioopens on gojek.io

    FCM and APNs alone measured 75 to 85 percent delivery. Over the MQTT path with push retained as fallback: 95 percent or better on the consumer app, 99 percent or better on driver and merchant apps.

    gojek.io, 2023-10-23

    downstackPush is a backstop, not a channel. The number that matters is not what the vendor reports as sent, it is what the device confirms it received.

  • Evolution of Bid Notifications to Courier

    Ajat Prabhagojek.ioopens on gojek.io

    99.9 percent or better delivery against 97 to 99 percent before, and about a 22 percent reduction in P99 bid acceptance latency.

    gojek.io, 2023-12-08

    downstackMigration method on a flow where a late message is a lost job: tier 3 cities first, then tier 2, then tier 1, with fallback kept on every hop.

Every one of these was self-published by the platform team that wrote it. Read them for the figures and the failure modes rather than for the conclusions.