Ten changes for a steadier booking service

4 sections, 10 ideas

Checkout carries the risk

The booking service is a Go service over Postgres that the rider app and the harbour office both call. Four people build it, with agents writing most of the code.

These ideas come from reading all of it: 212 files, about 31,000 lines of Go and SQL, and the last 90 days of traces and incidents. Most of the pain sits on one path, from search to a paid booking, and most incidents began with a change to that path.

Effort is agent time, not calendar time.

The path from search to a paid booking. Holds live inside one process, and payments retry without a key.

Mermaid source

flowchart LR
  app[Rider app] --> api[Gateway]
  api --> search[Search]
  api --> checkout[Checkout]:::accent
  search --> fares[Fares]:::violet
  checkout --> fares
  checkout --> holds[Seat holds]:::risk
  checkout --> pay[Payments]:::risk
  fares --> db[(Postgres)]
  holds -.-> mem[(In-process map)]:::risk
  pay -->|retries, no key| card[Card provider]

Where checkout slows down

At the Friday evening peak, a booking takes 3.2 seconds from tapping Book to the confirmation, and one attempt in forty times out. Payments and fares account for most of the wait.

Median milliseconds per checkout step, Friday evening peak

fares/quote.go: one query for every leg of the trip

func (q *Quoter) Quote(ctx context.Context, legs []Leg) (Quote, error) {
	var total Money
	for _, leg := range legs {
		fare, err := q.store.FareFor(ctx, leg.RouteID, leg.Class)
		if err != nil {
			return Quote{}, err
		}
		total = total.Add(fare.Price(leg.Party))
	}
	return Quote{Total: total}, nil
}

3.2 seconds Median from Book to the confirmation at the Friday peak.

1 in 40 Checkout attempts that time out at the same peak.

7 of 11 Incidents in 90 days that began with a change to checkout.

No tests Cover the payment retry path or the seat hold expiry.

A retry can charge twice

When the card provider is slow, checkout retries the charge without an idempotency key. Twice in 90 days a rider paid twice for one booking. Idea 2 closes this whichever way the cycle goes.

Ideas

At a glance10 ideas

#	Idea	Size	Effort	Depends on
1	Quote every leg in one query	minor	S	nothing
2	An idempotency key on every charge	minor	S	nothing
3	Typed errors from the payments client	minor	S	nothing
4	One request id in every log line	minor	S	nothing
5	Contract tests for the fares API	minor	M	nothing
6	Free seat holds when checkout gives up	minor	S	nothing
7	Cache quotes for thirty seconds	major	M	1
8	Seat holds in a lease table	major	L	nothing
9	Refunds on a queue with retries	major	L	2, 3
10	Ship checkout changes behind flags	major	M	nothing

Effort is agent time, not calendar time: S is under an hour, M is a few hours with review, L is a day or more across sessions.

Quote every leg in one query

Fares for a whole trip load in one query instead of one per leg, which removes most of the wait in the fares step.

minorEffort S impact 4/5

How it works: FareFor becomes FaresFor in fares/store.go, and Quote reads from the map it returns:

SELECT route_id, class, price FROM fares WHERE (route_id, class) IN (SELECT * FROM unnest($1::text[], $2::text[]))
Why: Fares is the second slowest step, and almost all of it is round trips. A return trip for a family of four makes eight queries today.
In use: The trace for a return booking shows one fares query, and the step drops to about 90 ms.
Unlocks: Idea 7 caches the same map.

An idempotency key on every charge

Each checkout sends one key with its charge and every retry, so a slow card provider can never charge a rider twice.

minorEffort S impact 5/5

Typed errors from the payments client

The payments client returns declined, retryable, and fatal errors as values the caller can test, not messages it has to match.

minorEffort S impact 3/5

One request id in every log line

A request id from the gateway reaches every log line and outgoing call, so one checkout reads as one story.

minorEffort S impact 3/5

Contract tests for the fares API

Recorded app requests pin what the rider app expects from the fares endpoint, so CI fails before a change breaks the app.

minorEffort M impact 3/5

Free seat holds when checkout gives up

A failed or abandoned checkout frees its held spaces at once instead of waiting fifteen minutes for them to expire.

minorEffort S impact 3/5

Cache quotes for thirty seconds

Search, the trip screen, and checkout share one short-lived quote, so a rider's fare is worked out once per trip.

majorEffort M impact 4/5

Seat holds in a lease table

Holds move from a map inside one process to a Postgres table, so deploys keep them and two instances can take bookings.

majorEffort L impact 5/5

Refunds on a queue with retries

Refunds leave the request path for a worker that retries with backoff and records every attempt.

majorEffort L impact 4/5

Ship checkout changes behind flags

A checkout change can go out switched off, reach a share of riders, and be turned off again without a rollback.

majorEffort M impact 4/5

Also considered

Rewrite checkout in Rust

The time goes to round trips and the card provider, not to Go, so a rewrite would move none of it.

Split booking into microservices

Four people run this, and more services would add the very network hops that slow checkout.

A GraphQL gateway for the apps

The app's calls are few and cheap, and idea 7 removes the repeated ones.

Your reply

Your choices, ready to send. Copy this reply back to your agent when you’re done.

Example speed, 2, 5, 7. Notes: 5: smaller first.

Choice: A faster checkout. Pick: ideas 2, 5, and 7. Note on idea 5: smaller first.

nothing.

Nothing decided yet.