Building a Payment Service in Go

Overview

Payment services are one of those systems where getting it wrong costs real money — literally. In this post I walk through the architecture decisions I made while building a backend payment processing service in Go, focusing on state machines, idempotency, database design, and observability.

This is a write-up of the Go Payment Infrastructure project, which I built as a production-grade exploration of payment orchestration.

State Machine: The Core of Payment Reliability

The most important architectural decision in a payment service is treating every payment as a state machine. A payment should only ever move forward through a defined set of states, and no transition should be possible unless the current state allows it.

type PaymentStatus string
 
const (
    StatusPending    PaymentStatus = "pending"
    StatusProcessing PaymentStatus = "processing"
    StatusSucceeded  PaymentStatus = "succeeded"
    StatusFailed     PaymentStatus = "failed"
    StatusRefunded   PaymentStatus = "refunded"
)
 
var allowedTransitions = map[PaymentStatus][]PaymentStatus{
    StatusPending:    {StatusProcessing, StatusFailed},
    StatusProcessing: {StatusSucceeded, StatusFailed},
    StatusSucceeded:  {StatusRefunded},
    StatusFailed:     {},
    StatusRefunded:   {},
}
 
func (p *Payment) CanTransition(to PaymentStatus) bool {
    allowed := allowedTransitions[p.Status]
    for _, s := range allowed {
        if s == to {
            return true
        }
    }
    return false
}

This prevents impossible state transitions — you can never mark a failed payment as succeeded, or move a refunded payment back to pending.

Idempotency: Retry-Safe by Design

Payment APIs are called from clients that may retry on network failure. Without idempotency, a retry could charge a user twice. The solution is an idempotency key — a unique client-supplied token stored alongside every payment request.

func (s *PaymentService) CreatePayment(ctx context.Context, req CreatePaymentRequest) (*Payment, error) {
    // Check if a payment with this idempotency key already exists
    existing, err := s.repo.FindByIdempotencyKey(ctx, req.IdempotencyKey)
    if err == nil && existing != nil {
        // Return the existing payment — safe to retry
        return existing, nil
    }
 
    payment := &Payment{
        ID:             newID(),
        IdempotencyKey: req.IdempotencyKey,
        Amount:         req.Amount,
        Currency:       req.Currency,
        Status:         StatusPending,
        CreatedAt:      time.Now(),
    }
 
    return s.repo.Create(ctx, payment)
}

The idempotency key is stored with a UNIQUE constraint in Postgres, so even if two concurrent requests arrive at the same time, only one will succeed at the database level.

Optimistic Concurrency Control

To handle concurrent updates safely without locking rows, I used optimistic concurrency control via a version column.

// Postgres schema
CREATE TABLE payments (
    id          UUID PRIMARY KEY,
    status      TEXT NOT NULL,
    amount      BIGINT NOT NULL,
    currency    TEXT NOT NULL,
    version     INT NOT NULL DEFAULT 0,
    created_at  TIMESTAMPTZ NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL
);

func (r *paymentRepo) UpdateStatus(ctx context.Context, id string, newStatus PaymentStatus, currentVersion int) error {
    result, err := r.db.ExecContext(ctx, `
        UPDATE payments
        SET status = $1, version = version + 1, updated_at = NOW()
        WHERE id = $2 AND version = $3
    `, newStatus, id, currentVersion)
 
    if err != nil {
        return err
    }
 
    rows, _ := result.RowsAffected()
    if rows == 0 {
        return ErrConcurrentUpdate // Another process won, retry
    }
    return nil
}

If two goroutines try to update the same payment simultaneously, only one will match the version check. The other receives ErrConcurrentUpdate and can safely retry.

Database Schema Design

CREATE TABLE payments (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key TEXT UNIQUE NOT NULL,
    status          TEXT NOT NULL DEFAULT 'pending',
    amount          BIGINT NOT NULL,      -- store in smallest unit (paise, cents)
    currency        TEXT NOT NULL,
    provider        TEXT NOT NULL,        -- 'stripe', 'razorpay', etc.
    provider_ref    TEXT,                 -- external payment ID
    metadata        JSONB,
    version         INT NOT NULL DEFAULT 0,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
 
CREATE INDEX idx_payments_status ON payments(status);
CREATE INDEX idx_payments_created_at ON payments(created_at DESC);

Key decisions:

Amount as integer — never store money as a float. Store in the smallest currency unit.
JSONB metadata — flexible field for provider-specific data without schema changes.
Indexed on status + created_at — the two most common query patterns.

Observability

Every payment operation emits structured logs with a consistent set of fields:

log.Info("payment status transition",
    "payment_id", payment.ID,
    "from_status", payment.Status,
    "to_status", newStatus,
    "amount", payment.Amount,
    "currency", payment.Currency,
    "duration_ms", time.Since(start).Milliseconds(),
)

I also exposed Prometheus metrics for:

payment_transitions_total — counter by from_status and to_status
payment_processing_duration_seconds — histogram
payment_failures_total — counter by failure reason

This makes it trivially easy to set up a Grafana dashboard that shows payment funnel drop-off, p99 processing latency, and failure rates at a glance.

Tradeoffs & What I Would Change

Decision	Why	Tradeoff
Optimistic concurrency	Avoids row locks under normal load	Requires retry logic in callers
Idempotency at service layer	Simple, explicit	Requires clients to generate stable keys
Postgres only	Simple ops, ACID guarantees	Harder to scale writes beyond ~10k TPS
State machine in app layer	Easy to test, readable	Not enforced at DB level

For a real production system, I would also add:

A dead letter queue for failed webhook deliveries
Distributed tracing with OpenTelemetry
A reconciliation job that catches stuck processing payments

Summary

Building a payment service is mostly about making failures boring. Idempotency means retries are safe. State machines mean invalid transitions are impossible. Optimistic concurrency means concurrent updates are handled gracefully. Structured logging and Prometheus metrics mean incidents have enough context to debug quickly.

The code for this project is on GitHub.

Tags