Cadence: Why Top Engineers Ditch Crons for This Workflow Engine

Cadence: Why Top Engineers Ditch Crons for This Workflow Engine

What if your most critical business process silently failed at 3 AM—and nobody noticed until customers started screaming?

Here's the brutal truth: most developers are still duct-taping their long-running operations together with cron jobs, fragile queues, and prayer. A payment reconciliation that spans six external APIs? A user onboarding flow that touches a dozen microservices? These aren't simple request-response cycles. They're orchestration nightmares that traditional infrastructure was never designed to handle.

When one service hiccups, your entire chain collapses. No automatic retries. No state visibility. No recovery path. Just you, frantically grep-ing through logs at 4 AM, hoping to reconstruct what happened.

But what if there was a battle-tested system that made these complex, asynchronous workflows indestructible by design?

Enter Cadence—the open-source workflow engine born at Uber that quietly powers some of the most demanding distributed systems on the planet. Since 2017, Cadence has been the secret weapon engineers reach for when "good enough" infrastructure crumbles under real-world load. And here's what should make you pay attention: it's not just surviving at scale. It's thriving where everything else falls apart.

Ready to understand why teams are abandoning their homegrown workflow hacks? Let's pull back the curtain.

What Is Cadence?

Cadence is a distributed, scalable, durable, and highly available orchestration engine designed to execute asynchronous long-running business logic in a resilient way. Originally developed by Uber Engineering and donated to the Cloud Native Computing Foundation (CNCF), Cadence solves a deceptively simple problem that breaks most systems: how do you reliably coordinate work that takes minutes, hours, or even days to complete?

The repository at github.com/cadence-workflow/cadence contains the core orchestration engine plus essential tooling: a command-line interface, schema management utilities, benchmark suites, and canary health-check workflows. This isn't a toy project or a proof-of-concept. It's production-hardened infrastructure that has processed billions of workflows across Uber's ride-sharing, food delivery, and freight platforms.

What makes Cadence genuinely different from a standard job scheduler or message queue? Stateful orchestration with automatic recovery. Traditional systems treat each task as an independent fire-and-forget operation. Cadence treats your entire multi-step process as a durable entity that survives crashes, maintains execution history, and automatically resumes exactly where it left off.

The timing couldn't be better for broader adoption. As microservices architectures proliferate, the "death by a thousand cuts" problem intensifies—every new service adds integration complexity, failure modes multiply exponentially, and operational visibility fragments across dozens of dashboards. Cadence centralizes orchestration logic, making complex flows observable, debuggable, and testable as first-class code rather than scattered configuration.

With official Go and Java SDKs, plus community-driven Python^{↗ Bright Coding Blog} and Ruby support, Cadence fits naturally into modern polyglot environments. The recent Kubernetes deployment improvements through KubeStellar Console and official Helm charts have dramatically lowered the operational barrier to entry.

Key Features That Separate Cadence from the Pack

Let's dissect what makes Cadence architecturally distinctive—and why engineers who've battled production fires become immediate evangelists.

Durable Execution State

Cadence automatically persists workflow state at every step. Server crash? Network partition? Deployment rolling restart? Your workflow resumes exactly where it paused, with full context intact. No manual checkpointing. No lost progress. This isn't optimistic recovery—it's guaranteed recovery.

Fault-Tolerant by Design

Every activity in your workflow gets automatic retries with configurable backoff strategies. Timeouts are explicit and enforceable. When external services fail—and they will—Cadence handles the transient fault patterns so your business logic stays clean. You write the happy path; Cadence manages the chaos.

Long-Running Native Support

Unlike systems that treat long duration as an edge case, Cadence optimizes for it. Workflows can run for days or weeks without holding resources. Activities execute on separate worker pools, decoupling scheduling from execution. This isn't bolted-on functionality; it's the foundational architecture.

Scalable Worker Architecture

Workers are horizontally scalable stateless processes. Add more instances to increase throughput. Remove them for maintenance. Cadence's task routing automatically redistributes work. The backend services themselves shard across your persistence layer for virtually unlimited scale.

Complete Operational Visibility

The built-in Web UI at localhost:8088 exposes execution histories, decision trails, and detailed traces. Every state transition is recorded. Every retry is logged. When something goes wrong—and eventually something always does—you have forensic-level detail without instrumenting a single line of application code.

Multi-Language SDK Ecosystem

Official Go and Java clients provide full feature parity. Community Python and Ruby SDKs extend reach. The iWF DSL framework from Indeed adds higher-level abstractions for teams wanting declarative workflow definitions. You're never locked into a single language stack.

Production-Grade Tooling

The CLI isn't an afterthought—it's a comprehensive operational interface organized by command hierarchies (workflow → batch → start, admin → workflow → describe). Schema tools handle database migrations for Cassandra, MySQL^{↗ Bright Coding Blog}, and PostgreSQL^{↗ Bright Coding Blog}. Benchmark and canary utilities validate performance and feature health continuously.

Real-World Use Cases Where Cadence Dominates

Theory is cheap. Let's examine where Cadence delivers transformational value against conventional approaches.

1. Financial Transaction Processing

Imagine a cross-border payment flow: validate sender, check sanctions lists, reserve funds, call FX service, notify recipient bank, confirm settlement. Seven services. Three time zones. Regulatory timeouts measured in business days. With cron jobs, any failure creates a reconciliation nightmare. With Cadence, each step is an activity with explicit timeouts and retries. The workflow state persists across days. Auditors get complete execution histories automatically.

2. E-Commerce Order Fulfillment

From inventory reservation through payment capture, warehouse picking, shipping label generation, carrier handoff, and delivery confirmation—modern order pipelines span dozens of internal and external systems. Cadence orchestrates these with saga-pattern compensation: if payment succeeds but inventory is unavailable, automatic rollback sequences execute. No orphaned charges. No angry customers.

3. Data Pipeline Orchestration

ETL workflows that extract from multiple sources, transform through sequential stages, and load to warehouses often run for hours. Traditional schedulers fail catastrophically when a mid-pipeline stage errors. Cadence enables fine-grained restartability: retry just the failed transformation, preserve completed upstream work, and maintain data lineage for compliance.

4. Infrastructure Provisioning and Lifecycle Management

Cloud resource provisioning—VPC creation, subnet allocation, security group configuration, instance deployment, load balancer attachment—must happen in strict order with verification gates. Cadence workflows encode these dependencies explicitly, with automatic cleanup on failure. Platform teams use this for self-service infrastructure APIs that actually work reliably.

5. Human-in-the-Loop Business Processes

Loan approvals, insurance claims, content moderation—these require automated processing interleaved with human decisions. Cadence supports asynchronous completions: a workflow can pause for hours or days waiting for human input, consuming zero execution resources, then resume instantly when signaled. Try that with your queue-based system.

Step-by-Step Installation & Setup Guide

Getting Cadence running locally takes minutes. Production deployment requires more planning, but the fundamentals are straightforward.

Prerequisites

• Docker^{↗ Bright Coding Blog} and Docker Compose installed
• Go 1.20+ or Java 11+ (for SDK development)
• One of: Cassandra, MySQL, or PostgreSQL (included in Docker Compose)
• Optional: Kafka and Elasticsearch for advanced visibility features

Local Development Setup

The fastest path to a running Cadence instance uses the provided Docker Compose configuration:

# Clone the repository
git clone https://github.com/cadence-workflow/cadence.git
cd cadence

# Start all backend services with default configuration
docker compose -f docker/docker-compose.yml up

This single command launches:

• Cadence frontend, matching, history, and worker services
• Cassandra as the default persistence store
• Elasticsearch for advanced search capabilities
• The Cadence Web UI

Once services report healthy, verify at http://localhost:8088.

CLI Installation

Multiple installation paths suit different environments:

# macOS via Homebrew (recommended for local development)
brew install cadence-workflow

# Docker-based execution (ideal for CI/CD pipelines)
docker run --rm ubercadence/cli:master

# Build from source for custom modifications
git clone https://github.com/cadence-workflow/cadence.git
cd cadence
make cadence  # Builds all tools including CLI

Database Schema Setup

For manual database initialization or upgrades:

# Schema tools install with the Homebrew package
cadence-sql-tool --help
cadence-cassandra-tool --help

# Schema files location (Homebrew installation)
ls /usr/local/etc/cadence/schema/

Critical upgrade note: When updating Elasticsearch schemas, first archive the old version to prevent conflicts:

mv /usr/local/etc/cadence/schema/elasticsearch /usr/local/etc/cadence/schema/elasticsearch.old
brew upgrade cadence-workflow

Kubernetes Production Deployment

For production environments, use the official Helm chart via KubeStellar Console:

1. Navigate to KubeStellar Console's Cadence mission
2. Execute the guided installation with pre-flight checks
3. Validate deployment health with built-in troubleshooting
4. Leverage rollback support for safe upgrades

The Helm charts reside at github.com/cadence-workflow/cadence-charts for custom configuration.

Worker Implementation Setup

With backend running, implement your workflow logic using official SDKs:

# Go SDK
go get github.com/cadence-workflow/cadence-go-client

# Java SDK (Maven)
<dependency>
    <groupId>com.uber.cadence</groupId>
    <artifactId>cadence-client</artifactId>
    <version>LATEST</version>
</dependency>

REAL Code Examples from the Repository

Let's examine practical patterns using actual repository documentation and standard Cadence implementations.

Example 1: Docker Compose Local Launch

The repository's canonical quickstart demonstrates Cadence's operational simplicity:

# From repository root - launches complete local stack
docker compose -f docker/docker-compose.yml up

What's happening under the hood? This orchestrates multiple containerized services: the Cadence frontend handles API requests, the matching service routes tasks to workers, the history service persists execution state, and worker services process internal system workflows. Cassandra provides durable storage; Elasticsearch enables workflow search. The Web UI container exposes port 8088 for visualization. No manual service wiring required—the Compose file encodes all inter-service dependencies and health checks.

Example 2: CLI Workflow Operations

The Cadence CLI organizes commands hierarchically. Explore capabilities incrementally:

# Top-level help reveals command structure
cadence --help

# Drill into workflow operations
cadence workflow --help

# Batch operations for bulk management
cadence workflow batch --help

# Start a batch operation (example pattern)
cadence workflow batch start \
    --domain samples-domain \
    --query "WorkflowType = 'MyWorkflow'" \
    --reason "rerun failed workflows" \
    --batch-type terminate

Critical insight: The tab-completion-friendly structure (workflow → batch → start) mirrors Cadence's domain model. This isn't accidental—it's designed for operational muscle memory during incident response. The --help flag works at every level, making exploration self-documenting. For admin operations like inspecting internal workflow state:

# Describe workflow execution with full history
cadence admin workflow describe \
    --domain samples-domain \
    --workflow_id my-workflow-123 \
    --run_id 7f3e2a1b-...

Example 3: Docker-Based CLI Execution

For environments without local installation, the official CLI image provides identical functionality:

# Execute specific release version (deterministic builds)
docker run --rm ubercadence/cli:v1.2.3 workflow list \
    --domain samples-domain

# Track master for latest features (development only)
docker run --rm ubercadence/cli:master workflow list \
    --domain samples-domain

# Update to latest master image
docker pull ubercadence/cli:master

Production discipline: Pin to specific release versions for reproducible operations. The master tag receives continuous updates—valuable for feature evaluation, dangerous for production scripts. The --rm flag ensures container cleanup after execution, preventing disk accumulation in automated systems.

Example 4: Schema Tool Operations

Database schema management uses dedicated tools for each persistence backend:

# For Cassandra deployments
cadence-cassandra-tool setup-schema \
    --datacenter dc1 \
    --replication-factor 3 \
    --keyspace cadence

# For SQL deployments (MySQL/PostgreSQL)
cadence-sql-tool setup-schema \
    --dbtype mysql \
    --host localhost \
    --port 3306 \
    --user cadence \
    --password secret \
    --database cadence

Operational note: Schema tools handle versioned migrations automatically. The setup-schema command initializes fresh databases; update-schema applies incremental changes. For zero-downtime upgrades in production, Cadence supports schema versioning that allows mixed-version server fleets during rolling deployments.

Advanced Usage & Best Practices

After mastering fundamentals, apply these patterns from production deployments:

Worker Tuning for Throughput

Configure WorkerOptions with MaxConcurrentActivityExecutionSize and MaxConcurrentDecisionTaskExecutionSize matched to your host resources. Over-subscription causes context switching overhead; under-utilization wastes capacity. Profile with the benchmark tools in ./bench/.

Workflow Idempotency Design

Design workflow IDs as deterministic business keys (e.g., order-12345-onboarding) rather than UUIDs. This enables natural deduplication—restarting with the same ID resumes existing execution rather than creating duplicates.

Activity Timeout Strategy

Set three timeout tiers: ScheduleToClose (total SLA), ScheduleToStart (queue latency), and StartToClose (execution duration). This granularity distinguishes infrastructure problems from business logic failures.

Signal and Query Patterns

Use Signals for external events that advance workflow state ("payment received"). Use Queries for read-only inspection without affecting execution ("what's current status?"). Never mix these semantics—queries must be side-effect free.

Canary Deployment Validation

Leverage the built-in canary workflows (./canary/) to validate new server versions against production-like patterns before full rollout. These exercise critical paths continuously, surfacing regressions within minutes.

Comparison with Alternatives

Why Cadence specifically? If you need proven hyperscale (Uber's billions of workflows), CNCF governance preventing vendor capture, or direct control over every infrastructure layer without cloud lock-in, Cadence delivers. Temporal offers newer SDKs and active commercial support—evaluate both if those dimensions matter. Airflow excels at data engineering DAGs but lacks general workflow durability. Step Functions trades control for convenience.

FAQ

Q: Is Cadence the same as Temporal? Temporal forked from Cadence in 2020. They share core concepts but diverged architecturally. Cadence remains CNCF-governed; Temporal is a venture-backed company. Both are production-viable—choose based on governance preferences and specific feature needs.

Q: What's the maximum workflow execution time? Practically unlimited. Cadence workflows have run for months in production. Duration is bounded only by your persistence retention policies, not architectural constraints.

Q: Can I migrate from my existing cron-based system? Yes, incrementally. Start by wrapping cron jobs in Cadence activities for retry and visibility benefits. Gradually refactor into full workflows as you gain confidence.

Q: How does Cadence handle worker failures during activity execution? Activities time out based on configured StartToClose thresholds. Cadence automatically reschedules to healthy workers with configurable retry policies. Failed activities never silently disappear.

Q: Is the Web UI suitable for production operations? Absolutely. The UI at github.com/cadence-workflow/cadence-web provides execution history, decision tracing, and stack dump inspection. Many teams use it as their primary incident response interface.

Q: What database should I choose for persistence? Cassandra offers highest write throughput for massive scale. PostgreSQL and MySQL simplify operational expertise if your team lacks Cassandra experience. Benchmark your expected load.

Q: How do I get help with production issues? Join the #cadence-users channel on CNCF Slack, search StackOverflow, or open detailed issues on GitHub.

Conclusion

The infrastructure landscape is littered with tools that promise resilience but deliver complexity. Cadence is the rare exception: a system that genuinely simplifies building reliable long-running workflows while scaling to virtually unlimited demand.

After seven years of production hardening at Uber and beyond, the patterns are proven. The community is active. The CNCF governance ensures long-term accessibility. Whether you're orchestrating financial transactions, data pipelines, or human-in-the-loop processes, Cadence provides the durable foundation that lets you focus on business logic instead of failure recovery.

The alternative? Keep waking up to failed cron jobs, orphaned processes, and angry stakeholders. Or make the switch that engineering leaders at the most demanding companies already have.

Start today: clone github.com/cadence-workflow/cadence, run docker compose -f docker/docker-compose.yml up, and experience workflow orchestration that actually works when everything else breaks. Your future self—the one sleeping through the night while Cadence handles the 3 AM failures—will thank you.

Ready to dive deeper? Explore the official documentation, contribute to the docs project, or watch Maxim's architectural deep-dive from the Data@Scale Conference.