Jepsen Found Orkes Conductor Marking Workflows Complete That Never Ran

Jepsen's analysis of Orkes Conductor, published October 20, found workflows that were acknowledged, then abandoned in RUNNING forever, and others marked COMPLETED though their tasks never ran. Orkes shipped 5.5.6 with fixes for four of the nine findings and had fixed two more in 5.5.0.

Kyle Kingsbury, who runs Jepsen, tested Conductor versions 5.3.2 through 5.5.0-rc2. The report says Orkes Inc. funded the work under Jepsen's ethics policy. Orkes says seven of the nine issues are closed; Jepsen's own table lists two as still under investigation. The tested configuration used Postgres for storage and Redis for queues.

The setup and the one claim that broke

The cluster ran ten Debian nodes, containers and EC2 VMs alike. The workload was one workflow with one task that slept up to 30 seconds, while Jepsen injected pauses, kills and partitions between servers and the databases.

Conductor documents that "If a workflow has started, it will run to completion as long as all its tasks are completed." Jepsen found that sentence false in three cases: findings 5, 7 and 9 below.

Nine findings, and which are fixed

#	Finding	Status
1	Tasks run concurrently on multiple workers	Fixed in 5.5.6
2	Tasks stall up to the response timeout, default 1 hour	Orkes calls it designed behavior
3	startWorkflow returns null IDs after a Redis kill	Under investigation
4	startWorkflow returns an SQL syntax error	Fixed in 5.5.6
5	Acknowledged workflows stuck in RUNNING	Fixed in 5.5.0
6	Successful starts reported as "failed to connect"	Under investigation
7	Visible workflows later disappear	Fixed in 5.5.0
8	startWorkflow returns a duplicate key error	Fixed in 5.5.6
9	Workflows marked COMPLETED with tasks never run	Fixed in 5.5.6

The numbers behind the worst three

Finding 5 came down to a missed enqueue. Workflows were written to Postgres but never placed on the decider queue, and the sweeper that recovers stuck work relied only on that queue, so nothing ever found them.

Finding 7 has the clearest timeline. One workflow was visible from 239 seconds into a run and reported RUNNING until 313 seconds. From then until the test ended at 677 seconds, getWorkflow returned null. Jepsen says the server had deleted the record after failing to enqueue it.

Finding 9 hit three workflows in one run of 5.5.0-rc2. A retry wrote a new task record and marked the old one superseded in two separate Postgres writes with no transaction around them. A failure between the two left no retry task, and the sweeper then marked the workflow COMPLETED.

The duplicates problem

Finding 1 needed no faults at all. With the default one-minute lease, two servers, three workers and about 1,100 workflows a second, two workers ran the same task about once every six hours. Finding 6 is a client behavior. In a short run, 1,752 of 25,920 startWorkflow calls reported as failed had in fact succeeded. The Java client retried a non-idempotent request internally and returned only the retry's error. Orkes changed the client to stop retrying such requests and is considering an isDefinite() method. The fix for duplicate runs is on the caller: Jepsen advises treating every Conductor error as indefinite, since a retry may start a second workflow.

Orkes' own account

In a post the same day, Orkes software engineer Miguel Prieto wrote that the 5.5.0 fix replaced two separate writes with a transactional outbox, so a workflow and its queue entry now commit together. A background process recovers anything that committed but never reached the queue. The 5.5.6 fix, he wrote, closes a time-of-check/time-of-use race in the sweeper that caused extra task deliveries.

Jepsen's table lists six findings as fixed, one as designed behavior and two as under investigation. Orkes wrote that "Workers must be idempotent," because at-least-once delivery is by design. It also changed the Java SDK to throw instead of returning a null workflow ID, though it has not reproduced the root cause.

Related coverage of developer tooling: Orca and OpenCode.

What Jepsen tells operators to do

Upgrade to 5.5.6. Search for workflows sitting in RUNNING for long periods, and for COMPLETED ones with missing task output, and restart them by hand.

Jepsen says it covered only the Java client, single-task workflows, Postgres and single-node Redis. Multi-step workflows, failure workflows, high latency and clock skew were not tested, and the report notes that testing can show bugs but not their absence.

Findings 3 and 6 remain open.