System Design: From Requirements to a Reliable Workflow

TL;DR

System Design means deciding how the parts of a software system work together to meet a product’s needs. It connects the user’s request to data, processing, permissions and an observable outcome, including what happens when something fails.

• Define the promises: who can do what, how long it can take and what must survive.
• Choose responsibilities and contracts before choosing infrastructure.
• Make repetition, limited capacity and recovery part of the design.

We will design a fictional expense-report workflow. Someone uploads receipts, the system extracts fields, and a draft report becomes available for review. It does not approve expenses or move money. REPORT-42 is an invented example, not a company record.

1. Start with the promise to the user

“Process receipts” leaves important decisions open. Can a user close the browser and come back? Must files stay within their organization? What does a rejected upload look like? Does the draft require a person to check extracted amounts? Answers shape the design.

For this example, accepted work should survive a worker restart, users should see progress, and each organization should access only its own documents. Record limits for file size, supported formats and retention. Agree on a completion-time target and how to measure it; avoid inventing a percentage simply because it looks impressive. Google’s service-objective guidance connects useful targets to user-facing behavior.

Separate requirements from assumptions. Expected traffic is an estimate until measured. A proposed recovery mechanism is a plan until implemented and tested. Keep that distinction in the design document so a diagram cannot quietly become a reliability claim.

2. Give each part a clear responsibility

Start with an application that owns the upload, access checks and report status; durable storage for files and records; and a worker, a process that performs background tasks. These can share one codebase. File parsing belongs behind a clear interface so the page does not need to understand every file format.

An API, or application programming interface, is a contract for communication. Here it specifies valid input, report identity, possible states and errors. The application derives organization ownership from the signed-in session. A report ID is a lookup key, never permission to read a document. Check access on status and download requests too. OWASP recommends authorization checks on every request.

Keep large files in file storage and pass references to background work. Define who owns the authoritative report state and how it changes. A single clear owner is easier to reason about than several components independently deciding that a report is complete.

3. Separate acceptance from completion

Receipt extraction may take longer than a useful browser request. Return a report ID once the application has durably accepted responsibility, then expose its status separately. An accepted response means work was recorded; it does not mean the report is finished. The page can display queued, processing, completed or failed. Microsoft describes this separation in the asynchronous request-reply pattern.

Saving a database record and publishing a queue message are two writes. If the second fails, the report can remain stuck. One option is a transactional outbox: save the report and a dispatch record in one database transaction; a separate relay publishes pending work. Publication can repeat, so consumers still need duplicate protection. AWS documents the dual-write problem and this recovery pattern.

File upload needs its own lifecycle. Confirm the file is available before accepting work, and expire abandoned uploads. Record a useful failure when processing cannot continue. Refreshing the page should retrieve existing work, not silently create another report.

4. Make failure and repetition explicit

Suppose the result is saved, but the worker crashes before acknowledging the message. The queue may deliver it again. Idempotency means that repeating the same logical operation preserves its intended business effect.

Give a submission an operation key scoped to its owner. Match that key to the original input, define how long it is retained, and use a transaction or conditional update to protect the result against concurrent attempts. A preliminary “does it exist?” check alone can race with another worker. Amazon’s idempotency guidance covers request identity, changed intent and late retries.

A timeout means the caller stopped waiting; the remote operation may have succeeded. Before repeating an external effect, use the provider’s idempotency support or reconcile its status. Retry temporary failures with a deadline, an attempt limit and increasing delays with small random variation, called jitter. Invalid files need correction, not endless retries. Amazon explains how retries can amplify an outage without these controls.

5. Decide what happens under load

When many reports arrive together, a queue can buffer work while workers process at a controlled pace. It also creates waiting time. If arrivals keep exceeding completions, the backlog grows; a queue does not create processing capacity. Microsoft’s load-leveling pattern explains this buffering tradeoff.

Bound concurrent work and file expansion in memory. Combine admission limits with per-organization concurrency limits or fair scheduling so one large batch cannot consume every slot. Protect shared dependencies across all worker instances. When the system is saturated, delay or reject new work explicitly and explain the next step to the user. This is backpressure: feeding limited capacity back to producers instead of accepting an unlimited promise.

6. Check the promises in operation

Measure request failures, time to a usable report, oldest pending job and resource saturation. Separate queue waiting from processing time so a fast parser does not hide a long wait. Google’s monitoring guidance organizes signals around latency, traffic, errors and saturation. Correlate request and job events without logging receipt contents, access tokens or unnecessary personal data. OWASP describes what sensitive information to exclude from logs.

Exercise the boundaries before relying on them:

• Crash after saving a result, then redeliver the job and inspect the final state.
• Interrupt queue publication and verify that recorded work is eventually dispatched.
• Submit a burst of large files and inspect waiting time, memory and rejection behavior.
• Request another organization’s report and verify that no data is returned.

If extraction uses artificial intelligence, test the quality of its fields separately. Reliable delivery does not establish that an extracted amount is correct. The AI evaluations guide explains how to define and check that kind of expected behavior.

7. Add complexity for a reason

Begin with the smallest design that meets the requirements. A modular application and a worker can be enough. Separate services can provide independent scaling and releases, but introduce network failures, deployment coordination and harder data consistency. Microsoft’s architecture guide describes these microservices tradeoffs.

Frequently asked questions

Sources & further reading

The primary research behind the article, for readers who want to explore the details.

1. Service Level Objectives Google SRE Book
2. Authorization Cheat Sheet OWASP
3. Asynchronous Request-Reply pattern Microsoft Azure Architecture Center
4. Transactional outbox pattern AWS Prescriptive Guidance
5. Making retries safe with idempotent APIs Amazon Builders’ Library
6. Timeouts, retries, and backoff with jitter Amazon Builders’ Library
7. Queue-Based Load Leveling pattern Microsoft Azure Architecture Center
8. Monitoring Distributed Systems Google SRE Book
9. Logging Cheat Sheet OWASP
10. Microservices architecture style Microsoft Azure Architecture Center