From $1.56M to $55k: Scaling a Regulated-Finance Agentic Workflow

tl;dr

What we delivered

Over the past three months, we worked on two linked problems for the Korean crypto exchange Bithumb: make a regulated-finance deep-research service reliable enough for millions of daily readers, then make the same workflow inexpensive enough to scale with a customized model. Here is what we delivered:

• The product: Bithumb AI is the research layer in Bithumb’s app, turning technical, on-chain, social, news, and macro evidence into Korean market copy. The copy uses the right asset and source hour and follows Bithumb’s Korean financial style.
• The production workflow: At production scale, the service has processed 500M+ market and technical data points, 2M+ generation and validation tasks, and 3M+ Korean quality evaluations, with 99%+ API availability since launch (public launch materials).
• The benchmark: We treat eval as the release control plane: a frozen panel and release harness make each candidate change comparable, while separate scoreboards measure serving, canonical QA, grounding, Korean language, diversity, retries, and token consumption instead of collapsing the product into one quality score.
• The auto-research loop: An autonomous experiment loop proposes bounded recipe changes, launches parallel arms, and uses scorecards and failure profiles to choose the next round. We post-trained a customized Qwen3.8-27B model with SFT and used on-policy DPO to correct its residual failures.
• The payoff: Four stage-specific LoRA adapters on Fireworks H200s cut projected token consumption from 14.790B to 4.843B per 30 days. That moves annualized inference cost from $1.56M to about $55k, saving roughly $1.5M per year.
• Next: We are taking the same post-trained path to a single RTX 5090; the hosted H200 route remains the production path.

01 · product

What Bithumb required

A deep-research report with a release contract

Bithumb AI is the research layer inside Bithumb’s trading app, covering more than 450 KRW trading pairs. It gives users ranked asset signals, per-asset analysis, a daily market brief, and guided questions. Surf and Bithumb built these surfaces together; this article follows the per-asset report path, where fast-moving market, technical, on-chain, social, news, and macro evidence has to become fresh Korean copy under one release contract.

The same evidence discipline applies across all four surfaces:

The formats differ, but each surface still has to preserve evidence, asset identity, Korean financial language, and publication timing. The per-asset report is the clearest example because it fans out across the largest cohort.

What one report looks like

Here is a shortened BTC report example, with the Korean customer copy beside an English translation.

The report reads smoothly, but five details make it non-publishable. The annotated view keeps the copy intact and ties each marked span to the contract it breaks.

The constraints that made this hard

Those contracts had to hold across the whole cohort, not just in a demo. We built the agentic workflow around them and froze a benchmark before changing prompts, models, or cost. That gave every later change the same product bar.

02 · eval

A benchmark that measures the product

We score the service, not prose in isolation. Separate scoreboards cover freshness, grounding, compliance, Korean language, diversity, and delivery, so one aggregate number cannot hide a regression in another.

Human-grounded reports and labels

Gold examples. We built examples from real evidence envelopes with domain experts and aligned the accepted report shapes with Bithumb. Future examples and unseen assets stayed in a separate test panel, and every candidate went through the same checks.

Semantic review (advisory). Human reviewers compared reports with their source evidence and labeled four failure types: unsupported claim; contradiction; numeric / metric mismatch; wrong venue / time binding. Each label was marked minor or major. Those labels train a smaller Qwen reviewer for shadow analysis. Evaluating an evidence–report pair is slower and more complex than a binary copy check, so this reviewer surfaces issues without blocking a slot.

Korean language classifier (production hard gate). Human reviewers labeled customer-visible snippets against four practical standards: natural Korean; report register; approved terminology; clean, neutral copy. The classifier sees only the candidate copy and returns a binary pass/fail, so the smaller Qwen service is fast enough for a synchronous release decision; it does not judge facts or evidence.

Shadow and live parity

Customer SLA is the top priority. A generative model is a probability distribution over outputs, not a deterministic rule: a candidate can look strong on a fixed benchmark and still fail on a rare evidence combination or under live queue pressure. We therefore keep each new producer in a shadow lane first. It sees the same request and frozen evidence as the live producer, but its response is recorded for comparison and never served.

Shadow is the bridge from offline quality to controlled production. Once a candidate holds the release bar on live-shaped traffic, we can ramp it from 5% to 25% to 100% behind the same endpoint; one routing change returns traffic to any known revision.

03 · system

The production graph behind one slot

Writing a good English analysis and producing a native-sounding, compliant Korean financial report are related but different jobs. The first is about selecting and explaining evidence; the second is about preserving those claims while meeting Korean terminology, register, and regulatory conventions. That distinction shaped the system: a shared evidence package feeds the research agents first, then a separate Korean layer with its own checks and feedback.

Technical, on-chain, and community research runs in parallel, then Korean localization works from the accepted claims. Schema validation and deterministic checks provide targeted feedback; the Korean classifier makes a fast binary release decision, and only a failed localization path is retried. A slower semantic judge compares the copy with its evidence in advisory mode and feeds prompt and context improvements rather than blocking a slot.

The clock remains part of the contract. Data must be ready by T-40 and final release checks start at T-11. Provider failures stay local to their section, while late rows are dropped instead of filled with stale context. This lets the workflow preserve freshness and Korean quality at cohort scale.

04 · training

Scaling the workflow with a post-trained, customized model

The GPT-based workflow did what we needed: it assembled the evidence, enforced the release contract, and produced publishable reports. Those safeguards also exposed the scaling problem. With many customer rules and QA layers, a failed validation or checker could reopen a generation path. Every retry consumed more tokens and sometimes pushed rows into the T-11 release window.

That made SFT the right first move. The report has a repeatable shape, so accepted examples teach evidence binding, neutral Korean register, and missing-context handling directly. Our scaling target was a one-shot customized model that could replace this complex staged workflow, while deterministic policy and release checks stayed outside the weights. We froze the contract and benchmark first, then measured each update against the same gates.

Corpus construction and hygiene

We kept the training data in three distinct lineages: curated public market and financial data for factual coverage, synthetic cases for missing or policy-sensitive inputs, and expert-reviewed reports for semantic and Korean quality. Future asset and time combinations stayed out of training so evaluation measured generalization rather than memorization. The adapters learned the report's narrative; evidence, schema, and release remained service responsibilities.

The training recipe as an auto-research loop

We borrowed the loop from Karpathy's autoresearch: propose a small change, run it against a fixed harness, keep or discard it, and use the result to choose the next arm. Here the objective was a production contract, not a single loss metric. We searched prompts, data mixtures, adapter settings, and preference recipes, sending every arm back through the same frozen scorecard.

From Day 0 through iteration 12, we ran 80+ arms across roughly a dozen research rounds. Iteration 4 was the first near-release screen (6/9 axes); each later round targeted the failure profile it exposed.

Search began in a cheap 9B lane before a raw 27B checkpoint showed that capacity alone was not enough (0/200 on the strict panel). The selected recipe used no-thinking rendering, rank 32, a constant 2.5 × 10⁻⁴ learning rate, and three epochs. Four stage-specific LoRA adapters for the report and market-narrative surfaces now run on Fireworks H200s; evidence, schema, and release checks stay outside the weights.

SFT plateau, then on-policy DPO

Whole-report SFT learned the report shape, but its remaining failures were contract-sensitive rather than stylistic. The hard parts stayed concentrated in titles, asset identity, and localization. One source reported a +15.82B token change, but the draft recast it as a 10.1% increase and added a KRW amount (1,573억) it could not support. Elsewhere, a level-only figure was rewritten as a KRW amount without a valid conversion, or the summary and detail sections assigned different values to TVL. These edits sound plausible while changing the underlying claim, so another generic SFT pass was unlikely to solve them.

We used DPO to target those residual defects: the same prompt and evidence paired each flawed answer with a contract-clean repair. On a matched 100-case screen, interpretive findings fell from 8/100 to 3/100. The main failures dropped sharply: entity swaps, unsupported conversions, summary/detail conflicts. SFT remains the base recipe. DPO focuses on three residual areas: evidence fidelity, editorial quality, cross-section consistency. The service still owns evidence, schema, and release checks.

05 · cost

Cost comparison

Training spend. For the scheduled report and market brief, we ran roughly 100 training iterations across the 9B and 27B lanes, using about 3.3B training tokens in total. At the 27B lane's published $3/M serverless LoRA rate, that is approximately $10k in one-time training compute.

Inference spend. The production ledger contains 14.790B total tokens for 30 days. At the supplied GPT-5.6 Sol rate card, that workload prices to $130,328, including the other-model checks used for semantic review and the Korean hard gate.

For Qwen, the scheduled-report and real-time market-narrative proxies average 32.745% of the same workload. That is 4.843B total tokens and a modeled $4,604. Both totals include checker, retry, offline-evaluation, and QA-only calls.

On this token ledger, the Qwen projection is 96.5% cheaper. This is a cost comparison, not a latency measurement.

06 · local

Local inference on a single RTX 5090

Once the hosted route was efficient, we wanted to push the boundary further: what is the smallest local setup that can still complete the same report contract? That question motivates a single-RTX-5090 training-and-inference experiment. The provider inputs, evidence boundary, and release checks stay fixed; only model execution moves to one workstation.

A BF16 27B checkpoint is too large for 32 GB, so we use 4-bit weights and QLoRA. Rank-32 adapters fit comfortably, while the 128 GB host can carry optimizer state. The trade is context length and concurrency; the contract stays the same.

KV cache, rather than the weights alone, is the main limiter for long reports and parallel assets. A bandwidth calculation gives only a rough decode ceiling, so the 65–95 tokens/s range is a planning hypothesis, not a measured result. A follow-up post will share the local measurements after validation on the same workload. The hosted H200 route remains the production path.

07 · reuse

What we would reuse

1. 1 /Start with the customer contract and evidence boundary.
2. 2 /Treat corpus quality as model engineering: factual examples, synthetic edge cases, and expert-reviewed labels each serve a different purpose.
3. 3 /Freeze provider snapshots and keep missingness and provenance visible.
4. 4 /Build the benchmark and shadow path before changing the serving model.
5. 5 /Let SFT and targeted DPO learn narrative behavior; keep evidence, schema, semantic review, the Korean hard gate, and rollout controls in the service.
6. 6 /Separate quality from modeled cost, and validate candidate token projections against a persisted serving ledger before promotion.

The reusable pattern is straightforward: make the contract executable, use the benchmark to expose the failure families that matter, and let post-training remove repeated model work without moving compliance into the weights. Shadow rollout keeps the producer switch reversible, while the hosted H200 path remains production and the RTX 5090 run explores the next efficiency frontier.