Bithumb AI: GPT Sol-level intelligence at a fraction of the cost

tl;dr

• Bithumb AI sits at the center of Bithumb's trading app: market signals, a deep analysis of each asset, answers to live market questions, and a daily market brief, all compiled from complex on-chain data, social and news sentiment, and technical indicators.
• Since launch it has served over a million AI-generated reports and signals at 99.99%+ API availability.
• Each AI report is built from live market context, written in steps, and checked by guardrails before it goes out. We built an agentic workflow that handles deep research, localization and guardrails, and used GPT-5.6 Sol, the most intelligent model at the time, as the base model.
• As the data sources, rules and guardrails became more complex, so did our workflow. Failure edge cases increased, which drove up token consumption and the time to finish a report, to the point of missing the required latency threshold.
• We spent significant engineering effort on an internal financial benchmark for Bithumb. Concretely, it has six measures: serving, schema and policy, semantic review, Korean language, diversity, and performance (tokens, cost and speed).
• We trained a Qwen3.8-27B model that scored 88 against Sol's 87 on our internal benchmark. On the same 30-day workload it costs 96.5% less to run, and the average time to result dropped from 17.9 to 4.5 minutes.
• We built a recursive self-improvement (RSI) framework that lets the AI train the model on its own: it proposes bounded recipe changes and launches parallel arms.

01

Requirements

Four features

Example report

Here is a shortened BTC sample report from the per-asset analysis. It reads well, but we couldn't publish it because of five details. All failing examples in this note come from development and test runs; none were published. Each one is a different kind of mistake, and Bithumb's full rule set is much stricter.

02

Eval

Internal benchmark

Building a high-quality benchmark is one of the most important parts of a modern AI development process. We put 30–40% of our engineering effort into a complete eval system. It mainly serves three core goals:

1. Calibrate the customer's requirements.
2. Align the AI's output with the spec and cut hallucination.
3. Serve as the yardstick for post-training parity.

In more detail, we split the eval system into the dimensions below. Each looks at a different aspect and measures it in its own way.

Expert labels

About half of the benchmark effort went into building and labeling gold data. A general foundation model can already write a decent report from the provided data. But it is still a long way from what a financial expert writes and from what the customer's spec requires. General-purpose outputs do not consistently meet Bithumb's domain-specific standards, and the Korean output also requires further adaptation to Bithumb's preferred terminology and style. So we built three kinds of human-labeled data:

• Gold examples. Complete reports written by domain experts and agreed with Bithumb.
• Semantic labels. Human reviewers compared reports with their source evidence and labeled four failure types: unsupported claim; contradiction; numeric / metric mismatch; wrong venue / time binding. Each label was marked minor or major.
• Korean language labels. Human reviewers labeled the final report against standards such as Korean naturalness and terminology, as a binary pass/fail. This label does not judge facts or evidence.

We later used this human-labeled data to train one dedicated Qwen3 27B judge model, separate from the model that writes the reports. It runs both the semantic review and the Korean language check inside the benchmark.

03

Training

The first version ran GPT-5.6 Sol, then the frontier model, inside our multi-step harness. At generation time it pulled the live context for each asset: on-chain data, social and news sentiment, and technical indicators. Multiple research agents worked the evidence in parallel, a separate step wrote the Korean copy in the required report register, and a set of guardrails decided what went out: schema and policy, semantic review against the evidence, and Korean language. Every cycle has to turn a fresh data snapshot into reports for 200 assets within minutes, so the data is still current when the reports go out.

As the product grew, so did the report rules and the guardrails, and the prompts grew with them. Each failed check meant another pass through Sol, so a stricter rule set cost tokens twice: longer prompts on every call, and more calls per report. The retries also ate into the time before publishing. So we went back and questioned the whole design. Is such a workflow on a frontier model the most time- and cost-efficient way to do this? We already had gold examples and synthetic data. Why not post-train an efficient model and have it generate the distribution we want directly? Our scaling target was a one-shot customized model that could replace this complex staged workflow. We froze the guardrails and benchmark first, then measured each update against the same gates.

Training data

We kept the training data in three separate lineages: curated public digital-asset data for factual coverage, synthetic cases for inputs that are missing or policy-sensitive, and expert-reviewed reports for semantic and Korean quality. We held out future asset and time combinations from training for testing. The training targets come from the labeled and synthetic lineages. The adapters only had to learn how to write the report. Evidence, schema, and release stayed with the service.

RSI

We borrowed the loop from Karpathy's autoresearch: propose a small change, run it against a fixed harness, and let the result pick the next arm. The difference is what the loop is chasing. Ours had to satisfy a production contract, not minimize a single loss number. It searched prompts, data mixtures, adapter settings, and preference recipes, and every arm went back through the same frozen scorecard.

We started with a 9B model and moved to a 27B model once the 9B stopped improving. Size alone wasn't the answer: the 27B model with no training did not reach the release bar on our test panel.

Over about a dozen iterations, we trained and scored more than 100 variants. By iteration 4, a variant reached the release bar on most of the six measures for the first time. Its overall score was still 76, against 87 for Sol. After that, each iteration targeted the mistakes the last one exposed. The result was four LoRA (low-rank adaptation) adapters on the 27B model, one for each report type.

SFT → on-policy DPO

Whole-report supervised fine-tuning (SFT) learned the shape of the report but wasn't perfect yet. The remaining failures were small: a draft would state something the evidence did not say, mostly in titles, asset identity and localization. In one typical case, the evidence said +15.82B tokens of net transfer volume. But the SFT draft wrote “transfers rose 10.1%, about ₩1,573억”, a percentage and a KRW figure that were not in the evidence. The sentence reads fine, yet it changes the claim. More SFT does not fix this, because the model is not short of good examples. What it lacks is a signal that this move is wrong.

Direct Preference Optimization (DPO) adds that signal, so we ran it after SFT, on the failures SFT left behind. For each flawed answer we built one pair, with the same prompt and evidence. Each pair holds two answers. The rejected one is the model’s own draft. The chosen one is that draft with the smallest edits needed to follow every rule. For the example above, the edit removes the invented percentage and KRW amount and keeps the raw +15.82B. From these pairs, the model learns where its drafts stop matching the evidence. On a matched test, these mistakes fell by more than half after DPO.

Shadow serving

The offline benchmark already measures the AI's ability well, but to keep the customer's SLA intact while data providers change and models are swapped, we built a shadowing and routing system. It is the bridge from offline quality to controlled production: once a release candidate holds the release bar on live-shaped traffic, we can ramp it from 5% to 25% to 100% of the assets behind the same endpoint.

04

Results

On the same 30-day workload, the post-trained Qwen3.8-27B route scored 88 to Sol's 87 on our benchmark and passed the same publish checks, at about 1/28th of the cost per report: a tenth of the price on a third of the tokens.

The saving comes in two layers. Per token, Qwen3.8-27B lists at roughly a tenth of GPT-5.6 Sol's price. Per report, the trained model also uses about a third of the tokens: one generation call instead of about eleven, and fewer retries. Fewer calls and retries also mean less waiting: the average time to result dropped from 17.9 to 4.5 minutes.