← Blog

Bulletin, part 1: 25 specs, 130 hours and $51 of DeepSeek

AIAgentsDeepSeekClaudeCodexOpenSpecTerminayBunPostgres

I wanted to know how far a cheap model can get on a big project if you give it good specs, a strict process and someone checking its work. So I picked a community forum in the style of oldschool web bulletin board systems, with forums, topics, private messages, moderation, permission masks, themes, plugins and a staff console. I planned it with Claude Opus 5.5, then handed it over: Claude ran the build unattended through a Terminay workflow, DeepSeek Flash wrote nearly all of the code, and Codex reviewed every change.

I called the project Bulletin, and the code is on GitHub at markwylde/bulletin. It ran from Monday morning to Saturday night:

133.6 hfirst message to last merge
25OpenSpec changes, 408 tasks
518commits, ~274k lines
9.96BDeepSeek tokens
591Codex findings, 198 P1
$50.91DeepSeek bill
Hover or tap a block to see a change, or a dot to see what happened.
MeDeepSeek buildingCodexClaude

This is part 1. It covers how the build ran, what each agent did, what DeepSeek kept getting wrong, what the same tokens would cost elsewhere, and a review of the code it produced. Part 2 rebuilds the UI, which is bad, and fixes a few bugs.

Who did what

I split the work across three models:

Planner and orchestrator

Claude Opus 5.5

Claude Code

Wrote the specs with me, built the harness that drove DeepSeek, read every review, decided what blocked a merge and wrote the fix prompts.

~790Mtokens
107/loop wake-ups
Implementer

DeepSeek Flash

opencode

Wrote the code, ran the tests, ticked tasks and committed, one task at a time.

9.96Btokens
~430commits
Reviewer

Codex

codex exec, GPT models

After every round of work, read the diff against the specs and reported P1 and P2 findings.

145reviews
591findings
Claudeprompt →DeepSeekdiff →Codexfindings →Claude… until clean, then merge

My instruction at the start was “implementor: deepseek / high level steering: claude / review/second opinion: codex”. I also asked for auto-merge once tests passed and Codex signed off, and for all 25 changes to run back to back, pausing only for a failed spike or a fundamental problem. It never paused, and DeepSeek never wrote the BLOCKED.md file it was told to write if a spec couldn’t be met. From there the build ran hands off.

1 · Planning

Planning before any code

Planning took a little over three hours on the first morning, all with Claude. I’ve written before about why I use spec-driven development and OpenSpec; this is the first time I’ve used it on something this big.

  1. 09:23
    The idea

    I described the forum and the stack I wanted: a Bun monorepo, an Astro user app, a Vite staff app, Bun APIs, Postgres with pg-boss, Docker Compose. I asked Claude for "a questionnaire on all the features you can think of".

  2. 09:24
    62 questions

    Claude opened a questionnaire in a Terminay window and I tapped through it. One of my notes became a rule for the whole project: "locally eventhing should run in 2 steps. 1: clone, 2: docker compose up --watch... only postgres is needed and stateful. no redis".

  3. 09:40
    17 ADRs

    Claude wrote 17 architecture decision records, a feature list and the agent instructions.

  4. 10:26
    First Codex review

    Five serious design problems, including cache entries shared between members of different groups, permission revocation that didn't take effect straight away, and plugin processes that weren't isolated. That added three ADRs, for 20 in total.

  5. 10:43
    25 OpenSpec changes

    I pointed out that ADRs aren't specs. Claude split the product into 24 OpenSpec changes in about 19 minutes: 55 capability specs, about 335 requirements and 404 scenarios. A review then split plugin-sandbox-spike into its own change, which made 25.

  6. 11:32
    Codex and agy together

    The last of seven review rounds: Codex three times, two external reviews I pasted in, a Codex check, then Codex and agy side by side. About 40 distinct findings, fixed by 11:56.

  7. 12:40
    Go

    I told Claude to start: DeepSeek implements, Claude steers, Codex reviews.

New to ADRs and OpenSpec?

An ADR (architecture decision record) is a short document that records one decision and why it was made: “Postgres is the only stateful service”, “every permission check goes through one function called can()”. The agent instructions made them binding: if a task needed to contradict an ADR, the agent had to stop and propose a new one.

OpenSpec organises work into changes. Each change folder has a proposal, a design, a tasks.md checklist, and spec files with requirements and scenarios (“GIVEN a guest, WHEN they open a members-only forum, THEN they get a 404”). The agent works through the checklist and ticks tasks off. The scenarios tell the reviewer exactly what to test.

I kept sending the documents back for review because a spec problem caught on day one costs a few minutes of Claude rewriting markdown, while the same problem found in change 20 costs hours of DeepSeek rework.

2 · The build loop

The build loop

Once I said go, Claude wrote a small Python driver (drive.py) and a few scripts in its scratchpad. By the middle of the build, every change went through this loop. The first four changes came before final.sh existed, and the last one was built by Claude after the DeepSeek balance ran out:

  1. Worktree. Create a git worktree and branch for the change.
  2. Task groups. For each group in tasks.md, start opencode run with DeepSeek Flash on a prompt: the base rules, the change name, and “do group N”. DeepSeek reads the ADRs and specs, writes code, runs the tests, ticks tasks, and commits one task at a time.
  3. Three review rounds. Run bun run check, then codex exec in read-only mode with a reviewer prompt. Codex returns numbered findings tagged P1 (security, data loss, spec violation, broken build) or P2 (bug, missing test, ADR deviation). DeepSeek gets the findings and fixes them. Repeat three times.
  4. Final gate. Claude runs final.sh: the full check, the unit tests, a fresh Docker stack, the database suite against real Postgres, both production builds (with a grep for Node imports leaking into browser bundles), the E2E suite, a curl of both sites, and one more Codex sign-off review.
  5. Steer. If the gate found anything, Claude writes a targeted prompt for DeepSeek in a fresh session (“## Your task now”, plus the findings) and runs the gate again.
  6. Merge. Merge to main, archive the OpenSpec change, tick the step in the Terminay workflow, start the next change.
The loop each change went throughOpen full screen

Claude tracked all 25 changes in a Terminay workflow and ticked each one off as it merged. A /loop kept it going from one change to the next. The loop fired 107 times. Claude handled 640 background notifications, made 352 workflow updates, and slept the rest of the time.

WORKFLOW
Bulletin build: 25 OpenSpec changes
25 of 25 done · 130h 15m from first DeepSeek commit to last merge
waiting on compute94h 01mwaiting on AI36h 39m
  1. 1foundation-dev-environment
    2h 41m
  2. 2plugin-sandbox-spike
    2h 10m
  3. 3i18n-foundation
    1h 00m
  4. 4identity-sessions
    2h 52m
  5. 5staff-console-shell
    2h 00m
  6. 6feature-flags
    1h 11m
  7. 7groups-permissions
    2h 04m
  8. 8rendering-and-caching
    5h 57m
  9. 9forum-structure
    4h 43m
  10. 10content-pipeline
    5h 24m
  11. 11topics-posts
    5h 24m
  12. 12attachments-avatars
    4h 04m
  13. 13events-notifications
    5h 54m
  1. 14reactions-reputation
    4h 39m
  2. 15member-profiles
    6h 27m
  3. 16conversations
    3h 58m
  4. 17moderation
    12h 25m
  5. 18search
    3h 23m
  6. 19themes
    6h 35m
  7. 20plugin-platform
    12h 56m
  8. 21public-api-webhooks
    11h 08m
  9. 22pwa-push
    8h 32m
  10. 23privacy-gdpr
    6h 22m
  11. 24seo-feeds
    4h 45m
  12. 25operations-observability
    4h 06m
Rebuilt from the session logs, archive commit to archive commit. Compute is builds, tests, Docker and the gate scripts; AI is DeepSeek thinking and writing plus Codex reviewing. Compute also absorbs short idle gaps and Claude's steering. The DeepSeek balance ran out during seo-feeds, so operations-observability's AI time is a Claude subagent writing plus Codex reviewing. The 24-minute dev-stack-isolation change wasn't a workflow step and isn't shown.

The rules grew as things broke

The base prompt is short, about 40 lines. Most of its “Headless run” section, and a few changes to the driver, came from something going wrong in the first 31 hours:

  1. Mon 19:21
    DeepSeek killed its own agent

    It ran pkill -f "docker compose up --watch", which matched its own command line and killed opencode.

    Never use pkill -f or killall with broad patterns.
  2. Mon 19:37
    A blank staff app

    A shared package barrel had pulled node:crypto into the browser bundle, so the staff app loaded blank. Nothing in the tests ran a real browser yet, so Claude added a browser gate to final.sh.

    Browser code must import client-safe subpaths only.
  3. Mon 21:50
    DeepSeek took down my stack

    It ran docker compose down in its worktree. Every checkout shared the compose project name, so that stopped my own running stack. This led to a 26th change, dev-stack-isolation, which made the project name and port configurable. It took 24 minutes, and the agents moved to port 18080.

    Never start, stop or touch the default bulletin project or port 8080.
  4. Tue 00:02
    A task group committed onto main

    opencode's shared background service ran a whole task group in the main checkout instead of the worktree. Claude moved the commits back.

    The driver runs opencode with --standalone.
  5. Tue 03:05
    Flaky E2E tests

    Tests clicked buttons that were still disabled.

    Wait for selector:not([disabled]) before clicking.
  6. Tue 08:54
    Sessions stopped silently

    Long opencode sessions sometimes "silently stop with exit 0 and no changes" once the context is huge.

    Every steering prompt gets a fresh session.
  7. Tue 13:51
    450 tests weren't running

    bun run check skipped the database integration tests, about 450 of them.

    Run the DB suite before finishing any group that touches the database.
  8. Tue 19:15
    A cap on final gates

    Content-pipeline kept failing its final gates, and each Codex sign-off found new P2s (it took six gates in all). Final gates were capped at two, except that a P1 or a security finding always blocks the merge. Non-security P2s go into a follow-up file for a later change.

    Never merge with a P1 or anything tagged [security], whatever the cap says.

That last rule is why some changes took 12 hours, as the breakdown below shows.

3 · Change by change

Change by change

Click any change to see its numbers, the Codex results round by round, and the notes from the logs. Blue is time spent in DeepSeek’s driver loop; orange is time spent in final gates and Claude’s steering.

Sort:
DeepSeek driver (task groups + 3 review/fix rounds)Final gates and Claude steeringBuilt by Claude subagents

17. moderation

Queues, reports, warnings, bans, IP tools, multi-moderation.

12h25wall clock
9 / 26task groups / tasks
19opencode runs (7 steers)
8final gates
25 / 22Codex P1 / P2 (raw)
24tagged [security]
862MDeepSeek tokens
$3.57DeepSeek cost (opencode)
43commits
+29,740lines added

Codex reviews in order (P1/P2)

6/25/54/41/22/22/21/11/01/12/00/3

From the logs

Largest change (9 groups, 26 tasks, 43 commits, ~29k lines). Driver alone 8h28m. Then 8 final gates / 7 steers in 4h: almost every gate found 'one more security P1' (IP access, session-unaware checks, ban bypass via edit, approval bypass), plus E2E regressions and beacon 429s.

Solid pills are the driver's review rounds; dashed pills are final gates. Red means at least one P1.

The same build as a timeline, with the incidents along the way:

Five and a half days, from first message to last mergeOpen full screen

Why moderation took 12h25m

Most changes took two to six hours. Three took more than eleven, and the gate cycles explain all three. Topics-posts is a useful comparison first: it took 5h24m, but only because its third task group spent 1h54m fighting Docker (stuck test containers, experiments with DOCKER_HOST) before Docker filled the disk and stalled. Its gate phase was short: 2 gates and 1 steer, about the same as the other mid-sized changes. The mini PC is also a runner for my Gitea Actions, so CI jobs from other projects were filling the same disk at the same time. That skews the timings a little, by no more than an hour across the whole build.

Moderation was the biggest change and the hardest to get past Codex. With 9 task groups, 26 tasks, 43 commits and about 29,700 lines, DeepSeek’s driver alone took 8h28m. The first seven of its eight final gates each found one or two new security P1s:

  • IP lookups ignored password-forum unlocks.
  • Edits bypassed bans.
  • Bans and approvals imposed by warnings had gaps (found at four gates for bans, two for approvals).
  • Temporary grants to see a member’s IP address survived the loss of the access that created them, or could be redeemed by another staff member (five findings).

Each of those cost a fix by DeepSeek plus a full gate, about 30 minutes to an hour. From gate 4, Claude’s prompts told DeepSeek to audit the whole change for the same class of bug instead of fixing the cited line (“Before you finish, audit the whole moderation change for the same class of bug.”). When gate 5 still found “one more security P1, again in IP access”, Claude ran its own security review. Moderation took 8 final gates and 7 steers to merge.

Plugin-platform (12h56m) was the same story with sandboxes. Codex’s first round found 15 P1s, and the gates later found app roles holding pg_read_file, a broker that never checked approved capabilities, and redirects that escaped the egress allowlist. Public-api-webhooks (11h08m) spent most of its time on flaky tests and an intermittent 500; its code was mostly fine. The 500 came from the core theme hitting the 50 ms template render limit under load.

A gate cycle is a fresh stack build, the unit, DB and E2E suites, and a Codex review, which is 15 to 30 minutes before any fixing starts. So the number of gate cycles mattered much more than code size, and the security-heavy changes needed the most cycles. Switch the chart between the two:

Hours per change against:
0h2h4h6h8h10h12h14h0123456789final gates (full stack + all suites + Codex sign-off)i18n-foundationrendering-and-cachingcontent-pipelinemoderationplugin-platformpublic-api-webhookspwa-push
Correlation with hours: r = 0.75. Red dots needed five or more final gates; orange was built by Claude subagents.

Most of the time went on shell commands

DeepSeek’s assistant messages span 103.9 hours. Shell commands account for 77.3 of them:

23.7h19.5h14.7h15.9h26.6h
Waiting on shell commands: 77.3 h
The model: ~27 h
  • Docker builds and stack restarts 23.7h
  • Database integration suite (950 runs)19.5h
  • E2E suite (1,307 runs)14.7h
  • Unit tests and checks 3.5h
  • Other shell commands 15.9h
  • The model thinking and writing 26.6h

That leaves the model itself thinking and writing for roughly 27 to 30 hours of the 130. It all ran on a GMKtec mini PC (Ryzen 9, 8 cores, 32GB of RAM) running Linux. On a more powerful machine, the build would have been a lot quicker.

What I’d change about the compute

The model waited on shell commands for more than twice as long as it spent thinking. Looking at which commands, most of that wait was avoidable:

  • Every test run rebuilt the test image. DeepSeek’s most expensive command was docker compose run --build --rm test, which runs the whole suite. It ran it 236 times, for 14.7 hours, nearly four minutes a go. The test Dockerfile copies the whole repo before bun install, so any code change invalidates the cache and reinstalls every dependency. Copying package.json and bun.lock first, or mounting the source into a long-lived test container, would remove most of that.
  • It ran the whole suite when one file would do. Targeted runs (bun test <path>) were 488 runs and only 3.5 hours. Running just the affected tests during a task group, and the full suite at the gate, would have saved hours per change.
  • The stack restarted a lot. docker compose up -d ran 444 times, averaging nearly two minutes. A stack started once with Compose watch would sync code into the running containers, so it would only need a restart when dependencies change. The final gate could still start a fresh stack.
  • Test Postgres wrote to disk. Each integration test already clones a migrated template database, which is quick. Running the test Postgres on tmpfs with fsync=off would make the 950 integration runs faster still.
  • The machine was shared. The mini PC was also a Gitea Actions runner. A dedicated machine, and a faster one, helps every item above.
4 · Codex

What Codex caught

Those repeated reviews cost time, but they caught what DeepSeek’s own tests missed. In 145 review sessions Codex raised 591 findings, or about 533 distinct issues once you remove the ones raised again because a fix was incomplete. About 507 were fixed before merge, 24 non-security P2s were deferred, and 2 were overridden: one was out of scope and one was a flaky test. None was rejected as technically wrong.

Every one of DeepSeek’s 73 driver review rounds found at least one P1 or P2. Round 1 across all changes found 87 P1s, 40 of them security. After two fix rounds, round 3 still found 31. Only 12 reviews in the whole build came back clean, and all of them were final gates.

Every finding

Explore every finding below. I assigned the categories by hand from the titles, so treat the counts as estimates.

P1P2Click a category to list its findings.
28 findings
  • Saved requests survive revocation and account switches.
    P1line from a fix commitidentity-sessionspackages/domain/src/auth/fetch.ts:198
  • Hidden forum activity leaks publicly.
    P1forum-structure
  • Deleting a target group makes restricted content public.
    P1forum-structurepackages/db/migrations/0032_announcements.sql:40
  • Public history bypasses content permissions.
    P1securitytopics-postsapps/user-api/src/topics.ts:259
  • Poll results leak across forum permissions.
    P1securitytopics-postspackages/domain/src/polls/index.ts:325
  • Overlay accepts arbitrary topic IDs.
    P1securitytopics-postspackages/domain/src/auth/service.ts:418
  • Hidden poll HTML contains voter totals.
    P1securitytopics-postsapps/user-app/src/polls/fragment.ts:43
  • Topic redirects disclose restricted titles.
    P1securitytopics-posts
  • Hidden topics remain accessible directly.
    P1securitytopics-posts
  • Permission checks use stale forum IDs.
    P1securityevents-notificationspackages/domain/src/notifications/repository.ts:688
  • Notifications and digests expose hidden posts/topics.
    P1securityevents-notificationspackages/domain/src/notifications/repository.ts:128
  • Unavailable follows still disclose current titles.
    P1securityevents-notificationspackages/domain/src/follows/repository.ts:287
  • Saved reauthentication drafts leak across accounts.
    P1securityline from a fix commitevents-notificationspackages/ui/src/islands/Editor.tsx:98
  • Unapproved-post reactions leak to ordinary readers.
    P1securityreactions-reputationapps/user-api/src/reactions.ts:100
  • Reputation giver identities escape the staff-only boundary.
    P1securityreactions-reputationpackages/domain/src/notifications/repository.ts:423
  • Staff reputation listings disclose inaccessible topic titles.
    P1securityreactions-reputationpackages/domain/src/reputation/repository.ts:294
  • Private-forum activity leaks through board totals.
    P1securitymember-profiles
  • Friend requests reveal who ignores the requester.
    P1securitymember-profiles
  • Deleting a group can publish restricted profile fields.
    P1securitymember-profilespackages/db/migrations/0073_profile_fields.sql:71
  • Leaderboards retain contributions moved into private forums.
    P1securitymember-profilespackages/domain/src/leaderboards/repository.ts:87
  • Bulk topic deletion leaves replies public and cannot restore the deleted opener.
    P1securitymoderationpackages/domain/src/topics/tools/repository.ts:706
  • Deleted topics remain readable through other surfaces.
    P1securityline from a fix commitraised againmoderationpackages/domain/src/topics/moderation.ts:388
  • Hidden-forum metadata leaks through prefix options.
    P1securitysearch
  • Core reads can disclose hidden content.
    P1securityplugin-platform
  • Hidden-topic posts leak by ID.
    P1securitypublic-api-webhookstopic.visibility
  • Export bypasses current forum permissions.
    P1securityprivacy-gdprpackages/domain/src/privacy/export.ts:148
  • Export leaks restricted poll text.
    P1securityprivacy-gdpr
  • Feeds and sitemaps bypass board-offline permissions.
    P1securityseo-feedsapps/user-app/src/middleware.ts:51

Three themes make up 38% of everything Codex raised:

  • Permissions and visibility (87 findings). The main page called can(), but totals, notifications, exports, leaderboards, prefixes, redirects, edit history and poll results skipped the check. “Private-forum activity leaks through board totals.” “Notifications and digests expose hidden posts/topics.” “Any authenticated member can retrieve private, unapproved or deleted posts and revisions by ID.”
  • Not actually done (82). DeepSeek ticked tasks in tasks.md without wiring the code up, despite the rule “No placeholder or stub implementations”. In plugin-platform round 1: “The production runtime still runs only the demo.” In reactions-reputation, “Task 2.3 falsely claims profile display is complete” came up in all three rounds. There were also 34 tests that couldn’t fail, like a permission test whose fixture already filtered out the thing it was checking for.
  • Caching (54). Every new piece of data shown in cached HTML needs its cache tag bumped when it changes. DeepSeek rarely did this unprompted: moderator lists, emoticons, ranks, badges, signatures, poll results. One P1 found that author-only edit forms were cached by permission bits alone, so another member with the same permissions got the author’s form.

A few more that stood out:

A sanitiser bypass

<noscript><p title="</noscript><img src=x onerror=…>"> got through both sanitiser layers. Only the Content Security Policy stopped it.

A WebP quota bypass

A 17 KB PNG became a 230 KB WebP after re-encoding, getting round the upload quota.

Identity by whitespace

The user ID from DarkAuth, the identity provider, was trimmed before lookup, so "admin" and " admin " resolved to the same member.

CSRF by query string

GET …?method=POST became a plugin POST that skipped the CSRF and Origin checks.

SMTP in plain text

AUTH PLAIN was sent over plain TCP.

Plugin roles that collide

demo-owner's runtime role had the same name as demo's owner role, so uninstalling one plugin could drop another plugin's objects.

Fixes that created new bugs

For this post, Claude ran git blame on each cited line at the branch head just before that review, to find the last commit that touched it. That worked for 455 findings. For 71 of them, the line was last written by a commit fixing an earlier review rather than one implementing a task, and 22 of those were P1 or security. That doesn’t prove the fix caused the bug, but some cases are clear:

  • In plugin-platform, the round-1 fix for a security finding added migration 0115. Gate 2 found that migration had given every app role, including the read-only one, file-read and large-object functions.
  • In events-notifications, the fix for “unsaved edits are lost during reauth” created “saved reauthentication drafts leak across accounts”.
  • Four of moderation’s ten final-gate security P1s sit on lines written by earlier fixes.

Codex isn’t deterministic

Forum-structure gate 1 came back clean, and the gate was only rerun because E2E failed on stale Docker images. Gate 4, on essentially the same code, found a new P1: deleting a target group made restricted content public. Public-api-webhooks alternated clean, P2, P2, clean, clean, P2 over six gates. Pick a change below to see its gates.

Codex also changed models during the build: gpt-6.1-sol on low effort for the first ten changes, gpt-6-astra on medium until plugin-platform’s last gate, then gpt-6.1-sol on medium to the end. Low-effort reviews took 1 to 2 minutes and medium ones 3 to 5. All 145 reviews used 181.6M tokens and about 5.8 hours.

P1s found in the driver's review rounds, all changes
Round 187 P1 · 40 security
Round 331 P1 · 19 security

Dark red is the share tagged [security]. Clean driver rounds in the whole build: 0 of 73.

Final gates, one box per Codex sign-off
gate 1clean
gate 22 P2
gate 31 P2
gate 4clean
gate 5clean
gate 61 P2
Which model reviewed each change
12345678910111213141516171819202122232425
gpt-6.1-sol, low effort (1–2 min)gpt-6-astra, mediumgpt-6.1-sol, medium (3–5 min)
5 · DeepSeek, good and bad

What DeepSeek got wrong and right

Across all those reviews, DeepSeek kept making the same few mistakes. It was also good at a lot of things.

Kept coming back · changes affected, of 26
Ticked tasks that weren't done 14
Forgot cache invalidation for new data 13
Permission check on the main path only 12
Permission logic outside can() 5
Caught and fixed itself
Test and check commands: 3,874
1,135 (29.3%) failed, every one worked back to green
Tool calls: 26,736
154 (0.6%) errored
243TypeScript errors hit and fixed
17 / 20changes passed bun run check first time
0invented blockers

Mistakes that kept coming back

  • Permission checks on the main path only. The same hole appeared in 12 of 26 changes. Permission logic written outside can() was flagged 6 times in 5 changes, even though the agent instructions say it in bold.
  • Tests run as the database owner. Code passed tests run as the owner, then failed under the real database roles. Digests and orphan cleanup failed under the production worker role, and timed bans never restored. Codex gave the same advice every time: run the test as workers.
  • Check-then-act races. The code checked authorisation before taking a lock and trusted the result afterwards, so concurrent topic moves, splits and uploads could race a staff block. There were also 25 plain races, like flood control that checks first and records later, so concurrent requests all pass.
  • Fixing the line instead of the class of bug. Claude’s prompts had to say “Do not fix only the cited lines” and “Treat each as a class”, and moderation still kept turning up the next one.

Mistakes it caught and fixed itself

DeepSeek was good at anything its own tests could see. Of its 3,874 test or check commands, 1,135 (29%) failed, and it worked every one back to green. It also fixed the TypeScript errors it hit 243 times. After its task groups, bun run check passed straight away in 17 of the 20 changes that logged it. It was honest about what it couldn’t run: “Note: the checkout’s pre-existing incomplete node_modules still prevents a full workspace bun run check”.

It missed anything its tests didn’t exercise: security and visibility checks, database roles, and code that was never connected to the running app. Codex caught almost all of those. Claude caught the infrastructure traps. One React StrictMode bug, which double-posted a single-use login challenge, only showed up in a real browser.

What went well

Only 154 of 26,736 tool calls errored (0.6%), and there was one hallucinated path. It made a commit per task with a consistent message, ticked tasks only after tests (mostly), and shut its stacks down afterwards. It never invented a blocker or worked around an ADR. Smaller changes went straight through: i18n-foundation in 1h00m, feature-flags in 1h11m, groups-permissions in 2h04m. And it built 23½ of the 25 roadmap changes, over 300k added lines and about 430 commits, for about $51.

Where it wasted effort

  • The whole context was sent on every request. The driver reused one opencode session per change across every group and fix round. The context grew to 300K–560K tokens on average (1.04M at most), and every call sent it again. 99.7% of DeepSeek’s tokens were cache reads. That’s cheap on DeepSeek, at $0.003 per million, but it’s why the token count is so large. A fresh session per group would probably have cut it by about half.
  • Edits through Python heredocs. DeepSeek made 3,240 edits by writing Python scripts in the shell, against 3,777 uses of the edit tool.
  • Flaky tests were chased across gates. One steer on public-api-webhooks ran for 3 hours, including 20-minute full E2E runs inside opencode, before Claude killed it, reviewed the work and committed it itself.

DeepSeek ran out of balance twice. The first time, at 01:25 on day five, a top-up cost 35 minutes. The second time, at 18:04 on the last day, Claude stopped using opencode and finished the build itself. Claude subagents fixed the last seo-feeds findings and built all of operations-observability, which is the missing half of the 23½.

6 · Costs

What it cost

This is DeepSeek’s usage page at the end of the build:

DeepSeek platform usage page: topped-up balance -$0.01, total cost $50.00 USD plus ¥6.14 CNY, 24,837 API requests, 9,955,557,125 tokens over the last 30 days

That’s 9,955,557,125 tokens and 24,837 requests for $50.00 plus ¥6.14, about $50.91 in total, and the balance went to −$0.01. A small part of it wasn’t Bulletin: the 30-day view includes a little earlier usage around 4 November, and 30 of the requests weren’t on deepseek-flash. opencode’s own records for the build come to 9.83 billion tokens, 98.8% of the dashboard figure.

How it’s that cheap

DeepSeek Flash charges $0.003 per million tokens for a cache hit, off-peak. A cache miss is $0.15 and output is $0.60. Peak hours cost double. opencode recorded 99.7% of its tokens as cache hits, which comes to $39.96 at off-peak prices. The bill is about $11 higher; some peak-hour traffic, or a slightly lower cache rate on the requests opencode didn’t record, would explain that.

What’s a cache hit?

An agent sends the whole conversation to the model on every request: the instructions, every file it has read, every command output. Most of that is identical to the previous request. Providers keep a copy of that prefix, and when the next request starts with the same text, they charge a fraction of the normal input price for it. That’s a cache read, or cache hit. Only the new part at the end is charged in full.

For a long agent session, cache reads are almost all of the tokens. Here they were 99.7%. So when you compare models for agent work, the cache-read price matters far more than the headline input price.

The same tokens on Claude or OpenAI

Here’s what the same 9.96 billion tokens would cost at API list prices. Move the cache slider to see how much the cache-hit rate matters.

Presets:
What I actually paid$50.91
DeepSeek Flash (off-peak)$53
DeepSeek Flash (peak)$107
Claude Haiku 5.5 (>100K prompt)$592
Claude Sonnet 5.5$1.4K
gpt-6.1-sol (≤272K prompt)$1.4K
gpt-6.1-sol (>272K prompt)$2.7K
Claude Opus 5.5$2.8K

9,955,557,125 tokens from DeepSeek's dashboard, priced at list rates with no batch discounts. Uncached input is billed as a cache write. This assumes the same token count on every model, which isn't true: a stronger model might loop less and use fewer tokens.

At a 99% cache rate (a little below what opencode recorded, to match the bill), the same tokens cost about $1.4K on Sonnet 5.5 or short-context gpt-6.1-sol, and about $2.8K on Opus 5.5 or long-context gpt-6.1-sol. That’s 27 to 54 times what I paid. The average request carried about 400K tokens of context, so many gpt-6.1-sol calls would land in the >272K price tier.

A stronger model wouldn’t use the same number of tokens, though. It might loop less, get the permission checks right the first time, and need fewer gate cycles. These numbers price DeepSeek’s workload on other models; what Bulletin would cost to build with Opus depends on how much work Opus needed.

On subscriptions instead

I pay for Claude Max ($100 a month, previously called Max 5x) and ChatGPT Pro ($100). Neither vendor publishes limits in tokens, so the weekly figures below are community estimates from small samples. Pick a combination of plans:

Time to use 9.96B tokens
DeepSeek, actual5.6 days
Claude Max $1007.9–10.2 wk
ChatGPT Pro $1006.1–12.3 wk
Selected together3.5–5.6 wk
What that costs
DeepSeek bill$51
Selected subscriptions$159–$256
Sonnet 5.5 API$1.4K
Opus 5.5 API$2.8K

Tokens per week are community estimates from small samples; neither vendor publishes limits in tokens. Faded bars show the slow end of each range. Subscription cost is the monthly fee prorated over the weeks, and assumes you can keep every weekly cap saturated. API prices assume a 99% cache-read share, like this build.

My guess before I looked was “about 10 times as long on one subscription, 5 times with two”. Compared with the 5½ days this took, the estimates say roughly 10–13 times as long on Max alone and 4–7 times with both. The fees for that time come to about $160–260, three to five times the DeepSeek bill but far below API prices. All of that assumes you can keep every weekly cap saturated, which the five-hour windows on Claude make hard in practice.

What Claude and Codex used

Claude and Codex used far fewer tokens than DeepSeek, and both fitted inside the subscriptions I already pay for:

~790Mtokens, 98.7% cache reads
Claude Opus 5.5: about $247 at API prices

The main session plus two build subagents. It ran in one context for 5½ days, auto-compacted 12 times and made about 3,700 API calls in 134 hours (2,900 of them in the main session), because it spent most of the time waiting on background events. By the community estimates that's two-thirds to four-fifths of one week of Max.

181.6Mtokens, 91% cache reads
Codex: about $55 at gpt-6.1-sol prices

145 reviews in about 5.8 hours. By the community estimates that's 11–22% of one week of Pro.

7 · The codebase

The codebase

So what did all that produce? Bulletin is a Bun monorepo with 7 apps, 12 packages, 9 local mocks and a 16-service compose file. It has 144 SQL migrations, 2,294 translation keys in each of English, Spanish and French, and 34 runtime dependencies. It uses no ORM, no HTTP framework and no Redis.

Bulletin's services and how they connectOpen full screen

Caddy splits traffic across four origins: the forum, the staff console, files and plugins. The forum app (Astro) reads Postgres directly with a read-mostly role and renders HTML, and every write goes through an API (user-api or staff-api). In the same transaction as the write, the API bumps version numbers on cache tags and appends an event row. Postgres NOTIFY then wakes the other processes so they drop stale HTML. A workers process turns events into jobs: emails, push, webhooks, CDN purges. Plugins run in their own locked-down container and only reach the rest of the system through a broker.

Caching by what you can see

This is one of the parts DeepSeek built well. A forum page looks almost the same for most members. Caching a copy per user wastes memory, and caching one copy for everyone leaks private forums. Bulletin caches by a visibility key instead: a hash of the permission answers the page actually depends on. Two members in different groups who can read and reply in the same forums get the same key and share one cached copy. Personal bits (unread markers, notification counts, “you reacted”) are fetched afterwards through a small overlay request and patched into the page.

A page view: authorise, build the key, check the cacheOpen full screen

Each page has to declare which permission bits it reads. In dev and test, reading an undeclared bit throws an error, so a missing cache dependency becomes a failing test. And if a process stops hearing invalidation messages from Postgres, it stops serving from the cache and renders every page fresh.

A reply: one transaction, then NOTIFY and the outboxOpen full screen

The plugin sandbox

Plugins are untrusted server code, so each one runs as a Bun process inside nsjail, a Linux process jail. It gets its own user, an empty network namespace, a read-only bundle, cgroup limits on memory, processes and CPU, and a seccomp policy that blocks every system call it doesn’t need. Anything it wants goes through a broker. The broker parses SQL into a syntax tree and only allows the plugin’s own schema. Outgoing HTTP is limited to the hosts in the plugin’s manifest, with checks for DNS rebinding and redirects. Reads from the core are signed calls that run can() as the viewer. A 17-case malicious-plugin suite (fork bomb, output flood, DNS rebinding, reading sibling files and so on) runs in CI inside the real container.

Plugin host: jail, broker and the only ways outOpen full screen

CI found a real sandbox bug that local runs never hit: on the x86_64 runners, Bun calls the legacy symlink() syscall, the seccomp policy didn’t allow it, and the kill action left the plugin hanging instead of crashing.

What’s good

763comment lines cite an ADR
The ADRs are followed in the code

There is one can(). Every database role has its own grants in migrations. Job payloads carry IDs only, and emit() rejects payloads that don't match the event catalog.

Personal data is fenced off

IP addresses and emails can only be read through SECURITY DEFINER functions with a pinned search_path, and reading one needs a committed audit row first.

Themes are locked down

Liquid templates with raw unregistered, filesystem tags removed, strict variables and render limits.

30of 1,532 exports unused
Little dead code

Almost every exported function is used somewhere.

What’s weak

One headline feature isn't real

ADR 0009 says admins install plugins from a marketplace at runtime, and the install task is ticked. But plugin-host runs exactly one bundle, from an environment variable, and the lifecycle code is only used in a test with a fake runtime. The sandbox works; there's no way to install a plugin into it. Codex flagged "not wired" in plugin-platform three times, and this one still slipped through.

Demo data seeds on every start

There's no production guard.

53files define their own isRecord
Copy-paste everywhere

40 define their own json() helper and 7 their own escapeHtml. A dev secret is hard-coded as a fallback in 17 places. Each agent run wrote its own helpers instead of finding the shared ones.

1,418lines of if-chain
Two API styles

staff-api uses a typed route table with a permission per route. user-api is one if (method && pathname) chain with 70 copies of if (session === null) return 401.

2,221lines in one file
Big files

staff-api/src/api.ts is the biggest. The notifications and themes repositories are about 1,600 lines each, and the themes admin page is one 1,000-line React component with 18 useStates.

144migrations
Migrations read like a review log

pii.read_post_ip is redefined in five of them. They should be squashed before anyone deploys this.

Every invalidation re-reads every cache tag

Fine for a demo, slow at scale.

For six days of agent output it holds together well. The rules that matter (authorisation, visibility-keyed caching, personal data, sandboxing) are enforced in code and tested. The debt is what you’d expect from many separate agent runs: duplicated helpers, inconsistent style, oversized files, and a couple of tasks marked done that you only spot by reading the code against the specs.

8 · Testing

Testing

Those rules hold because the test suite checks them. There are 483 test files with about 2,632 test cases, and about 0.71 lines of test for every line of code.

Layer Files Cases How it runs
Unit 220 1,333 bun test, part of bun run check
Database 171 1,062 Real Postgres. Each test file gets its own database, cloned from a migrated template
Full-stack integration 7 26 Over HTTP against the running compose stack
E2E 84 194 Bun.WebView driving Chrome at 360, 390 and 1440 px
Malicious plugin 1 17 Inside the real plugin-host container

Two kinds of test are specific to this design:

  • Permission-denial tests. About 463 test names are refusals: “a guest is refused…”, “denies…”, “403…”. The overlay test asks for a private staff topic’s ID and checks it doesn’t leak into unread markers, editable posts or anything else.
  • Cache-variant tests. A shared harness renders each registered route twice for each viewer case. The variants must differ, the second render must be a cache hit, and “leak markers” planted in privileged content must never appear in a less privileged variant.

There are also architecture tests that check structure statically (no direct writes to the cache tag table, content queries only in permission-filtered repositories) and, during the build, a nightly load test with a million posts.

During the build, CI ran on my Gitea runners. It failed on every run for the first four days, because it needed sudo the runners didn’t have, a Postgres health check raced, and E2E took 14 minutes. Claude fixed it on the last day: each run gets its own compose project and port, and E2E is split into eight shards plus a full-stack shard. Getting CI green found four real bugs that local Docker never hit, including the seccomp one above and a race in marking conversations as read.

Some integration test files are huge (one is 1,911 lines) and redeclare the same helpers, and the E2E tests average 2.3 cases per file because each one walks through a long journey.

The tests cover behaviour, permissions and speed. Nothing in the specs or the tests said what the UI should look like.

9 · The UI

The UI

Here’s everything, as it is. I haven’t fixed anything. The forum app is on the first tab, the staff console on the second.

The member-facing site, rendered by the default Liquid theme. Desktop at 1440 px, captured at 2x. Click any screenshot to open it full size; arrow keys move between them.

The UI is bad, and for a real forum it’s unacceptable, even though everything works. You can log in, post, react, quote, start a conversation, report a post, ban someone, edit a permission mask, export your data and switch to French, and the seed data is rich. But it all looks like a default form with a blue button. The staff console puts every setting in one column and makes every action a full-width button.

I expected this before I started. The specs did say “mobile-first at 360 px” and listed the screens and fields, but they didn’t say what good looks like, and DeepSeek didn’t fill the gap. I think that apart from Claude and GPT, basically no model builds a decent UI without detailed prompting.

Where part 1 ends

DeepSeek’s $51 forum works and has real security engineering, but nobody would want to use it as it looks today. There are also a few bugs, like the plugin marketplace that can’t install a plugin. I’d expect a few bugs from any model, even the best ones, so that’s fine.

Part 2

In part 2 I’m going to throw away both front ends, the forum theme and the staff console, and have Claude Opus 5.5 rebuild them on top of the same APIs, view models and tests. It will also fix a few of the bugs from part 1. The back end stays.

It will be the same approach as part 1: plan it with Claude first, then hand it off and let it run unattended. We’ll see what Claude does with the UI, and what that costs.

markwylde/bulletinAll 25 specs, the ADRs, the code and the tests.View on GitHub →