Bulletin, part 1: 25 specs, 130 hours and $51 of DeepSeek
I wanted to know how far a cheap model can get on a big project if you give it good specs, a strict process and someone checking its work. So I picked a community forum in the style of oldschool web bulletin board systems, with forums, topics, private messages, moderation, permission masks, themes, plugins and a staff console. I planned it with Claude Opus 5.5, then handed it over: Claude ran the build unattended through a Terminay workflow, DeepSeek Flash wrote nearly all of the code, and Codex reviewed every change.
I called the project Bulletin, and the code is on GitHub at markwylde/bulletin. It ran from Monday morning to Saturday night:
This is part 1. It covers how the build ran, what each agent did, what DeepSeek kept getting wrong, what the same tokens would cost elsewhere, and a review of the code it produced. Part 2 rebuilds the UI, which is bad, and fixes a few bugs.
- 1 Planning
- 2 The build loop
- 3 Change by change
- 4 Codex
- 5 DeepSeek, good and bad
- 6 Costs
- 7 The codebase
- 8 Testing
- 9 The UI
Who did what
I split the work across three models:
Claude Opus 5.5
Claude CodeWrote the specs with me, built the harness that drove DeepSeek, read every review, decided what blocked a merge and wrote the fix prompts.
DeepSeek Flash
opencodeWrote the code, ran the tests, ticked tasks and committed, one task at a time.
Codex
codex exec, GPT modelsAfter every round of work, read the diff against the specs and reported P1 and P2 findings.
My instruction at the start was “implementor: deepseek / high level steering: claude / review/second opinion: codex”. I also asked for auto-merge once tests passed and Codex signed off, and for all 25 changes to run back to back, pausing only for a failed spike or a fundamental problem. It never paused, and DeepSeek never wrote the BLOCKED.md file it was told to write if a spec couldn’t be met. From there the build ran hands off.
Planning before any code
Planning took a little over three hours on the first morning, all with Claude. I’ve written before about why I use spec-driven development and OpenSpec; this is the first time I’ve used it on something this big.
- 09:23The idea
I described the forum and the stack I wanted: a Bun monorepo, an Astro user app, a Vite staff app, Bun APIs, Postgres with pg-boss, Docker Compose. I asked Claude for "a questionnaire on all the features you can think of".
- 09:2462 questions
Claude opened a questionnaire in a Terminay window and I tapped through it. One of my notes became a rule for the whole project: "locally eventhing should run in 2 steps. 1: clone, 2: docker compose up --watch... only postgres is needed and stateful. no redis".
- 09:4017 ADRs
Claude wrote 17 architecture decision records, a feature list and the agent instructions.
- 10:26First Codex review
Five serious design problems, including cache entries shared between members of different groups, permission revocation that didn't take effect straight away, and plugin processes that weren't isolated. That added three ADRs, for 20 in total.
- 10:4325 OpenSpec changes
I pointed out that ADRs aren't specs. Claude split the product into 24 OpenSpec changes in about 19 minutes: 55 capability specs, about 335 requirements and 404 scenarios. A review then split
plugin-sandbox-spikeinto its own change, which made 25. - 11:32Codex and agy together
The last of seven review rounds: Codex three times, two external reviews I pasted in, a Codex check, then Codex and
agyside by side. About 40 distinct findings, fixed by 11:56. - 12:40Go
I told Claude to start: DeepSeek implements, Claude steers, Codex reviews.
New to ADRs and OpenSpec?
An ADR (architecture decision record) is a short document that records one decision and why it was made: “Postgres is the only stateful service”, “every permission check goes through one function called can()”. The agent instructions made them binding: if a task needed to contradict an ADR, the agent had to stop and propose a new one.
OpenSpec organises work into changes. Each change folder has a proposal, a design, a tasks.md checklist, and spec files with requirements and scenarios (“GIVEN a guest, WHEN they open a members-only forum, THEN they get a 404”). The agent works through the checklist and ticks tasks off. The scenarios tell the reviewer exactly what to test.
I kept sending the documents back for review because a spec problem caught on day one costs a few minutes of Claude rewriting markdown, while the same problem found in change 20 costs hours of DeepSeek rework.
2 · The build loopThe build loop
Once I said go, Claude wrote a small Python driver (drive.py) and a few scripts in its scratchpad. By the middle of the build, every change went through this loop. The first four changes came before final.sh existed, and the last one was built by Claude after the DeepSeek balance ran out:
- Worktree. Create a git worktree and branch for the change.
- Task groups. For each group in
tasks.md, startopencode runwith DeepSeek Flash on a prompt: the base rules, the change name, and “do group N”. DeepSeek reads the ADRs and specs, writes code, runs the tests, ticks tasks, and commits one task at a time. - Three review rounds. Run
bun run check, thencodex execin read-only mode with a reviewer prompt. Codex returns numbered findings tagged P1 (security, data loss, spec violation, broken build) or P2 (bug, missing test, ADR deviation). DeepSeek gets the findings and fixes them. Repeat three times. - Final gate. Claude runs
final.sh: the full check, the unit tests, a fresh Docker stack, the database suite against real Postgres, both production builds (with a grep for Node imports leaking into browser bundles), the E2E suite, a curl of both sites, and one more Codex sign-off review. - Steer. If the gate found anything, Claude writes a targeted prompt for DeepSeek in a fresh session (“## Your task now”, plus the findings) and runs the gate again.
- Merge. Merge to main, archive the OpenSpec change, tick the step in the Terminay workflow, start the next change.
Claude tracked all 25 changes in a Terminay workflow and ticked each one off as it merged. A /loop kept it going from one change to the next. The loop fired 107 times. Claude handled 640 background notifications, made 352 workflow updates, and slept the rest of the time.
- 2h 41m1foundation-dev-environment
- 2h 10m2plugin-sandbox-spike
- 1h 00m3i18n-foundation
- 2h 52m4identity-sessions
- 2h 00m5staff-console-shell
- 1h 11m6feature-flags
- 2h 04m7groups-permissions
- 5h 57m8rendering-and-caching
- 4h 43m9forum-structure
- 5h 24m10content-pipeline
- 5h 24m11topics-posts
- 4h 04m12attachments-avatars
- 5h 54m13events-notifications
- 4h 39m14reactions-reputation
- 6h 27m15member-profiles
- 3h 58m16conversations
- 12h 25m17moderation
- 3h 23m18search
- 6h 35m19themes
- 12h 56m20plugin-platform
- 11h 08m21public-api-webhooks
- 8h 32m22pwa-push
- 6h 22m23privacy-gdpr
- 4h 45m24seo-feeds
- 4h 06m25operations-observability
The rules grew as things broke
The base prompt is short, about 40 lines. Most of its “Headless run” section, and a few changes to the driver, came from something going wrong in the first 31 hours:
- Mon 19:21DeepSeek killed its own agent
It ran
Never usepkill -f "docker compose up --watch", which matched its own command line and killed opencode.pkill -forkillallwith broad patterns. - Mon 19:37A blank staff app
A shared package barrel had pulled
Browser code must import client-safe subpaths only.node:cryptointo the browser bundle, so the staff app loaded blank. Nothing in the tests ran a real browser yet, so Claude added a browser gate tofinal.sh. - Mon 21:50DeepSeek took down my stack
It ran
Never start, stop or touch the defaultdocker compose downin its worktree. Every checkout shared the compose project name, so that stopped my own running stack. This led to a 26th change,dev-stack-isolation, which made the project name and port configurable. It took 24 minutes, and the agents moved to port 18080.bulletinproject or port 8080. - Tue 00:02A task group committed onto main
opencode's shared background service ran a whole task group in the main checkout instead of the worktree. Claude moved the commits back.
The driver runs opencode with--standalone. - Tue 03:05Flaky E2E tests
Tests clicked buttons that were still disabled.
Wait forselector:not([disabled])before clicking. - Tue 08:54Sessions stopped silently
Long opencode sessions sometimes "silently stop with exit 0 and no changes" once the context is huge.
Every steering prompt gets a fresh session. - Tue 13:51450 tests weren't running
Run the DB suite before finishing any group that touches the database.bun run checkskipped the database integration tests, about 450 of them. - Tue 19:15A cap on final gates
Content-pipeline kept failing its final gates, and each Codex sign-off found new P2s (it took six gates in all). Final gates were capped at two, except that a P1 or a security finding always blocks the merge. Non-security P2s go into a follow-up file for a later change.
Never merge with a P1 or anything tagged[security], whatever the cap says.
That last rule is why some changes took 12 hours, as the breakdown below shows.
Change by change
Click any change to see its numbers, the Codex results round by round, and the notes from the logs. Blue is time spent in DeepSeek’s driver loop; orange is time spent in final gates and Claude’s steering.
17. moderation
Queues, reports, warnings, bans, IP tools, multi-moderation.
Codex reviews in order (P1/P2)
From the logs
Largest change (9 groups, 26 tasks, 43 commits, ~29k lines). Driver alone 8h28m. Then 8 final gates / 7 steers in 4h: almost every gate found 'one more security P1' (IP access, session-unaware checks, ban bypass via edit, approval bypass), plus E2E regressions and beacon 429s.
Solid pills are the driver's review rounds; dashed pills are final gates. Red means at least one P1.
The same build as a timeline, with the incidents along the way:
Why moderation took 12h25m
Most changes took two to six hours. Three took more than eleven, and the gate cycles explain all three. Topics-posts is a useful comparison first: it took 5h24m, but only because its third task group spent 1h54m fighting Docker (stuck test containers, experiments with DOCKER_HOST) before Docker filled the disk and stalled. Its gate phase was short: 2 gates and 1 steer, about the same as the other mid-sized changes. The mini PC is also a runner for my Gitea Actions, so CI jobs from other projects were filling the same disk at the same time. That skews the timings a little, by no more than an hour across the whole build.
Moderation was the biggest change and the hardest to get past Codex. With 9 task groups, 26 tasks, 43 commits and about 29,700 lines, DeepSeek’s driver alone took 8h28m. The first seven of its eight final gates each found one or two new security P1s:
- IP lookups ignored password-forum unlocks.
- Edits bypassed bans.
- Bans and approvals imposed by warnings had gaps (found at four gates for bans, two for approvals).
- Temporary grants to see a member’s IP address survived the loss of the access that created them, or could be redeemed by another staff member (five findings).
Each of those cost a fix by DeepSeek plus a full gate, about 30 minutes to an hour. From gate 4, Claude’s prompts told DeepSeek to audit the whole change for the same class of bug instead of fixing the cited line (“Before you finish, audit the whole moderation change for the same class of bug.”). When gate 5 still found “one more security P1, again in IP access”, Claude ran its own security review. Moderation took 8 final gates and 7 steers to merge.
Plugin-platform (12h56m) was the same story with sandboxes. Codex’s first round found 15 P1s, and the gates later found app roles holding pg_read_file, a broker that never checked approved capabilities, and redirects that escaped the egress allowlist. Public-api-webhooks (11h08m) spent most of its time on flaky tests and an intermittent 500; its code was mostly fine. The 500 came from the core theme hitting the 50 ms template render limit under load.
A gate cycle is a fresh stack build, the unit, DB and E2E suites, and a Codex review, which is 15 to 30 minutes before any fixing starts. So the number of gate cycles mattered much more than code size, and the security-heavy changes needed the most cycles. Switch the chart between the two:
Most of the time went on shell commands
DeepSeek’s assistant messages span 103.9 hours. Shell commands account for 77.3 of them:
- Docker builds and stack restarts 23.7h
- Database integration suite (950 runs)19.5h
- E2E suite (1,307 runs)14.7h
- Unit tests and checks 3.5h
- Other shell commands 15.9h
- The model thinking and writing 26.6h
That leaves the model itself thinking and writing for roughly 27 to 30 hours of the 130. It all ran on a GMKtec mini PC (Ryzen 9, 8 cores, 32GB of RAM) running Linux. On a more powerful machine, the build would have been a lot quicker.
What I’d change about the compute
The model waited on shell commands for more than twice as long as it spent thinking. Looking at which commands, most of that wait was avoidable:
- Every test run rebuilt the test image. DeepSeek’s most expensive command was
docker compose run --build --rm test, which runs the whole suite. It ran it 236 times, for 14.7 hours, nearly four minutes a go. The test Dockerfile copies the whole repo beforebun install, so any code change invalidates the cache and reinstalls every dependency. Copyingpackage.jsonandbun.lockfirst, or mounting the source into a long-lived test container, would remove most of that. - It ran the whole suite when one file would do. Targeted runs (
bun test <path>) were 488 runs and only 3.5 hours. Running just the affected tests during a task group, and the full suite at the gate, would have saved hours per change. - The stack restarted a lot.
docker compose up -dran 444 times, averaging nearly two minutes. A stack started once with Composewatchwould sync code into the running containers, so it would only need a restart when dependencies change. The final gate could still start a fresh stack. - Test Postgres wrote to disk. Each integration test already clones a migrated template database, which is quick. Running the test Postgres on tmpfs with
fsync=offwould make the 950 integration runs faster still. - The machine was shared. The mini PC was also a Gitea Actions runner. A dedicated machine, and a faster one, helps every item above.
What Codex caught
Those repeated reviews cost time, but they caught what DeepSeek’s own tests missed. In 145 review sessions Codex raised 591 findings, or about 533 distinct issues once you remove the ones raised again because a fix was incomplete. About 507 were fixed before merge, 24 non-security P2s were deferred, and 2 were overridden: one was out of scope and one was a flaky test. None was rejected as technically wrong.
Every one of DeepSeek’s 73 driver review rounds found at least one P1 or P2. Round 1 across all changes found 87 P1s, 40 of them security. After two fix rounds, round 3 still found 31. Only 12 reviews in the whole build came back clean, and all of them were final gates.
Every finding
Explore every finding below. I assigned the categories by hand from the titles, so treat the counts as estimates.
- Saved requests survive revocation and account switches.
- Hidden forum activity leaks publicly.
- Deleting a target group makes restricted content public.
- Public history bypasses content permissions.
- Poll results leak across forum permissions.
- Overlay accepts arbitrary topic IDs.
- Hidden poll HTML contains voter totals.
- Topic redirects disclose restricted titles.
- Hidden topics remain accessible directly.
- Permission checks use stale forum IDs.
- Notifications and digests expose hidden posts/topics.
- Unavailable follows still disclose current titles.
- Saved reauthentication drafts leak across accounts.
- Unapproved-post reactions leak to ordinary readers.
- Reputation giver identities escape the staff-only boundary.
- Staff reputation listings disclose inaccessible topic titles.
- Private-forum activity leaks through board totals.
- Friend requests reveal who ignores the requester.
- Deleting a group can publish restricted profile fields.
- Leaderboards retain contributions moved into private forums.
- Bulk topic deletion leaves replies public and cannot restore the deleted opener.
- Deleted topics remain readable through other surfaces.
- Hidden-forum metadata leaks through prefix options.
- Core reads can disclose hidden content.
- Hidden-topic posts leak by ID.
- Export bypasses current forum permissions.
- Export leaks restricted poll text.
- Feeds and sitemaps bypass board-offline permissions.
Three themes make up 38% of everything Codex raised:
- Permissions and visibility (87 findings). The main page called
can(), but totals, notifications, exports, leaderboards, prefixes, redirects, edit history and poll results skipped the check. “Private-forum activity leaks through board totals.” “Notifications and digests expose hidden posts/topics.” “Any authenticated member can retrieve private, unapproved or deleted posts and revisions by ID.” - Not actually done (82). DeepSeek ticked tasks in
tasks.mdwithout wiring the code up, despite the rule “No placeholder or stub implementations”. In plugin-platform round 1: “The production runtime still runs only the demo.” In reactions-reputation, “Task 2.3 falsely claims profile display is complete” came up in all three rounds. There were also 34 tests that couldn’t fail, like a permission test whose fixture already filtered out the thing it was checking for. - Caching (54). Every new piece of data shown in cached HTML needs its cache tag bumped when it changes. DeepSeek rarely did this unprompted: moderator lists, emoticons, ranks, badges, signatures, poll results. One P1 found that author-only edit forms were cached by permission bits alone, so another member with the same permissions got the author’s form.
A few more that stood out:
<noscript><p title="</noscript><img src=x onerror=…>"> got through both sanitiser layers. Only the Content Security Policy stopped it.
A 17 KB PNG became a 230 KB WebP after re-encoding, getting round the upload quota.
The user ID from DarkAuth, the identity provider, was trimmed before lookup, so "admin" and " admin " resolved to the same member.
GET …?method=POST became a plugin POST that skipped the CSRF and Origin checks.
AUTH PLAIN was sent over plain TCP.
demo-owner's runtime role had the same name as demo's owner role, so uninstalling one plugin could drop another plugin's objects.
Fixes that created new bugs
For this post, Claude ran git blame on each cited line at the branch head just before that review, to find the last commit that touched it. That worked for 455 findings. For 71 of them, the line was last written by a commit fixing an earlier review rather than one implementing a task, and 22 of those were P1 or security. That doesn’t prove the fix caused the bug, but some cases are clear:
- In plugin-platform, the round-1 fix for a security finding added migration 0115. Gate 2 found that migration had given every app role, including the read-only one, file-read and large-object functions.
- In events-notifications, the fix for “unsaved edits are lost during reauth” created “saved reauthentication drafts leak across accounts”.
- Four of moderation’s ten final-gate security P1s sit on lines written by earlier fixes.
Codex isn’t deterministic
Forum-structure gate 1 came back clean, and the gate was only rerun because E2E failed on stale Docker images. Gate 4, on essentially the same code, found a new P1: deleting a target group made restricted content public. Public-api-webhooks alternated clean, P2, P2, clean, clean, P2 over six gates. Pick a change below to see its gates.
Codex also changed models during the build: gpt-6.1-sol on low effort for the first ten changes, gpt-6-astra on medium until plugin-platform’s last gate, then gpt-6.1-sol on medium to the end. Low-effort reviews took 1 to 2 minutes and medium ones 3 to 5. All 145 reviews used 181.6M tokens and about 5.8 hours.
P1s found in the driver's review rounds, all changes
Dark red is the share tagged [security]. Clean driver rounds in the whole build: 0 of 73.
Final gates, one box per Codex sign-off
Which model reviewed each change
What DeepSeek got wrong and right
Across all those reviews, DeepSeek kept making the same few mistakes. It was also good at a lot of things.
Kept coming back · changes affected, of 26
can() 5Caught and fixed itself
bun run check first timeMistakes that kept coming back
- Permission checks on the main path only. The same hole appeared in 12 of 26 changes. Permission logic written outside
can()was flagged 6 times in 5 changes, even though the agent instructions say it in bold. - Tests run as the database owner. Code passed tests run as the owner, then failed under the real database roles. Digests and orphan cleanup failed under the production worker role, and timed bans never restored. Codex gave the same advice every time: run the test as
workers. - Check-then-act races. The code checked authorisation before taking a lock and trusted the result afterwards, so concurrent topic moves, splits and uploads could race a staff block. There were also 25 plain races, like flood control that checks first and records later, so concurrent requests all pass.
- Fixing the line instead of the class of bug. Claude’s prompts had to say “Do not fix only the cited lines” and “Treat each as a class”, and moderation still kept turning up the next one.
Mistakes it caught and fixed itself
DeepSeek was good at anything its own tests could see. Of its 3,874 test or check commands, 1,135 (29%) failed, and it worked every one back to green. It also fixed the TypeScript errors it hit 243 times. After its task groups, bun run check passed straight away in 17 of the 20 changes that logged it. It was honest about what it couldn’t run: “Note: the checkout’s pre-existing incomplete node_modules still prevents a full workspace bun run check”.
It missed anything its tests didn’t exercise: security and visibility checks, database roles, and code that was never connected to the running app. Codex caught almost all of those. Claude caught the infrastructure traps. One React StrictMode bug, which double-posted a single-use login challenge, only showed up in a real browser.
What went well
Only 154 of 26,736 tool calls errored (0.6%), and there was one hallucinated path. It made a commit per task with a consistent message, ticked tasks only after tests (mostly), and shut its stacks down afterwards. It never invented a blocker or worked around an ADR. Smaller changes went straight through: i18n-foundation in 1h00m, feature-flags in 1h11m, groups-permissions in 2h04m. And it built 23½ of the 25 roadmap changes, over 300k added lines and about 430 commits, for about $51.
Where it wasted effort
- The whole context was sent on every request. The driver reused one opencode session per change across every group and fix round. The context grew to 300K–560K tokens on average (1.04M at most), and every call sent it again. 99.7% of DeepSeek’s tokens were cache reads. That’s cheap on DeepSeek, at $0.003 per million, but it’s why the token count is so large. A fresh session per group would probably have cut it by about half.
- Edits through Python heredocs. DeepSeek made 3,240 edits by writing Python scripts in the shell, against 3,777 uses of the edit tool.
- Flaky tests were chased across gates. One steer on public-api-webhooks ran for 3 hours, including 20-minute full E2E runs inside opencode, before Claude killed it, reviewed the work and committed it itself.
DeepSeek ran out of balance twice. The first time, at 01:25 on day five, a top-up cost 35 minutes. The second time, at 18:04 on the last day, Claude stopped using opencode and finished the build itself. Claude subagents fixed the last seo-feeds findings and built all of operations-observability, which is the missing half of the 23½.
6 · CostsWhat it cost
This is DeepSeek’s usage page at the end of the build:
That’s 9,955,557,125 tokens and 24,837 requests for $50.00 plus ¥6.14, about $50.91 in total, and the balance went to −$0.01. A small part of it wasn’t Bulletin: the 30-day view includes a little earlier usage around 4 November, and 30 of the requests weren’t on deepseek-flash. opencode’s own records for the build come to 9.83 billion tokens, 98.8% of the dashboard figure.
How it’s that cheap
DeepSeek Flash charges $0.003 per million tokens for a cache hit, off-peak. A cache miss is $0.15 and output is $0.60. Peak hours cost double. opencode recorded 99.7% of its tokens as cache hits, which comes to $39.96 at off-peak prices. The bill is about $11 higher; some peak-hour traffic, or a slightly lower cache rate on the requests opencode didn’t record, would explain that.
What’s a cache hit?
An agent sends the whole conversation to the model on every request: the instructions, every file it has read, every command output. Most of that is identical to the previous request. Providers keep a copy of that prefix, and when the next request starts with the same text, they charge a fraction of the normal input price for it. That’s a cache read, or cache hit. Only the new part at the end is charged in full.
For a long agent session, cache reads are almost all of the tokens. Here they were 99.7%. So when you compare models for agent work, the cache-read price matters far more than the headline input price.
The same tokens on Claude or OpenAI
Here’s what the same 9.96 billion tokens would cost at API list prices. Move the cache slider to see how much the cache-hit rate matters.
9,955,557,125 tokens from DeepSeek's dashboard, priced at list rates with no batch discounts. Uncached input is billed as a cache write. This assumes the same token count on every model, which isn't true: a stronger model might loop less and use fewer tokens.
At a 99% cache rate (a little below what opencode recorded, to match the bill), the same tokens cost about $1.4K on Sonnet 5.5 or short-context gpt-6.1-sol, and about $2.8K on Opus 5.5 or long-context gpt-6.1-sol. That’s 27 to 54 times what I paid. The average request carried about 400K tokens of context, so many gpt-6.1-sol calls would land in the >272K price tier.
A stronger model wouldn’t use the same number of tokens, though. It might loop less, get the permission checks right the first time, and need fewer gate cycles. These numbers price DeepSeek’s workload on other models; what Bulletin would cost to build with Opus depends on how much work Opus needed.
On subscriptions instead
I pay for Claude Max ($100 a month, previously called Max 5x) and ChatGPT Pro ($100). Neither vendor publishes limits in tokens, so the weekly figures below are community estimates from small samples. Pick a combination of plans:
Time to use 9.96B tokens
What that costs
Tokens per week are community estimates from small samples; neither vendor publishes limits in tokens. Faded bars show the slow end of each range. Subscription cost is the monthly fee prorated over the weeks, and assumes you can keep every weekly cap saturated. API prices assume a 99% cache-read share, like this build.
My guess before I looked was “about 10 times as long on one subscription, 5 times with two”. Compared with the 5½ days this took, the estimates say roughly 10–13 times as long on Max alone and 4–7 times with both. The fees for that time come to about $160–260, three to five times the DeepSeek bill but far below API prices. All of that assumes you can keep every weekly cap saturated, which the five-hour windows on Claude make hard in practice.
What Claude and Codex used
Claude and Codex used far fewer tokens than DeepSeek, and both fitted inside the subscriptions I already pay for:
The main session plus two build subagents. It ran in one context for 5½ days, auto-compacted 12 times and made about 3,700 API calls in 134 hours (2,900 of them in the main session), because it spent most of the time waiting on background events. By the community estimates that's two-thirds to four-fifths of one week of Max.
gpt-6.1-sol prices145 reviews in about 5.8 hours. By the community estimates that's 11–22% of one week of Pro.
The codebase
So what did all that produce? Bulletin is a Bun monorepo with 7 apps, 12 packages, 9 local mocks and a 16-service compose file. It has 144 SQL migrations, 2,294 translation keys in each of English, Spanish and French, and 34 runtime dependencies. It uses no ORM, no HTTP framework and no Redis.
Caddy splits traffic across four origins: the forum, the staff console, files and plugins. The forum app (Astro) reads Postgres directly with a read-mostly role and renders HTML, and every write goes through an API (user-api or staff-api). In the same transaction as the write, the API bumps version numbers on cache tags and appends an event row. Postgres NOTIFY then wakes the other processes so they drop stale HTML. A workers process turns events into jobs: emails, push, webhooks, CDN purges. Plugins run in their own locked-down container and only reach the rest of the system through a broker.
Caching by what you can see
This is one of the parts DeepSeek built well. A forum page looks almost the same for most members. Caching a copy per user wastes memory, and caching one copy for everyone leaks private forums. Bulletin caches by a visibility key instead: a hash of the permission answers the page actually depends on. Two members in different groups who can read and reply in the same forums get the same key and share one cached copy. Personal bits (unread markers, notification counts, “you reacted”) are fetched afterwards through a small overlay request and patched into the page.
Each page has to declare which permission bits it reads. In dev and test, reading an undeclared bit throws an error, so a missing cache dependency becomes a failing test. And if a process stops hearing invalidation messages from Postgres, it stops serving from the cache and renders every page fresh.
The plugin sandbox
Plugins are untrusted server code, so each one runs as a Bun process inside nsjail, a Linux process jail. It gets its own user, an empty network namespace, a read-only bundle, cgroup limits on memory, processes and CPU, and a seccomp policy that blocks every system call it doesn’t need. Anything it wants goes through a broker. The broker parses SQL into a syntax tree and only allows the plugin’s own schema. Outgoing HTTP is limited to the hosts in the plugin’s manifest, with checks for DNS rebinding and redirects. Reads from the core are signed calls that run can() as the viewer. A 17-case malicious-plugin suite (fork bomb, output flood, DNS rebinding, reading sibling files and so on) runs in CI inside the real container.
CI found a real sandbox bug that local runs never hit: on the x86_64 runners, Bun calls the legacy symlink() syscall, the seccomp policy didn’t allow it, and the kill action left the plugin hanging instead of crashing.
What’s good
There is one can(). Every database role has its own grants in migrations. Job payloads carry IDs only, and emit() rejects payloads that don't match the event catalog.
IP addresses and emails can only be read through SECURITY DEFINER functions with a pinned search_path, and reading one needs a committed audit row first.
Liquid templates with raw unregistered, filesystem tags removed, strict variables and render limits.
Almost every exported function is used somewhere.
What’s weak
ADR 0009 says admins install plugins from a marketplace at runtime, and the install task is ticked. But plugin-host runs exactly one bundle, from an environment variable, and the lifecycle code is only used in a test with a fake runtime. The sandbox works; there's no way to install a plugin into it. Codex flagged "not wired" in plugin-platform three times, and this one still slipped through.
There's no production guard.
isRecord40 define their own json() helper and 7 their own escapeHtml. A dev secret is hard-coded as a fallback in 17 places. Each agent run wrote its own helpers instead of finding the shared ones.
staff-api uses a typed route table with a permission per route. user-api is one if (method && pathname) chain with 70 copies of if (session === null) return 401.
staff-api/src/api.ts is the biggest. The notifications and themes repositories are about 1,600 lines each, and the themes admin page is one 1,000-line React component with 18 useStates.
pii.read_post_ip is redefined in five of them. They should be squashed before anyone deploys this.
Fine for a demo, slow at scale.
For six days of agent output it holds together well. The rules that matter (authorisation, visibility-keyed caching, personal data, sandboxing) are enforced in code and tested. The debt is what you’d expect from many separate agent runs: duplicated helpers, inconsistent style, oversized files, and a couple of tasks marked done that you only spot by reading the code against the specs.
8 · TestingTesting
Those rules hold because the test suite checks them. There are 483 test files with about 2,632 test cases, and about 0.71 lines of test for every line of code.
| Layer | Files | Cases | How it runs |
|---|---|---|---|
| Unit | 220 | 1,333 | bun test, part of bun run check |
| Database | 171 | 1,062 | Real Postgres. Each test file gets its own database, cloned from a migrated template |
| Full-stack integration | 7 | 26 | Over HTTP against the running compose stack |
| E2E | 84 | 194 | Bun.WebView driving Chrome at 360, 390 and 1440 px |
| Malicious plugin | 1 | 17 | Inside the real plugin-host container |
Two kinds of test are specific to this design:
- Permission-denial tests. About 463 test names are refusals: “a guest is refused…”, “denies…”, “403…”. The overlay test asks for a private staff topic’s ID and checks it doesn’t leak into unread markers, editable posts or anything else.
- Cache-variant tests. A shared harness renders each registered route twice for each viewer case. The variants must differ, the second render must be a cache hit, and “leak markers” planted in privileged content must never appear in a less privileged variant.
There are also architecture tests that check structure statically (no direct writes to the cache tag table, content queries only in permission-filtered repositories) and, during the build, a nightly load test with a million posts.
During the build, CI ran on my Gitea runners. It failed on every run for the first four days, because it needed sudo the runners didn’t have, a Postgres health check raced, and E2E took 14 minutes. Claude fixed it on the last day: each run gets its own compose project and port, and E2E is split into eight shards plus a full-stack shard. Getting CI green found four real bugs that local Docker never hit, including the seccomp one above and a race in marking conversations as read.
Some integration test files are huge (one is 1,911 lines) and redeclare the same helpers, and the E2E tests average 2.3 cases per file because each one walks through a long journey.
The tests cover behaviour, permissions and speed. Nothing in the specs or the tests said what the UI should look like.
The UI
Here’s everything, as it is. I haven’t fixed anything. The forum app is on the first tab, the staff console on the second.
The member-facing site, rendered by the default Liquid theme. Desktop at 1440 px, captured at 2x. Click any screenshot to open it full size; arrow keys move between them.
The UI is bad, and for a real forum it’s unacceptable, even though everything works. You can log in, post, react, quote, start a conversation, report a post, ban someone, edit a permission mask, export your data and switch to French, and the seed data is rich. But it all looks like a default form with a blue button. The staff console puts every setting in one column and makes every action a full-width button.
I expected this before I started. The specs did say “mobile-first at 360 px” and listed the screens and fields, but they didn’t say what good looks like, and DeepSeek didn’t fill the gap. I think that apart from Claude and GPT, basically no model builds a decent UI without detailed prompting.
Where part 1 ends
DeepSeek’s $51 forum works and has real security engineering, but nobody would want to use it as it looks today. There are also a few bugs, like the plugin marketplace that can’t install a plugin. I’d expect a few bugs from any model, even the best ones, so that’s fine.
Part 2
In part 2 I’m going to throw away both front ends, the forum theme and the staff console, and have Claude Opus 5.5 rebuild them on top of the same APIs, view models and tests. It will also fix a few of the bugs from part 1. The back end stays.
It will be the same approach as part 1: plan it with Claude first, then hand it off and let it run unattended. We’ll see what Claude does with the UI, and what that costs.
markwylde/bulletinAll 25 specs, the ADRs, the code and the tests.View on GitHub →