feat(utils): add WriteAheadLog classes #1210

BioPhoton · 2026-01-09T21:39:57Z

Related: #1197

This PR includes:

types for source sink flows
wal.ts - general WriteAheadLog and sharded WAL inc format and filenames
wal-trace-json.ts traceEvent specific login for a WAL
profiler constants to control WAL leader
moved trace utils into profiler

Note

This PR addresses a comment related to logging.
It could also be used as file output for the logger

Warning

All filesystem operations in WAL—and indirectly in the profiler—are synchronous by design.
The eslint rules are disables over file name pattern

Why synchronous fs is used for WAL code?

Profilers are side-channels, not business logic
Data must never be lost, reordered, or partially written
process.on('exit') cannot safely run async I/O
Sync writes guarantee ordering + durability
Profiler I/O volume is small → overhead is negligible
Crash-safety > throughput

nx-cloud · 2026-01-09T21:41:17Z

View your CI Pipeline Execution ↗ for commit a3641d3

Command	Status	Duration	Result
`nx run ci:code-pushup -- merge-diffs --files=/h...`	✅ Succeeded	5s	View ↗
`nx run-many --targets=code-pushup --parallel=fa...`	✅ Succeeded	1m 28s	View ↗
`nx run-many --targets=code-pushup --parallel=fa...`	✅ Succeeded	12m 53s	View ↗
`nx run-many -t unit-test,int-test`	✅ Succeeded	7s	View ↗

☁️ Nx Cloud last updated this comment at 2026-01-18 23:17:23 UTC

pkg-pr-new · 2026-01-09T21:41:34Z

Open in StackBlitz

@code-pushup/ci

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/ci@1210

@code-pushup/cli

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/cli@1210

@code-pushup/core

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/core@1210

@code-pushup/create-cli

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/create-cli@1210

@code-pushup/models

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/models@1210

@code-pushup/nx-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/nx-plugin@1210

@code-pushup/axe-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/axe-plugin@1210

@code-pushup/coverage-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/coverage-plugin@1210

@code-pushup/eslint-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/eslint-plugin@1210

@code-pushup/js-packages-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/js-packages-plugin@1210

@code-pushup/jsdocs-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/jsdocs-plugin@1210

@code-pushup/lighthouse-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/lighthouse-plugin@1210

@code-pushup/typescript-plugin

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/typescript-plugin@1210

@code-pushup/utils

npm i https://pkg.pr.new/code-pushup/cli/@code-pushup/utils@1210

commit: a3641d3

github-actions · 2026-01-09T21:48:09Z

Code PushUp

🤨 Code PushUp report has both improvements and regressions – compared current commit f5a0e21 with previous commit 270b474.

🕵️ See full comparison in Code PushUp portal 🔍

🏷️ Categories

🏷️ Category	⭐ Previous score	⭐ Current score	🔄 Score change
Documentation	🔴 42	🔴 46
Code coverage	🟢 93	🟢 93
Bug prevention	🟡 75	🟡 75
Axe Accessibility	🟡 88	🟡 88
Code style	🟢 99	🟢 99	–
Security	🟡 51	🟡 51	–
Updates	🟡 78	🟡 78	–
Type Safety	🟡 67	🟡 67	–
Miscellaneous	🟡 67	🟡 67	–
Performance	🔴 37	🔴 37	–
Accessibility	🟢 92	🟢 92	–
Best Practices	🟢 100	🟢 100	–
SEO	🟢 92	🟢 92	–

👍 2 groups improved, 👍 12 audits improved, 👎 1 audit regressed, 14 audits changed without impacting score

🗃️ Groups

🔌 Plugin	🗃️ Group	⭐ Previous score	⭐ Current score	🔄 Score change
JSDocs coverage	Documentation coverage	🔴 42	🔴 46
Code coverage	Code coverage metrics	🟢 93	🟢 93

32 other groups are unchanged.

🛡️ Audits

🔌 Plugin	🛡️ Audit	📏 Previous value	📏 Current value
JSDocs coverage	Classes coverage	🟥 6 undocumented classes	🟨 6 undocumented classes
JSDocs coverage	Properties coverage	🟥 30 undocumented properties	🟥 40 undocumented properties
Lighthouse	Speed Index	🟥 6.0 s	🟨 5.7 s
Lighthouse	Time to Interactive	🟥 13.0 s	🟥 12.5 s
Lighthouse	Total Blocking Time	🟥 1,890 ms	🟥 2,080 ms
JSDocs coverage	Functions coverage	🟥 577 undocumented functions	🟥 580 undocumented functions
JSDocs coverage	Types coverage	🟥 268 undocumented types	🟥 258 undocumented types
JSDocs coverage	Variables coverage	🟥 268 undocumented variables	🟥 276 undocumented variables
Lighthouse	First Contentful Paint	🟥 3.1 s	🟥 3.1 s
JSDocs coverage	Methods coverage	🟨 13 undocumented methods	🟨 15 undocumented methods
Code coverage	Function coverage	🟩 94.4 %	🟩 94.6 %
Code coverage	Line coverage	🟩 92.7 %	🟩 92.9 %
Code coverage	Branch coverage	🟨 89 %	🟨 89.2 %
Lighthouse	Avoids enormous network payloads	🟩 Total size was 2,104 KiB	🟩 Total size was 2,102 KiB
Lighthouse	Largest Contentful Paint	🟥 11.5 s	🟥 10.6 s
Lighthouse	Uses efficient cache policy on static assets	🟨 31 resources found	🟨 31 resources found
Lighthouse	Metrics	🟩 100%	🟩 100%
Lighthouse	Minimizes main-thread work	🟥 9.2 s	🟥 8.8 s
Lighthouse	Reduce unused CSS	🟥 Potential savings of 95 KiB	🟥 Potential savings of 95 KiB
Lighthouse	Reduce unused JavaScript	🟥 Potential savings of 224 KiB	🟥 Potential savings of 225 KiB
Lighthouse	Server Backend Latencies	🟩 1,200 ms	🟩 930 ms
Lighthouse	Remove duplicate modules in JavaScript bundles	🟥 Potential savings of 93 KiB	🟥 Potential savings of 93 KiB
Lighthouse	Initial server response time was short	🟩 Root document took 540 ms	🟩 Root document took 390 ms
Lighthouse	Max Potential First Input Delay	🟥 980 ms	🟥 900 ms
Lighthouse	JavaScript execution time	🟥 4.3 s	🟥 4.3 s
TypeScript	Semantic errors	🟥 25 errors	🟥 26 errors
Lighthouse	Network Round Trip Times	🟩 10 ms	🟩 10 ms

652 other audits are unchanged.

# Conflicts: # packages/utils/src/lib/sink-source.types.ts

# Conflicts: # testing/test-utils/src/lib/utils/perf-hooks.mock.ts

github-actions · 2026-01-16T22:26:47Z

Code PushUp

🥳 Code PushUp report has improved – compared current commit f5a0e21 with previous commit 270b474.

💼 Project `utils`

🥳 Code PushUp report has improved.

🕵️ See full comparison in Code PushUp portal 🔍

🏷️ Category	⭐ Previous score	⭐ Current score	🔄 Score change
Documentation	🔴 45	🟡 51
Code coverage	🟢 95	🟢 95

4 other categories are unchanged.

👍 2 groups improved, 👍 9 audits improved

🗃️ Groups

🔌 Plugin	🗃️ Group	⭐ Previous score	⭐ Current score	🔄 Score change
JSDocs coverage	Documentation coverage	🔴 45	🟡 51
Code coverage	Code coverage metrics	🟢 95	🟢 95

13 other groups are unchanged.

🛡️ Audits

🔌 Plugin	🛡️ Audit	📏 Previous value	📏 Current value
JSDocs coverage	Classes coverage	🟨 3 undocumented classes	🟨 3 undocumented classes
JSDocs coverage	Properties coverage	🟥 27 undocumented properties	🟥 37 undocumented properties
JSDocs coverage	Types coverage	🟥 66 undocumented types	🟨 56 undocumented types
JSDocs coverage	Functions coverage	🟥 239 undocumented functions	🟥 242 undocumented functions
JSDocs coverage	Variables coverage	🟥 49 undocumented variables	🟥 57 undocumented variables
JSDocs coverage	Methods coverage	🟨 13 undocumented methods	🟨 15 undocumented methods
Code coverage	Function coverage	🟩 96.2 %	🟩 96.6 %
Code coverage	Branch coverage	🟩 91.8 %	🟩 92.2 %
Code coverage	Line coverage	🟩 97.8 %	🟩 98 %

435 other audits are unchanged.

13 other projects are unchanged.

hanna-skryl

The justification for sync file operations is well-reasoned 👍 I've added a few minor suggestions and comments.

hanna-skryl · 2026-01-19T16:07:54Z

packages/utils/src/lib/wal.ts

+  SHARD_ID: /^\d{8}-\d{6}-\d{3}(?:\.\d+){3}$/,
+} as const;
+
+export function soratebleReadableDateString(timestampMs: string): string {


Suggested change

export function soratebleReadableDateString(timestampMs: string): string {

export function sortableReadableDateString(timestampMs: string): string {

hanna-skryl · 2026-01-19T16:23:14Z

packages/utils/src/lib/wal.ts

+   * @throws Error if recovery encounters decode errors
+   */
+  repack(out = this.#file) {


The JSDoc says that repack() throws if recovery encounters decode errors, but the implementation logs to the console and continues without throwing. Am I missing something?

hanna-skryl · 2026-01-19T16:27:56Z

packages/utils/mocks/sink.mock.ts

+import type { Codec } from '../src/lib/wal.js';

-export class MockSink implements Sink<string, string> {
+export class MockFileSink implements WriteAheadLogFile<string> {


The TypeScript plugin reports an error here. Maybe MockFileSink should implement AppendableSink<string> instead?

hanna-skryl · 2026-01-19T16:31:46Z

packages/utils/src/lib/profiler/wal-json-trace.unit.test.ts

+  it('should create WAL format with custom directory', () => {
+    const format = traceEventWalFormat({ dir: '/custom/path' });
+
+    expect(format.baseName).toBe('trace');
+    expect(format.walExtension).toBe('.jsonl');
+    expect(format.finalExtension).toBe('.json');
+  });
+
+  it('should create WAL format with groupId', () => {
+    const format = traceEventWalFormat({ groupId: 'session-123' });
+
+    expect(format.baseName).toBe('trace');
+    expect(format.walExtension).toBe('.jsonl');
+    expect(format.finalExtension).toBe('.json');
+  });


traceEventWalFormat doesn't take any parameters 🤔

hanna-skryl · 2026-01-19T16:40:04Z

packages/utils/src/lib/wal.ts

+    const recordsToWrite = hasInvalidEntries
+      ? (r.records as T[])
+      : filterValidRecords(r.records);
+    fs.mkdirSync(path.dirname(out), { recursive: true });


There's an existing ensureDirectoryExists utility, but it's async. Maybe worth adding a sync variant (ensureDirectoryExistsSync) since this pattern repeats multiple times?

hanna-skryl · 2026-01-19T16:46:56Z

packages/utils/src/lib/wal.ts

+   * @throws Error if any shard contains decode errors
+   */
+  finalize(opt?: Record<string, unknown>) {


Same question as in repack() - the JSDoc says it throws on decode errors, but there is no explicit Error.

hanna-skryl · 2026-01-19T16:48:55Z

packages/utils/src/lib/wal.ts

+  format: Partial<WalFormat<T>>,
+): WalFormat<T> {
+  const {
+    baseName = 'trace',


There is a mismatch between the JSDoc and the actual implementation - should baseName default to 'trace' or Date.now().toString()?

hanna-skryl · 2026-01-19T16:53:44Z

packages/utils/src/lib/wal.ts

+/**
+ * Initialize the origin PID environment variable if not already set.
+ * This must be done as early as possible before any user code runs.
+ * Set's envVarName to the current process PID if not already defined.


Suggested change

* Set's envVarName to the current process PID if not already defined.

* Sets envVarName to the current process ID if not already defined.

hanna-skryl · 2026-01-19T16:54:15Z

packages/utils/src/lib/wal.ts

+/**
+ * Pure helper function to recover records from WAL file content.
+ * @param content - Raw file content as string
+ * @param decode - function fo decoding records


Suggested change

* @param decode - function fo decoding records

* @param decode - function for decoding records

hanna-skryl · 2026-01-19T16:57:17Z

packages/utils/src/lib/wal.ts

+ * @param dir - The directory to store the shard file
+ * @param format - The WalFormat to use for the shard file
+ * @param groupId - The human-readable group ID (yyyymmdd-hhmmss-ms format)
+ * @param shardId - The human-readable shard ID (readable-timestamp.pid.threadId.count format)


Suggested change

* @param dir - The directory to store the shard file

* @param format - The WalFormat to use for the shard file

* @param groupId - The human-readable group ID (yyyymmdd-hhmmss-ms format)

* @param shardId - The human-readable shard ID (readable-timestamp.pid.threadId.count format)

* @param opt.dir - The directory to store the shard file

* @param opt.format - The WalFormat to use for the shard file

* @param opt.groupId - The human-readable group ID (yyyymmdd-hhmmss-ms format)

* @param opt.shardId - The human-readable shard ID (readable-timestamp.pid.threadId.count format)

John Doe added 2 commits January 8, 2026 20:11

refactor: add general file sink logic

c03b2c4

feat: add file sink classes

756f8c0

github-actions bot assigned BioPhoton Jan 9, 2026

github-actions bot added 🔬 testing writing tests 🧩 utils labels Jan 9, 2026

BioPhoton mentioned this pull request Jan 9, 2026

feat(utils): add performance observer #1206

Merged

BioPhoton mentioned this pull request Jan 10, 2026

Implement performance profiling and trace file generation #1197

Open

John Doe and others added 16 commits January 10, 2026 01:51

Merge branch 'main' into feat/utils/file-sink

4c95897

# Conflicts: # packages/utils/src/lib/sink-source.types.ts

refactor: add trace json file

b0c9cc4

refactor: wip

1f6e326

refactor: wip

6bcb73b

Merge remote-tracking branch 'origin/main' into feat/utils/file-sink

dfb43be

refactor: wip arc

3fe6871

refactor: fix lint

67e004c

Merge branch 'main' into feat/utils/file-sink

a4a9f10

# Conflicts: # testing/test-utils/src/lib/utils/perf-hooks.mock.ts

refactor: wip

0b1fd3a

refactor: wip

bf0ecb9

refactor: wip

c88ffe9

refactor: wip

d6e75e6

refactor: wip

21575b3

refactor: wip

09437a9

refactor: wip

4705394

refactor: wip

e0c210e

BioPhoton added 3 commits January 17, 2026 20:34

refactor: wip

29016b0

refactor: wip

e60ea2b

refactor: wip

5ea2ac4

BioPhoton added 8 commits January 17, 2026 21:39

refactor: wip

73ff371

refactor: wip

34b0eb8

refactor: fix lint

dd2e959

refactor: fix tests

64501ec

refactor: add tests

eb559f5

refactor: wip

e20e804

refactor: wip

8f812c5

refactor: fix lint

a839b9b

BioPhoton marked this pull request as ready for review January 17, 2026 21:44

BioPhoton requested a review from hanna-skryl January 17, 2026 21:44

BioPhoton added 2 commits January 17, 2026 22:53

refactor: fix lint

27b1506

refactor: wip

592fb89

BioPhoton requested a review from matejchalk January 17, 2026 22:34

BioPhoton added 3 commits January 18, 2026 00:35

refactor: wip

a19070e

refactor: wip

4f8cd2d

refactor: wip

bc86d08

BioPhoton changed the title ~~feat(utils): add file sink classes~~ feat(utils): add WriteAheadLog classes Jan 18, 2026

BioPhoton added 4 commits January 18, 2026 23:46

refactor: wip

a530134

refactor: remove path logic

d9eb84b

refactor: fix lint

67f0c1e

refactor: fix unit tests on windows

a3641d3

hanna-skryl reviewed Jan 19, 2026

View reviewed changes

	export function soratebleReadableDateString(timestampMs: string): string {
	export function sortableReadableDateString(timestampMs: string): string {

	* Set's envVarName to the current process PID if not already defined.
	* Sets envVarName to the current process ID if not already defined.

	* @param decode - function fo decoding records
	* @param decode - function for decoding records

feat(utils): add WriteAheadLog classes #1210

Are you sure you want to change the base?

feat(utils): add WriteAheadLog classes #1210

Conversation

BioPhoton commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nx-cloud bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code PushUp

🏷️ Categories

🗃️ Groups

🛡️ Audits

Uh oh!

github-actions bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code PushUp

💼 Project utils

🗃️ Groups

🛡️ Audits

Uh oh!

hanna-skryl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BioPhoton commented Jan 9, 2026 •

edited

Loading

nx-cloud bot commented Jan 9, 2026 •

edited

Loading

pkg-pr-new bot commented Jan 9, 2026 •

edited

Loading

github-actions bot commented Jan 9, 2026 •

edited

Loading

github-actions bot commented Jan 16, 2026 •

edited

Loading

💼 Project `utils`