Process large JSON: a runnable NDJSON filter

Streaming is useful when your task can finish one record before reading the next. It does not make every operation small: sorting all records, retaining a large output array or parsing one enormous record can still consume substantial memory.

Reproduce an adult-user filter

Download users.ndjson and filter-ndjson.mjs into the same directory. The input contains three synthetic users:

{"id":1,"name":"Ana","age":28}
{"id":2,"name":"Ben","age":16}
{"id":3,"name":"Cyr","age":20}

Run with Node 22 or later:

node filter-ndjson.mjs users.ndjson

Expected output:

{"id":1,"name":"Ana","age":28}
{"id":3,"name":"Cyr","age":20}

The downloadable script uses built-in file streams. It reads UTF-8 chunks, carries an incomplete line across chunks, checks the example’s 64 KiB record limit, parses one record, and waits if stdout applies backpressure. It also accepts a final line without a newline. It never collects all users in an output array.

This is NDJSON: one complete JSON value per line. A newline escaped inside a JSON string is fine; a pretty-printed object spread over multiple physical lines is not the same format. Blank lines are ignored by this script. A malformed record or invalid age fails the run; previously written output may already exist. Write to a temporary destination and publish it only after a successful exit when partial results would be misleading.

A JSON array needs a different parser

For { "users": [...] }, splitting on newlines is incorrect. Use a token-stream parser for your language or change an export you control to NDJSON. Convenience layers that emit one object at a time must still materialize each object; an enormous field may exceed your budget.

jq --stream emits path/value events, not complete user objects. For the small array-shaped equivalent, this query reconstructs the array elements before filtering:

jq -nc --stream 'fromstream(2|truncate_stream(inputs)) | select(.age >= 18)' users.json

Here users.json must contain exactly {"users":[...]} with user objects in that array. The depth argument is specific to that shape. It is not a universal extractor for unrelated documents. Check the jq streaming reference before adapting it.

Choose by workload, not one file-size threshold

Measure the largest record, nesting, destination buffering and available memory. A mobile browser and a backend worker have different budgets. If you only need an example to inspect, extract a small sample locally, then open that valid JSON in the viewer or JSON-to-CSV converter. These browser tools are not advertised as streaming multi-gigabyte processors.

For Node stream semantics and flow control, see the official stream documentation. For Python, ijson offers iterator-based parsing; check its backend and numeric behavior for your environment. Compression and parser choice are separate decisions: see the compression benchmark.