stream-gotchas

List of corner/edge cases when streaming JSON over HTTP,
i.e when doing something like:

// this uses 'express' and 'pg' but could be any database, any framework
db.select('*').from('messages').stream().pipe(res)

Examples:

...and others.

A draft description of each case can be found here.
Each case includes tests for its failure and tests for it's solution.

The test suite tests each case (both failure and solution) for:

Memory pressure and memory leaks.
- Are all streams amenable to garbage collection?
Runaway queries
- Are database queries still running when they shouldn't?
Database connection release.
- Does we unnecessarily hold-on to a database connection?

Install

git clone, then:

# Run a local Postgres server, then:
export DATABASE_URL=postgres://postgres:123@localhost:5432/repro

# install
npm i

# create a test DB with test data
npm run initdb

Test

To collect accurate runtime statistics, the requests are run in a separate process, via httpie, which you need to install.

Run the test cases:

npm test

Plot the Garbage Collector

The tests attempt to stress an endpoint so that Oilpan, the garbage collector, kicks in.

As the heap limits are reached, Oilpan will start freaking out and run ever more frequent compaction cycles.

Because it's a stop-the-world type of GC, it tries to avoid unnecessarily running unless it believes it's about to be OOM-ed.

Heap compaction is a process based on heuristics so sometimes it's best to view it visually.

To plot the heap while running the tests:

npm test --plot-gc

prints something like this:


client aborts request while in-flight
  when we send one request
    ✔ sends an HTTP 200 and a 20 MB response
  when we send a lot of requests
    ✔ releases database connections back to the pool
*
                                                              -- Heap size following GC --

   Cur: 17 MB
   Max: 42 MB                                                                                   ─── heap size      No leakage
          ╷
    42.00 ┼                     ╭───────╮                    ╭───────╮             ╭───────╮                                                  
    35.17 ┤                     │       │                    │       │             │       │             ╭─────────────────────╮              
    28.33 ┤       ╭─────────────╯       │      ╭─────────────╯       ╰─────────────╯       ╰──────╮      │                     ╰──────╮       
    21.50 ┼───────╯                     ╰──────╯                                                  ╰──────╯                            │       
    14.67 ┤                                                                                                                           ╰──────
     7.83 ┤                                                                                                                                   
     1.00 ┤                                                                                                                                   
          ┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬

   Initial: 18 MB                                                                                                               GC Cycles: 18


    ✔ exhibits memory spikes that return to baseline

Heap statistics are collected immediately following a collection/compaction.

Additional Reading:

Authors

@nicholaswmin