Performance concern of spawning a new worker

[josteph]

So I became very interested in this library after stumbled upon this thread. But I have several questions about the performance concern of spawning a new worker.

I have a function as following:

import { gzip } from 'fflate';

const gzipStuff = (data) => {
  const gzip = new Promise((resolve, reject) => {
      const callback = (error: Error, compressed) => {
        error ? reject(error) : resolve(compressed);
      };

      const buffer = Buffer.from(
        isDataString ? data : JSON.stringify(data),
        'utf-8',
      );

      // gzip the buffer into an archive
      gzip(buffer, callback);
  });

  return gzip
      .then((compressed) => {
            // gzip op finished, calculate the end timing

           // will be sent to backend
           return compressed
       })
}

This function is repeatedly called by many API call operations, so we can expect several Workers could be spawned at the same time.

Then I tried to throttle the CPU to be 6x slower and was quite surprised that this method is ~2x slower than zlib.gzip function. I'm aware it's because the function will spawn a new Worker to every single gzip call and it's terminated after the operation is done, so that is more costly than running the gzip operation in the same thread.

It's also mentioned here: https://nodejs.org/api/worker_threads.html

The above example spawns a Worker thread for each parseJSAsync() call. In practice, use a pool of Workers for these kinds of tasks. Otherwise, the overhead of creating Workers would likely exceed their benefit.

---

What could be improved here? Thanks!

[101arrowz]

For some buffers fflate will always be substantially slower than zlib. In this case the overhead of the worker call is probably not the only reason fflate is slower. However it is probably worth looking into whether a worker pool could be integrated into fflate, I'll do that for the next release.

[robatwilliams]

Related - #58