Handling browser limitations on Blob storage

[hrishikesh-k]

Hi, I was wondering what's the general recommendation to handle the blog storage limitation? I have read that on Chrome, 500 MiB is the max amount of data that can be stored? In that case, I'd like my app to be able to handle situations in which the zip might be more than that.

Before I try to implement my own logic to handle this, does this really apply to zips created by fflate? I'm not sure about the implementation logic, so if I created zips as I have mentioned here: #92, would I run into problems for large zips?

If yes, here's my plan:

1. Before adding the file to the zip, I'd check for the current zip size.
2. If by adding the current file to it, the zip size is going over or close to 500 MiB, I'll close that zip and add the file to a new zip.
3. I would then continue the process for all files.

Would this be a correct approach? If yes, how can I check for the current size of the zip or should I use my current file size to determine the size of the zip?

[101arrowz]

If you're using ZIP streams you don't even need to do this, you can save memory by not accumulating into a blob and instead piping to streamsaver. You can do the following:

zipReadableStream.pipeTo(streamSaver.createWriteStream('filename.zip'))

I've tested this on multi-gigabyte files, works like a charm with very low memory usage.

[hrishikesh-k]

Hi @101arrowz,

Thank you once again for responding! This seems like it could work, however I'm a little confused as to where exactly it should go. I mean, do I run this after zip.end() or after each file?

This is my current code:

downloadAll(index, zip, zipReadableStream) {

  zip = zip ? zip : new Zip()

  zipReadableStream = zipReadableStream ? zipReadableStream : new ReadableStream({
    start(controller) {
      zip.ondata = (error, data, final) => {
        if (error) {
          controller.error(error);
        } else {
          controller.enqueue(data)
          if (final) {
            controller.close()
          }
        }
      }
    }
  })

  callApi('url').then(file => {
    (async () => {
      const zippedFileStream = new ZipPassThrough(file.path)
      zip.add(zippedFileStream)
      const fileReader = file.stream().getReader()
      while (true) {
        const {done, value} = await fileReader.read()
        if (done) {
          zippedFileStream.push(new Uint8Array(0), true)
          break
        }
        zippedFileStream.push(value)
      }
    })().then(() => {
      downloaded++
      if (downloaded < files.length - 1) {
        downloadAll(zip, zipReadableStream)
      } else {
        zip.end()
        zipReadableStream.pipeTo(streamSaver.createWriteStream('file.zip'))
      }
    })
  })

}

I'm not sure if I'm doing it right, because at the moment, I'm saving it after the zip is all done, so it would probably share the same fate as before. Do I do this in a different stage than my current implementation? I hope so.

On a side note, unrelated to fflate, the above code gives me a 404 and points to this address when trying to save the file: https://jimmywarting.github.io/StreamSaver.js/localhost:3000/217602/file.zip. Maybe you have an idea of what I'm missing. No problem if not, I'll figure it out later when I sit for some more time with the project.

[101arrowz]

Call it when you create the zip, i.e. just after the first assignment to zipReadableStream (you probably need to replace your ternary with an if statement for this). As fflate pushes chunks to the zipReadableStream, streamsaver will consume those chunks, save them to the disk, and free them, meaning you should peak at around 1MB of memory or so used even for multi-gigabyte files. That error suggests you might not be using a supported browser, the latest Chrome works for me.

[hrishikesh-k]

Awesome, thank you so much for the explanation. This works well!

I figured out the 404 too. I was doing import streamSaver from 'streamsaver' and using it as streamSaver.createWriteStream('file.zip') which was giving me 404 for some reason. I changed it to import { createWriteStream } from 'streamsaver' and used it as createWriteStream('file.zip') and now it works - not sure why though. But that's unrelated to this discussion. I'd have to make it work the first way as I need to change the download URL instead of the default one.

But thank you so much for your help!