[help] Render RMarkdown Files in Parallel, but each RMarkdown file also uses parallel processing?

[emstruong]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I know we can use crew to render R-Markdown files in parallel, but if the parallel processing is used within the R-Markdown file, then should any special pre-cautions be taken for everything to work properly?

For example, I imagine that if you have 8 CPU cores and set crew::crew_controller_local(workers = 8, ...), then you could render 8 R-Markdown files in parallel. However, if each R-Markdown file itself requests 4 workers, then will targets only render 2 R-Markdown files in parallel, so that there isn't an over-allocation of CPUs?

[wlandau]

A crew worker is an R process, and you could potentially have more R processes than CPU cores. Neither targets nor crew checks for strict limits on the hardware. So it's technically possible to run 8 R Markdown reports in parallel, each of which requests 4 workers, even on a machine with fewer than 32 CPU cores. The code should still run (unless you run out of physical memory or something). But if the machine has fewer physical cores than requested workers, then your code probably won't run 32 times faster, even with CPU hyperthreading. It's generally good practice to either reduce the number of workers requested by crew or reduce the number of workers requested by the R Markdown code.

[emstruong]

Thanks!

In my own testing, I am getting the impression that R-Markdown files rendered via tar_render() seem to be done serially as opposed to in parallel. Would you happen to know if this is expected behavior or if it's more likely that I've configured something incorrectly?

My non-R-Markdown targets are dispatching and completing in parallel, so I'm not sure what's going on...

[wlandau]

deployment = "main" is the default in tar_render(). You can set deployment = "worker" to run it on a crew worker.