Allow the use of SLURM clusters which forbid --mem and --mem-per-cpu options

[phupe]

New feature

Allow the use of SLURM clusters which forbid --mem and --mem-per-cpu options.

Use case

We have been unable to use nextflow of some national supercomputers from tier-0 which do not allow the use of the --mem and --mem-per-cpu options. This SLURM configuration is made on purpose to make easier the accounting of the CPU time which have been used the users. As this is the policy of these supercomputer, we have to deal with this setting that they will not change.

As a consequence, any nextflow process which sets the memory directive just fails since the --mem option is added in the .command.runfile. Therefore, there is no way to run a nextflow pipeline unless we rewrite the it without the memory directive, which is not a realistic solution

Suggested implementation

We introduced the environment variable NXF_EXECUTOR_MEM_GIGA_PER_CORE which sets how many memory in GB is available per core on the supercomputer. For example, if this variable is set to 4, any nextflow process requiring 40 GB of memory will make sure that the job requests a least 10 cores. The file modules/nextflow/src/main/groovy/nextflow/executor/SlurmExecutor.groovy is modified to consider thi variable if defined.

[bentsherman]

I think this is going a bit too far trying to accommodate every variation of SLURM cluster out there

If you need something this custom, I would just create a plugin and duplicate / extend the slurm executor. nf-ospool is a recent example of this, where someone adapted the condor executor for a particular condor cluster with specific requirements

This way you can maintain your custom executor without going through us, and we don't have to bother you to test the code every time it is changed.