The job always hangs at one particular step in the Slurm system.

[xyz202310]

Description of the bug

Hello!
Thanks for your great work! I am running scnanoseq on an HPC (slurm system), but the job always hangs at one particular step. The last screen output looks like this (The complete output is provided in the attachment.

out.txt

execution_trace_2025-09-24_15-11-55.txt

):

executor > local (71)
[- ] NFC…ANOSEQ:SCNANOSEQ:CAT_FASTQ -
[dc/6b0124] NFC…_TRIM:NANOPLOT (mou_brain) | 0 of 1
[6c/27b59f] NFC…TRIM:TOULLIGQC (mou_brain) | 0 of 1
[d1/71ff35] NFC…RE_TRIM:FASTQC (mou_brain) | 0 of 1
[- ] NFC…Q:SCNANOSEQ:NANOCOMP_FASTQ | 0 of 1
[3a/250e2f] NFC…E_FAIDX (GRCm39.genome.fa) | 1 of 1 ✔
[75/9eed5d] NFC…ncode.vM37.annotation.gtf) | 1 of 1 ✔
[4b/dfee70] NFC….vM37.annotation.genepred) | 1 of 1 ✔
[dc/13a128] NFC…SCNANOSEQ:GUNZIP_FASTQ (1) | 1 of 1 ✔
[9a/31f3ff] NFC…NOSEQ:NANOFILT (mou_brain) | 0 of 1
[- ] NFC…ANOPLOT_POST_TRIM:NANOPLOT -
[- ] NFC…NOPLOT_POST_TRIM:TOULLIGQC -
[- ] NFC…_NANOPLOT_POST_TRIM:FASTQC -
[- ] NFC…_SCNANOSEQ:SCNANOSEQ:BLAZE -
[- ] NFC…SCNANOSEQ:PREEXTRACT_FASTQ -
[- ] NFC…SCNANOSEQ:CORRECT_BARCODES -
[- ] NFC…PLOT_POST_EXTRACT:NANOPLOT -
[- ] NFC…LOT_POST_EXTRACT:TOULLIGQC -
[- ] NFC…NOPLOT_POST_EXTRACT:FASTQC -
[- ] NFC…OSEQ:SCNANOSEQ:READ_COUNTS -
[e5/4a982a] NFC…SCRNA_ISOQUANT:SPLIT_FASTA | 1 of 1 ✔
[0d/776473] NFC…AIDX_SPLIT (chrX.split.fa) | 61 of 61 ✔
[2a/bcef09] NFC…Y_SCRNA_ISOQUANT:SPLIT_GTF | 1 of 1 ✔
Plus 42 more processes waiting for tasks…

And the final lines of the execution_trace file are as follows:

69 0d/776473 1429737 NFCORE_SCNANOSEQ:SCNANOSEQ:PROCESS_LONGREAD_SCRNA_GENOME:QUANTIFY_SCRNA_ISOQUANT:SAMTOOLS_FAIDX_SPLIT (chrX.split.fa) COMPLETED 0 2025-09-24 15:14:16.739 812ms 1s 88.1% 3.9 MB 10.2 MB 164.5 MB 5 KB
4 75/9eed5d 1414439 NFCORE_SCNANOSEQ:SCNANOSEQ:UCSC_GTFTOGENEPRED (gencode.vM37.annotation.gtf) COMPLETED 0 2025-09-24 15:14:03.971 13.8s 13s 99.3% 513.8 MB 524.9 MB 1 GB 41.4 MB
71 4b/dfee70 1430392 NFCORE_SCNANOSEQ:SCNANOSEQ:UCSC_GENEPREDTOBED (gencode.vM37.annotation.genepred) COMPLETED 0 2025-09-24 15:14:18.766 848ms 0ms 91.9% 4.9 MB 18.7 MB 41.4 MB 30.1 MB
5 dc/13a128 1414454 NFCORE_SCNANOSEQ:SCNANOSEQ:GUNZIP_FASTQ (1) COMPLETED 0 2025-09-24 15:14:03.985 10m 13s 10m 13s 121.7% 5.3 MB 31.9 MB 57.7 GB 127.3 GB

At first I suspected insufficient memory, so I increased the memory allocation, but the result was the same—the job remains stuck at this step. How can I solve this? Thank you very much!

Command used and terminal output

#!/bin/bash
#SBATCH --job-name=scnano
#SBATCH --partition=cpu
#SBATCH -n 30 (8 GB of memory allocated per core)
#SBATCH --ntasks-per-node=30
#SBATCH --output=%j.out
#SBATCH --error=%j.err

module load miniconda3
source activate nf
export NXF_SINGULARITY_CACHEDIR=/dssg/home/sy/scnano/sing

cd /dssg/home/sy/scnano/data
nextflow run nf-core/scnanoseq \
  --input ./samplesheet.csv \
  --outdir ./results_new \
  --genome_fasta /dssg/home/sy/GRCm39.genome.fa \
  --gtf /dssg/home/sy/gencode.vM37.annotation.gtf \
  --quantifier "isoquant" \
  --barcode_format 10X_3v3 \
  -profile singularity

Relevant files

out.txt

execution_trace_2025-09-24_15-11-55.txt

System information

nextflow 25.04.7
nf-core 3.4.0.dev0
nf-core/scnanoseq 1.2.1
HPC Centos Linux

[atrull314]

Thanks for your interest in scnanoseq!

It looks like the pipeline is stuck on Nanofilt. Nanofilt is not able to be multi-threaded and can take a very long time as a result, we have implemented a way to get around this via the split_amount parameter (we have notes about this in the README in the pipeline here and I recommend reading through the first bullet point before enabling this as there are some caveats when using the parameter). Can you try rerunning the pipeline with that parameter?

I'll also note we are in the process of reviewing PRs that swap out Nanofilt for Chopper, which is multi-threaded so shouldn't have the same issues.

Thanks!

[xyz202310]

Hello!
Thank you for your quick reply! After adding the parameter --split_amount 500000, the program no longer gets stuck at the Nanofilt step. However, it has now been lingering on a later step (apparently BLAZE) for almost 20 hours. The last line of output is as follows:

executor > local (823)
[- ] NFC…ANOSEQ:SCNANOSEQ:CAT_FASTQ -
[db/1c56d9] NFC…_TRIM:NANOPLOT (mou_brain) | 1 of 1 ✔
[cb/93a5ad] NFC…TRIM:TOULLIGQC (mou_brain) | 1 of 1 ✔
[a0/37173a] NFC…RE_TRIM:FASTQC (mou_brain) | 1 of 1 ✔
[69/02bdfe] NFC…Q:SCNANOSEQ:NANOCOMP_FASTQ | 1 of 1 ✔
[52/dff326] NFC…E_FAIDX (GRCm39.genome.fa) | 1 of 1 ✔
[24/1e0d1b] NFC…ncode.vM37.annotation.gtf) | 1 of 1 ✔
[bb/1d0aca] NFC….vM37.annotation.genepred) | 1 of 1 ✔
[a8/645f7e] NFC…SCNANOSEQ:GUNZIP_FASTQ (1) | 1 of 1 ✔
[ed/544f02] NFC…SEQ:SPLIT_FILE (mou_brain) | 1 of 1 ✔
[82/b9c718] NFC…NOSEQ:NANOFILT (mou_brain) | 746 of 746 ✔
[5c/0f8c78] NFC…ANOSEQ:CAT_CAT (mou_brain) | 1 of 1 ✔
[60/b98ba0] NFC…_TRIM:NANOPLOT (mou_brain) | 1 of 1 ✔
[a0/530795] NFC…TRIM:TOULLIGQC (mou_brain) | 1 of 1 ✔
[72/6637b5] NFC…ST_TRIM:FASTQC (mou_brain) | 1 of 1 ✔
[- ] NFC…_SCNANOSEQ:SCNANOSEQ:BLAZE | 0 of 1
[e2/445a37] NFC…_FILE_BC_FASTQ (mou_brain) | 1 of 1 ✔
[- ] NFC…CNANOSEQ:SPLIT_FILE_BC_CSV -
[- ] NFC…SCNANOSEQ:PREEXTRACT_FASTQ -
[- ] NFC…SCNANOSEQ:CORRECT_BARCODES -
[d7/c9dbd4] NFC…SCRNA_ISOQUANT:SPLIT_FASTA | 1 of 1 ✔
[95/70ff40] NFC…AIDX_SPLIT (chrX.split.fa) | 61 of 61 ✔
[6b/c59746] NFC…Y_SCRNA_ISOQUANT:SPLIT_GTF | 1 of 1 ✔
Plus 49 more processes waiting for tasks…

Is it normal for this step to run for so long? How can I resolve this? Thank you!

[lianov]

@xyz202310 , glad that enabling the splitting param helped you. Regarding BLAZE, did you increase the amount of cpus ?

We show how to do this via custom config file in https://nf-co.re/scnanoseq/1.2.1/#troubleshooting (for BLAZE and a few other steps. I definitely recommend increasing is for ISOQUANT as well - this is usually the bottleneck step). Could you do this and let us know how it goes?

[xyz202310]

Hi, Thank you for your reply! I noticed this point in Troubleshooting, so I also added the parameter -c ./proc.config to my command:

#!/bin/bash
#SBATCH --job-name=scnano
#SBATCH --partition=cpu
#SBATCH -n 30 (8 GB of memory allocated per core)
#SBATCH --ntasks-per-node=30
#SBATCH --output=%j.out
#SBATCH --error=%j.err

module load miniconda3
source activate nf
export NXF_SINGULARITY_CACHEDIR=/dssg/home/sy/scnano/sing

cd /dssg/home/sy/scnano/data
nextflow run nf-core/scnanoseq
--input ./samplesheet.csv
--outdir ./results_new
--genome_fasta /dssg/home/sy/GRCm39.genome.fa
--gtf /dssg/home/sy/gencode.vM37.annotation.gtf
--quantifier "isoquant"
--barcode_format 10X_3v3
-profile singularity
--split_amount 500000
-c ./proc.config

The content of proc.config is as follows:

process
{
withName: '.*:.FASTQC.'
{
cpus = 20
}
}

process
{
withName: '.*:BLAZE'
{
cpus = 30
}
}

process
{
withName: '.*:TAG_BARCODES'
{
memory = '60.GB'
}
}

process
{
withName: '.*:SAMTOOLS_SORT'
{
cpus = 20
}
}

process
{
withName: '.*:MINIMAP2_ALIGN'
{
cpus = 20
}
}

process
{
withName: '.*:ISOQUANT'
{
cpus = 30
memory = '85.GB'
}
}

The complete output file with -c ./proc.config added is attached.

outfile.txt

execution_trace_2025-09-25_17-17-15.txt

[lianov]

Thanks - one question and a suggestion:

1. What read depth approximately, is your sample(s)? That would give us an idea since we certainly have not experienced BLAZE taking this long. If you read depth is similar to what we have tested our runs on, it points to a miss-configuration of resources somewhere.

2. Suggestion: Currently you are running the pipeline under local mode. Since you are submitting this to a cluster, I highly advice to use Nextflow's executor mode to enable the jobs to be further parallelized. The easiest way to pull this into your run would be if your institution already has an Institutional profile, which you can check at: https://nf-co.re/configs . If your institution does not, you could write one following the examples and add this configuration to your existing proc.config

3. I also suggest to go ahead and add --skip_bam_nanocomp since this is one process we noted can also take (we will likely deprecate this one in future version since additional / complementary QCs have been added - so you don't miss out on much by skipping nanocomp)

@atrull314 : any other thoughts or question to aid in this troubleshooting issue?

[atrull314]

I think the only other thing I can add is IsoQuant is also going to be a point where there's a major time sink. So if you don't need the gene-level matrices, I would definitely recommend only using the oarfish quantifier instead of both or just isoquant to help speed the pipeline up as well.

[lianov]

@xyz202310 : just wondering if you tried any of our suggestions above to see if it resolved your issue. Let us know. Thanks!

[xyz202310]

Hi @lianov ,
So sorry for the late response. The ONT data I used were generated with the SQK-LSK114 kit and sequenced on the PromethION platform. After adding the --skip_bam_nanocomp parameter, the previous issue persists.

[atrull314]

Hi @xyz202310,

Just wanted to check if you were still using the local executor mode or if you had an updated command using an institutional profile? That would definitely speed up the pipeline to process the data.

Thanks,
Austyn