Anyone know any clever snakemake/SLURM tricks to run a big analysis with limited storage?

Anyone know any clever snakemake/SLURM tricks to run a big analysis with limited storage?

1

I am using a SLURM HPC to run jobs and have ran into issues with storage. I have 3TB storage, and want to run over 1000 publicly available RNAseq data through my pipeline, which includes aligning with STAR. Obviously I must download the data in sections and run the pipeline multiple times.

Does anybody know any clever tricks to streamline this process?

Is there any way to configure a snakemake/slurm pipeline to lets say, run for 30 files, with all files expect counts being temp files, then once completed, run again for the next 30 files in a download list, and so on?

Any advice or guidance would be greatly appreciated !


SLURM


genomics


RNAseq


Snakemake

• 160 views

Read more here: Source link