Anyone know any clever snakemake/SLURM tricks to run a big analysis with limited storage?
I am using a SLURM HPC to run jobs and have ran into issues with storage. I have 3TB storage, and want to run over 1000 publicly available RNAseq data through my pipeline, which includes aligning with STAR. Obviously I must download the data in sections and run the pipeline multiple times.
Does anybody know any clever tricks to streamline this process?
Is there any way to configure a snakemake/slurm pipeline to lets say, run for 30 files, with all files expect counts being temp files, then once completed, run again for the next 30 files in a download list, and so on?
Any advice or guidance would be greatly appreciated !
• 160 views
Read more here: Source link