Dada2 denoise-paired running time over 24hrs and taking more than 4TB temporary space on HPC – Technical Support

I am running DADA2 through QIIME and my demux.qza file is 110GB. After 24Hr I saw it consumed more than 4TB of disk space and the log file is not updating from a certain point.
Here I am curious whether it is normal or if there is any glitch and I need to restart the analysis.
Kindly suggest.
I am sharing the log file and command:

time mpirun qiime dada2 denoise-single
–i-demultiplexed-seqs coin_single-end-demux.qza –p-trim-left 0 –p-trunc-len 0 –p-n-threads 0 –o-representative-sequences rep-seqs-dada2.qza –o-table table-dada2.qza –o-denoising-stats stats-dada2.qza –verbose

log:
Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/temp/qiime2-archive-6eqn0u1c/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp41b9srao/output.tsv.biom /home/test12/scr4-test1/temp/tmp41b9srao/track.tsv /home/test12/scr4-test1/temp/tmp41b9srao 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-77rgs2vk/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp21t11680/output.tsv.biom /home/test12/scr4-test1/temp/tmp21t11680/track.tsv /home/test12/scr4-test1/temp/tmp21t11680 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  1. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  2. Filtering Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-aa9mg1yb/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmplb8rfki7/output.tsv.biom /home/test12/scr4-test1/temp/tmplb8rfki7/track.tsv /home/test12/scr4-test1/temp/tmplb8rfki7 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  1. Filtering Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-_04iljtf/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpcf0ah59e/output.tsv.biom /home/test12/scr4-test1/temp/tmpcf0ah59e/track.tsv /home/test12/scr4-test1/temp/tmpcf0ah59e 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  1. Filtering Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-7tntpq9u/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp5by7zk2j/output.tsv.biom /home/test12/scr4-test1/temp/tmp5by7zk2j/track.tsv /home/test12/scr4-test1/temp/tmp5by7zk2j 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-uzqat370/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp14bzcc08/output.tsv.biom /home/test12/scr4-test1/temp/tmp14bzcc08/track.tsv /home/test12/scr4-test1/temp/tmp14bzcc08 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-h5gbfr0u/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpv4dcttbh/output.tsv.biom /home/test12/scr4-test1/temp/tmpv4dcttbh/track.tsv /home/test12/scr4-test1/temp/tmpv4dcttbh 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-wkzb__vf/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmps0q3oqpz/output.tsv.biom /home/test12/scr4-test1/temp/tmps0q3oqpz/track.tsv /home/test12/scr4-test1/temp/tmps0q3oqpz 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-rzvzazvv/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp9gwx0asc/output.tsv.biom /home/test12/scr4-test1/temp/tmp9gwx0asc/track.tsv /home/test12/scr4-test1/temp/tmp9gwx0asc 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-uj9_hgn2/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpwvcsfo7t/output.tsv.biom /home/test12/scr4-test1/temp/tmpwvcsfo7t/track.tsv /home/test12/scr4-test1/temp/tmpwvcsfo7t 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-mvsy8es8/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpzzj2awct/output.tsv.biom /home/test12/scr4-test1/temp/tmpzzj2awct/track.tsv /home/test12/scr4-test1/temp/tmpzzj2awct 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-9tbq9434/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp21t7tboh/output.tsv.biom /home/test12/scr4-test1/temp/tmp21t7tboh/track.tsv /home/test12/scr4-test1/temp/tmp21t7tboh 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-opq32q9v/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpipict_40/output.tsv.biom /home/test12/scr4-test1/temp/tmpipict_40/track.tsv /home/test12/scr4-test1/temp/tmpipict_40 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-iknv8kse/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp0jjtcyc9/output.tsv.biom /home/test12/scr4-test1/temp/tmp0jjtcyc9/track.tsv /home/test12/scr4-test1/temp/tmp0jjtcyc9 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-5wvq4xvr/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmppcmt8xrb/output.tsv.biom /home/test12/scr4-test1/temp/tmppcmt8xrb/track.tsv /home/test12/scr4-test1/temp/tmppcmt8xrb 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-fnmu2sza/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp2gg9mye1/output.tsv.biom /home/test12/scr4-test1/temp/tmp2gg9mye1/track.tsv /home/test12/scr4-test1/temp/tmp2gg9mye1 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)Loading required package: Rcpp

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-plhy_j3/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpyq76ump/output.tsv.biom /home/test12/scr4-test1/temp/tmpyq76ump_/track.tsv /home/test12/scr4-test1/temp/tmpyq76ump_ 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-u52hhet6/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpwbwq3u4h/output.tsv.biom /home/test12/scr4-test1/temp/tmpwbwq3u4h/track.tsv /home/test12/scr4-test1/temp/tmpwbwq3u4h 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-6g8q03cl/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpg42c3dw0/output.tsv.biom /home/test12/scr4-test1/temp/tmpg42c3dw0/track.tsv /home/test12/scr4-test1/temp/tmpg42c3dw0 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

R version 4.1.3 (2022-03-10)

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

Loading required package: Rcpp

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-ftgam1va/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp8ty_u96i/output.tsv.biom /home/test12/scr4-test1/temp/tmp8ty_u96i/track.tsv /home/test12/scr4-test1/temp/tmp8ty_u96i 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-csx4v9me/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmppqeq3n9a/output.tsv.biom /home/test12/scr4-test1/temp/tmppqeq3n9a/track.tsv /home/test12/scr4-test1/temp/tmppqeq3n9a 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-nh4ydv2l/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpi4zyyg7_/output.tsv.biom /home/test12/scr4-test1/temp/tmpi4zyyg7_/track.tsv /home/test12/scr4-test1/temp/tmpi4zyyg7_ 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-f_bitfru/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpht9gqygf/output.tsv.biom /home/test12/scr4-test1/temp/tmpht9gqygf/track.tsv /home/test12/scr4-test1/temp/tmpht9gqygf 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

Loading required package: Rcpp

Running external command line application(s). This may print messages to stdout and/or stderr.

The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-s2p8jb4g/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpewcwwqn3/output.tsv.biom /home/test12/scr4-test1/temp/tmpewcwwqn3/track.tsv /home/test12/scr4-test1/temp/tmpewcwwqn3 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

R version 4.1.3 (2022-03-10)

Loading required package: Rcpp

DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  1. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  2. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  3. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  4. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  5. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  6. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  7. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  8. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  9. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  10. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  11. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  12. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  13. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  14. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  15. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  16. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  17. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  18. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  19. Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5

  20. Filtering

Read more here: Source link