I am running DADA2 through QIIME and my demux.qza file is 110GB. After 24Hr I saw it consumed more than 4TB of disk space and the log file is not updating from a certain point.
Here I am curious whether it is normal or if there is any glitch and I need to restart the analysis.
Kindly suggest.
I am sharing the log file and command:
time mpirun qiime dada2 denoise-single
–i-demultiplexed-seqs coin_single-end-demux.qza –p-trim-left 0 –p-trunc-len 0 –p-n-threads 0 –o-representative-sequences rep-seqs-dada2.qza –o-table table-dada2.qza –o-denoising-stats stats-dada2.qza –verbose
log:
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/temp/qiime2-archive-6eqn0u1c/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp41b9srao/output.tsv.biom /home/test12/scr4-test1/temp/tmp41b9srao/track.tsv /home/test12/scr4-test1/temp/tmp41b9srao 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-77rgs2vk/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp21t11680/output.tsv.biom /home/test12/scr4-test1/temp/tmp21t11680/track.tsv /home/test12/scr4-test1/temp/tmp21t11680 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-aa9mg1yb/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmplb8rfki7/output.tsv.biom /home/test12/scr4-test1/temp/tmplb8rfki7/track.tsv /home/test12/scr4-test1/temp/tmplb8rfki7 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
- Filtering Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-_04iljtf/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpcf0ah59e/output.tsv.biom /home/test12/scr4-test1/temp/tmpcf0ah59e/track.tsv /home/test12/scr4-test1/temp/tmpcf0ah59e 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
- Filtering Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-7tntpq9u/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp5by7zk2j/output.tsv.biom /home/test12/scr4-test1/temp/tmp5by7zk2j/track.tsv /home/test12/scr4-test1/temp/tmp5by7zk2j 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-uzqat370/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp14bzcc08/output.tsv.biom /home/test12/scr4-test1/temp/tmp14bzcc08/track.tsv /home/test12/scr4-test1/temp/tmp14bzcc08 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-h5gbfr0u/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpv4dcttbh/output.tsv.biom /home/test12/scr4-test1/temp/tmpv4dcttbh/track.tsv /home/test12/scr4-test1/temp/tmpv4dcttbh 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-wkzb__vf/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmps0q3oqpz/output.tsv.biom /home/test12/scr4-test1/temp/tmps0q3oqpz/track.tsv /home/test12/scr4-test1/temp/tmps0q3oqpz 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-rzvzazvv/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp9gwx0asc/output.tsv.biom /home/test12/scr4-test1/temp/tmp9gwx0asc/track.tsv /home/test12/scr4-test1/temp/tmp9gwx0asc 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-uj9_hgn2/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpwvcsfo7t/output.tsv.biom /home/test12/scr4-test1/temp/tmpwvcsfo7t/track.tsv /home/test12/scr4-test1/temp/tmpwvcsfo7t 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-mvsy8es8/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpzzj2awct/output.tsv.biom /home/test12/scr4-test1/temp/tmpzzj2awct/track.tsv /home/test12/scr4-test1/temp/tmpzzj2awct 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-9tbq9434/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp21t7tboh/output.tsv.biom /home/test12/scr4-test1/temp/tmp21t7tboh/track.tsv /home/test12/scr4-test1/temp/tmp21t7tboh 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-opq32q9v/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpipict_40/output.tsv.biom /home/test12/scr4-test1/temp/tmpipict_40/track.tsv /home/test12/scr4-test1/temp/tmpipict_40 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-iknv8kse/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp0jjtcyc9/output.tsv.biom /home/test12/scr4-test1/temp/tmp0jjtcyc9/track.tsv /home/test12/scr4-test1/temp/tmp0jjtcyc9 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-5wvq4xvr/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmppcmt8xrb/output.tsv.biom /home/test12/scr4-test1/temp/tmppcmt8xrb/track.tsv /home/test12/scr4-test1/temp/tmppcmt8xrb 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-fnmu2sza/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp2gg9mye1/output.tsv.biom /home/test12/scr4-test1/temp/tmp2gg9mye1/track.tsv /home/test12/scr4-test1/temp/tmp2gg9mye1 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)Loading required package: Rcpp
Loading required package: Rcpp
R version 4.1.3 (2022-03-10)
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-plhy_j3/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpyq76ump/output.tsv.biom /home/test12/scr4-test1/temp/tmpyq76ump_/track.tsv /home/test12/scr4-test1/temp/tmpyq76ump_ 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-u52hhet6/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpwbwq3u4h/output.tsv.biom /home/test12/scr4-test1/temp/tmpwbwq3u4h/track.tsv /home/test12/scr4-test1/temp/tmpwbwq3u4h 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-6g8q03cl/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpg42c3dw0/output.tsv.biom /home/test12/scr4-test1/temp/tmpg42c3dw0/track.tsv /home/test12/scr4-test1/temp/tmpg42c3dw0 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
R version 4.1.3 (2022-03-10)
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
Loading required package: Rcpp
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-ftgam1va/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmp8ty_u96i/output.tsv.biom /home/test12/scr4-test1/temp/tmp8ty_u96i/track.tsv /home/test12/scr4-test1/temp/tmp8ty_u96i 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-csx4v9me/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmppqeq3n9a/output.tsv.biom /home/test12/scr4-test1/temp/tmppqeq3n9a/track.tsv /home/test12/scr4-test1/temp/tmppqeq3n9a 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-nh4ydv2l/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpi4zyyg7_/output.tsv.biom /home/test12/scr4-test1/temp/tmpi4zyyg7_/track.tsv /home/test12/scr4-test1/temp/tmpi4zyyg7_ 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-f_bitfru/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpht9gqygf/output.tsv.biom /home/test12/scr4-test1/temp/tmpht9gqygf/track.tsv /home/test12/scr4-test1/temp/tmpht9gqygf 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
Loading required package: Rcpp
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_single.R /home/test12/scr4-test1/temp/qiime2-archive-s2p8jb4g/dca13352-e90c-43f0-a44b-6cab29210258/data /home/test12/scr4-test1/temp/tmpewcwwqn3/output.tsv.biom /home/test12/scr4-test1/temp/tmpewcwwqn3/track.tsv /home/test12/scr4-test1/temp/tmpewcwwqn3 0 0 2.0 2 Inf independent consensus 1.0 0 1000000 NULL 16
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
R version 4.1.3 (2022-03-10)
Loading required package: Rcpp
DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering DADA2: 1.22.0 / Rcpp: 1.0.8.3 / RcppParallel: 5.1.5
-
Filtering
Read more here: Source link