- what is the point of using n-jobs (and joblib) if the the library uses all cores anyway?
It does not, if you specify n_jobs to -1, it will use all cores. If it is set to 1 or 2, it will use one or two cores only (test done scikit-learn 0.20.3 under Linux).
The documentation says:
This parameter is used to specify how many concurrent processes or
threads should be used for routines that are parallelized with joblib.
n_jobs is an integer, specifying the maximum number of concurrently
running workers. If 1 is given, no joblib parallelism is used at all,
which is useful for debugging. If set to -1, all CPUs are used. For
n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For example with
n_jobs=-2, all CPUs but one are used.
n_jobs is None by default, which means unset; it will generally be
interpreted as n_jobs=1, unless the current joblib.Parallel backend
context specifies otherwise.
For more details on the use of joblib and its interactions with
scikit-learn, please refer to our parallelism notes.
You should either use
joblib, don’t use both simultaneously.
Read more here: Source link