Tag: SLURM

[slurm-users] Questions about scontrol reconfigure / reconfig

Hello, I have some questions about adding nodes in configless mode. My version of SLURM is 21.08.5. I gave logs below to ease the read of the message. First, is “scontrol reconfigure” equal to “scontrol reconfig” ? Then, I have a strange behaviour at node addition. I have an healthy…

Continue Reading [slurm-users] Questions about scontrol reconfigure / reconfig

[slurm-users] Strange sbatch error with 21.08.2&5

Running test job with srun works: wayneh@login:~$ srun -G16 -p v100 /home/wayne.hendricks/job.sh 179851 Linux dgx1-1 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux 179851 Linux dgx1-2 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Submitting the same with sbatch does…

Continue Reading [slurm-users] Strange sbatch error with 21.08.2&5

Question: total memory usage (–mem) exceeded the RealMemory in pcluster with SLURM

Hello, I have set RealMemory=58000 (though they are 64G instances) in the file slurm_parallelcluster_queue_partition.conf When I run the following commands: sbatch -N 1 -n 1 –mem=40000 –wrap=”srun script.sh first_task” sbatch -N 1 -n 1 –mem=40000 –wrap=”srun script.sh second_task” and more They will be assigned to single instance which lead to…

Continue Reading Question: total memory usage (–mem) exceeded the RealMemory in pcluster with SLURM

Running Jobs on Titan

Running Jobs on Titan Table of Contents Titan’s Job Scheduler – SLURM Documentation Translating to SLURM commands from other workload managers Basic SLURM Commands squeue sinfo scontrol sbatch scancel Titan’s Environment Module System – LMOD Listing all available modules on Titan Loading a module into your environment Listing all modules…

Continue Reading Running Jobs on Titan

runing a command multiple times in different nodes in SLURM

I want to run three instances of GROMACS mdrun on three different nodes. I have three temperatures 200,220 and 240 K and I want to run 200 K simulation on node 1, 220 K simulation on node 2 and 240 K simulation on node 3. I need to do all…

Continue Reading runing a command multiple times in different nodes in SLURM

biopython – Github Help

1 1 0 biopython,How to rescue failed project ? To do: 1. The wrapper of the KEGG gene orthology database should obtain gene names. 2. Pandas should be replaced by other software more appropriate for data mining by counting lines in tables ( see towardsdatascience.com/surprising-sorting-tips-for-data-scientists-9c360776d7e). i User: dariusz-izak-doktorat pandas python…

Continue Reading biopython – Github Help

kegg – Github Help

1 1 0 kegg,How to rescue failed project ? To do: 1. The wrapper of the KEGG gene orthology database should obtain gene names. 2. Pandas should be replaced by other software more appropriate for data mining by counting lines in tables ( see towardsdatascience.com/surprising-sorting-tips-for-data-scientists-9c360776d7e). i User: dariusz-izak-doktorat pandas python…

Continue Reading kegg – Github Help

Installation times out – githubmate

I’m not able to get this running. The cloud formation template times out with this step seemingly the problem: SlurmManagementEC2WaitCondition | im-slurm-2-SlurmManagementEC2WaitCondition-PNDUOLXY4N9W | AWS::CloudFormation::WaitCondition | CREATE_IN_PROGRESS | Resource creation Initiated Details: Region: eu-central-1 AMI: ami-0e8286b71b81c3cc1 Slurm version: 20.02.5 It seems that the slurm binaries get created but slurm never becomes…

Continue Reading Installation times out – githubmate

Anyone know any clever snakemake/SLURM tricks to run a big analysis with limited storage?

Anyone know any clever snakemake/SLURM tricks to run a big analysis with limited storage? 1 I am using a SLURM HPC to run jobs and have ran into issues with storage. I have 3TB storage, and want to run over 1000 publicly available RNAseq data through my pipeline, which includes…

Continue Reading Anyone know any clever snakemake/SLURM tricks to run a big analysis with limited storage?

Big Slurm | FINAL FANTASY XIV, The Lodestone

PlayStation, the “PS” family logo, “PS5” and “PS4” are registered trademarks or trademarks of Sony Interactive Entertainment Inc.ESRB and the ESRB rating icon are registered trademarks of the Entertainment Software Association. MAC is a trademark of Apple Inc., registered in the U.S. and other countries.Windows is either a registered trademark…

Continue Reading Big Slurm | FINAL FANTASY XIV, The Lodestone

How does slurm work? – IT & Software development Q&A

Hi, I was relatively recently told about slurm, however, as usual, I did not have time to go into details and had to quickly write scripts and subsample them “to not waste time”. Well, since recently, constantly goes out of timeout and scripts “do not finish” as simply not enough…

Continue Reading How does slurm work? – IT & Software development Q&A

SLURM and tailoring walltime for different jobs –

Hi, so finally, I have access to a big cluster that uses SLURM as scheduler for Matlab. So far so good. Now, I would need to understand if I am planning the execution of my program properly. I have a Main file, with several batch jobs. At the moment, it…

Continue Reading SLURM and tailoring walltime for different jobs –

Install Slurm 19.05 on a standalone machine running Ubuntu 20.04

Use apt to install the necessary packages: sudo apt install -y slurm-wlm slurm-wlm-doc Load file:///usr/share/doc/slurm-wlm/html/configurator.html in a browser (or on WSL2), and: 1. Set your machine’s hostname in `SlurmctldHost` and `NodeName`. 2. Set `CPUs` as appropriate, and optionally `Sockets`, `CoresPerSocket`, and `ThreadsPerCore`. Use command `lscpu` to find what you…

Continue Reading Install Slurm 19.05 on a standalone machine running Ubuntu 20.04

docker – SLURM cluster inside k8s cannot run srun command

I’m a beginner k8s user, I’m trying to recreate this docker-compose SLURM cluster with kubernetes. First I converted the docker-compose.yaml file into k8s yaml file in order to use kubectl apply -f . to create pods and services. I’m using minikube on my computer with the none driver (like this…

Continue Reading docker – SLURM cluster inside k8s cannot run srun command

Webinar: Batch System at OSC

OSC will host a webinar on the SLURM batch system on Owens and Pitzer, on January 19, 2022, from 1:00pm to 3:00pm. We will discuss SLURM syntax and go through hands-on activities on submitting, running, and managing jobs on the clusters using SLURM batch scripts and interactive batch sessions. Who…

Continue Reading Webinar: Batch System at OSC

How to make SLURM use gres.conf

I distribute jobs using SLURM, and I have a generic resource called “cards”. In slurm.conf there is a line: GresTypes=cards I do not include this resource in the node configuration lines. Instead, I try and configure it in gres.conf: NodeName=mynode-01 Name=cards Count=2 Unfortunately, scontrol show node mynode-01 shows Gres=(null). Both…

Continue Reading How to make SLURM use gres.conf

[workshop] Building HPC clusters with LXD, Slurm & GPU:s – community-workshop

Welcome to join another exciting community workshop with Juju and related projects! This time @mmrezaie will take us through an interesting deployment of Slurm charms to build a HPC cluster with juju and lxd. Meeting link: meet.google.com/ceh-zber-jnf Date: Fri 2021-12-17 09:00 – 10:00 (UTC) Abstract In this workshop we will…

Continue Reading [workshop] Building HPC clusters with LXD, Slurm & GPU:s – community-workshop

How to get count of failed and completed jobs in an array job of slurm ( Slurm, Sbatch )

Problem : ( Scroll to solution ) I am running multiple array jobs using slurm. For a given array job id, let’s say 885881, I want to list the count of failed and completed number of jobs. Something like this: Input: <some-command> -j 885881 Output: Let’s say we have 200…

Continue Reading How to get count of failed and completed jobs in an array job of slurm ( Slurm, Sbatch )

NoMachine Forums – NoMachine and slurm created cgroups

Hello everyone, We installed NoMachine on clusters we use remotely. These clusters are handled with a scheduler called slurm. Every time a user allocate a node with slurm, its session enters a cgroup which bounds the resources the user can use. Slurm allows for example one user to use a…

Continue Reading NoMachine Forums – NoMachine and slurm created cgroups

[slurm-users] smap on Centos7

Hi, This is my first post. :-) Where can I find the ‘smap’ gui for Cenos7 ? I can’t seem to locate the package that provides this. I didn’t get any results with ‘yum whatprovides smap’. In contrast, ‘yum whatprovides sview’ gives me ‘slurm-gui’. Shouldn’t smap be part of…

Continue Reading [slurm-users] smap on Centos7

Running SortMeRNA on Multiple Files

Running SortMeRNA on Multiple Files 0 Hi all, I am VERY new to SortMeRNA (I’m a PhD student taking a bioinformatics class that has been very poorly taught). I have 27 paired samples for a total of 54 samples named like this: SRR13711719_1_val_1.fq SRR13711719_2_val_2.fq. So the format is _1_val_1.fq and…

Continue Reading Running SortMeRNA on Multiple Files

Slurm installation for multiple users – Install, Configure and Update

Welcome to the forum @bzuber.As you already suspected, jobs submitted through the web UI are counted against the quota of the system user that owns the cryoSPARC instance. Cluster resources and quota need to be allocated according to the combined needs of all cryoSPARC users. You may still capture information…

Continue Reading Slurm installation for multiple users – Install, Configure and Update

[slurm-users] A Slurm topological scheduling question

Hi David, The topology.conf file groups nodes into sets such that parallel jobs will not be scheduled by Slurm across disjoint sets. Even though the topology.conf man-page refers to network switches, it’s really about topology rather than network. You may use fake (non-existing) switch names to describe the topology. For…

Continue Reading [slurm-users] A Slurm topological scheduling question

In slurm; count number of folders in directory, encounter directory error

I have a lot of data to run through using slurm and I figured I could use a for loop sequence as it’s based on a range. The data takes a long time to generate so altering the output structure is not an option. The problem: When running my job…

Continue Reading In slurm; count number of folders in directory, encounter directory error

[slurm-users] Failed to forward X11 with a remote scheduler

Hi, I’m unsuccessful in running an X11 application with a remote SlurmctldHost. Let us call myfrontalnode the node from which the user is running the slurm commands that is different from the host SlurmctldHost. What fails is the following : ssh -X myfrontalnode srun –x11 xclock …

Continue Reading [slurm-users] Failed to forward X11 with a remote scheduler

Unable to download fastq files in parallel / SOS

Unable to download fastq files in parallel / SOS 0 Hi! Very new to all this so bear with me if I’m using incorrect terminology. Also english is my second language. I’m trying to download my fastq files in parallel but it doesn’t work and I keep receiving this error:…

Continue Reading Unable to download fastq files in parallel / SOS

Computational Scientist-Bioinformatics in Chicago, IL for University of Chicago (UC)

  UPDATE: We are in process of fixing some issues with our job board search function. If you are experiencing any problems, we apologize for the inconvenience and thank you for your patience. Job Seekers, Welcome to HERC Jobs…

Continue Reading Computational Scientist-Bioinformatics in Chicago, IL for University of Chicago (UC)

POSTDOCTORAL FELLOW IN TRANSCRIPTOMICS AND BIOINFORMATICS

We are looking for an enthusiastic and ambitious postdoctoral fellow to study the pathophysiological bases of vascular anomalies via an integrated analysis of “omics” data (incl. single cell and bulk transcriptome, methylome, genome). The postdoctoral fellow will participate in all the omic aspects of the studies, from data analysis and…

Continue Reading POSTDOCTORAL FELLOW IN TRANSCRIPTOMICS AND BIOINFORMATICS

Phylogeographic reconstruction of the marbled crayfish origin

Procambarus fallax collections and PCR genotyping Animals were collected from various wild populations (Table S1) in compliance with state and local regulations (Georgia department of natural resources scientific collection permit 115621108, state of Florida collection permits S-19-10 and S-20-04). DNA was isolated from abdominal muscle tissue using SDS-based extraction and precipitation…

Continue Reading Phylogeographic reconstruction of the marbled crayfish origin

Beagle 5.2 for imputation, outofmemory or ‘bus error’ any help?

Beagle 5.2 for imputation, outofmemory or ‘bus error’ any help? 0 Hi, I was trying to use beagle 5.2 for imputation of genotype data (N=62k). I was using chr21 with chunks of 3MB (total 12 chunks running in parallel). This tool is very fast. The problem is every time I…

Continue Reading Beagle 5.2 for imputation, outofmemory or ‘bus error’ any help?

Average Amino Acid Identity (AAI) analysis manually

Average Amino Acid Identity (AAI) analysis manually 1 Hi all, I need to perform Average Amino Acid Identity (AAI) analysis for 422 genome using the SLURM system that only allows jobs to run for 3 days. Tool like compareM can’t finish the job on time. Therefore I wish to run…

Continue Reading Average Amino Acid Identity (AAI) analysis manually

samtools server vs cluster error

Using inspiration from this thread HISAT2 output direct to bam, I’m attempting to run this command. The shell variables in this case represent paths to files/locations that make sense and in fact this command runs fine on my Ubuntu 18.04 LTS server using hisat 2.1 and samtools 1.10 (this seems…

Continue Reading samtools server vs cluster error

HPC Engineer at European Molecular Biology Laboratory (EMBL)

About the team/job We are seeking a HPC engineer to join our Compute team within our Technical Services Cluster (TSC), serving an institute of over 800 researchers and technical staff. You will be working closely with members of the department, and more widely with technical users across EMBL-EBI, playing an…

Continue Reading HPC Engineer at European Molecular Biology Laboratory (EMBL)

MarkduplicatesSpark How to speed-up ?

MarkduplicatesSpark How to speed-up ? 0 Hello all, I would like to know if there is any good option to speed up MarkduplicatesSpark ? I work with human genome with arround 900 millions reads (151 bp). I work on a cluster (with slurm). The command that i used is (with…

Continue Reading MarkduplicatesSpark How to speed-up ?

Senior Programmer Analyst/Bioinformatics in New York, NY for Columbia University

  Job Seekers, Welcome to HERC Jobs Senior Programmer Analyst/Bioinformatics Columbia University Columbia University Job Type: Officer of Administration Bargaining Unit: Regular/Temporary: Regular End Date if Temporary: Hours Per Week: 35 Salary Range: Commensurate with experience  …

Continue Reading Senior Programmer Analyst/Bioinformatics in New York, NY for Columbia University

IT System Engineer at European Molecular Biology Laboratory (EMBL)

About the team/job EMBL is Europe’s flagship research organization for the life sciences – an intergovernmental organization with more than 80 independent research groups covering the spectrum of molecular biology. EMBL is international, innovative and interdisciplinary – its approx. 2000 employees of more than 80 nationalities, operate across six sites…

Continue Reading IT System Engineer at European Molecular Biology Laboratory (EMBL)

Staff Data Scientist

  Staff Data Scientist – 93444 Organization: JG-Joint Genome Institute Lawrence Berkeley National Lab (LBL, lbl.gov/) Joint Genome Institute (JGI, biosciences.lbl.gov/area-organization/jgi/) Division has an opening for a Staff Data Scientist to join the team. In this exciting role, you will work in the JGI Division as part of the User…

Continue Reading Staff Data Scientist