- claix18-slurm-pilot - lists.rwth-aachen.de

#BSUB -P abcd4321
by simon＠isf.rwth-aachen.de 24 Apr '19

24 Apr '19

Hi all, Is there an equivalent of the #BSUB -p command in SLURM as well? Best Wishes, Marek

4 3

HPCWORK files unavailable
by Wickel, Noah Frederic 23 Apr '19

23 Apr '19

Dear Admins and CLAIX-18 users, as of today I found, from both login18-1 and login18-2 nodes, the /hpcwork/ directory to be completely empty. That is, i don't see my own folder nor any other users folders. The terminal tells me that $HPCWORK is not a directory. Is this a temporary incident, or did I miss an important announcement? Has anyone else encountered this? Best, Noah

3 3

$CFXHOSTS: unable to find master host
by simon＠isf.rwth-aachen.de 18 Apr '19

18 Apr '19

Dear all, i attach my jobsubmission file with the error: :::::::::::::: slurm_ansys.sh :::::::::::::: #!/usr/local_rwth/bin/zsh ### Job name #SBATCH -J CFX_OpenMPI ### File / path where output will be written, the %J is the job id #SBATCH -o CFX_OpenMPI.%J ### Request the time you need for execution in minutes ### The format for the parameter is: [hour:]minute, ### that means for 80 minutes you could also use this: 1:20 #SBATCH -t 24:00:00 ### Request virtual memory you need for your job in MB #SBATCH --mem-per-cpu 2G #SBATCH --nodes=1 ### Request the number of compute slots you want to use #SBATCH --cpus-per-task=8 ### Enable Verbose Output #SBATCH -V #SBATCH --mail-type=end #SBATCH --mail-user=sim(a)isf.rwth-aachen.de #SBATCH --account=rwth0398 ### Change to the work directory cd $WORK/Droplet CFX/ more slurm_ansys.sh ### Purge default MPI and Compiler to use the CFX MPI implementation module purge ### load modules and execute module load TECHNICS module load ansys ### start non-interactive batch job cfx5solve -def droplets_scaled2_sim.def -par-dist "$CFXHOSTS" -start-method "Intel MPI Distributed Parallel" -ccl R.ccl ### (OK) Unloading intelmpi 2018.4.274 (!!) neither LSF nor SLURM environment present, environment broken? (OK) Unloading Intel Suite 19.0.1.144 (WW) ATTENTION!!! Unloading this module can result in inoperable software!!! (OK) Loading TECHNICS environment (!!) Please notice: Using ansys requires payment. (!!) If in doubt, please contact your institute's IT-administrator or servicedesk(a)itc.rwth-aachen.de. (!!) Run the workbench GUI using runwb2 (OK) Loading ansys 19.3 Warning! The parallel start method specified in the input file: IBM MPI Local Parallel, is not a default parallel start method for this platform and has therefore been preserved. A different start method may be specified using the -start-method option. An error has occurred in cfx5solve: Unable to find the master host ncm0067.hpc.itc.rwthaachen.de (ncm0067.hpc.itc.rwth-aachen.de) in the host list: at least one partition must be assigned to the master host. I dont understand why im receiving this error. Did somehthing change in $CFXHOSTS?? Best Regards, Marek

2 4

/hpcwork unstable
by Johannes Sauer 12 Apr '19

12 Apr '19

Dear all, I keep having problems with (input) files on /hpcwork. I'm submitting my jobs on login18-2.hpc.itc.rwth-aachen.de Sometimes its not possible to ls a directory which has files linked to hpcwork, the system just hangs. I also had jobs not starting or failing between different steps due to files that could not be found. Is this an issue with hpcwork, should I move my files to work or my home directory? Best Johannes -- M.Sc. Johannes Sauer Researcher Institut fuer Nachrichtentechnik RWTH Aachen University Melatener Str. 23 52074 Aachen Tel +49 241 80-27678 Fax +49 241 80-22196 sauer(a)ient.rwth-aachen.de http://www.ient.rwth-aachen.de

2 1

Problem with GUI on login18-x-1, login18-x-2
by simon＠isf.rwth-aachen.de 11 Apr '19

11 Apr '19

Dear all, I have noticed a problem with my GUI applications (Ansys CFX, cfx5pre, etc). Following protocol i wanted to use these applications via the "GUI"-dialogue nodes login18-x-1, login18-x-2. The application started but the part of the GUI which displays the 3D mesh, the 3D geometry or the 3D domain set up could not be displayed (it gave a glitch, showing only the fixed screen "behind" the GUI, when the GUI was loaded). I am not very much bothered by this, since it works perfectly fine on login18-1, but i feel like im "abusing" the system, so i wanted to report this issue. I case more information is required, because for the isse to show an actual case needs to be loaded in Ansys CFX, i can attach screenshots the next time. Best Regards, Marek Simon

1 0

unable to use project-based compute cluster
by Zhijian Li 09 Apr '19

09 Apr '19

Dear all, I applied a project, called rwth0233, and would like to use it on the new system. When I set -A=rwth0233, and submit the jobs, I got below error information: >> sbatch: error: [E] Given account rwth0233 not found in config. >> sbatch: error: Batch job submission failed: Unspecified error Previously, I could use it on Bull Cluster by setting -P rwth0233. So how can I use this computation resource now? or the project-based quota has not been transferred to the new system. Thanks, Zhijian *______________________________* Zhijian Li Institute for Computational Genomics RWTH Aachen University Pauwelsstrasse 19 52074 Aachen, Germany

2 3

CUDA on non-gpu machines
by Wilfried Michel 01 Apr '19

01 Apr '19

Hi all, I am using some piece of software that internally uses tensorflow to do some model training and evaluation. In the training phase I'd like to profit from the benefits of GPGPU and am therefore using a gpu version of tensorflow. In the inference phase, the gpu would be heavily underutilized and an evaluation of the models on cpu is totally acceptable. The gpu version of tensorflow in that case detects automatically that no gpu is present and runs all computation on cpu. It is however still linked against the cuda libraries and crashes if they are not present. My problem now is, that on the non-gpu part of the Claix18 cluster no cuda libraries seem to be available. steps to reproduce: $ ssh login18-1 ls /usr/local_rwth/sw/cuda/8.0.44 #No such file or directory $ ssh login18-g-1 ls /usr/local_rwth/sw/cuda/8.0.44 #Directory contents $ ssh login ls /usr/local_rwth/sw/cuda/8.0.44 #Directory contents $ sbatch --wrap="ls /usr/local_rwth/sw/cuda/8.0.44" #No such file or directory $ sbatch --gres=gpu:1 --wrap="ls /usr/local_rwth/sw/cuda/8.0.44"#Directory contents (I have taken this path from the environment variable LD_LIBRARY_PATH after loading the module cuda/80 on login18-g-1) I realize that there is the possibility to compile my software twice, one version using tensorflow-gpu and one using tensorflow(-cpu). But I think that having cuda module available also on cpu nodes is also usefull for other settings (e.g. compiling gpu-software without the need to block a gpu slot). So I would like to ask if cuda modules on cpu-nodes have been forgotten, or if this was a deliberate design decision. In the latter case I'd like to open the discussion to change this decision. Or am I missing some crucial point in the module system? Thank you very much and best regards Wilfried Michel

2 1

Bond to cores required?
by Julien Guénolé 29 Mar '19

29 Mar '19

Hi all, is the binding to cores automatic with SLURM? The command --cpu-bind=cores is not recognized. I tried to force the binding by using #SBATCH -B 2:24:1 but it says that there is no such nodes available. However #SBATCH -B 2:12:1 works unexpectedly fine! Thanks! Bests, Julien -- ************************************************** Dr. Julien GUÉNOLÉ Research Group Head ---------------------- Institute of Physical Metallurgy and Metal Physics RWTH Aachen University Kopernikusstrasse 14 52074 Aachen, GERMANY ---------------------- Room 202 Phone [office] +49 241 80 26866 Email guenole(a)imm.rwth-aachen.de Web http://www.julien-guenole.fr Twitter @nanouayeur **************************************************

2 2

pam_slurm_adopt
by Marcus Wagner 29 Mar '19

29 Mar '19

Dear users, we have reached another, important milestone. We implemented pam_slurm_adopt, which means, that you can ssh to a computenode, where your job is running. This does also mean, that you can ssh from one compute node in the jobs to another compute node in the job. * CFX users, please test multinode jobs again, they should be functional now. * starccm+ users, please test your jobs again, they should be functional now. * This milestone is also a prerequisite for full X11-Forwarding ** You can do something like that already now. Submit a job with e.g. "sleep 100000" as command. Do a "ssh -Y <nodename>". Then you can start there the GUI, might it be xterm or whatever else. ** Full X11-Forwarding would mean something like sbatch --x11 xterm (in short) Kind regards Marcus -- Marcus Wagner, Dipl.-Inf. IT Center Abteilung: Systeme und Betrieb RWTH Aachen University Seffenter Weg 23 52074 Aachen Tel: +49 241 80-24383 Fax: +49 241 80-624383 wagner(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Semi-repeatable low CPU usage
by Gereon Kremer 14 Mar '19

14 Mar '19

Hi, (as of now) we mainly use the system for benchmarking, that is we measure how long it takes a solver to run on a particular input. I'm not so much interested in the result of the solver, but rather want to know the runtime. I honestly don't really care about a difference of a second or whether we use wall-clock or CPU time. After our solver has loaded the input file (of only a few KB) there is no further IO and the whole process is entirely CPU bound -- so we assume 100% CPU load and just assume that CPU time and wall-clock time are essentially the same. All tasks are run with a timeout (here: two minutes + 3 seconds grace time accounting for CPU vs. wall clock etc. measured with date) and a memout (here: 8GB) The corresponding part of the script looks like this (with $cmd being the command that is run): start=`date +"%s%3N"` ulimit -c 0 && ulimit -S -v 8388608 && ulimit -S -t 123 && time $cmd end=`date +"%s%3N"` echo "time: $(( end - start ))" I however observer that from time to time a task takes way longer than it should, i.e. the time that is output is way beyond 120. I currently have an example with above 5 minutes and have already seen instances with almost 10 minutes. About every second run or so (one run being an array job with 1000 individual jobs running 12 tasks each) I hit a case where one individual task takes way longer. The time output would then look like this: 122.24s user 0.53s system 40% cpu 5:05.67 total Unfortunately I cannot really reproduce it: it happens with seemingly random inputs and only once or twice on a run. It however happens rather consistently every second run or so. Running this particular input (on the login node) is just fine with 100% CPU load and stopped by ulimit after 123 seconds. As I run multiple tasks within one array job this also leads to this array job being canceled (as I compute the overall time limit from the timeouts and assume that every task actually finishes within its timeout), for example: slurmstepd: error: *** JOB 491231 ON nihm017 CANCELLED AT 2019-03-11T23:57:48 DUE TO TIME LIMIT *** (The issue happened about 6-8 minutes earlier than this message) Can you trace this to something happening on the nodes? Or do I simply have to rerun stuff until it did not happen anymore? Best, Gereon -- Gereon Kremer Lehr- und Forschungsgebiet Theorie Hybrider Systeme RWTH Aachen Tel: +49 241 80 21243

2 3