Dear users,
thanks to several reports we have discovered a problem when trying to submit multi-node jobs that request more than 24 tasks per node. In general a resource request looking like this should work perfectly fine:
(...) #SBATCH --nodes=5 #SBATCH --ntasks-per-node=48 (...)
Theoretically this would allow you to make full use of 5 nodes. Currently, however, sbatch rejects such job scripts claiming that there were no hosts suitable for dispatchment. Despite this, the following request
(...) #SBATCH --ntasks=240 (...)
works as intended while being semantically equal in this scenario. We are not sure exactly what is causing this problem but do suspect a bug in slurm, possibly in conjunction with the Skylake-SP CPUs. If you are affected, we recommend to use --ntasks only for the time being. We will change the documentation accordingly so that you can build your job scripts upon correct templates. The problem has been relayed to the developers, we will have to wait for their assessment.
Please excuse any inconvenience.
Best, Sven