Skip to content

Running SOLPS-ITER

This will guide you through a successful run of SOLPS-ITER. It is assumed that you have completed all the steps in Creating a new SOLPS-ITER run.

It is assumed that you have loaded the SOLPS-ITER environment.

See also Questions and Answers, section Running, restarting and transferring SOLPS-ITER runs.

Running the simulation using b2run

The simplest way to run SOLPS-ITER is using the b2run command in the run/ folder: 

b2run b2mn >& run.log &
- The b2run b2mn part is essential; this is what runs SOLPS-ITER. - The >& run.log part redirects both STDOUT and STDERR to a log file named run.log; the extra & suppresses EIRENE pre-processing printout. - The final & lets the simulation run in the background so that you can use the command line for other things in the meantime.

In this form, this command will persist in running your simulation even after you have closed the command line, lost your internet connection or shut down your computer. This applies, however, only when you're running SOLPS-ITER on an external server (like Soroban)!

You can submit several simulations at once using b2run in this form. Beware, however, that submitting two runs at once might not be twice as efficient. In this bare form, unprotected by submission scripts (see the next section), each b2run will hog all the computational power it can to itself. Consequently, simultaneous runs will compete. They won't get in each others' way and crash, but the cpu time and elapsed time, which are usually very close to each other in an isolated run, will be different. For instance, elapsed time will be 1 hour on both runs but cpu time will be only 40 minutes. Correspondingly, the number of iterations SOLPS-ITER has managed to do in the appointed time will be lower than if the simulation was running alone.

You will find all information about your run (whether it ended successfully, how many iterations were performed, type of errors etc.) in the run.log file.

Running the simulation using submission scripts

A tidier alternative to the b2run command is submitting the job to a resource management system using a submission script. In the context of SOLPS-ITER, these are traditionally called (cluster_name)submit. For instance there's itmsubmit for Marconi Gateway and sorobansubmit for Soroban.

The submission scripts are essentially a wrapper around b2run. They are run in the run/ folder. Many simulations can be run simultaneously from different folders. Based on whether the input.dat file is present, the command will perform either a coupled B2.5-EIRENE run or a B2.5 standalone run.

itmsubmit

On the Marconi Gateway cluster, SLURM is used to perform resource management. To submit a simulation, initiate the SOLPS-ITER work environment, go to the run/ folder of the simulation you'd like to run and use the following command: 

itmsubmit -j my_first_job -T 1
Thus you submit a job named my_first_job that will run at most for 1 hour. Use 
itmsubmit  -z -m "-np 24"
to submit a run where EIRENE is parallelised with MPI and running on 24 cores. (EIRENE calls take up approximately 85 % of CPU time in my pure D COMPASS simulations, so the parallelisation speeds the whole thing up.) Check out 
itmsubmit -h
for more flags.

Furthermore, there is the "loop" option, which I have not figured out how to use yet, but apparently it breaks up the simulation into many smaller parts (16 hours of CPU time or so) which are submitted to the cluster as individual runs. Apparently this way the cluster computation power is used more efficiently. Plus, you don't have to wait in the queue because you requested 3 weeks of computation time and SLURM thinks you're greedy.

To check which jobs are running: 

squeue
To list only your own jobs: 
squeue -u username
When your run is done, you will receive a notification e-mail to the address which is written when the script starts. If you don't, check your spam settings.

The commands which are executed by the itmsubmit script are listed in the runs/ folder.

Finally, there isn't that much difference between running SOLPS-ITER manually in the command line and submitting it as a job - except that submitted job will take care of itself and you can just periodically check in on it. See also Questions and answers: itmsubmit and file updating.

sorobansubmit

Soroban is one of the IPP Prague servers designed for user simulations. (More info on Soroban can be found at the COMPASS wiki and in the Soroban section of Remote access).) Its resource management system is called qsub. The sorobansubmit command was written by Aleš Podolník and it must be copied into the SOLPS-ITER scripts folder during the installation (see Installing SOLPS-ITER at COMPASS Soroban). Note that sorobansubmit only works in a user's private installation, not in the central installation in /scratch/solps/solps-iter! (It's a bug.) To submit a job:

  1. Log in to the Soroban front, which acts as a gateway to the Soroban nodes.

    ssh -X username@soroban

    (You can also write soroban.tok.ipp.cas.cz if you aren't in the internal network of IPP Prague.)

  2. Go to your SOLPS-ITER installation directory and initiate the SOLPS-ITER work environment.

    cd /net/soroban-node-06/scratch/jirakova/solps/solps-iter

tcsh
source setup.csh
setenv DEVICE compass

  3. Go to the run folder of the simulation you wish to run.

    cd compass-16908/run

  4. Use sorobansubmit:

    sorobansubmit -m "-np 8" -j "my_job_with_no_blank_spaces" -H "soroban-node-06" -T "1"

  5. -m specifies the MPI (Message Passing Interface) execution options. "-np 8" will instruct the script to use 4 processors in parallel. Don't use "-n 8", it will break the script. If -m is not set, the code will run on 1 thread.

  6. -j is followed by the job name. This parameter is mandatory. If it isn't set, a default name will be used and the queue will be a mess. The name should contain alphanumeric characters only, meaning the underscore character is fine but a blank space will result in the error set: Variable name must begin with a letter..
  7. -H specifies the Soroban node which should carry out the job. If nothing is selected, the resource management system will choose on its own.s
  8. -T specifies the approximate number of hours the simulation will take, so that the resource management system knows what priority to assign it. Always leave extra time compared to the planned simulation run time, so that the computation doesn't stop before b2fstate is written.

The source code of sorobansubmit is located in $SOLPSTOP/scripts/ (where you have copied it during the installation). It uses the qsub command to submit a job of a given size, so you need not initiate the interactive qsub -IX environment prior to sorobansubmit.

By default, sorobansubmit submits the jobs into the queue long, which has the lowest priority. To change this behaviour, open the file $SOLPSTOP/runs/QSUP.soroban_coupled and replace the line 

#PBS -q long
with 
#PBS -q medium

or another queue visible at soroban:44444 in your browser (more info below). Only use the medium queue for short simulations (Aleš Podolník says 2 days at maximum).

You can check which simulations are in the queue with: 

qstat
To print out only your jobs: 
qstat -u username
A prettier alternative to obtain the same information is to connect to the IPP network via X2Go (pure VPN doesn't do the trick), open a web browser and type into the address bar: 
soroban:44444
Don't expect immediate updates, it takes a minute or two to see your submitted (or deleted) job. To find how busy each Soroban node is, you can use the command 
pbsnodes -a
or go to the address https://prometheus.tok.ipp.cas.cz/grafana/d/525hCXDMz/soroban-cluster-utilization?orgId=1.

To delete a job, use:

qdel NNNNN

where NNNNN are the first 5 numbers printed by qstat -u username in the column Job id.

Restarting a SOLPS-ITER run

In practice, you'll sometimes run a case for a shorter amount of time to see what it will do, or to change up input values and see the effect. It is plain nonsense to be starting from scratch, "flat profiles", every time. You'll need to start the next run where the previous one ended. These instructions show you how.

Important: When manipulating b2fstati and b2fstate, always use the cp command! Using the file browser to copy the files, renaming the files etc. causes the b2fstati file to be ignored and the simulation begins again from the flat profiles state.

Go into the run/ folder and:

  1. Check if your run has concluded successfully. This can include plotting the simulation output (echo "phys te surf" | b2plot) or the residuals (resco), reading the end of run.log and checking the size and contents of b2fstate (if the run crashes, it may be nearly empty).
  2. Delete b2mn.prt. Failing to do so will prevent your simulation from running.

  3. Rename b2fstate (e for "end") to b2fstati (i for "initial"). Always use the command line to manipulate b2fstate/b2fstati, otherwise the b2fstati could be ignored.

    cp b2fstate b2fstati

  4. If you want to keep a copy of the simulation results, archive b2fstate (rename it to e.g. b2fstate_before_gas_puff). If you're running a coupled run and want to keep the EIRENE results as well, archive also fort.13, 14, 15, 44 and 46. (Refer to the source code of the $SOLPSTOP/runs/SLURM.itm scripts group.)

  5. Run b2run b2mn >& run.log & or a submission script again.

When resubmitting a run, don't use the setup_baserun_eirene_links command. It will cause your simulation to start from the very beginning (flat profiles).

Branching out a SOLPS-ITER run

Sometimes, you'll want to tweak a simulation a little (like changing the perpendicular diffusion coefficient) to see what happens, but you don't want to lose the original simulation. Other times, you'll want to conduct a parameter scan. In those cases, there's no need to create a new simulation (baserun/ from scratch) or start simulating from the flat profiles. What you need is branching an existing run out.

The simplest way to do this is to copy the entire run folder. 

cd $SOLPSTOP/runs/my_simulation
cp -r run_existing run_new
This should transfer every aspect of the run.

If you wish for a cleaner start, create the run_new folder and copy inside the following files (using the cp command, see below):

b2.boundary.parameters
b2.neutrals.parameters
b2.transport.inputfile   #if using anomalous diffusion coefficient profiles
b2.transport.parameters

b2mn.dat
input.dat
b2fstati

To my knowledge, b2fstate/b2fstati files are fully portable between simulations of the same mesh size. Supplying a new b2fstati will take the run instantly to that plasma state. Finally, run setup_baserun_eirene_links in the run_new folder.

Important! When making copies of the files, use the cp command and do not copy the files in a file browser! Using the file browser has been found to produce input.dat which prevented the simulation from running and b2fstati which was ignored in favour of starting from the flat profiles. To file browser lovers like me, it has repeatedly happened that the simulation started from scratch without my knowledge, due to badly copied b2fstate/b2fstati. I believe the reason is somewhere along the lines of time stamps.

Interrupting a running SOLPS-ITER run

touch b2mn.exe.dir/.quit
This will end the simulation after the current iteration is finished, write the output files and quit SOLPS-ITER. But general advice - you should leave the situations running for a bit longer after they seem to have converged, as they might be in a meta-stable state that is about to explode after a few more days.

Defining how long SOLPS-ITER should run

There are several options to define this, all specified in b2mn.dat. To set the number of iterations (1 EIRENE call + 14 B2.5 calls, see also Questions and answers: What is the meaning of residual plots): 

'b2mndr_ntim' '100'
To set the target value of residuals: 
'b2mndr_min_areshe' ' 1.00E-03'
'b2mndr_min_areshi' ' 1.00E-03'
'b2mndr_min_aresco' ' 1.00E-03'
To set the target CPU time (in seconds): 
'b2mndr_cpu' '3600'
To set the target real elapsed time (in seconds): 
'b2mndr_elapsed' '3600'
Out of the last two, using the real elapsed time is more reliable when running SOLPS-ITER within limited time windows on a cluster.

Checking if the run has converged

See also Questions and answers: Convergence, section 3.5.11 of the manual, While the code is running, and Common pitfalls: Simulation divergence.

Unless you're doing time-dependent modelling (such as ELM modelling), code convergence basically means you run SOLPS-ITER again and again until you reach a stable solution. (Whether this solution is the globally best one, that's a whole different question.) Such a convergence is reached when further running the code makes no difference on the solution. There are several ways to check the convergence:

  1. Check the residuals using commands such as resco, resall_D and others (section 2.5 of the manual). Further information can be found in Questions and answers: What is the meaning of residual plots.

  2. Plot several important quantities, such as the separatrix temperature or the total ion energy, in time using the 2dt command (appendix E of the manual). If the line you see is flat, your run isn't evolving anymore.

    2dt tesepm

  3. Perform a sanity check by looking at the code output and, ideally, compare it to diagnostic data. Refer to Questions and answers: How can I tell if my converged simulation is "physical".

  4. Check the particle balance. This is done by setting the following switches in b2mn.dat:

    'tallies_netcdf' '10'       #save the tallies for every 10 steps
'balance_netcdf'  '1'

    Then, in the run directory, call:

    particle_balance.py

    This will plot the difference between the puffed and pumped particle flux. It should be very small compared to the whole flux.

  5. Check the energy fluxes with

    energy_analysis

Convergence criteria are also discussed in section 4 of [Wiesen 2015].

b2run troubleshooting

It is good to know that b2run is actually an alias for gmake, thus many error messages are the same.

Target does not exist

You have probably omitted a step in B2.5 hierarchy, something failed or whatever. A target mentioned in the message is needed to be run beforehands (depends on the stage). Do not run the target by yourself, try to find out why it was not created (easy solution: you forgot; hard solution: something failed).

Faulty internal parameter jxa

There are control parameters for indexing outer midplane etc. in b2mn.dat. Their values are marked as b2mwti_jxa, b2tqna_ixa in the file. They can be omitted to use default values. The probable cause is that you have created different grid (different number of grid points or surfaces). Look those names up in the manual for more information.

Transferring a SOLPS-ITER run using b2yt

b2yt is useful in the following situations:

  • You have a converged run and want to increase the grid resolution.
  • You want to build a new case whose geometry is similar but not the same as a previously converged run, and you don't want to wait until the "flat profiles" solution converges.
  • You want to check if the converged result is the same between two SOLPS version (4.3, 5, another release of SOLPS-ITER...).

The following materials concern b2yt:

Transferring a run to another case or server is described in Remote access: Transferring SOLPS-ITER runs.