Module run_several

source code

This script runs a bunch of processes in parallel. You tell it what to run on the standard input. It spawns off a specified number of subprocesses, and hands the jobs to the subprocesses.

Usage: run_several.py [-v] Ncpu Oom_adjust <list_of_commands >log_of_outputs

Flags: -v means "be more verbose".

Ncpu is the maximum number of cores to devote to running these tasks (you can enter zero to have it use all the cores of the local machine), or +N or -N to have it use N more (or less) cores than the machine has. Further x and * allow you to multiply the number of CPUs by a factor. (NB: The +, -, x, * forms always leave at least one core running, no matter what N is or how few cores you have.) All this can be useful if you want to avoid loading the machine too heavily, or if you want to boost the loading because your jobs are limited by disk seek time.

Oom_adjust is a number given to Linux's OOM killer. When the system starts running out of memory, the OOM killer kills processes. Positive numbers (e.g. 3) make it more likely that a process will be killed. So, if your subtasks are likely to use too much memory and are not critical, set Oom_adjust positive.

The commands in list_of_commands are just piped into Bash's standard input one at a time. Each line is (typically) processed by a different instance of bash, so don't try any multi-line commands. The resulting standard outputs and standard errors are kept separate and each processes' outputs are printed as a lump when it completes. Stderr from a subprocess comes out as stdout from run_several.py; various blocks of output are separated by lines with hash marks and strings of "----" (see write_msgs()).

Note that the processes may finish in any arbitrary order. The integer on the stdout and stderr separator lines gives you the (zero-based) line number in the input file. Run_several.py returns failure if any of its subprocesses fail, and it will terminate on the first failure.

Example 1:

       $ echo "pwd" | run_several.py 0 0
       # command 0: pwd
       # stdout 0 ----------------(exited with 0)
       /home/gpk/speechresearch/gmisclib/bin
       # stderr 0 ----------------
       $

Example 2:

       $ { echo uname -m; echo uname -p; echo uname -o;} | run_several.py 0 0
       # command 0: uname -m
       # stdout 0 ----------------(exited with 0)
       x86_64
       # stderr 0 ----------------
       # command 1: uname -p
       # stdout 1 ----------------(exited with 0)
       unknown
       # stderr 1 ----------------
       # command 2: uname -o
       # stdout 2 ----------------(exited with 0)
       GNU/Linux
       # stderr 2 ----------------
       $ 

Imports: sys, time, tempfile, subprocess, die, SL