[Last version at December 11, 2014]
0. Confirmed running environment

OS  : CentOS 6.6
MPI : IntelMPI 5.0.2 runtime

[Contents]
1. Compilation
2. Execution
2.1 System environment
2.2 Setting on ssh
2.3 Setting of running hosts
2.4 Starting daemon
2.5 Checking job status and stopping daemon
2.6 Execution of mpiTaogen using mpirun
2.7 Execution of mpiTaogen using mpiexec
2.8 Configuration file for analysis parameters
2.9 Configuration file for running hosts
2.10 Configuration file for initial parameters
2.11 Input file (gene expression data)
2.12 Output file (last results, log)
2.13 On the execution and ending of processes
A. Script for setting environmental variables
B. Output format
B.1 .out (last results)
B.2 .log (log for exchanges)
B.3 .measure (log for measurements)
B.4 .param (log for parameters at samplings)
B.5 .hist (histgram for parameters at samplings)

-----------------------------------------------------------------------------
1. Compilation
Following library needs to be installed and added to the PATH.

- intel fortran & library
- intel MPI

To add the PATH, please refer to the last section in this document, "A. Script for setting environmental variables". If these settings are done, all you will do for compilation is to execute "make", as follows.

$ make

As a result, taogen, extra, and mpiTaogen will be made. The mpiTaogen is the load module for MPI parallel version of taogen.

Note: The compiler, mpifort, will give an error message, '[: 478: no: unexpected operator', when it is executed on linux ubuntu. It seems that this is due to the discordance between intel compiler and ubuntu ( the ubuntu is the outside of intel compiler support ). There is no practical problem.
-----------------------------------------------------------------------------
2. Execution
There are various ways for execution, however, only two will be described here.

- execution by mpirun
- execution by mpiexec
-----------------------------------------------------------------------------
2.1 System environment
Following library needs to be installed in all of the hosts of use.

- intel fortran & library
- intel MPI

2.2 Setting on ssh
This will explain on how to use ssh for connection, omitting pass phrase. Use ssh-agent. Assume that there are four computers, red1-4, which share the home with NFS. When you put the private key(~/.ssh/id_rsa) and the public key(~/.ssh/authorized_keys) in ~/.ssh/, then the pass phrase will not be requested by running ssh-agens as follows:

$ eval `ssh-agent`
$ ssh-add 
Enter passphrase for /home/.../id_rsa: [pass phrase]

It is successful if you type as:

red1> $ ssh red2 uname -a

2.3 Setting of running hosts
Enumerate a group of running hosts in the file called mpd.hosts. Place it in the running directory.

Example) the case using red1-4

$ cat mpd.hosts
red1
red2
red3
red4

* Intel fortran & library & MPI must be installed on the hosts.

2.4 Starting daemon
Start MPI daemon on the MPI parallel computers. Starting it with mpdboot, two options are required.

-r ssh : specify to use ssh for interprocess communications.
-n # : the number of computer hosts (specify 4 if four hosts (four lines) exist in mpd.hosts.

$ mpdboot -r ssh -n 4

If ssh-agent is not started, the pass phrase will be requested for each host.

2.5 Checking job status and stopping daemon
The method to check the running status of MPI daemon is as follows.

$ mpdtrace [-l]
red4_58802 (192.168.236.93)
red1_60294 (192.168.236.90)
red3_45501 (192.168.236.92)
red2_46728 (192.168.236.91)

The method to halt MPI daemon is as follows.

$ mpdallexit

* It seems better to perform mpdallexit upon logout. If you logout without this procedure, the daemon will survive in most cases and be usable at re-login. There are also non-usable cases.

2.6 Execution of mpiTaogen using mpirun
To execute mpiTaogen, make a shell script for it, and pass it to mpirun. When the path to mpiTaogen is /home/tany/MCMC/test/mpiTaogen and the path to configuration file (to be described later) that contains analysis parameters is /home/tany/MCMC/test/conf.txt, The content of shell script will become as follows (suppose that the file name is test.sh).

$ cat test.sh
exec /home/tany/MCMC/test/mpiTaogen < /home/tany/MCMC/test/data.txt

Pass it to mpirun and excute mpiTaogen as follows.

$ mpirun -n 5 ./test.sh

-n indicates the number of processes to run, which is the total number of parent and child processes. The number of child processes will be set by the configuration file for analysis parameters (Chapter 2.7). Please specify this number+1.

2.7 Execution of mpiTaogen using mpiexec
Pass the configuration file on running hosts (to be described later) and the configuration file that contains analysis parameters (the same as the one used in mpirun; to be described later for details) to mpiexec. If the configutation file on running hosts is host.txt and the configuration file on analysis parameters is data.txt, implement will be as follows.

$ mpiexec -configfile host.txt < data.txt

2.8 Configuration file for analysis parameters

There are parent and child processes in mpiTaogen. The child process performs samplings. This configuration file describes hyper parameters to be used in each child process, a path to input data (gene expression profiles), methods of measurements to use, values on output format. The configuration format is indicated as follows.

Line 1 : the number of child processes
Line 2 : an array of hyper parameter degree1 (variation of beta with edges, to be described later)
Line 3 : an array of hyper parameter degree2 (variation of beta without edges, to be described later)
Line 4 : an array of hyper parameter degree3 (dummy and not used, to be described later)
Line 5-: parameters to pass to input_data (namelist)
Line end: configuration on sampling parameters for histogram output

Lines 2-4 are arrays to specify hyper parameters. The size of the array is the same as the number of child processes.

The arguments to pass to input_data are as follows.

- sample        --- the number of samples in the gene expression profile
- gene          --- the number of genes in the gene expression profile
- sampling      --- the number of samplings
- array_data    --- path to a file of gene expression profile (relative path is allowed)
- replica_exchang --- swith of replica exchange(0:OFF, 1:ON)
- exchange      --- the number of exchanges in replica exchange
- replica_exchang_order 
            --- the order of replica exchange
                (0:random, 1:broader to narrower of search range)
- r             --- upper limit of exchange probability (real in [0,1]. basicaly 1. In the process of real calculation, generate a random value in [0,1] and exchange if it is min(r,r(k,k+1)) or smaller, otherwise not exchange.
- measure       --- method of measuremnt(0:log likehood, 1:log posterior probability)
- burn_in       --- the number of samplings to ignore in the collection of summary (specify the number to ignore from the first sampling)
- count_in      --- the number of samplings to use in the collection of summary (specify the number to use from the last sampling. To be ignored if burn_in is specified)
- verbose       --- log output format(0:.out+.log, 1:.out+.log+.param+.measure, 2:.out+.log+.param+.measure+.hist)
- varbose_measure   --- writing interval of measurements(step number)
- verbose_parameter --- writing interval of measurements(step number)
- verbose_histogram --- writing interval of histogram, beta(step number)

Hyper parameters (degree1-3) specify the variations of beta in samplings. Each hyper parameter corresponds to the object as follows.

- degree1 --- When an edge exists in sampling at one step before, this value will be used for the variance of proposed distribution for sampling the beta, which is sigma(beta)^2 with the symbols of "Inference for Bayesian network via Gibbs sampling". When an edge exists, the prior distribution of beta is set as mean of 0 and variance of sigma(beta)^2. Although this depends on the model and data, if extremely small value is set, all of beta values come close to 0. From the viewpoint of regularization, a value around 1 seems good. If it increases, the effect of prior distribution will be decreased, which can disturb edges even if it exits.
- degree2 --- Reversely, When an edges did not exist, this value will be used for the variance of proposed distribution for sampling the beta, which is (sigma(beta)*C)^2 with the symbols of "Inference for Bayesian network via Gibbs sampling". As the degree1, the prior distbituion of beta is set as mean 0 and variance of (sigma(beta)*C)^2. This is the case where an edge did not exist, and it will be good if the value is decreased when you think beta should be close to 0. If you would disturb more, you should increase this value, i.e., it is possible to sample a large value of beta, which will affect the next sampling. However, if more than 1 is set, it becomes larger than degree1, and it will be in vain to set both parameters, degree1 and degree2.
- degree3 --- This is a dummy. Not to be used in this version.

In the case of printing histograms of parameter samplings, the settings need to be described at the last line. The following parameters are set by comma-separated arrays:
min, max, step


Ex.) A realistic example ( comments after - )
$ cat data.txt
4 ---- # procs (next line degree2 array which length is # procs)
10.0,10.0,10.0,10.0
0.001,0.01,0.1,1.0
0.0001,0.001,0.01,0.1
&input_data
sample=100,
gene=8, 
sampling=1000, 
array_data="../pseudo_para/data/sample8-8/sample_data8.dat",
replica_exchange=1,
exchange=10,
r=1.0,
measure=1,
burn_in=5000,
verbose=2,
verbose_measure=10,
verbose_parameter=50,
verbose_histogram=500
/
-3.0,3.0,0.1


2.9 Configuration file for running hosts

In mpiexec, execution options can be set on each running host.

If the running hosts are red1-4, this settings will be:

$ cat host.txt
-s all
-n 2 -host red1 ./mpiTaogen
-n 1 -host red2 ./mpiTaogen
-n 1 -host red3 ./mpiTaogen
-n 1 -host red4 ./mpiTaogen

Each option corresponds to:

-s all : pass stdin to whole processes
-n N -host XXX LM : run LM on host XXX with the number of running process of N

According to the specified order, rank (which consider as process ID) will be numbered. The rank=0 specified first in red1 becomes a parent process, another process in red1 becomes rank=1, a process in red2 becomes rank=2, and so on. Because we need a parent proces, the sum of process numbers specified here will become the number of child process, which is set in the cofiguration file for analysis and etc. parameters (Chapter 2.8), plus one.

Also, you can set the initial values for parameters, beta and T. Store the initial values in the file (the formatting will be described later), setting by the following options:

-cmat [file name of initial T values]
-cbet [file name of initial beta values]

In the configuration file, the settings will be:

$ cat host.txt
-s all
-n 2 -host red1 ./mpiTaogen -cmat cmat_data8.dat -cbet cbet_data8.dat
-n 1 -host red2 ./mpiTaogen -cmat cmat_data8.dat -cbet cbet_data8.dat
-n 1 -host red3 ./mpiTaogen -cmat cmat_data8.dat -cbet cbet_data8.dat
-n 1 -host red4 ./mpiTaogen -cmat cmat_data8.dat -cbet cbet_data8.dat

2.10 Configuration file for initial parameters
It is possible to set initial beta and T values. Since these are also values for pairs of genes, they will be described in array format, as .out or .log. The example is as follows.

 Ex.) the case of beta
 0.00    0.00    0.00    0.01   -0.01   -0.01   -0.02   -0.93 
-0.01    0.00    0.01    0.00    0.01   -0.01    0.00   -0.01 
 0.00    0.01    0.00    0.02   -0.01    0.00   -0.01    0.00 
 0.06   -0.04   -0.06    0.00   -0.01    0.97    0.04    3.43 
 0.00   -0.61    0.00   -0.02    0.00    0.00    0.01    1.21 
-0.01    1.57   -0.01    0.00    2.05    0.00    0.03    0.13 
 0.01   -0.01   -0.01    0.01   -0.01    0.26    0.00    0.01 
 0.00    0.00    0.00    0.01    0.00    0.00   -0.01    0.00 

 Ex.) the case of T
 0  0  1  0  0  0  0  1
 0  0  0  0  0  0  0  0
 0  0  0  0  0  0  0  0
 0  0  0  0  0  1  0  1
 0  1  0  0  0  0  0  1
 0  1  0  0  1  0  0  1
 0  0  0  0  0  1  0  0
 0  0  0  0  0  0  0  0

2.11 Input file (gene expression data)
Input file should contain gene expression profile data. Rows are genes, and columns are samples. An practical example is as follows.

1.53485849353176        1.79157470989697        2.44855888944983      ... 
4.22290886760429        5.030559490864          6.9099493377697       ...
2.70657410181092        3.39558916186227        4.49803590653045      ...
-1.71328717000842       -1.77374402167112       -2.45228036917643     ...
-2.90522273721228       -3.65885302972593       -4.85261663253319     ...
 

In the inside of mpiTaogen, the orders of rows and columns are recognized as gene IDs and sample IDs. The gene ID of the first row is 1, the second is 2, the third is 3, and so on. The sample ID of the first column is 1, the second is 2, the third is 3, and so on. The result output prints parameters, sampling results, and its expected values in array format. Their rows and columns are genes which order is the same as the order of IDs.

* The input data are required to be normalized by row and column, providing normal distribution.

2.12 Output file (last results, log)
The output results are as follows.
- mpiTaogen.out       : A file contains expected values of sampling parameters (.out).
- mpiTaogen.log       : A file to be generated at replica exchange, containing parameters and measurements (.log). The   ##   means a rank.
- mpiTaogen.param##   : A file contains values of sampling parameters (.param). The   ##   means a rank. This file is generated when verbose is set to 1 or 2 in the configuration file for analysis parameters (Chapter 2.7). The sampling results will be printed by specified interval.
- mpiTaogen.measure## : A file contains measurements (.measure).The   ##   means rank. This file is generated when verbose is set to 1 or 2 in the configuration file for analysis parameters (Chapter 2.7).  The measurements will be set with interval specified by   verbose_measure".
- mpiTaogen.hist##    : A file contains histograms of parameters (.hist). The   ##   means a rank. This file is generated when verbose is set to 2 in the configuration file for analysis parameters (Chapter 2.7). The histograms will be printed by interval specified by "verbose_histogram  . The targets of numbering will be sampling results after steps specified by   burn_in  .

Matrices of sampling parameters (or expected values for .out) will be printed into .out, .log, and .param. These are for gene pairs, and a row corresponds to a specific gene. For example, the orders of rows and columns will correspond to gene IDs (Chapter 2.10).

Histograms of sampling parameters will be printed into .hist. A histogram for T will be binary, but beta is a continuous value thus a user must specify the step value. This configuration is set in the configuration file for analysis parameters (Chapter 2.7). In this configuration, if the minimum, maximum, and the step value are -1, 1, and 0.5, respectively, the histogram will be 4 districts. The following is a matrix to explain this format. Values will be input to (). The content is ([gene ID],[gene ID],[histogram step ID]). In this case, the gene IDs correspond to row IDs, but not to column IDs. In the case that a histogram has four districts for a gene, after values of districts 1, 2, 3, and 4, the following histogram will be printed. The   ,   becomes the delimiter between genes.

(1,1,1) (1,1,2) (1,1,3) (1,1,4), (1,2,1) (1,2,2) (1,2,3) (1,2,4), ...
(2,1,1) (2,1,2) (2,1,3) (2,1,4), (2,2,1) (2,2,2) (2,2,3) (2,2,4), ...
(3,1,1) (3,1,2) (3,1,3) (3,1,4), (3,2,1) (3,2,2) (3,2,3) (3,2,4), ...
.
.

2.13 On the execution and ending of processes
The executed results (.out, .log, .measure, and .hist) are printed by a parent process (rank=0). After a parent process gives a message as follows, there is a case that the child process may end.

MPI processers nomrmaly terminated!

In this case, although there may be a case that MPI daemon outputs warnings such as signal kill (=9) or else, there is no critical problem. However, if a message such as error at daemon part before the above message is displayed, it may be configuration errors or program bugs.

-----------------------------------------------------------------------------
A. Script for setting environmental variables

# Intel Fortran and MKL
if [ -e /opt/intel/Compiler/11.1/056/bin/intel64/idbvars.sh ]; then
 . /opt/intel/Compiler/11.1/056/bin/intel64/idbvars.sh
fi
if [ -e /opt/intel/Compiler/11.1/056/tbb/bin/tbbvars.sh ]; then
 . /opt/intel/Compiler/11.1/056/tbb/bin/tbbvars.sh intel64
fi
if [ -e /opt/intel/Compiler/11.1/056/Frameworks/tbb/bin/tbbvars.sh ]; then
 . /opt/intel/Compiler/11.1/056/Frameworks/tbb/bin/tbbvars.sh intel64
fi
if [ -e /opt/intel/Compiler/11.1/056/mkl/tools/environment/mklvarsem64t.sh ]; then
 . /opt/intel/Compiler/11.1/056/mkl/tools/environment/mklvarsem64t.sh
fi
if [ -e /opt/intel/Compiler/11.1/056/Frameworks/mkl/tools/environment/mklvarsem64t.sh ]
; then
 . /opt/intel/Compiler/11.1/056/Frameworks/mkl/tools/environment/mklvarsem64t.sh
fi
if [ -e /opt/intel/Compiler/11.1/056/ipp/em64t/tools/env/ippvarsem64t.sh ]; then
 . /opt/intel/Compiler/11.1/056/ipp/em64t/tools/env/ippvarsem64t.sh
fi
if [ -e /opt/intel/Compiler/11.1/056/Frameworks/ipp/tools/env/ippvarsem64t.sh ]; then
 . /opt/intel/Compiler/11.1/056/Frameworks/ipp/tools/env/ippvarsem64t.sh
fi
if [ -e /opt/intel/Compiler/11.1/056/bin/intel64/ifortvars_intel64.sh ]; then
 . /opt/intel/Compiler/11.1/056/bin/intel64/ifortvars_intel64.sh
fi

# MPI
if [ -e /opt/intel/impi/3.2.2/bin64/mpivars.sh ]; then
 . /opt/intel/impi/3.2.2/bin64/mpivars.sh
fi

-----------------------------------------------------------------------------
B. Output format
[Special Instructions]

Some output results contain a string of "****************".
In the output format from fortran, this will be displayed when more than the specified digits are printed. The above case indicates 16 or more digits.

B.1 .out (last results)
$ cat mpiTaogen.out
#############################################################################
# Setting and parameters
#############################################################################
array data=./sample_data8.dat
CPU(#)          : 4               
Sample(#)       : 100             
Gene(#)         : 8               
Sampling        : 1000            
Measure         : 1(log posterior probability(lpp))
Burn out        : 5000            
Replica exchange: 1
exchange       : 10              
r              : 1.00
Verbose         : 2
measure        : 10              
parameter      : 50              

CPU1
degree1 : 10.0000
degree2 :  0.0010
degree3 :  0.0001

CPU2
degree1 : 10.0000
degree2 :  0.0100
degree3 :  0.0010

CPU3
degree1 : 10.0000
degree2 :  0.1000
degree3 :  0.0100

CPU4
degree1 : 10.0000
degree2 :  1.0000
degree3 :  0.1000

##############################################################################
# Sampling :  10
##############################################################################
## CPU1,10.0000, 0.0010, 0.0001
# Mean of transition(T)
0.00  0.02  0.03  0.29  0.02  0.08  0.02  0.00 
0.01  0.00  0.01  1.00  0.01  0.02  1.00  0.01 
...
# Mean of beta(B)
0.00   0.00  -0.01   0.07   0.00   0.02   0.00   0.00 
0.00   0.00   0.00   5.90   0.00  -0.01   1.63   0.00 
...
# Log posterior probability
final :        -859.8087

## CPU2,10.0000, 0.0100, 0.0010
# Mean of transition(T)
0.00  0.03  0.03  0.14  0.04  0.08  0.03  0.00 
0.03  0.00  0.03  1.00  0.03  0.05  1.00  0.03 
...
# Mean of beta(B)
0.00   0.00  -0.02   0.04   0.01   0.04  -0.01  -0.01 
0.01   0.00   0.01   5.90   0.02  -0.04   1.64   0.00 
...
# Log posterior probability
final :        -827.5890

## CPU3,10.0000, 0.1000, 0.0100
# Mean of transition(T)
0.00  0.07  0.11  0.20  0.07  0.16  0.08  0.00 
0.09  0.00  0.08  0.99  0.08  0.10  0.98  0.07 
...
# Mean of beta(B)
0.00   0.00  -0.24   0.10   0.07   0.27  -0.09  -0.12 
0.02   0.00   0.03   5.78   0.04  -0.10   1.61   0.01 
...
# Log posterior probability
final :         -82.9500

## CPU4,10.0000, 1.0000, 0.1000
# Mean of transition(T)
0.00  0.15  0.29  0.25  0.16  0.71  0.17  0.14 
0.19  0.00  0.25  0.82  0.18  0.37  0.27  0.16 
...
# Mean of beta(B)
0.00  -0.02  -1.27   0.04  -0.06   2.25  -0.59  -0.67 
-0.31   0.00  -0.47   3.90   0.02   0.82   0.89  -0.23 
...
# Log posterior probability
final :          75.0476

B.2 .log (log for exchanges)
The output results when exchange=1000.
$ cat mpiTaogen.log
#############################################################################
# Setting and parameters
#############################################################################
Array data=./sample_data8.dat
CPU(#)          : 4               
Sample(#)       : 100             
Gene(#)         : 8               
Sampling        : 1000            
Measure         : 1(log posterior probability(lpp))
Burn out        : 5000            
Replica exchange: 1
exchange       : 10              
r              : 1.00
Verbose         : 2
measure        : 10              
parameter      : 50              

CPU1
degree1 : 10.0000
degree2 :  0.0010
degree3 :  0.0001

CPU2
degree1 : 10.0000
degree2 :  0.0100
degree3 :  0.0010

CPU3
degree1 : 10.0000
degree2 :  0.1000
degree3 :  0.0100

CPU4
degree1 : 10.0000
degree2 :  1.0000
degree3 :  0.1000

##############################################################################
# Exchange : 1(Step:1000)
##############################################################################
Parameters:
## CPU1,10.0000, 0.0010, 0.0001
# Transition(T)
0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00 
0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00 
 ...
# Beta(B)
 0.00    0.00    0.01    0.00   -0.03    0.05    0.00    0.00 
 0.03    0.00    0.04    0.05   -0.03    0.00    0.03    0.02 
 ...
## CPU2,10.0000, 0.0100, 0.0010
# Transition(T)
0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00 
0.00  0.00  0.00  0.00  1.00  1.00  0.00  0.00 
...
# Beta(B)
 0.00   -0.04   -0.02    0.09   -0.08    0.05    0.01   -0.01 
-0.01    0.00    0.01   -0.12   -0.54    1.35    0.00    0.00 
...
## CPU3,10.0000, 0.1000, 0.0100
# Transition(T)
0.00  0.00  0.00  1.00  0.00  0.00  0.00  0.00 
0.00  0.00  0.00  1.00  0.00  0.00  1.00  0.00 
...
# Beta(B)
 0.00    0.05   -0.14   -0.81    0.08   -0.01   -0.35   -0.05 
 0.01    0.00   -0.02    5.82   -0.05    0.10    1.61    0.00 
 ...
## CPU4,10.0000, 1.0000, 0.1000
# Transition(T)
0.00  1.00  1.00  1.00  0.00  0.00  0.00  0.00 
0.00  0.00  1.00  1.00  0.00  1.00  0.00  0.00 
...
# Beta(B)
 0.00   -0.21   -1.26    1.17   -0.06    2.25   -0.27   -0.66 
-0.57    0.00   -0.77    2.68   -0.01    1.53    0.38   -0.43 
...
Replica exchange:
# CPU 1 vs CPU 2
Without parameter swapping : CPU1:       -846.9095, CPU2:       -865.8279
With parameter swapping    : CPU1:       -870.0413, CPU2:      -1212.0500
r:1.0000, r(k,k+1):   1.4381, r/r(k,k+1):   0.3477
Exchanged!
# CPU 2 vs CPU 3
Without parameter swapping : CPU2:       -865.8279, CPU3:       -448.8446
With parameter swapping    : CPU2:       -969.5050, CPU3:       -436.9587
r:1.0000, r(k,k+1):   1.0901, r/r(k,k+1):   0.4587
Exchanged!
# CPU 3 vs CPU 4
Without parameter swapping : CPU3:       -448.8446, CPU4:        176.9399
With parameter swapping    : CPU3:       -551.5436, CPU4:        264.6506
r:1.0000, r(k,k+1):   1.8379, r/r(k,k+1):   0.2720
Exchanged!
##############################################################################
Parameters:
## CPU1,10.0000, 0.0010, 0.0001
# Transition(T)
0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00 
0.00  0.00  0.00  0.00  1.00  0.00  0.00  0.00 
...

B.3 .measure (log for measurements)
The output results for rank=1 when verbose=1, and verbose_parameter=50.
$ cat mpiTaogen.param1 
##Step:      50
Parameters:
# Transition(T)
0  0  0  0  0  0  0  1
0  0  0  0  0  0  1  0
0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  1
0  0  0  0  0  0  0  1
0  0  0  0  1  0  0  0
0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0
# Beta(B)
 0.00    0.02   -0.01    0.00   -0.01   -0.01    0.02   -1.21 
 0.01    0.00    0.01    0.01    0.01    0.01    0.70   -0.01 
-0.02    0.02    0.00    0.00    0.00    0.00    0.02    0.00 
 0.00    0.05   -0.03    0.00   -0.03   -0.07   -0.04    3.25 
 0.02    0.01    0.02    0.01    0.00    0.01   -0.02    1.47 
-0.02   -0.05   -0.03    0.00    1.99    0.00    0.00   -0.04 
-0.01    0.02   -0.02   -0.01   -0.01   -0.02    0.00    0.01 
 0.00    0.01   -0.01   -0.01    0.00    0.00    0.01    0.00 

##Step:     100
Parameters:
# Transition(T)
0  0  1  0  0  0  0  1
0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0
0  0  0  0  0  1  0  1
0  1  0  0  0  0  0  1
0  1  0  0  1  0  0  1
0  0  0  0  0  1  0  0
0  0  0  0  0  0  0  0
# Beta(B)
 0.00    0.00    0.00    0.01   -0.01   -0.01   -0.02   -0.93 
-0.01    0.00    0.01    0.00    0.01   -0.01    0.00   -0.01 
 0.00    0.01    0.00    0.02   -0.01    0.00   -0.01    0.00 
 0.06   -0.04   -0.06    0.00   -0.01    0.97    0.04    3.43 
 0.00   -0.61    0.00   -0.02    0.00    0.00    0.01    1.21 
-0.01    1.57   -0.01    0.00    2.05    0.00    0.03    0.13 
 0.01   -0.01   -0.01    0.01   -0.01    0.26    0.00    0.01 
 0.00    0.00    0.00    0.01    0.00    0.00   -0.01    0.00 

B.4 .param (log for parameters at samplings)
The output results when verbose=1, and verbose_parameter=10.
$ cat mpiTaogen.measure1 
Step   Log posterior probability
   10       -1691.1120
   20       -1631.8041
   30       -1485.6158
   40       -1512.2003
   50       -1468.4915
   60       -1487.3927
   70       -1435.0071
   80       -1486.9147
   90       -1511.4319
  100       -1495.9154
  110       -1334.2992
  120       -1278.8822
  130       -1330.1334
  140       -1326.7853
  150       -1074.9012
  160        -945.2173
  170        -956.6725
  180       -1001.1356
  190       -1129.3547
  200       -1009.3109
  210       -1032.0119
  220       -1006.5198
  230        -886.9210
  240        -869.9092
  250        -879.9200
  260        -904.0355
  270        -790.0273
  280        -865.8678
  ...

B.5 .hist (histgram for parameters at samplings)


The output results for rank=1 when verbose=1, and verbose_histogram=500.
$ cat mpiTaogen.hist1
#############################################################################
# Histogram setting
#############################################################################

Histogram Transition(T) : 0 or 1
Histogram Beta(B)       : from -3.00 to  3.00,  step size is 0.10

#############################################################################
# Histogram
#############################################################################

## Step: 5000-5500            
# Histogram Transition(T):
   500        0,      414       86,      348      152,      393      107,      405       95,      158      342,      399      101,      426       74
   414       86,      500        0,      369      131,       76      424,      408       92,      303      197,      356      144,      422       78
   425       75,      437       63,      500        0,      394      106,      424       76,      259      241,      419       81,      451       49
   464       36,      494        6,      445       55,      500        0,      462       38,      423       77,      499        1,      492        8
   423       77,      434       66,      401       99,      370      130,      500        0,       97      403,      419       81,      433       67
   478       22,      466       34,      454       46,      407       93,      479       21,      500        0,      448       52,      500        0
   425       75,      409       91,      413       87,       11      489,      424       76,      376      124,      500        0,      437       63
   294      206,      397      103,      244      256,      164      336,      419       81,        4      496,      391      109,      500        0
# Histogram Beta(B):
     0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0      500,        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        1       43         ...
## Step: 5000-6000            
# Histogram Transition(T):
  1000        0,      831      169,      705      295,      772      228,      819      181,      278      722,      801      199,      871      129
   796      204,     1000        0,      736      264,      160      840,      823      177,      630      370,      711      289,      850      150
   845      155,      880      120,     1000        0,      791      209,      859      141,      509      491,      842      158,      910       90
   918       82,      982       18,      890      110,     1000        0,      917       83,      840      160,      992        8,      969       31
   835      165,      862      138,      796      204,      736      264,     1000        0,      198      802,      825      175,      857      143
   957       43,      926       74,      918       82,      831      169,      966       34,     1000        0,      919       81,     1000        0
   841      159,      833      167,      821      179,       38      962,      836      164,      760      240,     1000        0,      873      127
   589      411,      791      209,      467      533,      379      621,      830      170,       14      986,      772      228,     1000        0
# Histogram Beta(B):
     0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0     1000,        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        2       84       66        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0        0     ...
