Committing initial version of Kaldi

git-svn-id: https://svn.code.sf.net/p/kaldi/code/trunk@2 5e6a8d80-dfce-4ca6-a32a-6e07a63d50c8

INSTALL

+
+[for native Windows install, see windows/INSTALL]
+
+(1)
+go to tools/  and follow INSTALL instructions there.
+
+(2) 
+go to src/ and follow INSTALL instructions there.
+

README.txt

+
+This README has been created for those with whom we share the
+"pre-release" version of Kaldi.  Although the toolkit has not
+been "officially" released, I have been given the OK to share
+it privately for "non-commercial purposes" (whatever that means).
+The official release is scheduled for mid-March.
+
+The current version is not as polished as we would like, and contains
+some files that should eventually be deleted.  
+
+See http://merlin.fit.vutbr.cz/kaldi/ for documentation 
+(may not always be fully up to date).  This documentation
+is generated by running "doxygen" from the src/ directory,
+and appears in src/html/
+
+I assume that the reader would like to (1) build the toolkit
+and (2) run the example system builds.
+
+To build the toolkit: see ./INSTALL.  These instructions are valid for UNIX
+systems including various flavors of Linux; Darwin; and Cygwin (has not been
+tested on more "exotic" varieties of UNIX).  For Windows installation
+instructions (excluding Cygwin), see windows/INSTALL.
+
+To run the example system builds, see egs/README.txt
+
+If you encounter problems (and you probably will), your first point of contact
+should be Dan Povey (dpovey@microsoft.com).  In addition to specific questions,
+please let me know if there are specific aspects of the project that you feel
+could be improved, that you find confusing, etc., and which missing features you
+most wish it had.
+

egs/README.txt

+
+This directory contains example scripts that demonstrate how to 
+use Kaldi.  Each subdirectory corresponds to a corpus that we have
+example scripts for.  Currently these are both corpora available from
+the Linguistic Data Consortium (LDC).
+
+Explanations of the corpora are below:
+
+ wsj: The Wall Street Journal corpus.  This is a corpus of read
+    sentences from the Wall Street Journal, recorded under clean conditions.
+    The vocabulary is quite large. 
+    Available from the LDC as either: [ catalog numbers LDC93S6A (WSJ0) and LDC94S13A (WSJ1) ]
+    or: [ catalog numbers LDC93S6B (WSJ0) and LDC94S13B (WSJ1) ]
+    The latter option is cheaper and includes only the Sennheiser
+    microphone data (which is all we use in the example scripts).
+
+ rm: Resource Management.  Clean speech in a medium-vocabulary task consisting
+    of commands to a (presumably imaginary) computer system.
+    Available from the LDC as catalog number LDC93S3A (it may be possible to
+    get the same data using combinations of other catalog numbers, but this
+    is the one we used).

egs/rm/README.txt

+
+Each subdirectory of this directory contains the
+scripts for a sequence of experiments.
+
+  s1: This setup is experiments with GMM-based systems with various 
+      Maximum Likelihood 
+      techniques including global and speaker-specific transforms.
+      See a parallel setup in ../wsj/s1 

egs/rm/s1/NOTES

+
+Note RE decoding beams:
+
+WER
+    Beam     20    25    30
+ monophone  18.28        28.24
+  triphone  6.767  6.724 6.724   [tri1]
+Time [on svatava, xRT]
+  triphone  0.13  0.27   0.43    [tri1]
+
+

egs/rm/s1/conf/mfcc.conf

+--use-energy=false   # only non-default option.

egs/rm/s1/conf/topo.proto

+<Topology> 
+<TopologyEntry> 
+<ForPhones>
+NONSILENCEPHONES
+</ForPhones> 
+<State> 0 <PdfClass> 0 <Transition> 0 0.75 <Transition> 1 0.25 </State> 
+<State> 1 <PdfClass> 1 <Transition> 1 0.75 <Transition> 2 0.25 </State> 
+<State> 2 <PdfClass> 2 <Transition> 2 0.75 <Transition> 3 0.25 </State> 
+<State> 3 </State>
+</TopologyEntry> 
+<TopologyEntry> 
+<ForPhones>
+SILENCEPHONES
+</ForPhones> 
+<State> 0 <PdfClass> 0 <Transition> 0 0.25 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 </State> 
+<State> 1 <PdfClass> 1 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 <Transition> 4 0.25 </State> 
+<State> 2 <PdfClass> 2 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 <Transition> 4 0.25 </State> 
+<State> 3 <PdfClass> 3 <Transition> 1 0.25 <Transition> 2 0.25 <Transition> 3 0.25 <Transition> 4 0.25 </State> 
+<State> 4 <PdfClass> 4 <Transition> 4 0.25 <Transition> 5 0.75 </State> 
+<State> 5 </State>
+</TopologyEntry> 
+</Topology> 

egs/rm/s1/data_prep/make_trans.pl

+#!/usr/bin/perl
+# Copyright 2010-2011 Microsoft Corporation
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
+# WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+# MERCHANTABLITY OR NON-INFRINGEMENT.
+# See the Apache 2 License for the specific language governing permissions and
+# limitations under the License.
+
+
+# usage:  make_trans.sh prefix in.flist input.snr out.txt out.scp
+
+# prefix is first letters of the database "key" (rest are numeric)
+
+# in.flist is just a list of filenames, probably of .sph files.
+# input.snr is an snr format file from the RM dataset.  
+# out.txt is the output transcriptions in format "key word1 word\n"
+# out.scp is the output scp file, which is as in.scp but has the
+# database-key first on each line.
+
+# Reads from first argument e.g. $rootdir/rm1_audio1/rm1/doc/al_sents.snr
+# and second argument train_wav.scp 
+# Writes to standard output trans.txt
+
+if(@ARGV != 5) {
+    die "usage:  make_trans.sh prefix in.flist input.snr out.txt out.scp\n";
+}
+($prefix, $in_flist, $input_snr, $out_txt, $out_scp) = @ARGV;
+
+open(F, "<$input_snr") || die "Opening SNOR file $input_snr";
+
+while(<F>) {
+    if(m/^;/) { next; }
+    m/(.+) \((.+)\)/ || die "bad line $_";
+    $T{$2} = $1;
+}
+
+close(F);
+open(G, "<$in_flist") || die "Opening file list $in_flist";
+
+open(O, ">$out_txt") || die "Open output transcription file $out_txt";
+
+open(P, ">$out_scp") || die "Open output scp file $out_scp";
+
+while(<G>) {
+    $_ =~ m:/(\w+)/(\w+)\.sph\s+$:i || die "bad scp line $_";
+    $spkname = $1;
+    $uttname = $2;
+    $uttname  =~ tr/a-z/A-Z/;
+    defined $T{$uttname} || die "no trans for sent $uttname";
+    $spkname =~ s/_//g; # remove underscore from spk name to make key nicer.
+    $key = $prefix . "_" . $spkname . "_" . $uttname;
+    $key =~ tr/A-Z/a-z/; # Make it all lower case.
+     # to make the numerical and string-sorted orders the same.
+    print O "$key $T{$uttname}\n";
+    print P "$key $_";
+    $n++;
+} 
+close(O) || die "Closing output.";
+close(P) || die "Closing output.";
+
+

egs/rm/s1/data_prep/run.sh

+# This script should be run from the directory where it is located (i.e. data_prep)
+# Copyright 2010-2011 Microsoft Corporation
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
+# WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+# MERCHANTABLITY OR NON-INFRINGEMENT.
+# See the Apache 2 License for the specific language governing permissions and
+# limitations under the License.
+
+
+# The input is the 3 CDs from the LDC distribution of Resource Management.
+# The script's argument is a directory which has three subdirectories:
+# rm1_audio1  rm1_audio2  rm2_audio
+
+if [ $# != 1 ]; then
+   echo "Usage: ./run.sh /path/to/RM"
+   exit 1; 
+fi 
+
+RMROOT=$1
+if [ ! -d $RMROOT/rm1_audio1 -o ! -d $RMROOT/rm1_audio2 ]; then
+  echo "Error: run.sh requires a directory argument that contains rm1_audio1 and rm1_audio2"
+  exit 1; 
+fi  
+
+if [ ! -d $RMROOT/rm2_audio ]; then
+  echo "**Warning: $RMROOT/rm2_audio does not exist; won't create spk2gender.map file correctly***"
+  sleep 1
+fi  
+
+(
+  find $RMROOT/rm1_audio1/rm1/ind_trn -iname '*.sph';
+  find $RMROOT/rm1_audio2/2_4_2/rm1/ind/dev_aug -iname '*.sph';
+) | perl -ane ' m:/sa\d.sph:i || m:/sb\d\d.sph:i || print; '  > train_sph.flist
+
+
+
+# make_trans.pl also creates the utterance id's and the kaldi-format scp file.
+./make_trans.pl trn train_sph.flist $RMROOT/rm1_audio1/rm1/doc/al_sents.snr train_trans.txt train_sph.scp
+mv train_trans.txt tmp; sort -k 1 tmp > train_trans.txt
+mv train_sph.scp tmp; sort -k 1 tmp > train_sph.scp
+
+sph2pipe=`cd ../../../..; echo $PWD/tools/sph2pipe_v2.5/sph2pipe`
+if [ ! -f $sph2pipe ]; then
+   echo "Could not find the sph2pipe program at $sph2pipe";
+   exit 1;
+fi
+awk '{printf("%s '$sph2pipe' -f wav %s |\n", $1, $2);}' < train_sph.scp > train_wav.scp
+
+cat train_wav.scp | perl -ane 'm/^(\w+_(\w+)\w_\w+) / || die; print "$1 $2\n"' > train.utt2spk
+cat train.utt2spk | sort -k 2 | ../scripts/utt2spk_to_spk2utt.pl > train.spk2utt
+
+
+for ntest in 1_mar87 2_oct87 4_feb89 5_oct89 6_feb91 7_sep92; do
+   n=`echo $ntest | cut -d_ -f 1`
+   test=`echo $ntest | cut -d_ -f 2`
+   root=$RMROOT/rm1_audio2/2_4_2
+   for x in `grep -v ';' $root/rm1/doc/tests/$ntest/${n}_indtst.ndx`; do
+      echo "$root/$x ";
+  done > test_${test}_sph.flist
+done
+
+# make_trans.pl also creates the utterance id's and the kaldi-format scp file.
+for test in mar87 oct87 feb89 oct89 feb91 sep92; do
+  ./make_trans.pl ${test} test_${test}_sph.flist $RMROOT/rm1_audio1/rm1/doc/al_sents.snr test_${test}_trans.txt test_${test}_sph.scp
+   mv test_${test}_trans.txt tmp; sort -k 1 tmp > test_${test}_trans.txt
+   mv test_${test}_sph.scp tmp; sort -k 1 tmp > test_${test}_sph.scp
+
+  awk '{printf("%s '$sph2pipe' -f wav %s |\n", $1, $2);}' < test_${test}_sph.scp  > test_${test}_wav.scp
+
+  cat test_${test}_wav.scp | perl -ane 'm/^(\w+_(\w+)\w_\w+) / || die; print "$1 $2\n"' > test_${test}.utt2spk
+  cat test_${test}.utt2spk | sort -k 2 | ../scripts/utt2spk_to_spk2utt.pl > test_${test}.spk2utt
+done
+
+cat $RMROOT/rm1_audio2/2_5_1/rm1/doc/al_spkrs.txt \
+ $RMROOT/rm2_audio/3-1.2/rm2/doc/al_spkrs.txt | \
+ perl -ane 'tr/A-Z/a-z/;print;' | grep -v ';' | \
+     awk '{print $1, $2}' > spk2gender.map
+
+../scripts/make_rm_lm.pl $RMROOT/rm1_audio1/rm1/doc/wp_gram.txt  > G.txt 
+
+# Getting lexicon
+../scripts/make_rm_dict.pl  $RMROOT/rm1_audio2/2_4_2/score/src/rdev/pcdsril.txt > lexicon.txt
+
+echo Succeeded.

egs/rm/s1/data_prep/sph2wav.sh

+#!/bin/bash
+# Copyright 2010-2011 Microsoft Corporation
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
+# WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+# MERCHANTABLITY OR NON-INFRINGEMENT.
+# See the Apache 2 License for the specific language governing permissions and
+# limitations under the License.
+
+
+fake=false
+if [ "$1" == "--fake" ]; then
+    fake=true
+    shift
+fi
+
+sphdir=$1 # e.g. /mnt/matylda2/data/RM
+wavdir=$2 # e.g. /mnt/matylda6/jhu09/qpovey/kaldi_rm_wav
+flistin=$3 # e.g. train_sph.flist, contains sph files in sphdir
+flistout=$4 # e.g. train_wav.flist, contains wav files in wavdir
+
+
+if [ $fake == false ]; then
+    for x in `cat $flistin`; do 
+        y=`echo $x | sed s:$sphdir:$wavdir: | sed s:.sph:.wav:`;
+        mkdir -p `dirname $y`
+        ../../tools/sph2pipe_v2.5/sph2pipe -f wav $x $y || exit 1;
+    done 
+fi
+
+cat $flistin | sed s:$sphdir:$wavdir: | sed s:.sph:.wav: > $flistout || exit 1;
+

egs/rm/s1/path.sh

+export PATH=$PATH:../../../src/bin:../../../tools/openfst/bin:../../../src/fstbin/:../../../src/gmmbin/:../../../src/featbin/:../../../src/fgmmbin:../../../src/sgmmbin

egs/rm/s1/run.sh

+#!/bin/bash
+
+# Copyright 2010-2011 Microsoft Corporation
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
+# WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+# MERCHANTABLITY OR NON-INFRINGEMENT.
+# See the Apache 2 License for the specific language governing permissions and
+# limitations under the License.
+
+exit 1 # Don't run this... it's to be run line by line from the shell.
+
+# This script file cannot be run as-is; some paths in it need to be changed
+# before you can run it.
+# Search for /path/to.
+# It is recommended that you do not invoke this file from the shell, but
+# run the paths one by one, by hand.
+
+# the step in data_prep/ will need to be modified for your system.
+
+# First step is to do data preparation:
+# This just creates some text files, it is fast.
+# If not on the BUT system, you would have to change run.sh to reflect
+# your own paths.
+#
+
+#Example arguments to run.sh: /mnt/matylda2/data/RM, /ais/gobi2/speech/RM, /cygdrive/e/data/RM
+# RM is a directory with subdirectories rm1_audio1, rm1_audio2, rm2_audio
+cd data_prep
+#*** You have to change the pathname below.***
+./run.sh /path/to/RM
+cd ..
+
+
+mkdir -p data
+( cd data; cp ../data_prep/{train,test*}.{spk2utt,utt2spk} . ; cp ../data_prep/spk2gender.map . )
+
+# This next step converts the lexicon, grammar, etc., into FST format.
+steps/prepare_graphs.sh
+
+
+# Next, make sure that "exp/" is someplace you can write a significant amount of
+# data to (e.g. make it a link to a file on some reasonably large file system).
+# If it doesn't exist, the scripts below will make the directory "exp".
+
+# tempdir should be set to some place to put training mfcc's
+# where you have space.
+#e.g.: tempdir=/mnt/matylda6/jhu09/qpovey/kaldi_rm_mfccb
+mfccdir=/path/to/mfccdir
+steps/make_mfcc_train.sh $mfccdir
+steps/make_mfcc_test.sh $mfccdir
+
+steps/train_mono.sh    
+steps/decode_mono.sh  &
+steps/train_tri1.sh
+steps/decode_tri1.sh  &
+
+steps/train_tri2a.sh
+steps/decode_tri2a.sh  &
+
+# Then do the same for 2b, 2c, and so on
+# 2a = basic triphone (all features double-deltas unless stated).
+# 2b = exponential transform
+# 2c = mean normalization (cmn)
+# 2d = MLLT
+# 2e = splice-9-frames + LDA
+# 2f = splice-9-frames + LDA + MLLT
+# 2g = linear VTLN (+ regular VTLN); various decode scripts available.
+# 2h = splice-9-frames + HLDA
+# 2i = triple-deltas + HLDA
+# 2j = triple-deltas + LDA + MLLT
+# 2k = LDA + ET (equiv to LDA+MLLT+ET)
+
+
+# To train and test SGMM systems:
+
+steps/train_ubma.sh
+
+# train and test unadapted system
+steps/train_sgmma.sh
+steps/decode_sgmma.sh
+
+# train and test system with speaker vectors.
+steps/train_sgmmb.sh
+steps/decode_sgmmb.sh
+
+
+
+

egs/rm/s1/scripts/add_disambig.pl

+#!/usr/bin/perl
+# Copyright 2010-2011 Microsoft Corporation
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
+# WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+# MERCHANTABLITY OR NON-INFRINGEMENT.
+# See the Apache 2 License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Adds some specified number of disambig symbols to a symbol table.
+# Adds these as #1, #2, etc.
+# If the --include-zero option is specified, includes an extra one
+# #0.
+if(!(@ARGV == 2 || (@ARGV ==3 && $ARGV[0] eq "--include-zero"))) {
+    die "Usage: add_disambig.pl [--include-zero] symtab.txt num_extra > symtab_out.txt ";
+}
+
+if(@ARGV  == 3) {
+    $include_zero = 1;