Files and string manipulation reminder

Play with string in bash

It is easy to simply manipulate string in bash with the following:

• ${var: :n}, to get the n first characters of a string
• ${var: n}, to get characters from the n character in the string
• ${var: -n}, to get the last n characters of the string
• ${var: n:m}, to get characters between positions n and m.

Here is a simple example where data are copied from one serie of directories to another where only a part of the name is changing:

for i in $(ls rep_s1*}; do cp $i/{data1, data2} rep_s2${i: 5}; done

In the same range of ideas, you can easily get what is before or after a given pattern, typically a file extension:

• ${fname##*.} to get a file extension
• ${fname%.*} to get the content before the extension
• ${fname%%.*} for a greedy version.

Transpose lines and columns in shell script

The following script can be improved but definitely works. It takes a file as input where columns are separeted by spaces.

#!/bin/bash

file=$1

# get the number of columns
num=$(awk -F" " 'NR==1 { print NF }' $file)
echo $num

# transpose
i=1
while (( $i <= $num ))
do
    newline=''
    for val in $(cut -d" " -f$i $file)
    do
    newline=$newline$val" "
    done
    nline=`echo ${newline%?}`
    echo $nline >> tmpdata
    (( i = i + 1 ))
done
mv tmpdata test.dat

Else, if the file contains a sole column, an elegant solution is to use tr:

~$ tr -s '\n' ' ' < [oldfile] > [newfile]

Retrieve part of a path

Here are three different possibilities:

• the "I'm learning bash" way:

function RootDir(){
    listdir=($(echo $1 | tr '/' ' '))
    length=${#listdir[@]}
    length=$(printf $(($length - 3)))
    DIR=
    i=0
    while [ $i -le $length ]; do
        DIR=$DIR/${listdir[$i]}
        i=$(($i + 1))
    done
    echo $DIR
}

newpath=$(RootDir $DIR)

• the "I've just discovered dirname" way:

newpath=$(dirname $(dirname $DIR))

• and, lastly, the cleaver way:

newpath=${$DIR%/*/*}

Fill a number with zeros in bash script

Here is a one-liner to rename massively files indexing them, using bc:

j=1; for i in $(ls *png); do cp $i slice_yz-`printf %02d $j`.png; j=$(echo "$j+1" | bc); done

Rename files with leading zeros

It can be useful to manipulate files indexed as 001, 010, 100. It is possible using rename (on Debian-like distributions), with regular expressions:

rename 's/\-([1-9]\.)/\-0$1/g' *

One can also use printf:

for i in $(ls -d *); do for j in $(more $i/list.dat); do scp -r user@remote:/path/$i/output_`printf %05d $j` $i/.; done; done

Beware of the single quotes that are ` and not '!

Merge files

It is possible to merge files with awk selecting columns:

pr -m -t -s\ file1 file2 | awk '{print($1 $5 $3)}' > fileout

The pr -m -t -s\ `` (don't forget the space after ``-s option) can be replaced by paste.

Convert music files

One should favor lossless formats, however all media players don't read these formats, and sometimes these files can be quite big. File conversion is easy with lame.

for file in *.wav; do lame -h -b [bitrate] "$file" "${file%.wav}.mp3"; done

for file in *.flac; do $(flac -cd "$file" | lame -h -b [bitrate] - "${file%.flac}.mp3"); done