.. title: Files and string manipulation reminder
.. slug: files-string-manipulation-reminder
.. date: 2017-12-25 18:25:52 UTC+01:00
.. tags: bash, fichiers, chaînes de caractères
.. category: blog
.. link: 
.. description: 
.. type: text

One of the strength of Unix-like systems are files and strings manipulation that are pretty straightforward with no other softwares required. It is not always easy to remember all the useful commands, hence this reminder.

.. TEASER_END

Play with string in ``bash``
============================

It is easy to simply manipulate string in ``bash`` with the following:

- ``${var: :n}``, to get the ``n`` first characters of a string
- ``${var: n}``, to get characters from the ``n`` character in the string
- ``${var: -n}``, to get the last ``n`` characters of the string
- ``${var: n:m}``, to get characters between positions ``n`` and ``m``.

Here is a simple example where data are copied from one serie of directories to another where only a part of the name is changing:

.. code-block:: bash

    for i in $(ls rep_s1*}; do cp $i/{data1, data2} rep_s2${i: 5}; done

In the same range of ideas, you can easily get what is before or after a given pattern, typically a file extension:

- ``${fname##*.}`` to get a file extension
- ``${fname%.*}`` to get the content before the extension
- ``${fname%%.*}`` for a *greedy* version.
    

Transpose lines and columns in shell script
===========================================

The following script can be improved but definitely works. It takes a file as input where columns are separeted by spaces.

.. code-block:: bash

    #!/bin/bash

    file=$1
    
    # get the number of columns
    num=$(awk -F" " 'NR==1 { print NF }' $file)
    echo $num
    
    # transpose
    i=1
    while (( $i <= $num ))
    do
        newline=''
        for val in $(cut -d" " -f$i $file)
        do
        newline=$newline$val" "
        done
        nline=`echo ${newline%?}`
        echo $nline >> tmpdata
        (( i = i + 1 ))
    done
    mv tmpdata test.dat


Else, if the file contains a sole column, an elegant solution is to use ``tr``:

.. code-block:: bash

    ~$ tr -s '\n' ' ' < [oldfile] > [newfile] 


Retrieve part of a ``path``
===========================

Here are three different possibilities:

- the "I'm learning ``bash``" way:

.. code-block:: bash

    function RootDir(){
        listdir=($(echo $1 | tr '/' ' '))
        length=${#listdir[@]}
        length=$(printf $(($length - 3)))
        DIR=
        i=0
        while [ $i -le $length ]; do
            DIR=$DIR/${listdir[$i]}
            i=$(($i + 1))
        done
        echo $DIR
    }

    newpath=$(RootDir $DIR)

- the "I've just discovered ``dirname``" way:

.. code-block:: bash

    newpath=$(dirname $(dirname $DIR))

- and, lastly, the cleaver way:
 
.. code-block:: bash

   newpath=${$DIR%/*/*}


Fill a number with zeros in ``bash`` script
===========================================

Here is a one-liner to rename massively files indexing them, using ``bc``:

.. code-block:: bash

    j=1; for i in $(ls *png); do cp $i slice_yz-`printf %02d $j`.png; j=$(echo "$j+1" | bc); done 


Rename files with leading zeros
===============================

It can be useful to manipulate files indexed as ``001``, ``010``, ``100``. It is possible using ``rename`` (on Debian-*like* distributions), with regular expressions:

.. code-block:: bash

    rename 's/\-([1-9]\.)/\-0$1/g' * 

One can also use ``printf``:

.. code-block:: bash

    for i in $(ls -d *); do for j in $(more $i/list.dat); do scp -r user@remote:/path/$i/output_`printf %05d $j` $i/.; done; done 

Beware of the single quotes that are ````` and not ``'``! 


Merge files
===========

It is possible to merge files with ``awk`` selecting columns:

.. code-block:: bash

    pr -m -t -s\ file1 file2 | awk '{print($1 $5 $3)}' > fileout

The ``pr -m -t -s\ `` (don't forget the space after ``-s`` option) can be replaced by ``paste``. 


Convert music files
===================

One should favor lossless formats, however all media players don't read these formats, and sometimes these files can be quite big. File conversion is easy with ``lame``.


wave/MP3 conversion
-------------------
    
.. code-block:: bash

    for file in *.wav; do lame -h -b [bitrate] "$file" "${file%.wav}.mp3"; done 


FLAC/MP3 conversion
-------------------
    
.. code-block:: bash

    for file in *.flac; do $(flac -cd "$file" | lame -h -b [bitrate] - "${file%.flac}.mp3"); done