Part 7: Learning with Linux

7.1 Linux Advanced Text Processing Tools

script
Log my current session in the text terminal into a text file typescript (the default filename). The log finishes when I type exit or press <Ctrl>d.

emacs&
(in X-terminal) The emacs text editor. Advanced and sophisticated text editor. Seems for gurus only: "emacs is not just an editor, it is a way of living". Emacs surely seems rich or bloated, depending on your point of view. There are likely 3 versions of emacs installed on your system: (1) text-only: type emacs in a text (not X-windows) terminal (I avoid this like fire); (2) graphical-mode: type emacs in an X-windows terminal (fairly usable even for a newbie if you take some time to learn it); and (3) X-windows mode: type "xemacs" in an X-windows terminal. 

vi
The famous (notorious?) "vi" text editor (definitely not recommended for newbies). To exit "vi" (no changes saved) use these five characters: <ESC>:q!<Enter> I use the "kate&"  (under X) or "pico" (command line) or "nano" (command line) text editors and don't ever need vi (well, unless I have to unmount the /usr subsystem and modify/edit some configuration files, then vi is the only editor avialable). To be fair, modern Linux distributions use vim (="vi improved") in place of vi, and vim is somewhat better than the original vi. The GUI version of vi is also available (type gvim in an X terminal). Here is one response I have seen to the criticism of vi interface being not "intuitive":  "The only intuitive interface is the nipple. The rest must be learned."  (Well, so much for MS Windows being an "intuitive" interface.)

Experts do like vi, but vi is definitely difficult unless you use it very often. Here is a non-newbie opinion on vi (http://linuxtoday.com/stories/16620.html):

"I was first introduced to vi in 1988 and I hated it. I was a freshman in college... VI seemed archaic, complicated and unforgiving... It is now 12 years later and I love vi, in fact it is almost the only editor I use. Why the change? I actually learned to use vi... Now I see vi for what it really is, a powerful, full featured, and flexible editor..."

nano
This is a brand new (March 2001) GNU replacement for pico. Works and looks like pico, but it is smaller, better, and licenced as expected for a decent piece of Linux software (i.e., General Public Licence, GPL).  Not included with RH7.0 or MDK7.2, but expect it soon.

khexedit
(in X terminal) Simple hexadecimal editor. Another hexadecimal editor is hexedit (text based, less user friendly).  Hex editors are used for editing binary (non-ASCII) files.

diff file1 file2 > patchfile
Compare contents of two files and list any differences. Save the output to the file patchfile.

sdiff file1 file2
Side-by-side comparison of two text files. Output goes to the "standard output" which normally is the screen.

patch file_to_patch patchfile
Apply the patch (a file produced by diff, which lists differences between two files) called patchfile to the file file_to_patch. If the patch was created using the previous command, I would use: patch file1 patchfile to change file1 to file2.

grep filter
Search content of text files for matching patterns. It is definitely worth learning at least the basics of this command.

A simple example. The command:

cat * | grep my_word | more

will search all the files in the current working directory (except files starting with a dot) and print the lines which contain the string "my_word".

A shorter form to achieve the same may be:

grep my_word * |more

The patterns are specified using a powerful and standard notation called "regular expressions".

There is also a "recursive" version of grep called rgrep. This will search all the files in the current directory and all its subdirectories for my_word and print the names of the files and the matching line:

rgrep -r my_word . | more

In regular expressions, most characters just match themselves. So to search for string "peter", I would just use a searchstring "peter". The exceptions are so-called "special characters" ("metacharacters"), which have special meaning.

The regexpr special characters are:  "\" (backslash), "." (dot), "*" (asterisk), "[" (bracket), "^" (caret, special only at the beginnig of a string), "$" (dollar sign, special only at the end of a string). A character terminating a pattern string is also special for this string.

The backslash, "\" is used as an "escape" character, i.e., to quote a subsequent special character.
Thus, "\\" searches for a backslash, "\." searches for a dot, "\*" searches for the asterisk, "\[" searches for the bracket, "\^" searches for the caret even at the begining of the string, "\$" searches for the dollar sign even at the end of the string.

Backslash followed by a regular (non-special) character may gain a special meaning. Thus, the symbols \<  and  \>  match  an  empty string at the beginning and the end of a word, respectively.  The symbol  \b  matches  the empty  string  at the edge of a word, and \B matches the empty string provided it's not at the edge of a word.

The dot, ".", matches any single character. [The dir command uses "?" in this place.] Thus, "m.a" matches "mpa" and "mea" but not "ma" or "mppa".

Any string is matched by ".*" (dot and asterisk). [The dir command uses "*" instead.]  In general, any  pattern followed by "*" matches zero or more occurences of this pattern. Thus, "m*" matches zero or more occurances of "m". To search for one or more "m", I could use "mm*".

The * is a repetition operator. Other repetition operators are used less often--here is the full list:
*       the proceding item is to be matched zero or more times;
\+      the preceding item is to be matched one or more times;
\?      the preceding item is  optional and matched at most once;
\{n}    the preceding item is to be matched exactly n times;
\{n,}   the preceding item is to be matched n or more times;
\{n,m}  the  preceding item is to be matched at least n times, but not more than m times.

The caret, "^", means "the beginning of the line". So "^a" means "find a line starting with an "a".

The dollar sign, "$", means "the end of the line". So "a$" means "find a line ending with an "a".

Example. This  command searches the file myfile for lines starting with an "s" and ending with an "n", and prints them to the standard output (screen):

cat myfile | grep '^s.*n$'

Any character terminating the pattern string is special, precede it with a backslash if you want to use it within this string.

The bracket, "[" introduces a set.  Thus [abD] means: either a or b or D. [a-zA-C] means any character from a to z or from A to C.

Attention with some characters inside sets. Within a set, the only special characters are "[", "]", "-", and "^", and the combinations "[:", "[=", and "[.". The backslash is not special within a set.

Useful categories of characters are (as definded by the POSIX standard): [:upper:]  =upper-case letters, [:lower:] =lower-case letters, [:alpha:]  =alphabetic (letters) meaning upper+lower, [:digit:] =0 to 9, [:alnum:] =alphanumeric meaning alpha+digits, [:space:] =whitespace meaning <Space>+<Tab>+<Newline> and similar, [:graph:] =graphically printable characters except space, [:print:] =printable characters including space, [:punct:] =punctuation characters meaning graphical characters minus alpha and digits, [:cntrl:] =control characters meaning non-printable characters, [:xdigit:] = characters that are hexadecimal digits.

Example. This command scans the output of the dir command, and prints lines containing a capital letter followed by a digit:

dir -l | grep '[[:upper:]][[:digit:]]'

Example :

cat my_file | tr 1 2 > new_file

This command takes the content of the file my_file, pipes it to the translation utility tr,  the tr utility replaces all instances of the character "1" with "2", the output from the process is directed to the file new_file.

For example, to print lines containing the string "1024", I may use:
cat filename | sed -n '/1024/p'

Here, sed filters the output from the cat command. The option "-n" tells sed to block all the incoming lines but those explicitly matching my expression.  The sed action on a match is "p"= print.

Another example, this time for deleting selected lines:

cat filename | sed '/.*o$/d' > new_file

In this example, lines ending the an "o" will be deleted. I used a regular expression for matching any string followed by an "o" and the end of the line. The output (i.e., all lines but those ending with "d") is directed to new_file.

Another example. To search and replace, I use the sed 's' action, which comes in front of two expressions:

cat filename | sed 's/string_old/string_new/' > newfile

A shorter form for the last command is:
sed 's/string_old/string_new/' filename > newfile

To insert a text from a text file into an html file, I may use a script containing:
sed '/text_which_is_a_placeholder_in__my_html_file/r text_file_to_insert.txt' index_master_file.html > index.htmll

gawk is particularly suitable for processing text-based tables. A table consists of records (each line is normally one record). The records contain fields separated by a delimiter. Often used delimiters are whitespace (gawk default), comma, or colon. All gawk expressions have a form:  gawk 'pattern {action}' my_file.  You can ommit the patern or action: the default pattern is "match everything" and the default action is "print the line". gawk can also be used as a filter (to process the output from another command, as used in our examples).

Example. To print lines containing the string "1024", I may use:
cat filename | gawk '/1024/ {print}'
Like in sed, the patterns to match are enclosed in a pair of "/  /".

What makes gawk more powerful than sed is the operations on fields. $1 means "the first field", $2 means "the second field", etc. $0 means "the entire line". The next example extracts fields 3 and 2 from lines containing "1024" and prints them with added labels "Name" and "ID". The printing goes to a file called "newfile":
cat filename | gawk '/1024/ {print "Name: " $3 "ID: " $2}' > newfile

The third example finds and prints lines with the third field equal to "peter" or  containing the string "marie":
cat filename | gawk '$3 == "peter" || $3 ~ /marie/ '
To understand the last command, here is the list of logical tests in gawk:  == equal,  != not equal,  < less than,  > greater than, <= less than or equal to,  >=  greater than or equal to, ~ matching a regular expression,  !~  not matching a regular expression,  ||  logical OR, &&  logical AND,  ! logical NOT.

cervisia
(in X-terminal). A GUI front-end to the cvs versioning system.

file -z filename
Determine the type of the file filename. The option -z makes file look also inside compressed files to determine what the compressed file is (instead of just telling you that this is a compressed file).

To determine the type of content, file looks inside the file to find particular patterns in contents ("magic numbers")--it does not just look at the filename extension like MS Windows does. The "magic numbers" are stored in the text file /usr/share/magic--really impressive database of filetypes.

There are three date/time values associated with every file on an ext2 filesystem:
- the time of last access to the file (atime)
- the time of last modification to the file (mtime)
- the time of last change to the file's inode (ctime).
Touch will change the first two to the value specified, and the last one always to the current system time. They can all be read using the stat command (see the next entry).

strings filename | more
Display the strings contained in the binary file called filename. "strings" could, for example, be a useful first step to a close examination of an unknown executable.

od
(=octal dump). Display contents as octal numbers. This can be useful when the output contains non-printable characters. For example, a filename may contain non-printable characters and be a real pain. This can also be handy to view binary files.

Examples:
dir | od -c | more

(I would probably rather do: ls -b to see any non-printable characters in filenames).

cat my_file | od -c |more
od my_file |more

Comparison of different outputs:
Show 16 first characters from a binary (/bin/sh) as ASCII characters or backslash escapes (octal):
od -N 16 -c /bin/sh
output:
0000000 177   E   L   F 001 001 001  \0  \0  \0  \0  \0  \0  \0  \0  \0
Show the same binary as named ASCII characters:
od -N 16 -a /bin/sh
output:
0000000 del   E   L   F soh soh soh nul nul nul nul nul nul nul nul nul
Show the same binary as short hexcadecimals:
od -N 16 -t x1 /bin/sh
output:
0000000 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Show the same binary as octal numbers:
od -N 16 /bin/sh
output:
0000000 042577 043114 000401 000001 000000 000000 000000 000000

Examples:
dir | wc
cat my_file | wc
wc myfile

md5sum filename
Compute a md5 checksum (128-bit) for file filename to verify its integrity.

mkpasswd -l 10
Make a hard-to-guess, random password of the length of 10 characters.

sort -f filename
Arrange the lines in filename according to the ascii order. The option -f tells sort to ignore the upper and lower character case. The ascii character set is (see man ascii):

Dec  Hex  Char          Dec  Hex  Char     Dec  Hex  Char    Dec   Hex  Char
---------------------------------------------------------------------------
 0    00   NUL '\0'     32   20   SPACE     64   40   @       96    60   `
 1    01   SOH          33   21   !         65   41   A       97    61   a
 2    02   STX          34   22   "         66   42   B       98    62   b
 3    03   ETX          35   23   #         67   43   C       99    63   c
 4    04   EOT          36   24   $         68   44   D       100   64   d
 5    05   ENQ          37   25   %         69   45   E       101   65   e
 6    06   ACK          38   26   &         70   46   F       102   66   f
 7    07   BEL '\a'     39   27   '         71   47   G       103   67   g
 8    08   BS  '\b'     40   28   (         72   48   H       104   68   h
 9    09   HT  '\t'     41   29   )         73   49   I       105   69   i
 10   0A   LF  '\n'     42   2A   *         74   4A   J       106   6A   j
 11   0B   VT  '\v'     43   2B   +         75   4B   K       107   6B   k
 12   0C   FF  '\f'     44   2C   ,         76   4C   L       108   6C   l
 13   0D   CR  '\r'     45   2D   -         77   4D   M       109   6D   m
 14   0E   SO           46   2E   .         78   4E   N       110   6E   n
 15   0F   SI           47   2F   /         79   4F   O       111   6F   o
 16   10   DLE          48   30   0         80   50   P       112   70   p
 17   11   DC1          49   31   1         81   51   Q       113   71   q
 18   12   DC2          50   32   2         82   52   R       114   72   r
 19   13   DC3          51   33   3         83   53   S       115   73   s
 20   14   DC4          52   34   4         84   54   T       116   74   t
 21   15   NAK          53   35   5         85   55   U       117   75   u
 22   16   SYN          54   36   6         86   56   V       118   76   v
 23   17   ETB          55   37   7         87   57   W       119   77   w
 24   18   CAN          56   38   8         88   58   X       120   78   x
 25   19   EM           57   39   9         89   59   Y       121   79   y
 26   1A   SUB          58   3A   :         90   5A   Z       122   7A   z
 27   1B   ESC          59   3B   ;         91   5B   [       123   7B   {
 28   1C   FS           60   3C   <         92   5C   \  '\\' 124   7C   |
 29   1D   GS           61   3D   =         93   5D   ]       125   7D   }
 30   1E   RS           62   3E   >         94   5E   ^       126   7E   ~
 31   1F   US           63   3F   ?         95   5F   _       127   7F   DEL

If you wondered about the control characters, here is the meaning of some of them on the console (Source: man console_codes). Each line below gives the code mnemonics, its ASCII decimal number, the key combination to produce the code on the console, and a short description:
BEL (7, <Ctrl>G) bell (=alarm, beep).
BS  (8, <Ctrl>H) backspaces one column (but not past the beginning of the line).
HT  (9, <Ctrl>I) horizonal tab, goes to the next tab stop or to the end of the line if there is no earlier tab stop.
LF (10, <Ctrl>J), VT (11, <Ctrl>K) and FF (12, <Ctrl>L) all three give a linefeed.
CR (13, <Ctrl>M) gives a carriage return.
SO (14, <Ctrl>N) activates the G1 character set, and if LF/NL (new line mode) is set also a carriage return.
SI (15, <Ctrl>O) activates the G0 character set.
CAN (24, <Ctrl>X) and SUB (26, <Ctrl>Z) interrupt escape sequences.
ESC (27, <Ctrl>[) starts an escape sequence.
DEL (127) is ignored.
CSI (155) control sequence introducer.

fold -w 30 -s my_file.txt > new_file.txt
Wrap the lines in the text file my_file.txt so that there is 30 characters per line. Break the lines on spaces. Output goes to new_file.txt.

fmt -w 75 my_file.txt > new_file.txt
Format the lines in the text file to the width of 75 characters.  Break long lines and join short lines as required, but don't remove empty lines.

nl myfile > myfile_lines_numbered
Number the lines in the file myfile. Put the output to the file myfiles_lines_numbered.

indent -kr -i8 -ts8 -sob -l80 -ss -bs -psl "$@" *.c
Change the appearance of "C" source code by inserting or deleting white space. The formatting options in the above example conform to the style used in the Linux kernel source code (script /usr/src/linux/scripts/Lindent). See man indent for the description of the meaning of the options. The existing files are backed up and then replaced with the formatted ones.

rev filename > filename1
Print the file  filename, each line in reversed order. In the example above, the output is directed to the file filename1.

shred filename
Repeatedly overwrite the contents of the file filename with garbage, so that nobody will ever be able to read its original contents again.

paste file1 file2 > file3
Merge two or more text files on lines using <Tab> as delimiter (use option "d=" to specify your own delimiter(s).

Example. If the content of file1 was:
1
2
3
and file2 was:
a
b
c
d
the resulting file3 would be:
1    a
2    b
3    c
     d

Example. if the content of file1 was:
1 Barbara
2 Peter
3 Stan
4 Marie
and file2 was:
2 Dog
4 Car
7 Cat
the resulting file3 would be:
2 Peter Dog
4 Marie Car

des -e plain_file encrypted_file
(="Data Encryption Standard") Encrypt plain_file. You will be ask for a key that the program will use for encryption. Output goes to encrypted_file. To decrypt use
des -d encrypted_file decrypted_file.

gpg
"Gnu Privacy Guard"--a free equivalent of PGP ("Pretty Good Privacy"). gpg is more secure than PGP and does not use any patented algorithms. gpg is mostly used for signing your e-mail messages and checking signatures of others. You can also use it to encrypt/decrypt messages. http://www.gnupg.org/ contains all the details, including a legible, detailed manual.

To start, I needed a pair of keys: private and public. The private key is used for signing my messages. The public key I give away so that others can use it to verify my signatures. [One can also use a public key to encrypt a message so it can only be read using my private key.] I generated my keypair using this command:
gpg --gen-key

My keys are stored in the directory ~/.gnupg (encrypted using a passphrase I supplied during the key generation). To list my public key in plain text file, I use:
gpg --armor --export my_email_address > public_key_stan.gpg

which created a file public_key_stan.gpg containing something like this:

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.0.1 (GNU/Linux)
Comment: For info see http://www.gnupg.org

mQGiBDmnzEYRBACoN438rxANaMfCy5bfj6KWM0/TR6x6HZ0gpmhGeuouM/SOR2IU
/G30NdCuzHeFs93BhtY0IdzoEMtMyZHnvdhZC2bx/jhgaaMbEaSsXwRhVB0xVPYx
rHbsgSULHYzRFF34MS3/Lse3QWfWxzA7I0lbXB7nLwZKZqaNONRFRR42owCg60hV
TDPEB2N0llMyt12R4ZByFSsEAJ1tE7pb9b6TP7cw21vkIjc+BI2uSzn/B4vNlCWK
TTuZHVv0w0jFcbd8DB0/1tlZUOrIzLSqJyQDGiNn258+7LetQ+LKG/1YKbiAcosz
4QirBuLIeF2M9GuXYCwZypE3Dwv+4YupvybR31CgLTJ8p4sKqC5n0eSr2oSrtdHZ
yuJtA/9v2HcebOncfCNOK+cVRmcTB1Frl/Gh/vNCfeZyXaJxlqDfCU2vJHtBemiE
AtcfZHB/iHy0DM68LfRJSAIFAa5um9iWHh5/vWCGZLqtpwZ7kyMw+2D6CFkWATsy
wQA1g1VcGkNc14Crrd36qf60bI+b8pn2zDhwZtLsELsXyXkNhbQmU3RhbiBKIEts
aW1hcyA8U3RhbktsaW1hc0B3ZWJoYXJ0Lm5ldD6IVgQTEQIAFgUCOafMRgQLCgQD
AxUDAgMWAgECF4AACgkQt+ZBooH8bHd2kwCghAt9aKIk0mRJv+g7YcRPotVtrwkA
n1a4xEVEyaKgKoMaJnopf69K9+vouQENBDmnzH4QBADgFpLP+tWZPnVYg47cn+9b
XQRjdOtNsDE6BYH872/sR1oCrdH6k+gXFOiZxRZ3PElK2/olo59kh5xa9aBxNdEC
FuXJN0UelmhOFbDtqVksIqVWyYfXnLz+wtcXg0Q0L0q8vY4IuTzw2WkV6EkM+/x8
6UhA2XVaMJKBdRKFSVilbwADBQP+JCzLj5HDgpRvf+KM72nzSg7sp8Tki7nF9wNA
PODK0SeQgI3dwXYyF6AVenlETE/3xRWoYQN1bxVZsOex9vzqPrQC3dR0NBljd74r
kfXwUTl2fNQX4N9iuVCo2gCGbi5+gfEk1GhsWDsq0z40f+18k+XBdWmY8sCNiolT
tnvm1QeIRgQYEQIABgUCOafMfgAKCRC35kGigfxsd9SGAJ9/FWSkEfgbE/Yc46d8
Ef1gYg3I1ACff3oLeAMeGGO79gW6UGp9RJ6mRao=
=X1k2
-----END PGP PUBLIC KEY BLOCK-----

Now, I can e-mail my public key to the people with whom I want to communicate securely. They can store it on their pgp system using;
gpg --import public_key_stan.gpg

Even better, I can submit my public key to a public key server. To find a server near me, I used:
host -l pgp.net | grep wwwkeys

and to submit the key, I did (can take a couple of minutes, and I am connected to the Internet):
gpg --keyserver wwwkeys.pgp.net --send-keys linux_nag@canada.com

The "wwwkeys.pgp.net" is the key server I selected, and "linux_nag@canada.com" is my email address that identifies me on my local key ring. I need to submit myself only to one public key server (they all synchronize).

Now, I can start using gpg. To manually sign a plain text file my_message, I could use:
gpg --clearsign my_message

This created file my_message.asc which may contain something like:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello World!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.1 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE5p9+3t+ZBooH8bHcRApn/AJ9kx9+pU3GJBuvJN9Bo3bW3ku/5PwCgquht
mfrPrt7PQtdmGox72jkY0lo=
=rtK0
-----END PGP SIGNATURE-----

To verify a signed message, I could do:
gpg --verify my_message.asc

If the contents of the signed section in my_message.asc was even slightly modified, the signature will not check.

Manual signing can be awkward. But, for example, kmail can sign the electronic signatures automatically for me.

"docbook" tools
Docbook is the incoming standard for document depository. The docbooks tools are included with RH6.2 (and later) in  the package "jade" and include the following converters: db2ps, db2pdf, db2dvi, db2html, db2rtf which convert docbook files to: postscript (*.ps), Adobe Portable Document Format (*.pdf), device independent file format (*.dvi), HyperText Markup Language (*.html), and Rich Text Format (*.rtf), respectively.

"Document depository" means the document is in a format that can be automatically translated into other useful formats. For example, consider a document (under development) which may, in the future, need to be published as a report, a journal paper, a newspaper article, a webpage, perhaps a book, I (the author) am still uncertain.  Formatting the document using "hard codes" (fonts, font sizes, page breaks, line centering, etc.) is rather a waste of time--styles vary very much between the particular document types and are publisher-dependent.  The solution is to format the document using "logical" layout elements which may include the document title, chapter titles, subchapters, emphasis style, picture filenames, caption text, tables, etc. Thats what "docbook" does--it is a description of styles (using xml, a superset of html, and a close relative of sgml)--a so-called stylesheet. The logical layout is rendered to a physical appearance when the document is being published.

This section will be expanded in the future as we learn to use docbook.
 

7.2 Simple Programming under Linux

How do I write a simple perl script?

I may use pico (or any other text editor of my choice) to type in a simple perl script:

pico try_perl

The example script below does nothing useful, except illustrates some features of perl:

#!/usr/bin/perl -w
# a stupid example perl program
# the lines starting with # are comments except for the first line
# names of scalar variables start with $
$a=2;
$b=3;
# each instruction ends with a semicolon, like in "c"
print $a**$b,"\n";
$hello_world='Hello World';
print $hello_world,"\n";
system "ls";

The first line tells the shell how to execute my text file. The option "-w" causes perl to print some additional warnings, etc. that may be useful for debugging your script. The next 3 lines (starting with #) are comments.  The following lines are almost self explanatory: I assign some values to two variables ($a and $b), put $a to power $b and print the result.  The "\n" prints a new line, just like in the "c" programming language.  Then I define another variable to contain the string "Hello World" and, in the next line,  I print it to the screen. Finally, I execute the local operating system command "ls", which on Linux prints the listing of the current directory content.  Really stupid script.

After saving the file, I make it executable:
chmod a+x try_perl

Now, I can run the script by typing:
./try_perl

Here is somewhat longer script that does something very simple yet useful to me. I take a long text file which is generatated by a data acquisition system. I need to erase every other line (or so) so that the file can be crammed into MS Excel (as required):

#!/usr/bin/perl -w
# Create a text file containing a selection of lines from an original file. This is needed
# so that data for manual postprocessing are fewer.
#
# Prompt the user for the filename, and the selection of lines to preserve in the output.
print STDOUT "Enter the filename: ";
chomp($infile=<STDIN>);
open(INFILE,"<$infile");  # open the file for reading.
print STDOUT "Enter the number of initial lines to preserve: ";
chomp($iskip=<STDIN>);    # the first lines may contain column headings etc
print STDOUT "Enter the skip: ";
chomp($skip=<STDIN>);
#
# The name of the output file is created automatically on the basis of the
# input file and the selection of lines. It is always of type CSV, so preserve is so.
$outfile=$infile.'-pro'.$iskip.'-'.$skip.'.csv';  #glue strings together using the dot operator.
open(OUTFILE,">$outfile"); # open file for writing.
#
# write the "initial" lines to the output file.
for($a=0;$a<$iskip;$a++) {
    $line=<INFILE>;
    print OUTFILE $line;
}
#
# do the rest of the file
$c=0;$w=0;$skip++;
while($line=<INFILE>){
    $c++;
    if(!($c%$skip)) {    #use % for remainder of integer division
        print OUTFILE $line;
        $w++;
    }
}
#
close(OUTFILE);
print STDOUT "Read Lines: ", $c+$iskip," Wrote lines: ", $w+$iskip,"\n";
 

How do I write a simple Python program?

Edit a text file that will contain your Python program. I can use the kde "kate" editor to do it (under X):
kate try_python.py &

Type in some simple python code to see if it works:

#!/usr/bin/env python
print 2+2

The first line (starting with the "pound-bang") tells the shell how to execute this text file--it must be there (always as the first line) for Linux to know that this particular text file is a Python script. The second line is a simple Python expression.

After saving the file, I make it executable:
chmod a+x try_python.py

after which I can run it by typing:
./try_python.py

Python is an excellent, and very modern programming language. Give it a try,  particularly if you like object oriented programming. There are numerous libaries/extensions available on the Internet. For example, scientific python (http://starship.python.net/crew/hinsen/scientific.html) and numeric python (http://sourceforge.net/projects/numpy) are popular libraries used in engineering.

Here is a slightly longer, but still (hopefully) self-explanatory python code. A quick note:  python flow control depends on the code indentation--it makes it very natural looking and forcing legibility, but takes an hour to get used to.

#!/usr/bin/env python
# All comments start with a the character "#"
# This program converts human years to dog years

# get the original age
age = input("Enter your age (in human years): ")
print           # print a blank line

# check if the age is valid using a simple if statement
if age < 0:
        print "A negative age is not possible."
elif age < 3 or age > 110:
        print "Frankly, I don't believe you."
else:
        print "That's the same as a", age/7, "year old dog."

A simple tcl program?

#!/usr/bin/tclsh
puts stdout {Hello World!}

How do I write a simple GUI program (using Tk)?

Tk is a GUI extension of the easy yet powerful tcl programming language.  For example, I may use pico to create a text file that will contain a simple tk program:

pico try_tk

and type in a simple example of tk code to see if it works:

#!/usr/bin/wish
button .my_button -text "Hello World" -command exit
pack .my_button

The first line (starting with the "#!" pound-bang) tells the shell what utility to use to execute my text file. The next two lines are an example of a simple tk program. First, I created a button called "my_button" and placed it at the root of my class hierarchy (the dot in front of "my_button").  To the button, I tied the text "Hello World" and a command that exits the program (when the button is pressed).  Last line makes my program's window adjust its size to just big enough to contain my button.

After saving the file, I make it executable:
chmod a+x try_tk

after which I can run it by typing (in the X-terminal, because it requires X-windows to run):
./try_tk

Tk is very popular for building GUI front ends.

To write a simple program in ruby, I open my favorite text editor and start a program with the following first line:
#!/usr/bin/ruby

Here is an example of a program that I wrote to help me understand the basics of the ruby language:

#!/usr/bin/ruby
#This is a comment
 a = Array.new
 print "Please enter a few words (type EXIT to stop):\n"

 i = 0
 while enterWord = STDIN.gets
   enterWord.chop!
   if enterWord == "EXIT"
     break
   end
   a[i] = enterWord
   i += 1
 end

#sort the array
for i in 0...a.length-1 do
  for j in i+1...a.length do
    if a[j] < a[i]
      tmp = a[i]
      a[i] = a[j]
      a[j] = tmp
    end
  end
end

#Output the results
print "You entered "  + a.length.to_s + " entries.\n\n"
for i in 0...a.length do
  print "Entry " + (i+1).to_s + ": "+ a[i] + "\n"
end

I save my ruby script to file "myprogram". To execute it, I need to type on the command line:
./myprogram

How do I compile a simple C program?

Start your favourite text editor and type in your source code. For example, I may use pico:
pico hello.c

and type in the Kerningham and Richie (the creators of "c") intro example of a C program:

#include <stdio.h>
void main(void) {
printf("hello world\n");
}

I save the file and then invoke the GNU C compiler to compile the file "hello.c":
gcc hello.c

The gcc compiler produces an executable binary file "a.out", which I can run:
./a.out

How do I compile a simple C++ program?

Just like in c, I open a text editor and write my program. For example, using pico, I write the following program:

//This is a comment (to the end of line, C++ style)
#include <stdio.h>
#include <math.h>
#include <iostream.h>
#include <stdlib.h>

//define a function
double wheeldrop (double dGap, double dDiameter) {
    double dDrop, dRadius, dNotDrop;

    dRadius = dDiameter * 0.5;
    dDrop = dRadius - sqrt( (dRadius*dRadius)-(0.25*dGap*dGap) );

    return (dDrop);
}   //end of the function

//The function main is the entry point to the program
void main(void) {
    double dGap, dDiameter, dDrop, dRadius, dNotDrop;  //variables

    for (;;) {                                         //infinite loop
        cout << "Please enter gap between track segments and \n"
      << "diameter of train wheel in inches (-1 -1 to exit): ";
        cin >> dGap >> dDiameter;

        if ((dGap == -1) && (dDiameter == -1))
            break;
        else if (dGap < dDiameter) {   //do calculations
            dDrop = wheeldrop (dGap, dDiameter);
            printf ("The wheel will drop %f inches.\n\n", dDrop);
        }
        else {
            printf ("Error, your train is going to crash.\n Gap bigger then wheel!\n\n");
        }
    }
}

I save the source to the file "train.c", and then invoke the GNU C++ compiler to compile the file "train.c" to an executable called "traincalc":
g++ -o traincalc train.c

I can then run the executable by typing:
./traincalc

kdevelop
(type in X-terminal) Integrated development environment for K. It is really worth downloading (if it does not come with your distribution).

glade
(type in X-terminal) A graphical builder of user interfaces.

 "Glade is an interface builder developed by Damon Chaplin. It allows graphical and interactive construction of Gnome/Gtk graphical user interfaces. From Glade, the generated interface can be saved in a xml file or directly exported to C code to be included in a C source tree. Glade also allows to define the name of the handlers - functions - to be attached to the various event of the interface. For example the function (name) to be called when a specific menu item is pressed." (From: http://linuxtoday.com/news_story.php3?ltsn=2000-07-16-013-04-PS-GN)

cd /usr/include
grep -H "cosh" *.h

There are also many interesting libraries that are not a part of a typical distribution, e.g., GNU libraries for scientific computing (GSL): http://sources.redhat.com/gsl/. Also check: http://www.phy.duke.edu/~hsg/sci-computing.html.

as
nasm
ndisasm
(3 commands). Assembler, an alternative "native assembler" and a disassembler. Send in your newbie examples how to use those :)  E.g.: ndisasm /bin/sh |more produces a long output of "assembler mnemonics" from a binary on my system (/bin/sh in this example) but I cannot understand it anyway :(   To learn more about nasm, you may want to see:  file:///usr/share/doc/nasm-doc-0.98/html/nasmdoc0.html

Example Intel assembly for Linux 2.2.17 or higher:
;; hello.asm: Copyright (C) 2001 by Brian Raiter, under the GNU
;; General Public License (version 2 or later). No warranty.

BITS 32

  org 0x05936000

  db 0x7F, "ELF"
  dd 1
  dd 0
  dd $$
  dw 2
  dw 3
_start:  inc eax   ; 1 == exit syscall no.
  mov dl, 13       ; set edx to length of message
  cmp al, _start - $$
  pusha            ; save eax and ebx
  xchg eax, ebx    ; set ebx to 1 (stdout)
  add eax, dword 4 ; 4 == write syscall no.
  mov ecx, msg     ; point ecx at message
  int 0x80         ; eax = write(ebx, ecx, edx)
  popa             ; set eax to 1 and ebx to 0
  int 0x80         ; exit(bl)
  dw 0x20
  dw 1
msg:  db 'hello, world', 10

After saving this to a plain-text file hello.asm,  I can build it to an output file "hello" and make the output executable using the command:

nasm -f bin -o hello hello.asm && chmod +x hello

and execute it using:

./hello

The example above is borrowed from http://www.muppetlabs.com/~breadbox/software/tiny/.

Why would somebody use assembler?  After building from assemebler, this example executable is 56 bytes on my system. The "C" language example with identical functionality (see one page above) is 13.7 kB.

Here is brief info to help me understand the above program:
";" marks comments (to the end of the line).
"msg:" -- is an example of a label (like in fortan).
org (="origin")--declares where in the memory the program begins (after it is loaded to memory for execution).
db, dd, dw are nasm "pseudoinstructions" used to insert initialized data into the output file.
"$" evaluates to the assembly position at the beginning of the line containing the expression; so you can code an infinite loop using "JMP $".   "$$" evaluates to the beginning of the current section.
The general-purpose 32-bit registers in the 80x86 ("Intel") processor are: EAX, EBX, ECX, EDX, ESI, EDI, EBP, and ESP.  (The "E" stands for extended. It is there because the processor can instead  "overlay" the registers and treat them as 16-bit registers with names: AX, BX, CX, CX, SI, DI, BP, and SP.  Still underlying those, there are also eight 8-bit registeres: AL, AH, BL, BH, CL, CH, DL, DH.  Here, the "L" and "H" stand for "high" and "low" byte.).
Mnemonics for some common 80x86 processor instructions:
Name  Syntax                    Comment
NOP   NOP                       No operation (do nothing).
MOV   mov destination, source   Move (copy, set)data.
XCHG  XCHG operand1,operand2    Exchange the values.
CMP   CMP operand1,operand2     Compare the two operands.
PUSH  PUSH source               Push onto stack.(Place the value on stack and increment the stack pointer).
PUSHF PUSHF                     Push flags.
PUSHA PUSHA                     Push all general-purpose registers.
POP   POP destination           Pop from stack(take the value from stack, and decrement the stack pointer). Pop is reverse to push.
POPF  POPF                      Pop flags.
POPA  POPA                      Pop all general-purpose registers.
INC   INC operand               Increment (increase by 1).
DEC   DEC operand               Decrement (decrease by 1).
ADD   ADD Dest,Source           Add.
ADC   ADC Dest,Source           Add with carry.
SUB   SUB Dest,Source           Subtract.
INT   INT number                Execute an interrupt.
CALL  CALL subroutine           Call a subroutine.
RET   RET                       Return from this (current, innermost) subroutine.
JMP   JMP destination           Jump (start executing code starting at the the address "destination")
JE    JE destination            Jump if equal.
JNE   JNE destination           Jump if not equal.
JZ    JZ destination            Jump if zero.
JNZ   JNZ destination           Jump if not zero.
JP    JP destination            Jump if parity (parity is even).
JNP   JNP destination           Jump if no parity (parity is odd).
JPE   JPE destination           Jump if parity even.
JPO   JPO desitination          Jump if parity odd.
JCXZ  JCXZ destination          Jump if CX zero.
JECXZ JECXZ destination         Jump if ECX zero.

Silly examples for the guile interpreter.
guile
(+ 1 1)
(define a 2)
(/ a 3)
(= a 7)
(display "hello\n")
(system "ls")
(exit)
The first command runs the guile interpreter.  The next four commands do addition, definition, division, and comparison using the so-called Polish notation (operator in front of the operants). See the section on reverse Polish notation on this page. The last command exits the guile interpreter.

Silly example of a fortran code. It prints squares and cubes of numbers from 1 to 20:
PROGRAM TRY_FORTRAN
  INTEGER X
  PRINT 200, "X", "X^2", "X^3"
  DO X=1, 20
     PRINT 100, X, X**2, X**3
  END DO
100 FORMAT (I20, I20, I20)
200 FORMAT (A20, A20, A20)
END
To compile this file, I run the fortran compiler with the option that recognizes the "free-form" source code (I don't like the fixed-code source):
g77 -ffree-form try_fortran.f
and now I can run the resulting executable (which has the default name is a.out):
./a.out

kylix
This is a brand-new (Feb.2001) commercial offering from Borland (aka Inprise). In short, it is a Linux port of the famous object-oriented Pascal ("Delphi").  kylix is unlikely to be on your Linux distribution CD, you must pay for it, but if you want the best rapid application development (RAD) platform with a code portablity between Linux, MS Windows and the Web, large number of pre-built components, etc., kylix is likely the best. In my opinion, Delphi is significanly better than MS Visual Basic.

javac
java
The Java language compiler and interpreter. Under Linux, both javac and java are actually scripts which call kaffe with appropriate options (try cat /usr/bin/java).

A trivial example for a java "standalone" program. I use my favourite text editor, e.g. kate (in X terminal) to type in the following java code:
/* Comments are marked like in C++
 * A java class to display "Hello World"
 */
class HelloWorldApp {
    public static void main(String[] args) {
        System.out.println("Hello World."); // Print "Hello World." followed by a newline
    }
}
I save this into a file named try_java.java. Now, I can compile it using:
javac try_java.java
This creates a file called HelloWorldApp.class which contains the "bytecode" (semi-compiled code which requires an interpreter to run). I can run it on the command line using:
java HelloWorldApp
For an example on how to embed a simple java applet into an html web page, have a look at http://java.sun.com/docs/books/tutorial/getStarted/cupojava/unix.html from which my java "standalone" program was borrowed.

make is used to bring a system "up to date", whenever a change in one file requires an action elsewhere.  make is "intelligent" in that it will not make changes unless the change is required, as determined by the file modification date/time. Normally used for buiding software packages (compiling, linking ...),  make is also used for other tasks, e.g., system administration. Makefile looks as follows:

target :  prerequisites
[Tab]commands

where target is usually a file (but does not have to be, it can be a "phony" target), and prerequisites are files on which target depends. If target does not exist or is older than any prerequisites, "commands" are executed.  The first line above is called "the rule", the second "the action". Please note that any action line must start with the tab character. Here is an example Makefile that makes an executable file called "edit":

my_program : main.o command.o
    cc -o my_program main.o command.o
main.o : main.c defs.h
    cc -c main.c
command.o : command.c defs.h command.h
    cc -c command.c
clean :
    rm my_program main.o command.o

To use this Makefile to create an executable file called "my_program', I type:  make. It works backwards to determine the dependencies,  so first it compiles "command.c" to the object file "command.o", then it compiles "main.c" to "main.o", and finally it links "main.o" and "command.o" into the executable "my_program".
One could also use this makefile to delete the executable file and all the object files from the directory by typing: make clean.  Since the target "clean" does not depend on any prerequisites, it is not normally executed unless explicitly called. The target "clean" is an example of a "phony" target.

yes
Generate a never-ending output of strings containing "yes" (it does end when <Ctrl><c> is pressed or when electricity goes off).  Sounds like a silly utility, but it can be used to write simple programs on the command line. For example, the following amusing on-liner determines the frequency of digits in 100 000 radom numbers (the whole command is a single line):
yes | sed '100000q' | awk 'BEGIN{srand();u=6*log(10)}{printf"%e\n",rand()*exp(rand()*u)}'| cut -c1 | sort | uniq -c
I hope this example does not scare you too much--it surely shows that the old-fashioned UNIX command line can be as complicated (and powerful) as you wish to make it. If you are interested why the frequency of digits varies (it seems intuitively that it is should be constant if the numbers are random), try the website from which I borrowed the example: http://www.newscientist.com/ns/19990731/letters4.html

7.3 Math Tools

dc is based on the concept of a stack, which is central to the operations of modern digital computer. A computer stack is not unlike a stack of kitchen plates, the last to come on stack, is the first to go out (this is also known as LIFO="last-in, first-out"). This contrasts with a queue (another important concept) where the first in is the first out (FIFO).

You can perform operations only on the number(s) which is on the top of the stack. The two basic operations are: push and pop (put on the top of stack, and retrieve from the top of stack). Unary operations pop one value off the stack ("unary" means "requiring one operand"). Binary operations pop two values off the stack ("binary" means "requiring two operands"). Tertiary operations pop three values off the stack ("tertiary" means "requiring three operands"). In all cases, the result is always pushed back onto the top of stack.

RPN calculators (regular, hand-held) are very popular among technically oriented people and in academia. The RPN notation never requires parentheses.

History. The parentheses-free logic was developed by Polish mathematician Jan Lukasiewicz (1878-1956) before the WWII. Originally, the operator preceded the values. For computer applications, it's been modified so that the operator goes after the values, hence "reversed" in the name "reversed Polish notation".

To exercise some operations on stack, try this:
dc [start the arbitrary precision reverse Polish notation calculator]
1 [push "1" on the stack]
2 [push another number on the stack]
3 [push yet another number on the stack]
4 [push yet another number on the stack]
f [print the entire stack; you should see 1 2 3 4]
p [print the number on the top of the stack without affecting the stack; you should see 4]
+ [perform addition (binary operation), therefore pop two last values off the stack (4,3), and push the result (7) on the stack]
p [print the number on the top of the stack, i.e. the results of the last addition (7).].
p [print again the number on the top of the stack to see that the stack wasn't affected by printing (7)]
* [perform multiplication (binary operation), therefore pop two last values, and push the result (14)]
p [print the result of the multiplication (14)]
P [pop the last number off the stack (14)]
p [print the number on the top of the stack]
2000 [push a large integer on the stack]
k [set the precision to the value which is on the top of the stack, i.e. 2000]
1 [push another number on the stack]
f [print the content of the entire stack]
701 [push another number on the stack]
/ [divide last two numbers on the stack, i.e. "1/701" with 2000 decimal places of precision]
p [print the result of the last division]
q [quit the arbitrary precision reverse Polish notation calculator]

Please note that when using the reverse Polish notation (RPN) you never need parentheses. Try man dc to read about other capabilities of dc.

kcalc&
xcalc&
(in X terminal) Standard, GUI calculators.

e  '2*3/4+sin(pi/2)'
The "e" expression evaluator did not come on my RH7.x CDs. Yet, it is my favourite of all the "command line" calculators. Try: http://www.softnet.tuc.gr/~apdim/projects/e/ to download.

expr 1 + 1 / 3
Evaluate an integer-type expression. The "expr" is not meant to be a calculator, but is mostly for flow control in shell scripts. The above example will return the result "1", which is correct because 1/3 is 0 as far as integer division is concerned.

octave
Octave is a high-level interactive language for numerical computations, closely resembling "matlab". Included with any fine Linux distribution.

scilab&
(in X terminal) Another large and sophisticated system for numerical computing, somewhat resembling "matlab", but with a rather clampsy graphical interface. Don't even try it unless you have rather sophisticated math needs else you won't appreciate it. It is included on RH7.0 "powertools" CD. The hompage is http://www-rocq.inria.fr/scilab/

A silly example session showing some matrix algebra. My input is shown bold.
-->a=[1 1 1;2 3 4]
 a  =
!   1.    1.    1. !
!   2.    3.    4. !

-->b=[1 1; 2 2;3 3]
 b  =
!   1.    1. !
!   2.    2. !
!   3.    3. !

-->c=a*b
 c  =
!   6.     6.  !
!   20.    20. !

-->d=inv(c)
 d  =

   1.0E+14 *

!   11.258999  - 3.3776997 !
! - 11.258999    3.3776997 !

-->

head -c 8 /dev/random
cat /dev/random | od
cat /dev/urandom | memencode
(3 commands.)  Examples on how to generate random characters on the Linux command line by reading from the device "random" or "urandom". The first command produces approximately 8 characters by reading from the device "random", which  generates high quality (difficult to predict) random numbers. It will become slow once the "entropy" on your computer is exhausted (e.g., when producing lots of random characters). The solution then is to wait or type something on the keyboard, move your mouse, switch the terminal, make your hard drive to read or write, etc.,  to generate more random noise ("entropy").  The second command keeps producing random characters, but displays them as octal numbers (using the "octal dump", od, filter). It has to be interrupted with <Ctrl><c>. The third command uses the device "urandom", which is faster then "random" when generating lots of random characters. But when the system enthropy is low, the randomness of its output from "urandom" might be compromised, yet it probably is still good for all but most demanding applications. The output is filtered to the mime format (the Internet mail-oriented 7-bit encoding standard) so it is all printable ASCII. The detailed description of the theorry and implementation of the Linux algorithm for generating the random numbers can be found in the source code file://usr/src/linux/drivers/char/random.c on your Linux system.

factor 10533858466222239345
Find all the prime factors of an integer number. Factors of an integer are numbers by which the integer is divisible without a remainder. For example, the factors for 6 are: 1, 2, 3, and 6.

R
A programming language / environment for advanced statistical computing.  Type "quit()" to exit.

gnuplot
Utility for creating graphs and plots. Very good for non-interactive (batch) work, but not very simple for interactive use. A good introduction to gnuplot can be found at http://www.duke.edu/~hpgavin/gnuplot.html.
 

7.4 Miscellaneous

How do I run an MS Windows Application (using "wine")?

Still, the Linux-based "wine" library lets me execute some MS Windows binaries, although with a rather severe speed penalty. On my system (Wine installed), I can execute MS Solitaire by typing in the X-windows terminal:

wine /mnt/dos_hda1/windows/sol.exe

The /mnt/dos_hda1 is the mount point of the harddrive partition that contains MS Windows, and it is mounted.

If you don't have wine installed, put your Mandrake cd into the cdrom, mount it, and then do something like this (as root):
cd /mnt/cdrom/Mandrake/RPMS/
rpm -ihv wine-991212-1mdk.i586.rpm

Mandrake packages are RedHat-compatible so you can use the Mandrake CD to install software that RedHat lacks.
 

Can I have a RAID if my computer has two or more IDE (or other) harddrives?

RAID operates by joining two or more disks into a single logical device. There are several layers of RAID:

RAID 0 layer ("striping") just joins two or more disks into a single logical device, without giving any redundancy. It is often used to join  RAID 1 or RAID 5 layers.  RAID 0 + RAID 1 is called RAID 10.  RAID 0 + RAID 5 is called RAID 50.

RAID 1 (mirroring) combines two disks, each containing the same data.

RAID 4 combines three or more disks, with one of the disks dedicated to parity. If any disk fails, the whole logical device remains available, but with degraded performance. It is not used very often because of the performance.

RAID 5 combines three or more disks, with parity distributed accross the disks. Functionality similar to RAID 4 but apparently better performance.

Try http://www.osfaq.com/vol1/linux_softraid.htm  for more information.

RH7.2 gives me an option to set up a software raid almost automatically during the initial operationg system installation procedure.  The (simple) procedure is outlined at http://www.redhat.com/docs/manuals/linux/RHL-7.2-Manual/custom-guide/software-raid.html. Briefly, during RH installation, part "partition the hard drive" :
    (1) Create new partition(s) of the type "software RAID" You will not be able to specify a mount point for the individual "RAID-type" partitions. You can specify the size for each partition, make it "growable", or force it to be the primary partition.
    (2) Press the "Make RAID" button on the Disk Druid screen.
    (3)  Into the dialog box which appears enter: the mount point for the RAID array, the partition type for the RAID array,  the RAID type (you can select between RAID 0, RAID 1, and RAID 5), the RAID member partitions (which you created in (1)), and the number of spares (for RAID 1 or 5). (The "spare" is the partition that will be automatically used should the the software RAID fail. In (1), you should have created at least one "RAID-type" partition + one "RAID-type" partition for each spare.)
    (4) Click "OK", and inspect the "Drive Summary" if your RAID array appears correctly.

 Note:  "If you are making a RAID partition of /boot, you must choose RAID level 1 and it must use one of the first two drives (IDE first, SCSI second). If you are not creating a RAID partition of /boot, and you are making a RAID partition of /, it must be RAID level 1 and it must use one of the first two drives (IDE first, SCSI second)"

Network traffic shaping using shapecfg

Go to Appendix: How to Upgrade the Kernel
Back to Main Menu