This chapter lists common tools referred to within the rest of the book. The purpose is to make you aware of the tools; not to teach you how to use the tools. While some setup and basic usage instructions may be given, you can normally find detailed instructions by searching online, or within the tool help or manual pages.
Almost all of the software-based examples or demonstrations in this book are performed using a Linux operating system, especially using command-line tools (as opposed to tools using a graphical user interface).
This book uses the Ubuntu distribution of Linux, although most tools will work equally well on other distributions (e.g. Red Hat, Fedora, Debian, Slackware).
It is assumed you have access to a Linux command line and having basic knowledge of common file and directory operations in Linux. My other book Network and Security in Linux motivates the use of Linux in the field (Chapter 2) and provides an introduction to the command line (Chapter 4).
The following lists selected command line tools which are normally available in common Linux distributions.
We often deal with binary values in cryptography. Therefore it is useful to be able to view the contents of binary files (as opposed to text files). xxd is one tool that allows viewing any file as binary or hex (default).
To demonstrate, let’s first create a plaintext file called demo.txt:
$ echo -n "This is a super secret message. " > demo.txt
$ ls -l demo.txt
-rw-r--r-- 1 sgordon sgordon 32 Dec 5 17:58 demo.txt
While your output of the ls command may be different it should show 32, meaning the file size is 32 bytes (note the text contains 32 characters, including the space at the end; the -n option means no new line character is added).
We can view the text file with cat:
$ cat demo.txt
This is a super secret message. $
The command prompt does not start on a new line since our text file does not finish with a new line character.
Now let’s look at the file in hex and then binary format (using the -b option):
$ xxd demo.txt
00000000: 5468 6973 2069 7320 6120 7375 7065 7220 This is a super
00000010: 7365 6372 6574 206d 6573 7361 6765 2e20 secret message.
$ xxd -b demo.txt
00000000: 01010100 01101000 01101001 01110011 00100000 01101001 This i
00000006: 01110011 00100000 01100001 00100000 01110011 01110101 s a su
0000000c: 01110000 01100101 01110010 00100000 01110011 01100101 per se
00000012: 01100011 01110010 01100101 01110100 00100000 01101101 cret m
00000018: 01100101 01110011 01110011 01100001 01100111 01100101 essage
0000001e: 00101110 00100000 .
By default, xxd shows three pieces of information per line of output:
xxd has a variety of command line options, which are well described in the man page. For example, to show 8 bytes per line (column size), group into 2 sets of 4 bytes, displaying a length just the first 16 bytes:
$ xxd -c 8 -g 4 -l 16 demo.txt
00000000: 54686973 20697320 This is
00000008: 61207375 70657220 a super
Combined with other commands, such as cut and grep, you can extract information of interest. For example, show the binary representation of the first 64 bytes of the file /bin/ls, 8 bytes per line:
$ xxd -b -l 64 -c 8 -g 8 /bin/ls | cut -d " " -f 2
0111111101000101010011000100011000000010000000010000000100000000
0000000000000000000000000000000000000000000000000000000000000000
0000001100000000001111100000000000000001000000000000000000000000
0101000001011000000000000000000000000000000000000000000000000000
0100000000000000000000000000000000000000000000000000000000000000
1010000000000011000000100000000000000000000000000000000000000000
0000000000000000000000000000000001000000000000000011100000000000
0000100100000000010000000000000000011100000000000001101100000000
No doubt you have access to a software calculator on your computer or phone, or even a traditional calculator on your desk. While these calculators are convenient, they often make approximations when dealing with very large numbers. An arbitrary precision calculator will give exact answers, no matter how big numbers are. Several important ciphers in cryptography rely on calculations with big numbers, so it is nice to have an arbitrary precision calculator available. In Linux, bc is a command-line arbitrary precision calculator.
The following shows an example of starting bc in Linux, and then performing normal arithmetic operations.
$ bc
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
1 + 2
3
20 - 13
7
7 * 6 + 2
44
7 * (6 + 2)
56
56 / 8
7
54 / 10
5
Note that the last division gives the quotient as the answer, not a fraction. By default, fractions are not used, but can easily be enabled by setting the scale parameter as follows:
scale=2
54/10
5.40
Exponentiation is also supported using the “hat” or “carat” operator:
10^2
100
2^3
8
Modular arithmetic (Section 5) is commonly used in cryptography. The mod operator is the percent sign (be sure to set the scale back to 0 first):
scale=0
13 % 10
3
7 * 6 % 10
2
The real use of bc in this book comes when performing operators on large numbers. As bc is an arbitrary precision calculator, it will perform any calculation without approximating (although beware, some calculations will take a long time).
2^10
1024
2^100
1267650600228229401496703205376
2^1000
10715086071862673209484250490600018105614048117055336074437503883703\
51051124936122493198378815695858127594672917553146825187145285692314\
04359845775746985748039345677748242309854210746050623711418779541821\
53046474983581941267398767559165543946077062914571196477686542167660\
429831652624386837205668069376
2^10000000
...
While there are faster algorithms than what bc uses, it can be used for modular exponentiation.
29401^19231
11791936741673782277951361412655628509750802626058442595065879112837\
30645660979602186783941907308557893020948598603221372351480244103370\
...
27540919410894776657722419140083914356072020143002078956241640716425\
878094269792146304397724529078575760209188791401
29401^19231 % 37669
35694
To exit bc, type quit:
quit
To perform (normal) logarithms, you need to start bc using the -l option to load the math library. Then the l() function can be used to calculate the natural logarithm, or find the logarithm in any base. Although be careful with the scale.
$ bc -l
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
l(2.718)
.99989631572895196894
l(100)/l(10)
2.00000000000000000000
l(32)/l(2)
5.00000000000000000004
l(1024)/l(2)
10.00000000000000000010
Discrete logarithms are not directly supported in bc.
Random numbers are important for creating shared secret keys (as well as other use in other cryptographic operations). There are different ways to generate a random value in Linux. Two approaches are demonstrated in the following; a third approach is to use OpenSSL (introduced in Section 3.2 and demonstrated in Section 3.2.4.
The Bash shell has a built-in random number generator, which is accessed from the shell variable $RANDOM. It uses a Linear Congruential Generator (LCG) to return a value between 0 and 32,767. This is not a cryptographically strong Pseudo Random Number Generator (PRNG) and should not be used to create keys.
$ echo $RANDOM
4086
$ echo $RANDOM
11809
$ echo $RANDOM
6018
To see the details of the LCG algorithm used, look in the Bash source code; after downloading and unpacking the source, look in the file variables.c, search for the function brand. You can also see that the seed is based on the current time and process ID.
The Linux kernel has a pseudo-device /dev/urandom which is considered cryptographically strong PRNG for most applications. The device produces a continuous stream of random bytes, so while it is possible to view the stream in real-time using cat, it is common to pipe the output to select a specific number of bytes in an easy to read format. We can use xxd to do this.
First grab 8 Bytes, output in binary:
$ cat /dev/urandom | xxd -l 8 -b
0000000: 10000111 11110111 01001101 10011100 01111110 10110110 ..M.~.
0000006: 01010110 11010001
If we want 16 Bytes of hex output:
$ cat /dev/urandom | xxd -l 16 -g 16
00000000: 75619f0688497b213c5db43d49210c4d ua...I{!<].=I!.M
A little bit of text processing will return just the random value (omitting the other output produced by xxd). Let’s use cut to grab the 2nd field, considering the output as space separated/delimited:
$ cat /dev/urandom | xxd -l 16 -g 16 | cut -d " " -f 2
313be197c436bebf074a2da3599a0ce0
Read the man pages for an explanation of the Linux kernel random number source device /dev/urandom and the related /dev/random. The section 7 man page gives an overview, while the section 4 man page gives more technical details on the two devices.
$ man -S7 random
$ man -S4 urandom
Linux usually includes several commands for applying common hash functions on data. These are the “sum” commands, i.e. used for calculating checksums. The following commands demonstrate applications of the hash functions MD5, SHA1 and SHA2 (256).
$ cat demo.txt
This is a super secret message. $
$ md5sum demo.txt
7899f47eb650b40ae9156f6664304281 demo.txt
$ sha1sum demo.txt
87992f407ab94d05f64131db482067f9ffe42044 demo.txt
$ sha256sum demo.txt
12e38182116f070ef1a4d8961692787aa57add87d5496c4daf402279bc71c0b6 demo.txt
You can write the hash value to a file, and then use that file to perform a check:
$ sha256sum demo.txt > demo.sha256
$ cat demo.sha256
12e38182116f070ef1a4d8961692787aa57add87d5496c4daf402279bc71c0b6 demo.txt
$ sha256sum -c demo.sha256
demo.txt: OK
A change to the file should result in failure of the check (if the hash is not recomputed):
$ cat demo.txt
This is a super secret message. $
$ echo -n "This is a super secret message! " > demo.txt
$ sha256sum -c demo.sha256
demo.txt: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
You can also use OpenSSL (Section 3.2) to apply hash functions.
A useful advantage of the (Linux) command line is that it is easy to write a series of commands into a file, and then running them all by executing the file. This file is called a script. Once you have a basic grasp of the line command line (see Chapter 4 of my other book Network and Security in Linux), you can then start writing scripts to automate tasks. The basics of scripting, with many examples, is covered in Chapter 6 of my other book Network and Security in Linux.
https://www.openssl.org/ is a program and library that supports many different cryptographic operations, including:
While the primary purpose of OpenSSL is as a library, i.e. you write software that calls OpenSSL to perform cryptographic operations for your software, it also is a standalone program with a command-line interface. While we only use the standalone program, once you are familiar with it, you should be able to use the library.
OpenSSL supports different operations or commands, with the name of the command following openssl. For example, to perform symmetric key encryption the command is enc and on the command line you run:
$ openssl enc
Each of the operations supported by OpenSSL have a variety of options, such as input/output files, algorithms, algorithm parameters and formats. To start learning the details of OpenSSL, read the man page, i.e. man openssl. You’ll soon learn that each of the operations (or commands) have their own man pages. For example, the operation of symmetric key encryption is enc, which is described in man enc.
There are other websites that give an overview of OpenSSL operations, as well as programming with the API. Check them out for more details.
OpenSSL takes an operation or command as first input, and that command may have it’s own set of parameters. Parameters are usually specified starting with a dash (-). To see all the commands available, use the help command:
$ openssl help
Standard commands
asn1parse ca ciphers cms
...
Message Digest commands (see the `dgst' command for more details)
blake2b512 blake2s256 gost md4
...
Cipher commands (see the `enc' command for more details)
aes-128-cbc aes-128-ecb aes-192-cbc aes-192-ecb
...
In reverse order:
For cipher and message digest commands, you can read the common format in the man pages man enc and man dgst, respectively. Most of the standard commands have their own man page, e.g. man rsa, man x509. Note that there are sometimes multiple commands that can be used to perform the same cryptographic operation. For example, you can generate Rivest Shamir Adleman cipher (RSA) key pairs using either genrsa or genpkey commands. This is mainly for compatibility reasons, that is, over time new commands have been added and the old command maintained.
Some common commands you will see in this book include:
To see the (symmetric key) ciphers and (hash functions or) digests supported by your version of OpenSSL, you can use the list command. OpenSSL distinguishes between the algorithms and the commands used to call those algorithms. The following shows the version of OpenSSL, then lists algorithms and commands for ciphers and digests.
$ openssl version
OpenSSL 1.1.1 11 Sep 2018
$ openssl list -cipher-algorithms
AES-128-CBC
AES-128-CBC-HMAC-SHA1
...
SM4-ECB
SM4-OFB
$ openssl list -cipher-commands
aes-128-cbc aes-128-ecb aes-192-cbc aes-192-ecb
aes-256-cbc aes-256-ecb aria-128-cbc aria-128-cfb
aria-128-cfb1 aria-128-cfb8 aria-128-ctr aria-128-ecb
aria-128-ofb aria-192-cbc aria-192-cfb aria-192-cfb1
aria-192-cfb8 aria-192-ctr aria-192-ecb aria-192-ofb
aria-256-cbc aria-256-cfb aria-256-cfb1 aria-256-cfb8
aria-256-ctr aria-256-ecb aria-256-ofb base64
bf bf-cbc bf-cfb bf-ecb
bf-ofb camellia-128-cbc camellia-128-ecb camellia-192-cbc
camellia-192-ecb camellia-256-cbc camellia-256-ecb cast
cast-cbc cast5-cbc cast5-cfb cast5-ecb
cast5-ofb des des-cbc des-cfb
des-ecb des-ede des-ede-cbc des-ede-cfb
des-ede-ofb des-ede3 des-ede3-cbc des-ede3-cfb
des-ede3-ofb des-ofb des3 desx
rc2 rc2-40-cbc rc2-64-cbc rc2-cbc
rc2-cfb rc2-ecb rc2-ofb rc4
rc4-40 seed seed-cbc seed-cfb
seed-ecb seed-ofb sm4-cbc sm4-cfb
sm4-ctr sm4-ecb sm4-ofb
openssl list -digest-algorithms
RSA-MD4 => MD4
RSA-MD5 => MD5
...
ssl3-sha1 => SHA1
whirlpool
$ openssl list -digest-commands
blake2b512 blake2s256 gost md4
md5 rmd160 sha1 sha224
sha256 sha3-224 sha3-256 sha3-384
sha3-512 sha384 sha512 sha512-224
sha512-256 shake128 shake256 sm3
Section 3.1.3 shows different ways to generate random numbers in Linux. OpenSSL has its own PRNG which is also considered cryptographically strong. This is accessed using the rand command and specifying the number of bytes to generate. To get hex output, use the -hex option:
$ openssl rand -hex 8
89978d4960720a750f35d569bcf28494
You can also output to a file and view the file with xxd:
$ openssl rand -out rand1.bin 8
$ ls -l rand1.bin
-rw-rw-r-- 1 sgordon sgordon 8 Jul 31 15:14 rand1.bin
$ xxd rand1.bin
0000000: 7d12 162f 1a18 c331 }../...1
$ xxd -b -g 8 -c 8 rand1.bin | cut -d " " -f 2
0111110100010010000101100010111100011010000110001100001100110001
On Linux, the OpenSSL rand command normally uses output from /dev/urandom to seed (initialise) it’s PRNG. Read the man page for more information.
Python (www.python.org) is a programming language that is seeing increasing use in networking applications that require cryptography. We use it for examples in this book as it is relatively quick to pick up and start building prototype applications with custom or existing cryptographic mechanisms.
Like most programming languages, including Java, PHP and C++, libraries are available that already implement common cryptographic mechanisms; you can focus your efforts on developing applications, not implementing encryption ciphers and hash algorithms. However there is currently no single standard cryptography library for Python; several are available. In this book we use the cryptography package, as introduced in Section 3.3.1. Another common library, not used in this book, is based on PyNaCL (https://pynacl.readthedocs.io/), which is based on libsodium and NaCL.
To use classical ciphers, the PyCipher package is used, which is introduced in Section 3.3.2.
For installation and quick usage guide, see https://cryptography.io/.
To learn some of the concepts and approaches used by current encryption algorithms (ciphers), it can be useful to first study how some of the original, simpler ciphers work (e.g. Caesar cipher, Playfair, Vigenere). With these simpler ciphers, often referred to as classical ciphers, it is quite easy to understand the algorithm and even perform encryption/decryption by hand. Although it is valuable to initially perform the encryption steps by hand, sometimes its useful to use software to speed things up. pycipher is a Python package that implements many classical ciphers. It has good documentation on how to use it, including installation instructions. Below I give two alternatives to install pycipher in a virtnet node. The first is the default and easiest that uses git. The second is an alternative if git is not available and you want a specific version of pycipher.
$ sudo apt-get update
$ sudo apt-get install git python-pip
$ sudo pip install git+git://github.com/jameslyons/pycipher
If the recommended method above does not work (e.g. you don’t have or want to use git or pip), then you could try the following:
$ sudo apt-get install unzip python-setuptools
$ wget https://github.com/jameslyons/pycipher/archive/master.zip
$ unzip master.zip
$ cd pycipher-master/
$ sudo python setup.py install
$ python setup.py test
This installs and tests the latest version. Depending on the version, some tests my fail. In my case it ran 41 tests, but 2 tests failed (using the Porta algorithm). Do not use the algorithms that failed the tests.
A quick example of encrypting and decrypting with pycipher is below. Other ciphers include: Beaufort, Foursquare, Enigma, Polybius, Bifid, ADFGVX, Coltrans, Playfair, and Vigenere. Details on the ciphers supported and how to use them are in the latest documentation.
$ python
Python 2.7.3 (default, Feb 27 2014, 20:00:17)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
> import pycipher
> pycipher.Caesar(3).encipher("hello")
'KHOOR'
> pycipher.Caesar(3).decipher("khoor")
'HELLO'
> quit()