Chapter 3
Software Tools

 3.1 Linux and Ubuntu
  3.1.1 Hex and Binary Viewer: xxd
  3.1.2 Arbitrary Precision Calculator: bc
  3.1.3 Random Numbers
  3.1.4 Hash Functions
  3.1.5 Bash Scripts
 3.2 OpenSSL
  3.2.1 Overview of OpenSSL
  3.2.2 Common Operations
  3.2.3 Listing Ciphers and Algorithms
  3.2.4 Random Number Generation with OpenSSL
 3.3 Python
  3.3.1 Cryptography Package
  3.3.2 PyCipher Package

File: crypto/tools.tex, r1962

This chapter lists common tools referred to within the rest of the book. The purpose is to make you aware of the tools; not to teach you how to use the tools. While some setup and basic usage instructions may be given, you can normally find detailed instructions by searching online, or within the tool help or manual pages.

3.1 Linux and Ubuntu

Almost all of the software-based examples or demonstrations in this book are performed using a Linux operating system, especially using command-line tools (as opposed to tools using a graphical user interface).

This book uses the Ubuntu distribution of Linux, although most tools will work equally well on other distributions (e.g. Red Hat, Fedora, Debian, Slackware).

It is assumed you have access to a Linux command line and having basic knowledge of common file and directory operations in Linux. My other book Network and Security in Linux motivates the use of Linux in the field (Chapter 2) and provides an introduction to the command line (Chapter 4).

The following lists selected command line tools which are normally available in common Linux distributions.

3.1.1 Hex and Binary Viewer: xxd

We often deal with binary values in cryptography. Therefore it is useful to be able to view the contents of binary files (as opposed to text files). xxd is one tool that allows viewing any file as binary or hex (default).

To demonstrate, let’s first create a plaintext file called demo.txt:

$ echo -n "This is a super secret message. " > demo.txt
$ ls -l demo.txt
-rw-r--r-- 1 sgordon sgordon 32 Dec  5 17:58 demo.txt

While your output of the ls command may be different it should show 32, meaning the file size is 32 bytes (note the text contains 32 characters, including the space at the end; the -n option means no new line character is added).

We can view the text file with cat:

$ cat demo.txt
This is a super secret message. $

The command prompt does not start on a new line since our text file does not finish with a new line character.

Now let’s look at the file in hex and then binary format (using the -b option):

$ xxd demo.txt
00000000: 5468 6973 2069 7320 6120 7375 7065 7220  This is a super
00000010: 7365 6372 6574 206d 6573 7361 6765 2e20  secret message.
$ xxd -b demo.txt
00000000: 01010100 01101000 01101001 01110011 00100000 01101001  This i
00000006: 01110011 00100000 01100001 00100000 01110011 01110101  s a su
0000000c: 01110000 01100101 01110010 00100000 01110011 01100101  per se
00000012: 01100011 01110010 01100101 01110100 00100000 01101101  cret m
00000018: 01100101 01110011 01110011 01100001 01100111 01100101  essage
0000001e: 00101110 00100000                                      .

By default, xxd shows three pieces of information per line of output:

  1. The address, in hex, of first byte on the line
  2. The hex (or binary) values, with some spacing to ease the readability
  3. The ASCII character (if it is printable) for the corresponding byte. A dot is shown if the ASCII character is not printable (e.g. the DELete or ESCape characters).

xxd has a variety of command line options, which are well described in the man page. For example, to show 8 bytes per line (column size), group into 2 sets of 4 bytes, displaying a length just the first 16 bytes:

$ xxd -c 8 -g 4 -l 16 demo.txt
00000000: 54686973 20697320  This is
00000008: 61207375 70657220  a super

Combined with other commands, such as cut and grep, you can extract information of interest. For example, show the binary representation of the first 64 bytes of the file /bin/ls, 8 bytes per line:

$ xxd -b -l 64 -c 8 -g 8 /bin/ls | cut -d " " -f 2
0111111101000101010011000100011000000010000000010000000100000000
0000000000000000000000000000000000000000000000000000000000000000
0000001100000000001111100000000000000001000000000000000000000000
0101000001011000000000000000000000000000000000000000000000000000
0100000000000000000000000000000000000000000000000000000000000000
1010000000000011000000100000000000000000000000000000000000000000
0000000000000000000000000000000001000000000000000011100000000000
0000100100000000010000000000000000011100000000000001101100000000

3.1.2 Arbitrary Precision Calculator: bc

No doubt you have access to a software calculator on your computer or phone, or even a traditional calculator on your desk. While these calculators are convenient, they often make approximations when dealing with very large numbers. An arbitrary precision calculator will give exact answers, no matter how big numbers are. Several important ciphers in cryptography rely on calculations with big numbers, so it is nice to have an arbitrary precision calculator available. In Linux, bc is a command-line arbitrary precision calculator.

The following shows an example of starting bc in Linux, and then performing normal arithmetic operations.

$ bc
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
1 + 2
3
20 - 13
                                                                                                                                                  
                                                                                                                                                  
7
7 * 6 + 2
44
7 * (6 + 2)
56
56 / 8
7
54 / 10
5

Note that the last division gives the quotient as the answer, not a fraction. By default, fractions are not used, but can easily be enabled by setting the scale parameter as follows:

scale=2
54/10
5.40

Exponentiation is also supported using the “hat” or “carat” operator:

10^2
100
2^3
8

Modular arithmetic (Section 5) is commonly used in cryptography. The mod operator is the percent sign (be sure to set the scale back to 0 first):

scale=0
13 % 10
3
7 * 6 % 10
2

The real use of bc in this book comes when performing operators on large numbers. As bc is an arbitrary precision calculator, it will perform any calculation without approximating (although beware, some calculations will take a long time).

2^10
1024
2^100
1267650600228229401496703205376
2^1000
10715086071862673209484250490600018105614048117055336074437503883703\
51051124936122493198378815695858127594672917553146825187145285692314\
04359845775746985748039345677748242309854210746050623711418779541821\
53046474983581941267398767559165543946077062914571196477686542167660\
429831652624386837205668069376
2^10000000
                                                                                                                                                  
                                                                                                                                                  
...

While there are faster algorithms than what bc uses, it can be used for modular exponentiation.

29401^19231
11791936741673782277951361412655628509750802626058442595065879112837\
30645660979602186783941907308557893020948598603221372351480244103370\
...
27540919410894776657722419140083914356072020143002078956241640716425\
878094269792146304397724529078575760209188791401
29401^19231 % 37669
35694

To exit bc, type quit:

quit

To perform (normal) logarithms, you need to start bc using the -l option to load the math library. Then the l() function can be used to calculate the natural logarithm, or find the logarithm in any base. Although be careful with the scale.

$ bc -l
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
l(2.718)
.99989631572895196894
l(100)/l(10)
2.00000000000000000000
l(32)/l(2)
5.00000000000000000004
l(1024)/l(2)
10.00000000000000000010

Discrete logarithms are not directly supported in bc.

3.1.3 Random Numbers

Random numbers are important for creating shared secret keys (as well as other use in other cryptographic operations). There are different ways to generate a random value in Linux. Two approaches are demonstrated in the following; a third approach is to use OpenSSL (introduced in Section 3.2 and demonstrated in Section 3.2.4.

Generating Random Numbers with Bash

The Bash shell has a built-in random number generator, which is accessed from the shell variable $RANDOM. It uses a Linear Congruential Generator (LCG) to return a value between 0 and 32,767. This is not a cryptographically strong Pseudo Random Number Generator (PRNG) and should not be used to create keys.

$ echo $RANDOM
4086
$ echo $RANDOM
11809
$ echo $RANDOM
6018

To see the details of the LCG algorithm used, look in the Bash source code; after downloading and unpacking the source, look in the file variables.c, search for the function brand. You can also see that the seed is based on the current time and process ID.

Generating Random Numbers with /dev/urandom

The Linux kernel has a pseudo-device /dev/urandom which is considered cryptographically strong PRNG for most applications. The device produces a continuous stream of random bytes, so while it is possible to view the stream in real-time using cat, it is common to pipe the output to select a specific number of bytes in an easy to read format. We can use xxd to do this.

First grab 8 Bytes, output in binary:

$ cat /dev/urandom | xxd -l 8 -b
0000000: 10000111 11110111 01001101 10011100 01111110 10110110  ..M.~.
0000006: 01010110 11010001

If we want 16 Bytes of hex output:

$ cat /dev/urandom | xxd -l 16 -g 16
00000000: 75619f0688497b213c5db43d49210c4d  ua...I{!<].=I!.M

A little bit of text processing will return just the random value (omitting the other output produced by xxd). Let’s use cut to grab the 2nd field, considering the output as space separated/delimited:

$ cat /dev/urandom | xxd -l 16 -g 16 | cut -d " " -f 2
313be197c436bebf074a2da3599a0ce0

Read the man pages for an explanation of the Linux kernel random number source device /dev/urandom and the related /dev/random. The section 7 man page gives an overview, while the section 4 man page gives more technical details on the two devices.

$ man -S7 random
$ man -S4 urandom

3.1.4 Hash Functions

Linux usually includes several commands for applying common hash functions on data. These are the “sum” commands, i.e. used for calculating checksums. The following commands demonstrate applications of the hash functions MD5, SHA1 and SHA2 (256).

$ cat demo.txt
This is a super secret message. $
$ md5sum demo.txt 
7899f47eb650b40ae9156f6664304281  demo.txt
$ sha1sum demo.txt 
87992f407ab94d05f64131db482067f9ffe42044  demo.txt
$ sha256sum demo.txt 
12e38182116f070ef1a4d8961692787aa57add87d5496c4daf402279bc71c0b6  demo.txt

You can write the hash value to a file, and then use that file to perform a check:

$ sha256sum demo.txt > demo.sha256
$ cat demo.sha256
12e38182116f070ef1a4d8961692787aa57add87d5496c4daf402279bc71c0b6  demo.txt
$ sha256sum -c demo.sha256
demo.txt: OK

A change to the file should result in failure of the check (if the hash is not recomputed):

$ cat demo.txt
This is a super secret message. $
$ echo -n "This is a super secret message! " > demo.txt
$ sha256sum -c demo.sha256 
demo.txt: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

You can also use OpenSSL (Section 3.2) to apply hash functions.

3.1.5 Bash Scripts

A useful advantage of the (Linux) command line is that it is easy to write a series of commands into a file, and then running them all by executing the file. This file is called a script. Once you have a basic grasp of the line command line (see Chapter 4 of my other book Network and Security in Linux), you can then start writing scripts to automate tasks. The basics of scripting, with many examples, is covered in Chapter 6 of my other book Network and Security in Linux.

3.2 OpenSSL

3.2.1 Overview of OpenSSL

https://www.openssl.org/ is a program and library that supports many different cryptographic operations, including:

While the primary purpose of OpenSSL is as a library, i.e. you write software that calls OpenSSL to perform cryptographic operations for your software, it also is a standalone program with a command-line interface. While we only use the standalone program, once you are familiar with it, you should be able to use the library.

OpenSSL supports different operations or commands, with the name of the command following openssl. For example, to perform symmetric key encryption the command is enc and on the command line you run:

$ openssl enc

Each of the operations supported by OpenSSL have a variety of options, such as input/output files, algorithms, algorithm parameters and formats. To start learning the details of OpenSSL, read the man page, i.e. man openssl. You’ll soon learn that each of the operations (or commands) have their own man pages. For example, the operation of symmetric key encryption is enc, which is described in man enc.

There are other websites that give an overview of OpenSSL operations, as well as programming with the API. Check them out for more details.

3.2.2 Common Operations

OpenSSL takes an operation or command as first input, and that command may have it’s own set of parameters. Parameters are usually specified starting with a dash (-). To see all the commands available, use the help command:

$ openssl help
Standard commands
asn1parse         ca                ciphers           cms
...

Message Digest commands (see the `dgst' command for more details)
blake2b512        blake2s256        gost              md4
...

Cipher commands (see the `enc' command for more details)
aes-128-cbc       aes-128-ecb       aes-192-cbc       aes-192-ecb
...

In reverse order:

Cipher commands
perform symmetric key encryption
Message Digest commands
apply hash functions
Standard commands
provide a variety of operations related to public key generation and key management

For cipher and message digest commands, you can read the common format in the man pages man enc and man dgst, respectively. Most of the standard commands have their own man page, e.g. man rsa, man x509. Note that there are sometimes multiple commands that can be used to perform the same cryptographic operation. For example, you can generate Rivest Shamir Adleman cipher (RSA) key pairs using either genrsa or genpkey commands. This is mainly for compatibility reasons, that is, over time new commands have been added and the old command maintained.

Some common commands you will see in this book include:

3.2.3 Listing Ciphers and Algorithms

To see the (symmetric key) ciphers and (hash functions or) digests supported by your version of OpenSSL, you can use the list command. OpenSSL distinguishes between the algorithms and the commands used to call those algorithms. The following shows the version of OpenSSL, then lists algorithms and commands for ciphers and digests.

$ openssl version
OpenSSL 1.1.1  11 Sep 2018
$ openssl list -cipher-algorithms
AES-128-CBC
AES-128-CBC-HMAC-SHA1
...
SM4-ECB
SM4-OFB
$ openssl list -cipher-commands
aes-128-cbc       aes-128-ecb       aes-192-cbc       aes-192-ecb
aes-256-cbc       aes-256-ecb       aria-128-cbc      aria-128-cfb
aria-128-cfb1     aria-128-cfb8     aria-128-ctr      aria-128-ecb
aria-128-ofb      aria-192-cbc      aria-192-cfb      aria-192-cfb1
aria-192-cfb8     aria-192-ctr      aria-192-ecb      aria-192-ofb
aria-256-cbc      aria-256-cfb      aria-256-cfb1     aria-256-cfb8
aria-256-ctr      aria-256-ecb      aria-256-ofb      base64
bf                bf-cbc            bf-cfb            bf-ecb
bf-ofb            camellia-128-cbc  camellia-128-ecb  camellia-192-cbc
camellia-192-ecb  camellia-256-cbc  camellia-256-ecb  cast
cast-cbc          cast5-cbc         cast5-cfb         cast5-ecb
cast5-ofb         des               des-cbc           des-cfb
des-ecb           des-ede           des-ede-cbc       des-ede-cfb
des-ede-ofb       des-ede3          des-ede3-cbc      des-ede3-cfb
des-ede3-ofb      des-ofb           des3              desx
rc2               rc2-40-cbc        rc2-64-cbc        rc2-cbc
rc2-cfb           rc2-ecb           rc2-ofb           rc4
rc4-40            seed              seed-cbc          seed-cfb
seed-ecb          seed-ofb          sm4-cbc           sm4-cfb
sm4-ctr           sm4-ecb           sm4-ofb

openssl list -digest-algorithms
RSA-MD4 => MD4
RSA-MD5 => MD5
...
ssl3-sha1 => SHA1
whirlpool
$ openssl list -digest-commands
blake2b512        blake2s256        gost              md4
md5               rmd160            sha1              sha224
sha256            sha3-224          sha3-256          sha3-384
sha3-512          sha384            sha512            sha512-224
                                                                                                                                                  
                                                                                                                                                  
sha512-256        shake128          shake256          sm3

3.2.4 Random Number Generation with OpenSSL

Section 3.1.3 shows different ways to generate random numbers in Linux. OpenSSL has its own PRNG which is also considered cryptographically strong. This is accessed using the rand command and specifying the number of bytes to generate. To get hex output, use the -hex option:

$ openssl rand -hex 8
89978d4960720a750f35d569bcf28494

You can also output to a file and view the file with xxd:

$ openssl rand -out rand1.bin 8
$ ls -l rand1.bin 
-rw-rw-r-- 1 sgordon sgordon 8 Jul 31 15:14 rand1.bin
$ xxd rand1.bin
0000000: 7d12 162f 1a18 c331                      }../...1
$ xxd -b -g 8 -c 8 rand1.bin  | cut -d " " -f 2
0111110100010010000101100010111100011010000110001100001100110001

On Linux, the OpenSSL rand command normally uses output from /dev/urandom to seed (initialise) it’s PRNG. Read the man page for more information.

3.3 Python

Python (www.python.org) is a programming language that is seeing increasing use in networking applications that require cryptography. We use it for examples in this book as it is relatively quick to pick up and start building prototype applications with custom or existing cryptographic mechanisms.

Like most programming languages, including Java, PHP and C++, libraries are available that already implement common cryptographic mechanisms; you can focus your efforts on developing applications, not implementing encryption ciphers and hash algorithms. However there is currently no single standard cryptography library for Python; several are available. In this book we use the cryptography package, as introduced in Section 3.3.1. Another common library, not used in this book, is based on PyNaCL (https://pynacl.readthedocs.io/), which is based on libsodium and NaCL.

To use classical ciphers, the PyCipher package is used, which is introduced in Section 3.3.2.

3.3.1 Cryptography Package

For installation and quick usage guide, see https://cryptography.io/.

3.3.2 PyCipher Package

To learn some of the concepts and approaches used by current encryption algorithms (ciphers), it can be useful to first study how some of the original, simpler ciphers work (e.g. Caesar cipher, Playfair, Vigenere). With these simpler ciphers, often referred to as classical ciphers, it is quite easy to understand the algorithm and even perform encryption/decryption by hand. Although it is valuable to initially perform the encryption steps by hand, sometimes its useful to use software to speed things up. pycipher is a Python package that implements many classical ciphers. It has good documentation on how to use it, including installation instructions. Below I give two alternatives to install pycipher in a virtnet node. The first is the default and easiest that uses git. The second is an alternative if git is not available and you want a specific version of pycipher.

Install pycipher (Recommended Method)

In a terminal in Linux run:

$ sudo apt-get update
$ sudo apt-get install git python-pip
$ sudo pip install git+git://github.com/jameslyons/pycipher

Install pycipher (Alternative Method)

If the recommended method above does not work (e.g. you don’t have or want to use git or pip), then you could try the following:

$ sudo apt-get install unzip python-setuptools
$ wget https://github.com/jameslyons/pycipher/archive/master.zip
$ unzip master.zip
$ cd pycipher-master/
$ sudo python setup.py install
$ python setup.py test

This installs and tests the latest version. Depending on the version, some tests my fail. In my case it ran 41 tests, but 2 tests failed (using the Porta algorithm). Do not use the algorithms that failed the tests.

Using pycipher

A quick example of encrypting and decrypting with pycipher is below. Other ciphers include: Beaufort, Foursquare, Enigma, Polybius, Bifid, ADFGVX, Coltrans, Playfair, and Vigenere. Details on the ciphers supported and how to use them are in the latest documentation.

$ python
Python 2.7.3 (default, Feb 27 2014, 20:00:17)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
> import pycipher
> pycipher.Caesar(3).encipher("hello")
'KHOOR'
> pycipher.Caesar(3).decipher("khoor")
'HELLO'
> quit()