3b-programming-cover

Training an ANN with multiple inputs

In this programming exercise, we shall continue our ANN build that we programmed in Exercise 1. The aim of this exercise is to extend the functionality of the network such that it can recognise and identify two inputs by completing the following tasks:

Changing the structure of our ANN.
We shall adapt the code such that on each training cycle both training pairs are fed into the ANN consecutively.
We shall plot the error values of each neuron output for each pattern and the total error per training cycle, a measure of overall ANN performance, and plot that separately.

In later exercises, we shall expand this build to:

Integrate the capability to read training pair inputs from an external text file.
Extend the build’s functionality to recognise and identify 5x7 images of alphanumeric characters.
Devise and program how we want the ANN to stop training

In this exercise we shall be building the following network:

ANN for programming exercise 2

Figure 3: ANN for programming exercise 2

Each neuron shall be activated using the Sigmoid Activation Function.

The network shall be trained to recognise the following two input patterns.

input pattern 1a_1

input pattern 1b

Training Pair
I1	0	O1	1
I2	1	O2	0

input pattern 2a_1

input pattern 2b

Training Pair
I1	1	O1	0
I2	0	O2	1

Step 1: Changing the structure of the network

In the previous programming exercise, we defined the size of the network by defining the number of Inputs, Neurons in the Mid Layer and Neurons in the Output Layer (nInputs, nNeurMid and nNeurOut respectively). We then created arrays to store the networked interconnections the size of which was based on functions of these variables (“nlayer1w = nInputs*nNeurMid” & “nlayer2w = nNeurMid*nNeurOut”). Note that the manner in which we programmed the ANN by referencing all subsequent variables, arrays, functions and formulae to these structural variables means that we can quickly change the structure of our network without having to change any of the subsequent programming.

We are can quickly change the structure of our network from a 2 x 2 x 2 network to a 2 x 4 x 2 network by simply changing the variable nNeurMid from [2] to [4].

% Structure of ANN
nInputs = 2;
nNeurMid = 4;
nNeurOut = 2;

Step 2: Entering two training pairs

In the previous programming exercise, we defined the training pair in simple terms: (KnownIn = [0;1] & TargetOut = [1;0]) and encapsulated these arrays within the training loop such that at the start of each cycle, the training pair would be re-established. To process two training pairs, we want to run the forward pass and backward pass for each training pair, in each training cycle.

Defining the number of Training Pairs in Code Setup

Before we create the embedded training pair loop we will add the new variable nTPairs, which we will set to the value [2] in the setup code.

We also need to expand our error log such that there is space to calculate the error of each output neuron for each training pair.

ErrLOG		TP1		TP2
ErrLOG		O1	O2	O1	O2
TCycle	1	-	-	-	-
	2	-	-	-	-
	3	-	-	-	-

% Training pairs, cycles and training rate (N)
nTPairs = 2;
nTCycles = 10;
N = 1;

% Array to store output error of each training cycle
ErrLog = zeros (nTCycles, nNeurOut*nTPairs, "double");

Embedding the Training Pair Loop

We now need to create the second embedded loop within the training cycle:

Note that the end of the Training Pair loop should be after the error log is populated, but before the figure is plotted, thus encapsulating the full forward and backwards pass set of code.

for Tloop = 1:1:nTCycles
 
    for TPair = 1:1:nTPairs
 
        % -----------------------------------------------%
        % 2: Define 2 Training Pairs (TPair)
        % -----------------------------------------------%

        % -----------------------------------------------%
        % 3: Forward Pass
        %- ----------------------------------------------%
 
        % -----------------------------------------------%
        % 4: Backward Pass
        %- ----------------------------------------------%
    end
end

Defining the Training Pairs

Now that we have done that, we need to define the two training pairs:

Note: this code should be located within the training pair loop defined as illustrated in the previous section.

Training Pair 1
I1	0	O1	1
I2	1	O2	0

Training Pair 2
I1	1	O1	0
I2	0	O2	1

if TPair == 1
    KnownIn = [0;1];
    TargetOut = [1;0];
elseif TPair == 2
    KnownIn = [1;0];
    TargetOut = [0;1];
end

Step 3: Plotting the error values

In step 2, we expanded the size of the error log array, such that it was able to store the error value calculated on each Output Neuron (of which there are 2) for each training pair (also 2), i.e. the width of the error log array is 4 columns deep.

First, we need to correctly store the error values in the correct location in the error log array.

Note we could do this with an IF statement that assesses the value of TPair and assigns the error value appropriately, however, a quicker way is to do it arithmetically. This method is both easier to expand upon in future and requires less processing power.

% Calculates error (and populates error log)
for o = 1:1:nNeurOut
    squash = NeurOut(o,2) * (1-NeurOut(o,2));
    NeurOut(o,3) = squash * (TargetOut(o,1)-NeurOut(o,2));
    ErrLog(Tloop, o + ((TPair-1) * nTPairs)) =
          abs(NeurOut(o,3) * 100);
end

We shall simply plot all the values in the Error Log, which requires a minor adjustment to our FOR loop.

figure
hold on;
for o = 1:1:nNeurOut*nTPairs
    plot(ErrLog(:,o));
end
title('Output error exposed to training');
xlabel('Number of training loops');
ylabel('Error');

Output error exposed to training

Figure 4: Example plot of ANN output error following training using 2 patterns

Finally, MATLAB provides some very quick functions to calculate the total sum value of a series of values in an error.

Note in another language, the programmer may have to loop through each value of the array to create the new array of totals.

S = sum(ErrLog,2);
figure
plot(S);
title('Total error exposed to training');
xlabel('Number of training loops');
ylabel('Total Error');

Total error exposed to training

Programming Exercise 2