Assessing performance and stopping
In this programming exercise, we shall complete the build of the ANN that we have been working on through Exercises 1 to 3. In review, our ANN to date has the following functionality:
- Our program accesses an external file containing a string of characters representing the pixels that create the letters ‘A’, ‘B’ and ‘C’
- Note that this file is pre-processed into a format recognisable to our ANN. The task of pre-processing input data is often very time consuming.
- From this input data, the program creates a set of Training Pairs
- Again, using the structure of the input data along with some programmer specified parameters, the program establishes the dimensions of its ANN.
- The program creates a log in which to store the errors calculated after each training loop.
- Entering the ANN, the program executes a forward pass and calculates the error value of each output neuron.
- Based on the error values calculated, the program executes a backwards pass using the back propagation algorithm.
- The ANN loops through all its training pairs and all of its training cycles, modifying its weights on each cycle.
- Finally, the program creates graphs of each output neuron error and the total error as it changes with each training cycle.
In this exercise, we shall complete our ANN build by adding the capability to:
- determine when to stop training and
- report upon the network’s performance
Assessing network performance and when to stop training
In Reusable Learning Object 2 we were introduced to several techniques for determining how well a network performs and how to decide when to stop training. To summarise, these included:
- Considering how well the network performs against a validation set
- Accomplishing multiple layers of success criteria, e.g. success in individual training pair accuracy and overall accuracy for all training pairs
- Reaching a plateau in network performance, whereby many additional training cycles do not result in a significant further performance gain
Note that these are just some simplistic concepts of and are in no way exhaustive.
In this exercise, we shall select the second of these techniques and create a set of criteria that, once met, shall stop the network from training. The layered set of criteria can be illustrated in the following decision tree:
Figure 1:Programming exercise 4 - Decision tree
Considering this decision tree, it is clear that we are going to require three nested conditional statements and that these should be inserted after the forward pass and before the backward pass. These conditional statements should seek to confirm:
- Have all the training pairs been completed in the current training cycle?
- Is the output error for each individual training pair less than 1%?
- Note that this would be a sum of the errors from each individual output neuron, e.g. where the target for ‘A’ is [1 0 0].
- Is the total error (the sum of errors for each individual training pair less than 2%?
- Note that this would be a sum of the errors from each training pair, calculated previously.
If all of these conditional statements are satisfied, we would then like to terminate the ANN and save the current set of weights.
One final consideration is that we also want to put a limit on the maximum number of training cycles that the ANN could run through before self-terminating and flagging a failure to train to the user.
Step 1: The conditional statements
In this section, we shall program each of the conditional statements illustrated in the decision tree. Note that these statements are to be nested, i.e., each statement shall be contained within one preceding it.
As noted, the location of these statements should be inserted after the forward pass and before the backward pass
for Tloop = 1:1:nTCycles
for TPair = 1:1:nTPairs
% -----------------------------------------------%
% 2: Define 2 Training Pairs (TPair)
% -----------------------------------------------%
% -----------------------------------------------%
% 3: Forward Pass
%- ----------------------------------------------%
<< location of nested conditional statements >>
% -----------------------------------------------%
% 4: Backward Pass
%- ----------------------------------------------%
end
end
The first conditional statement is:
- Have all the training pairs been completed in the current training cycle?
This can be achieved by checking whether the current Training Pair ‘TPair’ is the final Training Pair ‘nTPairs’.
%--------------------------------------------------------%
% Termination Criteria
%--------------------------------------------------------%
if TPair == nTPairs
% << >>
end
The second conditional statement is:
- Is the output error for each individual training pair less than 1%?
In many software environments we would have to write a loop to check each entry in a row of our error log, but MATLAB provides an easier method of doing this check. In MATLAB the reference ‘Arr (nth,:)’ refers to all items the ‘nth’ row of the array ‘Arr’.
if TPair == nTPairs
if ErrLog(Tloop,:) < 1
% << >>
End
end
The third and final conditional statement is:
- Is the total error (the sum of errors for each individual training pair less than 2%?
Once again MATLAB provides an easy method. The command ‘sum (Arr (nth,:))’ calculates the sum of all values contained in the brackets, i.e. all values in the ‘nth’ row of the array ‘Arr’.
if TPair == nTPairs
if ErrLog(Tloop,:) < 1
if sum(ErrLog(Tloop,:)) < 2
% << >>
end
end
end
Step 2: Terminating the program
The command ‘break’ is a common syntax (used across many programming languages) that terminates the execution of a for or while loop and MATLAB also employs this syntax. However, an important consideration is that in nested loops, ‘break’ exits only from the loop in which it occurs. In our case, we are nesting a Training Pair loop inside a Training Cycle Loop; we therefore need to terminate both loops. A simple method to break from two loops simultaneously is to create a ‘flag’ and then use that ‘flag’ to trigger a second ‘break’ command. This concept is illustrated below:
Simple method for breaking from nested loops in MATLAB.
for Tloop = 1:1:nTCycles
for TPair = 1:1:nTPairs
% -----------------------------------------------%
% 3: Forward Pass
%- ----------------------------------------------%
if (condition is met)
<< set flag >>
break 1: TPair
end
% -----------------------------------------------%
% 4: Backward Pass
%- ----------------------------------------------%
end
<< this is where control is passed to after ‘break 1’
if (flag is met)
break 2: Tloop
end
end
Note that first, we need to initialise ‘flag’ to zero in setup section of the ANN.
%--------------------------------------------------------%
% 1: Setup ANN Structure
%--------------------------------------------------------%
...
% initialise flag for break
flag = 0;
Writing the first break and setting the flag.
%--------------------------------------------------------%
% Termination Criteria
%--------------------------------------------------------%
if TPair == nTPairs
if ErrLog(Tloop,:) < 1
if sum(ErrLog(Tloop,:)) < 2
flag = 1;
break
end
end
end
Using the flag to trigger the second break
for Tloop = 1:1:nTCycles
for TPair = 1:1:nTPairs
...
end
if flag == 1
break
end
end
Step 3: Setting limits and reporting data
As you have likely noticed when testing out your ANN, there are occasions when then network does not successfully train. This is likely because it has found itself stuck in a local minimum or has entered some cyclically stable condition.
Figure 2: Example of a local minima
Figure 3: Example of a cyclically stable untrained condition
In step 2, we programmed a method for terminating the program once the ANN has been trained. In this final step we shall increase the number of training cycles for our program but set an appropriate upper limit so that the program does not find itself stuck in an unbounded loop. We shall also program a variety of program closing reports, that provide the user with information pertaining to the success or failure of the ANN in training.
There is no definitive guide to choosing a maximum number of loops for your ANN. On some occasions, you may be able to run the program several times to get an idea of the meantime to train. The maximum should be set above this number to allow the program to train, but not so far above this value that the compute time for training is unacceptably loop. For our program, I am going to set the maximum number to 50, with a training weight of 0.1.
% Training pairs, cycles and training rate (N)
nTPairs = size(InputsABC,1);
nTCycles = 50;
N = 0.1;
We shall write a section of code at the end of our program to display termination information to the user.
Recollect that in the previous Step we created a flag (necessary to terminate nested loops) that we can now use to indicate whether training has been successful.
%--------------------------------------------------------%
% 8. Output Report
%--------------------------------------------------------%
if flag == 1
% Output on Success in Training
elseif flag == 0
% Output on Failure to Train
end
First, we shall create a simple message that tells the user when training has failed.
elseif flag == 0
% Output on Failure to Train
disp (...
strcat('The ANN has failed to train.', ...
' The user should consider re-running', ...
' the program or adjusting the ANN', ...
' setup parameters.'));
We shall also create a similar message that tells the user when training has been successful and how many training cycles were completed.
if flag == 1
% Output on Success in Training
disp (...
strcat('Training completed following', ...
" ", num2str(Tloop), ...
' training cycles.'));
This disp command sends an output message to the command line:
To close the program, we want to be able to write the parameters of the ANN to an external ‘.txt’ file so that the user could re-create the network at a later date, pre-trained for letter recognition.
We shall once again use the commands ‘fopen’, ‘fprintf’ and ‘fclose’. Further information in the use of these commands can be found at the MATLAB resource:
if flag == 1
% Output on Success in Training
disp (...
strcat('Training completed following', ...
" ", num2str(Tloop), ...
' training cycles.'));
% write output file
fileOut = fopen('OutputReport.txt', 'w');
fprintf(fileOut, 'The ANN has been successfully
trained using the following structure: \n');
fprintf(fileOut, 'Input Neurons:\t\t');
fprintf(fileOut, '%i\n', nInputs);
fprintf(fileOut, 'Midlayer Neurons:\t');
fprintf(fileOut, '%i\n', nNeurMid);
fprintf(fileOut, 'Output Neurons:\t\t');
fprintf(fileOut, '%i\n', nNeurOut);
fprintf(fileOut, '\nThe following weights are
applied to the input to midlayer neurons: \n');
fprintf(fileOut, '%f\n', layer1w);
fprintf(fileOut, '\nThe following weights are
applied to the midlayer to output neurons: \n');
fprintf(fileOut, '%f\n', layer2w);
fclose(fileOut);
The user can open and review the output file from within MATLAB or directly through the PCs file explorer:
