Multi-agent models (MAS) are increasingly popular as a tool to provide support for scientists and industrialists to develop their understanding of the natural systems, to make forecasts, and to predict the impact of potential future changes. The increasing popularity of the paradigm strongly suggests the need for rigorous model testing and testing tools, not least because simulation errors can potentially have very high consequences. In the past, scientific software errors have led to paper retractions, financial loss, fatal consequences such as damage to costly infrastructure and loss of lives.

In order to gain confidence that a model is implemented correctly, several simulation executions need to be performed and inspected. In practice, however, the inspection process tends to be performed manually, which is a tedious, time-consuming and prone-to-error process.

In this project, we developed a MAS testing tool that aims to automatically identify 'suspicious' simulation executions, which may be due to implementation faults in an underlying MAS based model. The testing tool works by observing a series of simulation executions believed by the modeller to represent the 'normal' behaviour of the model. The modeller then specifies a set of assertions that place bounds on particular simulation properties of the model which may deviate from that already observed in the normal behaviour. In addition, a series of so-called `facts' about the model may also be specified - as states of the simulation which should never occur. The tool then extensively monitors further, incorporating statistical analysis, simulation executions of the model automatically flagging up executions that deviate from normal behaviour or violate some specified fact.

Two case studies were used to demonstrate the effectiveness of the testing tool. For the first case study, a Flockers model was used. It is supplied with MASON (a widely used multi-agent development framework in Java). Our testing tool successfully identified 'suspicious' simulation executions when the model's code was mutated to introduce small faults. On the second case study, a model for simulating skin tissue (known as the Keratinocyte) was implemented using FLAME (a multi-agent development framework in C language). Noise was injected into the model, the presence of which was correctly identified by our testing tool.

The testing tool evaluation was performed by applying fault injection techniques on the data of the MAS models under testing (both noise injection and mutation were applied).

As part of this project, a seminar on applications of MAS was organised across the entire University of Sheffield. The seminar was very successful and attracted a considerably large audience among several academic departments and research groups including social sciences, politics, biology, biochemistry, genetics, neuroscience, physics, computer science, mathematics, economics and psychology.

The skin tissue model is used to simulate colonies of skin cells on a laboratory culture plate. The simulation begins with a few randomly seeded individual cells, which form the epicentre of a colony. In each time step of the model, cells progress through the cell cycle and divide, producing new cells. Colonies grow outwards from the initial cell to eventually covering the entire plate. One important aspect of the model is the so-called 'differentiation' of a skin cell from one type to another (e.g. 'corneocyte' skin cells are found in the uppermost layers of skin tissue). In the model, cells change type based on their distance from the centre of the skin cell colony of which they are a part.

For the purpose of evaluating the testing tool, a function was introduced into the model which applied a random proportion of noise to this distance property, thus introducing a source of potential simulation error into the model.

The model was then run for 1000 time steps, with 50 runs performed for tracking and 10 repetitions with three proportional noise levels (low, medium and high). Six skin cells were initially seeded for each simulation run at random locations on the culture plate. In tracking and testing the model, the distance attribute before a cell makes its first differentiation (into another skin cell type) is monitored. The results showed the number of observation violations that occurred when the distance attribute strayed over 2 standard deviations from the mean found for the property during the tracking step. Low levels of noise (up to 1% of proportional noise applied to the attribute during testing) resulted in little difference from the original model (without noise), but several violations occurred with higher levels of noise (10-100%) and as such are easily detected by our testing tool.

The Flockers model in MASON simulates a number of agents exhibiting coordinated movement with one another, as seen with natural flocks of birds or shoals of fish. Each flocker agent takes into account local spatial information when deciding which coordinates to move to in the next time step, including the direction and momentum of the flockers around it, and the need to avoid colliding with other Flockers, coupled with a small degree of random movement. The model also includes optional 'dead' flockers that do not move; but live flockers must try to avoid colliding with them.

An experiment was performed with the Flockers model using Mutation Analysis (MA) to insert, into program code, small syntactic changes designed to mimic typical errors made by programmers (such as typing "&" or "=" instead of "&&" or "==" respectively). The use of mutation allowed us to artificially inject errors into models, resulting in potentially faulty simulation runs.

The testing tool successfully identified 'suspicious' simulation executions when the model's code was mutated to introduce small faults.