Dangers and Pitfalls of Interpretion

Modelling results can be easily misused. The findings gained through their use can be misintepreted like any other result from research, but model results are typically not exposed to the same degree of statistical rigour as, say, analysis of field results, because it is often assumed that the data upon which the models have been built have already received the necessary level of examination. There are real dangers of focussing on results from your simulations without evaluating them within the strict context of your simulation scenarios. If your results are telling you a message, then that message is only true for the circumstances for your simulation, and in our case, particularly the conditions defined by your climate data.

To generalize to the broader context, for example, independent of specific time-dependent climate conditions, e.g., the timing and severity of droughts given by your data, then you need to accumulate simulation results for a broad range of climate conditions, and then derive summary statistics from them. But the broad range of data might not be readily available to you. No worries.

John von Neumann
One of the pioneers of simulation concepts was John von Neumann. In the late 1940's he conceived of the idea of running multiple repetitions of a model, gathering statistical data, and deriving behaviors of the real system based on these models. This came to be known as the Monte Carlo method because of the use of randomly generated variates to represent behaviors that could not be modeled exactly, but could be characterized statistically.
Source: Simulation Article by Roger D. Smith, modelbenders.com
Taking inspiration from von Neumann, one approach for modelling livestock in semi-arid regions reported in Illius et al. (1998) has been to run replicate simulations varying only the sequence of annual rainfall, such that for a ten year sequence of rainfall data with years
1, 2, 3, 4, 5, 6, 7, 8, 9 and 10
the first simulation will use rainfall data for each year in the order
1, 2, 3, 4, 5, 6, 7, 8, 9, 10
and the second may be in the random sequence
7, 4, 10, 9, 8, 3, 2, 1, 5, 6
and the third might be
6, 10, 7, 2, 3, 5, 8, 4, 1, 9
and so on. This has the effect of breaking up actual patterns of historic climate such as runs of good and bad years. Averaging across all of the randomized replicate simulations gives a result that is applicable to the range of climatic conditions generated through randomization of the rainfall sequence, and if the number of replicates is sufficiently large, this result will tend towards a general result that is applicable for all combinations of rainfall. A useful rule-of-thumb is to select the number of replicates based on the percentage variation of a key output variable, such as animal density, between successively larger numbers of replicates. As this coefficient of variation tends towards perhaps 5% say, then this may be a suitable cutoff for the required number of replicates to obtain stable output estimates for your model.

Therefore, the steps that you should consider to avoid such problems are as much part of the cautious simulation process as they are part of applying the appropriate analysis techniques. Therefore we are posed with a problem that forces us to tradeoff our confidence in how accurate our model is in representing the system in question, and how general are our findings that result from using it.

Limitations of Your Site Area

Consider a group of pastoralists who traditionally tend primarily cattle with a few goats. They herd their livestock to-and-from water and grazing daily, and keep them safe in pens over night. Modelling this scenario is a straight forward prospect in SimSAGS and we can be confident in our abilities to comment on the capacities of these people to survive unfavourable rainfall conditions.

Now introduce a bit if history - traditionally these people have used various coping mechanisms to deal with drought. Vital to their long-term survival has been the seasonal movement from their wet season grazing sites to more persistant dry season sites that support better forage resources outside the growing season, especially because they have been left to rest during the wet season. This now poses our model with a problem. Either we,

(a) ignore this influence that is external to our current site area and undoubtedly end up with overly pessimistic predictions from our model,


(b) set up another region within our model to account for this external area

but there may be data limitations for this other area, etc. etc. etc. Problems, problems, but you see what the trouble is - it is important to interpret your results only within the context of your model parameters, including its geographic boundaries. If external factors are important in the real world but are omitted from your model, then your predictions for the real world may be compromised.
XHTML 1.0 6sm Get PFE