Lethal Automated Robotic Systems and Automation Bias

Lethal Autonomous Robotic Systems (LARS) are machines that are capable of initiating a lethal attack on individuals or other targets. Based on its programming, a LARS can determine whether an individual is a valid target and whether engaging that target is a proportional action, and act upon its own assessment. Such sophisticated systems have long been in the realm of science fiction, but today they are not only a possibility, but a reality. For example, Samsung has developed the SGR-A1, which is currently deployed in the Korean demilitarised zone. Although, for now, that device leaves the final decision to engage to a human.

The debate on the use of such systems is heating up (see for instance the various reports by Human Rights Watch, the Oxford Martin Policy Paper, or discussions on the topic in relation to the CCW). These systems have been criticised from moral, political and legal perspectives. Leaving aside the moral and political objections, the development of a LARS is extremely problematic from the perspective of international humanitarian law. In particular, questions have been raised about the ability of such systems to make distinctions between civilians and combatants, as well as computing the proportionality of an attack. Furthermore, there are complex responsibility questions that are as yet not fully answered.

In response to these problems, the US has issued a directive that all robotic systems of this type will in fact not be operated in a fully autonomous mode, but will always function with a ‘human in the loop’. This statement is apparently intended to undermine at least the legal, and possibly the other criticisms relating to the deployment of LARS.

Human in the loop

It could be argued, however, that the deployment of a LARS with a human in the loop is just as problematic as a fully automated version. While the decision to engage a target will always be overseen by a human being, I will argue that it is not a given that this will in fact influence the functioning of the system sufficiently to adequately safeguard against the problems associated with the fully automated settings.

Firstly, the term ‘human in the loop’ is not very specific. There are a variety of ways in which a system can operate with a human in the loop. For instance, each LARS could be operated by a ‘handler’ who continuously monitors the actions of the LARS, and whose authorisation would be required for any engagement. Thus, the LARS can function without input, but is constantly monitored, and any decision to engage must be approved by the handler. This is a scenario that is closely related to the current drone operators. The main difference is that the system does not only respond to input from the handler, but has a certain level of independence: in particular, it is capable of making its own assessment of the situation and communicating it to the handler.

An alternative would be to have the human in the loop only at the critical time at which the engagement decision needs to be made, this is often referred to as a ‘human on the loop’. In this case, a human, arguably the commander, will determine if there is indeed a situation that would warrant or allow for the use of force only after a request for permission from the LARS. This would mean that the LARS is deployed and can act independently, be it stationary or otherwise, and will only ‘call home’ in a situation where it believes a target should be engaged. In this scenario, it would be possible for one commander to operate a relatively large number of units simultaneously. This approach has clear benefits for military operational costs. The operator does not necessarily monitor the entire process, but is only notified of a target identified by the system, and is given a short window to override the system and abort the engagement (such as the Phalanx system, as well as a reported function on the Samsung SGR-A1).

This latter approach combines two controversial elements, namely that the human in the loop has not necessarily been monitoring the decision making process of the system, and that the human is in fact not needed to engage the target, but is only able to overrule that decision within a short window. However, the former approach, which on the face of it looks like only a small step from the current drone operations, may in fact be just as controversial, due to a phenomenon known as automation bias.

Automation bias

The development of LARS is in part based on the idea that the use of such automated systems could eliminate any human error from the field of warfare. No more battle fatigue (or PTSD any re-iteration of it you prefer), no spectre of death resulting in bad decisions, no unpredictable emotional responses. This theory relies on the presumption that machines are unlikely to make mistakes, as long as they are functioning properly, and are therefore to be trusted.

The benefit of LARS over current drone systems for instance, is that it is capable of making assessments on its own, it can make decisions or advise certain actions. Thus, the operator of the system is not merely making his or her own decision based on video and audio input, but is to a large extent relying on the analysis made by the system itself.

This reliance on automated systems leads to a documented human response known as Automation Bias (see for example Skitka et al. 2000; Mosier et al. 1998; Cummings 2004). Simply put, human beings have the tendency to believe automated systems are infallible. We rely on them, and we expect their information to be accurate, even when our senses tell us differently. This response is exacerbated in stressful situations, such as aircraft piloting or, related to the present topic, combat situations.

The effect of introducing automated systems has been researched in flight control scenarios (Mosier et al. 1998), where experienced pilots were placed in a simulator which included a new automated detection system for engine fires, which would also advise the pilot on the appropriate action to take. Although they had been informed that the system was imperfect, and that relying on it alone would be dangerous, the pilots tended to follow the systems advice, even in cases where no other indicators in the cockpit suggested taking action. Interestingly, in the debriefing, the vast majority of pilots said they remembered other indicators being present, despite this objectively not having been the case.

This is but one of the many experiments that show that human beings, with all their complex brain capacity, have a tendency to rely heavily on automated systems when they are available, with possible detrimental or even fatal effects.

As this phenomenon tends to increase in cases that are considered stressful or time sensitive (Cummings 2004), there is a very high likelihood that operators of LARS would be subject to this phenomenon. This would mean that, although the human in the loop makes the decision, effectively, this person does so by relying blindly on the analysis and suggestions provided by the LARS, and as such the addition of the human would not necessarily have an appreciable impact on the autonomous nature of the LARS.

The manner in which the system and the human interact is thus of vital importance. Such an interaction must be carefully designed to avoid Automation Bias occurring. A LARS could in theory provide the human with all the relevant data it has gathered to come to its decisions that the target is of a military nature and that the attack would be proportionate. This approach would overwhelm any individual to the extent that it would become impossible to go through all the data within a reasonable time frame, and is therefore unpractical.

Consequently, data will need to be presented in a manner that allows for a proper analysis to be made by the human. The latter, more likely approach, has several pitfalls. Firstly, it will not be clear what assumptions the LARS has made, or which biases may have influenced its conclusion. Thus, the manner in which the LARS collects and represents data would become even more important. The difficulty with Artificial Intelligence systems is that, while they do not have a tendency to go quite as ‘rogue’ as in the movies, their learning pattern cannot be accurately predicted either. Errors may be made, and should be corrected, a process which would not only require complex analysis of data, but also rather more time than is usually available in any real world scenario.

Furthermore, and perhaps more importantly, if the LARS provides data to its handler that it considers it likely that there are currently three individuals who it considers a valid target, and that engaging those targets is likely to be proportional, then due to the automation bias of its handler, it is unlikely that within the short timeframe available to allow successful engagement, the handler will do anything other than follow the assessment of the LARS. This effect is exacerbated in the case of a commander operating multiple machines, as in that case the additional information that the LARS does not transmit may be of great value. All these contextual elements would be lost, meaning that the commander would be even more likely to trust the decision made by the LARS.

With a permanent handler (or shifts of the same), this may be somewhat mitigated. Nonetheless, it may still not be possible for the human to operate at the same speed as the LARS, thus resulting in the handler once again succumbing to automation bias. This would be particularly likely if the LARS is capable of engaging multiple threats at the same time, as it would remain extremely difficult for the handler to check each assessment. Even if it would only move from one target to the next, the timeframe in which the operator would need to make the decision would likely lead to automation bias on the part of the handler.

Conclusion

The deployment of LARS in the field is likely to lead to violations of IHL as the rules on distinction and proportionality require qualitative decision-making, which is very challenging to program. Merely including a ‘human in the loop’ does not in fact diminish this problem as the effects of automation bias can take away the effectiveness of this safeguard.

Thus, while the problems associated with automation bias do not mean that it would be impossible to design a weapons system that is highly autonomous, the effects of human psychology should not be overlooked in weapons testing and certification.