So demands “The Physician’s Pledge” (the “Declaration of Geneva”¹) in its new 2017 version, also known as the Hippocratic Oath. But are the procedures used in medicine still compatible with this – or is our society now threatened by technical selection? Given the recent use of various forms of triage methods, among them algorithm-based and data-driven ones, serious concerns arise.

Discussion of Some ML-Related Issues in the Health Care System

In view of the possible or even actual use of algorithms for life-or-death decisions³ and the serious issues with the underlying predictive models, there looms a dangerous diffusion of responsibility with unforeseeable consequences for our society. To illustrate the problem, we describe below – in a simplified way – some crucial elements of the process chain in the health care system and how they interact with each other.

• The reality in hospitals includes capacity constraints, as the COVID-19 crisis has revealed again. In such circumstances, prioritisation decisions are implicitly or explicitly made. In the worst case, these can be triage decisions resulting in the death of people, who would otherwise be saved under normal circumstances.

• Prioritisation decisions are often serious and difficult. It cannot be assumed that all people involved in the related decisions do always fully understand all their ethical and legal implications, so mistakes can easily happen.

• For such and other reasons, medical personnel is increasingly supported in their decisions by measured data and expert systems or even AI systems. This is often very helpful. However, for certain decisions, such decision support can be problematic, because such tools are often created by software developers, who have limited knowledge of medical, legal, ethical, and societal issues.

• The algorithms are typically subject to business secrecy and are frequently updated, such that their effects are partially unknown and, in some cases, are not verifiable due to a lack of algorithm transparency. Accordingly, their results may change unpredictably. Nevertheless, algorithms may exert “epistemic authority”: the compliance with algorithmic recommendations often seems to be advised. It may also appear to be an advantage, in case a lawsuit later occurs.

• Business secrecy is not the only reason why expert or AI systems are often lacking transparency. Machine learning systems have often been characterised as “black boxes” [20], because in contrast to what is common in science and in medical research, ML/AI-based systems are typically not based on transparent, validated, and reproducible causal relations. Moreover, as machine learning progresses, ML-based decision-making and its outcomes may change, without any explanation given. Although machine learning programs have high potential for specific tasks such as skin cancer detection [21], they can also sometimes fail dramatically. “Black-box medical algorithms” should, therefore, be monitored closely; the accuracy of predictions is not granted and requires continuous external validation.

• In concrete applications, medical decisions are often based on criteria such as the expected effectiveness of the measures taken. In this context, decisions are recommended by algorithms that judge between different treatment options – including the possibility of no treatment. The underlying machine decisions are typically based on mean values, such as those obtained in extensive medical studies. However, such mean values are not suited to make precise forecasts on the individual level, because the variation in the data is often large. In other words, a treatment, which is better on average, may actually be worse for certain individuals.

• Triage decisions may also be made on the basis of data that was never collected and intended for triage decisions. As a result, data quality can be insufficient or the application context of the data can be inappropriate. Therefore, the application in a Triage context is likely to be problematic.

• Data-driven decisions are often very sensitive to details of the algorithm used or the dataset evaluated. In other words, taking a data-driven decision using a different dataset or a different algorithm (as other research teams might provide them) may result in pretty different priorities. Consequently, the selection of patients who are disadvantaged by triage decisions or medically prioritised can greatly vary with the procedures, algorithms, or datasets used. Hence, the approach of data-driven decisions may suffer from an undesirable, but hidden degree of randomness and arbitrariness, which is particularly worrying when it comes to life-and-death decisions. Therefore, “data-driven” or “evidence-based” does not automatically mean that the resulting decisions are scientifically objective and sound at the level one would require for grave decisions. Despite these methodological shortcomings, vital decisions are increasingly taken on the basis of algorithms.

• The legitimacy and meaningfulness of the algorithms and personal data used in such contexts may be questioned as well. For example, some of these algorithms seem to be based on life expectancy ⁴, even though this may vary a lot.⁵ There are at least two undesirable consequences: (1) While life expectancy is actually quite variable [22],⁶ the application of algorithms to decide the level of life support may determine the lifespan pretty much in the sense of a self-fulfiling prophecy. (Thereby, “good” and “bad luck” would be largely eliminated – and hope as well.) (2) While the expectation of a long life may be rewarded, the expectation of a short life would be almost like a ”death sentence”, particularly when the prediction is incorrect.⁷

• If algorithms were developed with the aim to meet limited budgets or reduce costs,⁸ this could imply that AI-based technological selection might shorten lifespans systematically. This is a serious threat since, after all, Big Data-driven medical decisions are often also supposed to ensure the efficient use of available funds. Such a development can shift the focus away from human dignity and equality to economic profit, and it may obfuscate the rationing of health-care resources.

• If, in addition, individual insurance coverage (the treatments reimbursed by the respective health insurance) is taken into account, social selection could happen.⁹ Furthermore, as known from recent AI research, such discriminatory effects [23] can even occur where not intended at all, namely as a result of the opacity of the algorithms or the training data used.¹⁰

• Despite all this, in the context of the current pandemic, triage decisions involving life-and-death decisions are increasingly being made.¹¹ They are often presented as inevitable in view of insufficient medical capacities.¹² Nevertheless, triage decisions are considered to be extremely problematic and acceptable only in war times or extreme disasters.

In conclusion, the application of algorithms for life-and-death decisions can imply technological selection and a mechanisation of death¹³ [6], particularly if these are allowed to act autonomously.

• Nevertheless, there is a trend to transfer problematic decisions to data-driven algorithms and autonomous systems. In some countries, algorithm-based triage decisions seem to be already in use.¹⁴ However, this makes human beings objects of machine decisions, which is not compatible with human dignity. Besides, the transfer of human responsibility to machines undermines the principle of accountability.

• Political decisions limiting the health-care budget will inevitably determine how many people are ultimately affected by triage decisions, even if not intended.¹⁵ However, algorithm-based triage will also depend on a person’s health condition and, hence, on the person’s social conditions and genetics. Therefore, algorithm-based life-death decisions, i.e. technological selection, may also imply social and eugenic selection.

• The COVID-19 pandemic has demonstrated the threat of introducing (at least partially) automated procedures for life-death decisions into everyday clinical practice. Thereby, triage measures would no longer be limited to situations of wars and serious disasters, but become a “new normal”. We consider such uses of triage procedures as unconstitutional, which must be prevented by political and legal oversight.

Threats of Algorithmic Decisions

We have explained above why undesirable, potentially hidden, genetic and social selection effects are to be expected in a health-care system, which is increasingly operated in a data-driven way. To minimise these, we suggest that medical institutions should regularly produce statistical evaluations of any genetic, social, and behavioural selection effects. Moreover, the application of algorithms as well as the algorithms’ functioning should be regularly audited. Discrimination as well as the non-compliance with state-of-the-art quality standards should be sanctioned. Otherwise, there is an acute danger that those people, who need our medical support and solidarity most, will be particularly disadvantaged by the data-driven systems emerging today.

In our opinion, current data-based approaches do not adequately meet constitutional requirements (principle of equality, self-determination, right to life) and policy goals (such as equity) today. They are also not scientifically sound enough (given the sensitivity of many data analytics approaches). To unfold their full benefits, it must be ensured that Big Data, Artificial Intelligence and other technological innovations are used in a fair way – and that problematic applications are avoided. Otherwise, it is to be feared that the democratic principle of equality will increasingly be replaced by discriminatory scoring systems, which would ultimately lead to a fundamentally different society.

Although constitutional and fair data-based methods may be possible to develop, the methods currently in use do not meet all justified expectations. If one takes the principle of equality seriously, according to which no human life should count more than another, then the blessings of modern medicine would have to benefit all people equally. This fairness requirement would imply that health-disadvantaged people with a shorter life expectancy should get greater medical support per year, not less. This, however, would require a completely different use of data-based methods and different goal functions.

Note that our assessment is supported by the recent WHO Guidance on “Ethics and Governance of Artificial Intelligence for Health”, which points out:¹⁶ “Use of computerized decision-support programs – AI or not – to inform or guide resource allocation and prioritization for clinical care has long raised ethical issues. They include managing conflicts between human and machine predictions, difficulty in assessing the quality and fitness for purpose of software, identifying appropriate users and the novel situation in which a decision for a patient is guided by a machine analysis of other patients’ outcomes. In some situations, well-intentioned efforts to base decisions about allocations on an algorithm that relies only on a rules-based formula produce unintended outcomes”. In its section on “The ethics of resource allocation and prioritization” it also warns that “if an AI technology is trained to ‘maximize global health’, it may do so by allocating most resources to healthy people in order to keep them healthy and not to a disadvantaged population”. Last, but not least, “Use of AI tools for triage or rationing is one of the most compelling reasons for ensuring adequate governance or oversight. Although intentional harm is not ethically controversial – it is wrong – the possibilities of unintended bias and flawed inference emphasize the need to protect … people and processes from computational misadventure”.

Technological and Ethical Awareness Needed

AI systems will certainly play an increasing role in the medical profession. Augmented intelligence has a large potential in assisting medical doctors, medical personnel, and patients in health-related decision-making. These systems can also raise the quality and cost-effectiveness of health care. However, there are a lot of caveats concerning fairness, equality, and bias in the treatment of certain patient groups, regarding the accuracy and validity of results, unintended side effects, and the transparency of algorithms. Therefore, the use of AI systems in practice will typically raise serious ethical questions.

Health care professionals must be aware of the underlying principles, possible weaknesses, unintended consequences, and ethical problems when applying AI systems. After all, they are responsible for the results. Thus, in a recent report, the US National Academy of Medicine demands: “Develop and deploy appropriate training and educational programs to support health care AI” [24]. These programs have to be integrated in the medical education curricula and they should become a high priority in the continued education of health care professionals.

AI courses for continued medical studies are already offered by certain medical schools (e.g. Massachusetts Medical Society, The Radiological Society of North America, Mayo Clinic, Stanford University School of Medicine) [25]. However, courses in continued medical education should, besides a focus on engineering and technical issues, place a higher emphasis on understanding the principles of algorithms and on reflection of related ethical problems. Consequently, not only medical experts and engineers should be responsible for teaching AI courses in medicine. It is highly important that AI programs in medical studies and in continued education “must be multidisciplinary and engage AI developers, implementers, health care system leadership, frontline clinical teams, ethicists, humanists, patients, and caregivers” [24].