Multi-criteria decision analysis explained: methods, a Python example and tools

Most real decisions have more than one right answer. The cheapest laptop is the heaviest. The most effective treatment has the most side effects. The site with the best transport links has the highest rent. When the options trade one good thing against another, there is rarely a single option that is best on every count, and someone has to decide how much each count matters.

Multi-criteria decision analysis (MCDA) is the set of methods for making that kind of decision in a structured, transparent way. This post covers the basics: the vocabulary, the main families of methods, a worked example in a few lines of Python, and the tools and libraries that do the work for you. Part 2 then shows how to check whether a ranking is trustworthy, with my open-source library mcdakit.

What makes a decision multi-criteria

Every MCDA problem has the same four ingredients:

• Alternatives: the options you are choosing between. Here, five laptops.
• Criteria: what you judge them on. Here, price, battery life, performance and weight.
• Direction: whether more is better (a benefit criterion, like battery life) or less is better (a cost criterion, like price).
• Weights: how much each criterion counts relative to the others.

Written down, they form a decision matrix: one row per alternative, one column per criterion.

Three kinds of question

Before choosing a method, it helps to be clear about the question. Bernard Roy, one of the founders of the field, distinguished several kinds of decision problem. Three of them cover most practical cases:

• Choosing: find the best option, or a short list of good ones.
• Ranking: put all the options in order, for example research proposals competing for funding.
• Sorting: put each option into a predefined class, such as accept, review or reject. Sorting judges each option against fixed standards rather than against the others, so adding a new option does not change where the old ones belong.

Sorting is less well known than ranking, but it is often what people really need. In my own early work we used ELECTRE Tri, a sorting method, to sort the association rules found by data-mining algorithms into classes from “interesting” to “useless”, because there were far too many to rank one by one (paper, follow-up).

The first step: remove what no one would choose

One option dominates another if it is at least as good on every criterion and strictly better on at least one. A dominated option can never be the right choice, whatever the weights, so it is worth finding before anything else. It is also the only step in MCDA that needs no weights at all.

import numpy as np

laptops = ["Atlas", "Breeze", "Cobalt", "Dune", "Ember"]
criteria = ["price (EUR)", "battery (h)", "performance", "weight (kg)"]
X = np.array([[1400, 14, 82, 1.3],       # Atlas
              [ 900, 10, 65, 1.4],       # Breeze
              [1900, 12, 95, 1.6],       # Cobalt
              [ 700,  6, 60, 2.2],       # Dune
              [1500, 11, 80, 1.5]])      # Ember
benefit = np.array([False, True, True, False])   # more battery and performance is better;
                                                 # lower price and weight are better
weights = np.array([0.35, 0.25, 0.25, 0.15])

def dominates(a, b):
    """a is at least as good as b on every criterion, and strictly better on one."""
    at_least = np.where(benefit, X[a] >= X[b], X[a] <= X[b])
    better = np.where(benefit, X[a] > X[b], X[a] < X[b])
    return at_least.all() and better.any()

for a in range(len(laptops)):
    for b in range(len(laptops)):
        if a != b and dominates(a, b):
            print(f"{laptops[a]} dominates {laptops[b]}")

Atlas dominates Ember

Atlas is cheaper, longer-lasting, faster and lighter than Ember, so nobody who understood the table would buy Ember. We keep it in the example anyway, because real decision matrices often contain options like this one.

Putting criteria on the same scale

Price is measured in euros, battery in hours and weight in kilograms. Before they can be combined, they have to be put on a common scale. This is normalisation, and there are several ways to do it:

• Min-max: the best value on each criterion becomes 1 and the worst becomes 0, with everything else in between.
• Vector: each value is divided by the length of its column, which keeps the proportions between values.
• Sum and max: each value is divided by the column’s total or by its largest value.

The choice sounds technical, and it can change the answer, as we will see below.

Weights

Weights say how much each criterion counts. There are two broad ways to get them.

From people. The decision-makers say what matters to them. They can give the weights directly, or answer easier questions from which the weights are calculated. In AHP, the Analytic Hierarchy Process, they compare criteria two at a time (“is price more important than battery life, and by how much?”), and the method checks whether their answers are consistent with each other. In swing weighting, they imagine every criterion at its worst level and say which one they would most like to move to its best level. That one gets the largest weight.

From the data. Objective methods such as entropy and CRITIC give more weight to criteria on which the alternatives differ more, or which carry information the other criteria do not. They are useful when there is no decision-maker to ask. They measure the data, though, not anyone’s priorities.

One common mistake is worth knowing about. A weight is not an abstract “importance”; it is a trade-off over the range of the data. If every laptop weighs between 1.3 and 1.4 kg, weight cannot decide anything, however much you say you care about it. Swing weighting builds this in, which is why decision analysts like it.

Five families of methods

There are dozens of MCDA methods, but most belong to a handful of families.

• Value scores. Turn each criterion into a score and add them up, weighted. The weighted sum is the simplest MCDA method there is, and MAUT (multi-attribute utility theory) is its rigorous form, with carefully built value functions for every criterion.
• Distance to a reference. TOPSIS prefers options that are close to an imaginary ideal option, the best on every criterion, and far from the anti-ideal, the worst on every criterion. VIKOR looks for a compromise close to the ideal. SPOTIS measures distance to an ideal defined by fixed bounds, which keeps rankings stable when options are added or removed (more on that in Part 2).
• Outranking. Compare the options in pairs, criterion by criterion, and ask how strongly the evidence says A is at least as good as B. PROMETHEE turns these comparisons into preference flows and a ranking. ELECTRE also asks whether B is much better on any single criterion: such a veto can stop A outranking B, however good A is elsewhere.
• Pairwise judgements. AHP and its generalisation ANP derive weights and scores from people’s pairwise comparisons. MACBETH asks for qualitative judgements of how much more attractive one option is than another.
• Decision rules. DRSA (the dominance-based rough set approach) learns if-then rules from examples of past decisions. DEX uses rules written by experts over qualitative scales.

Compensation: can a strength make up for a weakness?

The most important difference between these families is compensation. In a weighted sum, a very good score on one criterion can make up for a very bad score on another. Look at Dune: it is the cheapest laptop by far, and the worst on everything else.

Whether that is acceptable depends on the decision. For a laptop, perhaps it is. For a medical treatment, a severe side effect should not be “compensated” by a low price. Non-compensatory methods, such as ELECTRE with a veto, or a simple minimum requirement, prevent it:

ok = X[:, 1] >= 8                       # a requirement, not a trade-off: at least 8 hours
print("passes the battery requirement:", [l for l, keep in zip(laptops, ok) if keep])

passes the battery requirement: ['Atlas', 'Breeze', 'Cobalt', 'Ember']

A minimum requirement is not a weight. It is a rule that removes options before any trade-off is made, and it is often the most honest way to express a criterion that is not really negotiable.

A worked example: weighted sum and TOPSIS

Here are the two most widely used methods, written out in plain numpy so that nothing is hidden. First, min-max normalisation and the weighted sum:

# Min-max normalisation: the best value on each criterion becomes 1, the worst 0
low, high = X.min(axis=0), X.max(axis=0)
N = np.where(benefit, (X - low) / (high - low), (high - X) / (high - low))

wsm = N @ weights                                  # the weighted sum
for i in np.argsort(-wsm):
    print(f"{laptops[i]:<7} {wsm[i]:.3f}")

Atlas   0.703
Breeze  0.586
Cobalt  0.537
Ember   0.532
Dune    0.350

Next, TOPSIS, with the vector normalisation it was originally defined with:

def topsis(X, weights, benefit):
    V = X / np.sqrt((X ** 2).sum(axis=0)) * weights          # vector normalisation, weighted
    ideal = np.where(benefit, V.max(axis=0), V.min(axis=0))  # best value on every criterion
    anti = np.where(benefit, V.min(axis=0), V.max(axis=0))   # worst value on every criterion
    d_ideal = np.sqrt(((V - ideal) ** 2).sum(axis=1))
    d_anti = np.sqrt(((V - anti) ** 2).sum(axis=1))
    return d_anti / (d_ideal + d_anti)                       # 1 = at the ideal, 0 = at the worst

closeness = topsis(X, weights, benefit)
for i in np.argsort(-closeness):
    print(f"{laptops[i]:<7} {closeness[i]:.3f}")

Breeze  0.665
Dune    0.574
Atlas   0.573
Ember   0.444
Cobalt  0.371

The two methods agree on very little. The weighted sum picks Atlas. TOPSIS picks Breeze, and moves Dune from last place to second. The two normalisations explain much of this. Min-max stretches every criterion to the full range from 0 to 1, so Dune’s poor battery and weight count fully against it. Vector normalisation keeps the original proportions. In those proportions, the spread of prices is about twice the spread of performance scores, so Dune’s price advantage carries more weight.

Notice also that Dune and Atlas are separated by less than 0.002 under TOPSIS. A difference that small is noise, not a preference, and a small change in any weight would reverse it.

The same, with a library

You rarely need to write these methods yourself. With the open-source library pymcdm (pip install pymcdm), TOPSIS is one line, and it is worth comparing it with ours:

from pymcdm.methods import TOPSIS
from pymcdm.normalizations import vector_normalization

types = np.where(benefit, 1, -1)        # pymcdm marks benefit criteria 1, cost criteria -1
print("pymcdm, default:     ", TOPSIS()(X, weights, types).round(3))
print("pymcdm, vector norm.:", TOPSIS(vector_normalization)(X, weights, types).round(3))
print("our function:        ", topsis(X, weights, benefit).round(3))

pymcdm, default:      [0.617 0.575 0.477 0.477 0.494]
pymcdm, vector norm.: [0.573 0.665 0.371 0.574 0.444]
our function:         [0.573 0.665 0.371 0.574 0.444]

With vector normalisation, pymcdm gives exactly our numbers. With its default settings it gives different ones, because pymcdm’s TOPSIS uses min-max normalisation by default. Neither is wrong, but they answer slightly different questions, and with min-max normalisation pymcdm ranks Atlas first, not Breeze.

This is the main practical lesson of MCDA software: the same method name does not guarantee the same procedure. Defaults for normalisation, for ties and for how cost criteria are handled differ from one tool to another. Always report which tool, which version and which settings you used.

Tools and libraries

MCDA is well served by free software. These are the tools I would look at first. All the links were checked when this post was written.

Python

R

• topsis: TOPSIS for R (small, and unchanged since 2013).
• ahpsurvey: AHP for survey data, where many respondents each give their own comparisons.

Software with a graphical interface

• Super Decisions: free software for AHP and ANP.
• DEXi: free software for the DEX method, with qualitative rules; its successor is DEXiWin.
• M-MACBETH: software for the MACBETH method.
• 1000minds: a commercial web tool that derives weights from a series of simple pairwise choices (the PAPRIKA method).
• diviz: from the Decision Deck project, for building MCDA workflows from ready-made components.

Researchers often also build platforms for one domain. With colleagues, I built DM-MCDA, a web platform that combines data mining with MCDA to analyse road-accident data. We also used MCDA to choose between symmetric encryption algorithms for securing smart grids (paper).

How to choose

• For a one-off decision with stakeholders in the room, a graphical tool built around one well-understood method (AHP, MACBETH or DEX) is usually best. People trust what they can see.
• For research, or for repeated decisions, use a library in a script, so that the analysis can be rerun, checked and reported exactly.
• Whatever the tool, check its defaults, and report the method, the normalisation, the weights and where they came from.

Usability matters too: a decision-support tool that people cannot use correctly produces confident nonsense. We recently looked at quick ways to test the usability of such tools (paper).

What comes next

This post ended with two methods disagreeing, two options separated by less than 0.002, and two libraries giving different answers to the same question. That is the normal state of an MCDA study, not a sign that something went wrong. The real question is whether your conclusion survives:

• other reasonable weights;
• other methods;
• options being added or removed.

Part 2 answers those questions step by step with mcdakit.

Further reading

• Belton and Stewart (2002). Multiple Criteria Decision Analysis: An Integrated Approach. The best introduction to the field as a whole.
• Greco, Ehrgott and Figueira, editors (2016). Multiple Criteria Decision Analysis: State of the Art Surveys. A chapter on every major family of methods.
• Keeney and Raiffa (1976; Cambridge reprint 1993). Decisions with Multiple Objectives. The classic on value and utility.
• Roy (1991). The outranking approach and the foundations of ELECTRE methods.
• Brans and Vincke (1985). A Preference Ranking Organisation Method. PROMETHEE.