Are algorithms too complex?

It is no news that algorithms and mathematical models have become a force that shapes (and sometimes drives) our choices these days. From the results from your favourite search engine, the models that are responsible for most of the transactions in the stock markets, the pricing in airlines and uber or our suggestions at Netflix or Amazon, algorithms are everywhere.

If you like TED talks and you like podcasts then, first check Pint of Science. Our podcasts are more about science and less about entertainment and design (maybe more T and little ED). But if you have spare time then check the TED podcasts at NPR. One of the more recent ones is called Solve for X and includes excerpts of a talk by Kevin Slavin warning us about the increase in the complexity of the algorithms that shape (and will soon rule) our lives.

http://www.npr.org/player/embed/388518887/390992496

And here is the thing: as a computer scientist and mathematical modeller I believe in the potential that these algorithms have to change our lives for the better. But I am also quite concerned about increasing the complexity of certain algorithms when they are effectively black boxes. As an example of this problem, Slavin refers to a recent and massive market crash at the NYSE that was caused by one of these algorithms. Having a complex algorithm in charge of, say, Netflix recommendations is not a big deal: it is not the end of the world if some of us get recommended Wolf of Wall Street when we would rather watch Margin Call for instance. It is a different matter when an algorithm can make billions of dollars disappear from the stock market or, closer to home, determine (or guide) the clinical strategy for a patient at Moffitt. In those situations we need algorithms and models whose complexity can be understood so that when something wrong happens (as it is bound to happen) then we can learn and adapt the algorithm accordingly. Modelling is not about capturing all the complexity of a system (be it the stock market, our your taste in movies and music) but about having a framework in which to place all the key variables. This allows us to get not only predictions but also understanding which is a lot more powerful.

2 thoughts on “Are algorithms too complex?

  1. I think the matter is a bit more complicated. In the case of healthcare we are replacing doctors with algorithms (that are too complex to understand). This is only an issue if we, in the first place, understand the reasoning of the doctors. And I don’t think we do. A lot of clinical practice today is, for good and bad, based on experience passed down from other doctors, and not on simple understandable rules.

  2. Thanks Philip, I guess one person’s complexity is another’s simplicity. I think that the TED talk I was referring too mentioned these highly complex and non-heuristic. A weighted combination of heuristic rules could be useful in many respects…even if they seem incoherent, as at least we can keep adjusting their weights, adding more or subtracting some as we see fit.

Leave a reply to David Basanta Cancel reply