Steve Furber | Computing & Neuroscience

In this inaugural episode of Computing Across Disciplines, we talk with Steve Furber, Professor of Computer Engineering at the University of Manchester. He’s probably most well known for designing the ARM processor, but his more recent research has led him in a surprising new direction: simulating the human brain. In our conversation, we talked about his role on the EU Human Brain Project, his latest architecture SpiNNaker, and his experiences working at the intersection of computer science and neuroscience.

Bio

Photo of Steve Furber Steve Furber CBE FRS FREng is ICL Professor of Computer Engineering in the School of Computer Science at the University of Manchester, UK. After completing a BA in mathematics and a PhD in aerodynamics at the University of Cambridge, UK, he spent the 1980s at Acorn Computers, where he was a principal designer of the BBC Microcomputer and the ARM 32-bit RISC microprocessor. Over 100 billion variants of the ARM processor have since been manufactured, powering much of the world’s mobile and embedded computing. He moved to the ICL Chair at Manchester in 1990 where he leads research into asynchronous and low-power systems and, more recently, neural systems engineering, where the SpiNNaker project is delivering a computer incorporating a million ARM processors optimised for brain modelling applications.

Transcript

Andrew Miller:               So, thank you for talking with us. I guess to just get started, I’m interested in what really first got you interested in computers or computer science.

Steve Furber:               Yeah, my background is in mathematics, I started as an undergraduate reading maths, but I’d always had an interest in aviation and flying, and so for my PHD I slid sideways into the engineering department and did a PHD on fairly theoretical aerodynamics. In the course of that I was sort of intrigued by the idea of flight simulation, and sort of got this idea in my head that it might be possible to build a simple flight simulator.

And in following that idea, I heard about a bunch of students who were forming a new university student society called the Cambridge University Processor Group, and that basically was a bunch of students who built computers for fun. And so, I wasn’t a founder of that group, but I got involved from the outset and started building microprocessor-based systems, initially with the objective of using these to implement some kind of flight simulation, but fairly rapidly it became an objective in its own right, I just got interested in the process of building computers from scratch.

Andrew Miller:             Would you classify yourself as a mathematician first, a computer engineer, an electronics engineer, some other something?

Steve Furber:               I think I’m primarily a microchip architect or designer. I still of course use mathematics from time to time, but my formal mathematics is now a long time in the background and I’ve spent quarter of a century in academia, and 10 years before that in computer engineering.

Andrew Miller:             Yeah, so let’s not bury the headline. You were the original designer of the Acorn Risk Machine, ARM processor. I imagine that most of the people listening to this are listening to it on a device that’s powered by that processor. Could you give us just sort of a quick high-level explanation of what distinguishes that design from previous systems?

Steve Furber:               Yes, the ARM grew out of Acorn Computers, which was the small British company that built educational desktop machines, a bit along the lines of Apple in the US. And we got the contract to build the BBC microcomputer, which was the first computer to go in significant scale into UK schools. And based on the success of the BBC micro, we were looking to design a successor machine, and we surveyed the microprocessors that were on the market at the time and for various reasons, we didn’t particularly like the way any of those microprocessors were put together. So we started toying with the idea of designing our own microprocessor.

Now at that time we thought microprocessor design was a bit of a black art, it was done by big semiconductor companies with lots of resource, and Acorn even at its peak was only about 400 people. But then we were introduced to the papers from Berkeley in Stanford on reduced instruction set computing. So we’re talking here about the early ’80s, about 1983, and the RISC papers had been out for a year or two. And here, microprocessors that were designed to have simplified instruction sets and could be put together by a postgraduate class of students in a year.

So we thought this looked like a great line to go. Of course, they were academic prototypes and we were interested in a microprocessor for commercial use, but we put the RISC idea together with some other ideas we had ourselves and came up with the architecture for what was then called the Acorn RISC Machine, RISC was its middle name. And that of course became ARM, and as you say, in quarter of a century since then, ARM has become extremely pervasive.

Andrew Miller:             I’m interested also in your experience. You were in academia and then sort of moved into this corporate environment for lack of a better word for about 10 years, and then now have been back in academia for a significant time. How do you characterize the way that collaboration happens or the way that development happens in those different environments?

Steve Furber:               Yes, I went through from undergraduate to PHD students to research fellow in academia following the conventional path, but towards the end of that, I was beginning to feel a little bit lost in the academic world of research. I wasn’t really feeling sufficiently strongly based to devise my own research programs and so on, and this computer revolution was just starting up and I’d been dabbling in it as a sort of part-time interest. And so when Acorn got the BBC micro-contract, it was a fairly easy decision to switch from academia to industry.

At that time, Acorn was not particularly corporate, it was only about 25 people. And then my attention of course moved from research to very much focusing on development. I enjoyed that industrial environment where the day to day priorities were much clearer to me than they’d been in academia.

10 years later after spending that decade in Acorn, I’d developed; If you like; a foundation of ideas to take forward and I was beginning to get a bit frustrated about the difficulty of getting interesting things going in industry. Acorn had gone from exponential growth to a fairly flat growth profile. And so, moving back to academia, taking those ideas with me became attractive at that point.

Andrew Miller:             And it seems; Just from my brief research; that it looks like at the beginning of this century you made a shift in your research or your career towards this biologically inspired design. I’m wondering if you can give us some insight into what motivated that shift or if you even see that yourself.

Steve Furber:               I very much see that. When I came to Manchester in 1990, I brought these ideas with me, which were very much focused around building asynchronous implementations of the ARM processor, clock-less designs of the ARM, and we spent the ’90s doing this. By the end of the ’90s, I was beginning to get a bit frustrated with conventional computing, processors were a thousand times faster than when I started, but there were still things that young humans find easy that computers struggled to do, such as recognizing faces and so on. So, that lead me to start thinking about the differences between information processing in a computer and information processing in the brain.

As it happened, I was given a small grant, a condition of which was to explore a new direction of research, and I used that grant to explore the idea of inexact associative memories. I’ve always liked associative memories, CAM’s, we use them on a number of ARM designs, but they’re very brittle. You give them exactly the right input and they give you exactly the right output, but you make any mistake in the input and you get nothing useful out.

And so, I started to think about how you could soften that functional characteristic, and what I realized after looking at this for a year or two, was that every angle I took just resulted in me reinventing artificial neural networks. So I thought, “Well if that’s the answer, I’d better go and look into these neural networks properly.” And that set the course if you like for the next 20 years of my research.

Andrew Miller:             Yeah, I’m interested in this moment of what happens when you realize that your research is going in a direction that touches another area or it’s perhaps a new body of knowledge within your existing area. Is it purely a matter of literature review, do you start going to other conferences, do you start trying to find other people in other departments?

Steve Furber:               Well, I think the first thing you do is you try and absorb the literature in the new area. That turns out to be extremely difficult if the new area is neuroscience, because firstly the literature is vast, there are something like 100,000 papers published every year on neuroscience, so even neuroscientists don’t read them all. And they’re also written in a foreign language. The neuroscience papers are full of complex biochemical terms which are meaningless to computer engineers, so you have to begin the work of learning a new vocabulary.

Obviously mixing with people and talking with people helps. It’s also good to find people who are on the interface. So, instead of diving straight into the deep neuroscience literature, look for the people who are closer to your discipline, the computational neuroscientists are at an intermediate point. There are some books out there written by computer engineers, Jeff Hawkins’ book called ‘On Intelligence’, I found great because it’s basically written by an engineer about this subject, the brain, but because it’s written by an engineer, it uses the vocabulary that I’m already familiar with.

But that’s basically the process. You have to be prepared for I think a few years that are relatively unproductive in terms of research output and publication, and then when you do have some ideas to publish, you can again expect for that to be a bit of a struggle because you’ll be unfamiliar with the refereeing principles and practices in that area that you’re not moving fully into, but touching upon. So it was quite some time before we started generating papers that were at the standard that’s expected by the neuroscience community.

Andrew Miller:             And was that something … Were you building a team of your own at that same time, or was this primarily you venturing off into this new area?

Steve Furber:               I was doing it with a number of people in my team here who had similar interests, and also I had colleagues in the department here who were in the machine learning or AI area where they knew a lot more about artificial neural networks than I did. So, working with them and bringing one or two of the people in my team in this direction was how we got initial momentum.

Andrew Miller:             And do you primarily still publish in venues that you would consider your own discipline? Do you have a mix of … Are you ever publishing in computational neuroscience journals or going to their conferences?

Steve Furber:               Yes, I think we publish across a fairly broad range of disciplines now. So, we’re still designing microchips, for example, we still have papers in TVLSI. But we’re increasingly also publishing in the Frontiers journals, which are based in the neuroscience area, and everything in between. And of course, quite a lot of our publications are in collaboration with other parties who have their roots in different disciplines. So we get into a surprising number of different journals and conference outlets.

Andrew Miller:             Would you mind just sort of giving us an example of one of those collaborations where you’ve been partnering with people in a different discipline and how that’s worked?

Steve Furber:               I guess a recent one is through the Human Brain Project, we work closely with a group in Germany, who are well embedded in computational neuroscience. We did some work where we were comparing the numerical accuracy of a particular network model running on a supercomputer and running on SpiNNaker, where SpiNNaker has no floating point, it’s a fixed point engine, looking at the performance and accuracy issues. And that ultimately lead to a joint publication going to one of the Frontiers journals.

Andrew Miller:             Yeah. So I think this is a perfect time to sort of … if you can talk us through the development of SpiNNaker and what distinguishes it from previous processors, and perhaps what distinguishes it from other neuromorphic architectures.

Steve Furber:               SpiNNaker is, in many ways, it’s very like a conventional massively powered old computer. The current machine that’s online and offering a service through the human brain project has half a million processor cores, all of which are coupled in a way that means that they can collectively run real-time models of biological systems.

The thing that distinguishes it from conventional computer architectures is the way the communication infrastructure is built. Because biological neurons communicate principally by sending pure impulses. Spikes, they’re called and they just go, “Ping”, every so often. And in SpiNNaker, each ping becomes a tiny packet, which flows through a packet switch fabric to reach many destinations. So the communication is optimized for conveying very large numbers of very small packets, and each of those packets goes to many destinations. The connectivity in the brain is such that each ping goes to thousands, tens of thousands, sometimes hundreds of thousands of other neurons. So in SpiNNaker, the communication is intrinsically multi-cast.

It describes connection networks, which are either static, or at most slowly changing. And that’s one of the things that makes it possible. So every time a neuron goes, “Ping,” that packet is delivered to the same set of target neurons. SpiNNaker is truly … It’s fully digital. The packet switching is its main feature. Neurons are modeled in software, so we have a lot of flexibility. We can introduce new models of neurons. We can introduce new models of sign-up says the new plasticity rules just by changing the software in the machine.

It’s fairly constrained. The software has to run in real time, and each processor has very limited resource. But otherwise, it looks like a big parallel computer. Now if you compare that with other neuromorphic platforms, there are many different categories but, for example, the IBM TrueNorth system, which is a very impressive piece of silicon, is again entirely digital like SpiNNaker. But there the neural models are implemented in digital hardware and the synapses, likewise, are relatively simple. So if the neural model you want is reachable within the parametrized hardware on TrueNorth and the synapses do what you want, it’s a very efficient platform.

But if you want to explore outside that space, then you can’t do it without redesigning the chip. So there’s a trade-off. TrueNorth gives you much better density and much better power efficiency, but less flexibility in terms of the range of neural models and synapse models it will support. Beyond TrueNorth, our collaborators in HBP at Heidelberg are building analog neuromorphic systems. So they use analog sic, it’s to module the equations in the neural run and, again, that constrains the range of equations that you can support on that model. They are, in their latest version, using a digital system for implementing plasticity rules. And they’re also integrating these analog circuits at wafer-scale. So they have very large-scale integration, as large numbers of analog neurons.

Because their analog circuits are above threshold, it’s very difficult to slow them down to biological speed and, in fact, the Heidelberg brain-scale system runs 10,000 times faster than biology. Which has its advantages if you want to model five years of learning, you can do it in five hours. It had its disadvantages if you want to couple it to a real time robot for instance. And finally the fourth point in this space and it’s not the only other point, it’s the work that’s been going on at Stanford where they use subthreshold analog circuits. And by going subthreshold they can then bring them down to biological speeds and of course very impressive energy efficiency. So there’s four points in the consolation of large-scale neuromorphic systems.

Andrew Miller:             There’s so many interesting things there, I guess the high-level one is, do you feel like the aim is to mimic and simulate the neuron pathways or is it more inspired? How do you do that trade-off between fidelity to the way the human brain works and then taking an inspiration but making it more computery, for lack of a better word?

Steve Furber:               Well, this is one of the big issues in this area. How much biological fidelity do you have to retain in order to be able to match the capabilities of the biological system? And we just don’t know the answer to that. The four systems I’ve talked about actually all greatly simplify the details of the biological neuron and synapse. In IBM’s case, I think they are really focusing on engineering applications of the spiking neural system. In our case, we’re kind of somewhere between wanting to use it to understand biology and wanting to use it to explore applications.

So, people look at this from all perspectives. If you really want to do a highly detailed biological model with all the equations, all the iron channels, all the complexity, then you’re probably best off using a conventional supercomputer in the way that the Blue Brain Project does at EPFL. The neuromorphic systems all require significant simplification of the biological system in order to make the model more efficient so that we can scale up to larger networks, with the hypothesis that a lot of what’s interesting about the biology is in the network rather than the component details, but of course that’s an assumption.

Andrew Miller:             I mean, I assume in all of this, also you have these partners in either computational neuroscience or just in neuroscience in general. And I’m wondering, how do you explain … The traditional problem that we have, I’m a human computer interaction researcher and we always have this problem when we’re explaining to other partners, that we’re gonna have to simplify these things and we can’t actually simulate all of the things that you want. And I’m assuming that you have a similar set of discussions with the people who are interested in using something like SpiNNaker or you’re working with to develop the next iterations. How do you manage that relationship?

Steve Furber:               This can be quite complex. If you’re talking to a neuroscientist whose major focus is the 15,000 proteins that determine the functionality of an individual biological synapse, then the only sensible response is that SpiNNaker is not the right machine for building models at that level of detail. So you have to be talking to neuroscientists who are thinking more at the system level, where whatever computing system they’re using for modeling, they’re having to do some abstraction. So they’re bought into the requirement to abstract some of the details away to make the computation feasible.

And across the neuroscience population, there’s a whole range of interests and really it’s a matter of picking the right people so that the approach that you can offer is an approach that’s suitable for the problem they’re trying to solve.

Andrew Miller:             The other … Just the last sort of thing on SpiNNaker that I think would be really interesting to our audience, is it sounds like your approach is more of a hardware-software fusion perhaps than others, or certainly that’s a main component of it. How do you handle that diversity of specializations within your own team, or is that something that everyone’s cross-trained on?

Steve Furber:               No, everyone isn’t cross-trained, and my research group here was originally very focused on the hardware side, on microchip design. Microchip design is our core skill from the asynchronous processors of the ’90s. So, to make SpiNNaker a usable platform, we’ve had to bring in significant amounts of software skills, and we’ve had to do quite a lot of learning. Because SpiNNaker is an usual architecture and because the resources on the processors are very limited, we’ve effectively had to program the system from the bare metal upwards.

So we’ve had to write the entire software stack, at the bottom from the realtime operating system and the realtime code that actually runs on these cores, right up through a big Python stack to the Pine language, which comes out of the European BrainScales project at the top. And so, we can take a Pine model and map it onto the machine and run it without any change, but that requires many layers of software in between. But we’ve been fortunate in being able to recruit half a dozen people with significant software skills and experience, and they’ve integrated into the project very effectively.

Andrew Miller:             So, I guess the other piece of this is the Human Brain Project. It sounds like SpiNNaker predates the Human Brain Project, but I’m interested in how you got involved and the extent of your involvement in the creation of the Human Brain Project itself.

Steve Furber:               Yes, we began building SpiNNaker, we got significant funding to begin the design and construction back in 2005/6. So we’ve been building it for quite some time, and it’s only really because of that that we were in a position to offer a platform that found us a way into the Human Brain Project. I was not one of the original proposers of HBP, that was a result of Henry Markram and Karlheinz Meier coming together, with Henry coming from the high performance computing brain modeling side, and Karlheinz coming from the analog neuromorphic side. And those two decided to join forces to put in this flagship proposal to the EU, and we had worked with Karlheinz on the BrainScales project itself in a very small capacity, and so we were invited to join in a more substantial capacity when HBP got going. So of course we had to contribute to the proposal, but in no sense did we cause HBP to set off, to get going.

Andrew Miller:             It seems to me like you’ve worked in teams for your whole career. Is there something … Is there some special secret that makes for a good team, or what are the things that you look for when you’re trying to recruit or join a team?

Steve Furber:               I think I’ve learnt a fair bit about what makes an effective engineering team. The first thing of course is that any major engineering project has to involve a great deal of teamwork. The idea that you can work in microchip design as a sole researcher I think is barely credible these days.

Working with people, it’s about assembling a group of people, each of whom has their own competence, they don’t all have to be good at the same things, but they all have to have a competence which earns the respect of the other members of the team. And they all have to be reasonably open about how they see the strengths and deficiencies in their particular contribution, because one of the most effective things I’ve found in teamwork is the process of close-coupled peer review, where when person A designs their component, they then openly discuss it with a couple of other team members in order to get some more input and to get some improvement through that.

And the thing that makes that difficult is if you have somebody in the team who’s feeling a little bit inadequate and then gets defensive, and then they don’t expose their work properly and then mistakes creep in. But a balanced team of competent people seems to me the right way to get significant engineering jobs done.

Andrew Miller:             And just returning back to the collaboration with computational neuroscientists or neuroscience in general, it sounds like you’ve learned a lot about the way the brain processes information and you’re applying that to your own designs. How do you feel you teach neuroscientists and what are they teaching you now that this collaboration has been ongoing for many years?

Steve Furber:               I think we’re still learning a lot from neuroscientists. They of course come up with new results in their own domain which have implications for how the brain processes information. But they do have a tendency to get bogged down in detail, because it’s the biological detail that interests them. So, I find I’m constantly having to have discussions about what’s the key abstraction, yes there may be this very exciting chemical that’s involved in this very interesting process, but what does it actually do functionally?

And I think we have something to offer neuroscience in terms of understanding how to construct levels of abstraction in order to understand the complex system. Of course, that’s what computer science has been doing from the very origins of the subject, but it’s much easier in computer science because we basically design our artifacts to provide very clean support for these different levels of abstraction. Biology has not worked so tidily as far as we can tell, and therefore every attempt to find a level of abstraction in the explanation of the biological system seems to always trip up on exceptions, biology sort of keeps going round the rules to make things work.

But I still think the neuroscientists can learn about abstraction from the computer scientists, and of course we have a great deal to learn about the diversity of ways that biology solves problems.

Andrew Miller:             Yeah, I guess pulling back to the view now over your career in computing and computer science, what do you feel like has been the biggest shift in either computing research or just computer science as an area?

Steve Furber:               Well, there have been many huge shifts in computing. Although interestingly of course, the fundamental principles change very little over half a century. Here in Manchester, we … Not we, because I wasn’t born then, but the university developed the first operational store program computer, basically implementing Alan Turing’s fundamental ideas from the 1930s. But this was a machine that was 32-bit binary machine, and until very recently, the ARM processor was a 32-bit binary machine, and an instruction still did very much what an instruction did half a century ago.

But of course on top of that, we’ve learned all sorts of interesting advances. In architecture we found new ways to cope with the different rate at which processing and memory moves over time. One of the most fundamental shifts I think is the shift that happened in machine learning about 10 years ago, where finally after decades of being the second best way to solve any problem, suddenly neural networks came to the fore, and that has driven this huge explosion in machine learning we’ve seen over the last decade.

Interestingly, running pretty much over the same decade is the development of the neuromorphics in SpiNNaker. And clearly these two things are in some sense parallel and yet intersect very little. So I think the really exciting developments for the next decade or two will be to bring more biology across into those machine learning systems, and thereby we’ll understand more about the biology and deliver significantly greater functionality in the machine learning systems, which I believe have only taken the first step along this biological line.

Andrew Miller:             And I’m interested in your thoughts as someone who’s been able to build and maintain these projects, what kinds of structures or projects would you recommend if we were to really try and integrate these areas more tightly? How do we do it?

Steve Furber:               The areas being neuromorphics and machine learning?

Andrew Miller:             Sure, yeah.

Steve Furber:               I think that’s a very, very open question at the moment. It’s only a decade since this explosion in machine learning started, and only much more recently that people started thinking about how you build efficient hardware support, starting with Google’s TensorFlow ASIC and so on. Now we’re seeing whole new architectures come out from interesting companies such as Graphcore here in the UK, that are effectively reconsidering in a radical way the architecture balance that’s required to deliver efficient machine learning.

So far there’s been very little intersection between that and neuromorphics, with the possible exception of the work that IBM has been doing in porting deep networks onto the TrueNorth architecture. But I think again, that’s a very small step in a direction where there’s a lot further to go. So I think it’s wide open, and as whenever the situation is wide open, we’re seeing lots of interesting startups emerge, most of which will disappear as quickly as they started, but some of which I think will really create a new agenda for computer architecture for the coming decades.

Andrew Miller:             Yeah, and I guess finally looking forward, what would you advise the next generation of computing researchers or engineers considering this sort of interdisciplinary work?

Steve Furber:               That’s a hard question to give advice on. I mean, I think there’s lots of scope for taking the current very effective machine learning algorithms and looking at how to architect hardware for that, but in a sense that all these startups and even big industry is beginning to tread that path. So, as a PHD student looking for a topic, you’d need to tread that route very carefully to avoid being run over by the larger players.

So I think there’s still more opportunity in continuing to look at biology and trying to pick up ideas that you can bring across, and see what value they add. I think this area is just going to grow and accelerate, so there’s huge opportunity there. But finding the right question to ask is of course always the key to doing the right research.

Andrew Miller:             Wonderful. Well, thank you so much for talking with us.

Steve Furber:               You’re welcome.