Post by Robert Myers
I got stopped on Bill Dally's first slide, where he says
He claims that his slide tells you everything you need to know about
the future of computing, AND THAT BELIEF AND PROSELYTIZING FOR THAT
BELIEF IS WHY I HAVE USED UP SO MUCH SPACE HERE.
THE MOST INTERESTING PROBLEMS ARE NOT LOCAL, BECAUSE THE MOST
INTERESTING PROBLEMS ARE NOT LINEAR.
A problem is embarrassingly parallel IFF it is local IFF it is linear.
If a problem is linear, then there is a representation in which it is
both local and embarrassingly parallel. If a problem is not linear,
then there is, in general, no such representation. Does one need a
PhD from Cal Tech to understand this?
(1) Is there a proof for this? Not that there are non-linear systems
that are not embarassingly parallel, but that there are no interesting
non-linear systems that are not amenable to parallel solutions.
E.g. the N-body problem, gravitational dynamics?
First, you have to understand that what I think of as a huge problem
the DoE (for example) does not even recognize as a problem at all,
never mind as a huge problem.
Bill Dally later whisks through the logic, where he talks about
calculating fluxes that never make it off the graphics card. If you
think way back to the divergence theorem, you can see what they're
doing. You reduce the nonlinear terms in the Navier-Stokes (or other
equation involving transport) to fluxes through the surface of a
control volume. Voila! The non-linear term is local.
Since you've got all these flops and not much bandwidth, you only ever
transmit and receiive fluxes across the boundary. What's more, if
you're the DoE, you can add all kinds of chemical reactions and
constitutive equations to keep the flops occupied, EVEN THOUGH YOU AR
USING TERRIBLE APPROXIMATION OF THE TRANSPORT EQUATIONS, even without
the complication of all those chemical reactions (or whatever).
Control volumes, of course, do not appear in the Navier-Stokes
equations, which contains differential operators. Any accurate
approximation to the differential operators is non-local, and, in
fact, the only one with quantifiable limitations in the face of non-
linearity is global (a spectral representation of the operator).
This all sounds very arcane, I'm sure, and the DoE wants you to think
it is arcane, but the physics are driven by the details of how the
shortest scales interact with the longest scales, and when you
misrepresent the differential operators with a crude appropximation,
you are changing the exact physics you want to look at.
If you can't build computers that can actually do global FFT's, then
you will never be able to examine what the numerics are actually
doing, and I do truly believe that that's the way the supercomputer
bandwagon wants things. That way, they can publish gorgeous pictures
of the Rayleigh-Taylor instability, even while using methods of
It's important enough that, in the middle of his sales pitch, Bill
Dally feels compelled to whisk through an admission of what's really
going on, even though I strongly suspect that he has no idea of the
mathematical consequences of the methods he is using. This whole
"supercomputer" shenanigan (and with it miraculous GPU's that will
unravel the secrets of the universe with practically no bandwidth) is
built on a crude fudge. The fudge is SO convenient, though, that the
empire builders and software writers just want to get on with it,
without worrying that they might have lost contact with reality before
they even started.
(2) If what you and Dally say is true, Robert, then you may be tilting
at windmills. There may be no computationally efficient way of doing
what you want.
I don't believe this, because I do not equate computationally efficient
to embarassingly parallel.
Also: even though locality really matters, nevertheless we have used
that as an excuse to pessimize non-local activities. At the very least
we can influence the constant factor by removing these pessimizations.
As I have attempted to do with my proposal for a scatter/gather based
I may well be tilting at windmills. I've understood that all along.
Even if you can build computers with sufficient global bandwidth (can
afford the hardware), the energy costs might kill you. I don't really
The FFT isn't local, but it diagonalizes the derivative (DOES TRULY
MAKE THE CALCULATION LOCAL), and it can be done with very great
efficiency, IFF you have sufficient global bandwidth.
Personally, I think that if I'm tilting at windmills, it's not because
of any actual physical limitation or any real financial limitation
that can't be overcome, but because dealing with the foundational
reality I keep harping on will interfere with the march of petaflops
and the endless flow of pretty pictures.