The idea was always appealing, but the implementation has always remained challenging.
For over a decade, "Mythic AI" was making accelerator chips with analog multipliers based on research by Laura Fick and coworkers. They raised $165M and produced actual hardware, but at the end of 2022 have almost gone bankrupt and since then there has been very little heard from them.
Much earlier, the legendary chip designers Federico Faggin and Carver Mead founded Synaptics with an idea to make neuromorphic chips which would be fast and power efficient by harnessing analog computation. Carver Mead published a book on that in 1989: "Analog VLSI and Neural Systems", but making working chips turned to be too hard, and Synaptics successfully pivoted to touchpads and later many other types of hardware.
Of course, the concept can be traced to an even older and still more legendary Frank Rosenblatt's "Perceptron" -- the original machine learning system from 1950s. It implemented the weights of the neural network as variable resistors that were adjusted by little motors during training. Multiplication was simply input voltage times conductivity of the resistor producing the current -- which is what all the newer system are also trying to use.
The idea of analog neural networks is appealing. I bought Analog VLSI and Neural Systems in 1989 and still have it as a trophy on my bookshelves. My gut feeling says one day analog neural networks will be a thing, if only for the reason of considerable lower power consumption.
I’m not saying that life is analog, DNA is two bits. IMHO life is a mix of Analog & Digital.
I know of only one real world successful product using analog computation in place of expensive high end micro. It was the first proper (no dedicated special mousepads) Optical Mouse designed and build by HP->Agilent->Avago and released by Microsoft in 1999 as IntelliMouse Optical. https://gizmodo.com/20-years-ago-microsoft-changed-how-we-mo... Afaik Microsoft bought 1 year explosivity for the sensor. Avago HDNS-2000 chip did all the heavy lifting in analog domain.
"each array element had nearest neighbor connectivity so you
would calculate nine correlations, an autocorrelation and eight cross-correlations, with each of your eight
nearest neighbors, the diagonals and the perpendicular, and then you could interpolate in correlation
space where the best fit was."
"And the reason we did
difference squared instead of multiplication is because in the analog domain I could implement a
difference-squared circuit with six transistors and so I was like “Okay, six transistors. I can’t do
multiplication that cheaply so sold, difference squared, that’s how we’re going to do it.”
"little chip running in the 0.8 micron CMOS could do the equivalent operations per second to 1-1/2 giga operations per second and it was doing this for under 200 milliwatts, nothing you could have approached at that time in the digital domain."
Faster than an H100 for solving 128x128 matrices. But it’s not clear to me how they tested this, code is only available on request.
> We have described a high-precision and scalable analogue matrix
equation solver. The solver involves low-precision matrix operations,
which are suited well to RRAM-based computing. The matrix operations
were implemented with a foundry-developed 40-nm 1T1R RRAM array
with 3-bit resolution. Bit-slicing was used to guarantee the high preci-
sion. Scalability was addressed through the BlockAMC algorithm, which
was experimentally demonstrated. A 16 × 16 matrix inversion problem
was solved with the BlockAMC algorithm with 24-bit fixed-point preci-
sion. The analogue solver was also applied to the detection process
in massive MIMO systems and showed identical BER performance
within only three iterative cycles compared with digital counterparts
for 128 × 8 systems with 256-QAM modulation.
This looks like one of many ideas for more efficient compute chips for machine learning. I'm waiting for the day some chip gets mass produced and works at scale for some large model and with sufficient reliability, but until then, I don't think there's anything particularly newsworthy here. I do think it'll eventually happen at some point maybe within a decade, but surely some alternative computing paradigm to the GPU will succeed. The analog chip in the article only seems to be a research prototype for now.
The idea was always appealing, but the implementation has always remained challenging.
For over a decade, "Mythic AI" was making accelerator chips with analog multipliers based on research by Laura Fick and coworkers. They raised $165M and produced actual hardware, but at the end of 2022 have almost gone bankrupt and since then there has been very little heard from them.
Much earlier, the legendary chip designers Federico Faggin and Carver Mead founded Synaptics with an idea to make neuromorphic chips which would be fast and power efficient by harnessing analog computation. Carver Mead published a book on that in 1989: "Analog VLSI and Neural Systems", but making working chips turned to be too hard, and Synaptics successfully pivoted to touchpads and later many other types of hardware.
Of course, the concept can be traced to an even older and still more legendary Frank Rosenblatt's "Perceptron" -- the original machine learning system from 1950s. It implemented the weights of the neural network as variable resistors that were adjusted by little motors during training. Multiplication was simply input voltage times conductivity of the resistor producing the current -- which is what all the newer system are also trying to use.
The idea of analog neural networks is appealing. I bought Analog VLSI and Neural Systems in 1989 and still have it as a trophy on my bookshelves. My gut feeling says one day analog neural networks will be a thing, if only for the reason of considerable lower power consumption.
I’m not saying that life is analog, DNA is two bits. IMHO life is a mix of Analog & Digital.
I know of only one real world successful product using analog computation in place of expensive high end micro. It was the first proper (no dedicated special mousepads) Optical Mouse designed and build by HP->Agilent->Avago and released by Microsoft in 1999 as IntelliMouse Optical. https://gizmodo.com/20-years-ago-microsoft-changed-how-we-mo... Afaik Microsoft bought 1 year explosivity for the sensor. Avago HDNS-2000 chip did all the heavy lifting in analog domain.
Travis Blalock Oral History https://www.youtube.com/watch?v=wmqa9XJED-Q https://archive.computerhistory.org/resources/access/text/20...:
"each array element had nearest neighbor connectivity so you would calculate nine correlations, an autocorrelation and eight cross-correlations, with each of your eight nearest neighbors, the diagonals and the perpendicular, and then you could interpolate in correlation space where the best fit was."
"And the reason we did difference squared instead of multiplication is because in the analog domain I could implement a difference-squared circuit with six transistors and so I was like “Okay, six transistors. I can’t do multiplication that cheaply so sold, difference squared, that’s how we’re going to do it.”
"little chip running in the 0.8 micron CMOS could do the equivalent operations per second to 1-1/2 giga operations per second and it was doing this for under 200 milliwatts, nothing you could have approached at that time in the digital domain."
Extra Oral History with inventor of the sensor Gary Gordon: https://www.youtube.com/watch?v=TxxoWhCzIeU
Faster than an H100 for solving 128x128 matrices. But it’s not clear to me how they tested this, code is only available on request.
> We have described a high-precision and scalable analogue matrix equation solver. The solver involves low-precision matrix operations, which are suited well to RRAM-based computing. The matrix operations were implemented with a foundry-developed 40-nm 1T1R RRAM array with 3-bit resolution. Bit-slicing was used to guarantee the high preci- sion. Scalability was addressed through the BlockAMC algorithm, which was experimentally demonstrated. A 16 × 16 matrix inversion problem was solved with the BlockAMC algorithm with 24-bit fixed-point preci- sion. The analogue solver was also applied to the detection process in massive MIMO systems and showed identical BER performance within only three iterative cycles compared with digital counterparts for 128 × 8 systems with 256-QAM modulation.
This looks like one of many ideas for more efficient compute chips for machine learning. I'm waiting for the day some chip gets mass produced and works at scale for some large model and with sufficient reliability, but until then, I don't think there's anything particularly newsworthy here. I do think it'll eventually happen at some point maybe within a decade, but surely some alternative computing paradigm to the GPU will succeed. The analog chip in the article only seems to be a research prototype for now.
Using all analog signal, why non analogue multiplying cells (operation amplifier)!
Huge if true, room temperature semiconductor if false
Semi or supra conductor ?
Seems a bit too good to be true.
Now put it in a guitar pedal!
Wo Fat has you covered: "Analog Man" => https://open.spotify.com/track/6KcM6et6Pn6UIna1o8Vl07?si=qFu...
[dead]
[dead]
What’s this good for?
Fear