I am setting up a gpgpu workstation for prototyping the neural nets. It's an interesting time in history, this will likely be the last digital, von Neumann machine I use for this work. It is a good, fast number cruncher, but it is the wrong type of machine for ANN's. It's a bad fit. 4 little islands of 4/6gb ram spinning wheels hyper-fast. The interconnects slow as mud in comparison. PCI3.0 doesn't come close, honestly PCI4.0 won't make any real difference.
My ideal is one, extremely large, homogenous 'brain'. That gives me much less work to do. Now I will have to come up with algo's to deal with very fast, slow-connect sub-nets. I will be using compression techniques to communicate between the gpu's and find ways to architect around the hardware. The 4/6gb sub-nets will have to adapt with minimal information from the rest of the network. Trying a rasterizing technique won't be efficient since the buses and HD's will be far too slow for the gpu's and their speed will not be utilized.
We are now coming into the era of neuromorphic hardware, Qualcomm is releasing "zeroth" processors http://www.arisplex.com/analysis/robotics-benefit-qualcoms-groundbreaking-neuromorphic-processors/, and I imagine in short time other gear in labs will become commercially available.
From now on the bulk of processing will be analog. Quite a change. When I was a kid I loved the 'Science & Invention' encyclopedia set my Dad got. When I was looking at the analog computer entry I thought 'how old fashioned'. Digital was new, modern, space-age and high-techy. Analog was rusty and a kludge. Digital has had the spotlight for decades now. But with the work of Carver Mead and his students (Sarpeshkar, Boahen, etc), analog is not just 'in vogue' but is inevitably the permanent winner for the majority of this type of low power, high throughput processing. To be clear, it's not just analog, but mixed analog/digital (mostly analog).
I'm also going to give litecoin mining a shot. If I can make back some money and pay off all this gear, then great. I could use it. So I bought the gpu's and other stuff early, I was going to bargain shop next year, but the litecoin hype has got me going. It's late getting into the game, we'll see what happens.
