Skip to content

Google’s tensor processing unit explained

There is this thing called marketing. A tensor processing unit or TPU is an ASIC with a fancy name. An ASIC is an application specific integrated circuit, a chip. In this case a system-on-chip more precisely. It is quite common. Apple makes systems-on-chips for their smartphones.

SoC or System-on-Chip

Apple’s A12 classifies as a processor or SoC on sales platforms that sell iPhones or iPads. They could have called it a separate class of processors. Apple Processing Unit. But for their marketing team, it wouldn’t make sense because the chip is for internal use only.

Google has a SoC that they built to suit their training needs and called it the tensor processing unit or TPU. It mainly processes tensors for their “tensorflow” framework. A tensor is comparable to a matrix so they need matrix operation specific hardware. The TPU hardware is the fastest architecture for this specific application. Processing tensors.

A CPU is general purpose, if we look at the Intel core chips and the AMD Ryzen. Application specific chips narrow down the use case and thus not general purpose anymore. Therefore a tensor processing unit has a narrow field of application.

A GPU was also application specific for graphics, but cryptocurrency miners and AI algorithm experts found out they could also use the massive parallel compute power in a GPU for their applications. NVidia specifically allowed people to use their CUDA cores, a specific multiply accumulates core which is fast for graphics but also for various other applications that need this fixed point or floating point calculations.

Back to goggles

To be able to claim the whole “inference to training” software stack, they had to also make an edge device, the eTPU (branded coral). But Google is not really a hardware company. What is crucial is that companies that were far away from the chip design business (ASIC and FPGA design) are now doing hardware. Apple took their GPU inside (Imagination almost went under due to this), they hired an ARM architect recently, they are focussing on hardware development now. Amazon itself and via Amazon capital are into hardware acceleration now as well. Google is into hardware acceleration. Facebook is doing FPGA and ASIC design.

Why are internet giants doing hardware?

Because it is crucial in many ways:

  • Keep your proprietary advantages secret, especially your AI algorithms that you use are your key differentiator versus competition. Plus, if you are doing data collection and processing that you shouldn’t do, better to keep it hidden for the world.
  • Dependency on semiconductor companies means slow progress (they are serving the whole market, not just you), too general purpose chips and they have pricing power. Two years ago, Intel owned the server market with Xeon, they had 99% market share. This means that innovation was replaced by high margin (for Intel) vendor locked-in purchases.
  • Very specific acceleration needs very specific IC design, analog, digital or mixed signal. Especially these giants that rose to the top fast and didn’t exist a few decades ago, they need very specific things. An optimal design for your needs. The only way you are going to have this fast is if you design it yourself.

Who is most experienced with VLSI design?

Apple does have the best chance here for hardware design because they based their main SoC on an ARM with several IP blocks around it in the same chip. The rest of the names mentioned aren’t experienced in hardware. If you are in software and you have grown fast, be able to scale very fast, your ego will be unmatched in the whole world. That could stand in the way of building a good hardware group.

Moreover, if your background is different (AI or pure software) how would you recognize a senior hardware designer which is bad, good, or exceptional? Even big semiconductor companies are most of the time unable to see the difference (they adapted, they don’t care anymore, quantity replaced quality hence the outrageous budgets of upto 500M USD for a state of the art tech node today). But starting and building a hardware team is crucial for the furture of that team. If you don’t have the experts in at the start, the rest will be as good (or bad) as their leads. But that is more what I talk about in my Quora space: HW accelerators eating AI (more than 15K professionals following my space).

Published inASICPOST