A GPU that is commercially available, is a PCB in mass production hence efficient in production from a cost standpoint. FPGA development boards are also available. But high-end boards will be much more expensive than an off-the-shelf GPU card for a desktop PC for example. FPGA also needs the implementation written in any HDL and it needs to be verified. Hence you will spend time on the custom FPGA implementation. Hence a GPU is the fastest and cheapest solution but not the most efficient.
Which is more cost effective as machine learning accelerators, FPGAs or GPUs?