An update from the USB DAC + Headphone Amp project by Yin Zhong (summivox):

USB Interface

Months into the project and I was still amazed at the lack of availability of a class of ASICs: USB-I²S bridges. Well I just lied — if you are fine with USB 1.1 and USB Audio Class (UAC) 1.0, which severely limits your (bit depth × resolution × channel count)

So I kept looking for ASICs, and I found one that is close enough: XMOS XHRA-2HPA. It even comes with a reference design that does exactly what I want!

summivox has shared the board on OSH Park:

20161229-1609 headphone-USB-v2.0.zip