The mesh has the standard normal/tangent/bitangent at each vertex, those are skinned in the usual way, and those decide how the heightfield is distorted.
The nice thing is this is all just the same authoring pipeline as we have today. But as well as a normal map, you get a heightfield/SDF.