I just woke up but have you tested this? The pubkey for the host key will be in PEM format, not the `<type> <fingerprint>` format won't it? Does SSH accept that for an authorized keys file? (neat if so)
I think also that it will fall apart if the selected config file has a match block in it, as it'll only match whatever the match block is and not "any user" as advertised, but I don't think that can be helped.