I tried limiting to "XMP:*", but I wasn't sure how to do it. I did SetNewValuesFromFile($file, ['XMP:*']) because I thought the second argument needed to be a list. The result was that I didn't get any of the metadata from the .xmp file.

The other thing I was sad is that WriteInfo() couldn't work in place. That means during ingest, I have to first convert the file and then copy it. Makes for a bit slower ingest. Seems like WriteInfo() should take an option to force overwrite.