So this boils down to set_data() not having a mode parameter, right? Alas, the obvious approach, adding it, breaks backward compatibility. The alternative is to use source_from_cache() in set_data() to get the source path, and then get the mode there. Of course, differentiating between the use cases for set_data() seems important there too. In short, "It may be rather hard to fix given the limitations of the set_data API".
Regardless, we'll need to be careful to use the loader's cache to avoid extra stat calls.
Also, we have to factor in issue6074 (basically do "mode | 0o600"). We'll need at least one more test to cover that (as Nick noted in that issue).
I have a patch that is close, but my eyes are getting a little heavy. Hopefully I'll have that up tomorrow.