I have been working on refactoring the API of mega_apply and friends. The purpose of that branch is to improve the performance of function calls. More specifically, in some cases (e.g. f_map) frame allocation, setup and deallocation could be avoided if the API permitted it. I will soon push a branch and ask for feedback on that. Now to the real questions:

Yes, separating frame handling from function calling would probably be a good idea (but it is a can of worms to get correct). (Re-)separating mega_apply() into its various tastes might also be good.

apply_low.h contains a FIXME notice talking about new-style tail recursion. Could anyone elaborate what that is, which code to look at and how that works?

That's a ~14 year old FIXME, and it probably refers to the F_RECUR et al opcodes in interpret_functions.h.