Angelfood cake is certainly appropriate, and maybe even the most popular choice here in the U.S. But you can really use any sort of cake or even a brownie or the same sort of "biscuit" type cookie you see in tiramisu.
The denser the cake, the more of of its own identity will be retained within the finished trifle.
If you're looking for a cake that will become one with the filling -- angel cake is perfect. It's usually cut into one inch cubes, to aid in layering.
If you're looking for more integrity think of a regular sponge cake, a genoise, a pound cake, a sugar free ladyfinger. (I believe there are some excellent ones available commercially. Stella d'Oro?, or even blondies or brownies. I've listed these in order of increasing density. All except the blondie/brownies are traditional trifle choices -- and brownies have been around long enough in chocolate trifles to possibly be "traditional" as well.
I think I'd choose ladyfingers. I like the way they line the bowl. I like the way they work with almond. Also they're lighter which seems to be the way you're trying to go. Or else, I'd choose pound cake. Or maybe both -- with the lady fingers on the side, and the pound cake doin' the ol' horizontal. But it's your trifle. One thing, if you do choose biscuits or blondies make sure to give a very good soaking before layering in.
Whatever you decide on, let us know and let us know how it works out for you.