help with this cake making issue

I've been trying to perfect making my mother's secret cake and I'm trying to overcome one particular problem. Sadly mom is no longer around so I'm on my own with this. This is a very delicious cake btw and people absolutely love it.

The cake is a non-bake cake -- which I guess technically makes it a dessert -- that is made using lady fingers dipped in warmed milk with a bit of cognac and cointreau in lieu of baked cake layers. The lady fingers are dipped in the milk to get soft and moist and then are arranged on a serving plate to form a layer. Each "cake layer" is 2 lady fingers high and there are 3 layers of "cake." In between each layer of lady fingers is a layer of chocolate-orange icing with a final layer of icing on top. The cake is then topped with a layer of crushed caramelized almonds.

The problem I have is with the lady fingers dipped in warm milk. They are very brittle and crumble very easily. The more you allow them time to drain the excess milk the more brittle they become. And still, after the cake is assembled, milk continues to leach out.

I bought a round pizza pan with holes thinking I could rest the dipped lady fingers on it, let the milk drain out, and then transfer them to the serving plate and build the cake.

Does anyone have any other suggestions to tackle this problem?

Thanks

jmclw