Two possibilities leap to mind.
You're not folding the egg whites into the base properly. After you've whipped the whites to soft peaks, add about a third to the base and mix them in -- not vigorously, but not folding them either. That will lighten the base. Then fold in the remaining egg whites, with the usual folding technique, until the whites are almost completely incorporated. That is, you want a just a few very thin streaks of white in the base. If you totally incorporate, you'll knock too much air out.
Your oven is too slow, and you're overcooking by cooking too long. At a guess this is more likely. 375F is a sort of dead minimum for souffle baking; I prefer 400F. A properly cooked souffle is something like a properly cooked omelette. In physics terms, at its center, the "matter phase" should be indeterminate. Is it liquid? Is it solid? Both and neither. In cooking terms, you want it very slightly undercooked in the center.
At a guess, yours is drying out, and in the course of completely drying the top dries faster, and separates from the center when it's no longer wet enough to hold it.
A common trick is to serve a sauce with the souffle (often creme anglaise). At service, the souffle is broken with a spoon and enough sauce poured in so that it just begins to run out of the souffle. This covers a multitude of sins.
On the whole, your lychee souffle sounds lovely. I may steal it.
BDL  | 
Why is My Sweet Souffle Hollow? 
