Anyone who has spent time in a professional kitchen will tell you that a degree in culinary arts/pastry does not guarantee that you know how to cook. It may (i stress may), give you a leg up, but it isn't a given.
For the most part, working in a kitchen is about one thing: consistency. Mindless repetition of the same things until you get fast, clean, and efficient (I think someone already said this). This true of both savory and pastry.
Also, like was said above, if you love it, do it. But:
Go to craigslist in your local area (or any area), and go to the Food service area. Count the number of jobs that are for pastry. There probably aren't many. While this isn't an exact representation of the job market, its a good benchmark.
Optimistically, you can expect $25k per year starting out. Maybe less. Do you want to go spend $80k on school, only to get out with an expected salary of $25k? Doctors/lawyers can afford expensive student loans because there's a pretty good chance they'll get paid back for it all. That isn't the case with foodservice.
I was in the same boat 4 years ago. What I decided to do was to work for a bit. I found a job (a terribly paying one at that), and worked. And worked. And worked. And I loved every minute of it, but at the end of it, I realized that I didn't really need what the CIA was offering, especially not at their price.
Before jumping in with pocketfuls of cash, try to get some work in a bakery. See if it's really for you, and then you will be better able to make a good decision. There's absolutely no rush to go to school if you want to be a baker. It will always be there if you change your mind