Reading your post has inspired me to join, just so I could share some advice with you.
What others have told you is true. It is possible, but it can be very difficult, especially early in your career.
When my wife became pregnant with our first child, I retired. Luckily, I could afford to because I had already been in the business for many, many years. I was the chef in a restaurant. I began cooking 43 years ago. Yes, I am older than my wife. She wanted to concentrate on her career and have a family. As a new Registered Dietitian, she had a lot of ambition of things she wanted to do, and she has been doing them over the last few years. Our oldest daughter is now 3 1/2 years old and our youngest is 8 months old. If I had not elected to stay at home with the children, we might not have made it.
Let me point something out... if you want to breast feed your children, then being a pastry chef may be even more difficult to achieve. If you aren't going to be with the baby, then you have to pump every four hours, which can be difficult on the best of jobs, much less a job with the responsibilities of a pastry chef. You must have a husband that is extremely supportive and is willing to pitch in and do things to help out without being asked to do them. If you have the kind of husband my sister-in-law has, where she comes home after working 2 jobs, he had the day off playing video games on the couch, and asks her what she is going to cook for dinner, then it may not be feasible.
In the absolute best of situations, you will lose a lot of sleep during the first few months, possibly six months of the child's life. Also realize that most women will not sleep well during the last months of a pregnancy, because they cannot get comfortable. This will take a toll on a career in a hurry. If you breast feed, you have to get up during the night to either feed the baby or use a breast pump, because it must be done every four hours for the first year of the baby's life, because it is best to best feed for the first year. That equates to over a year without proper sleep. My wife is putting herself through this torture while working as a Dietitian, because she is determined to have a successful career. On top of her normal job, she has been serving on the state board of dietitians, and is now also the editor for our state's dietitian manual. She's not a pastry chef, but has still taken on a lot of responsibilities. She is doing well at her job and all her other work, but partly because I am so supportive of it. She is still so tired at times she is ready to throw in the towel, but I won't let her and I keep encouraging her, because I know she can do it. Currently, I'm telling her she'll start getting sleep in October, just hang in there.
I do everything for her around the house. She doesn't have to lift a finger, except to give the baby a bath, because she wants to do that. I cook, clean, do laundry, mow the lawn, etc. I pack her lunch and get everything ready for her to leave in the morning. She only has to get up, get a shower, dress, eat the breakfast that I have timed to be on the table by the time she finishes her makeup, kiss us goodbye and leave. It works well. More than likely, at least in most cases, you may have to do a good amount of the work around the house, take care of the children, AND do a job that will most likely require 60+ hours per week. I used to work between 65 and 90 hours per week on a regular basis. I only worked the 65 hours at times I had to call in sick with the flu. Not everyone works that crazy, but I worked multiple jobs. Usually 60 hours as a chef, and 30 as a computer programmer. Yes, I have multiple loves in life. It's why I could afford to retire and stay home with my beautiful children. You probably can't afford to wait until your mid 40's to have children, and probably don't want to wait until then to start your career.
I don't want to discourage you from doing what you love. By all means, do it if you can! There is more to consider about it than how hard the job will be, because you also have to consider how job the hard of a mother is. I know how hard that part of the job is, because I've been playing the part of a stay-at-home-mom for over 3 years now (except I'm the Dad.)
Best of wishes in whatever you decide to do. And whatever you decide, stay determined and focused on doing the very best you can, whether as a pastry chef or as a mom, because both are awesome jobs.