Almost exactly a year ago, I started making chocolate truffles out of my home. I sold directly to friends, neighbors and coworkers, for special occasions like Mother's Day, Father's Day, Christmas, etc. In January I decided to go legit and start my own business.
I live in Michigan, which does not allow any food production from home (unlike our neighbor Ohio which has a cottage industry law). So I had to find a kitchen to use. I'm using the prep kitchen space at a place called Main Dish Kitchen - this is one of those places where you can go and assemble a variety of meals from pre-prepared ingredients, moving from station to station. They only use their kitchen from 6-2 weekdays, so it's available to me a lot of the time. It's fabulous! I love, love, love using a commercial kitchen. Just the dishwashing setup alone makes my life so much easier. You might think about checking out a place like this, if there are any near you - the one I use has just one oven and no stovetop (I use a portable burner and a microwave), so it wouldn't be very good for catering out of, but is great for chocolate and might be workable for baked goods at small volume.
There are churches that will rent to businesses - I know of a chocolatier here in Michigan that uses one - but it sounds like you've already checked out that option.
Labeling - labeling laws are determined by the Federal government. When selling at retail, you need to label your packages with weight and ingredients and sell by date. But if you are a small business, you don't have to do nutritional labeling. Selling wholesale, you just need to have an ingredient list available.
If you do get this started, the last thing I can say has to do with pricing your goods. As someone else said, you need to consider all the time you put into it - shopping, production, clean up. Plus rent, insurance, equipment. Food costs are a very small piece of the picture. Be sure you are pricing your goods appropriately from the start - if you sell yourself too short in the beginning trying to create a market, you will find it difficult to raise prices later. You can't compete on price with what can be purchased at through Sysco or at Costco, so you have to find a market that is willing to pay what it costs to actually make great tasting products out of real ingredients.