Not assumptions, BDL. Actual facts of life.
Trying to establish relative costs between now and then is an excercise in futility because the numbers don't mean anything. Plus there often are complete reversals of what things were worth, then and now. Examples would be cotten vs linen, and foodstuffs like lobster. So you have to look at value comparisons of the time.
There were, primarily, three slave-dependent crops in the 18th century south: Tobacco, in the Chesapeake; Rice, in the Lowlands; and Cotton in the Gulf States. Of the three, rice was the most profitable. Lowland planters were the most wealthy in the colonies. High profits, then and now, result from high prices as compared to production costs.
American rice actually originated in Africa. Slaves were essential not only because of the labor pool they provided, but because of their prior knowledge of rice cultivation. For various reasons, that one included, slaves involved in rice growing actually were self-supervised. They set their own work schedules, lived rather freely, and had no overseers. This is rather unique in 18th century southern agriculture.
Rice has been grown in South Carolina since 1680, and production grew with the growth of slavery. A look at exports shows the correlation: Charleston exported 10,000 lbs in 1698. That figure increased to 20 million pounds by 1730. I'll leave the number of slaves for those years as an exercise for the student.
While we find rice dishes (including rice pudding) in period cookbooks and manuscripts (all of which represent cookery of the well-to-do), I have never seen rice mentioned as part of the foods used by plain folks.
Also, talking about assumptions, you're making one with milk that is probably unjustified. In Amelia Simmons' American Cookery, the first American cookbook, published in 1796, there are six recipes for rice pudding. One of them is designated "a cheap one." It's primary difference is that it doesn't contain eggs, as the other five do. American Cookery, it should be noted, was written to be used by domestics who cooked for the upper classes.
As to where it was popular: I can't speak for the northern colonies, where it may or may not have been popular. But it was tremendously so among the well-to-do in the American south and in England. A relevant item of interest: in the puddings section of Thomas Jefferson's cookbook, the first four recipes deal with rice; three of them are rice puddings (including a French version---no surprise there), and one for a sauce to go with the Gateau au Riz. Whether that placement indicates anything deponent sayeth not.