No whipped cream ever. (Whip cream is for pie, ice cream, or cocoa.)
You can get non-fat milk, but to me I'd prefer the calories and the flavor of regular milk.
When I treat myself and order a flavored latte, I always ask them to put in 1/2 the numbers of pumps/flavoring that they would regularly put in. I like the taste of coffee, and I actually want to coffee just sometimes with a little somethin' somethin'.
In general, skip the sauces (drizzles) if you want to cut calories. One pump of caramel, toffee nut, or hazelnut syrup works. Or skip the syrup and ask for a drizzle of chocolate or caramel:
http://store.starbucks.com/more/syrups-and-sauces/