Depending on how much of a walk you’re up for, I went to La Pecora Bianca in Bryant Park recently and it was great. Jeans and sneakers would be perfectly appropriate.
https://www.lapecorabianca.com/
Friedmans on 31st is closer and also fits the bill.
https://www.friedmansrestaurant.com/