Beat butter (or coconut oil) and sugar in a mixing bowl with an electric mixer on medium-high until well combined. Add eggs, one at a time, beating well after each addition. Add banana, buttermilk, lemon zest and vanilla; beat until combined. Add the flour mixture and beat until just combined. Fold in walnuts (or chocolate chips), if using. Scrape the batter into the prepared pan.