Preheat oven to 350 degrees F. Generously coat a 9-by-5-inch nonstick loaf pan with cooking spray.
Whisk whole-wheat and oat flours, granulated sugar, baking powder, cardamom, ginger, allspice and salt in a medium bowl.
Whisk eggs, milk, oil (or butter) and vanilla in a large bowl until well combined. Add plums and almonds and stir to combine. Add the dry ingredients in four additions, mixing lightly to combine after each addition. Scrape into the prepared pan. Sprinkle with turbinado sugar.
Bake the loaf until golden brown and a wooden skewer inserted in the center comes out clean, 1 hour to 1 hour 10 minutes. Let cool in the pan for 10 minutes. Run a knife around the edges, then turn the loaf out onto a wire rack and place on its side. Let cool for 1 hour before cutting.
To make ahead: Store at room temperature for up to 2 days or freeze for up to 1 month.
2 fat, 1 other carbohydrate, 1 starch