Preheat oven to 350°F (175°C). Grease and flour a 9-inch round cake pan.
In a saucepan over medium heat, melt 1/2 cup butter. Stir in 1 cup sugar until dissolved and lightly caramelized. Pour this mixture into the prepared cake pan.
Arrange the peach slices evenly over the caramel layer. Scatter the raspberries over the peaches.
In a medium bowl, whisk together the flour, baking powder, and salt. Set aside.
In a large bowl, cream together 1/2 cup softened butter and 3/4 cup sugar until light and fluffy.
Beat in the eggs one at a time, then stir in the vanilla extract.
Alternately add the dry ingredients and milk to the butter mixture, beginning and ending with the dry ingredients. Mix until just combined.
Gently spread the cake batter evenly over the fruit in the pan.
Bake for 35-45 minutes, or until a wooden skewer inserted into the center of the cake comes out clean.
Let the cake cool in the pan for 10-15 minutes before inverting it onto a serving plate.