r – Define different start points to stacked bars with ggplot

I am trying to do a stacked bar chart, but each bar should start at the highest point of the last bar. I have no problem on adding the start point manually for each bar, but I have no idea where to specify this in ggplot.

Here is the dataframe and the plot

df <- data.frame(Name = c("A", "A", "A", "B", "B", "B", "C", "C", "C"), category = rep(1:3, times=3), value = c(5, 8, 4, 3, 1, 8, 5, 11, 3), difference_value = c(5, 3, -4, 3, -2, 7, 5, 6, -8))

ggplot(df, aes(fill=reorder(Name, difference_value), y=difference_value, x=category)) +
  geom_bar(position="stack", stat="identity")

Now, the first bar of category 1 should start at 0, but the subsequent bars should start at the end of the bar of the last category: the bar of category 2 should start in this case at 17, the stacked part of B should go down because it is negative and the values of A and C up, because they are positive.

It should be something like this:

Thanks a lot!