Who removes it in the woods? 
The ants take the seed to their nests and eat it, that how it gets dispersed, same method as a lot of other woodland species. But the elaisome contains a germination inhibitor which helps as it means the seed is delayed germination until it is dispersed away from the plant. The elaisome must be soluble, although perhaps the ants' digestive enzymes help break it down. Without ants, the elaisome must gradually dissolve or wear away in time, but then the seed is losing viability over time.
Very interesting to hear your experiences, Darren, with dry stored seed, contrary to the usual theories. I had a similar experience with Eranthis seed, also usually regarded as short viability. I sowed some from a seed exchange that had been dry stored for several months, they came up the following year the same week as the ones I had sown fresh in the previous May. I don't think the process has yet been fully researched, so I will keep trying different methods.
