As soon as the potato tops start looking a bit shabby, I take it as the signal to start harvesting. No trouble digging them up either, even with the sunflowers growing in right beside them.