There were over 1m millionaires heading into the pandemic but the number has steadily fallen since