By this logic, you would agree felons should be allowed to bear arms as well?
Yes. Not only based on principal, but because I would rather felons have legal firearms than enable a black market for them. Felons who want weapons already get them, so this would only change the legal status.
I should be clear that full restoration of rights after time served should go along with major reform of the penal system. As things are now prison is often training for future criminality, and we send too many petty drug offenders away to spend time with real criminals.