I first started out with the alto then after a few years started on bari but when I picked up a tenor it was a little difficult should I have started on tenor??

There's no reason why tenor should be harder than alto or bari.

The mouthpiece &/or reeds were quite different to those you have on the other instruments. Also there might be leaks on the tenor which will cause problems.