I lift the front on the corner jacks, retract the wheel so it's off the ground, swing away, and then lower the wheel to roll the TM the last few feet into position in the garage.
Perhaps I am misunderstanding, but I seem to hear you say that you put the full tongue weight of the TM on the wheel while the swing tongue is swung to the side. This is not a good idea, since the torsional rigidity of the A-frame and swing mechanism is compromised when you pull out the big pin, and apply a torque from the side. We had a big discussion of this a few years ago, in the context of broken frame welds in the swing tongue area. I found it in this thread
http://www.trailmanorowners.com/forum/showthread.php?t=3728
as well as an earlier thread which is referenced therein.
Bill