3500# rated TV owner here (Sienna), weighing in (pun intended!).
Toyota makes a substantial and durable TV in general. I would expect a Highlander with tow package (and all the necessary additions) would be able to tow the smaller TMs (even up to a 2720, which is my TM) in the most favorable of conditions (i.e. low altitude, flats). Tow dry, only take what you need, carry one propane tank if full (or two if near empty -- the weight will be close to the same), etc. etc. I have weighed my rig once, I know I'm just at the stated limits. I drive safely, and I am well aware of my limitations. Works for me, works for many others. May or may not work for you, you'll need to decide that for yourself. There have been many, many discussions on this topic in the past, and the summary is exactly what others have said: some agree, some disagree.
If you're a strictly by-the-numbers kind of person, go for a lighter/smaller model. If you're more of a "these are general guidelines, not hard-and-fast rules" and "the majority of people who tow do not do the diligent research we here do, and still manage (with some sense involved) to successfully enjoy their camping experience" then the numbers would support the use of a Highlander with a 2720-class TM. Larger than this, even I would really question, because now you're talking about a longer trailer, more weight, and therefore more risk.
Many vehicles have specific boards out there, much like this one, but dedicated to the TV. I know there is a Sienna board out there, with extensive discussions on towing, and that board, with this one, were my principle sources of research when I was looking. I chose the 2720SL, and it has perfectly met my needs, and matches perfectly with the Sienna. No question I'm at my limit, but I feel comfortable being there. I wouldn't want bigger, I wouldn't care for smaller. I also stay on the flats of the midwest, and usually I'm not driving more than 2 hours or so at a time. I don't need to travel with water, I usually camp for 3-4 days at a stretch. I have electric at the campsite, so I don't need a genny or anything like that. This fits my needs. Yours may be different.
It's all about being comfortable with your choices: some people need the "belt and suspenders" approach of a large margin between the tow rating and the towed camper, and their experiences bear out this decision. Others are quite satisfied with spec limits being just that, and 3500# TVs will work for these TMs under that philosophy, and their experiences bear out this decision as well. Each side thinks the other is a little crazy, but that's just human nature.
(For the record, the official Toyota towing guidelines stress that axle weights are the most important factor, more than tow ratings, hitch ratings, or just about any other numbers that are bandied about. Realistically, there is no standardization in how these numbers are determined, and no one knows for sure where they come from. It's all guesswork and supposition on our parts. That's part of why there are such differing opinions.) Literally: YMMV.