You bet, JP. I have been learning a lot about solar lately, most of it from the Wind-Sun forum at
http://www.wind-sun.com/forum/
The concern that you have expressed is legitimate. A charge controller takes care of it. It prevents any source of energy (including a battery charger) from feeding power backward through the panels.
You are good to go, with both connected - no need to disconnect anything.
Bill