In arena mode the enemy tower should not be able to affect your multiplier. Some towers can’t harm your tower but other ones can. Like Robot Factory couldn’t ever hurt your tower. But Hive Pylon can, even if you play proper defense it can hurt your tower. Sentinel Orb can sometimes hurt your tower but proper defense helps avoid multiplier issues.
I think no tower should be able to affect your arena multiplier.