My husband and I own a car and an SUV (ok, it's a crossover). The car is "mine" and the SUV is "his". Honestly, the biggest problem we have driving each other's cars is the fact that the SUV has a rear camera and the car doesn't. When I put the SUV in reverse, I automatically look over my shoulder, wonder for a moment why I can hardly see anything, and then check the camera display. When DH puts my car in reverse, he first checks the dashboard display, and wonders why there are numbers showing the time and temperature floating around behind the car.
I have driven a truck-based SUV. At the time, I owned a mid-size car. I won't lie to you - it was an adjustment going from one to the other. The vehicles handle differently, and you sit much higher in an SUV.
Best thing to do is take a test drive and judge for yourself.