I'll give you some pros and cons of each system then at the end I'll tell you my opinion.
PS Vita:
Pros: Amazing Graphics for a handheld, variety of apps ranging from Facebook to Netflix, 3G use (if paying through AT&T), Unique use of touch screen and back pad, Compatibility with Playstation 3 (able to transfer saves, use vita as a controller, etc), Wifi enabled, local and online play, Good Battery Life, 2 analog sticks!!!
Cons: Expensive compared to the 3DS (250$ for the system itself), SD card used to store games is also expensive depending on size (60$ + for a decent size), lack of games at the moment (only a few good ones: Persona 4 Golden, LBP, Gravity Rush, Murumasa Rebirth, Uncharted, and a few others that are not coming to mind at the moment).
3DS
Pros: Cheap compared to the PS Vita (around 150$), amazing library of games (Star Fox, Zelda, Fire Emblem, Luigi's Mansion, Mario Kart, and a lot more), Wifi enabled, local and online play, Good Battery Life. 3D (if you're into it, personally I'm not, but a lot of people are), Street Pass gives you rewards for you're games!
Cons: The normal 3DS is a little too small for my hands so I wish I got an XL, the graphics are not on par with PS Vita, but can be good or bad depending on the game, (for me) The 3D gives me a headache and therefore I never use it, The Wii store is terrible for purchases compared to Sony's Playstation Store, One analog stick does not work for certain games and feels and looks cheap compared to Vita.
Final Thoughts: Being the owner of both of these fine consoles, I have to say that the 3DS is the better investment. It has a lot more games that are extremely fun and it has online and local play as well as wife like the vita. Plus the 3DS is cheaper so the extra money that you would use on spending on a vita would be put towards some 3DS games. I would go for the XL if you have big hands, but if not then the normal size is fine. The Vita is a great system, it just does not have enough games to keep a consumer happy. I love both of them, but with the 3DS you get more bang for your buck.