That really depends on what service you are using. Companies such as Packet8 and VoiceGlo allow for calls to be made even through a 56 Kbps dial-up modem because they compress the digital packets that carry the voice streams in each direction. Companies such as Vonage and VoicePulse recommend a minimum of 90 Kbps, in both upload and download speeds, because they do not use compression in transmitting voice packets. It is generally agreed, however, that a broadband connection will always result in higher-quality voice over the phone. And if you plan on using advanced features such as three-way-calling and conference calls, you will need connection speeds higher than 90 Kbps. If you are uncertain, the "Great Links" section of Voxilla (reachable from the menu on the left hand column) lists a number of sites that allow you to test your upstream and downstream speeds.

[Cybertoy]

also bear in mind that many broadband services are asymetric - i.e. you have more download than upload speeds. For 3 or more-way conferencing that's important. Your slowest speed will be the culprit.