One of the reasons I opted for moving all my sites to a VPS was to be better able to control parts of the web server that hosted servers can't let you touch -- the Apache server, the MySQL server, things like that.
If I were just creating HTML pages, I wouldn't have make the change. But having a database-driven website, while opening up wide possibilities, also introduces performance issues. When I finally got Gallery2 running on my current setup, the whole thing slowly ground to a halt. But after tweaking the PHP, Apache and MySql config files, it hummed along better than before.
And I can't emphasize enough that you can gain wonderful quickness just by using Drupal's performance settings (admin/settings/performance). Especially with D6, it can be quick as a bunny.
One of the most useful sessions at the Do It With Drupal conference in Dec 08 was Matt Westgate & John VanDyk session on Performance and Scalability. In brief, here's an outline of what they discussed: