There are many reasons why you need a fast website. One important reason is Google considers a websites loading speed as part of its ranking algorithm. Google wants people to have a good experience with your website, not wait. You should feel the same way.
The last thing you want is for people to give up and leave before they can experience all your website has to offer. Here is a good article from copyblogger: Why You Need a Seriously Fast Website.