Ssh is very picky about permissions on the ~/.ssh directory and files. Sometimes you may do something to mess up these permissions. Run the following to fix most permissions problems. You may have to do this on both the remote host and local host.
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub
chmod 644 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/known_hosts