If you accept credit card payment without SSL protecting the entry of the card and CVV2, you are in violation of PCI DSS standard 4.1. Depending on your relationship with your acquiring bank or payment processor, that will result in a passthrough of fines and any fraudulent charges that are incurred, along with possible termination of your account.
Now, for the research on SSL and UI, there is good research on this. My favorite is The Emperor's New Security Indicators - http://www.usablesecurity.org/emperor/
"We confirm prior findings that users ignore HTTPS indicators: no participants withheld their passwords when these indicators were removed. We present the first empirical investigation of site-authentication images, and we find them to be ineffective: even when we removed them, 92% participants who used their own accounts entered their passwords. We also contribute the first empirical evidence that role playing affects participants' security behavior: role-playing participants behaved significantly less securely than those using their own passwords."
Users will be happy with no encryption than with a self signed cert.
Personally i think the best solution would be SSH style for most sites (the certificate is stored on first visit and changes are flagged as suspicious), with physical distribution of public keys for sites that really needed security (e.g. banks).
Now, for the research on SSL and UI, there is good research on this. My favorite is The Emperor's New Security Indicators - http://www.usablesecurity.org/emperor/
"We confirm prior findings that users ignore HTTPS indicators: no participants withheld their passwords when these indicators were removed. We present the first empirical investigation of site-authentication images, and we find them to be ineffective: even when we removed them, 92% participants who used their own accounts entered their passwords. We also contribute the first empirical evidence that role playing affects participants' security behavior: role-playing participants behaved significantly less securely than those using their own passwords."