I notice this tread is very old, but we ran into the exact same issue today with a vCloud 5.1.2 install and configure. Even though we had used the keytool located at /opt/vmware/jre/bin to generate the keystore we had the same issue when using the vcloud configure script
/opt/vmware/vcloud-director/bin/configure script.
the configure script kept on saying invalid keystore password, even though using the keytool to list the certificates worked. After a lot of digging we found that the keystore password that we had used when creating the keystore, was to complex. Although keytool understood the password for it's operations the vcloud director configure script did not.
Our solution was to create a new keystore using a simple keystore password. Once we did that the configure script worked.
Would be nice of the VMware documents had told us this.