You can find here common problems, and their most common causes. If you cannot solve your problem using this guide, you can try the irc channel, The mailing lists, or The bug database. When you solved your problem, please send me the description of the problem and the cause to m4gw4s at gmail dot com.
Problem: No messages are received from the peer.
Cause: Something is blocked, or the other end is dead. Check that the other end works, and no packets needed for communication are blocked. These are: udp 500 for ISAKMP, IP 50 for ESP, IP 50 for AH, udp 4500 for udp encapsulated esp. You probably do not need all for your setup.
Problem: sent MI2 (STATE_MAIN_I2: sent MI2, expecting MR2), but no answer received.
Cause: The other end does not like the encryption parameters you propose for ISAKMP. Check your ike= parameter. With certificate authentication, it might be the first phase where lost fragment problems manifest, especially if your certificate is too long, or you have too much CA certs. See also Debugging with tcpdump.
Problem: sent MI3 (STATE_MAIN_I2: sent MI3, expecting MR3), but no answer received.
Cause: Authentication problem. Check your certificate/secret/etc. With certificate authentication, it might be the first phase where lost fragment problems manifest, especially if your peers certificate is too long, or it have too much CA certs to show you. See also Debugging with tcpdump.
Problem: INVALID_KEY_INFORMATION
Cause: Openswan did receive a key, but could not establish its authenticity. With certificates, it might mean that not all CA certificates in the certificate chain are in /etc/ipsec.d/cacerts. FIXME(I do not know too much about the other auth methods): with dnssec, maybe the DNS is unreacheable, or does not contain the necessary data.
Problem: Quick mode initiated (STATE_QUICK_I1: initiate), but did not succeed
Cause: The peer does not like the parameters you have proposed for esp. Check your esp= and {right|left}subnet= parameters. Do you do nat traversal? UDP encapsulation? Tunnel mode? Reportedly there is a bug in some old cisco VPN3030 software which can also cause this symptom.
Problem: On the server side: next payload type of ISAKMP Hash Payload has an unknown value:xx malformed payload in packet On the Oakley.log - Windows: IKE failed to find valid machine certificate Looking for IPSec only cert Trust failed.
Cause: The default CA.sh provided with openssl in some cases may not have the proper configurations in openssl.cnf to let clients recognize the CA's self signed certificate as an authorized signer. Remember to change the values back once you start signing client certificates, otherwise they'll be recognized as CAs.
