IPsec Guide (Part 1) – Protocols

The purpose of this article is to give readers a basic guide to the world of the Internet Protocol Security (‘IPSec’). I’ve also included some exam tips; whether you are studying for CISSP, CEH, ISSAP or other certifications, the IPSec protocol suite appears on them very often.

IPsec is a protocol suite to secure the IP communication by authenticating and/or encrypting each IP packet of a communication. IPSec operates on the Network Layer of the OSI model. IPSec can be implemented in the following ways:

  • Network -> Host,
  • Network -> Network,
  • Host -> Host.

As mentioned earlier, IPSec is a protocol framework of open standards developed by the Internet Engineering Task Force (IETF) that provides security for transmission of sensitive information over public, untrusted networks. Before we look how the IPSec works, let me present you with the following protocols, which may be used, while configuring the IPSec link:

ISAKMP – Internet Security Association Key Management Protocol – RFC2408is a key management protocol suite. Think about ISAKMP as a framework for authentication and key exchange.

ISAKMP’s exam tips:

  • It establishes Security Associations (SA) (more about SA in Part 2),
  • It negotiates key exchange protocol.

IKE – Internet Key Exchange – RFC2409, is the key exchange protocol. IKE simplifies configuration of the IPSec and it is de facto the standard for configuring IPSec links. It helps in defining the SA.

IKE’s exam tips:

  • Eliminates the need to manually specify security parameters.
  • Allows to specify a lifetime for the IPSec security association (SA).
  • Allows to change the encryption keys during IPSec session.
  • It provides anti-replay services.
  • It can be configured with pre-shared keys.
  • Optionally it may be used with Public Key Infrastructure but it is not a requirement.
  • It allows dynamic authentication of peers.
  • It uses Diffie-Hellman algorithm.

Oakley – RFC2412, is the key exchange protocol. It was superseded by the IKE protocol. One difference to the IKE worth mentioning is that Oakley provides something called Perfect Forward Secrecy (PFS). PFS is a security property of the Oakley protocol that ensures that a session key (in the PKI’s deployment) will not be compromised if one of the private keys is compromised in the future.

Oakley’s exam tips:

  • It uses Diffie-Hellman algorithm.
  • It uses PFS.

SKEME – The Secure Key Exchange Mechanism, is the key exchange protocol. SKEME contains four distinct modes:

1. Basic mode – key exchange based on PKI and Diffie-Hellman with PFS.
2. Key exchange based on PKI without the usage of Diffie-Hellman.
3. Key exchange based on pre-shared keys with Diffie-Hellman.
4. Key exchange based on symmetric algorithms.

SKEME’s exam tips:

  • It can use PKI,
  • It can use symmetric keys,
  • It can use pre-shared keys,
  • It can use PFS.

SKIP – Simple Key Management for Internet Protocol, is the hybrid key exchange protocol similar to SSL, except that it establishes a long-term key once, and then requires no prior communication in order to establish or exchange keys on a session-by-session basis. SKIP uses the knowledge of its own secret key or private component and the destination’s public component to calculate a unique key that can only be used between them.

SKIP’s exam tips:

  • It is similar to the SSL protocol.

KEA – Key Exchange Algorithm – RFC2528is the key agreement protocol. Similar to Diffie-Hellman.

KEA’s exam tips:

  • It uses asymmetric keys.
  • Developed as a closed protocol by the NSA.

AH – Authentication Header – RFC4835, is the security protocol. It provides connectionless integrity and data origin authentication of IP packets.

AH’s exam tips:

  • Optionally, it protects against replay attacks.
  • It can not be used on NATted networks.
  • It uses the IP protocol number 51.

ESP – Encapsulating Security Payload – RFC4835is the security protocol. It provides origin authenticity, integrity and confidentiality protection of packets. ESP also supports encryption-only and authentication-only configurations.

ESP’s exam tips:

  • It can be used on NATted networks.
  • It operates directly on top of IP, using IP protocol number 50


Summarizing, the major components of the IPSec are:

  • Security Protocols: ESP and AH.
  • Key Management Protocols: ISAKMP, IKE, OAKLEY, SKEME, SKIP.
  • Key Agreement Protocols: Diffie-Hellman, KEA.

1. http://en.wikipedia.org/wiki/Ipsec
2. https://www.freepracticetests.org
3. Official (ISC)2 Guide to the ISSAP CBK
4. CISSP All-in-One Exam Guide, 6th Edition

Jake Eliasz

Jake is a Chartered Lead Security Consultant with over 15 years' experience in Information Technology. Jake has performed many consultative engagements for retail, banking and government sectors in the EMEA region. Jake is currently focused on designing security controls, PCI DSS, PA DSS, ethical hacking and security risk/compliance. Prior to working for NCC Group, Jake worked as a Lead Security Consultant - QSA (Ambersail), Security Specialist (CreditCall) and Security Analyst (Symantec), where he was designing, implementing and managing various security controls for large, distributed networks. Jake has graduated from the University of Plymouth with the MSc degree in Information Security.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.