What is IPSec?

A closer look at the internet's security protocol suite

IPsec and a series of security icon in small circles

IPsec, an extension of the widely adopted internet protocol (IP), encrypts network communications, protecting data against ‌theft‌ ‌and‌ ‌infiltration. Yet, history gives us a better understanding of IPsec.

In the 1970s and 80s, the US Department of Defense (DoD) developed the IP, laying the foundation for ARPANET, a precursor to the internet.

IP‌ ‌enabled data movement in discrete segments called‌ ‌packets‌ ‌from‌ ‌a‌ ‌source‌ ‌computer‌ ‌to‌ ‌a‌ ‌destination‌ ‌computer. Each packet, or datagram, contained control and user ‌information. The former included instructions to deliver the latter, which was known as the payload. Simply put, the IP encapsulates and routes data packets across IP networks.   

IP had its limitations, though.  The protocol lacked a mechanism to prevent data packets from being snooped on. TV shows in the past have repeatedly alluded‌ ‌to‌ ‌this‌ ‌flaw, indicating it wasn’t hidden for very long.

To end growing security concerns and prevent data sniffing across IP networks, the internet engineering task force (IETF) proposed IP security (IPsec) in 1995. The protocol has remained in use ever since.

The following guide provides insights into the inner workings of IPSec, its features, specific use cases, and more.

How does IPSec work?

IPsec is a layered security protocol that supports IP versions 6 and 4. There are three main protocols within IPsec: ‌‌authentication‌ ‌header‌ ‌(AH),‌ ‌encapsulating‌ ‌security‌ ‌payload‌s ‌(ESP),‌ and internet security association and key management protocol (ISAKMP)

Authentication Header

The authentication header (AH) does two things; it validates IP packets to source data origin and provides connectionless data integrity through hash keys. Hash functions allow arbitrary-sized data to be mapped to ‌fixed values, ensuring confidentiality.

Additionally, the AH layer offers protection against replay attacks. In a replay attack, data is retransmitted or delayed by malicious actors. AH‌ eliminates this problem by sequencing IP packets and discarding obsolete ones.

Encapsulating Security Payloads 

Like AH, the encapsulating security payloads (ESP) protocol authenticates sender and recipient identities and detects tampered IP packets, ensuring data integrity. But it is encryption that sets it apart. The ESP layer extends IP packets’ security by encrypting the entire payload.  

Internet Security Association and Key Management Protocol

ISAKMP adds to AH and ESP by defining security associations between two network entities or hosts exchanging information. Before data transmission, ISAKMP enables the sender and recipient hosts to decide what type of cryptographic algorithm to encrypt the IP packets with, session duration, network parameters, symmetric keys for decryption, and more. 

Notably, the IPsec protocol suite encrypts data exchanges between two hosts (host-to-host), networks (network-to-network), or between a security gateway and ‌host‌ ‌(network-to-host).

Methods of operation

IPSec supports two modes of operation: transport and tunnel.

Transport mode

An IP packet consists of two components: the header, which addresses and routes the packet, and the payload, which contains the data.

However, IP packets in transport mode only have their payload encrypted or authenticated. No changes are made to the header, which includes the source and target IP addresses. This is due to the lack of encryption by default in AH.

Transport mode is best suited for host-to-router network connection management.

Tunnel mode

As a stark contrast to transport mode, tunnel mode encrypts the entire IP packet. Essentially, a new IP header is added on top of the original IP header. This is also how private IP addresses or VPNs work. The IP header, which contains the source and the target IP address, is masked to prevent third-party interception. 

IPsec’s tunnel mode is best applied to network-to-network communications, host-to-network communications (remote user access), and host-to-host communications (private chat).


Depending on the operating system, IPSec may use one of three cryptic algorithms to perform authentication. They are:

  1. Rivest–Shamir–Adleman (RSA)
  2. Elliptic curve digital signature algorithm (ECDSA) 
  3. Pre-shared key (PSK)

Let’s go over each in detail.


Rivest–Shamir–Adleman (RSA) is a public-key cryptosystem named after its inventors Ron Rivest, Adi Shamir, and Leonard Adleman.

A public-key cryptosystem has one public encryption key and one‌ ‌private decryption‌ ‌key. Two large prime numbers and an auxiliary value are used to create the RSA public keys. 

Let’s say John wants to send a message to Clara. To encrypt the messages with RSA, John must know Clara's public key, and likewise, to decrypt them, Clara must use her private key. Clara sends her public key to John over a secure pathway so he can reliably send his encrypted messages. Remember, Clara's private key is never disclosed.

Elliptic curve digital signature algorithm

The ECDSA algorithm is a variant of the digital signature algorithm (DSA), a federal information processing standard for digital signatures. 

Technically, the ECDSA algorithm relies on the algebraic structure of elliptic curves over finite fields. Elliptic curves may be used for key agreement, digital signatures, or pseudo-random generators, among other things.

For elliptic curve cryptography to work, all participating devices must possess a private and public key pair. A message is signed with a private key by its sender, and the recipient uses the sender's public key to verify its authenticity.

Messages that have been altered on route to the recipient will not pass the signature verification test, as the signature only applies to the original message.

Pre-shared key 

In cryptography, a pre-shared key (PSK) is a shared secret that two parties have previously shared through a secure channel. 

Wi-Fi encryption standards, including wired equivalent privacy (WEP), Wi-Fi protected access (WPA) and the extensible authentication protocol (EAP) use PSK in their encryption methods.  The wireless access points (AP) and the clients share the same authentication key.

A PSK may be a password, a passphrase, or a hexadecimal string. 

IPsec versus SSL: What’s the difference?

The‌ ‌IPsec‌ ‌protocol‌ ‌suite‌ ‌operates‌ ‌at‌ ‌the‌ ‌network‌ ‌layer‌ ‌in‌ ‌the‌ open systems interconnection

(‌OSI‌) ‌model. The secure sockets layer (SSL), on the other hand, operates at the application layer of the OSI model.  Each serves a very different purpose: IPsec encrypts IP packers, while SSL encrypts HTTP traffic and web-based transactions.

Featured Resources

The ultimate law enforcement agency guide to going mobile

Best practices for implementing a mobile device program

Free download

The business value of Red Hat OpenShift

Platform cost savings, ROI, and the challenges and opportunities of Red Hat OpenShift

Free download

Managing security and risk across the IT supply chain: A practical approach

Best practices for IT supply chain security

Free download

Digital remote monitoring and dispatch services’ impact on edge computing and data centres

Seven trends redefining remote monitoring and field service dispatch service requirements

Free download


NETGEAR offers VPN support for branch offices and remote workers
wifi & hotspots

NETGEAR offers VPN support for branch offices and remote workers

14 Oct 2021
ONF and Deutsche Telekom debut fully disaggregated Open RAN

ONF and Deutsche Telekom debut fully disaggregated Open RAN

12 Oct 2021
Senators urge FTC to enforce child privacy laws

Senators urge FTC to enforce child privacy laws

8 Oct 2021
NSA and CISA offer new security guidance for VPNs
virtual private network (VPN)

NSA and CISA offer new security guidance for VPNs

29 Sep 2021

Most Popular

Alibaba unveils custom Arm-based server chip

Alibaba unveils custom Arm-based server chip

19 Oct 2021
What is cyber warfare?

What is cyber warfare?

15 Oct 2021
HPE wins networking contract with Birmingham 2022 Commonwealth Games
Network & Internet

HPE wins networking contract with Birmingham 2022 Commonwealth Games

15 Oct 2021