IT Pro is supported by its audience. When you purchase through links on our site, we may earn an affiliate commission. Learn more

What is IPSec?

A closer look at the internet's security protocol suite

IPsec and a series of security icon in small circles

IPsec, an extension of the widely adopted internet protocol (IP), encrypts network communications, protecting data against ‌theft‌ ‌and‌ ‌infiltration. Yet, history gives us a better understanding of IPsec.

In the 1970s and 80s, the US Department of Defense (DoD) developed the IP, laying the foundation for ARPANET, a precursor to the internet.

IP‌ ‌enabled data movement in discrete segments called‌ ‌packets‌ ‌from‌ ‌a‌ ‌source‌ ‌computer‌ ‌to‌ ‌a‌ ‌destination‌ ‌computer. Each packet, or datagram, contained control and user ‌information. The former included instructions to deliver the latter, which was known as the payload. Simply put, the IP encapsulates and routes data packets across IP networks.   

IP had its limitations, though.  The protocol lacked a mechanism to prevent data packets from being snooped on. TV shows in the past have repeatedly alluded‌ ‌to‌ ‌this‌ ‌flaw, indicating it wasn’t hidden for very long.

To end growing security concerns and prevent data sniffing across IP networks, the internet engineering task force (IETF) proposed IP security (IPsec) in 1995. The protocol has remained in use ever since.

The following guide provides insights into the inner workings of IPSec, its features, specific use cases, and more.

How does IPSec work?

IPsec is a layered security protocol that supports IP versions 6 and 4. There are three main protocols within IPsec: ‌‌authentication‌ ‌header‌ ‌(AH),‌ ‌encapsulating‌ ‌security‌ ‌payload‌s ‌(ESP),‌ and internet security association and key management protocol (ISAKMP)

Authentication Header

The authentication header (AH) does two things; it validates IP packets to source data origin and provides connectionless data integrity through hash keys. Hash functions allow arbitrary-sized data to be mapped to ‌fixed values, ensuring confidentiality.

Additionally, the AH layer offers protection against replay attacks. In a replay attack, data is retransmitted or delayed by malicious actors. AH‌ eliminates this problem by sequencing IP packets and discarding obsolete ones.

Encapsulating Security Payloads 

Like AH, the encapsulating security payloads (ESP) protocol authenticates sender and recipient identities and detects tampered IP packets, ensuring data integrity. But it is encryption that sets it apart. The ESP layer extends IP packets’ security by encrypting the entire payload.  

Internet Security Association and Key Management Protocol

ISAKMP adds to AH and ESP by defining security associations between two network entities or hosts exchanging information. Before data transmission, ISAKMP enables the sender and recipient hosts to decide what type of cryptographic algorithm to encrypt the IP packets with, session duration, network parameters, symmetric keys for decryption, and more. 

Notably, the IPsec protocol suite encrypts data exchanges between two hosts (host-to-host), networks (network-to-network), or between a security gateway and ‌host‌ ‌(network-to-host).

Methods of operation

IPSec supports two modes of operation: transport and tunnel.

Transport mode

An IP packet consists of two components: the header, which addresses and routes the packet, and the payload, which contains the data.

However, IP packets in transport mode only have their payload encrypted or authenticated. No changes are made to the header, which includes the source and target IP addresses. This is due to the lack of encryption by default in AH.

Transport mode is best suited for host-to-router network connection management.

Tunnel mode

As a stark contrast to transport mode, tunnel mode encrypts the entire IP packet. Essentially, a new IP header is added on top of the original IP header. This is also how private IP addresses or VPNs work. The IP header, which contains the source and the target IP address, is masked to prevent third-party interception. 

IPsec’s tunnel mode is best applied to network-to-network communications, host-to-network communications (remote user access), and host-to-host communications (private chat).


Depending on the operating system, IPSec may use one of three cryptic algorithms to perform authentication. They are:

  1. Rivest–Shamir–Adleman (RSA)
  2. Elliptic curve digital signature algorithm (ECDSA) 
  3. Pre-shared key (PSK)

Let’s go over each in detail.


Rivest–Shamir–Adleman (RSA) is a public-key cryptosystem named after its inventors Ron Rivest, Adi Shamir, and Leonard Adleman.

A public-key cryptosystem has one public encryption key and one‌ ‌private decryption‌ ‌key. Two large prime numbers and an auxiliary value are used to create the RSA public keys. 

Let’s say John wants to send a message to Clara. To encrypt the messages with RSA, John must know Clara's public key, and likewise, to decrypt them, Clara must use her private key. Clara sends her public key to John over a secure pathway so he can reliably send his encrypted messages. Remember, Clara's private key is never disclosed.

Elliptic curve digital signature algorithm

The ECDSA algorithm is a variant of the digital signature algorithm (DSA), a federal information processing standard for digital signatures. 

Technically, the ECDSA algorithm relies on the algebraic structure of elliptic curves over finite fields. Elliptic curves may be used for key agreement, digital signatures, or pseudo-random generators, among other things.

For elliptic curve cryptography to work, all participating devices must possess a private and public key pair. A message is signed with a private key by its sender, and the recipient uses the sender's public key to verify its authenticity.

Messages that have been altered on route to the recipient will not pass the signature verification test, as the signature only applies to the original message.

Pre-shared key 

In cryptography, a pre-shared key (PSK) is a shared secret that two parties have previously shared through a secure channel. 

Wi-Fi encryption standards, including wired equivalent privacy (WEP), Wi-Fi protected access (WPA) and the extensible authentication protocol (EAP) use PSK in their encryption methods.  The wireless access points (AP) and the clients share the same authentication key.

A PSK may be a password, a passphrase, or a hexadecimal string. 

IPsec versus SSL: What’s the difference?

The‌ ‌IPsec‌ ‌protocol‌ ‌suite‌ ‌operates‌ ‌at‌ ‌the‌ ‌network‌ ‌layer‌ ‌in‌ ‌the‌ open systems interconnection

(‌OSI‌) ‌model. The secure sockets layer (SSL), on the other hand, operates at the application layer of the OSI model.  Each serves a very different purpose: IPsec encrypts IP packers, while SSL encrypts HTTP traffic and web-based transactions.

Featured Resources

Activation playbook: Deliver data that powers impactful, game-changing campaigns

Bringing together data and technology to drive better business outcomes

Free Download

In unpredictable times, a data strategy is key

Data processes are crucial to guide decisions and drive business growth

Free Download

Achieving resiliency with Everything-as-a-Service (XAAS)

Transforming the enterprise IT landscape

Free Download

What is contextual analytics?

Creating more customer value in HR software applications

Free Download


Paessler PRTG Network Monitor 21.4 review: Hard to beat
Network & Internet

Paessler PRTG Network Monitor 21.4 review: Hard to beat

15 Mar 2022
Best paying tech jobs of 2022
Careers & training

Best paying tech jobs of 2022

10 Mar 2022
Edge-to-cloud security webinar

Edge-to-cloud security webinar

8 Mar 2022
Edge-to-cloud security webinar

Edge-to-cloud security webinar

8 Mar 2022

Most Popular

Europe's first autonomous petrol station opens in Lisbon

Europe's first autonomous petrol station opens in Lisbon

23 May 2022
Nvidia pauses hiring to help cope with inflation
Careers & training

Nvidia pauses hiring to help cope with inflation

23 May 2022
Windows 11's nifty new search feature has one major downside
Microsoft Windows

Windows 11's nifty new search feature has one major downside

23 May 2022