Hakin9 - Hard Core IT Security Magazine

Access:

» How IPSec works

Related categories: Networks | Low-level network security | Security | Networks

Benoni Martin
Viewed: 9702 | Article date: 2006-08-01 12:57:46

You will learn all about the IPSec protocol which is used to secure IP data transmissions and is one of the most complicated network protocols.

The IPSec protocol is used to secure IP data transmissions and is one of the most complicated network protocols. It combines a variety of other protocols (AH, ESP, ISAKMP, IKE and others) which you need to be familiar with before using IPSec. The complexity is reflected in the large number of RFCs related to the protocol.

About the author

Bénoni Martin has been a security professional for over 4 years now. Initially working in the banking sector, he then moved to a VPN and firewall vendor. He is currently working as computer systems architect for a cell phone operator in Gabon. In his spare time, he maintains a personal website dealing with cryptography, security, telecommunications, networking and physics.

IETF created the IPSec protocol in order to provide TCP/IP security at layer 3 (the network layer of the OSI model), which means that IPSec transmissions are not bound to a specific port (such as 22 for SSH or 443 for HTTPS). Other popular secure protocols, such as SSL/TLS or SSH, secure layers 6 and 7 respectively.

IPSec can be used for host-to-host, host-to-gateway and gateway-to-gateway connections.

You will learn...

how IPSec works.

You should know...

the basics of TCP/UDP and IP protocols,
basic cryptographic techniques (pre-shared keys, Diffie-Hellman key agreement, certificates and digital signatures).

The first connection type can use transport mode or tunnel mode, while the other two work in tunnel mode only. The IP packet authentication and encryption provided by IPSec allows TCP-based transmissions to be completely secured. IPSec provides the following features and services:

Authentication - based on cookies (as discussed below), pre-shared keys, IP addresses, fully qualified distinguished names (FQDNs) and X.509 certificates.
Data integrity - hash functions are used to make sure that data arrives at the target host without modification. Integrity checking is based on message authentication codes (MACs) and hash MACs (HMACs).
Irrefutability - formal sender identification means that message authorship is undeniable. Irrefutability is ensured using digital signatures.
Privacy - data is encrypted to ensure that unauthorised parties cannot read it.
Replay prevention - this feature will be discussed in detail in the context of using PFS to prevent packet replay.

This functionality is provided by IPSec’s two subprotocols:

AH (Authentication Header) - data integrity and authentication;
ESP (Encapsulating Security Payload) - data privacy through encryption, potentially also covering authentication; used more often than AH.

Glossary

Digital certificate - a certificate that securely presents a public key belonging to a subject (individual, company, organisation) that owns the corresponding private key. The certificate is digitally signed by a trusted Certification Authority (CA), and its format is defined by the international standard ITU-T X.509.
Hash function (one-way function) - a mathematical function that transforms an input message into a hashed digest of a specified length (usually shorter than the original message) using an algorithm that is universally considered practically irreversible. The digest is related to the input message in the sense that only this input can be used to predictably generate this particular digest. Hash functions are commonly used to:

- perform authentication using a digital signature, for example when connecting to a VPN over IPSec,
- ensure data integrity, for example for downloaded files or e-mail (PGP),
- store user password fingerprints in some operating systems (notably Unix-based systems), so what is stored are just the password digests, not the actual passwords.

MAC (Message Authentication Code) - one of a family of algorithms for verifying message integrity and ensuring that the data have not been tampered with. MACs are calculated using symmetric key algorithms, typically 3DES.
HMAC (Hash-based MAC) - a MAC combined with a hash function. Like MACs, HMACs are calculated using symmetric key algorithms, and the most common HMAC families are HMAC-SHAx and HMAC-MDx.

Digital signature - the result of encrypting a message digest. The message to be signed is first hashed to create a digest, and then the digest is encrypted using the sender’s private key and can only be correctly decrypted using the corresponding public key (hence the asymmetric nature of the signing process). A digital signature ensures:

- Authentication: the private key should be known only to the sender, so the sender is the only party that can generate a specific signature for a specific digest (unless the key is compromised). The public key should be available to all parties who want to verify the sender’s authenticity.
- Irrefutability: the sender cannot deny sending a signed message, because the correct signature for a specific message can be generated only using the sender’s private key (unless the key is compromised).
- Integrity: the message recipient can calculate a digest of the message and compare it to the digest calculated by decrypting the signature using the sender’s public key - if the two digests are identical, the message is exactly the one that was sent.

Key types and assignment

Keys can be assigned manually or automatically. Manual assignment requires the system administrator to physically configure a separate key for each managed host. Obviously, this is only workable for static and relatively small networks. Automatic key agreement can be done via DNS, using an asymmetric agreement algorithm - typical algorithms include ISAKMP, OAKLEY and IKE.

Three main types of keys exist:

Key encryption keys - used to encrypt large or long-lived keys.
Traffic encryption keys - used to encrypt and decrypt messages. Even if the data is very large, the encryption/decryption process should be as fast a possible, hence the use of fast symmetric keys. Their relative vulnerability is usually compensated by their short lifetime (typically no longer than 10 minutes, as for example in the default settings of the VPN feature of the NetASQ firewall).
Master keys - used to derive other keys, for example encryption or digital signature keys.

IPSec in detail

Let’s have a closer look at the components that contribute to an IPSec.

Frame 9 (354 bytes on wire, 354 bytes captured)     
Arrival Time: Jan 20, 2006 17:26:50.985181000     
Time delta from previous packet: 0.000016000 seconds     
Time since reference or first frame: 60.784231000 seconds     
Frame Number: 9     
Packet Length: 354 bytes     
Capture Length: 354 bytes     
Protocols in frame: eth:ip:udp:isakmp Ethernet II, 
Src: xxx.xxx.48.123 (00:02:3f:76:70:a9), 
Dst: xxx.xxx.48.122 (00:0b:cd:cc:22:5c)     
Destination: xxx.xxx.48.122 (00:0b:cd:cc:22:5c)     
Source: xxx.xxx.48.123 (00:02:3f:76:70:a9)     
Type: IP (0x0800) Internet Protocol, 
Src: xxx.xxx.48.123 (xxx.xxx.48.123), 
Dst: xxx.xxx.48.122 (xxx.xxx.48.122)     
Version: 4     
Header length: 20 bytes     
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)         
0000 00.. = Differentiated Services Codepoint: Default (0x00)         
.... ..0. = ECN-Capable Transport (ECT): 0         .... ...0 = ECN-CE: 0     
Total Length: 340     Identification: 0x038a (906)     Flags: 0x00         
0... = Reserved bit: Not set         .0.. = Don't fragment: Not set         
..0. = More fragments: Not set     Fragment offset: 0     Time to live: 128     
Protocol: UDP (0x11)     Header checksum: 0x33f1 [correct]     
Source: xxx.xxx.48.123 (xxx.xxx.48.123)     
Destination: xxx.xxx.48.122 (xxx.xxx.48.122) 
User Datagram Protocol, Src Port: isakmp (500), Dst Port: isakmp (500)     
Source port: isakmp (500)     Destination port: isakmp (500)     Length: 320     
Checksum: 0x8200 [correct] Internet Security Association and Key Management Protocol     
Initiator cookie: 0x9426A39D294F08B0     Responder cookie: 0x0000000000000000     
Next payload: Security Association (1)     Version: 1.0     
Exchange type: Identity Protection (Main Mode) (2)     Flags         
.... ...0 = Not encrypted         .... ..0. = No commit         
.... .0.. = No authentication     Message ID: 0x00000000     Length: 312    
 Security Association payload         Next payload: Vendor ID (13)         
Length: 200         Domain of interpretation: IPSEC (1)         
Situation: IDENTITY (1)         Proposal payload # 1             
Next payload: NONE (0)             Length: 188             Proposal number: 1             
Protocol ID: ISAKMP (1)             SPI size: 0             Number of transforms: 5             
Transform payload # 1                 Next payload: Transform (3)                 
Length: 36                 Transform number: 1                 
Transform ID: KEY_IKE (1)                 Encryption-Algorithm (1): 3DES-CBC (5) 
                Hash-Algorithm (2): SHA (2)                 
Group-Description (4): 2048 bit MODP group (14)                 
Authentication-Method (3): PSK (1)                 
Life-Type (11): Seconds (1)                 
Life-Duration (12): Duration-Value (28800)             
Transform payload # 2                 Next payload: Transform (3)                 
Length: 36                 Transform number: 2                 
Transform ID: KEY_IKE (1)                 
Encryption-Algorithm (1): 3DES-CBC (5)                 
Hash-Algorithm (2): SHA (2)                 
Group-Description (4): Alternate 1024-bit MODP group (2)                 
Authentication-Method (3): PSK (1)                 
Life-Type (11): Seconds (1)                 
Life-Duration (12): Duration-Value (28800)             
Transform payload # 3                 Next payload: Transform (3)                 
Length: 36                 Transform number: 3                 
Transform ID: KEY_IKE (1)                 Encryption-Algorithm (1): 3DES-CBC (5)
                 Hash-Algorithm (2): MD5 (1)                 
Group-Description (4): Alternate 1024-bit MODP group (2) 
                Authentication-Method (3): PSK (1)                 
Life-Type (11): Seconds (1)
Life-Duration (12): Duration-Value (28800)             
Transform payload # 4                 Next payload: Transform (3)                
 Length: 36                 Transform number: 4                 
Transform ID: KEY_IKE (1)                 Encryption-Algorithm (1): DES-CBC (1)   
              Hash-Algorithm (2): SHA (2)                 
Group-Description (4): Default 768-bit MODP group (1)                
 Authentication-Method (3): PSK (1)                
 Life-Type (11): Seconds (1)                 
Life-Duration (12): Duration-Value (28800)           
  Transform payload # 5                 Next payload: NONE (0)   
              Length: 36                 Transform number: 5         
        Transform ID: KEY_IKE (1)                
 Encryption-Algorithm (1): DES-CBC (1)             
    Hash-Algorithm (2): MD5 (1)                
 Group-Description (4): Default 768-bit MODP group (1)              
   Authentication-Method (3): PSK (1)                 
Life-Type (11): Seconds (1)                 
Life-Duration (12): Duration-Value (28800)     
Vendor ID payload         Next payload: Vendor ID (13)        
 Length: 24         Vendor ID: MS NT5 ISAKMPOAKLEY     
Vendor ID payload         Next payload: Vendor ID (13)         
Length: 20         Vendor ID: Microsoft L2TP/IPSec VPN Client     
Vendor ID payload         Next payload: Vendor ID (13)        
 Length: 20         Vendor ID: draft-ietf-ipsec-nat-t-ike-02    
 Vendor ID payload         Next payload: NONE (0)         
Length: 20         Vendor ID: unknown vendor ID: 0x26244D38EDDB61B3172A36E3D0CFB819

A d v e r t i s e m e n t

Page: 1 2 3 4 5 6 7

Buy article		Buy subscription

Standard price: 2€/$3		Standard price: 25€/$30
Buy article for as little as (2€/$3) each allow access to individual articles.		Buy a full access to our Hakin9 archive portal. You will be able to read the articles from all archive issues from year 2005 and 2006. For just 25€/$30 you get unrestricted access to the entire website for the whole year.