A Virtual Private Network (VPN) is probably the most popular method to set up connections to share sensitive data. The technology works by linking your device to a VPN server creating a secure tunnel. Before initiating the VPN connection, there has to be a shared definition of how it should look.
A VPN protocol provides clear instructions on how to handle this connection. It specifies all the details that your device and server on the receiving end must know. Protocols come in all shapes and sizes. Some require additional data integrity checks while others add more robust encryption. The variety is immense, and there's plenty of room for customization.
This article will introduce the most popular VPN protocols and assess their benefits, differences, and use cases.
What are the main VPN protocols?
Data passing through a tunneling protocol is split up into pieces and placed inside encrypted data packets. After reaching their intended destination, the server decrypts packets with its configured private key and unpacks the sent data. The process goes back and forth between your device and a VPN server for the whole session.
While the core principle of how tunneling works remains constant, specific properties distinguish tunneling protocols from one another. Most hardware devices natively support them, and some are also considered standard.
Here's a brief rundown of some of the most popular tunneling protocols:
NordLynx (WireGuard)
WireGuard is an open-source tunneling protocol with streamlined code to minimize the attack surface and drastically improve maintenance. It relies on User Datagram Protocol (UDP), which skips handshaking to benefit from significantly increased speeds.
WireGuard cannot dynamically assign IP addresses to everyone connected to a server. To handle this and to make connections possible, the server needs an internal ledger. This registers each user, making sure that their packets are not mixed up. In a data breach, this could backfire as it would be easy to identify the user's identity assigned by the server.
NordLynx was created by the Nord Security team. Representing an improvement to the core of WireGuard's code, it solves the issue regarding user identities.
NordLynx differs from WireGuard by adding a double Network Address Translation system. It creates a local interface for each session, assigning a unique IP address to users and their tunnels. This solves the ledger problem without logging current users to trace the destination of each data packet. IP addresses remain assigned only when the session is active, providing unparalleled performance and security when connected to a VPN server.
Only Nord Security products like NordVPN and NordLayer have this tunneling protocol.
IKEv2
Internet Key Exchange version 2 is a VPN encryption protocol that handles request and response actions. It relies on two mechanisms for authentication and encryption. For encryption, it uses Diffie–Hellman key exchange protocol to set up a shared session. IKEv2's authentication is processed by X.509 public key infrastructure certificate, based on the International Telecommunications Union (ITU) standard.
IKEv2 belongs to an Internet Security Protocol suite that handles security associations (SA). Their job is to find mutually agreed conditions to establish a VPN tunnel. As IKEv2 uses UDP, it has relatively low latency and will be a speedy option for most use cases. It also isn't very demanding for performance, so works well on weaker hardware devices.
The IKE VPN protocol can also maintain the connection when switching between wi-fi and mobile data. Therefore, it could be a good option for mobile devices that rely on cellular data but can switch to wi-fi. In addition, IKEv2 also implements features like auto-connect when the connection drops.
OpenVPN
OpenVPN is the most widely-used VPN protocol. It is an open-source tunneling protocol and system that allows the establishment of secure point-to-point and site-to-site connections. Transport Layer Security, combined with OpenSSL cryptographic library algorithms, handles private key exchange to increase the safety of virtual private networks.
From a networking perspective, the OpenVPN protocol can operate in two modes: UDP or Transmission Control Protocol (TCP).
UDP — doesn't use handshaking, meaning sending data packets without receiving the confirmation that they have reached their destination. It's the same method used by WireGuard, and IKEv2 prioritizing speed rather than connection stability.
TCP — uses a three-way handshake between the initiator and the receiver. The client sends a request, the server acknowledges receipt, and the client replies with a confirmation. While this may seem like many extra steps, this helps to achieve a reliable connection at the downside of more considerable latency.
Both UDP and TCP slice your data into smaller packets when sending. They include the sender's and receiver's IPs, the actual data, and other configuration data. So, while it's a much more advanced iteration, OpenVPN still relies on the same tunneling principles.
L2TP
The Layer 2 Tunneling Protocol (L2TP) combines the Point to Point Tunneling Protocol (PPTP) and Layer 2 Forwarding to create VPN tunnels. On its own, L2TP does not provide encryption. That is generally supplied by IPSec (see below).
Originally published in 1999, L2TP creates a connection between the Access Concentrator (LAC) and the Network Server (LNS). This connection can feature multiple sessions and operates across Layer 2 of the OSI model.
One of the positives of the protocol is compatibility. PPTP is built into Microsoft Windows and MacOS. Setting up tunnels is easy via both operating systems, while L2TP is also compatible with Linux.
On the downside, the protocol on its own lacks encryption and authentication systems. For this reason it is supported by few modern VPNs.
IPSec
Internet Protocol Security (IPSec) works with L2TP in the form of L2TP/IPSec. IPSec provides both authentication and encryption for data passing through L2TP tunnels. This results in a much more robust way to send data securely across the web.
IPSec uses the AES cipher to encrypt data. The VPN protocol runs a security association operation using shared IKE keys. When a secure connection is available, the protocol creates an Encapsulated Security Payload (ESP). This wraps data packets in two layers of information, allowing two devices to communicate. It also hides the identity of the original sender with an extra layer of IP address data.
IPSec is categorized as a secure protocol. However, it is known that the NSA sought to compromise 256 bit AES encryption as part of its Bullrun program. The actual security status of the cipher is therefore uncertain.
IPSec speeds can be compromised by double encapsulation. And both L2TP and IPSec struggle to navigate firewalls. This can result in performance issues.
TLS
Transport Layer Security (TLS) has two elements: the TLS Record Protocol and the TLS Handshake Protocol. The Record Protocol creates secure connections, while Handshake anonymizes data for transmission.
TLS is an updated iteration of the Secure Socket Layer Protocol (SSL). It goes beyond SSL, offering HMAC authentication, PRF key generation, and the use of AES ciphers. Connections operate via Port 443, which is available in most situations. This also makes TLS a good VPN protocol for navigating firewalls.
TLS is generally browser based. It is regularly used as the basis for the Secure Socket Tunneling Protocol (SSTP) - the protocol of choice for always-on web-based VPNs.
In this context, TLS is used to connect web applications and user devices or servers, although it can also be used to encrypt Voice-over-IP connections. Browser based connections may also be clientless , reducing the need for additional VPN software.
The downside of TLS is that it only operates via browsers or supported apps. For this reason, users often combine IPSec and TLS VPNs for comprehensive network coverage.
What are the key differences between VPN protocols?
Whether you're self-hosting a VPN server or turning to a VPN provider, you'll have to pick a VPN protocol for your connection. It's one of the essential components of a VPN connection.
Here's an overview of how the most common protocols compare. Choosing the proper VPN protocol can benefit your internet activity and won't waste your resources when it's not needed.
Speed comparison
Research from Nord Security finds that the NordLynx VPN protocol can ramp up to 1200 Mbps, while IKEv2 only reaches 600 Mbps, and OpenVPN manages a best-recorded speed of 400 Mbps. The same tendencies repeat across the board no matter the distance between the VPN server and the client's location.
Based on these findings, if you're looking for the fastest secure tunneling protocol, you should go with NordLynx (or WireGuard). The second fastest will be IKEv2, which can confidently hold its own even when connecting to the other side of the world.
All of the above protocols tend to out-perform L2TP/IPSec in terms of speed. TLS also suffers from longer negotiation times, placing it below IKEv2/IPSec, OpenVPN and NordLynx.
Encryption
OpenVPN offers the strongest encryption via the OpenSSL library. It supports various cryptographic algorithms like AES, ChaCha20, Poly1305, etc. OpenVPN can also use hashing algorithms for credentials like MD5, BLAKE2, and others. RSA, DSA, and many other algorithms can process the protocol's private key derivation. The extent of customization options makes OpenVPN a very adaptable VPN protocol for many use cases that could be relevant to your VPN setup.
NordLynx/WireGuard is less flexible regarding encryption. Its encryption rests solely on ChaCha20, (which is considered safer than AES). Authentication is processed by the Poly1305 cryptographic function using BLAKE2 for hashing. Its Transport Layer provides only the UDP option.
IKEv2 provides more choice than NordLynx but slightly less then OpenVPN. It provides a variety of robust encryption algorithms to choose from, like AES 256 encryption, Blowfish, and others. The main difference is that it only has a UDP setting.
OpenVPN, NordLynx, and IKEv2 should be considered secure tunneling protocols. This is less true with regard to L2TP/IPSec and TLS.
Layer 2 Tunneling Protocol on its own offers no encryption but users can add ciphers over-the-top if desired. Experts now doubt that IPSec is totally secure, following information disclosed by Edward Snowden. IPSec is also not open-source, and users must trust in the security credentials of its developers.
SSL/TLS provides secure AES encryption via public key cryptography. However, the TLS protocol is less flexible than more advanced alternatives.
Security
While WireGuard is open-source, NordLynx is proprietary, which can be a problem if users always need total awareness of security app code. Despite this, NordLynx's WireGuard has no known security vulnerabilities. The same applies to OpenVPN.
However, since OpenVPN was around longer, independent testers had more time to iron out most of the bugs, meaning that it could be considered the most secure VPN option. WireGuard is still a work in progress.
IKEv2 is slightly less secure. According to a leaked NSA presentation, exploiting its protocol vulnerabilities resulted in successful decryption. While viewed as speculation, OpenVPN and WireGuard will be better options when aiming for maximum security.
SSL/TLS has been linked to man-in-the-middle attacks via untrusted security certificates. Users may be exposed by configuration errors. But the 256 bit AES encryption used by TLS is deemed secure by most experts.
L2TP/IPSec suffers from the same problems as IKEv2. Doubts have been raised about the protocol's security credentials, and it lies below OpenVPN and NordLynx in the secure VPN rankings.
Network port requirements
VPN users may experience connectivity issues if networks do not support a specific port used by their chosen VPN protocol. Some network administrators or ISPs may close ports to limit the attack surface. In the process, they may limit access to work resources with a VPN.
NordLynx and IKEv2 use UDP — regular internet traffic doesn't. The network's administrator can therefore disable it. If the port is closed, the VPN protocol won't be able to initiate a connection, which can be a problem.
OpenVPN is one of the best options regarding compatibility. The VPN protocol uses TCP port 443, the same as HTTPS. Even if limited web browsing is allowed, users should be able to pass through with OpenVPN.
L2TP/IPSec can run into problems with firewalls due to the use of double encapsulation. On its own the Layer 2 Tunneling protocol handles firewalls well, but lacks security features. On the other hand, TLS uses port 443. This allows TLS tunnels to pass through most firewalls easily.
Data usage
Data consumption usually increases when using a VPN protocol. This is because each data packet is repacked. Adding additional data on top of it increases its size. Over time this adds up, and when working on a metered connection, users can reach monthly data caps faster.
However, not all protocols have the same data overhead. For instance, OpenVPN adds up to 20%, while WireGuard (and NordLynx) will add only 4% more. IKEv2 is in between, with added 7%, which is still a good result. So, if you want to minimize your user data, you should stick with NordLynx.
L2TP/IPSec will generally exceed all of these protocols in terms of data usage. Double encapsulation adds extra overheads to every packet transmission. TLS uses more data than IKEv2 and NordLynx, but should consume less than OpenVPN.
Device compatibility
IKEv2, L2TP/IPSec and OpenVPN are generally best when it comes to compatibility.
Most devices natively support IKEv2, and it will be relatively easy to set up manual connections on them. Usually, users must pick the VPN protocol from the list and enter specific addresses and credentials.
OpenVPN relies on supplementary cryptographic libraries. Most manufacturers of consumer products don't include them by default, so you may be required to look into firmware flashing if you're setting it up on a router.
NordLynx and WireGuard can only be used via their apps, meaning every primary operating system is supported—most VPN providers' apps support both IKEv2 and OpenVPN.
Native L2TP/IPSec functionality is included with MS Windows and MacOS, and compatibility reaches all the way back to Windows 2000. Users should experience few problems when combining the VPN protocol with computers or routers.
TLS is browser based. It works with all major web browsers, but can only be used with web connections. This makes it less useful for installing on routers or protecting enterprise networks, but does make SSL/TLS useful for remote working connections.
Which VPN protocol should you use and when?
L2TP
Pros:
Excellent compatibility with Windows, Linux and MacOS
Easy pairing with IPSec for 256-bit encryption
Cons:
Lacks encryption on its own
Supported by relatively few modern VPNs
Slow speeds due to double encapsulation in L2TP/IPSec mode
Easily blocked by firewalls
Use Cases:
Basic networking situations with a wide variety of devices and operating systems
Use with legacy software where security is not a priority
Misuse Cases:
Bypassing firewalls and protecting sensitive data
Bandwidth heavy activities
IPSec
Pros:
Uses 256-bit AES encryption
ESP and public keys anonymize user identities effectively
Operates at network layer, visible to all applications
Cons:
Encryption may have been compromised by the NSA
Often blocked by firewalls
Wide access range may put data at risk
High data usage due to double encapsulation
Use Cases:
Building site-to-site VPNs
Misuse Cases:
High-security data transmission
Reliably traversing firewalls
TLS
Pros:
Works well with web applications and browsers
Good choice for always-on VPNs
Suited to clientless setups
Secure AES 256-bit encryption
Use of Port 443 means TLS can pass through most firewalls
Cons:
Inflexible as TLS only works with browser-based services
Long negotiation times can reduce speed
Linked to man-in-the-middle attacks
Higher data consumption than recent alternatives
Use Cases:
Applying VPN protection to web applications and browser traffic
Always-on clientless VPN setups
Misuse Cases:
General network protection beyond browser activity
High bandwidth data transmission
