oreilly.comSafari Books Online.Conferences.


AddThis Social Bookmark Button

An Inside Look at IPSec in Vista

by Mitch Tulloch

IPSec has traditionally been used to secure remote access connections using virtual private network (VPN) tunneling protocols such as Layer 2 Tunneling Protocol (L2TP). In the last few years this has been changing, though, as IPSec moves from the WAN into the LAN to secure internal network traffic against eavesdropping and modification. When two machines want to communicate using IPSec, they mutually authenticate with each other first and then negotiate how to encrypt and digitally sign traffic they exchange. These IPSec communication sessions are called security associations (SAs), and Microsoft Windows platforms use IPSec policies to determine how these associations are formed. An IPSec policy consists of a series of rules made up of filter lists and filter actions. Any traffic that matches a specific filter is processed according to the filter action associated with it. The whole thing is pretty complex to set up and manage, and though IPSec management tools were improved in Windows XP, they're not really very intuitive to use.

Things are going to be better in Windows Vista, at least to a degree. Let's look at how IPSec support is improving with this new platform, and also how some of these improvements will still take a while to materialize.

IPSec and the Next Generation TCP/IP Stack

One big change in Vista is in the TCP/IP networking stack itself. Vista has a totally revamped Next Generation TCP/IP stack that has a ton of enhancements with regard to performance, scalability, and extensibility. There's also a new architecture called Windows Filtering Platform (WFP) that provides APIs for accessing packets at virtually any point in the path as they are processed by the stack. These changes to the stack affect how IPSec works because of the addition of built-in callout functions that can be used for IPSec communications. A list of APIs for this feature can be found on MSDN if you're a developer interesting in building IPSec-aware applications and tools. Note that these APIs, like any other feature of Vista, are subject to change before RTM.

IPSec and Windows Firewall

Another change in Vista is that management of IPSec and Windows Firewall now are tied closely together. This is accomplished by integrating the firewall filtering functions and IPSec protection settings and managing them using a single snap-in called Windows Firewall with Advanced Security. There are also unified command-line tools you can use as well to manage both Windows Firewall and IPSec settings. In fact, even the Group Policy settings for Windows Firewall and IPSec are now in the same place with Vista and are found under Computer Configuration\Windows Settings\Security Settings\Windows Firewall with Advanced Security.

So why is all this a good thing? Well, host-based firewalls like Windows Firewall and security protocols like IPSec both do the same thing: filter packets. That means in existing Windows XP and Windows Server 2003 platforms, it's possible to set up firewall filters that conflict with IPSec policies and prevent network traffic from working the way you intend it to. With a single console for configuring both Windows Firewall and IPSec settings, there's less chance for errors like this to occur, which is good since IPSec problems are notoriously difficult to troubleshoot. Finally, the new console and command-line tools for managing Windows Firewall and IPSec settings are designed to make it a heck of a lot easier to configure IPSec policies in the first place. We'll have to wait for the first release candidate to appear, though, to see how all this works out, since these tools are still evolving and are in a state of flux.

One cool enhancement in this area is that Windows Firewall exceptions can be created to specify that incoming or outgoing traffic has to be protected using IPsec, and if you use these exceptions then you can also define which user or computer accounts (or groups of accounts) are authorized to initiate such communication sessions. This new level of granularity lets you specify that only traffic from specific users or computers should be accepted by a particular server on your network. The question is whether these enhancements on the client side will work with current Windows servers, or whether we'll have to wait for Longhorn Server to see these benefits fully realized.

IPSec, NAP, and Domain Isolation

Finally, let's return to the starting point of this article, namely, the changing use of IPSec in the enterprise. First, a Microsoft PressPass news release concerning the December 2005 Community Technology Preview (CTP) of Windows Vista says that the new integrated firewall/IPSec console "centralizes inbound and outbound traffic filtering along with IPSec server and domain isolation settings in the user interface." What's domain isolation? It's a name for a set of technologies including IPSec that can be used to prevent rogue workstations from accessing resources on an Active Directory-based network. Microsoft is already using domain isolation internally to increase the level of protection of their own company network, and other large enterprises have started deploying it too. And Vista is designed to help make domain isolation easier to implement--though Longhorn Server will probably be required for domain isolation to be truly simple to configure.

And second, Vista supports Network Access Protection (NAP), a new security technology that extends the Network Access Quarantine Control feature of Windows Server 2003 to help protect Active Directory-based networks from infected, misconfigured, or otherwise unhealthy client computers. Again, the problem is that to realize the full benefit of this technology, Longhorn Server will probably be required.


Enterprises are steadily making progress toward using IPSec to secure their internal networks, but the technology is not quite there yet as far as Windows platforms are concerned. Vista will change some of that, and Longhorn Server will bring this elusive goal even closer. Meanwhile, the enhancements to TCP/IP and the IPSec management improvements found in Vista will make IPSec easier to use in the enterprise and likely lead to more organizations adopting it as an inside network protection technology.

Mitch Tulloch is the author of Windows 2000 Administration in a Nutshell, Windows Server 2003 in a Nutshell, and Windows Server Hacks.

Windows Server Hacks

Related Reading

Windows Server Hacks
100 Industrial-Strength Tips & Tools
By Mitch Tulloch

Return to the Windows DevCenter.