Private VLANs: where to deploy them.

On this blog I’ve often covered Private VLANs: how to configure them, work around them and deploy them in a larger network. Yet it’s rarely that you see an actual Private VLAN in a design. Part of the problem is covered in the article about deployment over multiple switches: you can’t connect a trunked device such as a firewall to it. Although the Nexus 7000 provides a solution, that doesn’t make it much easier (or cheaper).

Another important reason is that few are willing to take the risk to deploy a VLAN where hosts cannot communicate with each other, as this is usually the reason hosts are put in the same VLAN in the first place. There’s the hesitation because it would introduce complexity or limit scalability, as new servers later on may need to communicate in the same subnet after all.

So where would it be beneficial and with low risk to use a Private VLAN? Actually quite a few places.

E-commerce

Say you have an internet-facing business with e-commerce websites where anyone can log in, create an account, or do a purchase. A compromised e-commerce server in the DMZ means immediate access to the entire DMZ VLAN. This VLAN has the highest chance of being compromised from the internet, yet the servers in it rarely need to speak with each other. In fact, if properly designed, they will all connect to backend application and/or database servers that on their turn communicate with each other. This way the e-commerce data is synchronised without the DMZ servers requiring a connection to each other.

Stepping Stones

Some environments have a VLAN with Stepping Stone servers where users can log on to with pre-installed tools to access confidential resources. Access from one Stepping Stone server to another is not needed here. Sometimes it’s even not desired as there may be a Stepping Stone per application, environment or third-party.

Out-of-Band
A modern rackserver has an out-of-band port to a dedicated chip in the server that can power off and on the server, and even install the OS remotely. For example, HP iLO. Typical here is that the out-of-band port never initiates connections but only receives connections for management, usually though the default gateway. This makes for a good Private VLAN deployment without issues.

Backup

Similar to out-of-band, some environments use a dedicated network card on all servers for backup. This introduces a security issue as it’s possible for two servers in different VLANs to communicate without a firewall in between. Again a Private VLAN can counter this. Somewhat unusual in the design is that it’s best to put the servers taking the backup in the promiscuous VLAN, so they can communicate with all servers and the backup VLAN default gateway, and put the default gateway in an isolated VLAN, preventing any other server from using it.

Campus – Wired guests
Similar to the Stepping Stones: guests can access the network through a firewall (the default gateway) but don’t need to access each others computers.

Campus – Wireless APs
In a WLAN deployment with a central controller (WLC), all the Lightweight APs do is connect to the controller using the subnet default gateway. Any other services such as DHCP and DNS will be through this default gateway as well.

Campus – Utilities
Utilities such as printers, camera’s, badge readers,… will likely only need the default gateway and not each other.

Where not to use PVLANs
This should give some nice examples already. But for last, a couple of places where not to use Private VLANs:

• Routing VLANs: unless you want to troubleshoot neighborships not coming up.
• VLANs with any kind of cluster in it: still doable with community VLANs for the cluster synchronisation, but usually better off in their own VLAN.
• User VLANs, VOIP VLANs and the like: VOIP and videoconferencing may set up point-to-point streams.
• Database server VLANs: not really clusters but they will often require access to each other.

Advertisement

Private VLANs over multiple devices.

I’ve discussed Private VLANs before and I still consider them a useful technology. However, most mentions and implementations of PVLANs only use one switch, and it’s not clear what the interactions of PVLANs with other devices are.

The above setup features three switches and a firewall, all connected through trunk links. SW1 and SW3 can use PVLANs, and SW2 can use VLANs but has no awareness of Private VLANs, as it’s a transit switch. The firewall acts as default gateway for all the VLANs, including the PVLANs (a real-world example would be an ASA 5510 here). The requirements of the setup are that an isolated port on one switch can’t talk with an isolated port on another switch, an isolated port on one switch can talk with a promiscuous port on another switch, it works over transit switches, and the trunk link can be used as default gateway. Will this scenario work? Let’s examine.

Cisco recommends to define both the isolated and promiscuous VLANs on all trunk links and transit switches. To remain consistent between switches, frames are always sent over trunk links tagged with the originating VLAN. A frame coming from an isolated VLAN will go over a trunk link tagged as the isolated VLAN. This way, the next switch knows it came from an isolated port and will only allow communication with either a promiscuous port or another trunk link that allows the isolated VLAN.

This logic solves the first requirements: the information of the origin port is carried over to the next switch, and PVLANs can work perfectly over multiple switches. Does this also include the unaware transit switch? Yes it does: it treats the PVLANs as just a VLAN and allows traffic over the trunk links. Of course, if the switch is not aware of Private VLANs, it’s not possible to configure access ports in those VLANs, isolated or promiscuous.

But how does return traffic behave? If a frame is sent from an isolated port to a promiscuous one (a default gateway), how does it return? It returns tagged as the promiscuous VLAN over the trunk links and can communicate to all ports: trunk , isolated and promiscuous ports. And this is where the flaw in the PVLAN design is: over trunk links, the switch expects return traffic to arrive over another VLAN. A ping from a device towards the default gateway goes over the trunks in the isolated VLAN and returns in the promiscuous VLAN. And a firewall with trunk links (e.g. an ASA) can’t handle this: it has no awareness of Private VLANs. It will send return traffic down the same VLAN, and once this arrives on the switch, the switch will think it’s a frame from an isolated VLAN access port and deny communication with all other isolated ports. Return traffic cannot reach the destination anymore.

In defence of  the ASA, not a single vendor has PVLAN awareness in firewalls or routers. And this cripples the technology in larger environments, as you’re constrained to use one access port per VLAN in a Private VLAN environment, and this doesn’t scale when dealing with many VLANs.

Update 13/03/2014: the test setup for this blog post did not cover MAC address learning so I missed a very important fact: non-PVLAN aware switches will flood traffic in the Private VLANs, as they will never learn the destination MAC addresses of traffic in the same VLAN. This will not scale. Thanks to the networking-forum.net members again for the nudge in the right direction!

Thought experiment: using PVLANs as a multipoint network for routing.

I’ve described in an earlier post how routing works in a hub-and-spoke Frame Relay network. The idea is that you have to get layer 3 services (routing) to work in an environment where no direct layer 2 contact is possible. You do this by using the hub router as a relay for all data.

In this post, I’m going to attempt to do the same in an Ethernet network. Normally, an Ethernet is a broadcast network, but Private VLANs form an exception to this rule: any data, even broadcasts, sent from an isolated port will not be received on other isolated ports. This is very similar to the hub-and-spoke Frame Relay, where spokes can’t communicate directly with each other.

I’m using the following setup:

After configuring the PVLANs on the switch (for details about this, see an earlier blog post), I configure all routers with EIGRP. Just a simple setup:

Router(config)#hostname Rx
Rx(config)#router eigrp 1
Rx(config-router)#network 10.0.0.0 0.0.255.255
Rx(config-router)#exit
Rx(config)#interface Loopback 0
Rx(config-int)#ip address 10.0.1.x 255.255.255.255
Rx(config-int)#exit
Rx(config)#interface Ethernet0/0
Rx(config-if)#ip address 10.0.0.x 255.255.255.0
Rx(config-if)#no shutdown

The ‘x’ represents the router number (R1, R2, R3). The loopback is configured to show that the routing protocol is working. After configuring all this, EIGRP adjacencies are formed between R1 and R2, and R1 and R3, but not between R2 and R3, as they can’t see each other because of the isolated PVLAN. A ‘show ip route’ on R2 and R3 shows the connected subnet, and one EIGRP learned route: the loopback from R1. Clearly, R2 and R3 don’t exist for each other.

But just like with the hub-and-spoke Frame Relay, this can be solved by disabling split-horizon on R1: in interface configuration mode, this is the ‘no ip split-horizon eigrp’ command. Neighbor associations are lost after this command and formed again quickly. R2 and R3 still aren’t neighbors, but their routing tables now also have routes for each other’s loopback adapters, learned by EIGRP. But they can’t reach this loopback: pings don’t work. The reason for this is of course in the PVLAN configuration, which prevents direct communication. And this is where it gets interesting: in the hub-and-spoke Frame Relay configuration, this was solved by adding static mappings to the PVCs so all frames are sent to the hub router, who then relays it to the correct spoke router. Can this be done here too?

It can: configure router R2 and R3 with static ARP information. But not with the ‘correct’ information: if you, for example, do a static arp entry on the R2 router for 10.0.0.3 with the MAC address of router R3, it still won’t work, as no direct communication is possible. Just like with Frame Relay, where you use the PVC of the hub router instead of the PVC of the spoke router, you make a static arp entry on R2 for 10.0.0.3 with the MAC address of R1! So, first do a ‘show interface Ethernet0/0’ on R1 to reveal the MAC address, in my case 0030.85e0.1d40. Next, on the other routers, make static arp entries:

R2(config)#arp 10.0.0.3 0030.85e0.1d40 arpa

R3(config)#arp 10.0.0.2 0030.85e0.1d40 arpa

Now all pings work! R1 relays the frames to the correct router, and adds the correct destination MAC address to the frames. I initially said ‘thought experiment’ here because it has little real-world use, but it does allow for interesting configurations, like ACL filtering on the central router (R1) in the same subnet.

First time configuration of Private VLANs.

Today I tried implementing Private VLANs for the first time.

Small explanation for PVLANs: with Private VLANs, you can provide segmentation of your existing VLAN, providing isolation and security for end devices. Devices put on an isolated port can only talk with promiscuous ports: usually the port going towards the gateway router. Community ports can talk with the promiscuous port and all other ports in their own community.

The illustration below is what I have set up to test it.

The switch is a Cisco 3560 series, capable of PVLANs, currently configured with VLAN 1 on all ports. This means that the IP Phone and the computer are in the same VLAN. Not a good practice, but since the router (provided by the ISP) does not support multiple VLANs and trunking, that’s what I have to work with. So to provide some form of security to the IP Phone, I’m going to put it in it’s own isolated PVLAN. The port to the router will be the promiscuous port. This way, the IP Phone will only be able to communicate with the gateway, segmenting it from the rest of the network.

The IP Phone has IP 192.168.0.106 and is connected to FastEthernet 0/2. The router is on FastEthernet 0/1. Before we start implementing the PVLAN, it can be pinged from the computer connected on FastEthernet 0/3.

Warning! Always configure PVLANs through the console port, or through a switchport that will not be affected by the PVLANs, otherwise you’ll lose connectivity during configuration.

First thing to do is putting VTP in transparent mode, as VTP version 1 and 2 don’t support PVLANs:
Switch(config)#vtp mode transparent

Next, we implement the PVLANs on the switch. I’ve choosen VLAN 4 as the promiscuous VLAN. VLAN 41 will be set on the port going to the IP Phone, the isolated port. Finally, all other ports will be put into PVLAN 42, a community VLAN, so all other devices can communicate with each other. The naming is just to make it easy in case of troubleshooting.
Switch(config)#vlan 4
Switch(config-vlan)#name PRIMARY
Switch(config-vlan)#exit
Switch(config)#vlan 41
Switch(config-vlan)#name ISOLATED
Switch(config-vlan)#exit
Switch(config)#vlan 42
Switch(config-vlan)#name COMMUNITY
Switch(config-vlan)#exit

Returning to VLAN 4 and binding all PVLANs together:
Switch(config)#vlan 4
Switch(config-vlan)#private-vlan primary
Switch(config-vlan)#private-vlan association 41,42
Switch(config-vlan)#exit
Switch(config)#vlan 41
Switch(config-vlan)#private-vlan isolated
Switch(config-vlan)#exit
Switch(config)#vlan 42
Switch(config-vlan)#private-vlan community
Switch(config-vlan)#exit

The creation of the VLANs has to be done first, otherwise the ‘association’ command will not work. Once this is done, we start binding ports to PVLANs.
First the port towards the router:
Switch(config)#interface f0/1
Switch(config-int)#switchport mode private-vlan promiscuous
Switch(config-int)#switchport private-vlan mapping 4 41,42

Then the isolated port:
Switch(config)#interface f0/2
Switch(config-int)#switchport mode private-vlan host
Switch(config-int)#switchport private-vlan host-association 4 41

And last, the community ports:
Switch(config)#interface range f0/3 – 24
Switch(config-int)#switchport mode private-vlan host
Switch(config-int)#switchport private-vlan host-association 4 42

That’s it. I can’t ping the IP Phone anymore, but I still have internet connectivity. Calling from the IP Phone works as usual.

That’s another task completed on my check-list towards CCNP!