Tag Archive: WAN


Advantages of MPLS: an example.

While MPLS is already explained on this blog, I often still get questions regarding the advantages over normal routing. A clear example I’ve also already discussed, but besides VRF awareness and routing of overlapping IP ranges, there’s also the advantage of reduced resources required (and thus scalability).

WANEdgeDesign

Given the above design: two routers connecting towards ISPs using eBGP sessions. These in turn connect to two enterprise routers, and those two enterprise routers connect towards two backend routers closer to (or in) the network core. All routers run a dynamic routing protocol (e.g. OSPF) and see each other and their loopbacks. However, the two middle routers in the design don’t have the resources to run a full BGP table so the WAN edge routers have iBGP sessions with the backend routers near the network core.

If you configure this as described and don’t add any additional configuration, this design will not work. The iBGP sessions will come up and exchange routes, but those routes will list the WAN edge router as the next hop. Since this next hop is not on a directly connected subnet to the backend routers, the received routes will not be installed in the routing table. The enterprise routers would not have any idea what to do with the packets anyway.

Update January 17th, 2014: the real reason a route will not be installed in the routing table is the iBGP synchronisation feature, which requires the IGP to have learned the BGP routes through redistribution before using the route. Still, synchronisation can be turned off and the two enterprise routers would drop the packets they receive.

There are a few workarounds to make this work:

  • Just propagating a default route of course, but since the WAN edge routers are not directly connected to each other and do not have an iBGP session, this makes the eBGP sessions useless. Some flows will go through one router, some through the other. This is not related to the best AS path, but to the internal (OSPF) routing.
  • Tunneling over the middle enterprise routers, e.g. GRE tunnels from the WAN edge routers towards the backend routers. Will work but requires multiple tunnels with little scalability and more complex troubleshooting.
  • Replacing the middle enterprise routers by switches so it becomes layer 2 and the WAN edge and backend routers have a directly connected subnet. Again this will work but requires design changes and introduces an extra level of troubleshooting (spanning tree).

So what if MPLS is added to the mix? By adding MPLS to these 6 routers (‘mpls ip’ on the interfaces and you’re set), LDP sessions will form… After which the backend routers will install all BGP routes in their routing tables!

The reason? LDP will advertise a label for each prefix in the internal network (check with ‘show mpls ldp bindings’) and a label will be learned for the interfaces (loopback) of the WAN edge routers… After which the backend routers know they just have to send the packets towards the enterprise routers with the corresponding MPLS label.

And the enterprise routers? They have MPLS enabled on all interfaces and no longer use the routing table or FIB (Forwarding Information Base) for forwarding, but the LFIB (Label Forwarding Information Base). Since all packets received from the backend routers have a label which corresponds to the loopback of one of the WAN edge routers, they will forward the packet based on the label.

Result: the middle enterprise routers do not need to learn any external routes. This design is scalable and flexible, just adding a new router and configuring OSPF and MPLS will make it work. Since a full BGP table these days is well over 450,000 routes, the enterprise routers do not need to check a huge routing table for each packet which decreases resource usage (memory, CPU) and decreases latency and jitter.

Advertisements

In Europe, the cheapest WAN links start around 2 Mbps these days. While this makes some WAN optimizations covered in Cisco’s QoS guides unnecessary, it’s good to know them and the effects of slower links on traffic.

Serialization delay
Putting a frame on the wire from a switch or router requires time. The amount of time is directly related to the line speed of the link. Note ‘line speed’: the actual negotiated speed on layer 1. For a 1 Gbps interface negotiated to 100 Mbps Full Duplex which is QoS rate-limited at 10 Mbps, the line speed is 100 Mbps. The formula is the following:

SerializationDelay

This means that if you have a 1514 bytes frame (standard MTU of 1500 bytes plus the layer 2 header of 14 bytes) and send it out of a 100 Mbps interface it will take (1,500*8)/(10^8)= 0.12 ms or 121 µs. If a small voice frame arrives in the egress queue of a switch or router it can incur op to 121 µs of additional latency if a 1514 bytes frame is just being sent out. Consequence: even under near perfect conditions and good QoS configuration where voice frames are given absolute priority over the network, there’s a possible jitter per hop. The higher the general bandwidth throughout the network, the lower the jitter, so latency-sensitive traffic does benefit from high bandwidth and fewer hops. Over a 10 GE interface that same frame would be serialized in just 1.21 µs per hop.

There are some consequences for WAN links: at slower speeds, the serialization delay increases rapidly. At 2 Mbps for a 1514 byte frame it’s 6 ms. At 64 kbps, it’s 190 ms. And in case you’re enabling jumbo frames: 9014 bytes over 10 Mbps is 7.2 ms.

Link Fragmentation and interleaving
Generally, at 768 kbps and below, jitter for voice becomes unacceptable. This is where Link Fragmentation and Interleaving (LFI). It works by splitting up large frames into smaller parts and putting low-latency packets between these parts.

LFI

Configuration of LFI on a Cisco router is as following:

Router(config)#interface Multilink1
Router(config-if)#ip address 192.0.2.1 255.255.255.0
Router(config-if)#ppp multilink
Router(config-if)#ppp multilink fragment delay 3
Router(config-if)#ppp multilink interleave
Router(config-if)#ppp multilink group 1
Router(config-if)#exit
Router(config)#interface Serial0/0
Router(config-if)#
Router(config-if)#encapsulation ppp
Router(config-if)#clock rate 768000
Router(config-if)#ppp multilink
Router(config-if)#ppp multilink group 1
Router(config-if)#exit

First, create a Multilink interface. It will serve as an overlay for the actual physical interface, as this is where the LFI will be configured on. The Multilink interface will have all configuration: IP address, service policies,… Except for the layer 1 configuration (notice the clock rate command on the serial interface).

The ‘ppp multilink interleave’ activates LFI. The ‘ppp multilink fragment delay 3’ means LFI will automatically try to split up large packets so no packet has to wait longer than 3 ms while another is serialized. On the serial interface, encapsulation has to be set to ppp first. Next, it becomes possible to associate the interface with a Multilink overlay interface using the ‘ppp multilink group’ command.

The configuration has to be done on both sides of the WAN link, of course. The other side needs to use PPP encapsulation as well, and needs to have LFI enabled to reassemble to split up packets.

This concludes the series of QoS articles on this blog. Up next, I’ll try out different attacks on a Catalyst switch and see how it reacts.

Using MPLS on the WAN is a great way for multi-customer WAN connectivity, but so far it’s all layer 3. Layer 2 technologies like Ethernet over MPLS (EoMPLS) exist, but I have  little experience with it so far, so I’m not covering it (yet).

Instead, something else: layer 2 WAN services. Let’s assume you need to transport layer 2 VLANs from multiple customers over WAN links. Using a dedicated line per customer makes configuration easy, but it’s not cost-efficient. Instead, you want to use a mesh of switches that transport the customer data. The concept is simple enough, but it has some practical limitations: you can’t have overlapping VLAN numbers between customers.

This is where Metro Ethernet switches step into the picture. In particular, I had the chance to work with a ME-3600X-24CX-M switch, and it’s something entirely different from a Catalyst switch.

ME-3600X-24CX-M

The difference is that this switch is made for MAN and WAN networks: it can run MPLS, supports VRF and has support for Ethernet Virtual Circuits (EVC).

EVC is a technology where you no longer use the 802.1q header of a frame only to specify to which network a frame belongs, but you use it for a number of different things, such as identifying a customer for example. It even allows multiple tags on a frame. Configuration of EVC is done on an interface (physical or port-channel), by making it a trunk, and the command ‘switchport trunk allowed vlan none’. Why this command? Because it makes the interface step out of its typical trunk mode and into EVC, after which service instances can be defined that match on 802.1q headers. An example configuration, where G0/1 and G0/2 are customer links and G0/3 is a shared WAN link:

Switch(config)#interface GigabitEthernet0/1
Switch(config-if)#switchport trunk allowed vlan none
Switch(config-if)#switchport mode trunk
Switch(config-if)#service instance 1 ethernet
Switch(config-if-srv)#encapsulation dot1q 100-200
Switch(config-if-srv)#bridge-domain 5010
Switch(config-if-srv)#exit
Switch(config-if)#exit
Switch(config)#interface GigabitEthernet0/2
Switch(config-if)#switchport trunk allowed vlan none
Switch(config-if)#switchport mode trunk
Switch(config-if)#service instance 1 ethernet
Switch(config-if-srv)#encapsulation default
Switch(config-if-srv)#bridge-domain 5020
Switch(config-if-srv)#exit
Switch(config-if)#exit

This seems complicated at first sight, but let’s look at the configuration lines one by one:

  • As said before, ‘switchport trunk allowed vlan none’ activates EVC.
  • ‘service instance number ethernet’ defines a policy, matching on 802.1q headers.
  • The ‘encapsulation dot1q 100-200’ makes that this service instance will be applied to all incoming frames with an 802.1q header between 100 and 200.
  • ‘bridge-domain 5010’ places these frames in a bridge domain. A bridge domain is like a VLAN, but it doesn’t use tags. All frames of the service instance are put in bridge domain 5010. All frames of the service instance on the next port are put in bridge-domain 5020. This way, even if there are overlapping VLANs, the differing bridge-domains keep them separated inside the switch.
  • ‘encapsulation default’ means anything not defined before. If it is the only service instance on the link, as in this case, it means every incoming frame.

So the configuration of the customer links makes sure the frames stay separated (in bridge-domains) even if there is a VLAN overlap. The shared WAN uplink is now as following:

Switch(config)#interface GigabitEthernet0/3
Switch(config-if)#switchport trunk allowed vlan none
Switch(config-if)#switchport mode trunk
Switch(config-if)#service instance 1 ethernet
Switch(config-if-srv)#encapsulation dot1q 10
Switch(config-if-srv)#rewrite ingress tag pop 1 symmetric
Switch(config-if-srv)#bridge-domain 5010
Switch(config-if-srv)#exit
Switch(config-if)#service instance 2 ethernet
Switch(config-if-srv)#encapsulation dot1q 20
Switch(config-if-srv)#rewrite ingress tag pop 1 symmetric
Switch(config-if-srv)#bridge-domain 5020
Switch(config-if-srv)#exit
Switch(config-if)#exit

Note there are two service instances, one per customer in this case:

  • Both instances refer to different bridge-domains, keeping the separation per customer.
  • Both match one 802.1q tag on incoming frames: 10 for bridge-domain 5010, customer G0/1, 20 for bridge-domain 5020, customer G0/2
  • ‘rewrite ingress tag pop 1’ means incoming frames will have their outer 802.1q tag removed.
  • Finally, the ‘symmetric’ keyword means that outgoing frames will have an 802.1q tag added: the one defined in the service instance.

This means the following: a frame entering from G0/1 will have a second 802.1q applied, VLAN 10, if it goes over the WAN link. Same for G0/2 towards the WAN link, only with VLAN 20. This means that if both customers send a frame with VLAN 100, the G0/1 frame will be sent out of G0/3 with inner 802.1q header 100 and outer header 10. The G0/2 frame will have inner header 100 also, but outer header 20. Separation between both customers is ensured, even with overlapping VLANs! A remote Metro switch on the other end of the link can use similar service instances again to remove the outer header and place each frame in its own bridge-domain.

Dot1q2

This is just one of the things possible with EVC. You can add 802.1q headers, remove them, but also translate them, by matching different headers on different interfaces in the same bridge-domain:

Switch(config)#interface GigabitEthernet0/4
Switch(config-if)#switchport trunk allowed vlan none
Switch(config-if)#switchport mode trunk
Switch(config-if)#service instance 1 ethernet
Switch(config-if-srv)#encapsulation dot1q 30
Switch(config-if-srv)#rewrite ingress tag pop 1 symmetric
Switch(config-if-srv)#bridge-domain 5030
Switch(config-if-srv)#exit
Switch(config-if)#exit
Switch(config)#interface GigabitEthernet0/5
Switch(config-if)#switchport trunk allowed vlan none
Switch(config-if)#switchport mode trunk
Switch(config-if)#service instance 1 ethernet
Switch(config-if-srv)#encapsulation dot1q 60
Switch(config-if-srv)#rewrite ingress tag pop 1 symmetric
Switch(config-if-srv)#bridge-domain 5030
Switch(config-if-srv)#exit
Switch(config-if)#exit

This configuration translates frames incoming on G0/4, VLAN 30, to frames tagged as VLAN 60 on G0/5. Again, the ‘symmetric’ keyword makes it work in both directions.

Dot1qTranslated

The switch does all of this in hardware forwarding, so gigabit bandwidth while forwarding and changing headers is possible. When headers are added, the MTU of the link may need to be increased though. And what if you want a layer 3 VLAN interface on the switch to manage it remotely? Can you enter through a link using EVC? Yes, you can: so far, I’ve used bridge domains above 4094, but if you define a bridge-domain below 4094, the VLAN of the same number will automatically be created on the switch, and its layer 3 interface is automatically part of that bridge domain. You do need to remove the 802.1q header(s) for incoming frames, as the frames need to reach the layer 3 interface ‘untagged’ internally. Assume you come in over the WAN link, outer header VLAN 30, inner header 10:

Switch(config)#interface GigabitEthernet0/3
Switch(config-if)#switchport trunk allowed vlan none
Switch(config-if)#switchport mode trunk
Switch(config-if)#service instance 3 ethernet
Switch(config-if-srv)#encapsulation dot1q 30 second-dot1q 10
Switch(config-if-srv)#rewrite ingress tag pop 2 symmetric
Switch(config-if-srv)#bridge-domain 50
Switch(config-if-srv)#exit
Switch(config-if)#exit
Switch(config)#interface Vlan50
Switch(config-if)#description Management
Switch(config-if)#ip address address subnetmask
Switch(config-if)#exit

As you can see, both 802.1q headers will be stripped off and the frame is placed in bridge-domain 50. This way it reaches VLAN interface 50. Other customers can’t reach this VLAN interface because they’re not part of the same bridge-domain.

Cisco support forums has a nice document explaining the concept, also together with MPLS. Be sure to check it out if you want to know more!

MPLS, part II: VRF-aware MPLS-VPN.

Where MPLS part I explains the basics of labeling packets, it’s not giving any advantage over normal routing, apart from faster table lookups. But extensions to MPLS allow for more. In this article I’ll explain MPLS-VPN, and more specifically a Virtual Private Routed Network (VPRN).

A VPRN is a routed (layer 3) network over an MPLS cloud, that is VRF-aware, or customer-aware. This means several different routing instances (VRFs, remember?) can share the same MPLS cloud. How is this achieved when there’s only a point-to-point link between routers with one IP address? After all, an interface can only be assigned to one VRF. Solution: by adding a second MPLS label to the data: the ‘outer’ label (the one closest to the layer 2 header) is used to specify the destination router, the ‘inner’ label (closest to the original layer 3 header) is used to specify to which VRF a packet belongs. The outer label workings are identical to standard MPLS: these are learned by LDP and matched with a prefix in the routing table. But for the inner label, a VRF-aware process needs to run on each router that can handle label information and propagate it to other routers. That process is Multiprotocol-BGP, or MP-BGP.

MPLS-VPN-Header

The outer label is used to route the packet through the MPLS cloud, and the last router(s) use the inner label to see to which VRF a packet belongs. Let’s look at the configuration to understand it more. First, the basic setup:

MPLS-VPN-VRF

Notice the router names, as these are often used in MPLS terminology.

  • The Customer Edge router a router that directly connects to a customer network. It’s usually the demarcation point, where the equipment governed by the MPLS provider begins. Contrary to the name, the CE itself is often managed by the provider as well.
  • The Provider Edge router is the ‘first’ router (seen from a customer site point of view) that has MPLS enabled interfaces. It’s where the labels are applied for the first time.
  • A Provider router is a router completely internal to the MPLS cloud, having only MPLS enabled interfaces.

The connection between CE and PE is a point-to-point so a /30 subnet is logical. Routing between CE and PE is done using a simple routing protocol, like RIP, OSPF, EIGRP or even static or standard BGP. The only notable part is that the PE router has to use a VRF to place the customer in. Below an example configuration:

Router-CE(config)#interface G0/1
Router-CE(config-if)#ip address 10.1.0.1 255.255.255.252
Router-CE(config-if)#exit
Router-CE(config)#router ospf 1
Router-CE(config-router)#network 10.1.0.0 0.0.0.3 area 0

Router-PE(config)#vrf definition VRF
Router-PE(config-vrf)#address-family ipv4
Router-PE(config-vrf-af)#exit
Router-PE(config-vrf)#exit
Router-PE(config)#interface G0/1
Router-PE(config-if)#vrf forwarding VRF
Router-PE(config-if)#ip address 10.1.0.2 255.255.255.252
Router-PE(config-if)#exit
Router-PE(config)#router ospf 2 vrf VRF
Router-PE(config-router)#network 10.1.0.0 0.0.0.3 area 0
*Mar  1 00:06:20.991: %OSPF-5-ADJCHG: Process 2, Nbr 10.1.0.1 on GigabitEthernet0/1 from LOADING to FULL, Loading Done

If you happen to run a router on an IOS before 15.0, the commands for the VRF change: it becomes ‘ip vrf VRF’ to define a VRF, without the need to specify an address family, as 12.x IOS versions aren’t VRF aware for IPv6. On the interface, the command is ‘ip vrf forwarding VRF’.

So far so good. Now on to activating MPLS between the PE and the P router, and making sure the routers learn the MPLS topology:

Router-PE(config)#interface G0/2
Router-PE(config-if)#mpls ip
Router-PE(config-if)#ip address 10.0.0.1 255.255.255.252
Router-PE(config-if)#ip ospf network point-to-point
Router-PE(config-if)#exit
Router-PE(config)#interface Loopback0
Router-PE(config-if)#ip address 10.0.1.1
Router-PE(config-if)#exit
Router-PE(config)#router ospf 1
Router-PE(config-router)#network 10.0.0.0 0.0.1.255 area 0

Router-P(config)#interface G0/1
Router-P(config-if)#mpls ip
Router-P(config-if)#ip address 10.0.0.2 255.255.255.252
Router-P(config-if)#ip ospf network point-to-point
Router-P(config-if)#exit
Router-P(config)#interface Loopback0
Router-P(config-if)#ip address 10.0.1.2 255.255.255.255
Router-P(config-if)#exit
Router-P(config)#router ospf 1
Router-P(config-router)#network 10.0.0.0 0.0.1.255 area 0
*Mar  1 00:02:29.023: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.0.1 on GigabitEthernet0/2 from LOADING to FULL, Loading Done
*Mar  1 00:02:33.127: %LDP-5-NBRCHG: LDP Neighbor 10.0.0.1:0 (1) is UP

The ‘ip ospf network point-to-point’ is not really needed but used to reduce OSPF overhead. The loopbacks are needed for BGP later on.

Up until this point, we have MPLS in the default VRF and a separate VRF per customer for routing, but no routing of the VRFs over the MPLS. To exchange the inner labels needed to specify the VRF, MP-BGP between PE and P is configured:

Router-PE(config)#router bgp 65000
Router-PE(config-router)#neighbor 10.0.1.2 remote-as 65000
Router-PE(config-router)#neighbor 10.0.1.2 update-source Loopback0
Router-PE(config-router)#no address-family ipv4
Router-PE(config-router)#address-family vpnv4
Router-PE(config-router-af)#neighbor 10.0.1.2 activate
Router-PE(config-router-af)#neighbor 10.0.1.2 send-community extended

Router-P(config)#router bgp 65000
Router-P(config-router)#neighbor 10.0.1.1 remote-as 65000
Router-P(config-router)#neighbor 10.0.1.1 update-source Loopback0
Router-P(config-router)#no address-family ipv4
Router-P(config-router)#address-family vpnv4
Router-P(config-router-af)#neighbor 10.0.1.1 activate
Router-P(config-router-af)#neighbor 10.0.1.1 send-community extended
*Mar  1 00:11:31.879: %BGP-5-ADJCHANGE: neighbor 10.0.1.1 Up

Again some explanation. First off, BGP neighbors always need to be defined under the main process, after which they need to be activated for a specific address-family. The ‘no address-family ipv4’ command means that no conventional routing information for the default VRF will be exchanged (we already have OSPF for that). The ‘address-family vpnv4’ activates the VPRN capability, and the label exchange for VRFs. In this process, the neighbor is activated. The ‘send-community extended’ means BGP will exchange the community Path Attributes (PA), in which the label information is present. The loopbacks are used to connect to each other, not only for redundancy in case a physical interface should go down, but also because LDP does not exchange labels with another router on a connected subnet for that subnet. This means that if the directly connected interfaces are used as BGP neighbors, the BGP process can’t figure out the labeling properly.

So VRF-aware MPLS is running now, but on each PE router it needs to be specified which VRF needs to be injected in the MPLS cloud. MP-BGP does this using import and export of routing tables. For MP-BGP, a route needs to be uniquely identified, and explained to which VRF it belongs. This is done with a Route Distinguisher (RD) and Route Target (RT). Both are 64 bits, and usually in the format AS:nn, or the first 32 bits the AS number and the last 32 bits a unique chosen number.

  • RD uniquely identifies a route. Inside MP-BGP, a route is prepended with its RD, e.g. 65000:1:192.168.1.0/24. This way, if the route exists twice (in different VRFs), it’s still unique because the RD part of the prefix is different.
  • RT specifies to which VRF a route belongs. It handles the import and export of routes from a VRF to the BGP process. In its basic form, it’s the same number as the RD, and the same at all PE routers for a certain client.

Configuration of these parameters is done inside the VRF:

Router-PE(config)#vrf definition VRF
Router-PE(config-vrf)#rd 65000:1
Router-PE(config-vrf)#route-target both 65000:1

Now that the VRF can be used in the BGP process, it is imported in the process as following:

Router-PE(config)#router bgp 65000
Router-PE(config-router)#address-family ipv4 vrf VRF
Router-PE(config-router-af)#redistribute ospf 2

The redistribution, of course, needs to be mutual between OSPF and BGP, so a few more lines of configuration are needed to complete everything:

Router-PE(config)#router ospf 2 vrf VRF
Router-PE(config-router)#redistribute bgp 65000 subnets

MPLS-VPN-Forwarding

And now everything is completed: the PE router learns routes from the P router by OSPF, and redistributes it into BGP to propagate them over the MPLS cloud. From this point on, configuration is modular: on another PE router, the configuration is likewise. Adding a P router isn’t any different from the P router in this example, the processes and parameters are the same each time. Do remember that routers don’t advertise iBGP-learned routes to other iBGP peers, so the PE and P routers need to form a full mesh, unless you’re using route reflectors or confederations.

MPLS, part I: basic labeling

Multi Protocol Label Switching. Plenty of sites to explain it, yet I feel it’s something to cover on my blog, it just wouldn’t be complete without it. What is it? In the most simple form: it’s the use of labels, inserted on frames between layer 2 and 3, and use those labels to make forwarding decisions instead of the original layer 3 header.

Why is it done? Well, in the late 1990’s switching in hardware was possible, but routing (layer 3 forwarding) in hardware wasn’t. By using labels as a layer 3 IP header replacement, these labels could be used in more efficient forwarding tables, and eventually also in hardware. This meant lower latency in routed networks.

So that’s why up until this day, it is still discussed whether MPLS is a layer 2.5 or layer 3 protocol. I’m not going to make a judgement on that: by the standards it’s a layer 3 protocol (it’s above the layer 2 header, and ip-in-ip and gre over ip are not called layer 2.5 or 3.5), but on the other hand, the original use is to replace the use of the actual layer 3 header in forwarding, so 2.5 is logical in that regard.

MPLS-Header

The MPLS header is 4 bytes or 32 bits (just like 802.1q!). Routers need to agree on which labels they will use towards each other. For this, they need to set up a protocol that exchanges label information: Label Distribution Protocol (LDP). Cisco also has a proprietary Tag Distribution Protocol, but it’s no longer widely used. LDP sends Hello/Discover frames on UDP port 646 to 224.0.0.2 (all routers multicast). Once neighbor routers see each other, they build up a session on TCP port 646. The actual labels are automatically assigned and agreed on through these sessions. To enable MPLS and LDP on an interface, you’ll need one command:

Router(config-if)#mpls ip

But the labels by themselves aren’t enough. Each label represents a destination: in a basic environment, an IPv4 destination subnet. To know which label is needed for which destination, each MPLS router needs to know the possible destinations so it can assign a label to it: it needs a routing protocol. Generally speaking, it’s a better move to choose a routing protocol with awareness of the topology, so a link state protocol like OSPF or IS-IS.

MPLS-Switching

Once the routers know all routes and have set up TCP sessions for LDP, they can agree on which labels to use the reach a certain destination subnet. If the first router in the above picture receives a normal IPv4 packet, it sees in the forwarding table (the Forwarding Information Base, or FIB) that the next destination is out of a MPLS enabled interface, with a label value of 33 appended to it. If the second router receives a MPLS packet with label value 33, it can do a quick lookup using it’s Label Forwarding Information Base (LFIB) and find a next hop out of another interface with a new label value. This process continues until it reaches the last router of the MPLS region, where the LFIB next hop points to an interface without MPLS. The MPLS header is removed and the packet is forwarded as usual.

The above explains basic MPLS functionality. But it’s not everything: labels can stack, allowing for more complex configurations. In the next part, I’ll explain the use of multiple labels to create VRF-aware forwarding.

Routing through Frame Relay.

Frame Relay is a topic that is widely discussed inside the Cisco certifications, as you encounter it in both the CCNA and CCNP topics. Outside of that, however, I still have to see the first real-world Frame Relay circuit still in use, as MPLS is a more commonly used WAN technology these days.

As a full mesh, Frame Relay works quite similar to a normal Ethernet broadcast domain at layer 2 and layer 3. The complexity arises when Frame Relay is deployed in a partial mesh. I struggled a long time with the concepts of Frame Relay, so to help understand others the oddities of a partial mesh, I will explain things by the following topology:

Frame Relay Partial Mesh

In this topology  router R1 is the main site that has a Frame Relay PVC to all other sites. The branch sites, R2, R3, R4 and R5, each have one PVC back to R1. They don’t have PVCs to each other. This is a hub and spoke setup: R1 is the hub, the others are spokes. The routers use the 10.0.0.0/24 subnet, which each router an IP equal to it’s number, e.g. R1 has 10.0.0.1, R2 has 10.0.0.2. After configuring the Frame Relay switch, the initial setup for the routers is fairly simple:

Router(config)#interface Serial0/0
Router(config-if)#encapsulation frame-relay
Router(config-if)#ip address 10.0.0.x 255.255.255.0
Router(config-if)#no shutdown
Router(config-if)#exit

Since Cisco routers have the command ‘frame-relay reverse-arp’ activated by default, all routers find out the layer 3 address (IP) on each PVC (layer 2). R1 can ping every router, the others can ping R1 but not each other.

So far so good, now we configure EIGRP on all routers. We also configure a loopback interface that will be advertised: 10.0.1.1/24 on R1, 10.0.2.1/24 on R2, and so on.

Router(config)#interface Loopback 0
Router(config-if)#ip address 10.0.x.1 255.255.255.0
Router(config-if)#exit
Router(config)#router eigrp 1
Router(config-router)#network 10.0.0.0 0.0.255.255
Router(config-router)#end

Adjacencies from quickly, and R1 can see all routes now. But the four spoke routers can’t see each others routes. No problem, just disable the split-horizon rule of EIGRP on the hub router, so that received updates will be send out of the serial interface again to the others.

Router(config)#interface Serial0/0
Router(config-if)#no ip split-horizon eigrp 1
Router(config-if)#exit

After a graceful EIGRP restart, all spoke routers can see all routes. But despite this, a spoke router still can’t reach another spoke router’s loopback. Why not? Well, because the spoke routers have no idea how to get there. The frame-relay reverse-arp mentioned earlier finds out what’s at the other side of a PVC link, but it does not go further than that. Unable to gather the correct ARP information, packets will be dropped. To make it possible, you will need to add static mappings to the spoke routers. The PVC used is of course the only PVC possible on a spoke router.

Router(config)#interface Serial0/0
Router(config-if)#frame-relay map ip 10.0.0.x <pvc>
Router(config-if)#exit

Since the next hop defined in the routering protocol is now reachable, the spoke routers can ping each other’s loopbacks. Packets will be send to the hub router first, who will then relay these packets back out of the serial interface to another PVC. The single point of failure is the central hub, since everything depends on it.

In OSPF it’s the same: you will need to add these static mappings to make the actual forwarding work. The configuration of OSPF itself is different, as OSPF has no concept of split-horizon (that’s for distance vector protocols, not link-state). I personally prefer configuring the serial interface as broadcast, allowing for automatic neighbor discovery, and set the hub router as DR (router-id, and configured as first).

Router(config)#interface Serial0/0
Router(config-if)#ip ospf network broadcast
Router(config-if)#exit

The layer 2 topology now reflects the DR situation: a spoke router will send a link-state advertisement to the hub router (the only PVC), and the hub router send out a multicast to all spoke with the necessary information.

I hope that this is an informative post to those who want to better understand Frame Relay partial mesh.