Linux, FreeBSD, Juniper, Cisco / Network security articles and troubleshooting guides

Juniper Firefly (vSRX) 12.1X47 chassis cluster under Vmware ESXi5.5

vSRX cluster - Introduction

FireFly Perimeter, or vSRX, is the virtualized version of Juniper SRX branch firewalls. The first version released by Juniper was based on Junos 12.1X46 with limited feature support. At the time of this writing, FireFly evaluation version is based on 12.1X47-D10.

Some of the features supported by Juniper vSRX in X47 are: ALGs, IPSEC, MPLS, BGP, OSPF, ISIS, IDP, AppFW and others. LACP in chassis cluster is not yet supported.
For a full list of features, visit page 18 from the Firefly Perimeter Getting Started Guide for VMware  at http://www.juniper.net/techpubs/en_US/firefly12.1x47/information-products/pathway-pages/security-virtual-perimeter-vmware-gs-guide-pwp.pdf.

In this post, we will configure a vSRX chassis cluster made of two 12.1X47-D10 FireFly virtual machines running in Vmware ESXi 5.5 hypervisor. Both will be running in the same hypervisor, but be advised this is not best practice. Juniper recommends both FireFly virtual machines to be deployed on separate physical hosts - for obvious reasons.

Before we begin, you should know that building a cluster of VSRX firewalls requires to understand and combine the Juniper chassis cluster reth interface concept as well as the functionality of Vmware standard vSwitch security policies.

Juniper chassis cluster - redundant ethernet (reth) interface concept

A redundant reth interface is a cluster specific virtual interface that is assigned to a cluster redundancy group (only one-to-one mapping is allowed). The reth interface, normally, has at least two members: a ge- or xe- interface from each of the cluster nodes. All redundancy groups in an srx chassis cluster (vsrx or physical firewalls) can only be active on one node at a time, thus allowing the physical ge/xe interface on that node, for that specific reth interface, to forward traffic only.
The RETH interface will not use the mac address of any of the member physical interfaces, but it will have it’s own virtual mac address based on the chassis cluster id, interface number and few other parameters. For more details on the reth interfaces mac address format in an srx HA cluster, visit: http://kb.juniper.net/InfoCenter/index?page=content&id=KB15911. An expressive image representation of how a reth interfaces looks in a network can be seen in the following image: .

So how does the srx/FireFly cluster notify the neighboring switches which cluster node is active ?
When the redundancy group fails over from one node to the other, the new active node sends multiple (configurable iirc) gratuitous arp requests (or replies ? Correct me if I’m wrong).

Vmware standard vSwitch security and settings

Understanding reth concept is key in world of Vmware esxi because, by default, esxi allocates mac addresses to all VMs, for each vmnic and does not allow incoming traffic from the VM with a different mac address. In order to change this behavior to successfully bring up a virtual srx cluster, note two settings that will need to be changed for the vSwitches involved in the setup:
MAC Address Changes: Accept
Forged Transmits: Accept

SRX Cluster - other special interfaces

The control and fabric links are special devices in any SRX based chassis cluster as they have special requirements: an MTU of 9000 bytes and no interference/garbage traffic > they need their special private vlans to ensure a proper cluster functionality. The control link is used for direct control plane communication between the two cluster nodes whereas the fabric link is the dataplane communication. Juniper explains these at http://www.juniper.net/documentation/en_US/junos11.4/topics/concept/chassis-cluster-fabric-links-understanding.html.

I will list below the steps required (in my case) to deploy the vsrx cluster in ESXi 5.5.

Step 1 - Setting up the control/fabric vSwitch in ESXi

The official guide uses two separate vSwitches for control and fabric links. For testing, I’m using only one with special care: the switch will have an increased MTU of 9000 bytes, will have two ports assigned to two different vlans (10 and 11) to separate them. Here is my esxi vswitch configuration:

Step 2 - Deploy the FireFly virtual machines using the OVF Template

In your vsphere client, go to File->Deploy OVF Template->Browse for the image (junos-vsrx-12.1X47-D10.4-domestic.ova) and hit NEXT.
You will have to choose the names of each FireFly VM. I will use “VM10-vSRX-1” and “VM11-vSRX-2”. The rest of the steps need no change, just use the preset choices.

Step 3 - Setup each vSRX initial networking configuration

.
Following configuration requires a reboot of the VMs.
As with any branch SRX firewall, after cluster is enabled and rebooted, the box stops behaving like an individual firewall and starts behaving like a cluster node. For FireFly this means that:
ge-0/0/0 - will be renamed to fxp0 - out of band management - mapped to the routing engine / control plane and does not go through flow (and security settings don’t apply to it).
ge-0/0/1 - will be renamed to fxp1 - cluster control link.
ge-0/0/2 - (optional) will be used as cluster fabric link.

The figure shows the following virtual machine network interfaces and their mapping to the vSRX ge- interfaces:
Network adapter 1 - ge-0/0/0 - fxp0 - connects to “VM Network” for the management LAN.
Network adapter 2 - ge-0/0/1 - fxp1 - connects to “FF-Control” port on vlan10.
Network adapter 3 - ge-0/0/2 - fab0/fab1 (fab0 on node0, fab1 on node1) - connects to “FF-Fab” port on vlan11.

Another thing to note is that node0 interfaces will numbered on FPC0 of the cluster (ge-0*) and node1 interfaces will be on PIC7 (ge-7*) AFTER the cluster has been rebooted.

Step 4 - vSRX enabling the cluster and initial configuration

This step requires to be performed at the vsrx console in the esxi client.
Since two ge- interfaces will be renamed, they will need remapping at Junos boot time so it is mandatory that the configuration DOES NOT contain ge-0/0/0, ge-0/0/1 nor ge-0/0/2 (ge-0/0/2 will be used in my case as fabric links, but this can be configured).
Activate root password (junos does not allow any commits if root user does not have a password)
From the console login as “root”, no password will be required. Start the “cli”, enter and go to configuration mode “conf”.


Note root password has to be at least 6 characters in length.
Preparing the nodes enable the chassis cluster
Warning: Console is still required.
From ESXi console, run the following commands for each node individually if you are still in configuration mode:
vSRX-1:

vSRX-2:

Note that any reference to ge-0/0/0 or ge-0/0/1 in the configuration, will render prevent the vsrx to act as a cluster node after reboot because PFE will not be initialized and the following error will be displayed:

Step 5 - initial chassis cluster configuration (post reboot)

After reboot, login in both nodes, start the CLI and their prompt will be “{primary:node0}” and “{secondary:node1}”. This means that they negotiated mastership and they are acting as one chassis.
This means that the rest of configuration can be deployed only from one node (preferably primary - node0 in my case) and the changes will be committed to the other node also.

From vSRX-1 console, commit the following configuration:

The above creates configuration group for each node, sets the hostname and management interface IP address, instructs both nodes to use the group assigned to it and commits.
Note: If any node has a wrong hostname, check that the system hostname is not configured globally (“# system host-name”). Remove it if it is.

Step 6 - Setting up the fabric interfaces

Step 7 - Checking the health state of the cluster interfaces so far

.
I’m interested in the control and fabric link status in the “> show chassis cluster interfaces” status. This output does not indicate physical connection, but rather node liveliness between the nodes (keepalives and heartbeats are sent and received correctly by both nodes).

Good. It is time for the rest of the article. Before continuing let’s add one more port ge-0/0/3 to each FireFly instance that will be a trunk and that will have access to the physical network as the vSwitch will also contain a physical network card. !!! It is important that both virtual machines have IDENTICAL network configuration.

Step 8 - Add “Network adapter 4” or “ge-0/0/3”/“ge-7/0/3” on both virtual machines

Before starting, power off both FireFly VMs to add extra interface.
In an esxi vSwitch that’s ready to accomodate the vSRX cluster forwarding interfaces and that has a physical nic added, add one more port. In my case this will be a trunk as I will use subinterfaces (IFL) that are vlan tagged.


Power on both VMs.

Step 9 - Enable redundant ethernet interfaces, add redundancy group and wrap up

It is time to set the “reth-count” to reflect our deployment. I will set it to 2, but in fact I only use one reth interface (with ge-0/0/3 and ge-7/0/3 members).
I will create to redundancy groups RG0 (routing engine) and RG1 (data plane 1) and will assign the reth interface to RG1.

Enable usage of reth interfaces in vSRX cluster and configure RG0 and RG1

Configure reth0 with ge-0/0/3 and ge-7/0/3 members as members


Reth0 interface will now terminate a trunk port (All Vlans) and reth0.10 will operate on vlan10 with an IP of 172.16.10.1.

Step 10 - Final checkups

The above shows that control and fabric links are up. We can manage both nodes through fxp0 management interface and the redundant interface (that forwards traffic) is up and can ping a host in it’s subnet (and learn it’s mac address).

I will not enable interface monitoring for reth0 member interfaces ge-0/0/3 and ge-7/0/3 as in a virtual environment it does not make much sense. However, ip-monitoring would make a lot.
Since it’s 2:30AM, I will let this rest and add further notes if they hit my head.