SRX GRE with loopback endpoints over Ipsec tunnel does not pass traffic

This has been discovered on Junos 12.1X44-D40 on branch SRX 240 when a GRE tunnel with loopback endpoints is configured over an IPSEC tunnel (one reason for this layout will be explained in future post) AFTER REBOOT (there is a workaround, read on).

It is a Juniper supported design and according the Juniper KB [SRX] GRE over IPsec configuration example, it is straight forward.

Here is SRX 240 relevant configuration.

SRX interfaces configuration:

Code:

[edit]
user@srx240# show interfaces lo0    
unit 0 {
    family inet {
        address 172.16.10.30/32;
    }
}

[edit]
user@srx240# show interfaces gr-0/0/0    
unit 0 {
    tunnel {
        source 172.16.10.30;
        destination 172.16.10.1;
    }
    family inet;
    family inet6 {
        mtu 1300;
    }
}
[edit]
user@srx240# show interfaces st0 
unit 0 {
    family inet {
        mtu 1350;
    }
}

[edit]
user@srx240# show interfaces ge-0/0/14 
##
## protect: interfaces ge-0/0/14
##
unit 0 {
    family inet {
        address 100.100.100.100/24;
    }
}

SRX ike and ipsec configuration:

Code:

[edit]
user@srx240# show security ike 
##
## protect: security ike
##
proposal phase-1-prop {
    authentication-method pre-shared-keys;
    dh-group group2;
    authentication-algorithm sha1;
    encryption-algorithm aes-256-cbc;
}
policy phase-1-policy {
    mode main;
    proposals phase-1-prop;
    pre-shared-key ascii-text "<PSK>"; ## SECRET-DATA
}
gateway SRX-IKE-GW {
    ike-policy phase-1-policy;
    address 11.11.11.11;
    external-interface ge-0/0/14.0;
}
[edit]
user@srx240# show security ipsec 
##
## protect: security ipsec
##
proposal phase-2-proposal {
    protocol esp;
    authentication-algorithm hmac-sha1-96;
    encryption-algorithm aes-256-cbc;
}
policy phase-2-policy {
    proposals phase-2-proposal;
}
vpn VPN_PMF {
    bind-interface st0.0;
    ike {
        gateway SRX-IKE-GW;
        no-anti-replay;
        proxy-identity {
            local 100.100.100.100/32;
            remote 11.11.11.11/32;
            service any;
        }
        ipsec-policy phase-2-policy;
    }
    establish-tunnels immediately;
}

SRX security zones and security policies:

Code:

[edit]
user@srx240# show security zones 
security-zone vpn {
    host-inbound-traffic {
        system-services {
            all;
        }
        protocols {
            all;
        }
    }
    interfaces {
        ge-0/0/14.0;
        ge-0/0/15.0;
    }
}
security-zone VPN {
    interfaces {
        st0.0;
        gr-0/0/0.0;
    }
}
security-zone loopback {
    host-inbound-traffic {
        system-services {
            all;
        }
        protocols {
            all;
        }
    }
    interfaces {
        lo0.0;
    }
}
[edit]
user@srx240# show security policies 
##
## protect: security policies
##
default-policy {
    permit-all;
}

[edit]

Troubleshooting a SRX GRE with loopback endpoints over ipsectunnel not working scenario:

After device boots and establishes ipsec tunnels and st0/gr-0/0/0 interfaces come up and before jumping to the workaround, here are some troubleshooting steps.

Checking GRE session on the SRX flow session table:

Code:

user@srx240# run show security flow session protocol gre extensive 
Session ID: 75, Status: Normal
Flag: 0x10000
Policy name: N/A
Source NAT pool: Null
Dynamic application: junos:UNKNOWN, 
Maximum timeout: N/A, Current timeout: N/A
Session State: Valid
Start time: 96, Duration: 371
   In: 172.16.10.1/1 --> 172.16.10.30/1;gre, 
    Interface: lo0.0, 
    Session token: 0x8, Flag: 0x621
    Route: 0x0, Gateway: 172.16.10.1, Tunnel: 0
    Port sequence: 0, FIN sequence: 0, 
    FIN state: 0, 
    Pkts: 0, Bytes: 0
Total sessions: 1

Looks fine. The session is between local and remote loopback addresses.

If we enable flow traceoptions (not shown here exact config), we see the following events:

Code:

Feb 21 15:52:19 15:52:19.270869:CID-0:RT:<172.16.10.1/1->172.16.10.30/1;47> matched filter 2:
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:packet [80] ipid = 2373, @0x422d8648
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:---- flow_process_pkt: (thd 2): flow_ctxt type 1, common flag 0x0, mbuf 0x422d8400, rtbl_idx = 0
Feb 21 15:52:19 15:52:19.270869:CID-0:RT: in_ifp <VPN:st0.0>
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:flow_process_pkt_exception: setting rtt in lpak to 0x51982550
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:pkt out of tunnel.Proceed normally
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:  st0.0:172.16.10.1->172.16.10.30, 47
Feb 21 15:52:19 15:52:19.270869:CID-0:RT: find flow: table 0x49178598, hash 55133(0xffff), sa 172.16.10.1, da 172.16.10.30, sp 1, dp 1, proto 47, tok 7
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:  no session found, start first path. in_tunnel - 0x4c39c168, from_cp_flag - 0
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:  flow_first_create_session
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:flow_first_create_session: Found invalid sess. Start first path
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:  flow_first_in_dst_nat: in <st0.0>, out <N/A> dst_adr 172.16.10.30, sp 1, dp 1
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:  chose interface st0.0 as incoming nat if.
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:flow_first_rule_dst_xlate: DST no-xlate: 0.0.0.0(0) to 172.16.10.30(1)
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:flow_first_routing: vr_id 0, call flow_route_lookup(): src_ip 172.16.10.1, x_dst_ip 172.16.10.30, in ifp st0.0, out ifp N/A sp 1, dp 1, ip_proto 47, tos 0
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:Doing DESTINATION addr route-lookup
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:Changing out-ifp from .local..0 to lo0.0 for dst: 172.16.10.30 in vr_id:0
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:  routed (x_dst_ip 172.16.10.30) from VPN (st0.0 in 0) to lo0.0, Next-hop: 172.16.10.30
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:flow_first_policy_search: policy search from zone VPN-> zone loopback (0x0,0x10001,0x1)
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:Policy lkup: vsys 0 zone(7:VPN) -> zone(8:loopback) scope:0
Feb 21 15:52:19 15:52:19.270869:CID-0:RT:             172.16.10.1/1 -> 172.16.10.30/1 proto 47
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:Policy lkup: vsys 0 zone(5:Unknown) -> zone(5:Unknown) scope:0
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:             172.16.10.1/1 -> 172.16.10.30/1 proto 47
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  app 0, timeout 1800s, curr ageout 1800s
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  permitted by policy default-policy-00(2)
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  packet passed, Permitted by policy.
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:flow_first_src_xlate:  nat_src_xlated: False, nat_src_xlate_failed: False
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:flow_first_src_xlate: src nat returns status: 0, rule/pool id: 0/0, pst_nat: False.
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  dip id = 0/0, 172.16.10.1/1->172.16.10.1/1 protocol 0
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  choose interface lo0.0 as outgoing phy if
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:is_loop_pak: Found loop on ifp lo0.0, addr: 172.16.10.30, rtt_idx: 0 addr_type:0x2.
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:flow_first_loopback_check: Setting interface: lo0.0 as loop ifp.
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf : Alloc sess plugin info for session 2786
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:[JSF]Normal interest check. regd plugins 19, enabled impl mask 0x0
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id  2, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id  3, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id  5, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id  6, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id  7, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id  8, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 12, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 15, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:+++++++++++jsf_test_plugin_data_evh: 3
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 16, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 22, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 23, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 26, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 27, svc_req 0x0, impl mask 0x0. rc 2
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf int check: plugin id 28, svc_req 0x0, impl mask 0x0. rc 4
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:[JSF]Plugins(0x0, count 0) enabled for session = 1280122100, impli mask(0x0), post_nat cnt 2786 svc req(0x0)
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:-jsf : no plugin interested for session 2786, free sess plugin info
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:flow_first_service_lookup(): natp(0x4c4d1c58): app_id, 0(0).
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  service lookup identified service 0.
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  flow_first_final_check: in <st0.0>, out <lo0.0>
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:flow_first_final_check: flow_set_xlate_vector.
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:flow_first_complete_session, pak_ptr: 0x48ac5b38, nsp: 0x4c4d1c58, in_tunnel: 0x4c39c168
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:construct v4 vector for nsp2
Feb 21 15:52:19 15:52:19.271301:CID-0:RT:  existing vector list 0x3004-0x45a70a10.
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  Session (id:2786) created for first pak 3004
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  flow_first_install_session======> 0x4c4d1c58
Feb 21 15:52:19 15:52:19.271798:CID-0:RT: nsp 0x4c4d1c58, nsp2 0x4c4d1cd8
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:flow_xlate_pak
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  post addr xlation: 172.16.10.1->172.16.10.30.
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:check self-traffic on lo0.0, in_tunnel 0x4c39c168
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:retcode: 0x1
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:pak_for_self : proto 47, dst port 1, action 0x0
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  flow_first_create_session
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  flow_first_in_dst_nat: in <lo0.0>, out <N/A> dst_adr 172.16.10.30, sp 1, dp 1
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  chose interface st0.0 as incoming nat if.
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  packet dropped: for self but not interested
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  packet dropped, packet dropped: for self but not interested.
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:  flow find session returns error.
Feb 21 15:52:19 15:52:19.271798:CID-0:RT:flow_process_pkt_exception: Freeing lpak 0x48ac5b38 associated with mbuf 0x422d8400
Feb 21 15:52:19 15:52:19.271798:CID-0:RT: ----- flow_process_pkt rc 0x7 (fp rc 0)

These traceoptions show what is happening inside the srx flow process. According to the first part of the traces, GRE traffic (172.16.10.1/1->172.16.10.30/1;47) from remote tunnel endpoint is matched from zone VPN(incoming interface st0.0) to zone loopback(out int lo0.0) and passed by default policy.
The second part of the traces shows that the actual GRE traffic is dropped: "packet dropped: for self but not interested", "packet dropped, packet dropped: for self but not interested.", "flow find session returns error.".

Ok. So I have failed to find a permanent solution (Juniper is working on this and by the time this post is read, a permanent fix will be avilable), see Junos 12.1X44 release notes for SRX, however it seems to affect srx branch models also.

Workaround: Juniper SRX GRE between loopbacks over ipsec tunnel not working (until next reboot):

The actual workaround implies removing the gre interface and from security zone, comitting and rollback 1 configuration and committing again:

Code:

[edit]
user@srx240# delete interfaces gr-0/0/0 

[edit]
user@srx240# delete security zones security-zone VPN interfaces gr-0/0/0.0 

[edit]
user@srx240# commit 
commit complete

[edit]
user@srx240# rollback 1 
load complete

[edit]
user@srx240# commit 
commit complete

After the second commit, the SRX box accepts traffic from the gre tunnel.

Update: This is fixed in 12.1X46-D35 and Junos releases past Feb 2015.

Linux, FreeBSD, Juniper, Cisco / Network security articles and troubleshooting guides