TDR cable testing with a Catalyst 3750E

February 22nd, 2012 Comments off

A very useful tool for determining simple wiring problems (Layer 1) without having to leave your desk is the Time Domain Reflector testing built in to some of the Catalyst switches including the 3750E.

One caveat is that it may cause connectivity problems for the particular tested interface.

Start the test by issuing the command:

Switch#test cable-diagnostics tdr interface gi2/0/2

After a few seconds the test results will be available by issuing this command:

Switch#show cable-diagnostics tdr interface gi2/0/2
TDR test last run on: February 22 10:00:46

Interface Speed Local pair Pair length        Remote pair Pair status
--------- ----- ---------- ------------------ ----------- --------------------
Gi2/0/2   auto  Pair A     N/A                Pair B      Normal              
                Pair B     N/A                Pair A      Open                
                Pair C     64   +/- 1  meters N/A         Normal              
                Pair D     65   +/- 1  meters N/A         Normal          
Categories: Cisco, Networking Tags: , , ,

Suppress %SNMP-3-AUTHFAIL Logging

March 23rd, 2011 Comments off

For a while now, security scanning software and users have been triggering an onslaught of %SNMP-3-AUTHFAIL messages in our device logs. This rather annoying message often obscures other events that may be more important. Old mentality would tell you to simply create a logging discriminator and be done with it. However, I recently learned of a little undocumented gem in IOS that saves the day:

hostname(config)#no logging snmp-authfail

Do not trust the almighty ‘?‘ command. It will not show up as a valid option. However, when actually executing the command, it has worked in every IOS case I have tried.

Categories: Cisco, Networking Tags: , , ,

Creating a lossless Ethernet network with Cisco NX-OS

January 14th, 2011 Comments off

For the purpose of this article, this configuration and discussion pertains to a Cisco Nexus 5020 switch running NX-OS version 4.1(3)N1(1a). Given what seems to be a variety of changes to the CLI configuration process for the Nexus, this may or may not work for you.

I felt it was important to write myself a reminder of how to create a lossless Ethernet network using Cisco Nexus devices as QoS on this platform seems to be similar, yet, take a divergent path from the methodology of configuring QoS on the Catalyst series. In this example we will assume we are working users that are requesting that any traffic marked with CoS 5 be placed in the no-drop queue. The flagship example of this is the Fiber Channel over Ethernet (FCoE) configuration present in the system configuration, although they use CoS 3, by default.

The first step is to create a qos class-map:

class-map type qos MATCH-COS-FIVE
  match cos 5

Next create a qos policy-map to set the qos-group to 2:

policy-map type qos SET-QOS-GROUP-TWO
    set qos-group 2

qos-group 1 is a system default for the FCoE configuration. Setting this to our own group allows us to have different parameters for FCoE should you need to manipulate it differently in the future.

Next create a network-qos class-map to match qos-group 2:

class-map type network-qos MATCH-QOS-GROUP-TWO
  match qos-group 2

Now, create a policy-map you wish to use to manage QoS on a system wide level and ensure traffic matching qos-group 2 is in the no-drop queue.

policy-map type network-qos SYSTEM-QOS-POLICY
  class type network-qos MATCH-QOS-GROUP-TWO
    pause no-drop
  class type network-qos class-default
    mtu 9216

In this case the class-default matches other traffic and permits the MTU to be jumbo framed at 9216 bytes.

Apply the system policy we created to system qos:

system qos
  service-policy type network-qos SYSTEM-QOS-POLICY

We are now at the point where matching CoS values can be done on a system-wide level or per interface. To apply the policy system-wide:

system qos
  service-policy type qos SET-QOS-GROUP-TWO

Be aware that if you are running within a VPC domain that this may very well cause a system compatibility failure and disable links within your VPC domain. Apply this configuration domain-wide if you are interested in have this feature across multiple switches.

Otherwise you can apply the CoS matching on a per interface level:

interface Ethernet1/1
  service-policy type qos input SET-QOS-GROUP-TWO

To verify packets are matching your newly defined qos-group:

switch# show queuing interface Ethernet 1/10

If traffic is being sent that is tagged with CoS 5 you should be seeing traffic on your qos-group:

qos-group  2:
    q-size: 81920, MTU: 9216
    drop-type: no-drop, xon: 128, xoff: 230
        Pkts received over the port             : 2767031
        Ucast pkts sent to the cross-bar        : 2767030
        Mcast pkts sent to the cross-bar        : 1
        Ucast pkts received from the cross-bar  : 270664
        Pkts sent to the port                   : 270664
        Pkts discarded on ingress               : 0
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

All should be well and drop-type should be reading no-drop. If you are interested in applying bandwidth shaping, then this can be accomplished in a similar manner using the queuing type class and policy maps to map and take action with traffic in qos-group 2. That service policy can also be applied system-wide or on an interface level as shown above.

Categories: Cisco, Networking Tags: , , ,

Stateful Cisco IOS Firewalls

April 5th, 2010 Comments off

Cisco IOS Firewall is capable of stateful packet filtering when it is configured to do so.

By applying inspection rules to an interface, you can configure the router to drop packets that could be potentially harmful given the state of a currently established session. For example, using a TCP inspection rule will not allow a TCP reset (RST) packet through the interface unless there is currently a TCP session established with the machine sending the reset.

To apply stateful packet filtering rules, you must apply an access control list (ACL) to the interface you wish to inspect. Packets traversing this interface will then be examined against the ACL first, then checked against the inspection rule(s) to determine whether the packet should be forwarded or dropped. Note that any packet may be rejected by the inspection rules, ACL, or both!

First, decide which interface you would like to do stateful inspection on. This is typically your WAN or untrusted network interface. Configure an IP ACL with this interface in mind. If you are configuring a firewall for an interface that uses DHCP, keep in mind that the destination IP for packets allowed in will be the IP address assigned to that interface. Therefore I generally use any and remember it is there for that purpose. This alleviates the issue of having to adjust ACLs every time your router receives a new IP address from your ISP. Also keep in mind that you must use an extended ACL when you are also using inspection rules:

Router(config)#ip access-list extended Internet-inbound-ACL
Router(config-ext-nacl)#10 permit tcp any any eq 22
Router(config-ext-nacl)#20 deny ip any any log

The above permits traffic on port 22 (SSH) to traverse the interface. The deny statement is added for clarity and logging.

Next define your inspection rules. IOS supports more than 170 protocols:
[no] ip inspect name inspection-name protocol [alert {on | off}] [timeout seconds]

Router(config)#ip inspect name FIREWALL tcp
Router(config)#ip inspect name FIREWALL udp

Finally apply the ACL and the inspection rules to the interface:

Router(config)#interface FastEthernet4
Router(config-if)#ip access-group Internet-inbound-ACL in
Router(config-if)#ip inspect FIREWALL out

It is important that the inspect statement is applied in the outbound direction. I found numerous documentation that stated it needed to be applied in the inbound direction, however I have found that this does not work and any new connection attempts are not stored in the session table or permitted through the interface!

You can verify the configuration that was just applied by issuing a show ip inspect session command. The output should look something like this:

Router#sh ip inspect sessions 
Established Sessions
 Session 83C60000 (>(76.73.X.X:22) ssh SIS_OPEN
 Session 83C65828 (>(74.X.X.X:5222) tcp SIS_OPEN
 Session 83C5AAB0 (>(208.43.X.X:80) tcp SIS_OPEN
 Session 83C685A8 (>(174.X.X.X:443) tcp SIS_OPEN
 Session 83C66EE8 (>(74.X.X.X:5222) tcp SIS_OPEN
 Session 83C63BB8 (>(208.89.X.X:80) tcp SIS_OPEN
Categories: Cisco, Networking Tags: , , ,

IPv6 addrconf: prefix with wrong length 48

February 15th, 2010 Comments off

If your hosting provider gives out an entire /48 for every hosted server, your syslog may get overwhelmed with messages concerning the subnet mask:

1 Time(s): [705959.619704] IPv6 addrconf: prefix with wrong length 48

To solve this temporarily you can disable auto-configuration and router advertisements:

echo 0 > /proc/sys/net/ipv6/conf/eth0/accept_ra
echo 0 > /proc/sys/net/ipv6/conf/eth0/autoconf

This seems to suppress the annoying log messages on my Ubuntu based environment. If you would like to make this persistent through reboots, add the following lines to /etc/sysctl.conf:

net.ipv6.conf.all.accept_ra = 0
net.ipv6.conf.all.autoconf = 0
Categories: IPv6, Linux, Networking Tags: , , ,

Enabling IPv6 on Cisco Catalyst 3750 Devices

February 9th, 2010 Comments off

I was recently baffled to see that when I entered configuration mode within a VLAN interface on a Catalyst access layer switch that I could not set any IPv6 options! It turns out that by default, SDM prefers what it calls the “desktop default” template. This means it is optimized for IPv4 and does not include IPv6 support. Fortunately a quick but painful fix to this is to change the prefered SDM template from “desktop default” to “dual-ipv4-and-ipv6″:

Switch#config t
Switch(config)#sdm prefer dual-ipv4-and-ipv6

Unfortunately, you will need to reload (reboot) the device in order for the changes to take place, which will obviously incur annoying downtime for your users. Once the device has reloaded you can verify by issuing a show sdm prefer command which should look something like this:

Switch#show sdm  prefer
 The current template is "desktop IPv4 and IPv6 default" template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1024 VLANs.

  number of unicast mac addresses:                  2K
  number of IPv4 IGMP groups + multicast routes:    1K
  number of IPv4 unicast routes:                    3K
    number of directly-connected IPv4 hosts:        2K
    number of indirect IPv4 routes:                 1K
  number of IPv6 multicast groups:                  1.125k
  number of directly-connected IPv6 addresses:      2K
  number of indirect IPv6 unicast routes:           1K
  number of IPv4 policy based routing aces:         0
  number of IPv4/MAC qos aces:                      0.5K
  number of IPv4/MAC security aces:                 1K
  number of IPv6 policy based routing aces:         0
  number of IPv6 qos aces:                          0.5K
  number of IPv6 security aces:                     0.5K

Now you should be able to configure IPv6 interfaces and ACLs.

Dual NAT Load-Balancing with Cisco

January 9th, 2010 Comments off

Load balancing through two NAT connections is now partially possible with the use of some tricky configurations within Cisco IOS running on integrated services routers. I recently was tasked with configuring this scenario and found that Cisco has documented a way of doing this.

Although their guide describes this process in enough detail to fit a lot of different situations, I felt it would be useful to revisit the configuration process in hopes of helping other people facing similar requirements and for my own documentation purposes.

Everything in this article was used specifically with a Cisco 1811 router running 12.4(24)T2 Advanced IP Services software, however the majority of this guide should work with any recent integrated service router with advanced IP services.


This example describes a router that has one DHCP-configured interface to one ISP and one static address to another ISP, each on different WAN interfaces of this router. You should configure policy-based routing will be configured for specific or secure traffic (such as HTTPS, SSH, etc) to ensure that it always uses one ISP connection. Fail-over redundancy will also be configured in the event one of these connections become unavailable. This configuration also assumes the interfaces and NAT has already been configured for each connection.


Specify the desired object tracking timer for interfaces in seconds. I used 5 as did the Cisco documentation:

track timer interface 5

Next create new tracked SLA objects – one for each of the ISP connections, and add desired delay change notifications in seconds:

track 100 ip sla 1 reachability
 delay down 15 up 10
track 200 ip sla 2 reachability
 delay down 15 up 10

For an interface running DHCP, route tracking can be enabled with the following interface configuration command:

ip dhcp client route track 100

*Where 100 is the desired tracking object number created above

Modify the static IP default route entry for the static IP connection to be tracked:

ip route interface FastEthernet1 [next hop] 254 track 200

The digits in orange signify the metric for this static route. Since the DHCP learned route has a metric of 254, I set this static default route with the same metric so that equal-cost load balancing could be achieved. The red digits should match the desired object track number we created earlier.

Next configure an OER tracking entry to monitor each ISP connection. The icmp-echo line can be a difficult choice. It is important to identify an address to ping that is reachable only through that connection. With the addition of the source-interface line referencing an outside interface with a global address, I do not believe this to be an issue in the majority of topologies as the echo reply would never make it back the device, however I have not had sufficient time to experiment with this case to see how this works in reality.

ip sla 1
 icmp-echo source-interface FastEthernet0
 timeout 1000
 threshold 40
 frequency 3
ip sla 2
 icmp-echo source-interface FastEthernet1
 timeout 1000
 threshold 40
 frequency 3

Schedule the new SLAs for start time and duration:

ip sla schedule 1 life forever start-time now
ip sla schedule 2 life forever start-time now

Fail-over redundancy

Tacked on to the route-maps configured to allow NAT, I added an additional permit sequence that set the next hop based on availability:

route-map NET1 permit 10
  match ip address NETWORK1
  set ip next-hop verify-availability [primary route] 10 track 100
  set ip next-hop verify-availability [secondary route] 20 track 200
route-map NET2 permit 10
  match ip address NETWORK2
  set ip next-hop verify-availability [primary route] 10 track 200
  set ip next-hop verify-availability [secondary route] 20 track 100

This will set specify which router should be primary for each network. In this case I set each network’s primary route to a different connection in an effort to spread the load but still ensure both networks work in the event either ISP connection were to fail.


So now that load-balancing has somewhat been configured on the device, it is time to talk about the feasibility. The one point to take away from this is that it is not true load-balancing even if there are two equal default routes to choose. This is because the nature of NAT will force the session on to one of the two connections.  My suggestion is to use this but limit the subnets to use one primarily and the other for backup, alternating the networks between the two WAN connections. In the end, this is what I did and has been a wonderful solution overall.

Categories: Cisco, Networking Tags: , , ,

How to create 6in4 tunnels in Arch Linux

January 25th, 2009 Comments off

Arch Linux uses a BSD-style network interface configuration located in /etc/rc.conf, which can make exotic network configurations interesting, to say the least. The following rc.d script is meant to help remedy the situation by providing a configurable wrapper to sanely manage a 6in4 link interface.

  • This script uses the route2 method; make sure that the iproute2 package is installed.

As root, write the following rc.d init script to /etc/rc.d/6in4-tunnel:


### begin user configuration

#                                            #
#  Stop this script before you reconfigure!  #
#                                            #

# if_name     - interface name that is to be created for the 6in4 link

# server_ipv4 - ipv4 address of the server that is providing the 6in4 tunnel

# client_ipv4 - ipv4 address of the client that is receiving the 6in4 tunnel

# client_ipv6 - ipv6 address of the client 6in4 tunnel endpoint

# link_mtu    - set the mtu for the 6in4 link

# tunnel_ttl  - set the ttl for the 6in4 tunnel

### end user configuration


. /etc/rc.conf
. /etc/rc.d/functions

case "$1" in
    stat_busy "Starting $daemon_name daemon"

    ifconfig $if_name &>/dev/null
    if [ $? -eq 0 ]; then
      stat_busy "Interface $if_name already exists"
      exit 1

    ip tunnel add $if_name mode sit remote $server_ipv4 local $client_ipv4 ttl $tunnel_ttl
    ip link set $if_name up mtu $link_mtu
    ip addr add $client_ipv6 dev $if_name
    ip route add ::/0 dev $if_name

    add_daemon $daemon_name

    stat_busy "Stopping $daemon_name daemon"

    ifconfig $if_name &>/dev/null
    if [ $? -ne 0 ]; then
      stat_busy "Interface $if_name does not exist"
      exit 1

    ip link set $if_name down
    ip tunnel del $if_name

    rm_daemon $daemon_name

    echo "usage: $0 {start|stop}"
exit 0

You will need to provide your 6in4 link configuration between the following sections of /etc/rc.d/6in4-tunnel:

### begin user configuration
### end user configuration

Once /etc/rc.d/6in4-tunnel has been configured properly, give it permission to be executed:

# chmod +x /etc/rc.d/6in4-tunnel

To create the 6in4 tunnel link and bring up the interface:

# /etc/rc.d/6in4-tunnel start

To delete the 6in4 tunnel link and remove the interface:

# /etc/rc.d/6in4-tunnel stop

The following method allows /etc/rc.d/6in4-tunnel to start automatically at system startup.

  • Verify that the 6in4 tunnel link is configured and working properly before doing this!

As root, insert 6in4-tunnel right after network in the DAEMONS line of /etc/rc.conf.

After this addition, the DAEMONS line in /etc/rc.conf should look something like this:


# -----------------------------------------------------------------------
# -----------------------------------------------------------------------
# Daemons to start at boot-up (in this order)
#   - prefix a daemon with a ! to disable it
#   - prefix a daemon with a @ to start it up in the background
DAEMONS=(syslog-ng iptables ip6tables network 6in4-tunnel openntpd ...

Categories: IPv6, Linux, Networking Tags: , , ,

How to create IPv4 GRE tunnels in Ubuntu

January 19th, 2009 1 comment

Here is an easy way to make use of IPv4 GRE tunnels in Ubuntu, or any other Debian based distro. You will need to edit /etc/network/interfaces.

Here is a template with the information you will need to add:

auto tun1
iface tun1 inet static
    address <tunnel IP>
    netmask <tunnel subnet mask>
    pre-up iptunnel add tun1 mode gre local <local IP> remote <remote IP> ttl 255
    up ifconfig tun1 multicast
    pointopoint <remote tunnel IP>
    post-down iptunnel del tun1

As with the previous post about 6in4 tunnels in Ubuntu, lets take a brief look at each line.

auto tun1 is used by the /etc/init.d/networking script. Just like the 6in4 tunnel, the auto parameter will instruct the script to automatically start or stop the interface. The script will get called during startup and will bring up this interface automatically. This line is entirely optional and depends on your personal preference.

iface tun1 inet static starts the configuration block for a new IPv4 interface. This is the interface of the tunnel we are about to create that will encapsulate traffic destined for the other side of the tunnel.

address <tunnel IP> is the IP address you wish to assign to this machine’s side of the GRE tunnel.

netmask <tunnel subnet mask> is the subnet mask of the tunnel. I highly suggest using a subnet mask as this tunnel will be point-to-point and there is no reason to waste address space, even if it is private addressing. If you are unfamiliar with how subnet masks are used, please refer to Subnetwork on Wikipedia.

pre-up iptunnel add tun1 mode gre local <local IP> remote <remote IP> ttl 255 – The pre-up parameter tells the init script to run the iptunnel command prior to bringing up the interface. This is where we are actually creating the tunnel and telling it to use GRE mode. <local IP> is this machine’s IP address of an interface on which you want to run this tunnel. For example, if eth0 had the address of, you could set the local IP address above to that address in order to have this tunnel use eth0. <remote IP> is the global IP address where the other side of the tunnel exists.

up ifconfig tun1 multicast – The up parameter tells the init script to run ifconfig tun1 multicast once the interface is up. In this case, we are enabling multicast on this interface. This is particularly useful if you wish to run a routing protocol over this tunnel, such as OSPF.

pointopoint <remote tunnel IP> – The remote IP here is the IP address of the other side of the GRE tunnel. For example, if your side of the tunnel is and their side is, then would be the IP address to specify here.

post-down iptunnel del tun1 – Just like pre-up, post-down tells the init script to run this command after the interface has been shutdown. In this case, we are deleting the tunnel we created earlier with the pre-up command.

As with the with the 6in4 tunnel, you can name this tunnel interface something that is more meaningful than tun1.

How to create 6in4 tunnels in Ubuntu

January 18th, 2009 Comments off

The simplest way to create a static 6in4 tunnel in Ubuntu, or any other Debian based distro, is to edit /etc/network/interfaces.

Here is a template with the information you will need to add:

auto tun1
iface tun1 inet6 v4tunnel
        address <your IPv6 address>
        netmask 64
        ttl 64
        endpoint <remote IPv4 tunnel address>
        up ip link set mtu 1280 dev tun1
        up ip route add 2000::/3 dev tun1

Lets take a brief look at what each line does.

auto tun1 is used by the /etc/init.d/networking script. The auto parameter will instruct the script to automatically start or stop the interface. The script will get called during startup and will bring up this interface automatically. This line is entirely optional and depends on your personal preference.

iface tun1 inet6 v4tunnel starts the configuration block for a 6in4 tunnel. That is, IPv6 traffic encapsulated in IPv4 packets. This is extremely similar to how GRE works with purely IPv4 traffic.

address <your IPv6 address> is where you need to specify the IPv6 address assigned to your machine. This is typically ends in ::2.

netmask 64 specifies the subnet mask for the IPv6 address you entered above. 64 is the smallest recommended subnet (see RFC3627 for why they no longer use numbers such as 127). 64 is what you should normally expect to use.

ttl 64 specifies the Time to Live value set for packets sent by your tunnel endpoint. This only affects the IPv4 packet that is used to encapsulate your v6 traffic. It does not change the original IPv6 packet. Time to Live is a number of iterations a packet can live through before it should be discarded. This number is reduced by one on every router it passes enroute to its destination. 64 is used here because it restricts the packet to roughly the same region.

Here is a quick break down of default TTL values:

  • 0 – restricted to same host
  • 1 – restricted to same subnet
  • 32 – restricted to same site
  • 64 – restricted to same region
  • 128 – restricted to same continent
  • 255 – unrestricted

endpoint <remote IPv4 tunnel address> is the IPv4 address to send encapsulated IPv6 traffic to. Your IPv6 provider will provide you with their IPv4 tunnel endpoint address.

up ip link set mtu 1280 dev tun1. This statement reconfigures the interface from the default MTU to 1280 bytes. This is desirable to prevent fragmentation because of the IPv6 packet being encapsulated in an IPv4 packet.
NOTE: SixXS seems to use 1280 as their default for tunnels, other providers likely use 1480 (which is default). If you leave this line out of your configuration, it should default to 1480.

up ip route add 2000::/3 dev tun1. This last statement adds a default route for all IPv6 traffic to be sent through device tun1 which in turn would be encapsulated in an IPv4 packet and sent to the endpoint address of your IPv6 provider which would send it on its way through the IPv6 network.

After you have saved the file, the next step is to up the interface using ifup tun1 or if you decided to put in the auto tun1 line you can restart your networking services using the init script: /etc/init.d/networking restart.

One final note. You can name this tunnel interface anything you want. For example, every time tun1 was used above it could have been replaced with something more meaningful. For example, in SIXXS documentation, they usually call the interface sixxs.

Categories: IPv6, Linux Tags: , , , ,