Sunday, October 25, 2015
At the All Things Open conference last year, one of the speakers talked of "dorm room ready software". It's wonderful that this plethora of software has become available, increasing the speed of prototyping. However, the only reason this is still possible is because those dorm rooms still have global IP addresses. The computer sitting on the desk can still serve content to the world. As we run out of IPv4 addresses, more and more of us will be closed off behind NATs beyond our control. Poor college students can ill afford hosting services that still have those scarce IPv4 addresses. This creates another barrier to innovation unless we do something.
Tuesday, September 1, 2015
At work today, I helped someone use
ping6 to figure out what to fix on a server that wasn't reachable over IPv6. I have a pretty basic formula that I thought I would share.
I will use this information for the problem host:
nethope@fixme$ ip -6 address show 1: lo:
mtu 65536 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qlen 1000 inet6 2001:db8:2015:3000::5/64 scope global valid_lft forever preferred_lft forever inet6 fe80::201:2ff:fe03:405/64 scope link valid_lft forever preferred_lft forever nethope@fixme$
When I start troubleshooting, I like to start small and expand to ever-wider scopes. One of my college professors said troubleshooting should start with a simple question to which you know the answer, and take small steps from there.
The smallest scope possible is loopback. Since I usually have multiple network interfaces,
ping6 needs to know which one to use. The syntax will vary between operating systems. I used '
ping6 -c3 -I lo ::1' on my Linux, and '
ping6 -c3 ::1%lo0' on my Mac.
ping6 -c3 -I lo ::1
That should work; it should be the question to which you know the answer. If it doesn't work, IPv6 probably isn't configured yet. This command should also work without specifying the interface; if you need to specify the loopback interface then route selection is broken.
Then I expand the scope ever so slightly, all the way up to another easy one, just so I don’t miss some strange condition. Try the link-local address (HINT: it starts with
fe80) for that interface. This traffic also shouldn't hit the network, just this computer and its network interface card.
ping6 -c3 -I eth0 fe80::201:2eff:fe03:405
If that fails, first bounce the interface (
ifdown; ifup if that's an acceptable outage), and then remove and reconfigure IPv6 for that interface. Again, this task is specific for your operating system. Reconfiguring the interface was the solution today!
The next small step: Try the global address for that interface (still on the same machine). Since it’s a global address, you shouldn’t have to specify the interface, but if you get unexpected results, drop it back in to be sure.
ping6 -c3 -I eth0 2001:db8:2015:3000::5
This should work if the previous two steps worked, but if it doesn't, try the same repair procedure as the previous step.
Now I’m finally ready to move beyond the scope of the NIC, all the way out to the router interface, still within the same VLAN. Again, you shouldn't have to specify which interface to use now, but we're troubleshooting so watch out for oddities. If you know it, start with the router's link-local address.
ping6 -c3 -I eth0 fe80::a8bb:ccff:fedd:eeff
ping6 -c3 -I eth0 2001:db8:2015:3000::1
If that fails, look for a problem within the VLAN. Find other IPv6-enabled hosts in the same VLAN (all of them, right? *grin*), and see if they can reach the default router. If no hosts can reach the router, the router needs to be fixed. (I've had to remove then reconfigure IPv6 on a VLAN on a Cisco router, and magically the very same configuration started working again. It hasn't happened in ages, thankfully.) If most hosts can reach the router, look for reasons why this one has problems. Perhaps it is missing IPv6 router advertisements, and can’t determine its IPv6 default router or thinks it can’t reach it. That would be a problem with ICMPv6 or multicast. Since link-local multicast is generally reliable (multicast querier on the router aside), it's probably the host-based firewall.
Next, if available, look outside your VLAN, but stay on your network. Pick anything.
ping6 -c3 -I eth0 2001:db8:2015:3001::101
If that works, your network is probably fine. If it fails, make sure the router has IPv6 routes. Are all VLANs affected, or just some VLANs? I usually test other VLANs at this step, too.
ping6 -c3 -I eth0 www.yoursite.edu
If you want to be sure that DNS isn't the problem, either don't use hostnames, or validate your hostnames with
dig first. However, we're entering the territory where I always use hostnames because I don't remember the numbers.
If the problem shows up at this scope, poke around. Pick two hosts in two VLANs, and try
traceroute6 in both directions (one to the other and other to the one; I detected an OSPFv3 problem that way). Our F5 load balancers make traceroute look odd, so pay more attention to the simple "good, yes, connected" versus "bad, no, couldn't connect" results at first. Try
mtr (a new enough version to support IPv6) and let it run for a while to look for path instability or intermittent packet loss (one weekend when an intermediate router was losing 60% of IPv6 packets).
And then try hosts that aren't on your network, like:
At what step in this process does
ping6 fail? There’s the scope of your problem, and where to concentrate your analysis.