Main Nav

NETMAN members,

 

In June of 2011, LSU encountered a peculiar issue with IPv6 while participating in World IPv6 Day.  To explain the cause of the issue, a brief overview of port configuration is necessary.

 

Every switchport on the LSU campus is set up with an untagged VLAN for data and a voice VLAN command.  This setup allows for the rapid deployment of VOIP phones and is also critical in the event of a disaster scenario.  This network implementation will also allow disaster personnel to attach VOIP phones without the aid of a network team to configure ports on-the-fly. With this configuration, Windows clients will receive an address from the untagged data VLAN and then later an address from the tagged VOIP VLAN.

 

Initially, a device connected to these ports will receive an ipv6 address from the untagged vlan. After a short, random amount of time some devices will obtain a second ipv6 address that corresponds to the tagged vlan. This behavior will cause ipv6 network connectivity to cease within the routed domain but works between routing domains.  This scenario only presents itself on routers that have the same MAC address (and thus same link-local address) on each switch virtual interface (SVI).  If the link-local addresses are different on each SVI this will not cause an issue, as the client will receive multiple link-local addresses for its default gateway and be able to route properly.

 

 

The issue described above was only encountered on Windows clients.  This behavior is a result of how VLAN tagging is handled according to Microsoft’s NDIS.  If VLAN tags are not stripped then SVI’s having the identical MAC addresses will not be an issue as shown with MAC and Linux operating systems.  It is important to note, however, that this issue only occurs with IPv6.  Windows seems to treat IPv6/IPv4 differently when it concerns VLAN tagging.

 

It is important for Microsoft to take a second look at how the NDIS is written and how they treat VLAN tagging.  LSU has already contacted Microsoft concerning the issue, but was informed that a design change request will only be entertained if there is enough business justification and interest backing the change.  Therefore, being that this group is part of a handful that has operational experience with ipv6, we encourage you to explore this issue. Then, contact your Microsoft representative to help create a backing for this request.

 

We do have several workarounds in place, which we will gladly share upon request. Thank you for your consideration.

 

 

----

 

Michael Fazely

Network Analyst II

Network Design

University Networking & Infrastructure

Louisiana State University

Baton Rouge, LA 70803

Office: (225)578-1971

Email: mfazel1@lsu.edu

 

 

Jeffry J.Handal

Convergence Specialist, M.S.E.E., P.E.

University Networking and Infrastructure - LSU

200 Computing Services Building

Baton Rouge, LA 70803

Office:  (225)578-1966

Fax:        (225)578-6400

 

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Comments

Message from aquerubin@hpu.edu

What kind of routers are you running that use the same MAC for each SVI?

 

Antonio Querubin

Network Engineer, Network Services

Information Technology Services

Hawai'i Pacific University

1164 Bishop Street, Suite #900

Honolulu. HI, 96813

p 808-356-5282 | f 808-544-1404

aquerubin@hpu.edu

 

Information Technology Services staff will never ask for your password.
Never email your password to anyone or share confidential information in emails.

 

Antonio,

 

They aren’t true routers, but are instead L2/L3 switches in most cases.  We have experienced the problem on Cisco Catalyst 4500’s and 6500’s.  It is important to note that Cisco Catalyst 3560’s and 3750’s have different MAC’s for each SVI and don’t experience the issue initially presented.  Thanks.

 

 

----

 

Michael Fazely

Network Analyst II

Network Design

University Networking & Infrastructure

Louisiana State University

Baton Rouge, LA 70803

Office: (225)578-1971

Email: mfazel1@lsu.edu

 

 

 

Message from aquerubin@hpu.edu

As a workaround couldn’t you set the mac address explicitly rather accept the default behavior?

 

Antonio Querubin

Network Engineer, Network Services

Information Technology Services

Hawai'i Pacific University

1164 Bishop Street, Suite #900

Honolulu. HI, 96813

p 808-356-5282 | f 808-544-1404

aquerubin@hpu.edu

 

Information Technology Services staff will never ask for your password.
Never email your password to anyone or share confidential information in emails.

 

Antonio,

 

Absolutely.  One of the workarounds that we have is to manually set a link-local address on each SVI.  This will keep the SVIs that have the same MAC address from using EUI-64 and handing out identical link-locals to the clients.  But, as I noted in my initial email to the group these are just fixes to the overall problem.  The goal is to encourage Microsoft to revisit how they treat VLAN tagging in IPv6.  Thanks.

 

 

----

 

Michael Fazely

Network Analyst II

Network Design

University Networking & Infrastructure

Louisiana State University

Baton Rouge, LA 70803

Office: (225)578-1971

Email: mfazel1@lsu.edu

 

 

 

Message from jeroen@zijndomein.nl

Hi Michael, > Every switchport on the LSU campus is set up with an untagged VLAN for > data and a voice VLAN command. This setup allows for the rapid > deployment of VOIP phones and is also critical in the event of a > disaster scenario. > > Initially, a device connected to these ports will receive an ipv6 > address from the untagged vlan. After a short, random amount of time > some devices will obtain a second ipv6 address that corresponds to the > tagged vlan. This behavior will cause ipv6 network connectivity to > cease within the routed domain but works between routing domains. Just to be sure: the voice VLAN is also IPv6 enabled and your routers are generating RA's? > This scenario only presents itself on routers that have the same MAC > address (and thus same link-local address) on each switch virtual > interface (SVI). If the link-local addresses are different on each > SVI this will not cause an issue, as the client will receive multiple > link-local addresses for its default gateway and be able to route > properly. Hmm, that's strange... link-local is link-local, it's only significant when combined with a specific link (interface). It's perfectly valid to have the same link-local address on more than one link. A route with a link-local address as a next hop should (afaik) always explicitly point to the 'nexthop%interface' and not just to the 'nexthop' value. I like to think of this like the primary key in a database: the combination of "link local addr" and "interface" is uniqe and is the correct way to refer to a specific entry. > The issue described above was only encountered on Windows clients. > This behavior is a result of how VLAN tagging is handled according to > Microsoft’s NDIS. If VLAN tags are not stripped then SVI’s having the > identical MAC addresses will not be an issue as shown with MAC and > Linux operating systems. It is important to note, however, that this > issue only occurs with IPv6. Windows seems to treat IPv6/IPv4 > differently when it concerns VLAN tagging. Well, it shouldn't :) Whether it's IPv4, IPv6 or IPX or whatever, the OS shouldn't care what runs over the L2 link. And a physical link with vlans should be treated as N virtual L2 links: each VLAN is a separate L2 broadcast domain ie a separate virtual L2 link. > It is important for Microsoft to take a second look at how the NDIS is > written and how they treat VLAN tagging. LSU has already contacted > Microsoft concerning the issue, but was informed that a design change > request will only be entertained if there is enough business > justification and interest backing the change. Therefore, being that > this group is part of a handful that has operational experience with > ipv6, we encourage you to explore this issue. Then, contact your > Microsoft representative to help create a backing for this request. We haven't run into this issue since our voice VLANs aren't IPv6 enabled yet; I'm afraid I can't offer much backing towards MS either. Still: this sounds very much like an implementation fault, going against the RFCs, which would constitute a bug. Still, I'm wondering about one little thing: why do your Windows machines see your voice VLAN at all? Are they explicitly VLAN-enabled to run a softphone application in the voice VLAN? Our VOIP deployment is still quite limited; currently our hardware VOIP phones (both regular SIP and MS Lync compatible gear) get a tagged voice VLAN, but if people decide they prefer to run a software solution, it just uses the regular (untagged) VLAN. I guess that, combined with the fact that our voice VLANs aren't IPv6 enabled yet (all data vlans are), is the reason why we haven't run in to this. Regards, Jeroen van Ingen ICT Service Centre University of Twente, P.O.Box 217, 7500 AE Enschede, The Netherlands ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Jeroen, > Just to be sure: the voice VLAN is also IPv6 enabled and your routers are generating RA's? Yes, the voice VLAN is IPv6 enabled and the routers are generating RA's. > Hmm, that's strange... link-local is link-local, it's only significant when combined with a specific link (interface). > It's perfectly valid to have the same link-local address on more than one link. A route with a link-local address > as a next hop should (afaik) always explicitly point to the 'nexthop%interface' and not just to the 'nexthop' value. > I like to think of this like the primary key in a database: the combination of "link local addr" and "interface" is uniqe > and is the correct way to refer to a specific entry. I agree with the statement "It's perfectly valid to have the same link-local address on more than one link", but only when VLAN tags are available. The issue here is that there are no VLAN tags associated with each address. > Well, it shouldn't :) Whether it's IPv4, IPv6 or IPX or whatever, the OS shouldn't care what runs over the L2 link. > And a physical link with vlans should be treated as N virtual L2 links: each VLAN is a separate > L2 broadcast domain ie a separate virtual L2 link. My assertion isn't that the OS itself should care about the VLANs, but rather that Microsoft's NDIS, which NIC manufacturers use to write drivers instructs them to handle VLAN tagging improperly for IPv6. I apologize if this wasn't communicated properly in my initial email. > We haven't run into this issue since our voice VLANs aren't IPv6 enabled yet; I'm afraid I can't offer much backing > towards MS either. Still: this sounds very much like an implementation fault, going against the RFCs, which would constitute a bug. > Still, I'm wondering about one little thing: why do your Windows machines see your voice VLAN at all? > Are they explicitly VLAN-enabled to run a softphone application in the voice VLAN? > Our VOIP deployment is still quite limited; currently our hardware VOIP phones (both regular SIP and MS Lync compatible gear) > get a tagged voice VLAN, but if people decide they prefer to run a software solution, it just uses the regular (untagged) VLAN. > I guess that, combined with the fact that our voice VLANs aren't IPv6 enabled yet (all data vlans are), is the reason > why we haven't run in to this. I understand that many people either aren't running IPv6 on any level or do not have voice VLANs with IPv6 enabled yet, but I felt that this should be brought to everyone's attention as it is a peculiar issue and more people will encounter it as IPv6 continues to grow. The Windows machines aren't VLAN-enabled to run a softphone application. We also were puzzled with why the Windows machines saw the voice VLAN as well, because it wouldn't receive an IPv4 address on the voice VLAN, but would receive one in IPv6. While troubleshooting with a vendor, it was explained to me that with IPv6, the client will receive whatever link-locals are being advertised, which is why suppressing RA's on the VLAN interface is one of the workarounds. We don't actually use that workaround, but have instead manually assigned link-local addresses to each SVI. This will allow every client to receive multiple link-locals and gets around the issue of VLAN tags not being present. I look forward to discussing this further with everyone. Thank you for your reply. ---- Michael Fazely Network Analyst II Network Design University Networking & Infrastructure Louisiana State University Baton Rouge, LA 70803 Office: (225)578-1971 Email: mfazel1@lsu.edu
Message from jeroen@zijndomein.nl

Michael ao, Sorry for continuing this thread after a month of silence, ignore if you're already past this subject :) >> Hmm, that's strange... link-local is link-local, it's only significant when combined with a specific link (interface). >> It's perfectly valid to have the same link-local address on more than one link. A route with a link-local address >> as a next hop should (afaik) always explicitly point to the 'nexthop%interface' and not just to the 'nexthop' value. > >> I like to think of this like the primary key in a database: the combination of "link local addr" and "interface" is uniqe >> and is the correct way to refer to a specific entry. > > I agree with the statement "It's perfectly valid to have the same link-local address on more than one link", but only when VLAN tags are available. The issue here is that there are no VLAN tags associated with each address. But if no VLAN tags are available, ie traffic from both VLANs is hitting the computer without at least one of them having a VLAN tag, then it's one link from the viewpoint of the computer. From a networking point of view, that's an extremely rare use case and almost always a misconfiguration. >> Well, it shouldn't :) Whether it's IPv4, IPv6 or IPX or whatever, the OS shouldn't care what runs over the L2 link. >> And a physical link with vlans should be treated as N virtual L2 links: each VLAN is a separate >> L2 broadcast domain ie a separate virtual L2 link. > > My assertion isn't that the OS itself should care about the VLANs, but rather that Microsoft's NDIS, which NIC manufacturers use to write drivers instructs them to handle VLAN tagging improperly for IPv6. I apologize if this wasn't communicated properly in my initial email. Personally I consider NDIS a part of the OS. If a part of the OS (or the OS manufacturer) doesn't observe proper layering in the networking code that's just asking for trouble. Do you happen to have any pointers to NDIS documentation, where it actually instructs NIC drivers to disregard VLAN tags under certain circumstances? > I understand that many people either aren't running IPv6 on any level or do not have voice VLANs with IPv6 enabled yet, but I felt that this should be brought to everyone's attention as it is a peculiar issue and more people will encounter it as IPv6 continues to grow. Thank you for that; I'll keep your observations in mind and I'm sure the other NETMAN readers will too. > The Windows machines aren't VLAN-enabled to run a softphone application. We also were puzzled with why the Windows machines saw the voice VLAN as well, because it wouldn't receive an IPv4 address on the voice VLAN, but would receive one in IPv6. While troubleshooting with a vendor, it was explained to me that with IPv6, the client will receive whatever link-locals are being advertised, which is why suppressing RA's on the VLAN interface is one of the workarounds. I've only seen this happening when we were testing a very specific network setup, where it could actually happen that traffic from several VLANs could simultaneously egress a port untagged. I'm still wondering what exactly happens in your case; whether it's two untagged vlans on a port, or the Voice VLAN being tagged but the tag disregarded by (a layer or software component in) Windows. In the first case however, you should see similar behaviour on other OS's. Regards, Jeroen van Ingen ICT Service Centre University of Twente, P.O.Box 217, 7500 AE Enschede, The Netherlands ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Close
Close


Annual Conference
September 29–October 2
View Proceedings

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.