Main Nav

This is related to a couple of other posts:

 

http://www.educause.edu/discuss/networking-and-emerging-technologies/network-management-constituent-group/physical-separation-networks

http://www.educause.edu/discuss/networking-and-emerging-technologies/network-management-constituent-group/core-distribution-and-access-layer

 

 

We have some service networks (i.e. environmental controls, security cameras, door locks) that are currently isolated campus-wide at layer 2.  A VLAN is associated with a  service and then trunked to wherever the service is required – across buildings and across dedicated fiber links between campuses.  The VLAN is trunked back to a core ASA, and access is granted as necessary. 

 

We are building a case to move to a design similar to that which Jeff Kell presents here:  http://net.educause.edu/content.asp?page_id=1026971&PRODUCT_CODE=SEC11%2FSESS26&bhcp=1  In our design, we plan to build the VRFs out at the distribution layer only (VRF-lite, no MPLS).  A point-to-point VLAN will connect the VRF SVI at the distribution layer to a sub-interface on the core ASA.  From that point, access will be granted in exactly the same way it was granted before.

 

The goal of the design it to be a simple, to isolate broadcast as much as possible, and to isolate the individual service networks from each other so that if a rogue device is connected – it does not have visibility to the entire service VLAN.

 

We certainly intend to reference the University of Tennessee presentation, but in building this case, the more examples we can present, the better off we’ll be.  Has anyone else make a similar transition from campus-wide L2 isolation to some form of L3 isolation using VRFs?

 

Thanks,

 

John Bartin

 

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Comments

What about routing at the edge and moving your RBAC away from the core and out closer to the user?

 
Tim Cappalli, Network Engineer
LTS | Brandeis University
x67149 | (617) 701-7149
cappalli@brandeis.edu


We do similar with L2 isolation. I’m wondering if you’ve experienced issues that make you want to change, or are looking at doing a preventative evolution kinda thing.

 

-Lee Badman

 

I echo Lee's question.  Do you have good reason to do this?

Having said that, we are sitting at routed vlans, with ACLs as needed.  We trust our users (for the most part) allowing us to keep the ACLs fairly easy to manage.

If we wanted to move towards inter-vlan communications lock-down, then, yeah, I would have to seriously consider VRF.

Great presentation.  Thanks for sharing it and bringing up a great conversation on the list!

-
Pete Hoffswell - Network Manager
pete.hoffswell@davenport.edu
http://www.davenport.edu
616-732-1101


Thanks for the replies.

 

I’ve attached some diagrams to illustrated the old and the new design.

 

Tim – We hope to use the VRFs to push routing out to the edge without having to give up centralized firewall services.  If we do this without VRFs, we’d have to purchase firewalls or implement access-lists at each building.  We like having a single point of maintenance for access to the services network.

 

Jeff – Currently, we manually put ports into the appropriate service VLAN.  The NOC is contacted when a new device is installed and the port is provisioned at that time.  Under the new system, we would use PVLANs to isolate different services within the building, but some level of manual configuration would still be required when devices are installed or replaced.   Using a MAC registration system would be a nice addition though.

 

If anyone else can provide us support for or criticism of the proposed design, I’d appreciate any input.

 

Thanks!

 

John

 

 

 

From: The EDUCAUSE Network Management Constituent Group Listserv [mailto:NETMAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Tim Cappalli
Sent: Wednesday, May 29, 2013 7:20 AM
To: NETMAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [NETMAN] moving away from extended L2 networks for service networks

 

What about routing at the edge and moving your RBAC away from the core and out closer to the user?


 

Tim Cappalli, Network Engineer
LTS | Brandeis University
x67149 | (617) 701-7149
cappalli@brandeis.edu

 

What about routing at the edge and moving your RBAC away from the core and out closer to the user?

 
Tim Cappalli, Network Engineer
LTS | Brandeis University
x67149 | (617) 701-7149
cappalli@brandeis.edu


We do similar with L2 isolation. I’m wondering if you’ve experienced issues that make you want to change, or are looking at doing a preventative evolution kinda thing.

 

-Lee Badman

 

I echo Lee's question.  Do you have good reason to do this?

Having said that, we are sitting at routed vlans, with ACLs as needed.  We trust our users (for the most part) allowing us to keep the ACLs fairly easy to manage.

If we wanted to move towards inter-vlan communications lock-down, then, yeah, I would have to seriously consider VRF.

Great presentation.  Thanks for sharing it and bringing up a great conversation on the list!

-
Pete Hoffswell - Network Manager
pete.hoffswell@davenport.edu
http://www.davenport.edu
616-732-1101


Thanks for the replies.

 

I’ve attached some diagrams to illustrated the old and the new design.

 

Tim – We hope to use the VRFs to push routing out to the edge without having to give up centralized firewall services.  If we do this without VRFs, we’d have to purchase firewalls or implement access-lists at each building.  We like having a single point of maintenance for access to the services network.

 

Jeff – Currently, we manually put ports into the appropriate service VLAN.  The NOC is contacted when a new device is installed and the port is provisioned at that time.  Under the new system, we would use PVLANs to isolate different services within the building, but some level of manual configuration would still be required when devices are installed or replaced.   Using a MAC registration system would be a nice addition though.

 

If anyone else can provide us support for or criticism of the proposed design, I’d appreciate any input.

 

Thanks!

 

John

 

 

 

From: The EDUCAUSE Network Management Constituent Group Listserv [mailto:NETMAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Tim Cappalli
Sent: Wednesday, May 29, 2013 7:20 AM
To: NETMAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [NETMAN] moving away from extended L2 networks for service networks

 

What about routing at the edge and moving your RBAC away from the core and out closer to the user?


 

Tim Cappalli, Network Engineer
LTS | Brandeis University
x67149 | (617) 701-7149
cappalli@brandeis.edu

 

Message from iam@st-andrews.ac.uk

We've some L2 some L3. L2 is only deployed where the service (bacnet springs to mind) expects/requires things to be in 1 broadcast domain.

HTH

--
ian

I realize, I had missed some messages when I sent my last reply.

 

Pete and Lee,

 

We see the following advantages in this design over the extended L2:

 

1.       Limit the scope of broadcast.  These networks are not huge in terms of number of hosts, but the broadcast adds up.  We’d prefer not have all the broadcast traffic hitting the core ASA.  We also think this  will somewhat limit the impact of any virus or security compromise.  (With the L2 approach, any poorly secured ports could allow a rogue device to easily see all devices on the service network – campus-wide.  The L3 approach allows for some level of isolation between buildings.)

 

2.       Simplified troubleshooting. We can use traceroute or routing table information to isolate the physical location of a down device.  We can derive physical location from the IP address.  With the L2 method, we have to chase the MAC address down – which is not possible if the device has been down longer than the CAM table timeout.

 

3.       Scalability.  We can scale this approach to handle remote locations, and we do not have to worry about exhausting IP space for a service because new subnets can easily be allocated.  With L2 networks terminating on an ASA, we depend on L2 links to all locations and we do not have the capability to add secondary IP space.  Allocating more IP space to a service is difficult.

 

4.       It also seems more in line with general network design recommendations.  Using L3 connections where possible, and limiting the scope of L2 broadcast domains.

 

All that being said, we are not experiencing any imminent failures with the L2 approach.  We do not intend for this to be a slash cut, but a gradual migration as devices are upgraded and as new buildings come online.

 

John

 

 

 

On 5/29/2013 6:29 PM, Bartin, John wrote:
We see the following advantages in this design over the extended L2:

 

1.       Limit the scope of broadcast.  These networks are not huge in terms of number of hosts, but the broadcast adds up.  We’d prefer not have all the broadcast traffic hitting the core ASA.  We also think this  will somewhat limit the impact of any virus or security compromise.  (With the L2 approach, any poorly secured ports could allow a rogue device to easily see all devices on the service network – campus-wide.  The L3 approach allows for some level of isolation between buildings.)


That is correct.  To sniff traffic between hosts covertly (e.g., dsniff) you must be on the same subnet as one of the endpoints, and do a MITM attack via ARP spoofing of the gateway.  You can reduce the risk with some network gear (e.g., ARP inspection or DHCP sniffing or private vlans) but it remains a risk.  Spreading this to L3 reduces the attack space significantly.
 

2.       Simplified troubleshooting. We can use traceroute or routing table information to isolate the physical location of a down device.  We can derive physical location from the IP address.  With the L2 method, we have to chase the MAC address down – which is not possible if the device has been down longer than the CAM table timeout.


Adjust your CAM timeout and/or ARP timeout so that your CAMs remain populated.  On Cisco switches, you can bump up the CAM timeout from the default (five minutes) and bring the ARP timeout down (default is like 4 hours, IIRC).  Preferably make them the same values.  One "significantly obscure" feature of Cisco layer-3 devices is that they will issue a directed ARP at any device prior to the ARP timeout removing the entry from the table.  This will repopulate your CAMs along the connected path.

 

3.       Scalability.  We can scale this approach to handle remote locations, and we do not have to worry about exhausting IP space for a service because new subnets can easily be allocated.  With L2 networks terminating on an ASA, we depend on L2 links to all locations and we do not have the capability to add secondary IP space.  Allocating more IP space to a service is difficult.


We use RFC1918 space exclusively, and use a 10.building.vlan.host scheme so that it is relatively easy to duplicate any service vlan in any building, and extend that so that VPN and wireless are simply another "building".

 

4.       It also seems more in line with general network design recommendations.  Using L3 connections where possible, and limiting the scope of L2 broadcast domains.


The L2 broadcast domains are currently only a pain on wireless.  We're an Aruba shop, and while Aruba supports "vlan pooling" to distribute clients across multiple L3 vlans, they don't yet support associating their "roles" with a vlan pool.  So we have some LARGE spaces on wireless, and particularly for unregistered wireless.  Most handheld devices and iThings will try to automatically associate with any open WiFi automagically, and they will associate and DHCP an address...  But we do have broadcast/multicast disabled between wireless devices, so the broadcast isn't as much an issue as it might otherwise be (there is however the AppleTV contingent trying to make us reverse that decision).

 

All that being said, we are not experiencing any imminent failures with the L2 approach.  We do not intend for this to be a slash cut, but a gradual migration as devices are upgraded and as new buildings come online.


We have a few L2s extended from the core, but we have gradually eliminated all but a few.  We used to run our wireless off of a Cisco 3560, but last fall we exceeded the mac-address capacity (6K with a routed SDM template) and had to punt :)

Jeff
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Message from mark.duling@biola.edu

On the troubleshooting and scalability matter, when everything is working well you can have large L2 networks that perform quite well.  But when something goes wrong figuring out what is the problem can be quite difficult.  I think of L2 networks as "fail open" by the way they spray frames around by their forwarding/flooding behavior, whereas an L3 segmented network is a "fail closed" scenario since routers don't forward packets if they can't determine where to send them.  Large L2 networks can have spectacular meltdowns.

By a few years ago we had slowly refreshed our campus gear to have one L3 capable switch in each building in preparation to go L3 segmented across campus.  But we were waiting for summer to pull the trigger and turn on L3.  Our network had been entirely stable for over a decade, but wouldn't you know it about a month before summer came the network became unstable at a certain time of the day disrupting the whole campus.  It didn't appear to be spanning-tree issues.  The problem would then vanish after a time until about the same time next day.  I'm not the best troubleshooter, but I couldn't tell what the problem was so it was faster to work at night to expedite the summer plan to turn on L3 in those switches to split up the campus and hopefully be able to determine the root cause in one of the segments after that.  But in fact after shrinking the L2 domains the problem went away entirely and never came back. 

All I'm saying is that you have a scalable L2 network right up until you don't.
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.