Main Nav

On Jan 17, 2014, at 22:16 , Mike Albano wrote: > > Would be nice if more technical details were available. For example, at what part of the EAP/PEAP packet exchange does this delay occur? Sounded like part of the issue was with CRL and/or OCSP. Also interesting was that they are advising folks to set trust settings on all the certs in the chain... -- Julian Y. Koh Acting Associate Director, Telecommunications and Network Services Northwestern University Information Technology (NUIT) 2001 Sheridan Road #G-166 Evanston, IL 60208 847-467-5780 NUIT Web Site: PGP Public Key: ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Comments

Absolutely! This is huge. They never, ever (ever ever ever) admit there is an issue. Maybe we’re seeing some change at the fruit?

 

 

(Unlikely, but it’s nice to dream)

 

 

Tim Cappalli  |  ACCP /  ACMP /  CCNA
Network Engineer  |  Brandeis University
cappalli@brandeis.edu | (617) 701-7149

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Joel Coehoorn
Sent: Friday, January 17, 2014 7:58 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

Even acknowledging the issue is a huge help for me: Mac people have a hard time believing Apple could possibly have done anything wrong with their device until you have something like this to point to. Until Apple own recommendation is to change the setting on the device, their view is the problem *must* be in the network.

Sent from my iPad


Is anyone working on (or successfully implemented) a scalable, automated(?) solution to change the SSL to 'Always Trust' for target certs and distributed this to their client devices en masse? x-press-con-nect folks offered a glimmer of hope for adding this feature to their routine but I was wondering if we could do something quicker. Has anyone tweaked Apple's command - suggested in their KB article - into an Applescript for distribution? As the cert is already installed on the devices I would thing some modification is needed. http://support.apple.com/kb/TS5258 Michael Dickson Network Analyst Office of Information Technologies University of Massachusetts Amherst Voice 413.545.9639
Message from iam@st-andrews.ac.uk

I'd be more interested in a method for doing this in a .mobileconfig file, or for them to fix it in a manner that doesn't involve us having to mess about on the clients. -- ian -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Michael Dickson Sent: 21 January 2014 17:06 To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue Is anyone working on (or successfully implemented) a scalable, automated(?) solution to change the SSL to 'Always Trust' for target certs and distributed this to their client devices en masse? x-press-con-nect folks offered a glimmer of hope for adding this feature to their routine but I was wondering if we could do something quicker. Has anyone tweaked Apple's command - suggested in their KB article - into an Applescript for distribution? As the cert is already installed on the devices I would thing some modification is needed. http://support.apple.com/kb/TS5258 Michael Dickson Network Analyst Office of Information Technologies University of Massachusetts Amherst Voice 413.545.9639
Anyone have concerns about making the trust setting changes to the certificate chain?  I'm thinking of the intermediate certs mostly.  Setting "always trust" on a client machine just makes me a little uncomfortable.
 - Don


+1 to that.

-dan


On 1/23/2014 9:28 AM, Wright, Don wrote:
Anyone have concerns about making the trust setting changes to the certificate chain?  I'm thinking of the intermediate certs mostly.  Setting "always trust" on a client machine just makes me a little uncomfortable.
 - Don


Message from iam@st-andrews.ac.uk

I certainly do have concerns about this being the right way to 'fix' the issue. Sticking plaster on the client behaviour this is..

Thanks

--
ian

Sent from my phone, please excuse brevity and misspelling.
From: Dan Brisson
Sent: ‎23/‎01/‎2014 14:41
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

+1 to that.

-dan


On 1/23/2014 9:28 AM, Wright, Don wrote:
Anyone have concerns about making the trust setting changes to the certificate chain?  I'm thinking of the intermediate certs mostly.  Setting "always trust" on a client machine just makes me a little uncomfortable.
 - Don


I am going to plead some ignorance here, and see if people can connect the dots…

 

We use 802.1X (TLS), and we use Godaddy Certs for our radius server.  The clients are set to verify the server certificates.  When I look at the installed certificates, I see information for CRLs.  Yet, I connect almost instantaneously with our SSIDs.  Why do some of you seem to be having such an issue with this, and I don’t seem to?

 

Ryan H Turner

Senior Network Engineer

The University of North Carolina at Chapel Hill

CB 1150 Chapel Hill, NC 27599

+1 919 445 0113 Office

+1 919 274 7926 Mobile

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Ian McDonald
Sent: Thursday, January 23, 2014 9:52 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

I certainly do have concerns about this being the right way to 'fix' the issue. Sticking plaster on the client behaviour this is..

Thanks

--
ian

Sent from my phone, please excuse brevity and misspelling.

From: Dan Brisson
Sent: ‎23/‎01/‎2014 14:41
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

+1 to that.

-dan



 

On 1/23/2014 9:28 AM, Wright, Don wrote:

Anyone have concerns about making the trust setting changes to the certificate chain?  I'm thinking of the intermediate certs mostly.  Setting "always trust" on a client machine just makes me a little uncomfortable.

 - Don

 

It doesn't happen for TLS....(where clients are authenticated using a cert your PKI infrastructure has provided) but appears specific for PEAP and TTLS - where the client uses a password to authenticate.

It also appears specific to certs based on 2048 bit keys.   Also there is no cert validation delay upon initial connect... only when attempting to reauth... ie after a death or a roam event.


-Travis


‘It also appears specific to certs based on 2048 bit keys.   Also there is no cert validation delay upon initial connect... only when attempting to reauth... ie after a death or a roam event.”

 

Correct.

 

FYI, Cloudpath (XPC) has a way to configure the SSL Trust settings now.

 

Marcelo Lew

Wireless Network Architect & Engineer

University Technology Services

University of Denver

Desk: (303) 871-6523

Cell: (303) 669-4217

Fax:  (303) 871-5900

Email: mlew@du.edu

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Travis Schick
Sent: Thursday, January 23, 2014 10:10 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

It doesn't happen for TLS....(where clients are authenticated using a cert your PKI infrastructure has provided) but appears specific for PEAP and TTLS - where the client uses a password to authenticate.

It also appears specific to certs based on 2048 bit keys.   Also there is no cert validation delay upon initial connect... only when attempting to reauth... ie after a death or a roam event.

-Travis

 

Thanks, all!

 

Ryan H Turner

Senior Network Engineer

The University of North Carolina at Chapel Hill

CB 1150 Chapel Hill, NC 27599

+1 919 445 0113 Office

+1 919 274 7926 Mobile

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Marcelo Lew
Sent: Thursday, January 23, 2014 1:19 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

‘It also appears specific to certs based on 2048 bit keys.   Also there is no cert validation delay upon initial connect... only when attempting to reauth... ie after a death or a roam event.”

 

Correct.

 

FYI, Cloudpath (XPC) has a way to configure the SSL Trust settings now.

 

Marcelo Lew

Wireless Network Architect & Engineer

University Technology Services

University of Denver

Desk: (303) 871-6523

Cell: (303) 669-4217

Fax:  (303) 871-5900

Email: mlew@du.edu

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Travis Schick
Sent: Thursday, January 23, 2014 10:10 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

It doesn't happen for TLS....(where clients are authenticated using a cert your PKI infrastructure has provided) but appears specific for PEAP and TTLS - where the client uses a password to authenticate.

It also appears specific to certs based on 2048 bit keys.   Also there is no cert validation delay upon initial connect... only when attempting to reauth... ie after a death or a roam event.

-Travis

 

‘It also appears specific to certs based on 2048 bit keys.   Also there is no cert validation delay upon initial connect... only when attempting to reauth... ie after a death or a roam event.”
 
Correct.
hehe... Not sure Apple can help with the delay after a death event.... but perhaps after a de-auth or any other event that causes a client to reconnect. :)
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Message from iam@st-andrews.ac.uk

Ahh, autocorrect errors nearly always cause amusement. A recent advertisement offered 'special pubic sector discounts'....

Thanks

--
ian

Sent from my phone, please excuse brevity and misspelling.
   Taking a slight tangent here, has client roaming and dropout problems motivated anyone to move to a WPA2-PSK model across their campus?  The second part of the question is if you have, is it any better or worse to manage than an 802.1X network?  
- Don


Message from normelton@gmail.com

> It also appears specific to certs based on 2048 bit keys. Also there is no > cert validation delay upon initial connect... only when attempting to > reauth... ie after a death or a roam event. Can anyone confirm the bug only affects certs with 2048 bit keys? I don't see that listed anywhere in Apple's release. It's an interesting twist. Thanks! Norman Elton College of William & Mary ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Message from normelton@gmail.com

And a follow up. Has anyone actually confirmed that this bug is actually causing client complaints? We do seem to riding a wave of complaints from MacBook owners. We are only just now starting to change cert trust settings. Hopefully we'll know more next week as students have a chance to test things out over the weekend. Norman Elton College of William & Mary
We've seen the cert issue, and OS 10.8 and 10.9 don't seem to like band/load-steering. The cert issue coupled with band-steering and/or load-steering make the Mac's very unhappy.
 
Jeff

>>> On Friday, January 31, 2014 at 10:05 AM, in message <CAPCnwUdAuZqKuFwOycKrGmXgiKCrb_Wy82=O5Xc3Be+o7Antuw@mail.gmail.com>, Norman Elton <normelton@GMAIL.COM> wrote:
And a follow up. Has anyone actually confirmed that this bug is
actually causing client complaints? We do seem to riding a wave of
complaints from MacBook owners. We are only just now starting to
change cert trust settings. Hopefully we'll know more next week as
students have a chance to test things out over the weekend.

Norman Elton
College of William & Mary

Message from ipe@algonquincollege.com

I agree with Jeff, we recently disabled band steering on our Aruba controllers and it has helped a bit.

Edward Ip

Algonquin College | 1385 Woodroffe Avenue | Room C316 | Ottawa | Ontario | K2G 1V8 | Canada

algonquincollege.com

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Jeffrey Sessler
Sent: Friday, January 31, 2014 1:40 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

We've seen the cert issue, and OS 10.8 and 10.9 don't seem to like band/load-steering. The cert issue coupled with band-steering and/or load-steering make the Mac's very unhappy.

 

Jeff

>>> On Friday, January 31, 2014 at 10:05 AM, in message <CAPCnwUdAuZqKuFwOycKrGmXgiKCrb_Wy82=O5Xc3Be+o7Antuw@mail.gmail.com>, Norman Elton <normelton@GMAIL.COM> wrote:

And a follow up. Has anyone actually confirmed that this bug is
actually causing client complaints? We do seem to riding a wave of
complaints from MacBook owners. We are only just now starting to
change cert trust settings. Hopefully we'll know more next week as
students have a chance to test things out over the weekend.

Norman Elton
College of William & Mary

Message from normelton@gmail.com

Interesting. What were the band-steering symptoms? Any way to pin the problem down to band-steering, or was it trial and error?

Norman


Message from ipe@algonquincollege.com

When we did a packet capture, we saw that the Mac clients would try to associate on the 2.4GHz band and then move over to the 5GHz band and then back. We saw this happen quite a bit. This made the re association longer when our clients roamed.

 

Sometimes in certain areas of our college if 2 APs are within range with about the same SNR, the Mac client would get shifted between the two APs due to load balance mode turned on. Due to the cert issue and the re association process it would drive our Mac clients (err…students) crazy around crunch time.

 

Edward Ip

Algonquin College | 1385 Woodroffe Avenue | Room C316 | Ottawa | Ontario | K2G 1V8 | Canada

algonquincollege.com

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Norman Elton
Sent: Friday, January 31, 2014 1:57 PM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

Interesting. What were the band-steering symptoms? Any way to pin the problem down to band-steering, or was it trial and error?

 

Norman

 

We noticed that the WLAN with band/load-steering enabled had a high report rate of Macintosh connectivity issues, and the WLAN that did not was trouble free.
 
I suspect what was happening was this: Mac would initially associate (Ent-WPA2), then the controller would force it to move to another band and/or AP. It's at this point (a roam) that the Apple certificate issue would kick in, and it was hit or miss as to the Mac re-associating or failing. This was especially problematic when a Mac client was equidistant from two AP's.
 
Turning off band/load steering pretty much eliminated the bulk of the connectivity issues, and trusting the certificate solved the rest.
 
Band/load steering is just problematic because you can never predict how a client will react to it.
 
Jeff

>>> On Friday, January 31, 2014 at 10:57 AM, in message <CAPCnwUdh-=JAwm78PJfuU1n9bhS9d_JApthBFNWRRGsrBZgtsg@mail.gmail.com>, Norman Elton <normelton@GMAIL.COM> wrote:
Interesting. What were the band-steering symptoms? Any way to pin the problem down to band-steering, or was it trial and error?

Norman


Message from normelton@gmail.com

Sorry for the spam today, one last question for those that have experienced the cert issue. What was the client symptom? That is, what does the user see during the 10 second delay? Wifi icon is all grey, laddering up and down, all dark?

Thanks,

Norman Elton
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

One other wrench in this at least from the Aruba standpoint.... check the cpu load on the Auth process....  we found back in late October that one of our heaviest used controller (M3 running 6.1.3.7) was pegging over 90% utilization for the Auth process which at the time
we believed the authentication was being additionally impacted (mostly drops).  It was indicated (source does not want to be mentioned) that there was a hard limit to the number of auth's per second the controller could handle (approx. 40 - 50/sec), at peak we were
running around ~100/sec.  We upgraded to the version 6.3.x to resolve other issues.  We noticed that the system now spawned 3 Auth processes, but we still getting complaints.  We then discovered through TAC and internal investigation that a new dot1x throttling 
mechanism had been introduced in the version of code. This new "Throttling" was still impacting our authentications but saving the cpu's on the auth process.  We were instructed to adjust the Watermarks to reach a balance point from the defaults.  This is a slidiing scale....
the higher the Watermarks, the higher the cpu process, but the less drops experienced.

to view the cpu process:  "Show cpuload current | include auth"

On 6.3x code:

to view the Throttle parameters: "show dot1x counters"  (There is some math involved when the system decides to drop packets)
to view the dropped auths: "show ap debug client-mgmt-counters" and look for the "Associations Dropped Due to Auth Throttling"

In the end, the old addage still holds true..."You can never please 100% of the people 100% of the time....Keep calm and carry on"

M



On 2014-01-31, at 2:11 PM, Jeffrey Sessler wrote:

We noticed that the WLAN with band/load-steering enabled had a high report rate of Macintosh connectivity issues, and the WLAN that did not was trouble free.
 
I suspect what was happening was this: Mac would initially associate (Ent-WPA2), then the controller would force it to move to another band and/or AP. It's at this point (a roam) that the Apple certificate issue would kick in, and it was hit or miss as to the Mac re-associating or failing. This was especially problematic when a Mac client was equidistant from two AP's.
 
Turning off band/load steering pretty much eliminated the bulk of the connectivity issues, and trusting the certificate solved the rest.
 
Band/load steering is just problematic because you can never predict how a client will react to it.
 
Jeff

>>> On Friday, January 31, 2014 at 10:57 AM, in message <CAPCnwUdh-=JAwm78PJfuU1n9bhS9d_JApthBFNWRRGsrBZgtsg@mail.gmail.com>, Norman Elton <normelton@GMAIL.COM> wrote:
Interesting. What were the band-steering symptoms? Any way to pin the problem down to band-steering, or was it trial and error?

Norman


You mentioned "load balance mode", were you running band-steering and spectrum-load-balancing at the same time on the same APs?  Check with Aruba, but I think they will tell you this is not a recommended practice and better to stick with band-steering only assuming your coverage will support both radios.
Also, I read somewhere that recent MacOS updates already try to "band-steer" the client to the 5GHZ radio?  If that's true, your controller setting really isn't having an affect anyway, and the client are getting themselves into trouble..  
- Don


Michael,  Are you using AAA Fastconnect allowing your controller to handle radius requests instead of using a backend server?  While we haven't done this ourselves, I know of others that have run into the same issue of not being able to keep up with the auth requests.  You'll notice this even more as smartphones will re-auth all the time.
- Don


Message from ipe@algonquincollege.com

We only used spectrum-load balance mode in very high dense classrooms, but we have this feature turned off now. We now also have band steering turned off based on recommendation we got from an Aruba ACE engineer. The band steering feature was causing problems mainly amongst the Apple smartphones.

 

Edward Ip

Algonquin College | 1385 Woodroffe Avenue | Room C316 | Ottawa | Ontario | K2G 1V8 | Canada

algonquincollege.com

 

From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Wright, Don
Sent: Monday, February 03, 2014 11:44 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

 

You mentioned "load balance mode", were you running band-steering and spectrum-load-balancing at the same time on the same APs?  Check with Aruba, but I think they will tell you this is not a recommended practice and better to stick with band-steering only assuming your coverage will support both radios.

Also, I read somewhere that recent MacOS updates already try to "band-steer" the client to the 5GHZ radio?  If that's true, your controller setting really isn't having an affect anyway, and the client are getting themselves into trouble..  

- Don

 

We use Radiator backend servers bound to each controller independently with failover to another Radius server.  At first we thought it was our radius servers having performance issues, but after checking response times and packet captures, we found the controller to be the lynch-pin.

M

On 2014-02-03, at 12:56 PM, Wright, Don wrote:

Michael,  Are you using AAA Fastconnect allowing your controller to handle radius requests instead of using a backend server?  While we haven't done this ourselves, I know of others that have run into the same issue of not being able to keep up with the auth requests.  You'll notice this even more as smartphones will re-auth all the time.
- Don


I would like to know the answer to this question as well.


------------------------
Walter Reynolds
Principal Systems Security Development Engineer
Information and Technology Services
University of Michigan
(734) 615-9438


Looks like Apple finally sort of “admitted” of an issue with 802.1x authentication, several months later and most of us already knew this work around, but better late than never J

 

http://support.apple.com/kb/TS5258

 

 

 

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Message from jcoehoorn@york.edu

Even acknowledging the issue is a huge help for me: Mac people have a hard time believing Apple could possibly have done anything wrong with their device until you have something like this to point to. Until Apple own recommendation is to change the setting on the device, their view is the problem *must* be in the network.

Sent from my iPad

Would be nice if more technical details were available. For example, at what part of the EAP/PEAP packet exchange does this delay occur? 
I've seen numerous times where the "Access-Challenge" is sent from RADIUS and received by the client (verified by 802.11 packets & WLC debugs)...and then it just sits there, doesn't respond, and eventually EAP starts over. 

Something tells me that level of detail won't be released.

Mike Albano

-----The EDUCAUSE Wireless Issues Constituent Group Listserv <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU> wrote: -----
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
From: Joel Coehoorn
Sent by: The EDUCAUSE Wireless Issues Constituent Group Listserv
Date: 01/17/2014 05:06PM
Subject: Re: [WIRELESS-LAN] OS X 802.1x auth issue

Even acknowledging the issue is a huge help for me: Mac people have a hard time believing Apple could possibly have done anything wrong with their device until you have something like this to point to. Until Apple own recommendation is to change the setting on the device, their view is the problem *must* be in the network.

Sent from my iPad

Close
Close


Annual Conference
September 29–October 2
Register Now!

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.