Main Nav

A question for folks with relatively large 802.1x (greater than 15,000 unique clients) wi-fi deployment (EAP-TTLS) with a FreeRADIUS infrastructure using Kerberos as the backend authentication …..

 

- how many FreeRADIUS servers do you deploy?, and

- have you changed any of the default eap.con/radius.conf performance parameters/values?

 

The good news is that we've started the year with a lot more folks finally using the 802.1x network than the last academic year.

The bad news is that we're getting long delays in connecting/authenticating -- not just a wireless issue as we're also getting lots of "RADIUS server FAILED" traps from our VPN concentrators throughout the day since the semester started (using the same RADIUS servers as the 1x wireless deployment)

 

We've also been seeing in the last three days HUGE numbers of:

Aug 22 19:25:00 calvin radiusd[21691]: Discarding duplicate request from client Wireless8021XResNET port 32769 - ID: 76 due to unfinished request 253745

Aug 22 19:25:00 calvin radiusd[21691]: Discarding duplicate request from client Wireless8021XResNET port 32769 - ID: 140 due to unfinished request 253705

Aug 22 19:25:00 calvin radiusd[21691]: Discarding duplicate request from client Wireless8021XResNET port 32769 - ID: 85 due to unfinished request 253758

and

Aug 19 03:30:14 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli 68-a8-6d-ae-fc-5d)

Aug 19 03:31:15 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli 28-6a-ba-6a-9d-6e)

Aug 19 03:31:35 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli c8-bc-c8-2e-52-13)

Aug 19 03:32:13 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli 10-40-f3-29-60-2c)

 

which, from what we can discern from the wonderful world of google, may be related to a "slow database", although our Kerberos folks don't see any issues on their end.

 

Any thoughts?     Responses to the two questions above would be appreciated … thanks!!

 

-- Jim Gogan / Univ of North Carolina at Chapel Hill

 

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Comments

Message from shuque@upenn.edu

Jim, We've been through this, and I'll describe what we did to address it. There are two problems with the freeradius code that cause performance problems with a Kerberos backend: 1) It doesn't disable the replay cache, which isn't needed for password verification operations (as opposed to native Kerberos), and causes an unnecessary performance bottleneck, as every authentication has to compare authenticators against a large cache of recently seen authenticators. The easiest way to disable the cache is to set the environment variable KRB5RCACHETYPE to "none" before starting freeradius. The MIT Kerberos software on our RADIUS servers though is so old (v1.3.x) that it didn't support this, so I had to disable it by writing a patch to the source code (in rlm_krb5.c). If you need that, let me know and I can send it. I think KRB5RCACHETYPE appeared in v1.4. 2) Although freeradius is multi-threaded, the Kerberos authentication module is single-threaded. I believe recent versions of both MIT and Heimdal Kerberos libraries are threadsafe, but I don't think the freeradius code has been updated for this. So the only way to scale the performance of the RADIUS infrastructure is to deploy more servers (they don't have to by physical, you can install multiple software instances on the same server if you have extra CPUs or cores). We currently have 8 freeradius servers running on 4 physical servers (2 per box). And we balance the wireless controllers across those. We actually implemented (1) first, and it instantly fixed the performance problems we were seeing at the time. But we wanted to head off the second problem and stay well ahead of demand, so started deploying multiple freeradius instances per server. One of our staff members is also looking at patching freeradius to multithread the Kerberos bits, but unless the freeradius folks accept those patches and maintain them, we likely won't deploy that option in production. --Shumon. On Wed, Aug 22, 2012 at 11:31:22PM +0000, Gogan, James P wrote: > A question for folks with relatively large 802.1x (greater than 15,000 unique clients) wi-fi deployment (EAP-TTLS) with a FreeRADIUS infrastructure using Kerberos as the backend authentication ..... > > - how many FreeRADIUS servers do you deploy?, and > - have you changed any of the default eap.con/radius.conf performance parameters/values? > > The good news is that we've started the year with a lot more folks finally using the 802.1x network than the last academic year. > The bad news is that we're getting long delays in connecting/authenticating -- not just a wireless issue as we're also getting lots of "RADIUS server FAILED" traps from our VPN concentrators throughout the day since the semester started (using the same RADIUS servers as the 1x wireless deployment) > > We've also been seeing in the last three days HUGE numbers of: > Aug 22 19:25:00 calvin radiusd[21691]: Discarding duplicate request from client Wireless8021XResNET port 32769 - ID: 76 due to unfinished request 253745 > Aug 22 19:25:00 calvin radiusd[21691]: Discarding duplicate request from client Wireless8021XResNET port 32769 - ID: 140 due to unfinished request 253705 > Aug 22 19:25:00 calvin radiusd[21691]: Discarding duplicate request from client Wireless8021XResNET port 32769 - ID: 85 due to unfinished request 253758 > and > Aug 19 03:30:14 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli 68-a8-6d-ae-fc-5d) > Aug 19 03:31:15 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli 28-6a-ba-6a-9d-6e) > Aug 19 03:31:35 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli c8-bc-c8-2e-52-13) > Aug 19 03:32:13 calvin radiusd[3507]: Login incorrect: [anonymous] (from client Wireless8021XResNET port 29 cli 10-40-f3-29-60-2c) > > which, from what we can discern from the wonderful world of google, may be related to a "slow database", although our Kerberos folks don't see any issues on their end. > > Any thoughts? Responses to the two questions above would be appreciated ... thanks!! > > -- Jim Gogan / Univ of North Carolina at Chapel Hill > > > ********** > Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. > -- Shumon Huque University of Pennsylvania. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Message from a.cudbardb@freeradius.org

On 23 Aug 2012, at 01:30, Shumon Huque wrote: > Jim, > > We've been through this, and I'll describe what we did to > address it. > > There are two problems with the freeradius code that cause > performance problems with a Kerberos backend: > > 1) It doesn't disable the replay cache, which isn't needed > for password verification operations (as opposed to native > Kerberos), and causes an unnecessary performance bottleneck, > as every authentication has to compare authenticators against > a large cache of recently seen authenticators. > > The easiest way to disable the cache is to set the environment > variable KRB5RCACHETYPE to "none" before starting freeradius. > The MIT Kerberos software on our RADIUS servers though is so > old (v1.3.x) that it didn't support this, so I had to disable > it by writing a patch to the source code (in rlm_krb5.c). If > you need that, let me know and I can send it. I think KRB5RCACHETYPE > appeared in v1.4. > So an interesting question would be - is anyone actually using EAP-Kerberos? If not, i'll disable caching by default and add a note to the configuration. AFAIK no supplicant has actually implemented any of the client side infrastructure to distribute tickets to other applications. It's annoying, SSO where you got your TGT from an 802.1X authentication would be really neat. > 2) Although freeradius is multi-threaded, the Kerberos authentication > module is single-threaded. I believe recent versions of both > MIT and Heimdal Kerberos libraries are threadsafe, but I don't > think the freeradius code has been updated for this. So the only > way to scale the performance of the RADIUS infrastructure is to > deploy more servers (they don't have to by physical, you can install > multiple software instances on the same server if you have extra > CPUs or cores). > > We currently have 8 freeradius servers running on 4 physical > servers (2 per box). And we balance the wireless controllers > across those. As the university I used to work at, we were handling a similar load with two Xserve G5s, but we were using LDAP and not Kerberos. > > We actually implemented (1) first, and it instantly fixed > the performance problems we were seeing at the time. But we > wanted to head off the second problem and stay well ahead of > demand, so started deploying multiple freeradius instances per > server. > > One of our staff members is also looking at patching freeradius > to multithread the Kerberos bits, but unless the freeradius > folks accept those patches and maintain them, we likely won't > deploy that option in production. > v2.1.x has gone into an unofficial feature freeze for anything not related to DHCP functionality. If you want this code to be included in function releases please submit the patches against the master branch (3.0). If the module uses long lived connection handles, it must use the connection API (src/main/connection.c). rlm_rest, rlm_sql and rlm_ldap2 (still in development) do this already, and all modules will be updated in time. If the patches are well formatted, documented, and non-duplicative we'll almost certainly accept them. -Arran ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
We used to have a setup where most all of our authentication went against 1 or two servers.  We did make some changes in radiusd.conf and did not have a problem with any of this. We have since also allowed PEAP but still do not see problems.  I found that when we did have problems it was never Kerberos related however.

Figure if the request is not handled in 5 seconds there is a problem, so lets free things up.
#max_request_time = 30
max_request_time = 5


Again, figure that losses of the reply packet on our network should be low so why hold on to the request
#cleanup_delay = 5
cleanup_delay = 2

Wanted to make sure we can handle enough request.  We did see problems at 1024 but not at the new 4096
#max_requests = 1024
max_requests = 4096

I have no notes that any of this was changed, but just in case I will list our settings.
reject_delay = 1

thread pool {
        start_servers = 5
        max_servers = 32
        min_spare_servers = 3
        max_spare_servers = 10
        max_requests_per_server = 0
}

------------------------
Walter Reynolds
Principal Systems Security Development Engineer
Information and Technology Services
University of Michigan
(734) 615-9438


Message from shuque@upenn.edu

On Thu, Aug 23, 2012 at 08:18:18AM +0100, Arran Cudbard-Bell wrote: > So an interesting question would be - is anyone actually using > EAP-Kerberos? If not, i'll disable caching by default and add a note > to the configuration. AFAIK no supplicant has actually implemented > any of the client side infrastructure to distribute tickets to other > applications. It's annoying, SSO where you got your TGT from an > 802.1X authentication would be really neat. Disabling the cache by default would be great. Thanks! EAP-Kerberos doesn't actually exist today as a documented spec - I'm sure that's why there's no client side code. I agree that it would be very nice, not only for the SSO function, but also because it would not be exposing the Kerberos password to the RADIUS servers. Many years ago, I tried to drum up interest in developing an EAP-Kerberos spec in the IETF. Ultimately, it didn't go anywhere, but you can read some of the discussion in the following archived thread: http://www.ietf.org/mail-archive/web/secmech/current/msg00041.html > > 2) Although freeradius is multi-threaded, the Kerberos authentication > > module is single-threaded. I believe recent versions of both > > MIT and Heimdal Kerberos libraries are threadsafe, but I don't > > think the freeradius code has been updated for this. So the only > > way to scale the performance of the RADIUS infrastructure is to > > deploy more servers (they don't have to by physical, you can install > > multiple software instances on the same server if you have extra > > CPUs or cores). > > > > We currently have 8 freeradius servers running on 4 physical > > servers (2 per box). And we balance the wireless controllers > > across those. > > As the university I used to work at, we were handling a similar load with two Xserve G5s, but we were using LDAP and not Kerberos. > The LDAP module is multithreaded, right? That would give it an advantage. One other mistake we made was that our RADIUS server hardware has many cores, but each core itself is quite slow - this was before we found out that the freeradius krb5 module was single threaded. If we'd known that earlier, we'd have instead purchased machines with fewer faster cores. > > One of our staff members is also looking at patching freeradius > > to multithread the Kerberos bits, but unless the freeradius > > folks accept those patches and maintain them, we likely won't > > deploy that option in production. > > > > v2.1.x has gone into an unofficial feature freeze for anything not related to DHCP functionality. If you want this code to be included in function releases please submit the patches against the master branch (3.0). > > If the module uses long lived connection handles, it must use the connection API (src/main/connection.c). rlm_rest, rlm_sql and rlm_ldap2 (still in development) do this already, and all modules will be updated in time. > > If the patches are well formatted, documented, and non-duplicative we'll almost certainly accept them. > > -Arran Thanks! I'll pass this info along to our team! --Shumon. -- Shumon Huque University of Pennsylvania. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Message from a.cudbardb@freeradius.org

> Disabling the cache by default would be great. Thanks! > > EAP-Kerberos doesn't actually exist today as a documented spec - Ah I guess I guess what I read wasn't an official IETF draft (it was years ago and I figured someone might have done something by now). > I'm sure that's why there's no client side code. I agree that it > would be very nice, not only for the SSO function, but also because > it would not be exposing the Kerberos password to the RADIUS > servers. Indeed. > > Many years ago, I tried to drum up interest in developing an > EAP-Kerberos spec in the IETF. Ultimately, it didn't go anywhere, > but you can read some of the discussion in the following archived > thread: > > http://www.ietf.org/mail-archive/web/secmech/current/msg00041.html Thanks :) > >>> 2) Although freeradius is multi-threaded, the Kerberos authentication >>> module is single-threaded. I believe recent versions of both >>> MIT and Heimdal Kerberos libraries are threadsafe, but I don't >>> think the freeradius code has been updated for this. So the only >>> way to scale the performance of the RADIUS infrastructure is to >>> deploy more servers (they don't have to by physical, you can install >>> multiple software instances on the same server if you have extra >>> CPUs or cores). >>> >>> We currently have 8 freeradius servers running on 4 physical >>> servers (2 per box). And we balance the wireless controllers >>> across those. >> >> As the university I used to work at, we were handling a similar load with two Xserve G5s, but we were using LDAP and not Kerberos. >> > > The LDAP module is multithreaded, right? That would give it an > advantage. Yes. > One other mistake we made was that our RADIUS server > hardware has many cores, but each core itself is quite slow - > this was before we found out that the freeradius krb5 module > was single threaded. If we'd known that earlier, we'd have instead > purchased machines with fewer faster cores. Additionally the control thread is also the only one that can insert packets into the request queue, though any worker can pick requests off it. >>> One of our staff members is also looking at patching freeradius >>> to multithread the Kerberos bits, but unless the freeradius >>> folks accept those patches and maintain them, we likely won't >>> deploy that option in production. >>> >> >> v2.1.x has gone into an unofficial feature freeze for anything not related to DHCP functionality. If you want this code to be included in function releases please submit the patches against the master branch (3.0). *future releases > Thanks! I'll pass this info along to our team! np -Arran ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Message from a.cudbardb@freeradius.org

> The easiest way to disable the cache is to set the environment > variable KRB5RCACHETYPE to "none" before starting freeradius. > The MIT Kerberos software on our RADIUS servers though is so > old (v1.3.x) that it didn't support this, so I had to disable > it by writing a patch to the source code (in rlm_krb5.c). If > you need that, let me know and I can send it. I think KRB5RCACHETYPE > appeared in v1.4. We've added an option to disable the cache to rlm_krb5, this will be available in 2.2.0, which should be released ~10/09/12. You will need to add 'cache = no' to your existing krb5 configuration file. In what will become 3.0 this is already set in the default configuration file. Thanks to Shumon Huque for testing. -Arran ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Ok, we all have different usage patters and number of users.  So can we do a quick check of what sort of authentications our servers are doing per second.  Yes this does not filter out failures and logs and.....  But at least it is an idea of how we stand to compared to others.

cat radius.log-[DATE] | tr -s " " | cut -d " " -f 4 | uniq -c | sort -n | tail -10

I did this for yesterday (first day of classes) and got the following.

     61 13:03:03
     62 13:01:03
     62 13:05:03
     62 14:50:11
     64 11:29:29
     64 12:50:13
     65 12:47:03
     65 12:50:08
     65 15:59:33
     68 13:02:58

Wondering what others get.  Thanks.


------------------------
Walter Reynolds
Principal Systems Security Development Engineer
Information and Technology Services
University of Michigan
(734) 615-9438


16 19:11:44 18 04:36:17 18 04:43:12 18 05:45:12 18 06:26:13 18 07:22:07 18 08:18:46 20 01:58:49 20 03:28:29 23 03:46:02
Message from dannyeaton@rice.edu

Here at Rice -bash-3.00$ cat today | tr -s " " | cut -d " " -f 4 | uniq -c | sort -n | tail -10 65 net3 68 net3 72 net3 74 net3 74 net3 76 net3 76 net3 78 net3 82 net3 107 net3 -----Original Message----- From: The EDUCAUSE Wireless Issues Constituent Group Listserv [mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of John Rodkey Sent: Wednesday, September 05, 2012 10:49 AM To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question 16 19:11:44 18 04:36:17 18 04:43:12 18 05:45:12 18 06:26:13 18 07:22:07 18 08:18:46 20 01:58:49 20 03:28:29 23 03:46:02
Message from craigsimons@sfu.ca

Using HiPath/Radiator Radius. Today is the first real day of classes though, so I would expect things to go higher.

  34 ns-ryu.its.sfu.ca
  34 ns-ryu.its.sfu.ca
  35 ns-ryu.its.sfu.ca
  36 ns-ryu.its.sfu.ca
  40 ns-ryu.its.sfu.ca
  41 ns-ryu.its.sfu.ca
  42 ns-ryu.its.sfu.ca
  45 ns-ryu.its.sfu.ca
  47 ns-ryu.its.sfu.ca
  50 ns-ryu.its.sfu.ca

Regards,
 Craig

SFU SIMON FRASER UNIVERSITY
Network Services

Craig Simons
Network and Systems Administrator

Phone: 778-782-8036
Cell: 604-649-7977
Email: craigsimons@sfu.ca
Twitter: simonscraig


From: "Danny Eaton" <dannyeaton@rice.edu>
To: WIRELESS-LAN@listserv.educause.edu
Sent: Wednesday, 5 September, 2012 09:09:47
Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question

Here at Rice

-bash-3.00$ cat today | tr -s " " | cut -d " " -f 4 | uniq -c | sort -n |
tail -10
     65 net3
     68 net3
     72 net3
     74 net3
     74 net3
     76 net3
     76 net3
     78 net3
     82 net3
    107 net3


-----Original Message-----
From: The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of John Rodkey
Sent: Wednesday, September 05, 2012 10:49 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question

     16 19:11:44
     18 04:36:17
     18 04:43:12
     18 05:45:12
     18 06:26:13
     18 07:22:07
     18 08:18:46
     20 01:58:49
     20 03:28:29
     23 03:46:02


Message from neil-johnson@uiowa.edu

We run RADIATOR and just had to add additional servers to handle the load. 


-Neil

-- 
Neil Johnson
Network Engineer
The University of Iowa
Phone: 319 384-0938
Fax: 319 335-2951
Mobile: 319 540-2081
E-Mail: neil-johnson@uiowa.edu


From: Craig Simons <craigsimons@sfu.ca>
Reply-To: Craig Simons <craigsimons@sfu.ca>
Date: Wednesday, September 5, 2012 11:45 AM
To: "WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU" <WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU>
Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question

Using HiPath/Radiator Radius. Today is the first real day of classes though, so I would expect things to go higher.

  34 ns-ryu.its.sfu.ca
  34 ns-ryu.its.sfu.ca
  35 ns-ryu.its.sfu.ca
  36 ns-ryu.its.sfu.ca
  40 ns-ryu.its.sfu.ca
  41 ns-ryu.its.sfu.ca
  42 ns-ryu.its.sfu.ca
  45 ns-ryu.its.sfu.ca
  47 ns-ryu.its.sfu.ca
  50 ns-ryu.its.sfu.ca

Regards,
 Craig

SFU SIMON FRASER UNIVERSITY
Network Services

Craig Simons
Network and Systems Administrator

Phone: 778-782-8036
Cell: 604-649-7977
Email: craigsimons@sfu.ca
Twitter: simonscraig


From: "Danny Eaton" <dannyeaton@rice.edu>
To: WIRELESS-LAN@listserv.educause.edu
Sent: Wednesday, 5 September, 2012 09:09:47
Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question

Here at Rice

-bash-3.00$ cat today | tr -s " " | cut -d " " -f 4 | uniq -c | sort -n |
tail -10
     65 net3
     68 net3
     72 net3
     74 net3
     74 net3
     76 net3
     76 net3
     78 net3
     82 net3
    107 net3


-----Original Message-----
From: The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of John Rodkey
Sent: Wednesday, September 05, 2012 10:49 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question

     16 19:11:44
     18 04:36:17
     18 04:43:12
     18 05:45:12
     18 06:26:13
     18 07:22:07
     18 08:18:46
     20 01:58:49
     20 03:28:29
     23 03:46:02


Message from ssmith@siu.edu

This list is one of several we have, and isn't our primary one.  I find it interesting, as I would have expected much higher numbers.  We have over 2,000 concurrent connections at any given moment throughout the day.

     12 09:57:11
     12 10:00:01
     12 10:14:36
     12 15:27:55
     13 10:01:18
     13 10:02:26
     13 10:35:30
     14 09:52:31
     14 10:37:31                                                                                                                                            
     15 16:51:20

Craig Simons wrote:
Using HiPath/Radiator Radius. Today is the first real day of classes though, so I would expect things to go higher.

  34 ns-ryu.its.sfu.ca
  34 ns-ryu.its.sfu.ca
  35 ns-ryu.its.sfu.ca
  36 ns-ryu.its.sfu.ca
  40 ns-ryu.its.sfu.ca
  41 ns-ryu.its.sfu.ca
  42 ns-ryu.its.sfu.ca
  45 ns-ryu.its.sfu.ca
  47 ns-ryu.its.sfu.ca
  50 ns-ryu.its.sfu.ca

Regards,
 Craig

SFU SIMON FRASER UNIVERSITY
Network Services

Craig Simons
Network and Systems Administrator

Phone: 778-782-8036
Cell: 604-649-7977
Email: craigsimons@sfu.ca
Twitter: simonscraig


From: "Danny Eaton" <dannyeaton@rice.edu>
To: WIRELESS-LAN@listserv.educause.edu
Sent: Wednesday, 5 September, 2012 09:09:47
Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question

Here at Rice

-bash-3.00$ cat today | tr -s " " | cut -d " " -f 4 | uniq -c | sort -n |
tail -10
     65 net3
     68 net3
     72 net3
     74 net3
     74 net3
     76 net3
     76 net3
     78 net3
     82 net3
    107 net3


-----Original Message-----
From: The EDUCAUSE Wireless Issues Constituent Group Listserv
[mailto:WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of John Rodkey
Sent: Wednesday, September 05, 2012 10:49 AM
To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [WIRELESS-LAN] FreeRADIUS performance question

     16 19:11:44
     18 04:36:17
     18 04:43:12
     18 05:45:12
     18 06:26:13
     18 07:22:07
     18 08:18:46
     20 01:58:49
     20 03:28:29
     23 03:46:02


 

  That is a fun exercise.  Here we are for yesterday September 4th.  We had load issues last semester with the addition of tons of wireless, but we scaled up to get ahead of it (all vmware).  We seem to be purring along this semester (at least AAA, NAC, wireless-wise).  I have been wanting to graph the Freeradius auths via Zenoss, but the project hasn’t made it to the top of the pile yet.

 

     98 10:54:22

     98 13:51:08

     98 13:51:20

    100 12:20:46

    100 13:52:25

    105 13:52:49

    107 12:09:18

    111 12:18:21

    114 10:56:28

    146 12:18:22

 

  Adam Ferrero

  adam@temple.edu

  Temple University

  Awesome at wireless?  We want you on our team!

  https://hospats.adminsvc.temple.edu/CSS_External/CSSPage_Referred.ASP?Req=TU-15524

 

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Message from shuque@upenn.edu

On Wed, Sep 05, 2012 at 10:43:25AM -0400, Walter Reynolds wrote: > Ok, we all have different usage patters and number of users. So can we do > a quick check of what sort of authentications our servers are doing per > second. Yes this does not filter out failures and logs and..... But at > least it is an idea of how we stand to compared to others. > > cat radius.log-[DATE] | tr -s " " | cut -d " " -f 4 | uniq -c | sort -n | > tail -10 Can you provide some sample lines from your radius.log, that this command line is intended to process? I'll have our RADIUS ops folks get our numbers, but I want to make sure we're measuring the same thing. We use syslog and customized log entries, so .... -- Shumon Huque University of Pennsylvania. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Message from rgc@buffalo.edu

>
Message from shuque@upenn.edu

On Thu, Sep 06, 2012 at 01:19:11PM -0400, Colantuoni, Robert wrote: > >
Message from shuque@upenn.edu

On Wed, Sep 05, 2012 at 02:34:35PM +0100, Arran Cudbard-Bell wrote: > > The easiest way to disable the cache is to set the environment > > variable KRB5RCACHETYPE to "none" before starting freeradius. > > The MIT Kerberos software on our RADIUS servers though is so > > old (v1.3.x) that it didn't support this, so I had to disable > > it by writing a patch to the source code (in rlm_krb5.c). If > > you need that, let me know and I can send it. I think KRB5RCACHETYPE > > appeared in v1.4. > > > We've added an option to disable the cache to rlm_krb5, this will be available in 2.2.0, which should be released ~10/09/12. > > You will need to add 'cache = no' to your existing krb5 configuration file. > > In what will become 3.0 this is already set in the default configuration file. > > Thanks to Shumon Huque for testing. > > -Arran Thanks again for the patch! There is one other point about freeradius and Kerberos performance that I forgot to make earlier, which may be of interest to other sites. MIT Kerberos client code in some circumstances makes a ridiculous number of serial DNS queries. Combined with the fact the the freeradius krb5 module is single threaded, this can pose a significant bottleneck. This might be true of Heimdal as well, but I haven't checked. When I was analyzing performance issues a while back on a new server we were bringing up, the RADIUS server was doing about 42 DNS queries for every authentication attempt (which translated to roughly 56ms of additional latency). Specifically in our environment (3 KDCs advertised in the DNS), 6 A, 24 AAAA, 4 SRV, 8 TXT. These are for KDC and realm discovery and lookups of address records for each server, and nxdomain responses for the AAAA lookups causing follow-on lookups via search list appending. You can get rid of almost all of this by tweaking krb5.conf to hardcode KDC names or addresses and realms. We didn't want to do that, so we stripped down the DNS domain search list and got rid of AAAA lookups (we don't support IPv6 on the KDCs yet) etc. We also deployed local caching-only DNS servers. -- Shumon Huque University of Pennsylvania. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Close
Close


Annual Conference
September 29–October 2
View Proceedings

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2015 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.