Main Nav

Message from vicutrone@vassar.edu

Hello, I am running multiple Cisco 4404 controllers Vers-7.0.116 and the majority of the WAPS are CAPWAP 1131's . The issue I seem to be having is that lately when I reboot an access layer switch, POE, 3750 series, with WAPS attached, I lose some of the WAPS. The WAPS never come back on line again and appear to be dead. They were completely operational prior to the reboot, but after reloading the switch, no switch configuration changes, the WAPS never come back on line. I have shut/no shut the interfaces of the switch, and the switches POST seems fine. Just wondering if anyone else has seen this issue, I have lose about 30 WAPS so far. Thanks!! Vikki Cutrone Network Admin Vassar College ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Comments

Are the 3750's the new "X" model or something older? Are you sure you're not running out of power allocation on the switch... that is, the initial call for power across all the ports is higher than the unit can supply, so the switch is leaving some ports down?
 
Jeff

>>> On Wednesday, January 11, 2012 at 11:39 AM, in message <4F0DE572.3000907@vassar.edu>, Vikki Cutrone <vicutrone@VASSAR.EDU> wrote:
Hello,  I am running multiple Cisco 4404 controllers Vers-7.0.116 and
the majority of the WAPS are CAPWAP 1131's .  The issue I seem to be
having is that lately when I reboot an access layer switch, POE, 3750
series, with WAPS attached, I lose some of the WAPS.  The WAPS never
come back on line again and appear to be dead.  They were completely
operational prior to the reboot, but after reloading the switch, no
switch configuration changes, the WAPS never come back on line.  I have
shut/no shut the interfaces of the switch, and the switches POST seems
fine.  Just wondering if anyone else has seen this issue, I have lose
about 30 WAPS so far.  Thanks!!

Vikki Cutrone
Network Admin
Vassar College

**********
Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Message from vicutrone@vassar.edu

Hi,
No the 3750's are the older model and are not fully populated with WAPS, ( about 18 total on a 48 port POE) so I don't think I am running out of power, also after the switch is up if you show int they appear disconnected down/down, attempt a no shut and they never come back on line.
Thanks

Vikki

On 1/11/2012 3:06 PM, Jeffrey Sessler wrote:
Are the 3750's the new "X" model or something older? Are you sure you're not running out of power allocation on the switch... that is, the initial call for power across all the ports is higher than the unit can supply, so the switch is leaving some ports down?
 
Jeff

>>> On Wednesday, January 11, 2012 at 11:39 AM, in message <4F0DE572.3000907@vassar.edu>, Vikki Cutrone <vicutrone@VASSAR.EDU> wrote:
Hello,  I am running multiple Cisco 4404 controllers Vers-7.0.116 and
the majority of the WAPS are CAPWAP 1131's .  The issue I seem to be
having is that lately when I reboot an access layer switch, POE, 3750
series, with WAPS attached, I lose some of the WAPS.  The WAPS never
come back on line again and appear to be dead.  They were completely
operational prior to the reboot, but after reloading the switch, no
switch configuration changes, the WAPS never come back on line.  I have
shut/no shut the interfaces of the switch, and the switches POST seems
fine.  Just wondering if anyone else has seen this issue, I have lose
about 30 WAPS so far.  Thanks!!

Vikki Cutrone
Network Admin
Vassar College

**********
Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Did you try a per port “power inline never” then “power inline auto”? simple “no shut” does line protocol but not power…

 

-Lee Badman

 

Message from dannyeaton@rice.edu

We had a similar problem, with the old 1000 series LWAPP APs (we’re replacing these, finally).  We had to physically unplug the ethernet cable from the AP.  In some cases, we “replaced” the APs with onhand spares, tested them in a known good location/switchport/cable, and then redeployed them as necessary.  This was with 4.2.209.0, on Cisco 3750’s with 12.2(33)SXI3 code. 

 

Message from dlpita@fiu.edu

Do a "show power inline | i AP", this will filter out the APs on the power inline output. Verify that they are all there, then check cdp details on each port to make sure they have an IP. David Pita Wireless Network Engineer UTS Network Management Services Florida International University 305.348.3658 On 01/11/2012 02:39 PM, Vikki Cutrone wrote: > Hello, I am running multiple Cisco 4404 controllers Vers-7.0.116 and > the majority of the WAPS are CAPWAP 1131's . The issue I seem to be > having is that lately when I reboot an access layer switch, POE, 3750 > series, with WAPS attached, I lose some of the WAPS. The WAPS never > come back on line again and appear to be dead. They were completely > operational prior to the reboot, but after reloading the switch, no > switch configuration changes, the WAPS never come back on line. I > have shut/no shut the interfaces of the switch, and the switches POST > seems fine. Just wondering if anyone else has seen this issue, I have > lose about 30 WAPS so far. Thanks!! > > Vikki Cutrone > Network Admin > Vassar College > > ********** > Participation and subscription information for this EDUCAUSE > Constituent Group discussion list can be found at > http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Vikki,

After the reboot are the APs still drawing power, just not showing a link light on the switch? Have you tried plugging a console cable into an AP when you do a reboot on the 3560 to see what messages the AP returns?

I have seen an issue on our 3560's before, and the condition was related to low power, though the switch did not throw a log message until I upgraded the firmware to a newer version. We had not exceeded the number of allowable APs/wattage on the switch, but we had undersized the UPS that the switch was connected to. Essentially when the switch rebooted and attempted to power on all of the APs, it would overload the UPS, cause a slight low voltage situation and so the switch would disable all POE power.



Message from vicutrone@vassar.edu

Hi, They are down/down never to come on line again. This is a show power inline-- Fa1/0/33 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/34 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/35 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/36 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/37 auto on 15.4 Ieee PD 3 15.4 Fa1/0/38 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/39 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/40 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/41 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/42 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/43 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/44 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/45 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/46 auto on 15.4 Ieee PD 3 15.4 Fa1/0/47 auto on 15.4 Ieee PD 3 15.4 Module Available Used Remaining (Watts) (Watts) (Watts) ------ --------- -------- --------- 1 370.0 231.0 139.0 2 n/a n/a n/a 3 n/a n/a n/a 4 n/a n/a n/a Did a power inline never power inline auto-- Fa1/0/33 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/34 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/35 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/36 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/37 auto on 15.4 Ieee PD 3 15.4 Fa1/0/38 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/39 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/40 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/41 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/42 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/43 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/44 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/45 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/46 auto on 15.4 Ieee PD 3 15.4 Fa1/0/47 auto on 15.4 Ieee PD 3 15.4 Samm result On 1/11/2012 3:32 PM, David Pita wrote: > Do a "show power inline | i AP", this will filter out the APs on the > power inline output. Verify that they are all there, then check cdp > details on each port to make sure they have an IP. > > David Pita > Wireless Network Engineer > UTS Network Management Services > Florida International University > 305.348.3658 > > > On 01/11/2012 02:39 PM, Vikki Cutrone wrote: >> Hello, I am running multiple Cisco 4404 controllers Vers-7.0.116 and >> the majority of the WAPS are CAPWAP 1131's . The issue I seem to be >> having is that lately when I reboot an access layer switch, POE, 3750 >> series, with WAPS attached, I lose some of the WAPS. The WAPS never >> come back on line again and appear to be dead. They were completely >> operational prior to the reboot, but after reloading the switch, no >> switch configuration changes, the WAPS never come back on line. I >> have shut/no shut the interfaces of the switch, and the switches POST >> seems fine. Just wondering if anyone else has seen this issue, I >> have lose about 30 WAPS so far. Thanks!! >> >> Vikki Cutrone >> Network Admin >> Vassar College >> >> ********** >> Participation and subscription information for this EDUCAUSE >> Constituent Group discussion list can be found at >> http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
For clarity- the power commands I referenced are port by port, not global. -Lee
Message from ceyre@mtroyal.ca

Vicki, If I remember correctly the power module in the cisco switch controls ports in groups of 4. Try and move one of the dead access points to a port in a group of 4 that does NOT already have an access point in the group. I've seen issues like this before but not on a 3560. Regards, Craig Eyre Network Analyst IT Services Department Mount Royal University 4825 Mount Royal Gate SW Calgary AB T2P 3T5 P. 403.440.5199 E. ceyre@mtroyal.ca "The difference between a successful person and others is not a lack of strength, not a lack of knowledge, but rather in a lack of will." Vincent T. Lombardi From: Vikki Cutrone To: WIRELESS-LAN@LISTSERV.EDUCAUSE.EDU Date: 01/11/2012 01:50 PM Subject: Re: [WIRELESS-LAN] WAPS seem to die after switch reboot Sent by: The EDUCAUSE Wireless Issues Constituent Group Listserv Hi, They are down/down never to come on line again. This is a show power inline-- Fa1/0/33 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/34 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/35 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/36 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/37 auto on 15.4 Ieee PD 3 15.4 Fa1/0/38 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/39 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/40 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/41 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/42 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/43 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/44 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/45 auto on 12.2 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/46 auto on 15.4 Ieee PD 3 15.4 Fa1/0/47 auto on 15.4 Ieee PD 3 15.4 Module Available Used Remaining (Watts) (Watts) (Watts) ------ --------- -------- --------- 1 370.0 231.0 139.0 2 n/a n/a n/a 3 n/a n/a n/a 4 n/a n/a n/a Did a power inline never power inline auto-- Fa1/0/33 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/34 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/35 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/36 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/37 auto on 15.4 Ieee PD 3 15.4 Fa1/0/38 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/39 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/40 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/41 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/42 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/43 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/44 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/45 auto on 15.4 AIR-LAP1131AG-A-K9 3 15.4 Fa1/0/46 auto on 15.4 Ieee PD 3 15.4 Fa1/0/47 auto on 15.4 Ieee PD 3 15.4 Samm result On 1/11/2012 3:32 PM, David Pita wrote: > Do a "show power inline | i AP", this will filter out the APs on the > power inline output. Verify that they are all there, then check cdp > details on each port to make sure they have an IP. > > David Pita > Wireless Network Engineer > UTS Network Management Services > Florida International University > 305.348.3658 > > > On 01/11/2012 02:39 PM, Vikki Cutrone wrote: >> Hello, I am running multiple Cisco 4404 controllers Vers-7.0.116 and >> the majority of the WAPS are CAPWAP 1131's . The issue I seem to be >> having is that lately when I reboot an access layer switch, POE, 3750 >> series, with WAPS attached, I lose some of the WAPS. The WAPS never >> come back on line again and appear to be dead. They were completely >> operational prior to the reboot, but after reloading the switch, no >> switch configuration changes, the WAPS never come back on line. I >> have shut/no shut the interfaces of the switch, and the switches POST >> seems fine. Just wondering if anyone else has seen this issue, I >> have lose about 30 WAPS so far. Thanks!! >> >> Vikki Cutrone >> Network Admin >> Vassar College >> >> ********** >> Participation and subscription information for this EDUCAUSE >> Constituent Group discussion list can be found at >> http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/. ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.
Message from caseb@purdue.edu

We're dealing with a similar issue right now too, but it seems to be AP-independent. We have a mix of Cisco 3500's and 1250's running on Cisco 3750EPs (running 12.2(53)SE2) and a sample of each type of AP experience the problem. Our 3750's are Gigabit so I've been using the 'test cable-diagnostic tdr' feature to see if there might be a cabling issue and in 95% of the cases it shows that pairs A and B are swapped between the local and remote ends. I'm still not sure how much I trust those results since they include 3 separate runs of cable (cross connect, infrastructure, and jumper at the AP) but it's still something. We figured out a workaround of sorts by hardcoding the ports to 1000M full duplex and it seems to work in most cases. For the rest, either reseating or replacing the cross connect has fixed it. In all cases, the tell-tale indicator is a value of 'IeeePD' in the Device field of a 'show power inline | ex off'. I've seen that happen whether or not the line protocol on the port is up, oddly enough. I've got an email out to Cisco to see if this might be some undocumented bug or...what. If I get anything worthwhile back I'll share. Thanks, -- Brandon Case, CCNA Network Engineer, ITaP Purdue University caseb@purdue.edu Office: (765)49-67096 Mobile: (765)479-7597 Fax: (765)49-46620 PGP Fingerprint: 99CB 02D6 983C 1E2A 015F 205C C7AA E985 A11A 1251
Message from vicutrone@vassar.edu

I am running IOS 12.2(55)SE3 on everything, and all of my 3560's seem to be fine, problem is with the 3750's. I do get the Ieee PD, show power inline- I can do a port by port power inline never/auto still stays stuck there. Vikki On 1/11/2012 3:56 PM, Case, Brandon J wrote: > We're dealing with a similar issue right now too, but it seems to be AP-independent. We have a mix of Cisco 3500's and 1250's running on Cisco 3750EPs (running 12.2(53)SE2) and a sample of each type of AP experience the problem. Our 3750's are Gigabit so I've been using the 'test cable-diagnostic tdr' feature to see if there might be a cabling issue and in 95% of the cases it shows that pairs A and B are swapped between the local and remote ends. I'm still not sure how much I trust those results since they include 3 separate runs of cable (cross connect, infrastructure, and jumper at the AP) but it's still something. We figured out a workaround of sorts by hardcoding the ports to 1000M full duplex and it seems to work in most cases. For the rest, either reseating or replacing the cross connect has fixed it. > > In all cases, the tell-tale indicator is a value of 'IeeePD' in the Device field of a 'show power inline | ex off'. I've seen that happen whether or not the line protocol on the port is up, oddly enough. I've got an email out to Cisco to see if this might be some undocumented bug or...what. If I get anything worthwhile back I'll share. > > Thanks, > -- > Brandon Case, CCNA > Network Engineer, ITaP > Purdue University > caseb@purdue.edu > Office: (765)49-67096 > Mobile: (765)479-7597 > Fax: (765)49-46620 > > PGP Fingerprint: > 99CB 02D6 983C 1E2A 015F 205C C7AA E985 A11A 1251 > >
Message from vicutrone@vassar.edu

Got it, just did it per port-- same thing On 1/11/2012 3:51 PM, Lee H Badman wrote: > For clarity- the power commands I referenced are port by port, not global. > > -Lee > > > >
Message from jarhelz@umich.edu

Are these 1131's LWAPP from the factory or did you convert these in the field from autonomous to LWAPP / CAPWAP? We have found that some of our converted 1230 series APs do this and show the same IEEE PD message on the switch. We had to go onsite or bring them back to the office to reload the autonomous image, convert it back to LWAPP, connect it to a controller and let them download the CAPWAP software version again. This would fix our issue which seemed to be a corrupted image. For the most part every failed AP image was preceded by some power event such as a power outage. -Jimmy James Helzerman Wireless Network Engineer University of Michigan ITS Communications Systems and Data Centers 4251 Plymouth Road, Building 2, #2224 Ann Arbor, Michigan 48105 Phone: 734-615-9541 Cell: 734-972-5095
Vikki,

Try this command at the port level:  power inline delay shutdown 20 initial 300 


Peter

Message from bosborne@liberty.edu

Please do NOT try that command.

 

The article is for Cisco PoE which predates the standard 802.3af PoE that is on 3750 switches.

 

If I recall correctly, that command can cause issues if a non-PoE device is plugged into the port.

 

Bruce Osborne

Network Engineer

IT Network Services

 

(434) 592-4229

 

LIBERTY UNIVERSITY

40 Years of Training Champions for Christ: 1971-2011

 

From: Peter Redhead [mailto:redheadp@NEWSCHOOL.EDU]
Sent: Thursday, January 12, 2012 12:13 PM
Subject: Re: WAPS seem to die after switch reboot

 

Vikki,

 

Try this command at the port level:  power inline delay shutdown 20 initial 300 

 

 

Peter

Message from vicutrone@vassar.edu

Thanks!!

On 1/13/2012 9:09 AM, Osborne, Bruce W wrote:

Please do NOT try that command.

 

The article is for Cisco PoE which predates the standard 802.3af PoE that is on 3750 switches.

 

If I recall correctly, that command can cause issues if a non-PoE device is plugged into the port.

 

Bruce Osborne

Network Engineer

IT Network Services

 

(434) 592-4229

 

LIBERTY UNIVERSITY

40 Years of Training Champions for Christ: 1971-2011

 

From: Peter Redhead [mailto:redheadp@NEWSCHOOL.EDU]
Sent: Thursday, January 12, 2012 12:13 PM
Subject: Re: WAPS seem to die after switch reboot

 

Vikki,

 

Try this command at the port level:  power inline delay shutdown 20 initial 300 

 

 

Peter

Close
Close


Annual Conference
September 29–October 2
Register Now!

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.