Main Nav

Message from mark.duling@biola.edu

We've had a couple of DNS response time issues with our IB appliances lately that are puzzling. I upgraded from 6.5.3 to 6.8.1 a month ago with no issues. But once our student population hit full strength a few days ago we're seeing sporadic response time issues and high cpu utilization. Most of the time at least the incidents have been preceded by adding or removing records. We're working with support in tracking the issue, but it feels kinda like a bug to me so I thought I'd ask if anyone else is running 6.8. I don't think we're under spec on this box at least on paper as far as I can tell, and it has always performed well in the past. Mark ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Comments

Do you have any performance metrics you can look at, such as number of queries per second?


--

Jeremy L. Gibbs
Systems Administrator / Network Engineer
Utica College IITS

T: (315) 223-2383
F: (315) 792-3814


Message from mark.duling@biola.edu

The GUI has a query per second graph, and I do snmp graphs with cacti too, and so it doesn't look like we've seen any big jump in queries. The GUI is showing less than 1k queries per second and the box is rated of 22k queries per second I think. Memory and disk use are low.
Message from mark.duling@biola.edu

Correction, the box is rated for 12k qps. We seem to be running under 2k qps.
Message from jemurray@zweck.net

We have been running 6.8.1 (upgraded from 6.6.8) for a month or so also.   We upgraded before the students came back from the summer break.  Our hardware is IB-1410's.   We have 4 DNS and 2 DHCP servers.   DNS is anycasted across all 4 servers, DHCP is load balanced between 2 servers.  

The load has gone up quite a bit since the students came back, but this is not causing any impact to performance. 

Do you have a separate grid master that you make all your changes from?



Here are some stats from one of our busiest DNS/DHCP servers:

Average DNS response time is ~1ms.
Average (daily) CPU: ~45%
Average (daily) Load: ~2.5
Average (daily) DNS queries/sec: ~500 (keep in mind anycast load balances this among all 4 servers)
Average (daily) DHCP queries/min: ~3000 (1/2 our DHCP traffic)


I am adding the weekly graphs inline to this message, I don't know if they will paste/attach properly (sorry for the HTML message).

































Hello, I am sure this is not much help but wanted to post our status. We are running 4 IB-1852As on version 6.5.8-195206. We have 1 as a grid master, a GM candidate, 3 run DNS servers, 2 run DHCP/DNS combined. All three DNS/DHCP run IPv6 and 4. Our students came back last week and we are not seeing any problems. In addition we do run a caching Bind Server that handles most of the DNS traffic from end-station clients. If you have a chance it would be great to get an update of what the problem and/or fix happens to be. Let us know if we can run any tests that may help. Thank you, Scott Kitterman UNH
Message from mark.duling@biola.edu

Thanks Jeremy and Scott.

Is that graph that displays system, user, and nice an Infoblox appliance or plain Unix/Bind?  Because I didn't think we get those exposed via SNMP from IB, though I could be wrong.  I only found a single CPU stat exposed by IB with SNMP and I guessed that was load average.

We don't actually have a grid, but an HA pair.  Yesterday I deleted an test DNS view when I learned that the total dns cache space is shared among them, increasing the cache available to the two live views we use.  No idea whether cache was an issue though.  We've had no performance problems since then, but then we've done almost no record changes and such as of yet that has seemed to kick off issues.

I guess it will take some time and experimentation to find out what the problem is/was.  We also turned of ddns and discovery too.  If the query levels weren't so low I guess I'd think our appliance was too small, but it just seems like we're way under the specs for it.

However this turns out I will update the list.



Hello,

 

The graphs are from Jeremy.  My guess is they are from SNMP.

 

At UNH we only use one view and our queries per second average anywhere from 130 to 200 per second on all three of the IB appliances running DNS.  We purchased the 1852-A model in anticipation of implementing DNSEC.  We used a smaller model appliance pair in testing.  When we enabled DNSEC in testing the database went over 150% which turned into a problem.

 

Our caching DNS servers are separate Linux OSes running Bind.  I only mentioned the Bind servers in the previous post to point out that much of our local clients do not rely on the Infoblox authoritative  DNS servers for name resolution.

 

 

-scott

UNH

 

 

From: The EDUCAUSE Network Management Constituent Group Listserv [mailto:NETMAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Mark Duling
Sent: Thursday, September 05, 2013 2:00 PM
To: NETMAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [NETMAN] Anyone running Infoblox NIOS 6.8.1?

 

Thanks Jeremy and Scott.

 

Is that graph that displays system, user, and nice an Infoblox appliance or plain Unix/Bind?  Because I didn't think we get those exposed via SNMP from IB, though I could be wrong.  I only found a single CPU stat exposed by IB with SNMP and I guessed that was load average.

 

We don't actually have a grid, but an HA pair.  Yesterday I deleted an test DNS view when I learned that the total dns cache space is shared among them, increasing the cache available to the two live views we use.  No idea whether cache was an issue though.  We've had no performance problems since then, but then we've done almost no record changes and such as of yet that has seemed to kick off issues.

 

I guess it will take some time and experimentation to find out what the problem is/was.  We also turned of ddns and discovery too.  If the query levels weren't so low I guess I'd think our appliance was too small, but it just seems like we're way under the specs for it.

 

However this turns out I will update the list.

 

 

Message from jemurray@zweck.net

If you are talking about the graphs I posted, they are all gathered via SNMP from the Infoblox servers.   We use CACTI to graph the data.   I believe I pulled the templates from this thread: http://forums.cacti.net/viewtopic.php?f=12&t=42129

Smokeping was used to get the RRT for the DNS queries.   


Have you worked with IB support to see if they have anyway of debugging your performance issues?   
-- Jason E. Murray Sr. Systems Engineer Washington University in St. Louis Phone: 314-935-4865 Email: jemurray@wustl.edu Web: http://nts.wustl.edu/~jemurray/



Message from mark.duling@biola.edu

Yes I've been working with them.  I put together my own Cacti template just for queries a while back, but I'll give the later one in the forum a try.  I'm interested in getting more detail now.


Message from jemurray@zweck.net

I was going to roll my own until I found this.   It was pretty painless, I would recommend it.   




Close
Close


Annual Conference
September 29–October 2
View Proceedings

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

Digital Badges
Member recognition effort
Earn yours >

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.