Mark's blog - Comments

David: Linux iostat monitoring with Cacti

nospam@example.com (David) — Tue, 16 Dec 2008 03:04:32 +0000

What's the realistic limit for number of devices? I'm trying to use this on a number of machines with over 1000 scsi devices attached to each of them. When running the snmpwalk manually, it does return proper data, albeit quite slowly. Attempting to use cacti to add the query as a graph returns nothing. Cacti complains about an snmp error:

+ Running data query [10].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.3.1'
+ No SNMP data returned
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/var/www/cacti/resource/snmp_queries/iostat.xml'

Any hints?

asdf: ZFS Replication

nospam@example.com (asdf) — Thu, 11 Dec 2008 21:28:02 +0000

use netcat instead of SSH as a transport for zfs send/receive speed improvement. (if your link is secure and local in your data center of course.)

Travis: Apache mod_proxy balancing with PHP sticky sessions

nospam@example.com (Travis) — Thu, 20 Nov 2008 20:09:38 +0000

Hmm, this might work. But the problem is that Apache is just responding to what it gets from the user's cookie value, which is set to the dead server. I need a way to do something different (i.e., go to the non-dead server) BEFORE the connection attempt is made. If maxattempts is set to 1, it's still going to try at least once on the bad server PER REQUEST, which will fail each time. Does that make sense (and am I correct?)

Since I wrote this post, I was also messing around with other load balancing options, so forgive me if my explanation is a bit incorrect, since I'm trying to put myself back into this configuration to fully understand it...

Andrew Nicols: ZFS Replication

nospam@example.com (Andrew Nicols) — Thu, 20 Nov 2008 10:45:14 +0000

An alternative, or an addition to mounting read only is to force a rollback of the file system:
zfs receive -F
From the zfs man page:

-F Force a rollback of the file system to the most
recent snapshot before performing the receive
operation. If receiving an incremental replication
stream (for example, one generated by "zfs send -R
-[iI]"), destroy snapshots and file systems that
do not exist on the sending side.

I use zfs send/receive to send hourly incremental snapshots between two X4500s. The issue we've come across is the time it takes to send the snapshots between servers.
We keep snapshots every hour for a month, and then every day until we run out of disk space (So far 9 months-worth). On our backup thumper, we've not destroyed any snapshots for some time and have about 3 months worth of hourly snapshots.

Brian Kearney: Apache mod_proxy balancing with PHP sticky sessions

nospam@example.com (Brian Kearney) — Thu, 30 Oct 2008 05:18:26 +0000

I think Apache 2.2 does natively support what you are trying to accomplish.

I see that maxattempts and nofailover parameters to the ProxyPass directive are described on this page of the HTTPD configuration site:

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

Does that look like what you need?

Mark Round: Linux iostat monitoring with Cacti

nospam@example.com (Mark Round) — Mon, 27 Oct 2008 21:02:24 +0000

Aha! Glad you worked it out

Typically, my hosts only have around 10-20 multipathed devices from the SAN.

I'll put some test cases together with this (and play with different partition table types etc.), and see what I can come up with. Thanks again for the comments!

Rael: Linux iostat monitoring with Cacti

nospam@example.com (Rael) — Mon, 27 Oct 2008 16:44:36 +0000

Yea, that made the difference.

Your scripts won't work well on a enterprise class host with many redundant path's, et al.

It works just fine returning results on smaller hosts.

How could you modify this script to return only valid paths and exclude disks with no valid partition tables?

IE: fdisk on dm-176 may return

Disk /dev/dm-176 doesn't contain a valid partition table

Rael: Linux iostat monitoring with Cacti

nospam@example.com (Rael) — Mon, 27 Oct 2008 16:34:15 +0000

This machine has well over 700 disks being presented to it. Perhaps this is an issue?

Mark Round: Linux iostat monitoring with Cacti

nospam@example.com (Mark Round) — Mon, 27 Oct 2008 14:44:06 +0000

Well, that's weird...

The verbose query (and the "Success [0 Items, 0 Rows]" bit) makes it appear as though no valid data is coming back.

You should have a list of indexes and descriptions in the verbose query, e.g.

+ Executing SNMP walk for data @ '.1.3.6.1.3.1.1'
+ Found item [ioIndex='1'] index: 1 [from value]
+ Found item [ioIndex='2'] index: 2 [from value]
+ Found item [ioIndex='3'] index: 3 [from value]
+ Found item [ioIndex='4'] index: 4 [from value]
+ Found item [ioIndex='5'] index: 5 [from value]
+ Found item [ioIndex='6'] index: 6 [from value]
+ Found item [ioIndex='7'] index: 7 [from value]
+ Found item [ioIndex='8'] index: 8 [from value]
...
... and so on...
...

But you say a SNMP walk provides valid data ? Are there perhaps any strange characters in your device names ? How many devices are there that you're trying to poll ?

Rael: Linux iostat monitoring with Cacti

nospam@example.com (Rael) — Mon, 27 Oct 2008 14:16:57 +0000

Cactiv: 0.8.7b

No, I'm not getting far enough to generate an RRD.

When I add the query to the device ir shows:

1) iostat - Queue size (Verbose Query) Uptime Goes Backwards Success [0 Items, 0 Rows]

I just added the Queue Size one first to see if it worked at all.

A verbose query execution returns:

"+ Running data query [22].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/proj/cacti/cacti-0.8.7b/resource/snmp_queries/iostat.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.3.1'
+ Located input field 'ioIndex' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.3.1.1'
+ Located input field 'ioDescr' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.3.1.2'
+ Found data query XML file at '/proj/cacti/cacti-0.8.7b/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/proj/cacti/cacti-0.8.7b/resource/snmp_queries/iostat.xml'
+ Found data query XML file at '/proj/cacti/cacti-0.8.7b/resource/snmp_queries/iostat.xml'
"

The snmpwalk you asked for returns valid data:

root@mivl02:/tmp > snmpwalk -v 2c -c CENSOR miux77 .1.3.6.1.3.1.13
SNMPv2-SMI::experimental.1.13.1 = STRING: "0.34"
SNMPv2-SMI::experimental.1.13.2 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.3 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.4 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.5 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.6 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.7 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.8 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.9 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.10 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.11 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.12 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.13 = STRING: "0.50"
SNMPv2-SMI::experimental.1.13.14 = STRING: "0.70"
SNMPv2-SMI::experimental.1.13.15 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.16 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.17 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.18 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.19 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.20 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.21 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.22 = STRING: "0.00"
SNMPv2-SMI::experimental.1.13.23 = STRING: "0.00"

Mark Round: Linux iostat monitoring with Cacti

nospam@example.com (Mark Round) — Mon, 27 Oct 2008 13:35:28 +0000

Hrm. What version of Cacti are you using ? Are the RRD files for the graphs being generated ?

Also, what happens when you do a SNMP walk for one of the data columns - for instance .1.3.6.1.3.1.13 for the utilisation ?

Simone: ZFS Replication

nospam@example.com (Simone) — Fri, 24 Oct 2008 15:19:43 +0000

Hi,

When you write:

"Now, I'll create a ZFS filesystem and add something to it", the command that worked for me is:

[root@solaris]$ zfs create master/data

Not zpool create master/data

Cheers

Rael: Linux iostat monitoring with Cacti

nospam@example.com (Rael) — Thu, 23 Oct 2008 15:30:19 +0000

So, this is exactly what I need. I've followed your steps to the letter and snmpwalk returns good data, however when I go to add the graphs the query is returning zero results.

Where could I be having my problem?

Mark Round: Apache mod_proxy balancing with PHP sticky sessions

nospam@example.com (Mark Round) — Mon, 20 Oct 2008 12:20:08 +0000

The only way I found to do this (it would be great if Apache natively supported it!) is to monitor the backend servers with something like Nagios. When a server goes down, an event handler removes that from the proxy pool and performs a graceful restart.

Alternatively, you could write a script that uses the balancer-manager page to dynamically add/remove backend servers.

Travis: Apache mod_proxy balancing with PHP sticky sessions

nospam@example.com (Travis) — Mon, 20 Oct 2008 10:23:13 +0000

Great post. I got this working, however, I have run into a practical issue.

If you have Apache running as a load balancer with two web nodes, and the user gets a cookie associating them with www1, there is an issue when www1 goes offline. It seems that Apache is smart enough to know that www1 is offline and requests should now go to www2, but that only happens with fresh requests. The previous user with the cookie pointing to www1 is now just getting sent to the www1 server, which throws them a Server Unavailable message (via the load balancer). Any way around this? I guess I would be looking for a way to check that a server is up before Apache uses that balance member. Thoughts?

-Travis