Vat of Knowledge

This post is about memory tests and memory graphing for Big Brother. Out of the box, BB 4.1 Professional tests memory fine on Windows boxes, but not Linux.

bb-memory 3.1

This external script will perform tests on memory for Linux, including physical memory, actual memory and swap memory. Download it here. Here are the instructions, pretty easy:

Linux

Big Brother Memory Test

This file details the installation of bb-memory to a standard *NIX
Big Brother installation. If you are installing bb-memory on a
Windows machine please see the file INSTALL.WIN

Installation
————

To install bb-memory into a standard *NIX Big Brother installation
perform the following steps:

1) Copy bb-memory.sh to the “ext” directory of your Big Brother installation

For HP-UX a helper-program is used to collect the memory information.
This program must be compiled:
a) Change into the source/hpux/ directory.
b) Type make
c) Copy bb-hp-memsz to $BBHOME/bin/

2) Add bb-memory.sh to the list of extensions running on this host. This
is done by adding “bb-memory.sh” to the host-entry in $BBHOME/etc/bb-bbexttab
and restarting Big Brother. If there is no host-entry in that file, you
simply add one like this:
localhost : : bb-memory.sh

3) Restart Big Brother. You should see a message about the bb-memory.sh
extension starting.

4) After a few minutes, a “memory” column will appear on your BBDISPLAY
server showing the memory status for this host.
Customization
————-

Within the bb-memory.sh script, there are a few variables that you may wish
to alter in order to better suit your environment. These customizeable values
are the warning levels for the various tests.

PERCENT_PHYS_WARN – Warning level for physical memory
PERCENT_PHYS_PANIC – Panic level for physical memory
PERCENT_ACT_WARN – Warning level for actual memory
PERCENT_ACT_PANIC – Panic level for actual memory
PERCENT_SWAP_WARN – Warning level for swap memory
PERCENT_SWAP_PANIC – Panic level for swap memory

I’m running RHES, so we need to go to around line 100 and make another edit. Change the line that reads

“case $BBOSTYPE in
linux | redhat | debian )”

to

“case $BBOSTYPE in
linux | redhat | debian | redhatES )”

Here is a link to the updated bb-memory.sh file.

Restart the client and memory tests appear in the mem column.

Windows

Big Brother Memory Test for Windows

This file details the installation of bb-memory on Windows. It should
work on all Win32 implementations, i.e. Win9x/ME/NT/2000/XP/2003.

Note that the bb_memory program will also pick up network statistics
and send these to Big Brother together with the memory statistics.
The network statistics can be trended by the LARRD add-on to Big
Brother, if you install the netstat-larrd-win32.patch patch. This
patch is against LARRD 0.43c.
Preparation
———–

bb-memory works in conjuction with the BBNT client. The BBNT client is
available from the Big Brother site at http://www.bb4.com. You must have
a working installation of the BBNT client before bb-memory will work.

Though not documented or specifically supported, the following directory
structure has been found to be reliable and easy to maintain:

C:\BB – Root of the Big Brother extensions
C:\BB\EXT – Extension programs (e.g. bb_memory.exe)
C:\BB\LOGS – Extension output directory
Installation
————

To install bb-memory onto a Windows machine, perform the following steps:

1) Copy bb-memory.exe to the desired location. (see above recommendation)

2) Using bbntcfg, configure the “Saved Logs” setting. This should be a
directory that is used only for Big Brother extensions. (C:\BB\LOGS
if you used the example above). This directory holds the results
from the extension scripts, until they have been sent off to the
BBDISPLAY server.

3) Add bb_memory.exe as an “external script”. In bbntcfg, click on the
“Add External” button and browse to the bb_memory.exe file (in
C:\BB\EXT\ if you use the directories above).

4) Restart the Big Brother service and view the results. A “memory”
column should appear on your BBDISPLAY server after a couple of
minutes.

Usage
—–

The warning values can be set at the command line invocation of the
external test. The command line switches and arguments are:

-p <PATH> – Set the PATH to place the output
(creates the file: <PATH>\memory)
bb_memory will normally determine this automatically.
To have the result shown on-screen, just
specify “-p” without any path.
-P “<WARN> <PANIC>” – Set WARN and PANIC values for physical memory.
-S “<WARN> <PANIC>” – Set WARN and PANIC values for paging memory.
-V “<WARN> <PANIC>” – Set WARN and PANIC values for virtual memory.
-F – Turn on “Fully Qualified Domain Names” (FQDN)

For the warning values, you will need to enclose them within quotes.

example:

* To run bb-memory using FQDN, directing output to C:\EXT_OUT, and using
the values 75 & 90 for physical memory run…

bb_memory.exe -F -p C:\EXT_OUT -P “75 90″

$Id: INSTALL.WIN,v 1.2 2004/09/28 21:23:04 henrik Exp $

The Windows client is necessary for LARRD graphing. The built in memory-larrd.pl seems to work with bb-memory-3.1, but does not work with the built in Big Brother memory test. This being the case, I want to disable my “mem” test in favor of the “memory” test that bb-memory-3.1 provides. I’m still finding out how to do this, and I will update this post when that information becomes available.

LARRD memory graphing

Another thing was I was unable to get the graphing to work properly. Thanks for this post over at http://support.bb4.com/archive/200704/msg00001.html

Also, here is a link to the updated file that has all the edits listed below already in it, memory-larrd.pl.

Thank you very much Philip, I've been trying to figure this out for a year
or so now. Finally my larrd page and my memory graphs work  

-----Original Message-----
From: owner-bb@bb4.com [mailto:owner-bb@bb4.com] On Behalf Of Philip Clark
Sent: Thursday, 29 March 2007 1:01 p.m.
To: bb@bb4.com
Subject: RE: {bb} LARRD and memory

On Wed, 2007-03-28 at 08:17 -0400, Calderon, Willy (NIH/NINDS) [C]
wrote:
> Thanks. I've taken your advice and gone with option B. I already get
> the memory (mem for Windows servers) button accumulate statistics.
> In bbdef-server.sh I've included the DATAMSG argument like so:
>
> RUNOPTS="EMBEDHTML","DATAMSG"
> export RUNOPTS

There's no need to do this if you are using scripts like bb-memory-3.1.
They use the "status" message type to transmit data. In any case,
multiple options here should be separated by spaces and only quoted
at the ends, so that would look like:

RUNOPTS="EMBEDHTML DATAMSG"

> I've added the bb-memory package from Deadcat on my SuSE and Red Hat
> ES 4 servers and it works. I've added "memory" in larrd-config.pl like
> so:
>
> @DATALIST=("la","disk","memory","bbnet","vmstat","netstat","procs",
> "users","up","WebApp")
>
> I've placed memory-larrd.pl in the ext directory and it's been added
> to the bb-bbextab list.

There's no need to do that. Larrd will run the script "auto-magically"
because of the DATALIST definition.

> After restarting Big Brother, I've checked LARRDOUT and find these
> errors coming up:
>
> /usr/local/larrd/memory-larrd.pl: ERROR creating /home/bbrother/bb/
> bbvar/rrd/donkeyhead.css.gov.real.rrd: you must define at least one
> Data Source
>
> Any ideas?

Actually, I gave installing this a shot and it turns out there are a
few problems with the memory-larrd.pl script. It looks more like a
"work in progress" that the finished article. Here is what I did to
get it working.

1. The variable $log is defined in the wrong scope so that, when it is
   referenced in a subroutine, it's empty. I added a line "my $log;"
   at the start of the script (just after the "# real work begins here"
   comment) and removed "my" from the line "my $log = $host;".

2. The "data source" is defined as a scalar and then referenced as an
   array. To match the other scripts, I change the definition to be
   an array too, so that the line:

     $DS="DS:realmempct:GAUGE:600:0:U";

   becomes:

     my @ds = ("DS:realmempct:GAUGE:600:0:U",
              );

3. The format of the RRD files it creates are out of step with the way
   the others are defined. It might not be strictly necessary to change
   this, but I added the line "$host =~ s/\,/\./g;" after the comment
   line "### create REAL-RRD if not exist and update data" and changed
   the two RRD definition lines to read like:

     $RRD="$ENV{'BBRRDS'}/$host.memory,real.rrd";
     $RRD="$ENV{'BBRRDS'}/$host.memory,swap.rrd";

And now it works  

When it comes to Windows machines, it appears there is another choice
to be made. You can either install the external executable that comes
with the package (which should provide the required data in a format
that the script can handle), or modify memory-larrd.pl so that it will
handle the default data returned by the standard client.

That said, I don't have any Windows clients to play with, so it's
a little difficult to debug.

Cheers, Phil.

Also not sure if this is 100% necessary, but I edited the following line in /usr/local/larrd/larrd-config.pl to include the memory test like:

@DATALIST=(“la”,”memory“,”disk”,”bbnet”,”vmstat”,”netstat”,”apache”,”temperature”,”iostat”,”procs”,”users”,”up”,”WebApp”);

Edit 04/08/10:

This helps to get the actual memory test into the graph:

Patch file is below.  Should be as simple as grabbing
larrd-graphs-1.5.tar.gz from deadcat.net and installing it, then

cd LARRDHOME
patch memory-larrd.sh <memory-larrd.sh.patch

Hope that helps. Good luck.

Tom

--- memory-larrd.sh.orig	2004-10-27 08:30:47.000000000 -0400
+++ memory-larrd.sh	2004-10-27 09:14:20.000000000 -0400
@@ -85,7 +85,10 @@
 	# pearcec added this line to match how larrd likes the rrds 11/9/2000
 	# I need to fix this
 	#HOST=`echo $HOST | $SED 's/,/\.//'`
-	RRD=$BBRRDS/$HOST.memory.real.rrd
+	# Changed RRD filename contents from "real" to "physical"
+	# so the plot name on the graph  matches bb-memory.sh
+	# output. Tom Georgoulias 10/27/04
+	RRD=$BBRRDS/$HOST.memory.physical.rrd

 	# if it doesn't exist, make it
 	if [ ! -f $RRD ]
@@ -161,6 +164,52 @@
 	fi
         $RRDTOOL update $RRD $TS:$VALUE

+# add support to graph "Actual" memory as well.
+# I just copied the section above for swap and edited the filename
+# for RRD, plus the HEAD line to grab the actual values.
+# Since I'm using larrd-graphs.cgi v1.5 from deadcat.net, I didn't
+# need to make any edits to that script.
+# Tom Georgoulias 10/27/04
+RRD=$BBRRDS/$HOST.memory.actual.rrd
+

+

+ # if it doesn't exist, make it
+ if [ ! -f $RRD ]
+ then
+ # $RRAS is defined in larrd.conf
+ $RRDTOOL create $RRD DS:realmempct:GAUGE:600:0:U $RRAS
+

+

+ if [ $STATUS -eq 1 ]
+ then
+ echo "$0: STATUS did not find $RRD, created."
+ fi
+ fi
+

+ # RRD acquisition co>plete, proceed to TS
+

+ # i am lazy right now and just using current -gra
+ # getting epoch from a file -grapht>p is a pain
+ # so the -grapht>p is from when the data is gathered
+ # not when it actually happened, ugh
+ TS=N
+ # TS=acquisition complete, proceed to VALUE
+

+ # now lets get the data
+ # look through the file and get what you want
+ # e.g. load average
+ VALUE=`$HEAD -4 $log | $EGREP 'Actual' |$AWK '{print $5}' |$SED 's/%//g'`
+ #VALUE=`$HEAD -4 $log | $GREP Swap |$AWK '{print $5}' |$SED 's/%//g'`
+ # VALUEoacquired, do the job
+

+ if [ $DEBUG -eq 1 ]
+ then
+ echo "$0: DEBUG $RRDTOOL update $RRD $TS:$VALUE"
+ fi
+ $RRDTOOL update $RRD $TS:$VALUE
+

+
+
 done

# NT logs the memory usage in .cpu