Wednesday, August 6, 2008

MySQL Cacti Templates finally working

After days of configuration, my Cacti MySQL template finally working.
At first, after I loaded MySQL template, the graphy is not updated by poller. It turns out MySQL template require spine poller than regular CMD poller. The spine poller has a bug , you will likely to get following error:

08/06/2008 12:20:02 PM - SPINE: Poller[0] ERROR: A database insert failed! Error:'1064', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (3,'mem_buffers','2008-08-06 12:20:01','70456'),(4,'mem_swap','2008-08-06 12:20:01','8388376'),(5,'','2008-08-06 12:20:01','1min:0.27 5min:0.29 10min:0.23'),(6,'users','2008-08-06 12:20:01','3'),(7,'proc','2008-08-06 12:20:01','140'),(8,'','2008-08-06 12:20:01','Key_read_requests:190457 Key_reads:148 Key_write_requests:79884 Key_writes:27194 history_list:3 innodb_transactions:479840 read_views:1 current_transactions:1 locked_transactions:0 active_transactions:0 pool_size:32000 free_pages:0 database_pages:31999 modified_pages:0 pages_read:5187008 pages_created:1153894 pages_written:6257069 file_fsyncs:185447 file_reads:4606447 file_writes:3070647 log_writes:90720 pending_aio_log_ios:0 pending_aio_sync_ios:0 pending_buf_pool_flushes:0 pending_chkp_writes:0 pending_ibuf_aio_reads:0 pending_log_flushes:0 pending_log_writes:0 pending_normal_aio_reads:0 pending_normal_aio_writes:0 ibuf_inserts:277316 ibuf_merged:277316 ibuf_merges:7235 spin_w'


the problem is default query statement buffer for insert is set to small, I manually changes buffer size in spine.h and recompiled spine.

/* #define BUFSIZE 1024 */
#define BUFSIZE 65535


Everything seems to be working after this last twick.

Here's some screen shots,




Friday, July 25, 2008

MySQL Cacti Templates installation

I was trying to install MySQL Cacti monitoring tool last few days. This is a cool RRDTOOL based monitoring package that can be downloaded from here.

http://code.google.com/p/mysql-cacti-templates

To get the template working, it requires Cacti to be installed. http://www.cacti.net/

Cacti is design around RRDTOOLs http://oss.oetiker.ch/rrdtool/

that in turn require many other packages installed,

pkgconfig, zlib, libpng, freetype, LibXML2, fontconfig, Pixman, Cairo, Glib, Pango

That's right every single of these packages need to downloaded, compiled and installed successfully.

The best way to enable wget for your server if you are behind a proxy, set following environment variable,

http_proxy=http://user:password@proxy2.yourcom.com:8080

Most package installation is pretty straight forward, just download the package, unzip it, configure, make and make install. I encountered some error message while compiling fontconfig, the error is

gcc -shared .libs/fcatomic.o .libs/fcblanks.o .libs/fccache.o .libs/fccfg.o .libs/fccharset.o .libs/fcdbg.o .libs/fcdefault.o .libs/fcdir.o .libs/fcfreetype.o .libs/fcfs.o .libs/fcinit.o .libs/fclang.o .libs/fclist.o .libs/fcmatch.o .libs/fcmatrix.o .libs/fcname.o .libs/fcpat.o .libs/fcserialize.o .libs/fcstr.o .libs/fcxml.o .libs/ftglue.o -Wl,--rpath -Wl,/usr/local/lib -Wl,--rpath -Wl,/usr/local/lib -L/usr/local/lib /usr/local/lib/libfreetype.so -lz -lexpat -Wl,--rpath -Wl,/usr/local/lib -Wl,-soname -Wl,libfontconfig.so.1 -o .libs/libfontconfig.so.1.2.0/usr/bin/ld: /usr/local/lib/libz.a(inflate.o): relocation R_X86_64_32 against `zcalloc' can not be used when making a shared object; recompile with -fPIC/usr/local/lib/libz.a: could not read symbols: Bad valuecollect2: ld returned 1 exit status

The error is actually because Zlib was not compiled properly. I end up recompiled Zlib with

env CFLAGS="-O3 -fPIC";./configure --shared;make;make install;

Clean the old fontconfig file, 'make clean' and recompile.

Bump into another error at final compile of RRDTOOL,

collect2: ld returned 1 exit status make[2]: *** [rrdtool] Error 1

It turns out because I skipped libxml2 installation because download link broken. Well, I hope the error message was little more informative.

Ok, next I need to make Cacti working. Which will not happen until PHP, MYSQL and Apache properly configured.
Then hit the DocumentRoot must be a directory problem, even I am pretty sure the document root is directory. This is because SELinux stopping apache taking it. This can be fixed by disable SELinux or

chcon -R -h -t httpd_sys_content_t /path/to/directory

Hit a php bug while compiling php5 with apache2. Get following error while start apache

Cannot load /etc/httpd/modules/libphp5.so into server:/etc/httpd/modules/libphp5.so: undefined symbol: zend_qsort

Have to recompile php5 with

./configure --enable-mbstring --with-mysql --with-apxs2=/usr/sbin/apxs --with-libdir=lib64

I was not using --with-libdir=lib64 switch.

Again, php seems not working with mysql properly. I doubt checked ./configure message of php , there's an error

configure: error: Cannot find libmysqlclient

Apparently, ./configure is looking for /lib64 under mysql installation but there's not any. Linked lib to lib64. I have to reconfigure php again :(


I was struggling in the last few days to figure out why my php page refuse connecting to local mysql database. Always has error from index.php complaining something is not right in 'include/config.php' . However I am very much sure everything is correct. I wrote a quick test php code to test the database connection. The interesting thing is I can run the code from command line without problem, but when I check it from web through apache it failed with "Can't connect to local MySQL server through socket " error. It's kind of frustrating.

Finally a note from google search helped me, I changed hostname from 'localhost' to '127.0.0.1' just to check if this will work. Guess what, it worked!

Ok, run into another problem trying to compile cacti-spine-0.8.7a,

*** [Gentoo] sanity check failed! ****** libtool.m4 and ltmain.sh have a version mismatch! ****** (libtool.m4 = 1.5.22, ltmain.sh = 1.5.24) ***Please run: libtoolize --copy --force

Running libtoolize --copy --force

create another error,

libtoolize: `/usr/share/aclocal/libtool.m4' is serial 47, less than 48 in `aclocal.m4'To remain compatible, you should update your `aclocal.m4' by running aclocal.Putting files in AC_CONFIG_AUX_DIR, `config'.

Having no idea what's going on here, I searched internet and the solution is

autoreconf --force --install --symlink

Tuesday, January 22, 2008

If you google mysql and youtube, there're many people talk about Paul Tuckfield's keynote presentation at MySQL world 2007.

However few noticed there's another great presentation of youtube mysql story by Cuong Do.
Seattle Conference on Scalability: YouTube Scalability

Cuong talked about the scaling problem of youtube's growing pain with in depth technoical detail.

Monday, January 21, 2008

Hidden feature of innodb monitor

It's quite fascinating to find out MySQL allow user to turn on/off extra monitoring information in a interesting way.

Create/Drop special tables like innodb_monitor and innodb_lock_monitor etc.

http://dev.mysql.com/doc/refman/5.0/en/innodb-monitor.html