Usually, you trace a single process that does not give a helpful error message. But if you have to trace e.g. a webserver with multiple processes you have to type a lot. Here is a one-liner that makes it a little bit easier:

strace -f -p $(ps -ea f| grep httpsd | cut -f 1 -d " " | xargs | sed -e 's/ / -p /g') 1>/tmp/strace.log 2>&1

When using web services, its sometimes hard to read the serialized json data. I wrote a small helper script to dump the contents

File: /usr/local/bin/json_decode

#!/usr/bin/php
<?php
error_reporting(E_ALL);
ini_set('display_errors',true);
if(!empty($argv[1])) {
        $json = file_get_contents($argv[1]);
}
else {
        $json = '';
        while(!feof(STDIN)) {
                $json .= fgets(STDIN, 1024);
        }
}

$data = json_decode($json);

if($data === null) {
        fwrite(STDERR, "Invalid json data\n");
        exit(1);
}

print_r($data);

Usage:

wget --quiet -O - "http://domain.tld/file.json" | json_decode

Usually you ignore tables when dumping your database. But when importing you can not ignore certain tables by the mysql command line tool.

I just had the challenge to find an easy way to do this or wait houres until the import completes and found an easy workaround. Just before you import the sql file, you can create a dummy table, which does not match the original schema. When you start the import an error will occur when processing the insert statements. If you supply a --force to your command line, mysql will ignore the error and continue with the import.

grep -v "DROP TABLE IF EXISTS" mysql-backup.sql | mysql --force

find -lname "*" | xargs ls -l

Less lines of code as in the unbindModelAll example:

class AppModel extends Model {
        function unbindModelAll() {
                foreach(array(
                        'hasOne' => array_keys($this->hasOne),
                        'hasMany' => array_keys($this->hasMany),
                        'belongsTo' => array_keys($this->belongsTo),
                        'hasAndBelongsToMany' => array_keys($this->hasAndBelongsToMany)
                ) as $relation => $model) {
                        $this->unbindModel(array($relation => $model));
                }
        }
}

If you set the security level in a cake application to high, an algorithmus regenerates the SessionID.

Usually, this is not a bad idea. But as sometimes several requests are started simoultaniously, I discovered that random requests didn't get a proper user-id in the session and further found out that during AJAX-Requests requests the sessioncookie got deleted and in the same request a new one was set. As a result, a few AJAX-Requests failed due to an invalid session id.

After reducing the security to medium, the errors were gone.

It seems that this is a known issue to the developers (https://trac.cakephp.org/changeset/5982) and should have been fixed for over two years. As far as I can tell from the current source, SessionIDs are regenerated each tenth request - even if a XML-HTTP-Request was made. Due to race conditions this will make many developers headaches and is a bug that should be fixed.

A slave recently began to report syntax errors in replicated queries. This happend about once a day and was completly random. As the master ran fine and the connection between the two hosts has no errors, the error must be somewhere located on the slave. I began testing the server including the discs without result. Later I accidently did something that made a service on the slave crash. In the error message I could see that the kernel discovered some kind of memory corruption in the process and terminated it.

After running memtest for a couple of houres on the system some memory errors were found. The server was tested before deployment and was running without errors for about two years so memory was the last thing I thought about. But now I can imagine why enterprise servers use ECC memory.

When connection to a mysql you can either connect by a TCP/IP connection or through a socket.

If you are connection though a socket the host value for the connecting user will always be 'localhost'. When using TCP/IP the host will always be '127.0.0.1'.

This is very important when specifiying privileges. If you allow a user to connect from 127.0.0.1 ('user'@'127.0.0.1') this user will not be able to connection through sockets:

# mysqladmin --user=zabbix_agent --password=xxx ping
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'zabbix_agent'@'localhost' (using password: YES)'

mysql> show grants for 'zabbix_agent'@'127.0.0.1';
+----------------------------------------------------------------------------------------------------------------------------------+
| Grants for zabbix_agent@127.0.0.1                                                                                                |
+----------------------------------------------------------------------------------------------------------------------------------+
| GRANT REPLICATION CLIENT ON *.* TO 'zabbix_agent'@'127.0.0.1' IDENTIFIED BY PASSWORD '*xxx' |
+----------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Note that the mysql client will automatically use a socket if you use -h localhost (default). So if you pass -h 127.0.0.1 things will work:

# mysqladmin -h 127.0.0.1 --user=zabbix_agent --password=x ping
mysqld is alive

If you get the error

MySQL said: Access denied; you need the SUPER privilege for this operation

when inserting data into an table with a trigger, you should check definier of the trigger. Mine was empty and caused this error. Re-Creating the trigger solved the problem and a proper definier was set.

To dump all you triggers you can use the following query

SELECT CONCAT_WS( " ", "CREATE TRIGGER",  TRIGGER_NAME,  ACTION_TIMING,  EVENT_MANIPULATION,  "ON",  EVENT_OBJECT_TABLE,  "FOR EACH ROW",  ACTION_STATEMENT) 'create_trigger' 
FROM information_schema.triggers

When using CakePHP with PHP 5.3.0 you will get tons of depricated errors like

Deprecated: Assigning the return value of new by reference is deprecated in /data/workspace/us/htdocs/tixi/cake/libs/debugger.php on line 99

You can fix this by changing the default error_reporting value of the debug mode. Open the file

cake/libs/configure.php

And replace the line 295:

$ diff configure.php configure.php.new
295c295
<                               error_reporting(E_ALL);
---
>                               error_reporting(E_ALL & ~E_DEPRICATED);

From time to time machines use some swap space due to heavy load. But Linux doesn't free the swap space itself if enough memory is available.

An easy method to flush all disk swap to memory is to disable and re-enable the swap. Eg:

# free -tm
             total       used       free     shared    buffers     cached
Mem:          4096       3269        826          0         31         76
-/+ buffers/cache:       3161        934
Swap:          509        114        395
Total:        4605       3384       1221

# swapoff /dev/xvdb1
# swapon /dev/xvdb1

# free -tm
             total       used       free     shared    buffers     cached
Mem:          4096       3375        720          0         31         78
-/+ buffers/cache:       3265        830
Swap:          509          0        509
Total:        4605       3375       1230

After installing a Xen Lenny on a Xen 5.5 host, the boot-process failed with the following error:

08.07.2009 15:55:47 Error: Starting VM 'vmname' - The bootloader for this VM returned an error -- did the VM installation succeed?  Error from bootloader: Return code: 1; Log = Traceback (most recent call last):
  File "/usr/bin/pygrub", line 719, in ?
    raise RuntimeError, "Unable to find partition containing kernel"
RuntimeError: Unable to find partition containing kernel

Cause of this error was, that I changed the root filesystem from ext3 to xfs during setup, which is the better choise for databases with large tables. But obiously Xen doesnt like that.

During a boot of a xen debian template you can see the following output:

Configuring network interfaces...Starting portmap daemon....
portmap: server localhost not responding, timed out
RPC: failed to contact portmap (errno -5).

If you press CTRL-D to cancel the portmap startup the loopback interfaces does not come up and several other services will not come up. I've been searching for the error for a long time but finally I found the wrong configuration. If you take a close look at the network configuration file at /etc/network/interfaces you will see the following startup order:

auto eth0 lo

This is wrong. Change the order to:

auto lo eth0

Now the system will bring up the interfaces in the proper order.

# netio -u location-somewhere-in-baden-württemberg


NETIO - Network Throughput Benchmark, Version 1.26

(C) 1997-2005 Kai Uwe Rommel

 

UDP connection established.

Packet size  1k bytes:  114245 KByte/s (0%) Tx,  110876 KByte/s (0%) Rx.
[...]

Will haben™

I was curious how fast our SAN is when heavy IO load hits the system by multiple processes.
I found a quite handy tool that can run benchmarks quite easily: www.iozone.org

I ran the test with a 5 simultaneous clients, each writes 1GiB to a file in 512KiB blocks.


Click to enlage

Command line:

./iozone -R -l 5 -u 5 -r 512 -s 1g /mnt/10_tib/tmp1 /mnt/10_tib/tmp2 /mnt/10_tib/tmp3 /mnt/10_tib/tmp4 /mnt/10_tib/tmp5

As I had no dedicated testing FCSAN, the tests were performed on a cluster that had more than a dozen VMs running. So the results don't reflect synthetic results but reallife performance. The RAID6 System usually performs backups, but is completely idle during work-time. The SATA System was a fresh installed mainstream rootserver without RAID.

You can see that the FCSAN outperforms a single SATA drive by about 4-10 times. So there is absolutely no way to compare these systems.

A bit more interesting is the RAID-6 performance with an ARECA controller. It nearly matches the simple read and write speeds, but can not match the performance in database-specific operations like random write or mixed workload.

I'm looking forward to repeat the tests, once the first SSD drives are considered to be stable for enterprise architectures.
"IO performance" vollständig lesen