Blog

Yesterday I updated to wicd -1.7.0 and experienced issues when connecting to my local wired network. I found out that the issue only comes up when using wicd-gtk. Wicd-curses manages to get a connection.

So I searched the web and found out that this problem only occurs when having (dis)connection scripts set. The issue is already filed at wicd’s launchpad upstream with a working patch by Jonathan (Comment #17). As I use wicd every day, the fix for this issue is quite important to me.
Gentoo currently only has the 1.7.0 (1.6.2 worked flawlessly) version in its tree and so I made an ebuild which includes Jonathan’s patch, fixes the issue and can be found in gentoo’s bugzilla.

If you are experiencing the same issues in gentoo, check out the new ebuild and the patch.

Happy emerging
Regards,
Phillip

My last post about server optimization dates February because the last weeks/months have been quite busy. I promised to continue with MySQL optimisation which I will be doing now. As with the other posts, I will only write down significant new information and otherwise link to the information in the web so that you can use this guide as a condensed view on the topic.
First of all visit documentation at mysql.com which contains a whole chapter about optimization. If you only want to
optimise the MySQL server
this subchapter will
suit your needs.

The MySQL Query Cache

The query cache, although not a panacea, can bring big performance improvements and is, at least with mysql 5.0, disabled by default. However you have to know that it only optimises queries. It cannot
“look” into your application and group all queries of the same request or similar.
The MySQL Performance Blog has a nice tutorial about configuration and background of the query cache.

Adapting MySQL Cache Sizes

MySQL has several caches which need to be adapted to your personal needs. If you are using PHPMyAdmin, it can give you hints about the cache sizes which need to be optimised in your current setup. Just open server_status.php which is linked on the start page as Show
MySQL runtime information. To change values edit your /etc/mysql/my.cnf and restart MySQL afterwards.
If you want to know how to view this information directly via SQL command, look at this optimisation guide underneath the subheading Getting information about current values or into the MySQL documentation. Both guides also lists other possible improvements.

Identifying slow queries

Optimising your web application or developing it with database performance in mind, normally should be the first step to do. Though it may often be the case that you cannot access the code of your application, have not the required skills, not enough time or that you do not want to change the standard application for easier updates. Problems in this area can often be avoided or at least minimized by using an object relational mapper like hibernate for Java as it optimises the queries on object level before executing them.
If you cannot use an ORM mapper for some reasons or if you want to know which queries use most resources you can activate the MySQL Slow Query Log. Pete Freitag directly states which entries are needed in the my.cnf file.

Optimising your application

After you identified slow queries or even before generating them, this blog post for PHP experts can help you finding database related performance hits in PHP with PHP.

As a general measure I strongly recommend using indexes for frequently used or searched attributes. Although from 2001 the following guide explains these topics and their background greatly.

Optimising the compilation

If you compile mysql by yourself, you can also get speed improvements by compiling MySQL with special
options. The MySQL
documentation lists several possibilities.

Optimising your Linux kernel for MySQL

Apart from MySQL
itself you can also optimise your kernel parameters (sysctl.conf) for
MySQL. If you want to, refer to this guide.

Other resources

If this still is not enough you will find plenty of other resources in this forum post. Some of the links listed there were already mentioned before.

Finally…

Before investing in new hardware be sure to check the configuration of your database and,  when you can, the database queries in your application. Even if the latter is not possible, simple server side tuning can bring huge improvements especially when the query cache is not activated or it or other caches are too small so that MySQL hast to write more data to disk than necessary.

I hope this small guide/list of links gave you an overview over MySQL performance tuning. For me it will serve as an aid to memory and thus it will be expanded if future issues arise.

Regards,
Phillip

I am using Latex for assignments, papers and presentations quite a long time. However, until recently, I did not know a solution for dual head presentation of pdfs created with the latex beamer class.
This changed with the finding of Impressive. So if you want to create a pdf with notes on the second screen you first have to set up your beamer class like this:

[code language=”latex”]\documentclass{beamer}

\usepackage[utf8x]{inputenc}
\usepackage{default}
\usepackage{pgfpages} %This is needed for notes presentation!
\setbeameroption{show notes on second screen}

\begin{document}
\begin{frame}
\frametitle{Note Test}
This is the frame text
\note[item]{Note for a itemized note list}
\note{Note for a continuous note text}
\end{frame}
\end{document}
[/code]

 

After that install impressives requirements and impressive itself. Then the following command will start the presentation on a beamer and notebook both with a 1024×768 resolution:

[code language=”bash”] ./impressive.py -g 2048×768 “pdfname.pdf”[/code]

Normally notes will be rendered on the right screen. So you have to set up the beamer as “left of” your notebook. With xrandr this can be accomplished like this or similar depending on your available outputs:

[code language=”bash”] xrandr –output VGA1 –left-of LVDS1 –output LVDS1 –mode 1024×768[/code]

Be sure to use the same aspect ratio and resolution on notebook and beamer. Otherwise you will get cropped slides and/or black borders.

Update: Someone pointed me to this useful article which achieved similar results by only using XrandR’s scaling and positioning parameters.

Happy presenting!
Phillip

I am using Latex for assignments, papers and presentations quite a long time. However, until recently, I did not know a solution for dual head presentation of pdfs created with the latex beamer class.
This changed with the finding of Impressive. To create a pdf with notes on the second screen you first have to set up your beamer class like this:

[code language=”latex”]\documentclass{beamer}

\usepackage[utf8x]{inputenc}
\usepackage{default}
\usepackage{pgfpages} %This is needed for notes presentation!
\setbeameroption{show notes on second screen}

\begin{document}
\begin{frame}
\frametitle{Note Test}
This is the frame text
\note[item]{Note for a itemized note list}
\note{Note for a continuous note text}
\end{frame}
\end{document}
[/code]

 

After that you have to install impressives requirements and impressive itself. Then you can start impressive on a beamer and notebook both with a 1024×768 resolution like this:

[code language=”bash”] ./impressive.py -g 2048×768 “pdfname.pdf”[/code]

Be sure to use the same aspect ratio and resolution on notebook and beamer. Otherwise you will get cropped slides and/or black borders.

Happy presenting!
Phillip

This HOWTO will show you how to upgrade from kde 4.3 to kde 4.4 (and possibly others) with as few problems as possible (It went through without errors for me). The whole process including merging will take 4 to 10 hours depending on your PC. Plan accordingly.

Before upgrading check if you use the latest packages. Additionally I will assume you have autounmask, eix and gentoolkit installed. Portage 2.2 might also come in handy because it can automatically resume builds of many packages (–keep-going) when only few of them fail.

[code lang=”bash”]
emerge –sync
emerge -nva gentoolkit autounmask eix
eix-update
[/code]

If you want to use your computer while updating KDE it is pretty neat to have an alternative login and windowmanager installed. I prefer gdm and enlightenment.

[code lang=”bash”] emerge -nva eix gdm enlightenment
[/code]

 

Preparing the upgrade

[code lang=”bash”]
#create a working directory
mkdir /root/kde-upgrade

#save the list of your kde 4.3 packages
eix -I –slot 4.3 -C kde-base –only-names > /root/kde-upgrade/kde-old
#save the list of packages installed by which are also avalaible for kde 4.4
eix -I –slot 4.4 -C kde-base –only-names > /root/kde-upgrade/kde-new
#save the list of installed qt-packages
eix –slot 4 -C x11-libs –only-names -I qt > /root/kde-upgrade/qt-packages
#KDE 4.4 needs QT-4.6 that is why we create a list of all qt-packages and keyword them for later use. Be sure to change ~amd64 to your architecture
eix –slot 4 -C x11-libs –only-names -I qt | while read line; do echo “$line ~amd64” >> /root/kde-upgrade/qt-keywords; done
[/code]

Copy the qt-keywords to your package.keywords directory

[code lang=”bash”]
cp /root/kde-upgrade/qt-keywords /etc/portage/package.keywords/
[/code]

Download and install kde-4.4 keywords

[code lang=”bash”]
cd /root/kde-upgrade
wget -O kde-4.4-keywords “http://git.overlays.gentoo.org/gitweb/?p=proj/kde.git;a=blob_plain;f=Documentation/package.keywords/kde-4.4.keywords;hb=HEAD”
cp -p /root/kde-upgrade/kde-4.4-keywords /etc/portage/package.keywords/
[/code]

 

Prepare your alternative Desktop environment

Now we are starting with the essential stuff. Log out from kde and shutdown kdm from one the Ctrl-ALT-Fx-Terminals and start your favorite alternative login manager.

[code lang=”bash”]
/etc/init.d/xdm stop
#Start other login manager
gdm
[/code]

After that you can login again on Ctrl-ALT-F7.

Upgrade QT and remove old KDE

Update your qt installation now because KDE 4.4 requires qt 4.6.

[code lang=”bash”]
emerge -va `cat /root/kde-upgrade/qt-packages`
[/code]

After that we are ready to remove the old KDE. Old KDE version’s packages would be removed by an upgrade anyway. When we remove them before our installation we will save us a lot of hassle concerning package blocks etc.

I heard unmerging the old kde version is completely unnessessary now, as newer portage versions will handle softblocks by themselves. So it is completely ok to skip the unmerging.

[code lang=”bash”]
emerge -C `cat /root/kde-upgrade/kde-old`
[/code]

 

(While emerging qt) Check for keyworded packages

While QT ist emerging switch to another terminal and check if your following KDE merge has all its dependencies in place.

[code lang=”bash”]
#try to
emerge -vau `cat /root/kde-upgrade/kde-new`
[/code]

If you get an error about missing keywords on dependant packages, execute:

[code lang=”bash”]
autounmask package_category/package_name-package_version
[/code]

on the corresponding package (the package mentioned in “dependency required by”). Personally I had to autounmask phonon-kde, kopete, kdebase-runtime-meta, kaddressbook and libiodbc. Libiodbc-3.52.6 failed on ~amd64 so I installed the keyworded version which built normally.

 

(After emerging QT) Rebuild Qt dependant packages

Rebuild all qt-dependendant packages.

[code lang=”bash”] emerge -1va sip PyQt4 qscintilla qscintilla-python
[/code]

Apart from these packages you will have to rebuild all installed kde-themes in portage. After that remerge Phonon. I had to do this and for you it will not hurt ;-).

[code lang=”bash”]
#emerge your new kde’s phonon version
emerge -1va phonon
[/code]

 

Emerge your new KDE

So sit back and relax. The biggest part of the work is done now. Emerging new KDE packages will take several hours depending on your CPU power. It is best to do this over night. If you peek at the following command you will see a –keep-going at the end. This tells Portage-2.2 to automatically resume the whole merge with packages not depending on the failed package when a package fails. If you use Portage-2.1 remove this option and beg that you will wake up with a fresh and shiny new version of KDE.

[code lang=”bash”]
#DO!
emerge -va `cat /root/kde-upgrade/kde-new` –keep-going
[/code]

 

Final work

After our work before kde should build without errors. At least it did for me… If there remain errors visit http://forums.gentoo.org in your spare desktop environment.

[code lang=”bash”]
#revdep-rebuild to rebuild packages like e.g. digikam or similar
revdep-rebuild
#update your config files
etc-update
#read announcements generated during upgrade
eread
[/code]

If all is done, you can log-out from your temporary desktop environment,
kill your other login manager and start kdm again. Then log-in to your new shiny kde version and enjoy.

If you have suggestions or improvements let me know.
Hope this helps,
Phillip

The last weeks I investigated server side optimization (not application optimization) issues further because our Joomla based web portal scales quite poorly. The server is still sufficient for our current load but I want to have some more resources for usage peaks and future development. I thought it would be nice to share my thoughts and experiences as appropriate tutorials are scattered all over the web. I will describe the main steps in brief and will link to further documentation at the web if possible and needed.

So far so good. Currently our application uses the following server software:

• Joomla 1.5 as web application
• PHP 5.2 with ZendOptimizer 3.3 as our only must have PHP extension
• Apache httpd 2.2 as web server
• Varnish 2 as reverse caching proxy for Java and PHP apps.
• MySQL 5.0 as database management system
• Gentoo Linux as server OS

Our website makes intensive use of Joomla extensions. Some of them were developed by our own IT department. The start page is news oriented and contains several extensions and also many images. Apart from that we have quite a few registered users which use interactive parts of the site like forums or lotteries. Additionally some content of the site is only available to privileged users.

With this information you should have a rough overview of our web site and the application comprehending my optimization efforts. The next part of the series will contain information about bringing MySQL to speed.

Comments are as always welcome
Phillip

Ah. Silly me. I experimented with the drupal caching options just before installing the WYSIWYG module and wondered why the editor was not showing up.

The solution was as obvious as simple: I had to disable or clear Drupal’s JavaScript and CSS cache after installing the WYSIWYG plugin.

Hopefully you read this before long searching. Additional clues can be found in this drupal forum post ( http://drupal.org/node/426398 ).

 

I recently stumbled upon a problem of a big evolutionary grown qmail/vpopmail installation when we moved the installation to a new server.

Virtual accounts created years ago were created with a different vpopmail installation located in a different filesystem path. For this reason accounts had different paths set up in the /var/vpopmail/domains/domain.tld/vpasswd.cdb file. Regrettably there is (at least afaik) no possibility to change these paths with vpopmail commands like vmoduser or similar.

To avoid always creating symbolic links from the old directories to the new account directories I found a way to modify/update these paths. Normally values in the plain text vpasswd (not the .cdb one mentioned before) file get overwritten every time you update/delete or create accounts with the values from the vpasswd.cdb file. However if only the vpasswd file and not its .cdb counterpart is present the vpasswd.cdb file will be regenerated with the values from the vpasswd file.

So if you want to change paths, do the following:


#copy old vpasswd to a new location
cp vpasswd vpasswd.new
# edit vpasswd.new to suit your needs
vim vpasswd.new
# shutdown your mailserver services for some seconds to prevent regeneration of the vpasswd.cdb file before you are ready (check if you may do this before!)
/etc/init.d/svscan stop
cp vpasswd.new vpasswd
#remove cdb file
rm vpasswd.cdb
# create a testuser for the appropriate domain, to regenerate cdb file
vadduser testuser@domain.tld
# delete testuser
vdeluser testuser@domain.tld
# start mailserver services again
/etc/init.d/svscan start


Normally a shutdown time of a few seconds should not be critical for a mailserver, because sending servers normally have queue lifetimes much longer than this and sending users can wait a few seconds anyway, eh ;-)?

You should however check this first!

If you put the lines above in a appropriately changed script downtime can be less than 5 seconds.

Hope this helps,
Regards,
Phillip

A few days before I stumbled upon a problem with our OpenSSH (net-misc/openssh-5.2_p1-r3) based SFTP solution. Although passwords were not changed SFTP logins did not work any longer whereas normal SSH logins with the same accounts continued working.

The concerning – now working – SSHD config looks like this (except AllowUsers):

Port 22
Protocol 2
LogLevel INFO
SyslogFacility AUTH
LoginGraceTime 60
PermitRootLogin no
PasswordAuthentication yes
KeepAlive yes
# this must be set to no and PasswordAuthentication to yes. Otherwise SFTP will not work!!!!
UsePAM no
PrintMotd no
PrintLastLog no
ClientAliveInterval 30
ClientAliveCountMax 10
Subsystem sftp internal-sftp
Match Group sftponly
ChrootDirectory /home/%u
ForceCommand internal-sftp
X11Forwarding no
AllowTcpForwarding no


After more than one hour of trial and error I found out, that UsePAM must be set to no and PasswordAuthentication must be set to yes. All other combinations of these two options kill sftp authentication (sys-auth/pambase-20090620.1-r1 with ssh USE flag enabled).

As I am not using pam’s advanced authentication functions this deactivation is not a problem to me.
So, finally, if you encounter strange authentication issues with sftp try to disable pam auth and see if sftp authentication is working again afterwards.

Regards,
Phillip

PS If you know another solution to this problem or if I somehow messed up my config please let me know.

My name is Phillip Merensky and I am currently studying master of applied computer science at the university of Bamberg/Germany.