Setting up file- and mediaserver with Debian GNU/Linux 7.8 and 6TB of RAID-5 space

This post a continuing saga about my home-server started in this post now more than 6 years ago.

Times have moved along and more and more I have realized that not only is file redundancy important, but having half-a-thousand CD's and DVD's lying around is not the nicest of one's room decoration.
Due to the need to clear one wall where currently most of the mentioned CD's and DVD's are located, I decided to move the more important content into my library server and eliminate the physical media. This, however, needs a lot of storage space. My Library server previously had three 1.0TB Western Digital Green hard drives in RAID-5, which amounted to 2.0TB of space. I decided to triple that to 6.0TB with three 3.0TB Western Digital Green drives. One funny note to make is that the new 3.0TB Green drive also weighs subjectively about three times as much as the previous 1.0TB Green drive.

In the end, the power consumption of the whole system went up as expected, but not that much - from about 58W average to 65W average at idle.

Along with the change of hard drives I also switched the processor and thus also the motherboard and memory. This decision was primarily caused by the inability of the old Athlon 64 3700+ to fully facilitate the Plex media server, especially the transcoding was unacceptably slow, but also the Plex client interface itself was lagging badly both in browser and in my Samsung TV. The one single Mebibyte of memory in the old server might've also played a part on the bad performance.

So the specs of the new server are as follows:

• AMD A8-6500T APU (4 cores, 45W, 2.1GHz (3.1GHz Turbo), 1.4GHz idle)
• ASUS A88XM-E
• 4.0GiB Crucial Ballistix Sport
• Tagan 420W i-Xeye TCF PSU
• Antec Three-Hundred case

Server installation

OS choice
The operating system was again a tried and true Debian GNU/Linux, this time around the version 7.8, "Wheezy".

Partitioning
As before, I decided to split the space up to three parts. An adequately sized (30GiB) root partition at the beginning, a 4GiB swap partition at the end and everything between them as the Data partition.

The first time I did the installation I ended up with an error at the point in the installer, where the grub was to be installed. A bit of digging in the net revealed that with 3.0TB drives, GPT must be used and for a boot drive, there has to be a small reserved space for the boot code left on the drive, with the "bios_grub" flag set

Since this kind of partitioning cannot be done from within the Debian installer, use an Ubuntu live CD, open up three terminals in order to be sure to do the exact same steps on all three disks and use the following steps in each:

sudo parted /dev/sda         # use sdb and sdc in the other windows.
mklabel gpt                  # create a new blank GUID partition table.
unit MiB                     # change the units to Mebibytes (notice the 'i' the middle).
print                        # prints the disk information, including the maximum MiB position.
mkpart GRUB 1 2              # create 1 MiB partition between 1 MiB and 2 MiB positions for BIOS boot code and set
set 1 bios_grub on           #   the "bios_grub" flag for that partition.
mkpart ROOT 2 15362          # create 15360 MiB (15 GiB) partition between 2 MiB and 15362 MiB positions for root.
mkpart HOME 15362 2859536    # create 2.7 TiB partition for Data; as the second position, use the max position you got
                             #   above, but subtract 2048 MiB for the swap partition and an additional 2 MiB for some
                             #   spare space at the end.
mkpart SWAP 2859536 2861584  # finally, create a 2 GiB partition for the swap at the end.
quit                         # exit from parted.

Notice that these partitions are used in RAID-5, therefore doubling their sizes, so root will be 30 GiB, swap will be 4 GiB and the Data partition will be 5.4 TiB.

Now, start the Debian installer and in the partitioning step...

• be sure to leave the first 1.0 MiB partition as it is, do not change it!
• mark the latter three partitions on each disk as volumes for RAID,
• combine the corresponding partitions on all three disks into RAID-5 arrays.
• use ext4 on the 30 GiB partition and use it as the / (root) partition with the noatime flag.
• also use ext4 on the ~5.4 TiB partition, but set the mount point to be /Data, also use the noatime flag.
• set the last 4 GiB partition as the swap partition.

After partitioning, at software selection step, I included the web server, file server and ssh server.

At the grub setup step, the Debian installer only installs grub into the first disk, therefore it is important to install grub manually into the other disks first! Switch to one of the other consoles, eg. with ALT+F2, and enter the following commands:

chroot /target /bin/bash
grub-install /dev/sdb
grub-install /dev/sdc

The installer will finish soon thereafter.

Server setup

Enable sudo for your user:

adduser username sudo

Note that this takes effect only after logging out and back in.

Reduce the boot manager delay:

sudo mcedit /etc/default/grub

Find and change the GRUB_TIMEOUT value from 5 to 1 and execute:

sudo update-grub

Change Gnome Classic to be the default graphical shell:

sudo update-alternatives --config x-session-manager

And select the fallback line, usually number 2.

Change the dynamic IP to be static:
In Gnome Classic, in the right at the upper bar, right-click the network icon and choose Edit to configure the network interface. In the window that opens, select your network interface and click Edit in the right. In the following window, at IPv4 settings tab, switch the method from DHCP into Manual, click Add, enter your static IP (eg. 192.168.10.2), netmask (255.255.255.0) and gateway (192.168.10.1), also enter the DNS below and click Save.

Make Debian boot into console mode:

sudo update-rc.d gdm3 disable

Setting up the environment:

mcedit ~/.bashrc

Add the following lines:

stty -ixon  # enables forward bash command history search CTRL-S
export LC_COLLATE="en_US.UTF-8"  # fixes the broken Estonian character map which breaks regexp [a-z]

File server setup

Setting up the Data directory:

sudo addgroup share
sudo adduser username share
cd /
chown nobody:share /Data
chmod 777 /Data
cd /Data
mkdir .transcodetemp  # this directory is for Plex Media Server
chmod 777 .transcodetemp

Setting up anonymous read-only Samba account:

sudo mcedit /etc/samba/smb.conf

Find the section "Share Definitions" and add the following right beneath it:

[library]
   comment = Library
   path = /Data
   browseable = yes
   guest ok = yes
   read only = yes

Setting up NFS share:

sudo mcedit /etc/exports

Add the following line:

/Data 192.168.10.0/24(rw,root_squash,async,no_subtree_check)

Making sure the file owner is displayed correctly over NFS:

sudo mcedit /etc/idmapd.conf

Uncomment the "Domain = xxxx" line and enter your domain in place of the xxxx.
Make sure to use the same domain in your other servers that connect here over NFS!

Reboot the server to make the changes take effect.

Media server setup

Setting up Plex Media Server:

echo "deb http://shell.ninthgate.se/packages/debian wheezy main" | sudo tee /etc/apt/sources.list.d/plexmediaserver.list
wget http://shell.ninthgate.se/packages/shell-ninthgate-se-keyring.key
sudo apt-key add shell-ninthgate-se-keyring.key
sudo apt-get update
sudo apt-get install plexmediaserver

This will also set the Plex Media Server to be automatically started when the server starts.

Other optional steps

Repairing the locales:
When you select a different language at the start of the Debian installer, it configures only the locale for this language. However, for many reasons the English locale should also exist, therefore:

sudo dpkg-reconfigure locales

And check also the en_US.UTF-8 locale in addition to your language locale.

Setting up redundant file- and mediaserver with Debian GNU/Linux 7.5 and RAID-5

This post is effectively a continuation of a saga I started now 5 years ago.

Times have moved on, electricity price has gone up and at some point I changed my Library box setup, swapping out the four 320GB hard drives with total of 960GB of RAID-5 storage for two 500GB hard drives in RAID-1, so retaining 500GB of redundant space.

I though back then that I would make the box consume less power that way. In some ways it worked, the power use went down about a dozen watts. In addition, I removed the videocard, as that motherboard had integrated graphics and I also removed a gigabit network card, as my intention was to put the machine into a closet, where I only had 100Mbit network connection anyway. With all those changes I managed to reduce the power usage from the original 70W down to 50W at idle (and in text mode).

As I now needed more redundant space, I swapped out the hard drives once again. In addition to a 1TB Western Digital Black drive, which I already owned, I bought additional two 1TB Western Digital Green drives (49€ a piece) to set up the system on RAID-5 again, giving me almost 2TB of redundant storage.

The other machine specs remained the same:

• AMD Athlon 64 3700+ San-Diego (clocks between 1.0 and 2.2GHz)
• Zalman CNPS7000B-Cu
• ASRock 939NF6G-VSTA (nForce 430)
• 1.0 GiB Kingston HyperX
• Tagan 420W i-Xeye TC PSU
• Antec Three-Hundred case

Server installation

As before, my choice was Debian GNU/Linux, though this time around the newest version 7.5, "Wheezy".

Partitioning

On each of the drives I created two partitions for software RAID.
The first partition is 998GB for the main data and the remaining 2GB will be used for swap.
NB! It is important to enable the boot flag on all of the three disks!
The bigger partitions were then combined into RAID-5 array to form the root partition of about 2TB using EXT4 and the smaller partitions formed the swap space of about 4GB, also using RAID-5.
This way, not only the data would survive a failure of any one of the drives, but also the whole setup should withstand such a failure.

After partitioning and loading some packages, the Software categories choice came up. I decided to leave the desktop choice, though I will later switch off the automatic boot into graphic mode.
I chose also web server, SQL server, file server and ssh server. I left also the print server, though I'm pretty sure I will not be using it for printer serving.

As I was using a "netinst" image, the installation took quite a while, which was no doubt dependent on my network speed.

When the time came to set up grub, I already knew in advance that the installer only installs grub on the first drive and I also need to install grub on the other two disks.
The necessary commands, to be entered before you allow the installer to install the grub into /dev/sda and call update-grub, are:

chroot /target /bin/bash
grub-install /dev/sdb
grub-install /dev/sdc

The install will finish shortly after that.

Reduce the boot manager delay:

mcedit /etc/default/grub 

and change the timeout value

GRUB_TIMEOUT=5

to something smaller, like 1, and then execute:

update-grub

Enable sudo for the main user:

adduser username sudo

this takes effect only after logging out and back in.

Change Gnome Classic to be default graphical shell:

update-alternatives --config x-session-manager

and selec the fallback line, usually number 2.

Change the IP to be static:

In Gnome Classic, in the right at the upper bar, right-click the network icon and choose Edit to configure the network interface.
In the window that opens, select your network interface and click Edit in the right.
In the following window, at IPv4 settings tab, switch the method from DHCP into Manual, click Add, enter your static IP, netmask and gateway, also enter the DNS below and click Save.

Make Debian boot into console mode

update-rc.d gdm3 disable

This will retain the possibility to start the graphical UI with the startx command.

Setting up the environment:

mcedit ~/.bashrc

add the following lines:

stty -ixon  # enables forward bash command history search CTRL-S
export LC_COLLATE="en_US.UTF-8"  # fixes the broken Estonian character map which breaks regexp [a-z]

File server setup

Creating a directory for the data to be served and group for users:

Create a new group called "share":

addgroup share

Add your user into that new group:

adduser username share 

Create and fix the rights to the /Data directory:

cd /
mkdir Data
chown nobody:share Data
chmod 775 Data

Set up an anonymous readonly Samba account:

mcedit /etc/samba/smb.conf

find the section "Share Definitions" and add the following beneath it:

[library]
   comment = Library
   path = /Data
   browseable = yes
   guest ok = yes
   read only = yes

Set up an NFS share:

mcedit /etc/exports

Add the following line:

/Data 192.168.10.0/24(rw,root_squash,async,no_subtree_check)

Making sure the file owner is displayed correctly over NFS:

mcedit /etc/idmapd.conf

Uncomment the "Domain = xxx" line, and enter your domain.
NB! Be sure to set that same domain name on the client aswell.
Reload the NFS configuration on the server:

/etc/init.d/nfs-kernel-server restart

Probably a better way for all of the changes to take effect is to reboot the machine now.

Media server setup

Setting up Plex Media Server

echo "deb http://shell.ninthgate.se/packages/debian wheezy main" > /etc/apt/sources.list.d/plexmediaserver.list
wget http://shell.ninthgate.se/packages/shell-ninthgate-se-keyring.key | apt-key add -
apt-get update
apt-get install plexmediaserver

This will also add the PMS to be automatically started when the server starts.

Other optional steps

Fixing the locale problem:

Somehow I managed to make all locales, except my own, to disappear. The fix:

dpkg-reconfigure locales

Setting mcedit as the default editor:

update-alternatives --config editor

How to make Avidemux work under Ubuntu 13.10

Ubuntu 13.10 Saucy Salamander
Avidemux 2.5.4

I was quite disappointed in Ubuntu 13.10, when I first tried it last year. Several apps were broken and many things did not function.

One of those apps was Avidemux, which I am using regularly at home to encode videos. Note that I'm talking about avidemux2-gtk version.

In Ubuntu 13.10, Avidemux would run, but after selecting some menu item which opens a dialogue window and then closing that dialogue would always hang Avidemux.

For example Help / About will open the About window and when you click OK, the window will hang with the button still appearing pressed and after a moment the main window will fade into black-and-white indicating an unresponsive window. Only xkill will help out and put Avidemux out of its misery.


I recently decided to try again. At my workplace I had upgraded to 13.10 a while ago, as I did not need Avidemux in there. A lot of time had passed and I thought perhaps it had been fixed already and Avidemux would now work.

I installed Avidemux and surprise - it worked. No more hangs. But as I had upgraded my machine at work from 13.04 to 13.10, I was not sure as of why it worked. I decided to try it out also in fresh installation into VirtualBox.

I installed a fresh 13.10 64-bit image into a new VM, then, without installing the VirtualBox additions, upgraded the packages and then installed Avidemux. The result - it hung, like before.

I started to look for differences between the two machines. Into my work machine, I had installed a mainline 3.12 kernel - maybe that was the solution? Nope, that did not help, the Avidemux in VirtualBox would still hang.

Then I noticed that the Avidemux in my main machine was displaying its menus inside the window, and not in the global menu bar. I also noticed, that gnome-terminal, for example, was also displaying its menus inside the main window, so I ran dpkg --get-selections|sort in both machines and compared the selections. And there was the answer - my main machine was missing the following packages:

unity-gtk2-module
unity-gtk3-module
unity-gtk-module-common

I removed these packages from my VirtualMachine installation and confirmed that Avidemux would no longer hang. Problem solved.

How I got the Precise Pangolin to play ball.

Ubuntu 12.04 Precise Pangolin

At first I thought that it would be impossible to get Ubuntu 12.04 to be usable in my workplace environment, but after a lot of tweaking and time, I have finally made the Precise Pangolin work exactly the way I wanted.

In the following I bring a set of changes I had to do to make it work.

1. The language problem (see my original post about this)
   The main problem is, that by choosing Estonian as your language, then in addition to the LANG environment variable, all of the LC_* variables are set to "et_EE.UTF-8" as well. This, as I wrote earlier, makes regexp [a-z] to exclude the letters coming after "s". The original solution I used previously was to set LANG back to English "en_US.UTF-8" locale in .profile, but this has the side-effect of turning everything into English.
   
   My current solution is much more elegant and will solve the regexp problem, while retaining the Estonian language - add these two lines into .profile:
   export LC_CTYPE="en_US.UTF-8"
export LC_COLLATE="en_US.UTF-8"

A quote to remember.

Babylon 5, “A Late Delivery from Avalon”, Marcus Cole:

 I used to think it was awful that life was so unfair. Then I thought, wouldn’t it be much worse if life were fair, and all the terrible things that happen to us come because we actually deserve them? So, now I take great comfort in the general hostility and unfairness of the universe.

WD My Book Live - a good external HDD and NAS

Western Digital My Book Live
AnandTech review

As I have previously posted, I have one of my older PC's (an Athlon 64 3200+ box) acting as a sort of NAS box, a Library, with four 320GB hard drives in RAID-5 configuration, resulting in approx 875GiB of useful redundant storage.
In addition to that, a few years ago, for media storing purposes, I put together a SATA-to-USB box and a 500GB WD Green drive in it.

However, now that I have bought Samsung's 46" LED-TV (EH5450W), which is also a PVR, I realized that I need an external HDD for the PVR functionality, and since it does not really have to be very big, the above mentioned 500GB box suit the purpose best.

I needed to move the stuff on that drive to somewhere else. The Library machine was out of the question, as my digital photos, documents and source code takes a lot of space there already and the RAID-5 redundancy is more important for this kind of content. It was therefore time to get a new external HDD.

At first I thought of getting WD My Book Essential drive, as I had some experience with it previously. Deciding to go with 3TB version, I was a bit worried about the max 30GiB/s transfer speeds inherent to USB2, as my machine lacks any USB3 ports.

Fortunately though, the Essential drives were currently out of circulation and I was offered a My Book Live instead. Back then, I only knew that the Essential is accessed through USB and the Live through the network. But in fact, they are very different beasts indeed. While the My Book Essential has only SATA-to-USB bridge in the box beside the drive, the Live is actually a whole independent PC, with 800MHz PowerPC 464 processor, 256MiB of memory and a Gigabit LAN port. It is running a ppc version of Debian Linux.

In addition to exposing the contents of the 3TB WD Green drive via Samba, also NFS is supported (though it is considerably slower apparently thanks to the NFS inherent encrypted data transfers). SSH access into the machine is also available. It also has web server, which includes nice UI for changing the settings of the NAS. And it has a media server, so my Smart TV picked it up automatically and was able to play back the media I had already copied there.

I was really surprised by this device.  The transfer speeds are much higher than USB2, reaching an average of 65MiB/s and going sometimes as high as 80MiB/s. The whole device is also almost totally silent, save some HDD head moving sound.
A bit worrying are the temperature readings, as after long coping stuff into the device, the hddtemp reports up to 65 degrees Celsius. Under normal operation, however, the drive is mostly sleeping and the temperature drops considerably.


I highly recommend this drive!

Setting up Flash Builder in Ubuntu 10.10

Adobe Flash Builder 4
Ubuntu 10.10 Maverick Meerkat

In a very sad news, Adobe has officially terminated the Linux Flex Builder project, which has stagnated in alpha status for more than three years now, olny receiving updates to extend the termination date. Regardless of the stated alpha status, the Linux Flex Builder plugin was very usable and quite stable, provided, of course, that you used it with Eclipse 3.3.

Since the current alpha 5 of the Linux Flex Builder plugin terminates at the end of this year and as Adobe has no intentions to allow anyone to use it past that termination date, I had to find an alternative.

Although JetBrains is welcoming all Flex programmers in Linux to switch to IntelliJ IDEA, I didn't want to learn yet an another IDE, especially one that is not open source nor free.

And finally I found the fb4linux project which works great (including the debugging).

The following is mostly a copy of the project wiki with only some additions to make it work with the Eclipse installed from Ubuntu repository.

1. Install Eclipse 3.5.x from Ubuntu repository;
   
2. Download the four FB4Linux* files from the fb4linux project download section, concatenate them to form FB4Linux.tar.bz2 and extract it somewhere.
   
3. Run Eclipse as root!
   sudo su; cd; eclipse
   
4. Select Window / Preferences and open the General / Capabilities section in the Preferences window.
   
5. Verify that Classic Update option is checked and close the preferences window.
   
6. Select Help / Software Updates / Manage Configuration
   
7. Right-click, select Add / Extension location... and browse to the Adobe Flash Builder 4 directory you extracted above.
   
8. Restart Eclipse as per its demands.
   

That's it, you can now open Eclipse from your user account and start coding Flex and AIR applications!

Small RegExp to reformat curly brackets Flex style

Adobe Flex 3

Find: [^( *)(.*)(\S+) *\{$]
Replace: [\1\2\3\n\1{]

In order to differentiate between Java code and JavaScript/ActionScript code I have defined for myself, that in Java the opening curly bracket should be at the end of the line where it starts, like this:
if (true) {
    System.out.println("true");
}

And in JavaScript/ActionScript code it should be alone in the next line, like this:
if (true)
{
    System.out.println("true");
}

The above code I am using in Eclipse to replace all such instances in the JavaScript/ActionScript code which does not conform to this rule.

How to remove higher version packages installed from another source in Ubuntu.

Ubuntu 10.04 Lucid Lynx
Linux Mint 9 Isadora

I am currently torn between plain Ubuntu 10.04 Lucid Lynx and Linux Mint 9 Isadora. Previously I have had both Linux Mint 7 Gloria and Linux Mint 8 Helena installed and liked them a lot. The main attraction for me is the MintMenu.

The last one was actually Ubuntu 9.10 with Linux Mint 8 repository set up and the packages from there installed over the Ubuntu packages. The reason for this was technical - I decided to try and set up RAID 0 array with my dual 750GB hard drives and unfortunately only Ubuntu had the so called alternate install available needed to install on RAID while the normal installer in Linux Mint 8 did not recognize this, even if it had been previously created.

Ubuntu 10.04 has made some considerable improvements in many areas over the previous version:

• My SB Audigy 2 ZS card works fine now even with the 5.1 setup switched on.
  
• The 3D desktop with my Radeon HD3870 works also out of the box (though it still recommends the ATI driver, but I have decided not to install it, see the next one for why).
  
• The LCD display powerdown works now (saving the backlight) with the automatic Mesa DRI R600 driver.
  Previously the ATI driver broke this LCD powerdown leaving the backlight on and I'm afraid it is still not fixed.
  
• The CD/DVD automatic mounting works now with correct volume labels appearing in /media instead of using the cdrom0, cdrom1 and cdrom2 directories previously (yes, I have three drives).
  The volume label automatic creation worked also in prior Ubuntu version but this functionality disappeared mysteriously after I installed Linux Mint packages over Ubuntu ones.
  
• The switching between text-mode and graphics is almost instantaneous - I wonder if the kernel graphics mode swithcing is already supported with ATI graphics or is this just very good MESA driver...?
  
• The resolution of the text-mode is perfect using the whole 1920x1200 screen with 240x75 characters!
  Previously in Linux Mint 8 or perhaps instead because of the ATI driver, the text-mode was an awful 80x25 and switching back messed up the graphics mode aswell...

After trying out the newly released Linux Mint 9 I felt that it looked and worked very similarly to the previous Linux Mint 8 while the Ubuntu 10.04 was considerable jump forward from 9.10 and therefore I decided to do similar configuration as before (though I'm not using RAID this time) to install Ubuntu 10.04 and then include the MintMenu from the Linux Mint 9 repositories.

What happened, unfortunately, is that I accidentally installed a lot of Linux Mint packages including grub and plymouth related things that affect the booting. I decided to remove these packages but alas, this seemed to be impossible without complete reinstall.
Most of the packages depended on others that depended finally on nearly all of the packages (the list was in hundreds of packages).
While each of these packages had counterpart in Lucid repositories, they all outranked the Lucid ones with higher version numbers.

The apt-get does not provide any way to downgrade nor to use different repository for reinstallation, the dpkg can reinstall any version, including downgrading to an earlier versions, but needs physical .deb files.

The answer was the forgotten aptitude package manager, which has somewhat similar syntax and with simple tasks can be used in place of apt-get, while having considerably more power.

And here is the step-by-step list of how I reverted these packages back to their Lucid Lynx counterpars:

1. First, in order to know which packages I need to switch back to Ubuntu versions, I listed those packages with dpkg:
   
   sudo dpkg --list | grep mint


2. Second, I downloaded the physical .deb files from lucid repository:
   
   aptitude -t lucid download libplymouth2 plymouth plymouth-label plymouth-theme-ubuntu-logo plymouth-theme-ubuntu-text plymouth-x11 python-software-properties software-properties-gtk


3. Third, I installed these versions with dpkg:
   
   sudo dpkg --install *.deb

That's it.

Quick howto: 32bit Firefox 2 within 64bit openSUSE 11.1

openSUSE 11.1

I am currently in the process of creating myself my very own openSUSE distribution using the excellent SUSE Studio.

As I've written previously, I am still using the old Firefox 2, which allows me to integrate with the KDE's excellent kprinter, something that was killed in the Firefox 3 branch.
Trying the distribution with VirtualBox I stumbled upon a problem with the Firefox 2 which I have actually seen previously. This 32bit Firefox 2 is unable to connect to any website when run from 64bit openSUSE 11.1 (probably in other versions as well).

The solution, however, is an easy one. The openSUSE is remarkably different from for example Ubuntu, as they actually include the 32bit environment out of the box in 64bit systems. This means that some 32bit versions of the various libraries are automatically installed. However, the distribution maintainers in openSUSE have, for some unknown reason, left out one of the necessary 32bit libraries: nss-mdns-32bit. Install this and the 32bit Firefox 2 works!

Gnome with Compiz and disabling Focus Prevention

Linux Mint 7 "Gloria"

One thing I've always hated about new Gnome after switching from openSuse and KDE is the focus prevention. I'm doing Flex development with Eclipse and whenever I run my application, a Firefox window is opened with the Flex compiled Flash file inside. The Compiz focus prevention, however, makes sure that the Firefox window is opened in the background. Also any and all Flash error dialogs (from the debug Flash player) appear only as the flashing taskbar button. It is ridiculous that I need to click on them to bring them forward.

Notoriously, neither Gnome nor Compiz ever mention the Focus Prevention in their setups. KDE 3.5 allowed the focus prevention settings in great detail to be modified via the KDE control center. Now, however, I've dug a bit deeper and I've found a rather simple solution to the problem.

It seems that all the hidden features are still present in Gnome and accessible via an app called gconf-editor. Just search for "prevention" in there (be sure to check the "Search also in key names" checkbox) and you should find a few matches.

Go to the key /apps/compiz/general/screen0/options/focus_prevention_level and change that value to zero to turn off the focus prevention. It's that simple!

How to make MPlayer from SVN work with Compiz?

Linux Mint 7 "Gloria"

One of the bigger problems when enabling Compiz for all these fancy UI tricks is the sad fact that video display will become, depending on which video output method your video player uses, either flashy, very slow or just totally unusable.

The root cause for this comes from the video output method. Those video output devices that render the graphics directly to the graphics card usually create a lot of flickering, since they effectively overwrite the rendered 3D graphics which in turn then overwrite the video window. The other video output methods think they render into a window become slow, since compiz will then have to copy the window contents to a 3D texture to be rendered by the video card.

With the Ubuntu 9.04 or (in my case) Linux Mint 7 "Gloria" x64 Edition the graphics driver installation is almost too trivial. At my workplace, with the ATI's Radeon X1650 card, the DRI R300 driver was installed automatically and effects were enabled out of the box with Gloria. At home I have ATI's Radeon HD3870 in which case I only had to confirm the installation of ATI's own Linux Catalyst drivers. The 3D effects along with full Compiz Fusion was already installed and ready to be swithced on.

But now became the real problem, how to make sure that watching video would still be possible with Compiz being active? I had heard a long time ago, at the dawn of Compiz, that there was one video player, which was patched to work with Compiz... I searched for it and was glad to find out that the player in question was MPlayer which is what I've always used in the past.

Since the patch is meant to modify source code, I had compile a patched version of MPlayer.

The compilation, however, blew up with some errors. The first error came from a module called IVTV... whatever that was I did not know. I simply disabled it with --disable-ivtv switch for ./configure script. The web told that there was a patch also to fix this, but I didn't care about that. The next error came from the x264 module which is a far worse thing, as this is one of the cornerstones of free MPEG-4 video. The web tells that this incompatibility of MPlayer 1.0rc2 and x264 is based on the fact that the aformentioned MPlayer source is outdated and does not support the much evolved current x264 version anymore. The only solution would be to compile the newest version of MPlayer, which currently is only available via an anonymous Subversion repository.

And here is the catch -- how to enable the fast Compiz video, yet keep x264 support also? The Compiz video patch is for the old MPlayer source, while the newer source supports the new x264?

I decided that the fortune favors the bold and tried to see if the patch could in any way be adopted to the new MPlayer source. The file in question is libvo/vo_xv.c. Of course there were numerous differences between the old and new versions of this file and this level of C code is well beyond my understanding (I generally dislike C).

Then I decided that it's worth a try to include the patched old vo_xv.c file with the rest of the new source code taken from Subversion. I expected it to blow up at ./configure script right away - it did not. I expected it to at least bail out with errors at compilation - it did not. It surely will crash once run - it does not. It works!

Since the following sources do a superb job at explaining at detail what packages you need to install, how to patch the file and how to compile the MPlayer, I will not copy their explanations, but merely add some remarks:

• A post in Ubuntu forums describing the original 1.0rc2 patching and compilation. Note that the patch file is no longer available from the link in that post (see below).


• A post in SmSpillaz with similar contents, but with a working link to the patch file. You can of course always use your friend Google to find alternate links to this file.


• An MPlayer download page with details how to download the new MPlayer from Subversion.

All you need to do really, is follow the first post up to the point of patching the original libvo/vo_xv.c file, back up the patched file, download the SVN version of MPlayer, copy the patched old file into libvo directory of the new source code overwriting the SVN version and follow the rest of the configuration, compilation and installation instructions from the first post but now within the new source code directory.

There's only one more glitch which needs to be rooted out. The fullscreen mode of MPlayer uses different screen mode which does not work anymore (reliably anyway). The solution is to turn off the special fullscreen mode by setting the property fstype=none. The only downside is that the Gnome panels remain visible that way.

I hate scripts and script languages!

Ubuntu 9.04

One of the big strengths of Linux is its open nature and the capabilities to customize everything to your personal needs and taste. Most of the times it requires a bit of learning, but other times it is a blessing to be able to create the needed tool yourself knowing that probably you are alone with your weird requirement anyway :-)

One way to make tasks less taxing and more convenient for yourself in Linux is to create scripts which do the tedious jobs for you. It is well known that in stark contrast to the Windows world, where the user interface of an app you see is the whole app, in Linux many apps you are using in the graphical desktop are actually only shells and the real work is done by other, command line apps. One of the better examples of this kind of shell is K3B, the KDE's CD and DVD burning application, which uses command line utilities such as cdrdao, wodim, growisofs, etc. to do all the dirty work.

This also means that you can use these same command line utilities to do that work directly without even launching the GUI or you can also create your own GUI if you prefer it that way. One big helper in creating scripts with better usability has been the kdialog command, giving a nice graphical touch.

During my time with Linux I have created a lot of scripts for various things I do. Since these scripts have been part of my activities for a number of years already, many having seen several revisions becoming more and more robust, I had no choice but to bring them all over to Ubuntu now that I'm using it. And this brings me to the point of this post - I hate scripts and script languages!

Though there are other bad traits with the scripts such as the lack of strong typing of variables, the most I hate scripts for their loose bindings. At its simplest a script -- as opposed to a compiled language program -- is a list of commands to be run in sequence.

But what happens if one of the commands is not installed and hence unavailable? Or if the command is of the wrong version and does not have the behavior you are using in your script? Or if the command itself depends on your environment and simply works differently? What if the command expects a file or directory to be at one place but in your OS they have been moved or installed in a different location? And since the UNIX doctrine specifies text to be the default data transmission medium, the commands become dependent on your current charset - is your $LANG set to UTF-8 or simply ISO-8859-1 or perhaps even ASCII instead?

Now, in case of a compiled program (unless it too uses external commands which is very unlikely), it should run just fine under different conditions (my experience here comes from years of Pascal and Delphi programming).
But in case of a script, you are most probably busted! Unless of course you test for every possible thing that can go wrong, but in such case you have probably spent 80% of time to write 80% of code that does not do what you wanted to do with your script in the first place. And the main power of the script is meant to be the speed and simplicity of doing what you want to do!

After my move from openSUSE to Ubuntu, I have stumbled upon this script nuisance several times already. The list of either direct or indirect problems encountered so far:

1. The regular expression [a-z] does not cover all of the alphabet if $LANG is set to et_EE.UTF-8 (as I've written before).


2. The KDE4 version of kdialog does not work with keyboard anymore (as I've also written before).


3. Ubuntu includes version 6.10 of the GNU's cut utility (used to cut chars away from a line of text) which does not work in UTF-8 space - special characters are counted as two. The solution is to get at least version of 6.11, which already supports UTF-8.


4. The KDE4 version of kfmclient (K File Manager) has a reduced set of commands. While previous KDE3.5 version could be used in scripts as the main file manipulation tool (coping, moving, renaming, etc.) the current version can only be used to open Konqueror browser windows. The new tool for file manipulation is kioclient and there's no backward compatibility - you have to change your scripts if you wish to port them to KDE4!

The fortunate thing is that KDE4 is installed directly into /usr, while KDE3 was always installed into /opt/kde3. So you can always install the KDE3.5 environment and point your $PATH first to /opt/kde3 to pick up the KDE3 version first. If for any reason you need to run KDE4 version, just use the full path, for example /usr/bin/konqueror.

Thats it for today.

A better way to stop ComboBox from resizing itself automatically...

Adobe Flex 3

This is an update to my earlier post about mx:ComboBox which, if set to a percentage width, will by default always take the width of the widest line it contains, widening possibly its container and/or changing the container to be scrollable.

The solution I suggested earlier was not enough for example in case the ComboBox was on a panel which was not visible when it was created. Now I am using a different solution.

Basically you extend the ComboBox and override the protected measure() method to cancel out the setting of minimum width set by the original code which is the cause the ComboBox will force its width upon its container.

The following code has some more changes, like resizing the dropdown list to the width of the longest line it will display and also refusing even programmatic focusing when either disabled or hidden.

Did you know, that if the focus already is on the standard ComboBox, then disabling or even hiding it does not block the keyboard input? For example, the [Down] and [Up] keys still change the items within the invisible ComboBox with all the corresponding events triggering, and [CTRL-Down] will still open the dropdown list even though the ComboBox itself is hidden and disabled?

public class ComboBoxEx extends ComboBox
{
   private var initDone:Boolean;
   public var preferredDropdownWidth:Number;

   public function ComboBoxEx()
   {
      super();
      this.preferredDropdownWidth = NaN;
      this.initDone = false;

      this.addEventListener("dropdownWidthChanged",
         function(event:Event):void { if (!initDone) preferredDropdownWidth = event.target.dropdownWidth; });

      this.addEventListener(FlexEvent.INITIALIZE,
         function(event:Event):void { initDone = true; });

   }

   /** A property indicating if the combobox should resize itself to the width of its contents */
   [Inspectable(category="General", enumeration="true,false", defaultValue="false")]
   public var resizeWidthToContent:Boolean = false;

   /**
    * A property indicating if the dropdown of the combobox should resize its width to the width of the longest
    * label within the combobox'es data, though never below the width of the combobox itself.
    * NOTE! that if you leave this property to its default value of true then the property dropdownWidth
    * will not be respected if it happens to be smaller than the width of the longest label in the combobox.
    */
   [Inspectable(category="General, enumeration="true,false", defaultValue="false")]
   public var resizeDropdownWidthToContent:Boolean = true;

   /**
    *  @private
    */
   override protected function keyDownHandler(event:KeyboardEvent):void
   {
      // Handle keyboard only if we are enabled and visible!
      if (super.enabled && super.visible)
      super.keyDownHandler(event);
   }

   /**
    *  @private
    */
   override public function setFocus():void
   {
      // Take focus only if we are enabled and visible!
      if (super.enabled && super.visible)
      super.setFocus();
   }

   /**
    * The super method determines the measuredWidth and measuredHeight
    * properties of the control.
    * This version will reset the measuredWidth property to the value of
    * UIComponent.DEFAULT_MEASURED_WIDTH unless the resizeWidthToContent
    * property is set.
    * @see mx.core.ComboBase#measure()
    */
   override protected function measure():void
   {
      super.measure();
      if (!this.resizeWidthToContent) measuredMinWidth = DEFAULT_MEASURED_MIN_WIDTH;
   }

   override protected function updateDisplayList(unscaledWidth:Number, unscaledHeight:Number):void
   {
      super.updateDisplayList(unscaledWidth, unscaledHeight);
      if (this.resizeDropdownWidthToContent && isNaN(this.preferredDropdownWidth))
      {
         var width:Number = getContentMaxWidth().width + getStyle("arrowButtonWidth");
         this.dropdownWidth = Math.max(this.width, width);
      }
   }

   public function getContentMaxWidth():Object
   {
      return calculatePreferredSizeFromData(this.collection.length);
   }

}

I also opened a bug about it.

Do not use your own language in Ubuntu!

Ubuntu 9.04

For crying out loud, I just spent a whole evening trying to figure out why the Gnome's Sensor Applet, which docks to the panel, does not work correctly and shows an error "Error compiling URL regex" instead of CPU core temperatures ?!?

The reason was simple - the function libsensors_plugin_init() within libsensors-plugin.c used regex which included [a-z0-9] in it and since in the alphabet of the Estonian language the letter "z" comes right after letter "s", any letters following them ("tuvxyz") will be excluded with the [a-z] regex. This included the "T" in "Core 0 Temp", so there you have it - "Error compiling URL regex" !

Aw gawd, I thought we are over this stupidity, but apparently not. Nobody probably knows the extent of this [a-z] bug and all the languages which are affected by it...

So the bottom line is:
Do not use your own language as the UI language if you want a working system!!!

My experiences trying out Ubuntu vs. openSUSE

Ubuntu 9.04 & openSUSE 11.1

I am and for a very long time already have been a die-hard SUSE user. However, most Linux people in my workplace are using Ubuntu.

Now that the Ubuntu 9.04, Jaunty Jackalope was released and my friend updated his laptop with it, I was excited to also try it out as it looked really good. The Gnome that comes with openSUSE somehow did not look that good...

Over the time I've almost always used KDE. I feel quite the same as Linus has once said - Gnome has taken the wrong turn in over simplifying its user interface. Over simplifying to the extent of removing also most of the necessary power-user stuff.

Unfortunately, nowadays the KDE 4 has also taken this bad route of over-simplifying the UI and probably for that cause, the KDE 4 team has re-written from the scratch many of the major KDE tools, such as Kate and KWrite, KDialog, Gwenview, Konsole, Amarok and many others. The new versions of the KDE 4 apps look, feel and are as useful as the namesakes of them at the time the KDE 2 started - raw and lacking many important features that I have grown accustomed to, plus introducing horde of bugs which should not be there at this point - an example would be KDialog, the new version of which does not return the correct response if you used only keyboard for selecting the button - with mouse it works, but not with keyboard. The old KDE 3.5 version, of course, works like a charm no matter how you use it.

One thing I found unbelievable in 64bit Ubuntu is the total lack of 32bit binary execution support out of the box. No 32bit ELF binary will run - bash, for example, says that the file you are trying to execute is not even there!
According to AMD64FirefoxAndPlugins page, in order to get the 32bit support you have to install several additional packages by hand and in order to run any 32bit binary you (apparently) need to modify some environment variables as well (better make a script). Additionally, for 32bit Firefox to be able to resolve names, you need some more installations to do.

Why on earth did the Ubuntu distro managers not create a normal dual architecture environment like openSUSE has - general 32bit support is given out of the box without any library path fiddling and only in case of more specific applications you need to select also the 32bit versions of any libraries they need.

Out of the box, there is one area where Ubuntu fares better than openSUSE - printers. I have two USB printers attached - an Epson Stylus Photo R220 and HP LaserJet 1020. Ubuntu noticed both of them, installing the Epson printer automatically with zero questions and for HP LaserJet only asking permission to install proprietary plugin direct from HP. Both work normally and to my biggest surprise, you can configure a ton of things for both printers including the CD printing capabilities of my Epson printer.

Right from the start I decided that I can not and will not use only Gnome tools and apps - I need at least Krusader, Gwenview, K3B, Konsole and KWrite. And I need the KDE 3.5 version of these apps - the ones that work as I've gotten used to. Again, out of the box there is no way to get KDE 3.5 apps. Fortunately there is the Pearson Computing KDE3.5 Repository for Ubuntu Intrepid and Above, which has a repository enabling the installation of KDE 3.5.10 environment and applications.

Another area where Ubuntu surprised me positively was the fact that KDE trash location was the same as the Gnome trash location. Deleting a file into trash from Krusader made it appear in Gnome's Trashcan... freedesktop.org has done a great job in unifying these two rivals making life easier for users of both.

So far so good, the system seems stable, printers work, audio works, my 32bit work environment (Java 6, Eclipse 3.4, Flex Builder, Firefox with Flash) is also working... so I guess I'll continue this experiment for a while longer.

How to make KPrinter work with Firefox 3?

openSUSE 11.1

In the recent past there have been several new software updates and upgrades which have almost infuriated me and forced me to revert to using old versions. The top two of these new things are:

1. Firefox 3 which cannot be persuaded to use kprinter anymore the way Firefox 2 was persuaded.


2. KDE 4 and its applications which totally and absolutely ignore the KDE 3 configuration with no apparent way to migrate the configuration either. Fortunately openSUSE still carries KDE 3.5 which I'm using currently.

The first of these items, however, is solved now, and here are the steps I took to make kprinter work with Firefox 3 (inspired by a post from this thread):

1. The first step is to edit the file /etc/gtk-2.0/gtkrc and add the following line to the end:
   

   gtk-print-backends = "lpr,file"

   
   This will make the lpr target appear in the Firefox 3 print dialog.


2. The second step is to switch the lpr command for kprinter command by executing:
   

   cd ~/bin
   ln -s `which kprinter` lpr
   



3. And the third and final step is to get rid of the Firefox 3 print dialog by opening the URL about:config and creating there the following new Boolean option:
   

   print.always_print_silent = true

And that's it - Firefox 3 now prints directly to kprinter.

NB! The only quirk I've found is that you cannot cancel the kprinter dialog - or it will hang Firefox 3 for some reason.

EDIT: After some testing, however, this way of printing seems to be very buggy - most pictures do not print at all and those that do end up on the printed page are heavily compressed with visible compression artifacts, the text is uneven and looks ugly.
It seems that the LPR printing uses the lowest possible resolution with no known way to fix this higher.

I've switched back to Firefox 2.0.0.20 for now until the Firefox 3 will play nice within KDE!

How to stop ComboBox from resizing itself automatically...

Adobe Flex 3

One of the many grievances I have with the otherwise wonderful Adobe Flex is the way the ComboBox component with percentage width resizes itself to accommodate the data which is loaded into it. There seems to be no switch to disable this behavior and yet retain the scalability of the ComboBox control itself.

The problem manifests itself best when you have, for example, two adjacent ComboBox widgets both set to be 50% width and you assign to the first a list of rows with short names and to the other a list of rows with long names - both widgets will resize themselves to a different width to accommodate their respective contents.

After searching for and failing to find a solution to this, I decided to try and write a workaround myself. I needed to fixate the width of the comboboxes right after they were first displayed and before they got their data. There was no need for the combobox to be resized later. In order to do this, I extended the original ComboBox and added the following code to it:

First I added a property via which you can turn on or off the new functionality:

/**
 * Property indicating if the initially measured width of the combobox will be
 * fixed as soon as the combobox has been layed out.
 * Only applies to combobox'es which have their width specified as a percentage.
 */
[Inspectable(category="General", enumeration="true,false", defaultValue="false")]
public var fixPercentWidthAfterInitialMeasurement:Boolean = false;

Second, to the constructor of the combobox, I added an event listener:

public function AComboBox()
{
   super();
   this.addEventListener(FlexEvent.UPDATE_COMPLETE, handleUpdateComplete);
}

Finally, I added the event handler method:

/**
 * This eventhandler is called right before the widget is drawn on screen - it's size has already been calculated,
 * but it has not yet received its contents, therefore now it is good time to convert its percentage width into
 * an explicit width, so that future data loaded into the combobox would not resize the compobox causing scrolling.
 */
private function handleUpdateComplete(event:FlexEvent):void
{
   if (this.fixPercentWidthAfterInitialMeasurement &&
      (!isNaN(this.percentWidth))) this.explicitWidth = this.width;
}

The idea here is to capture the event which is dispatched after the ComboBox size has been determined, right before the combobox is drawn on screen. Each ComboBox can have its width specified either explicitly in pixels or as a percentage of the containers width. When the width of the ComboBox is set as a percentage, I now assign its actual pixel width to the explicitWidth property. This will fixate the width and also stop the ComboBox from automatically resizing itself.

That solved the situation for me - the ComboBox retained its size after the initial layout.

A RAID success - with a Linux of course!

This is a continuation of my struggle to create a viable RAID-5 array for my sensitive data, started in this post and continued in this post
This saga continues in here.

The first tries at setting up Linux RAID system were failures for yet unknown reason. As suggested somewhere I created 20GiB partitions on each disk for /root filesystem to be RAID-1 (mirror) array and into the rest of the space I created partitions for /home filesystem in RAID-5. I tried this with both openSuSE 11.0 and the newest openSuSE 11.1, but in both cases the system was unable to boot, complaining about missing root filesystem. Sometimes the system would boot if I gave it the obvious root=/dev/md0 as the kernel parameter, other times even that did not help.

After some more experimenting I finally gave up trying to make the system totally redundant and resolved the situation by putting 10GiB on each of the four disks aside for /root, /swap, /usr and /tmp, accordingly, and using the rest of the space on disks as a RAID-5 /home partition. This worked as a charm of course.

I did extensive testing on the redundancy of the drive and found it to be rock solid. When one of the drive was disconnected, the MD reported an active RAID-5 system having with 3 out of 4 drives working... and that's it. Coping, moving and changing the content did not even seem to be any slower than before.

After some testing I re-attached the fourth drive and booted the system up again. This time the boot messages reported that the fourth drive of the array was rejected as not being fresh. Checking the mdadm command for any clues how to make the array complete again showed that the correct command was
mdadm --manage /dev/md0 --add /dev/sdd2
after that, the rebuilding of the fourth drive commenced in the background (it took a few hours) and the system was again working at peak efficiency.

Later I have been thinking what did I do wrong in the first two attempts and there are some things I can think of, but these must be verified first:

1. I used partition type Linux-RAID when creating partitions to be set up as RAID-1 later for the /root file system. Maybe I should have used the normal Linux partition type instead?
   
2. After creating the first 20GiB partition, I used the rest of the drive for the second partition on each drive. Since the drives were not identical, I probably should have created the second partition by specifying the size by hand, keeping it some gigs smaller than the rest of the space to insure the equality on all drives?
   

Anyway, now I am sure that I can restore all of my data, if one of the drives fails and as a bonus, I am also sure that I can insert a replacement drive and continue working without any need to reinstall the system.
I call that a success story!

More RAID rant

This post is an update to my previous post about my RAID-5 experience. Also see the continuation of this saga.

I found some disturbing problems with the nForce semi-software RAID-5 fault tolerance - without the fourth drive, the array seemed to work normally, but any modifications to the drive contents seemed to corrupt data on the drive. For example, in order to back up the data, after I had copied a few folders of data, I started to verify the copied data and delete it as I moved on. I always do it this way - first copy the data over, then run byte-per-byte verify (or MD5 hash verify) to see if the copy process worked and only after successful verify I will delete the data at the source.

Now, however, after I had verified and deleted the first folder of data from the RAID-5 array, the next folder showed an error in couple of files. Since they were picture files I was able to visually check the difference and the result was puzzling - the copied data seemed to be OK, but the source data on RAID-5 was corrupted. I tried a reboot, but the source file contents were still corrupted. I dismissed this as random happening and deleted the source (after all the errors were in the source data). Now the next folder showed even more corruptions and again the corrupted data was at the source.

A RAID-5 array with N drives saves actual data on N-1 of the drives and calculates a parity information to the remaining one drive. The data is saved in rotating stripes, so that the parity information of each next stripe is always on a different disk.
In case of reading data, if any one of the drives have failed, then N-1 times out of N, the the failed part of the data for each stripe is constructed by using the parity.

For example, when a stripe has parity on the drive 4, but drive 2 has failed, then the contents of drive 1 and 3 as well as the drive 4 with parity info is read and the contents of the failed drive 2 is constructed from the three other drives. Only in case the drive 2 held the parity data, is the reconstruction not needed.

The writing, however, is much more difficult. Assuming the above scenario, if the data is updated in the drive 1 region, then in addition to updating data on drive 1, the contents of drives 3 and 4 are read, the contents of drive 2 is temporarily reconstructed and new parity data is generated, which is updated on drive 4. If the data is updated in the failed drive 2 region, then similarly the data from drives 1, 3 and 4 are read, the original data of drive 2 is reconstructed, updated in memory and new parity is created and updated on drive 4.

Since the corruption was only few dozen bytes at a time and with no detectable pattern of changed bits I dismissed any further drive failure and assumed that one of the SATA cables could be faulty. I bought full set of new cables and tried again, recreated new RAID-5 array, copied a few dozen gigabytes of data on it. Made a copy of the data and then another copy of the same data. While the second copy was in progress, I disconnected one of the drives. At first, everything seemed to work normally. The array was in degraded mode and the copy was finished normally. Then I started to verify the copies of the data. At first things looked good, but then one of the bigger files had a few kilobytes of data totally differing in the middle of the file.

At this point I decided to power off the system, re-attach the drive I removed earlier and try the rebuilding. However, the nForce RAID BIOS reported error for the fourth drive and did not integrate it to the array. Booting Windows was also broken. After several attempts, somehow, I was able to boot and log into Windows at which point most things I tried to run either crashed, reported access violations or did not run at all.

For me, all this means that I will never trust the chipset semi-software RAID anymore. It could well be, that it is my motherboard that is at fault here, or the BIOS version... But still, having the RAID array is all about the ability to save important data on it without the need to worry if it will still be available after something happens to one of the drives. When the RAID array starts to corrupt data while in degraded mode then there really is no point in having a redundant RAID array.

If anyone has had a success using a chipset provided BIOS RAID array where a drive has failed in the middle of using it, please let me know the type and model of your motherboard.

Next I will try the software RAID offered by Linux. I'll try to do the same kind of trick - power down one of the disks while in use and see if and how much data I'll lose.