Actually, there were a few things that annoyed me on Telia. One was that I didn’t manage to change the machine that was connected to the ADSL connection. I followed all instructions I could find and even called support. All I had to do was switch off the modem for 20 minutes and turn it back on with a new box plugged in. Never worked. Three hours. Still no new IP. Of course, it may have been just me, but that has never stopped anyone from being annoyed before.

The other thing was that I upgraded to 24Mbit/s, paid more, but they didn’t send me a new modem. I also had a feeling that new customers got better deals than me.

So, I switched. The switch was surprisingly painless. The target date was April 30th, and I expected all connections to go down and I’d have to install the new shiny modem I received from Tele2. Nothing of the kind! All of a sudden my dhcp address changed and that was it!

Then my wife started complaining about her mail not reaching her friends. My first suspicion was user error, but then I realized I had set up my qmail mail server to route via Telia’s smtp-server. Doh. I can’t blame Telia for not routing mail from Tele2 customers.

The neat thing about my linux servers is that they never crave any attention. That’s also a bad thing. While documentation is fun of the highest degree, my own network is slightly less than 6 sigma standard in that regard. Luckily I remembered that all I had to do was change the server name in the /var/qmail/control/smtproutes file.

Unfortunately Tele2 have blocked all outbound traffic on port 25. Instead they require mail clients to use port 587 and also to authenticate.

I had received a paper with a user name and password. I also set up a mail account with a new password. Furthermore I spent significant time researching how to shove that into the smtproutes file.

All to no avail. Support closes at 8 pm, but then I’ll try to explain my problem. The support site mentions very little about how to configure an smtp-server, though it does mention a lot about configuring mail clients. They also have a robot help desk service named Sara, where you can presumably ask questions in natural language. She claims to answer questions about internet, broad band and so on. I asked “How can I relay from my smtp server?” and got the reply “Server… Not my thing really. I don’t know much about hardware”.

Tomorrow I’ll bother tech support. That’ll be fun…

–Jesper

My latest informational reorg included an idea to publish sound and video on a share that the kids could access. All that stuff used to sit on a share deep in a structure from a previous box I used to run.

Easy thing to figure out what to do; simply create a new share, call it Media and allow all the internal network read-access.  Mount it on a linux box with root access, then move everything from the old jesper/backup/jesper/bigdrive/private/media to that directory.

The only thing is that it’s not possible to simply rewrite the inodes when moving things from one nfs-mounted volume to another nfs-mounted volume. Every single bit needs to be copied.

It was a somewhat lengthy procedure to move everything onto the NAS, but at that time I had a gigabit card (still have it but it is not recognized by vmware esxi) and data was only going one way – from one box onto the NAS. This time, I was stuck with a 100MBit nic and data had to travel from the NAS, to the computer and then back to the NAS. Admittedly I didn’t have to watch, but I figured it would take me days to shuffle all of that data.

As the ReadyNAS runs a linux distro behind the scenes I figured the solution would be to gain terminal access to the NAS using ssh. Unfortunately that wasn’t on by default. I googled and found several solutions to fix the problem, ranging from opening the NAS and mounting the disks on another computer to using some odd “feature” in Apple File Protocol and hacking the crontab file.

Eventually I found a much simpler solution: Installing the official ‘enable ssh access’ patches.

On http://www.readynas.com/?page_id=94 you’ll find – at the bottom of the page – two patches; ToggleSSH and EnableRootSSH. Both need to be installed, and the NAS wants to reboot after each one.

ssh into the nas. Log in as root with your existing admin password. Voila! you’re in!

Now moving the remaining data was a few keystrokes away and obviously lightning fast using the good ol’ mv command.

Happy hacking!

–Jesper Hogstrom

I get logs mailed from my various systems on a daily basis. Several times a week they are filled with failed attempts to pick up mail from adam, annie, alonzo, bert, bill, caesar, daniel, david… as well as requests for /phpadmin/index.php /phpdbadmin/index.php, /phpsqladmin/index.php and so on.

The idea I dreamed up was fairly simple; either hook onto an existing intrusion detection system (IDS) like snort or write some other simple rules, maybe by analyzing log files from apache; “if an ip# requested 10 non-existing pages in 5 seconds, drop all packets from that source”.

I didn’t get very far in my thinking, before I realized someone else may have already done this. My first search yielded quite a few hits, including some stating it’s a bad idea to auto-block ip# based on IDS rules, mainly because of the many false positives.

However, one guy was right along my thinking with a Dynamic Firewall Daemon. The specs matched much of what I would eventually have set up myself (he said humbly). Other applications in the same area include fwsnort.

On the other hand, this article from linuxsecurity.com states “intrusion prevention is still at its early stage and there is no out-of-the-box product that will perfectly fits your requirements.” – though that was written ages ago (2006 I would guess based on the date of the first comment).

An insightful article from IBM gives some handy scripts, but not quite automation.

I readily admit to not being an expert in package filtering, but I do run firewalls, IDS and open only the ports required. What worries me a bit is that the majority of people out there are unaware of even the most basic facts about how to protect themselves from attacks. And even Pentagon have a problem securing their perimeter…

Before I completely take off on something else, I recommend the OnPoint Radio show on Cyber warfare. Much more high level than mere firewall scripts, but very eye-opening.

Anyway, if you have any experience on dynamic firewalls, open source or commercial, please feel free to share!

–Jesper Högström

To make things perfect, I wanted all the machines to run VMWare tools. On some machines that was as simple as selecting the appropriate menu item.

However, in some other cases it was simply not that simple. I needed to do the whole thing manually. For now, I’ll skip over the details of that and cut to the chase.

I tried to upgrade my kernel so I could recompile some vmware tools drivers when something went terribly wrong.

When I rebooted, all I had was a black screen saying “GRUB _” and it didn’t accept a single key stroke.

Luckily, it wasn’t the most important machine, but still – things didn’t look too good.

After spending some time on google, I found that others had had the same problem, and it seemed to having to do with the MBR (Master Boot Record) being messed up. It made kind of sense, but what then?

A thread I found on an Ubuntu site mentioned something called the Super Grub Disk. I downloaded it from http://www.supergrubdisk.org/ and uploaded it to the ESXi data storage.

Add a CD to the VM and point it to the ISO image.

Next – how to get the VM to change boot sequence?

Shut down the VM, edit settings and check the little box.

Save and reboot.

The interface is somewhat different – but plain menu driven! For me, it worked to simply select the menu options with help, select language, opt for Gnu/Linux and then “Fix boot of Gnu/Linux (GRUB)”. 0.1 seconds later I got an OK message, shut down the machine and switched back the boot order.

It just worked, The machine is back online. Now it’s time to beat the next generation in table tennis one more time before I install the VMWare tools.

A piece of advice: Take snapshots before you upgrade the kernel!

–Jesper Hogstrom

I alluded to my initial conversion issues in this post. While I learned more than I asked for, it seems it wasn’t quite necessary to use the lsilogic adapter. Buslogic worked just as fine…

First, credit where credit is due. I gathered all my information from http://hamed.dk/home/53-vm-ide2scsi and http://blog.creonfx.com/linux/howto-convert-vmware-windows-xp-ide-disk-to-lsilogic-scsi.

Having converted a couple of disks, I felt obliged to add the minor details I managed to add to the process.

Not that I’m a sissy, but one never knows. Close the virtual machine and backup!

My disks hadn’t been defragmented in a while – as I didn’t even know that was possible, so lets start doing that:

When I read the other posts, I was quite worried about not having a “small vmdk-file” that contained the disk description. Fear not! It will magically appear after the next step: Disk conversion. While we’re at it, give the disk a better name. Delete the old disk. We have a backup, remember?

The virtual physical layout of the disks will change when we convert it to a SCSI disk, but the number of sectors remain the same. Check the number of sectors you have now:

The number 25165824 will be used to create a dummy SCSI disk. YOUR NUMBER WILL NOT BE THE SAME!

If you’re like me and prefer to copy entire lines and avoid remembering numbers between screens, here goes:

Either way, look at the scsi.vmdk file and copy the lines the look similar to

Open virtualmachine1.vmdk and replace the lines that have the same keys.

Now we need to make some changes to the vmx-file. The vmx-files don’t seem to care about the order of values. My file was hard to navigate in, as keys were not collated. To the rescue:

Edit the vmx-file and remove/comment out the ide0:0* keys.

Find the section mentioning scsi, or place it in the correct alphabetical place:

I already had the key ’scsi0.present = “TRUE”‘ present. Having the key twice prevents the machine from starting!

If you changed names on the vmx-file, remove the old machine from the inventory in VMWare server console. Open the new vmx-file. Start it.

Important: If you did rename the files, you will get a question if you want to keep or replace the identifier. Most likely you want to KEEP the identifier. After all it is the same machine.

If everything worked as expected, delete the backup directory. You can also delete scsi.* from your VM directory. If it didn’t work out – restore the directory, start from top of page and read more carefully.

–Jesper Hogstrom

Looking at the file everything looked fine. The line

SV:123456:respawn:/command/svscanboot

was indeed present, but “qmailctl stat” kindly informed me the services weren’t running.

# qmailctl stat
/service/qmail-send: supervise not running
/service/qmail-send/log: supervise not running
/service/qmail-smtpd: supervise not running
/service/qmail-smtpd/log: supervise not running
/service/qmail-pop3d: supervise not running
/service/qmail-pop3d/log: supervise not running
messages in queue: 0
messages in queue but not yet preprocessed: 0

Finally I found a blog entry that pointed me in the right direction. It also included a pointer to this blog entry.
In short, it seems fc9 uses another boot sequence than fc8 and earlier versions. Note too that the two scripts in the above mentioned links look slightly different.
You probably want to read up on things by investigating

man events
man initctl

This guide also contains some useful information.
After some experimenting I got the script right, and it looks like

# svscan - daemontools
#
# This service starts daemontools from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5

stop on shutdown

respawn
exec /command/svscanboot

Put this in /etc/event.d/svscan and reboot – or simply test it by “start svscan”. Verify by invoking initctl list.

# initctl list
control-alt-delete (stop) waiting
logd (stop) waiting
prefdm (stop) waiting
rc0 (stop) waiting
rc1 (stop) waiting
rc2 (stop) waiting
rc3 (stop) waiting
rc4 (stop) waiting
rc5 (stop) waiting
rc6 (stop) waiting
rcS (stop) waiting
rcS-sulogin (stop) waiting
serial (instance)
sulogin (stop) waiting
svscan (start) running, process 1862
tty1 (start) running, process 1749
tty2 (start) running, process 1745
tty3 (start) running, process 1747
tty4 (start) running, process 1738
tty5 (start) running, process 1743
tty6 (start) running, process 1751
vpnc-cleanup (stop) waiting

–Jesper Högström

Then one day it struck me – what if the developer is away for a week and the customer files an urgent bug report… Who gets notified? The answer is: The developer who isn’t there.

A simple solution is to force all project managers to check on their projects every day to make sure reports don’t fall between chairs. There are two things wrong with that solution imho:

* It’s error prone as it assumes people will always do what they are told and never forget.
* It doesn’t entail a single piece of new technology…

As I already had some scripts doing data extraction from the mantis database I figured something along the following would work:

* For each project; list all managers with an internal email address.
* For each project, select new bugs and list them with some info about the bug.
* Send the bug reports to the managers for the project.

Sounds easy enough, doesn’t it.

Here’s the script to get all the project managers

Save it as projectmanagers.sql

Worth noting is the filtering on email addresses. In some cases we assign manager rights to the project champion on the customer side. Yu may or may not want them to get the bug report email. Your call.

Here’s a shell script to get the list and then call an awk script to do the rest.

Save it as bugreports and make it executable with chmod +x bugreports
There are some things worth noting:
* I specify the path to the mantis configuration file. This is to avoid having to specify the db credentials twice. I guess I could have extracted the php-variables using php, but I don’t know much php. This works, assuming you formatted your config_inc.php file in a reasonable way.
* The bulk of the work is obviously done elsewhere. You do know about awk, don’t you?
* Getting the slashes right in mantisbaseurl took a few minutes

Here comes the awk script.

Save it as bugreports.awk

This script will operate on all lines in the /tmp/projman file (with all the project managers in it) and do the following:
* Replace some stuff in a file named bugreports.rsql using sed.
* Put mail headers in a text file
* Execute the expanded sql-file and put the result in the text file that makes up the mail
* Could the number of lines in the mail
* If there are any lines above and beyond the headers, send the mail to the manager.

Save it as bugreports.rsql

This will select all bugs in status new (i.e. resolution = 10) that are younger than 5 days and come from a project with the number PROJNUM. We actually use sed to replace PROJNUM with the actual project id and also BASEURL with the g_path variable in config_inc.php.

The mail you send will look something like

123 (http://mydomain.tld/mantis/view.php?id=123)      Project_1       \
developer1 70      2008-10-09      1       \
System stops when power cord is removed.
148 (http://http://mydomain.tld/mantis/view.php?id=148)      Project_1       \
developer2  50      2008-10-08      2       \
Saving big images consumes a lot of disk.

I put line breaks in there manually

I admit it could be done somewhat more glitzy with fireworks and images and what have you. However, it’s merely a status report and there is no need to add more – possibly the column headers.

The scripts make very few assumptions about how you have configured things. Feel free to change what doesn’t suit you (like age of bugs, status, who gets mail).

I run my script nightly by symlinking it to /etc/cron.daily.

–Jesper Högström

Here’s a shell script I wrote to handle the situation.

What the script does is
* Get the list of registered machines
* For each machine,
* if it’s running – suspend, backup, restore
* else backup

Much ofthe time was spent handling the completely silly idea of putting spaces in file names.

Note that you must probably change the backup target directory high up in the script. I mounted my ReadNAS+ with 1.5TB space for backups…

Put the script somewhere, make it runnable and symlink it into /etc/cron.daily or cron.weekly

chmod +x backupvms
ln -s backupvms /etc/cron.daily

It is worth noting that I ran into problems as not all services on all machines handled the suspend/restore cycle well. The backing up worked just fine, though

–Jesper Högström

Rsync is is a protocol and a client/server pair that allows you to rapidly synchronize two file sets. The main use is to create a mirror image of set A onto box B. The neat thing is that it can be configured to skip files that are equal based on check sum, size or timestamp. And if the file is flagged for synchronizing, the rsync protocol copies only the changed bits. This article does not intend to describe the rsync protocol, so I recommend reading one or all of “how rsync works“, “the rsync algorithm” or the wikipedia article on rsync.

Here, we’re going to get a bit more pragmatic <g>

First, rsync is offered as an option when you install your linux box. You can easily check if it’s there:

# rpm -qa|grep rsync
rsync-2.6.9-5.fc8

Yep, I’ve got it. If you failed to make the correct selection then, fear not! Simply install it with yum.

yum -y install rsync

The -y switch prevents silly questions.

Now, rsync runs under xinetd, so you’ll need that too.

# rpm -qa | grep xinet
xinetd-2.3.14-15.fc8

If nothing, then

yum -y install xinetd

This xinetd/rsync combo means you need to turn on rsync under xinet. Edit /etc/xinetd.d/rsync and make it look like

# default: off
# description: The rsync server is a good addition to an ftp server, as it \
#       allows crc checksumming etc.
service rsync
{
disable = no # <== This is what needs to change!!
flags           = IPv6
socket_type     = stream
wait            = no
user            = root
server          = /usr/bin/rsync
server_args     = --daemon
log_on_failure  += USERID
}

Make sure the firewall doesn’t block the rsync port, per default 873.

If you have a firewall script, add incoming tcp traffic on port 873, or simply edit /etc/sysconfig/iptables and add the line

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 873 -j ACCEPT

Restart iptables with

service iptables restart

That’s it. You have an rsync server set up. Albeit not configured. To do that, open /etc/rsyncd.conf and add the folders you want to expose.

Naturally, it’ll look somewhat different for you, but essentially this will work:

uid = nobody
gid = nobody
use chroot = no
max connections = 4
log file = /var/log/rsync.log

[pearadio]
comment = Podcasts
path = /home/jhogstrom/podcasts
read only = true
dont compress = *.mp3
uid = jhogstrom
exclude = .peapod

Test the setup by using the rsync client:

rsync rsync://localhost

The output will look something like

# rsync rsync://localhost
pearadio        Podcasts

Next time, we’ll discuss rsync clients on windows boxes.

–Jesper Hogstrom

I spent some time pouring over the various things that I could have done wrong. Email server… No, emails were sent out on bug added. Maybe some new config setting I should have added? Browsing through config_defaults_inc.php I did indeed find some settings I really liked, but nothing that had an impact on the problem of sending of emails when bug notes were added.

Finally I made some wild google searches, and found the solution! The mantis bug database knew about this case as reported in this bug.

What you need to do is to update the file core/bugnote_api.php as per this subversion diff:

First, make a copy of bugnote_api.php to bugnote_api.php.org.

Open bugnote_api.php in some decent editor, like nano.

Find line 102 and replace

function bugnote_add ( $p_bug_id, $p_bugnote_text, $p_time_tracking =
'0:00', $p_private = false, $p_type = 0, $p_attr = '', $p_user_id =
null ) {

with

function bugnote_add ( $p_bug_id, $p_bugnote_text, $p_time_tracking =
'0:00', $p_private = false, $p_type = 0, $p_attr = '', $p_user_id =
null, $p_send_email = TRUE ) {

After saving your developers will get email notes when customers add notes to bugs.

The fact that this fix is checked in seems to indicate it wont happen in future releases.

–Jesper