The most crucial aspect in infrastructure management is to know what’s going on. During the course of evolution of systems, from dedicated servers to virtual machines to containers, the granularity of data changed drastically. Observability of systems really needs to follow some kind of evolution.

We usually distinguish 3 big stages of evolutions in recent infrastructure history: - the dedicated servers era, each having unique names, treated like pets, each server hosting various services, or sometimes dedicated to an application, alone or in cluster. - the virtual machines era, where servers are still almost like pets, but just in a very large number, as services get split in different servers as a more general rule of thumb. - the containers era, where systems are considered as cattle, they have names related to their function including some numbers, they can live or die and be replaced without human intervention. This is where Ayla wants to go, with the transition to the kubernetes kind of architecture.

The traditional monitoring, checking pulse on individual servers, still is the norm in most systems used everywhere. But now that we have various micro-services and soon short-lived instances of applications, we need another layer of observability, which will report on systems activity as a whole rather than individual data. First, there should be a way to group systems activity by functional usage. We already have such a way to group data for application metrics in the APM provided by NeewRelic, but not at system level. Second, when we reach the third of evolution of the infrastructure, we need to have a visibility on the lifecycle of the herd. It becomes more about crowd control than individual checks.

The main obstacle I see in that evolution is that the more the infrastructure advances in stages, the more it escapes to the scope of a dedicated infrastructure team. Various departments spawn servers on their own as the need requires. But then there is no uniformity on the observability of the systems. So the monitoring needs to really get deep into the automation or the tools that are used to spawn new resources, and discoverability needs to address the grouping and agglomeration automatically somehow.

About observability, there are 3 main ways to observe a system. - First from outside, in a black-box approach, measuring traffic, response time, availability of a service. - Second at system level, for resources accounting, like cpu, memory, disk IO, etc. - Third is at application level, where data is collected by the application itself during its runtime activity, typically using New Relic and the plugins they provide.

But whatever the ways to collect data, there is a layer that is sorely missed at the moment, and it’s true in most systems that I have had the occasion to explore. It’s an overlay that would give in a glance a view of the sanity of an infrastructure. Some kind of quick way to see an overall picture in a glance. The split in micro-services also makes it hard to find a standard way to give a view of the links between services. In my opinion there is no substitute to custom coding here.

There are various emerging solutions and services that could help us address the situation in the case of kubernetes, though. The environment is so fertile, there are new projects popping up every day. I really need to get into that.

Recently I found my self again in that situation on a linux server. The partition where logs are stored went 100%. In such case, It’s clever top purge old useless logfiles. Typical move for me would be to run logrotate manually with

logrotate -f /etc/logrotate.conf

But I had a case where that was not enough. A developer forgot to remove a debugging output and the logs were just gathering way too much information, more than what I could free with some janitoring.

To avoid losing logs, we can move the logfile where there is space and replace the file with a symbolic link. That’s good enough for until the partition gets resized of the logs get cleaned. But when it’s done on a live logfile, the running process that writes into it still has the same file descriptor. The process has to be relaunched so the new fd can be taken in account, on the new partition, as instructed by the symbolic link.

So a colleague pointed out that could be done without restart by using gdb. It’s a pretty neat trick (if you have gdb installed on your production server, which may not always be the case, and for good reasons). Anyways I had it at hand, and here is the sequence:

touch /path/to/new/logfile

gdb -p pid

(gdb) call dup(1)
$1 = 3
(gdb) call close(1)
$2 = 0
(gdb) call open("/path/to/new/logfile", 2)
$3 = 1
(gdb) call close($1)
$4 = 0
(gdb)

This gave me the taste of digging up a little bit more on how gdb can interact with live processes.

1000 subscribers

After the 100th Greenruby last week, we get the 1000th subscriber to the email newsletter this week. Welcome George :) So for the occasion I refreshed the subscribers map on cartodb. About half of the subscribers are in the US, but there is a total of 73 countries represented, which is pretty neat. But this is based on the ip used for subscribing, so it’s not totally accurate.

Fighting logs pollution

Some time ago, on edition 82, I posted a link to my webalizer stats for the greenruby website. Fatal mistake, I had referrers stats enabled and I got somehow listed somewhere. The consequence was a lot of fake traffic with only purpose to get links listed in my referrers section. After removing this section from webalizer config, and moving the url, the fake traffic was still there. Like a blind wave. This was pretty annoying.

So I began to take some drastic measures. Because I didn’t have referrers anymore in my stats, I went to my apache logs (yes it’s an old server, still running apache2) and fire up a:

tail -5000 greenruby.org-access.log | cut -d' ' -f11 | sort | uniq -c | sort -n

This give me a nice view of the recent referrers. Fake ones are easy to notice. And that the fake traffic was only coming from a handful of ips, so I just made a few:

iptables -A INPUT -s <ip> -j DROP

The result was radically efficient. Certainly I had less volume but it’s now much cleaner. But it makes it clear that having clean volume stats for traffic on your website is not that easy. There are a bunch of fake traffic sources that you may not suspect.

New server soon

I got a new job 2 month ago at gandi, and there is some nice VPS hosting there. I can have a machine just for Green Ruby there. It can be neat and open some options. Green Ruby website is still very static. Various attempts to improve it with a search engine didn’t end up to something concluding yet. Maybe this time it will.

That also may be the occasion to build up a distribution system and move out of mailchimp. I looked around and all I found was php based. So I will probably just write my own ruby scripts, the mail gem looks great and the rest is a question of configuring a postfix with correct SPF and DKIM setup.

Je suis Charlie

Well, this tragedy that happened this Thursday in France impacts a lot of people. Somehow, it perhaps impacts me more than the average. When I was younger I went to art school to become a cartoonist, and finally I changed my mind. But I knew the work of the victims of this slaughter. They were icons in the French cartoon world.

It’s really sad, first because this is murder, second because the irrational impact it will have on society. This is crazy how a handful of brain-dead punks can bend history. Now France is going to become even more paranoid. I already was so uncomfortable when they began to send military with assault rifles wander in the train stations in Paris. It’s not going to get better.

We host some dev boxes on Digitalocean, and I tried varioous CLI because I like to stay in the console. Recently I was pretty happy to find Tugboat, as it saves default droplet size, region and image, and convers the whole API. Its fuzzy droplet name matching also can find great usage.

The only problem it has, is its counter-intuitive damn name (I forgot how to launch it 2 hours after using it the first time haha).

• https://github.com/pearkes/tugboat

Recently we switched from chef-solo to a chef-server setup on our infrastructure, a good occasion to refactor our recipes to better practices. I spent some time figuring out how to replace the fabric scripts I had for remote execution of actions on various servers, by using a knife plugin. That way I can just use knife abilities and don’t need fabric anymore.

So I created a new file in .chef/plugins/knife/ named apt.rb for a test:

require 'chef/knife'

module KnifeOpenvpn
  class Apt < Chef::Knife

    banner "knife apt <update|upgrade|simulate> <nodename>"

    deps do
      require 'chef/knife/ssh'
      require 'chef/node'
      Chef::Knife::Ssh.load_deps
    end

    def run
      if name_args.size == 2
        command_arg = name_args.shift
        server = name_args.shift
      else
        ui.fatal "Syntax: knife apt <update|upgrade|simulate> <nodename>"
        ui.fatal "Where <nodename> is a node name."
        exit 1
      end
      command = case command_arg
                when 'update'
                  'update'
                when 'upgrade'
                  '-y upgrade'
                when 'simulate'
                  '-y -s upgrade'
                end
      knife_ssh(Chef::Node.load(server).ipaddress, "sudo apt-get #{command}").run
    end

    def knife_ssh(server, command)
      ssh = Chef::Knife::Ssh.new
      ssh.name_args = [ server, command ]
      ssh.config[:ssh_user] = Chef::Config[:knife][:ssh_user]
      ssh.config[:ssh_port] = Chef::Config[:knife][:ssh_port]
      ssh.config[:identity_file] = Chef::Config[:knife][:identity_file]
      ssh.config[:ssh_gateway] = Chef::Config[:knife][:ssh_gateway]
      ssh.config[:manual] = true
      ssh.config[:host_key_verify] = Chef::Config[:knife][:host_key_verify]
      ssh.config[:on_error] = :raise
      ssh
    end

  end

end

Then I just run a knife apt simulate my-server to execute a apt-get -s -y upgrade on the my-server client node. Pretty useful. But I guess that’s only a beginning, I should extend it to run on various nodes at the same time and maybe inside threads or something like that, to match the fabric power.

We use S3 to backup various kind of files on MB. We use the very convenient backup gem for that (we still use 3.9.0).

• http://meskyanichi.github.io/backup/v4/

But at some point it appeared that backing up our audio recording was hammering disk IO on our server, because the syncer is calculating md5 footprint for each file each time a backup happens. When you get thousands of big files that is pretty expensive process (in our case 20k files and 50G total).

So I added a small trick there:

in Backup/models/backup_audio.rb

module Backup::Syncer::Cloud
  class Base < Syncer::Base
    def process_orphans
      if @orphans.is_a?(Queue)
        @orphans = @orphans.size.times.map { @orphans.shift }
      end
      "Older Files: #{ @orphans.count }"
    end
  end
end

Backup::Model.new(:backup_audio, 'Audio files Backup to S3') do

  before do
    system("/Backup/latest_audio.sh")
  end

  after do
    FileUtils.rm_rf("/tmp/streams")
  end

  ##
  # Amazon Simple Storage Service [Syncer]
  #
  sync_with Cloud::S3 do |s3|
    s3.access_key_id     = "xxx"
    s3.secret_access_key = "xxx"
    s3.bucket            = "bucket_backup"
    s3.region            = "us-east-1"
    s3.path              = "/mb_audio_backup"
    s3.mirror            = false
    s3.thread_count      = 50

    s3.directories do |directory|
      directory.add "/tmp/streams"
    end
  end
end

and in Backup/latest_audio.sh

#!/bin/sh
# isolate files changed in the last 3 days

TMPDIR=/tmp/streams

mkdir $TMPDIR
for i in `find /storage/audio/ -type f -cmin -4320`; do
  ln -s $i $TMPDIR
done

It creates a fake backup dir with links to the files that actually changed in the last 3 days and patches the syncer to avoid flooding the logs with orphan files. When sometimes S3 upload fails on one file (and it happens from time to time for ‘amazonian’ reason) it will be caught on the next daily backup.

The result was pretty obvious on our disk usage with our daily backups:

in the logs:

[2014/06/10 07:00:25][info] Summary:
[2014/06/10 07:00:25][info]   Transferred Files: 5
[2014/06/10 07:00:25][info]   Older Files: 22371
[2014/06/10 07:00:25][info]   Unchanged Files: 16
[2014/06/10 07:00:25][info] Syncer::Cloud::S3 Finished!
[2014/06/10 07:00:25][info] Backup for 'Audio files Backup to S3 (backup_audio)' Completed Successfully in 00:00:22
[2014/06/10 07:00:25][info] After Hook Starting...
[2014/06/10 07:00:25][info] After Hook Finished.

BigBlueButton is an amazing piece of free software, designed for virtual classrooms but can also be used for webinars, conferences, technical support and other uses. It gathers in the same place a video-conference system, with a shared PDF on which the presenter can doodle, an SIP bridge that make people can join using a phone, and a deskshare java applet for showing others pieces of your screen. Pretty neat. It’s around since 2007 and I had occasion to play with it with the Tikiwiki community.

And now I need to install one at Codegreen for our customer support and maybe some other uses. Great. I go for the last version 0.80 and it only works on ubuntu 10.04 (there are 25 different free software projects embedded in the beast, quite hard to maintain all on many versions, so they stick to the old one for now it seems).

So I get a new server just for that, and I try the install of the old 10.04 image. But it fails miserably for some new SATA drivers reasons, despite the more recent 12.04 works fine, the 10.04 just can’t see the hard drive. Bummer. After some fight then I need to use the alternative install method, using the VM image that BBB provides for VMWare.

Install the BBB VM

I don’t really like VMWare but I go get the free player and install it. The VMWare server that I wanted is discontinued they now only offer the VMWare player which requires a GUI install. Very lame, but ok. I get the VMWare Player image from their site and I install it without problem (the .bundle that I download needs to be executed after a chmod +x). Then I get the BBB VM 0.80 and launch it in the VMWare Player and tadaaa, it just works out of the box, magic.

Note that I installed it under my own account, no need for root with that thing.

Setup the startup launch

Because the Player is a GUI application, I looked around and found the way to launch it from commandline using the VIX (that is downloadable on the same page as the player and installable the same way). Then I can use vmrun to control the player, feels much better. I add in the /etc/rc.local of the host:

sudo -u mose /usr/bin/vmrun -T player start /home/mose/vmware/bigbluebutton-vm/bigbluebutton-vm.vmx nogui

Handling the routing

Here is the tricky part. The Player has 3 ways to handle th routing of the VM (which is very poor compared to the options provided by VirtualBox). and the bridge method was not working for me, because I have a static IP for the server on one card, a dhcp address on internal network on a second card, and the bridge option in VMWare don’t let you specify which interface is going to be bridged.

BBB says that you should use bridge, so the VM has its own IP and is like a first class member of your network, but it seemed that the NAT option was possible as far as you redirect some ports. So I’m going that way.

So I setup the NAT option for the VM, in the VM I fix the ip to 192.168.96.6 (that on the host I aliased to vm-bbb in /etc/hosts), and I add the subdomain I plan to use in the /etc/hosts of the VM (that way the VM bypass the DNS search and assumes that his own natted IP is matching the subdomain reachable from outside). Note that this etc-hosts trick will spit some errors at next step that needs to be run on the VM:

bbb-conf --setip bbb.mydomain.com

Then I need to redirect some ports, like the 22 for ssh access, the 80 for access to the main application, and some others that I have no clue what they are useful for (but I got it from some blog posts). So I install rinetd and add to /etc/rinetd.conf:

0.0.0.0 2222 vm-bbb 22
0.0.0.0 80   vm-bbb 80
0.0.0.0 1935 vm-bbb 1935
0.0.0.0 9123 vm-bbb 9123

and all is good. It seems to work fine, and I probably will have to test it more, but besides the sysadmin setup, the install of BBB was pretty straightforward. Great job !

Links

• Get VMWare player and VIX : https://my.vmware.com/web/vmware/free#desktop_end_user_computing/vmware_player/5_0
• BBB website : http://bigbluebutton.org
• BBB VM instructions : http://code.google.com/p/bigbluebutton/wiki/BigBlueButtonVM
• BBB VM Download : http://sourceforge.net/projects/bigbluebutton/files/