This week is the 100th edition of Green Ruby, it’s also the first one of 2015. This week the Linux Outlaws broadcasted their last show, and Rebecca Watson left the Skeptics Guide to the Universe. Those are 2 podcasts that I never list in the Green Ruby list (they are not really on topic) but I rarely miss any of them. Too bad they leave. I thought about it too, but no, I will stay.

So, happy new year or, as we say in french: Bananier!

For 100 weeks already I gathered those links and sent them to whoever wants them. Every damn single week. Just for the fun of it. This is a kind of personal kata, it keeps me in the loop. I just make my exercise in a way I can share it. And my streak of never missing one week is a kind of personal challenge.

There is now 971 subscribers to the mail newsletter, a whole lot of people prefer to subscribe using the RSS feed, and the website has 8 or 9k visits per month. But this is still a side project, with no business behind it. Just one single guy and a couple of friends sharing the result of their weekly review of the news of their craft.

So, as you can notice, the edito becomes a rant, and is going to get a bit longer. For a while I was trying to keep it short just because it sits on the top of the letter. What matters are the links and not the ranting. By moving the blah blah at the bottom, I free myself from format constraints and can unleash my writing desires. muahaha. Not that I have much to say. But you can consider this new section of Green Ruby as the last screen displayed in The Big Bang Theory, where producers display some random thoughts, rants or anecdote. You need to pause the video for being able to read them. Yeah I know I could just blog like everybody else. But it’s just convenient for me to include this writing in my weekly process.

I also improved formatting for text-only diffusion. There is still something bugging me about the link tracking, which is mandatory when using the free tier of Mailchimp service. I guess I will have to find a way around: either I will get away from mailchimp, either I will find a sponsor to pay for removing the tracking (sounds funny when I say it that way, no?). Goddamn I hate tracking-by-default. Maybe I will end up setting up my own postfix with some mailman. Or use something? If you know a good free software self-hosted solution for newsletters feel free to drop me a mail.

It’s amazing how many times I saw dev add google analytics on their templates and at the end nobody was going to check stats except for volume metrics. If you only need volumetric data, use your web server logs! If you really need fine metrics, why not consider using self-hosted analytics like Piwik?

Don’t get me wrong, I don’t hate analytics. What bothers me is what is made of those data. Many companies chose as a business model to consider the user as a merchandise. And most of the time they are not clear about it.

Oh, one last word, next week I will prepare an option so you can chose to receive only the links and no rant. Or only the rant and no links. Maybe I will make each section of the letter opt-out too, if you never listen to podcasts why should you get those, right? I just like choice.

When you live in your console, it’s always good to find ways to stay there. The column command was recently brought to my attention by https://sysadmincasts.com/episodes/36-cli-monday-column-and-tr and despite years of console life, godamn I was unaware of this simple tool.

It is very useful for displaying csv files in a readable way, for example:

$ head members_export_0bfc48c38f.csv
LATITUDE,LONGITUDE,CC,REGION
24.7798,120.93,TW,HSZ
52.0927,5.1158,NL,UT
25.0389,121.509,TW,TPE
47.6836,-122.122,US,WA
24.1403,120.682,TW,TXG
37.4055,-122.078,US,CA
37.4055,-122.078,US,CA
37.4055,-122.078,US,CA
26.1405,-80.1738,US,FL

yuck. Let’s prettify it.

$ head members_export_0bfc48c38f.csv | column -t -s,
LATITUDE  LONGITUDE  CC  REGION
24.7798   120.93     TW  HSZ
52.0927   5.1158     NL  UT
25.0389   121.509    TW  TPE
47.6836   -122.122   US  WA
24.1403   120.682    TW  TXG
37.4055   -122.078   US  CA
37.4055   -122.078   US  CA
37.4055   -122.078   US  CA
26.1405   -80.1738   US  FL

ohh, my world just changed!

JSON prettifying is a reccurent topic, every coder needs it to keep some sanity. My way to do it has been something like this for a while:

ruby -rjson -ropen-uri -rawesome_print -e "ap JSON.load(open('http://jsonip.com/'))"

{
       "ip" => "59.115.120.166",
    "about" => "/about",
     "Pro!" => "http://getjsonip.com"
}

because when you are rubyist everything can be solve with ruby, obviously.

But well, then one day I did

pip install pjson

and then

curl -s http://jsonip.com | pjson

{
  "Pro!": "http://getjsonip.com",
  "about": "/about",
  "ip": "59.115.120.166"
}

Faster, easier, happier.

We host some dev boxes on Digitalocean, and I tried varioous CLI because I like to stay in the console. Recently I was pretty happy to find Tugboat, as it saves default droplet size, region and image, and convers the whole API. Its fuzzy droplet name matching also can find great usage.

The only problem it has, is its counter-intuitive damn name (I forgot how to launch it 2 hours after using it the first time haha).

• https://github.com/pearkes/tugboat

There is a shell tool that I install systematically on any new machine/server, it’s tree.

apt-get install tree

or

brew install tree

Its very basic usage is to show the content of a dir in an arborescent way, and there are a lot of options. It can be a good alternative to find sometimes. But I mostly use it for having a quick overview of the content of a dir.

mose@momac > ~ > tree projects/lolita
projects/lolita
├── Gemfile
├── Gemfile.lock
├── README.md
├── config.default.yml
├── config.yml
├── lita-faria
│   ├── Gemfile
│   ├── LICENSE
│   ├── README.md
│   ├── Rakefile
│   ├── lib 
│   │   ├── lita
│   │   │   └── handlers
│   │   │       └── faria.rb
│   │   └── lita-faria.rb
│   ├── lita-faria.gemspec
│   └── spec
│       ├── lita
│       │   └── handlers
│       │       └── faria_spec.rb
│       └── spec_helper.rb
├── lita_config.rb
├── log 
└── tmp 

9 directories, 15 files

Some options:

mose@momac > ~ > tree projects/lolita -hfih -I 'spec|lib' --du
projects/lolita
[ 164]  projects/lolita/Gemfile
[1.2K]  projects/lolita/Gemfile.lock
[1.2K]  projects/lolita/README.md
[ 177]  projects/lolita/config.default.yml
[ 222]  projects/lolita/config.yml
[2.6K]  projects/lolita/lita-faria
[  39]  projects/lolita/lita-faria/Gemfile
[1.0K]  projects/lolita/lita-faria/LICENSE
[ 357]  projects/lolita/lita-faria/README.md
[ 117]  projects/lolita/lita-faria/Rakefile
[ 769]  projects/lolita/lita-faria/lita-faria.gemspec
[1.2K]  projects/lolita/lita_config.rb
[ 102]  projects/lolita/log
[ 102]  projects/lolita/tmp

 7.4K used in 3 directories, 11 files

man tree for more (but the basic tree view is the real deal, for more detailled listings, find is better).

There is a package that is not installed by default that I now grab systematically, it’s called parallel.

apt-get install parallel

or on mac

brew install parallel

The parallel command is a pretty complex beast, but the package also comes with the sql command, which simplifies queries to any database from the console. There is just one command whatever the sql backend you use.

I first setup my dburl in .sql/aliases as:

:dbmb mysql:///managebac
:dboa mysql:///openapply
:dbis psql://isis:password@localhost/intersis

And I can then use those aliases for quick sql commands:

sql :dbmb "select count(*) from users"
sql --show-processlist :dbis

Check http://www.gnu.org/software/parallel/sql.html for more examples of use. Also check http://www.gnu.org/software/parallel/package it’s very powerful, but a bit complicated to catch it, maybe I will provide some examples of practical use in a later tip.

Recently we switched from chef-solo to a chef-server setup on our infrastructure, a good occasion to refactor our recipes to better practices. I spent some time figuring out how to replace the fabric scripts I had for remote execution of actions on various servers, by using a knife plugin. That way I can just use knife abilities and don’t need fabric anymore.

So I created a new file in .chef/plugins/knife/ named apt.rb for a test:

require 'chef/knife'

module KnifeOpenvpn
  class Apt < Chef::Knife

    banner "knife apt <update|upgrade|simulate> <nodename>"

    deps do
      require 'chef/knife/ssh'
      require 'chef/node'
      Chef::Knife::Ssh.load_deps
    end

    def run
      if name_args.size == 2
        command_arg = name_args.shift
        server = name_args.shift
      else
        ui.fatal "Syntax: knife apt <update|upgrade|simulate> <nodename>"
        ui.fatal "Where <nodename> is a node name."
        exit 1
      end
      command = case command_arg
                when 'update'
                  'update'
                when 'upgrade'
                  '-y upgrade'
                when 'simulate'
                  '-y -s upgrade'
                end
      knife_ssh(Chef::Node.load(server).ipaddress, "sudo apt-get #{command}").run
    end

    def knife_ssh(server, command)
      ssh = Chef::Knife::Ssh.new
      ssh.name_args = [ server, command ]
      ssh.config[:ssh_user] = Chef::Config[:knife][:ssh_user]
      ssh.config[:ssh_port] = Chef::Config[:knife][:ssh_port]
      ssh.config[:identity_file] = Chef::Config[:knife][:identity_file]
      ssh.config[:ssh_gateway] = Chef::Config[:knife][:ssh_gateway]
      ssh.config[:manual] = true
      ssh.config[:host_key_verify] = Chef::Config[:knife][:host_key_verify]
      ssh.config[:on_error] = :raise
      ssh
    end

  end

end

Then I just run a knife apt simulate my-server to execute a apt-get -s -y upgrade on the my-server client node. Pretty useful. But I guess that’s only a beginning, I should extend it to run on various nodes at the same time and maybe inside threads or something like that, to match the fabric power.

When working on a remote server, I use less instead of tail or vi to check files, the more I can.

filter

• with &searchstring it will hide all lines not matching searchstring, like a very simple grep from the console
• just & with nothing to get back to full view

edit file

• when in less /etc/hosts if I find out I need to edit it I just type v it opens it in vi
• when you quit the editor you are back in less

follow mode

• when in less production.log, if you want to follow incoming changes type F it will become like a tail -f but it keeps the syntax highlighting that was made with a / search - when in follow mode with F you can ctrl-c any time and examine the new stuff
• note that if you are in filter mode the follow respects it (like if you did a grep something/var/log/auth.log | grep 'Invalid user')

multifiles

• you can open several file and navigate like in vim with :n and :p
• when less is open you can open a new file with :e

call options inside less

• you can call any launch option from inside less, one I use often is -S that will enable/disable wrapping of long lines, very useful when parsing web server log files
• to disable/enable search highlighting just -G
• any other launch option can be called from inside less, pretty convenient

We use S3 to backup various kind of files on MB. We use the very convenient backup gem for that (we still use 3.9.0).

• http://meskyanichi.github.io/backup/v4/

But at some point it appeared that backing up our audio recording was hammering disk IO on our server, because the syncer is calculating md5 footprint for each file each time a backup happens. When you get thousands of big files that is pretty expensive process (in our case 20k files and 50G total).

So I added a small trick there:

in Backup/models/backup_audio.rb

module Backup::Syncer::Cloud
  class Base < Syncer::Base
    def process_orphans
      if @orphans.is_a?(Queue)
        @orphans = @orphans.size.times.map { @orphans.shift }
      end
      "Older Files: #{ @orphans.count }"
    end
  end
end

Backup::Model.new(:backup_audio, 'Audio files Backup to S3') do

  before do
    system("/Backup/latest_audio.sh")
  end

  after do
    FileUtils.rm_rf("/tmp/streams")
  end

  ##
  # Amazon Simple Storage Service [Syncer]
  #
  sync_with Cloud::S3 do |s3|
    s3.access_key_id     = "xxx"
    s3.secret_access_key = "xxx"
    s3.bucket            = "bucket_backup"
    s3.region            = "us-east-1"
    s3.path              = "/mb_audio_backup"
    s3.mirror            = false
    s3.thread_count      = 50

    s3.directories do |directory|
      directory.add "/tmp/streams"
    end
  end
end

and in Backup/latest_audio.sh

#!/bin/sh
# isolate files changed in the last 3 days

TMPDIR=/tmp/streams

mkdir $TMPDIR
for i in `find /storage/audio/ -type f -cmin -4320`; do
  ln -s $i $TMPDIR
done

It creates a fake backup dir with links to the files that actually changed in the last 3 days and patches the syncer to avoid flooding the logs with orphan files. When sometimes S3 upload fails on one file (and it happens from time to time for ‘amazonian’ reason) it will be caught on the next daily backup.

The result was pretty obvious on our disk usage with our daily backups:

in the logs:

[2014/06/10 07:00:25][info] Summary:
[2014/06/10 07:00:25][info]   Transferred Files: 5
[2014/06/10 07:00:25][info]   Older Files: 22371
[2014/06/10 07:00:25][info]   Unchanged Files: 16
[2014/06/10 07:00:25][info] Syncer::Cloud::S3 Finished!
[2014/06/10 07:00:25][info] Backup for 'Audio files Backup to S3 (backup_audio)' Completed Successfully in 00:00:22
[2014/06/10 07:00:25][info] After Hook Starting...
[2014/06/10 07:00:25][info] After Hook Finished.

Making and publishing your own gems is so incredibly simple. Here is my setp by step process when I want to publish one:

Make a Rubygems account

• first ensure create an ccount on https://rubygems.org/
• initialize your credentials: ``` curl -u username https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials Enter host password for user ‘username’:

chmod 0600 ~/.gem/credentials ```

Prepare the code

These steps are my typical path, nothing is really mandatory there.

• create a repo at github (or wherever you want)
• create a base gem skeleton bundle gem mycoolgem
• update the README.md, add some minimal information. The clearest you explain what it does the best chances people will adopt it. Add cypirght information
• create a CHANGELOG.md because when you publish something you need a clear log of your version changes
• enable the tracking on codeclimate, travis, coveralls, gemnasium
• add badges from http://shields.io in the README.md because it helps to qualify the state of the code quickly
• when the gem code don’t need to have access to the version number, I found convenient to just remove the version file and add version in the .gemspec from the changelog file with spec.version = File.read(File.expand_path('../CHANGELOG.md', __FILE__))[/([0-9]+\.[0-9]+\.[0-9]+)/] (so then when I version bump there is only one file to edit)
• write the code (or copy it from the private app you extract it from)
• add a spec/ or test/ dir and write some tests and try to reach a decent coverage

Release it

• add the date in the changelog for the current version
• rake release
• bump the version to the next increment in the changelog (and version file if any)

Overall with the initial bundle command and the final rake task for release, it’s pretty straightforward.

Also check http://guides.rubygems.org/make-your-own-gem/