Dev, Manage, Deploy

Open source efforts of january

Mon, 06 Feb 2012 11:55:55 +0100

I’m hoping this will be a returning post of mine, I’ve started a bit during the chrismas holidays and inventoring what open source projects I used the most and could most easily contribute to. The first aim was actually just to get involved in something that used git, so I could have some practice before switching to it entirely at work. But I realised this should be more continous than that. And also, its been a few years since I contributed or at least tried to squirrelmail and seagull.

Since I do mostly web development, the linux-part didnt feel so likely, or contributing to gnome seemed far away. :)

I’ve been checking out Openphoto on github for a while since I’ve been wanting a self-hosted good and flexible image hosting thingie. So I started there, and actually found a bug, which I could fix with one single simple line. So I sent a pull request, and just like that I was more or less started.

After that I realised phpmyadmin is my everyday tool more or less, I started by setting up the latest master revision and checking the bug-list, to see if there was something that I could easily fix and get acquainted with the codebase. Since I set out to use the dev-master-version in my everyday use I quickly found some inconsistencies(and also found some new neat features), so I’ve filed a few bugs and sent a few patches. Felt good, I’ll definitly try to keep this up.

And timewise, I might have put in like 8 hours during january for this. That seems reasonable, I’ll start tracking it with letsfreckle as well though, just to make sure it doesnt get to much and also to make sure I actually get something done on that time.

Every day, new tasks come in with email, and after a days work,...

Sun, 05 Feb 2012 21:45:00 +0100

Every day, new tasks come in with email, and after a days work, its hard to determine if you really accomplished anything. So I figured I would track everything that I mark as a todo in my inbox.

I also needed to separate private stuff with work-stuff, and since I have a privat email with all that stuff and a work-email for all work-related, it was easy.

I use labels in thunderbird to mark things with different status, the mailserver is running dovecot and figuring out where dovecot stored the labels required some googling. Since mails are stored in Maildir format, the label is stored in the filename of the email. I just needed to find what character represented my label. Theres a special file called dovecot-keywords in the Maildir-folder, that lists all labels and their character. Finding the correct one needed some trial and error, since the labels name is not synced over to imap.

See the script below, which uses grep and some bash stuff to find the right emails, and then uses wc to count them. Really simple and could probably count wrong in some cases, but its good enough for some basic tracking.

The big dip in the beginning of week 2 was basically a change in how the script worked, and some inboxcleaning.

Tracking redmine ticket overview with munin

Tue, 29 Nov 2011 22:03:03 +0100

I got frustrated over the overview in time with graphing plugins for redmine. The ones I tried only showed me a growing curve of added tickets, it didnt show any progress or positive feedback.

I wanted the positive feedback, so I created a quick script which took the atom ticket-list and parsed through it and counted all tickets depending on status:

Just add it to the munin plugin-folder, and it will “just” work. It would be possible to adjust it a bit, so the url comes from the munin plugin config.

Logging incoming udp-messages with node.js

Tue, 29 Nov 2011 21:20:00 +0100

Last week, I realized I needed a quick way to set up logging of incoming udp-messages. My first idea was using netcat which is kind of a swiss army knife when it comes to network data. It worked, but for small scale mostly.

$ nc -l -u -p 4711 » udp-stream.log

This makes netcat listen on port 4711 and append the output to the log file. However, after each connection, the daemon is closed and needs to be restarted. Not very practical. The demo scenario was going to include somewhere around 300-500 messages / minute.

Node.js eventually caught my attention, since its mentioned in at least two articles on the frontpage of HN each day. I decided to try to set it up for logging udp-messages and append them to a file. Some googleing of it turned up node.js documentation with a really good example, http://nodejs.org/docs/v0.3.1/api/dgram.html#dgram.bind.

The code for it is available on github, its really simple, https://gist.github.com/1406283

Easily tested by using netcat for sending a udpmessage

$ echo “test” | nc -u server.hostname 4711

Even though I copied and pasted most of the code, it was a nice experience, since javascript is more or less fluent, I wont hesitate to use node.js in the future.

Google chrome segfaults after upgrade on debian stable

Thu, 10 Nov 2011 21:50:00 +0100

GAH! Frustration, google chrome, my thrustworthy companion everyday stopped working today. Segfaulted.

It happened after a reboot, it might have been that I hadnt restarted chrome for a week or two or something like that. The battery went dead which made it hard-reboot. After boot, chrome segfaulted, likely because the version got updated(not entirely sure yet).

I realized that there was a release today in the stable update channel, and figured I might as well try to upgrade to that one(15.0.874.120), the upgrade I had was probably a week or two old. But, no change. I changed the apt cache, and noticed I had google-chrome-stable_15.0.874.106-r107270_i386.deb in there, so I tried to revert to that. No change.

After some googling, I came across this post which seems to indicate a similar problem, and it hinted me to check dmesg. Dmesg tells me its a segfault in ligGL,

[ 504.112676] chrome[9171]: segfault at 4 ip a841729c sp b1498600 error 4 in libGL.so.1.2[a83aa000+b4000]

I checked my version of libgl1-mesa-glx, and that was 7.7.1-5, I added backports for squeeze and installed 7.10.3-4~bpo60+1. No change though. Same segmentation fault.

I did some attempts with strace, but its mostly useless(for me) and no obvious place where it happens.

I even tried the beta channel of google chrome(16.0.912.32-r108990 now), but the same error is still there.

Not really sure where to go further, firefox feels like a downgrade when used to chrome imho. :)

Any hints?

Init.d-script for StatsD and Debian

Sun, 30 Oct 2011 21:35:13 +0100

I just deployed StatsD together with Graphite and Carbon according to http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ .

Seems to work nice so far, havent finetuned it so much yet though. Although, I didnt find a decent init-script for StatsD and Debian, so I modified one for RHEL found at https://gist.github.com/1071989 to this Debianized one: https://gist.github.com/1326359 I guess it would also work with ubuntu with minor tweaks(if any).

RFC: Chrooted ssh account with lamp-webhosting

Wed, 19 Oct 2011 21:53:00 +0200

I’ve been running LAMP-setups throughout the last 12 years or so, but security have always been a big concern. User convenience, simplicity and accessibility have also been big concerns, so security features and a locked down system can not decrease any of theese key points.

This is still a draft, needs to be updated. Consider it as a work in progress.

The setup below is a continous work and will probably be modified, but should be considered as a request-for-comment document. There will likely be lots of improvements possible, and it might even contain security holes that I havent thought about yet.

I expect to keep the following services:

- ssh/scp/sftp access

- apache

- mysql database

- php

- outgoing emails

- use of subversion and git through ssh

- possibly memcached(to be added later)

Common attack scenarios are

- sql injections

- execution of arbitrary phpcode

Goals:

- Limit successful attacks to a single user account

- Limit successful attacks so they can not cause much harm to affected user

First and foremost - ssh/scp/sftp

Allow users to ssh and scp into their chrooted home directory. Although, they should be able to use most stuff locally in there, like subversion, git and also access their weblog files. Uploading should be made with scp or preferrably managed with some version control system.

I found this script which sets up a chroot jail where users are sent after successful scp and ssh. Seems to work reasonably well.

http://www.fuschlberger.net/programs/ssh-scp-sftp-chroot-jail/

My biggest worry with it is that its really old. :)

My $APPS-list in it is:

/usr/bin/git-* /bin/chmod /usr/bin/git /usr/bin/file /bin/nc /usr/bin/giftopnm /usr/bin/jpegtopnm /usr/bin/bmptoppm /usr/bin/pnmscale /usr/bin/ppmtojpeg /usr/bin/ppmquant /usr/bin/ppmtogif /usr/bin/pngtopnm /usr/bin/pnmtopng /usr/bin/tail /usr/bin/php /bin/cat /usr/bin/crontab /usr/bin/svnadmin /usr/bin/grep /usr/bin/ssh-keygen /bin/nano /bin/hostname /usr/bin/svn /bin/bash /bin/cp /usr/bin/dircolors /bin/ls /bin/mkdir /bin/mv /bin/rm /bin/rmdir /bin/sh /bin/su /usr/bin/groups /usr/bin/id /usr/bin/rsync /usr/bin/ssh /usr/bin/scp /sbin/unix_chkpwd

This makes users locked into a “fake” system-chroot in /home/jail/. Right now I run multiple users in the same /home/jail, although, they have separate user homedirs in /home/jail/home/user but share the binaries.

The user-row in /etc/passwd looks like:

user-name:x:1032:1032::/home/jail/home/user-name:/bin/chroot-shell

And this is added to the /etc/sshd_config:

Match Group webuser
ChrootDirectory /home/jail/
AllowTCPForwarding no
X11Forwarding no

Running apache as the real user-user

I’ve been thinking about this a lot, whichever is the best. Now I’m running apache as the real user, within the chroot with help of the mpm-itk module for apache.

This is an example of the virtualhost used. Apache is running as the real jailed user, and documentroot is set as below.


        ServerName hostname.domain
	DocumentRoot /home/jail/home/user-name/www/
	AssignUserID user-name user-name
	php_value mysql.default_user db-user-name
	php_value mysql.default_password db-password
	php_value mysql.default_host localhost
    
	php_admin_value 	open_basedir 	/home/jail/home/user-name/	
	php_admin_value	    upload_tmp_dir 	/home/jail/home/user-name/tmp/
	php_admin_value	    session.save_path /home/jail/home/user-name/tmp/

The virtualhost-config-file can be owned and only readable by root, which means the user can access the environment variables in php with the getenv()-function. This is good, since the user doesnt need to store the database username and password in their homedirectory, makeing it a little bit more protected.

Read more about mpm itk at http://mpm-itk.sesse.net/

Using suhosin to limit PHP

Some things in PHP have so sharp edges that users are more or less expected to do mistakes with them, therefor, I’ve been disabling the following functions:

suhosin.executor.func.blacklist = “register_globals, enable_dl, show_source, system, shell_exec, passthru, exec, popen, proc_open, allow_url_fopen”

TODO:
- Lots of documentation and clarifications to make
- Add some example code to verify that its not possible to escape the jail. Would be nice with a thorough testsuite.

If you see problems somewhere, see good improvements, or have questions, leave a message.

Using rdiff for small mysqldumps

Mon, 10 Oct 2011 23:36:07 +0200

I’ve been pursuing the idea of using rdiff or something similar to manage sqldumps more effectively. We have a bunch of customers with the need and the requirement that they should store backups for a long time back in time.

As I like to keep it simple, using this wrapper together with Amazon S3 for storage seems simple and fairly robust. Its still in an early stage, and things might need adjustments, but its kept simple so it shouldnt hopefully be much, needs more testing though.

A quick and simple test shows the followng, on a testdatabase, a full sqldump is 1.3GB, gzipped its around 176MB. See below with an example patch created:

drwxr-xr-x   3 eric eric 4,0K 10 okt 23.30 .
drwxr-xr-x 175 eric eric  12K  8 okt 20.22 ..
-rw-r--r--   1 eric eric 7,6M 10 okt 23.06 full_20111010_230629.signature
-rw-r--r--   1 eric eric 1,3G 10 okt 23.06 full_20111010_230629.sql
-rw-r--r--   1 eric eric   11 10 okt 23.30 patch_20111010_233033_to_full_20111010_230629.rdiff
-rwxr-xr-x   1 eric eric 2,9K 10 okt 22.53 rdiff_mysqldump
-rw-r--r--   1 eric eric   21 10 okt 23.06 .rdiff_mysqldump_lastbase
-rw-r--r--   1 eric eric    2 10 okt 23.30 .rdiff_patches_since_full
-rw-r--r--   1 eric eric 1,1K 10 okt 23.11 README

The rdiff patchfile got 11 bytes on this one, not much, but not really a fair comparison since its not more than 2 hours of changes. Will redo it on a later backup to see the differences better.

Get the source at github, https://github.com/perssontm/rdiff_mysqldump

Good quick examples of rdiff can be found at http://beerpla.net/2008/05/12/a-better-diff-or-what-to-do-when-gnu-diff-runs-out-of-memory-diff-memory-exhausted/

I also came across this post, which is an attempt to shorten the locktime when dumping sqlite databases, possible thanks to rdiffs signature, quite interesting, http://ts1-en.blogspot.com/2009/06/backup-sqlite-database-with-rdiff.html

Realtime stats with google analytics

Sat, 08 Oct 2011 22:40:00 +0200

Google launched the realtime stats in analytics in beta a few weeks ago, I quickly signed up and after a few days got the confirm-email. Great!

My often biggest issue with analytics have been the terribly long delay from the visit to the site until it shows up in the statistics. Especially when creating funnels, goals and other more specialized tracking stuff, the debugging have been extremely lengthy and timeconsuming since I usually had to wait to verify it was working.

Will realtime solve that? No, as far as it looks now, nope. It includes the nice suboptions Overview, Locations, Traffic sources and Content. I would have liked to see goals and funnels. Imagine the possibility to see in realtime how people enter and exit your funnels, would have been fantastic.

Is it any good then, well, the live-view of contentpages is really nice, but its mostly neat to get away using yet another service for the realtimestuff than anything else.

Although, I hope they’ll improve it shortly.

Update 1: This is the results after a few minutes of putting up this post, a worlwide spread to 10 visitors. :)

Bring cookies from the web to my DVD

Tue, 04 Oct 2011 07:12:38 +0200

DVD is an annoying mediaformat, filled with barriers, obstacles and annoyances towards the enduser. The dvdplayer constantly tells you what you can and can not do, if you feel like fast forwarding through the “please do not pirate this movie”-text, youre immediatly shown a stop sign. Even though you bougt it, you cant fast forward. Its fair to assume that pirated material doesnt have this text still left in them, so they likely have a higher and much better user experience where the customer is in charge.

I wonder if theres a move somewhere in the industry trying to get rid of this, likely not since it seems to get worse for each movie i buy.

Putting on a dvd for your kid, needs 6 clicks on the remote and somewhere between 10-30 secs of wait between each, all because WB feel the need to spin their logo in my face and tell me that others use to pirate their movie. Oh, btw, thats a good idea, maybe Ill do that next time.

I would pay double for a dvd that did let me skip this junk, or double for the movie that just consisted of the movie. Or at least, give me some sort of “dvd-player-version-of-cookies” so we only have to see this junk once, at least once per player.

Is blueray any better perhaps?

Bounce rate for analytics..

Sat, 01 Oct 2011 12:26:18 +0200

I read about this a few weeks ago, and finally got around to implement and try it. Really interesting method to compensate for Google Analytics poor bounce rate calculation.

http://mindtheproduct.com/2011/08/data-driven-your-bounce-rate-and-time-on-site-are-wrong/

Multiple incognito windows in chrome...I wish..

Thu, 29 Sep 2011 22:34:00 +0200

Today I realized coincidentally that cookies between incognitowindows are shared, didnt think about this before, I just assumed that a newly opened incognitowindow from the main window was a new fresh one. It turns out not to be.

A quick googlesearch shows that its not just be, theres several others who came across this a long time ago. Theres even a bug filed for it, although the status wontfix is not very optimistic, see issue 24690.

Some of the comments on the issue refers to the multiple profile work thats beeing done instead, it looks like a good idea, but seems to need some preparations before use. I’ll definitly try this in the near future..

See the user dropdown in the right corner of the window.

Massive smartphone penetration

Wed, 28 Sep 2011 08:22:00 +0200

Yesterday getting home on the subway, the massive smartphone penetration struck me. Standing up between all those seats, i noticed that, in a group out of 16 seats in car, 12 people where using a smartphone. From the quick glance, more android ones than iphones. Out of the 12 people, 4 where as far as I could tell above 50 yo, all of those 4 where playing som sort of game, wordfeud was on 3, the fourth I couldnt see.

Impressive! Looking at the commercial banners in the car, one had a QRcode, during the 15 minutes I couldnt see anyone trying to use it. Would be interesting to know how much conversion they get from those QRcodes.

Update: This is a nice (american I think) summary of qr-code usage http://digitalbuzz.s3.amazonaws.com/wp-content/uploads/2011/08/Infographic-QR-Codes-Statistics-Hi-Res.jpg

Update2: Found the blog entry presenting the above graphics today, http://www.digitalbuzzblog.com/infographic-how-people-use-qr-codes/

Whats happening in tech..

Sun, 25 Sep 2011 08:00:05 +0200

Seems like the heat is on in tech, in multiple ways. Most significant as of late is the patent wars. The view from the sideline tells me its energy spent that could been used for something better, like improving products.

And the social platform chaos, g+ is moving ahead, and Facebook seem to be rolling out features faster than ever, allmost all at the cost of privacy of users and headaches of developers.

Gameification part 2: the quick demo

Wed, 31 Aug 2011 22:34:00 +0200

Makeing a quick demo prototype is always good to get a message across, although can be a bit more time- and energy- consuming than just talking. :)

If you read my previous post “Gameification part 1 - thoughts only…”, you might see where I’m going, getting a simple score for each simple achieved task throughout the day. Scores could be weighed on work amount and or time consumption etc, but remember, this is a quick demo. Its not even beautifully layouted, but perhaps it will be soon.

The data is not very fun, since its more ore less anonymized because details are important for clients etc. But you get the idea. The data will grow over time, I have only really full data from today.

And a monthly highscore on top of this would really add on to it, and push you to achieve higher scores. :) Better watch out not to touch the “Overlearn the game”-problem( http://andrewoneverything.com/the-overlearning-the-game-problem ) that I read about the other day.

So, tell me what you think? Worthy a pursuit or just information-waste? If it was friend/collegue-sharing only, I would expect it to share some more details.

Gameification part 1 - thoughts only...

Mon, 29 Aug 2011 22:11:00 +0200

How often do you perform repetitive(usually boring) work throughout your workweek? I usually think about this when sorting through my inbox, replying to simple emails that, this is fixed, this will be fixed and whatever else is needed.

And when done, the inbox is sometimes almost empty, and at other times you only managed to clear out the upper top of the iceberg. No matter what, at both of those occasions, I’ve figured it would be nice with some positive feedback. Something simple, like, “Great, you cleared out 18 emails”. Looking at the mail client, it only shows 534 left to do. :) Not very inspiring… Its the same with task managers, 13 done, 243 to go..

Because of this, I’ve been thinking about something of an activity dashboard, on to which you could hook almost anything. Some systems include it, for example redmine or some of those webbased task managers. It really helps you get a feeling of, “Good, I’ve really accomplished something here.”.

Add a level of comparison with your friends to that dashboard, and you should have a really good motivator. Its fine even if your friends doesnt do the exact same thing, you can always motivate each other to work harder.

Pushing the idea even further, you should perhaps be able to setup a race between a few friends. Earn X amount of points before 12:30. And each point is a checked item of your list, a replied email, a fixed bug, a tested functionality or whatever is a tiny action that you can perform in your work. Best thing is if the action can report itself as done, for example a commit-hook, a email hook or whatever.

Hmm..I’ll try to elaborate this and perhaps create a working example in a future post.

Update, if you read this far, dont miss part 2

Overloaded inbox? Chaos-email management?

Sat, 27 Aug 2011 22:17:13 +0200

Then this might be for you, for a while, I’ve been developing a new tool to handle overloaded inboxes and give the control back to the user. To get it going a bit, I’m interesting to see if more have the same issues as I have with email. Just plain to many to respond to, hard to focus on the most important ones and just get the stuff done.

Läser in…

I did actually manage to get out the door running today, and I...

Tue, 23 Aug 2011 22:00:57 +0200

I did actually manage to get out the door running today, and I fired up My Tracks to get some performance stats on it. Having the earphones plugged in I could hear the distances, towards the end it sounded like I really picked up the speed..or something where wrong.. It sure was, checking the stats afterwards, My Tracks claims the top speed was 65km/h at one point, and that was when climbing up a bit… There must be a setting to avoid this…

"Linux Journal goes 100% digital"

Sat, 20 Aug 2011 07:18:00 +0200

I just recieved an email with the subject above. Whats up with this? I’ve been a subscriber for the past 10 years or so, and have really appreciated the physical magazine.

Sure, I can read a PDF, both in my phone and on the computer, I could even get some sort of tablet or e-reader to get the reading done. But, I already spend 8, if not more, hours / day in front of a screen, the physical magazine is a nice change from that.

The last issue of LJ was sent digitally to me, since they had some problem with the printing or shipping, I havent even read it so far. And I’m guessing thats how the future will be like. Theres more digital material sent to me than I can be expected to read, and adding LJ to it will make it go quite low on the priority list unfortunately. But I guess, this demands I’ll go shopping for an e-reader.

Read more about the switch if you like: http://www.linuxjournal.com/content/linux-journal-goes-100-digital

ios ui design

Fri, 19 Aug 2011 21:00:44 +0200

Really interesting post about grids in the user interface of the iphone. I’ve always though of the interface as really good, but he actually makes suggestions for a few improvements.

http://aentan.com/design/new-visual-proportions-for-the-ios-user-interface/

Still, i prefer android, who could really benefit from some new and improved ui-design, nothing big, just tuning.