Planet Linux Australia

Just in case you haven't seen this brilliant explanation before (I'm a proud bureaucrat). Sorry, don't know who authored it, merely that I got it through James Cook on bicigaga.



21 Economic Models Explained With Cows



SOCIALISM

You have 2 cows.

You give one to your neighbour.





COMMUNISM

You have 2 cows.

The State takes both and gives you some milk.





FASCISM

You have 2 cows.

The State takes both and sells you some milk.





NAZISM

You have 2 cows.

The State takes both and shoots you.





BUREAUCRATISM (CY)

You have 2 cows.

The State takes both, shoots one, milks the other, and then throws the milk away...





TRADITIONAL CAPITALISM

You have two cows.

You sell one and buy a bull.

Your herd multiplies, and the economy grows.

You sell them and retire on the income.





SURREALISM

You have two giraffes.

The government requires you to take harmonica lessons





AN AMERICAN CORPORATION

You have two cows.

You sell one, and force the other to produce the milk of four cows.

Later, you hire a consultant to analyze why the cow has dropped dead.





ENRON VENTURE CAPITALISM

You have two cows.

You sell three of them to your publicly listed company, using letters of credit opened by your brother-in-law at the bank, then execute a debt/equity swap with an associated general offer so that you get all four cows back, with a tax exemption for five cows.

The milk rights of the six cows are transferred via an intermediary to a Cayman Island Company secretly owned by the majority shareholder who sells the rights to all seven cows back to your listed company.

The annual report says the company owns eight cows, with an option on one more.

You sell one cow to buy a new president of the United States , leaving you with nine cows.

No balance sheet provided with the release.

The public then buys your bull.





A FRENCH CORPORATION

You have two cows.

You go on strike, organize a riot, and block the roads, because you want three cows.





A JAPANESE CORPORATION

You have two cows.

You redesign them so they are one-tenth the size of an ordinary cow and produce twenty times the milk.

You then create a clever cow cartoon image called 'Cowkimon' and market it worldwide.





A GERMAN CORPORATION

You have two cows.

You re-engineer them so they live for 100 years, eat once a month, and milk themselves.





AN ITALIAN CORPORATION

You have two cows, but you don't know where they are.

You decide to have lunch.





A RUSSIAN CORPORATION

You have two cows.

You count them and learn you have five cows.

You count them again and learn you have 42 cows.

You count them again and learn you have 2 cows.

You stop counting cows and open another bottle of vodka.





A SWISS CORPORATION

You have 5000 cows. None of them belong to you.

You charge the owners for storing them.





A CHINESE CORPORATION

You have two cows.

You have 300 people milking them.

You claim that you have full employment, and high bovine productivity.

You arrest the newsman who reported the real situation.





AN INDIAN CORPORATION

You have two cows.

You worship them.





A BRITISH CORPORATION

You have two cows.

Both are mad.





AN IRAQI CORPORATION

Everyone thinks you have lots of cows.

You tell them that you have none.

No-one believes you, so they bomb the **** out of you and invade your country.

You still have no cows, but at least now you are part of Democracy....





AN AUSTRALIAN CORPORATION

You have two cows.

Business seems pretty good.

You close the office and go to the pub for a few beers to celebrate.





A NEW ZEALAND CORPORATION

You have two cows.

The one on the left looks very attractive.





A GREEK CORPORATION

You have two cows.

You borrow against the cows from the Germans

You kill the cows and make souvlaki

You can’t pay the interest so the Germans lend you more money

You can’t pay the interest so the Germans lend you more money

You can’t pay the interest so the Germans lend you more money

You can’t pay the interest so the Germans lend you more money

Based on my investigation of RAID reliability [1] I have determined that BTRFS [2] is the Linux storage technology that has the best potential to increase data integrity without costing a lot of money. Basically a BTRFS internal RAID-1 should offer equal or greater data protection than RAID-6.

As BTRFS is so important and so very different to any prior technology for Linux it’s not something that can be easily deployed in the same way as other filesystems. It is possible to easily switch between filesystems such as Ext4 and XFS because they work in much the same way, you have a single block device which the filesystem uses to create a single mount-point. While BTRFS supports internal RAID so it may have multiple block devices and it may offer multiple mountable filesystems and snapshots. Much of the functionality of Linux Software RAID and LVM is covered by BTRFS. So the sensible way to deploy BTRFS is to give it all your storage and not make use of any other RAID or LVM.

So I decided to do a test installation. I started with a Debian install CD that was made shortly before the release of Squeeze (it was first to hand) and installed with BTRFS for the root filesystem, I then upgraded to Debian/Unstable to get the latest kernel as BTRFS is developing rapidly. The system failed on the first boot after upgrading to Unstable because the /etc/fstab entry for the root filesystem had the FSCK pass number set to 1 – which wasn’t going to work as no FSCK program has been written. I changed that number to 0 and it then worked.

The initial install was on a desktop system that had a single IDE drive and a CD-ROM drive. For /boot I used a degraded RAID-1 and then after completing the installation I removed the CD-ROM drive and installed a second hard drive, after that it was easy to add the other device to the RAID-1. Then I tried to add a new device to the BTRFS group with the command “btrfs device add /dev/sdb2 /dev/sda2” and was informed that it can’t do that to a mounted filesystem! That will decrease the possibilities for using BTRFS on systems with hot-swap drives, I hope that the developers regard it as a bug.

Then I booted with an ext3 filesystem for root and tried the “btrfs device add /dev/sdb2 /dev/sda2” again but got the error message “btrfs: sending ioctl 5000940a to a partition!” which is not even found by Google.

The next thing that I wanted to do was to put a swap file on BTRFS, the benefits for having redundancy and checksums on swap space seem obvious – and other BTRFS features such as compression might give a benefit too. So I created a file by using dd to take take from /dev/zero, ran mkswap on it and then tried to run swapon. But I was told that the file has holes and can’t be used. Automatically making zero blocks into holes is a useful feature in many situations, but not in this case.

So far my experience with BTRFS is that all the basic things work (IE storing files, directories, etc). But the advanced functions I wanted from BTRFS (mirroring and making a reliable swap space) failed. This is a bit disappointing, but BTRFS isn’t described as being ready for production yet.

• [1] http://etbe.coker.com.au/2012/02/06/reliability-raid/
• [2] http://en.wikipedia.org/wiki/Btrfs

Related posts:

1. Discovering OS Bugs and Using Snapshots I’m running Debian/Unstable on an EeePC 701, I’ve got an...
2. Reliability of RAID ZDNet has an insightful article by Robin Harris predicting the...
3. How I Partition Disks Having had a number of hard drives fail over the...

I just finished my last day at Red Hat, where I’ve worked as a kernel hacker since 2003.   I’ve been fortunate to work with so many brilliant people there on challenging and rewarding projects—like SELinux.  If someone had told me in 1999 that Linux would by now be fitted with a mandatory access control system from the NSA, which was enabled by default in major distributions, and certified and deployed in the field, I would have been skeptical.  To play a direct role in that would have been a dream come true.  It was.

I’ve also had the opportunity to work extensively within the community, during which time I’ve co-maintained or maintained kernel networking, crypto, SELinux and, currently, the security subsystem.  This work has taken me around the world and allowed me to make many new friends.

It’s been a great adventure.

Recently, I decided to make some changes in my career path and seek out some new challenges.  I’ll be starting in a new role the week after next.  I can’t say much about that now, but I will be continuing with my current upstream commitments.

My previous blog post spoke about not second-guessing the compiler and other facilities provided by the system, such as the C Standard library. Prompted, in part, by a real-world example: see if you can spot the bug in the following code… void * memmove(const void *src, void *dst, uint32_t len) { const uint8_t *s = [...]

I was recently asked why so few inline methods appear in my open source C++ code, such as SRecord or Tardy. There are two main reasons. The first reason is that by using the inline keyword, you are second-guessing the compiler, a form of premature optimization. See the Quotes section of Program optimization on Wikipedia [...]

Many years ago, when my pet dinosaur was still alive, I recall the painful transition from C on 16-bit machines, to C on 32-bit machines.  Many lessons from that era were incorporated into the the C89 standard, which happened in almost the same time frame. Years later, and some folks are once again struggling, this [...]

My latest adventure into Gtkmm-3, http://canola.sourceforge.net/, left me wanting to animate a series of images, and I wanted to use a transparent borderless transient window, and move the images around within it. Strangely, this proved to be in the “far too hard” category.  First, you need a compositing window manager.  Second, you have to ignore [...]

Squid is one of the biggest and most used proxies on the interwebs. And generating reports from the access logs is already a done deal, there are many commercial and OSS apps that support the squid log format. But I found my self in a situation where I wanted stats but didn’t want to install a web server on my proxy or use syslog to push my logs to a centralised server which was running such software, and also wasn’t in a position to go buy one of those off the shelf amazing wiz bang Squid reporting and graphing tools.

As a Linux geek I surfed the web to see what others have done. I came across a list provided by the Squid website. Following a couple of links, I came across a awk script called ‘proxy_stats.gawk’ written by Richard Huveneers.

I downloaded it and tried it out… unfortunately it didn’t work, looking at the code.. which he nicely commented showed that he had it set up for access logs  from version 1.* of squid. Now the squid access log format from squid 2.6+ hasn’t changed too much from version 1.1. all they have really done is add a “content type” entry at the end of each line.

So as a good Linux geek does, he upgrades the script, my changes include:

• Support for squid 2.6+
• Removed the use a deprecated switches that now isn’t supported in the sort command.
• Now that there is a an actual content type “column” lets use it to improve the ‘Object type report”.
• Add a users section, as this was an important report I required which was missing.
• And in a further hacked version, an auto generated size of the first “name” column.

Now with the explanation out of the way, let me show you it!

For those who are new to awk, this is how I’ve been running it:



zcat <access log file> | awk -f proxy_stats.gawk > <report-filename>

NOTE: I’ve been using it for some historical analysis, so I’m running it on old rotated files, which are compressed thus the zcat.

You can pass more then one file at a time and it order doesn’t matter, as each line of an access log contains the date in epoch time:



zcat `find /var/log/squid/ -name "access.log*"` |awk -f proxy_stats.gawk

The script produces an ascii report (See end of blog entry for example), which could be generated and emailed via cron. If you want it to look nice in any email client using html the I suggest wrapping it in <pre> tags.:

<html>

<head><title>Report Title</title></head>

Report title<body>

<pre>

... Report goes here ...

</pre>

</body>

</html>

For those experienced Linux sys admins out there using cron + ‘find -mtime’ would be a very simple way of having an automated daily, weekly or even monthly report.

But like I said earlier I was working on historic data, hundreds of files in a single report, hundreds because for business reasons we have been rotating the squid logs every hour… so I did what I do best, write a quick bash script to find all the files I needed to cat into the report:

#!/bin/bash ACCESS_LOG_DIR="/var/log/squid/access.log*" MONTH="$1" function getFirstLine() { if [ -n "`echo $1 |grep "gz$"`" ] then zcat $1 |head -n 1 else head -n 1 $1 fi } function getLastLine() { if [ -n "`echo $1 |grep "gz$"`" ] then zcat $1 |tail -n 1 else tail -n 1 $1 fi } for log in `ls $ACCESS_LOG_DIR` do firstLine="`getFirstLine $log`" epochStr="`echo $firstLine |awk '{print $1}'`" month=`date -d @$epochStr +%m` if [ "$month" -eq "$MONTH" ] then echo $log continue fi #Check the last line lastLine="`getLastLine $log`" epochStr="`echo $lastLine |awk '{print $1}'`" month=`date -d @$epochStr +%m` if [ "$month" -eq "$MONTH" ] then echo $log fi done

So there you go, thanks to the work of Richard Huveneers there is a script that I think generates a pretty good acsii report, which can be automated or integrated easily into any Linux/Unix work flow.

If you interested in getting hold of the most up to date version of the script you can get it from my sysadmin github repo here.

As promised earlier here is an example report:

Parsed lines  : 32960 Bad lines     : 0 First request : Mon 30 Jan 2012 12:06:43 EST Last request  : Thu 09 Feb 2012 09:05:01 EST Number of days: 9.9 Top 10 sites by xfers           reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- 213.174.155.216                   20   0.1% 100.0%   0.0%        0.0   0.0%   0.0%       1.7       2.5 30.media.tumblr.com                1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%      48.3      77.4 28.media.tumblr.com                1   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      87.1       1.4 26.media.tumblr.com                1   0.0%   0.0%      -        0.0   0.0%      -         -         - 25.media.tumblr.com                2   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      49.2      47.0 24.media.tumblr.com                1   0.0% 100.0%   0.0%        0.1   0.0%   0.0%     106.4     181.0 10.1.10.217                      198   0.6% 100.0%   0.0%       16.9   0.9%   0.0%      87.2    3332.8 3.s3.envato.com                   11   0.0% 100.0%   0.0%        0.1   0.0%   0.0%       7.6      18.3 2.s3.envato.com                   15   0.0% 100.0%   0.0%        0.1   0.0%   0.0%       7.5      27.1 2.media.dorkly.cvcdn.com           8   0.0% 100.0%  25.0%        3.2   0.2%   0.3%     414.1     120.5 Top 10 sites by MB              reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- zulu.tweetmeme.com                 2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       3.1     289.6 ubuntu.unix.com                    8   0.0% 100.0% 100.0%        0.1   0.0% 100.0%       7.5     320.0 static02.linkedin.com              1   0.0% 100.0% 100.0%        0.0   0.0% 100.0%      36.0     901.0 solaris.unix.com                   2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       3.8     223.6 platform.tumblr.com                2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       1.1     441.4 i.techrepublic.com.com             5   0.0%  60.0% 100.0%        0.0   0.0% 100.0%       6.8    2539.3 i4.zdnetstatic.com                 2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%      15.3     886.4 i4.spstatic.com                    1   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       4.7     520.2 i2.zdnetstatic.com                 2   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       7.8    2920.9 i2.trstatic.com                    9   0.0% 100.0% 100.0%        0.0   0.0% 100.0%       1.5     794.5 Top 10 neighbor report          reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- www.viddler.com                    4   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.turktrust.com.tr              16   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.trendmicro.com                 5   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.reddit.com                     2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.linkedin.com                   2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.google-analytics.com           2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.facebook.com                   2   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.dynamicdrive.com               1   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 www.benq.com.au                    1   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 wd-edge.sharethis.com              1   0.0% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 Local code                      reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- TCP_CLIENT_REFRESH_MISS         2160   6.6% 100.0%   0.0%        7.2   0.4%   0.0%       3.4      12.9 TCP_HIT                          256   0.8% 100.0%  83.2%       14.0   0.8% 100.0%      56.0    1289.3 TCP_IMS_HIT                      467   1.4% 100.0% 100.0%       16.9   0.9% 100.0%      37.2    1747.4 TCP_MEM_HIT                      426   1.3% 100.0% 100.0%       96.5   5.3% 100.0%     232.0    3680.9 TCP_MISS                       27745  84.2%  97.4%   0.0%     1561.7  85.7%   0.3%      59.2      18.2 TCP_REFRESH_FAIL                  16   0.0% 100.0%   0.0%        0.2   0.0%   0.0%      10.7       0.1 TCP_REFRESH_MODIFIED             477   1.4%  99.8%   0.0%       35.0   1.9%   0.0%      75.3    1399.4 TCP_REFRESH_UNMODIFIED          1413   4.3% 100.0%   0.0%       91.0   5.0%   0.0%      66.0     183.5 Status code                     reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- 000                              620   1.9% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 200                            29409  89.2% 100.0%   2.9%     1709.7  93.8%   7.7%      59.5     137.1 204                              407   1.2% 100.0%   0.0%        0.2   0.0%   0.0%       0.4       1.4 206                              489   1.5% 100.0%   0.0%      112.1   6.1%   0.0%     234.7     193.0 301                               82   0.2% 100.0%   0.0%        0.1   0.0%   0.0%       0.7       1.5 302                              356   1.1% 100.0%   0.0%        0.3   0.0%   0.0%       0.8       2.7 303                                5   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.7       1.5 304                              862   2.6% 100.0%  31.2%        0.4   0.0%  30.9%       0.4      34.2 400                                1   0.0%   0.0%      -        0.0   0.0%      -         -         - 401                                1   0.0%   0.0%      -        0.0   0.0%      -         -         - 403                               47   0.1%   0.0%      -        0.0   0.0%      -         -         - 404                              273   0.8%   0.0%      -        0.0   0.0%      -         -         - 500                                2   0.0%   0.0%      -        0.0   0.0%      -         -         - 502                               12   0.0%   0.0%      -        0.0   0.0%      -         -         - 503                               50   0.2%   0.0%      -        0.0   0.0%      -         -         - 504                              344   1.0%   0.0%      -        0.0   0.0%      -         -         - Hierarchie code                 reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- DIRECT                         31843  96.6%  97.7%   0.0%     1691.0  92.8%   0.0%      55.7      44.3 NONE                            1117   3.4% 100.0% 100.0%      131.6   7.2% 100.0%     120.7    2488.2 Method report                   reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- CONNECT                         5485  16.6%  99.2%   0.0%      132.8   7.3%   0.0%      25.0       0.3 GET                            23190  70.4%  97.7%   4.9%     1686.3  92.5%   7.8%      76.2     183.2 HEAD                            2130   6.5%  93.7%   0.0%        0.7   0.0%   0.0%       0.3       1.1 POST                            2155   6.5%  99.4%   0.0%        2.9   0.2%   0.0%       1.4       2.0 Object type report              reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- */*                                1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       1.6       3.2 application/cache-digest         396   1.2% 100.0%  50.0%       33.7   1.8%  50.0%      87.1    3655.1 application/gzip                   1   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      61.0      30.8 application/javascript           227   0.7% 100.0%  12.3%        2.2   0.1%   7.7%       9.9      91.9 application/json                 409   1.2% 100.0%   0.0%        1.6   0.1%   0.0%       4.1       6.0 application/ocsp-response        105   0.3% 100.0%   0.0%        0.2   0.0%   0.0%       1.9       2.0 application/octet-stream         353   1.1% 100.0%   6.8%       81.4   4.5%   9.3%     236.1     406.9 application/pdf                    5   0.0% 100.0%   0.0%       13.5   0.7%   0.0%    2763.3      75.9 application/pkix-crl              96   0.3% 100.0%  13.5%        1.0   0.1%   1.7%      10.6       7.0 application/vnd.google.sa       1146   3.5% 100.0%   0.0%        1.3   0.1%   0.0%       1.1       2.4 application/vnd.google.sa       4733  14.4% 100.0%   0.0%       18.8   1.0%   0.0%       4.1      13.4 application/x-bzip2               19   0.1% 100.0%   0.0%       78.5   4.3%   0.0%    4232.9     225.5 application/x-gzip               316   1.0% 100.0%  59.8%      133.4   7.3%  59.3%     432.4    3398.1 application/x-javascript        1036   3.1% 100.0%   5.8%        9.8   0.5%   3.4%       9.7      52.1 application/xml                   46   0.1% 100.0%  34.8%        0.2   0.0%  35.1%       3.5     219.7 application/x-msdos-progr        187   0.6% 100.0%   0.0%       24.4   1.3%   0.0%     133.7     149.6 application/x-pkcs7-crl           83   0.3% 100.0%   7.2%        1.6   0.1%   0.4%      19.8      10.8 application/x-redhat-pack         13   0.0% 100.0%   0.0%       57.6   3.2%   0.0%    4540.7     156.7 application/x-rpm                507   1.5% 100.0%   6.3%      545.7  29.9%   1.5%    1102.2     842.8 application/x-sdlc                 1   0.0% 100.0%   0.0%        0.9   0.0%   0.0%     888.3     135.9 application/x-shockwave-f        109   0.3% 100.0%  11.9%        5.4   0.3%  44.5%      50.6     524.1 application/x-tar                  9   0.0% 100.0%   0.0%        1.5   0.1%   0.0%     165.3      36.4 application/x-www-form-ur         11   0.0% 100.0%   0.0%        0.1   0.0%   0.0%       9.9      15.4 application/x-xpinstall            2   0.0% 100.0%   0.0%        2.5   0.1%   0.0%    1300.6     174.7 application/zip                 1802   5.5% 100.0%   0.0%      104.0   5.7%   0.0%      59.1       2.5 Archive                           89   0.3% 100.0%   0.0%        0.0   0.0%      -       0.0       0.0 audio/mpeg                         2   0.0% 100.0%   0.0%        5.8   0.3%   0.0%    2958.2      49.3 binary/octet-stream                2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       5.5      14.7 font/ttf                           2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%      15.5      12.5 font/woff                          1   0.0% 100.0% 100.0%        0.0   0.0% 100.0%      42.5    3539.6 Graphics                         126   0.4% 100.0%   0.0%        0.1   0.0%   0.0%       0.6       2.5 HTML                              14   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.1       0.1 image/bmp                          1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       1.3       3.9 image/gif                       5095  15.5% 100.0%   2.4%       35.9   2.0%   0.7%       7.2       9.5 image/jpeg                      1984   6.0% 100.0%   4.3%       52.4   2.9%   0.6%      27.0      62.9 image/png                       1684   5.1% 100.0%  10.3%       28.6   1.6%   1.9%      17.4     122.2 image/vnd.microsoft.icon          10   0.0% 100.0%  30.0%        0.0   0.0%  12.8%       1.0       3.3 image/x-icon                      72   0.2% 100.0%  16.7%        0.2   0.0%   6.0%       3.2      15.0 multipart/bag                      6   0.0% 100.0%   0.0%        0.1   0.0%   0.0%      25.2      32.9 multipart/byteranges              93   0.3% 100.0%   0.0%       16.5   0.9%   0.0%     182.0     178.4 text/cache-manifest                1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.7       3.1 text/css                         470   1.4% 100.0%   7.9%        3.4   0.2%   5.8%       7.4      59.7 text/html                       2308   7.0%  70.7%   0.4%        9.6   0.5%   0.6%       6.0      14.7 text/javascript                 1243   3.8% 100.0%   2.7%       11.1   0.6%   5.2%       9.1      43.3 text/json                          1   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       0.5       0.7 text/plain                      1445   4.4%  99.4%   1.5%       68.8   3.8%   5.5%      49.0      41.9 text/x-cross-domain-polic         24   0.1% 100.0%   0.0%        0.0   0.0%   0.0%       0.7       1.7 text/x-js                          2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%      10.1       6.4 text/x-json                        9   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       3.0       8.5 text/xml                         309   0.9% 100.0%  12.9%       12.9   0.7%  87.5%      42.8     672.3 unknown/unknown                 6230  18.9%  99.3%   0.0%      132.9   7.3%   0.0%      22.0       0.4 video/mp4                          5   0.0% 100.0%   0.0%        3.2   0.2%   0.0%     660.8      62.7 video/x-flv                      117   0.4% 100.0%   0.0%      321.6  17.6%   0.0%    2814.9     308.3 video/x-ms-asf                     2   0.0% 100.0%   0.0%        0.0   0.0%   0.0%       1.1       4.7 Ident (User) Report             reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- -                              32960 100.0%  97.8%   3.5%     1822.6 100.0%   7.2%      57.9     129.0 Weekly report                   reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- 2012/01/26                     14963  45.4%  97.6%   3.6%      959.8  52.7%   1.8%      67.3     104.5 2012/02/02                     17997  54.6%  98.0%   3.4%      862.8  47.3%  13.2%      50.1     149.4 Total report                    reqs   %all %xfers   %hit         MB   %all   %hit     kB/xf      kB/s ------------------------- ------------------------------- ------------------------ ------------------- All requests                   32960 100.0%  97.8%   3.5%     1822.6 100.0%   7.2%      57.9     129.0 Produced by : Mollie's hacked access-flow 0.5 Running time: 2 seconds

Happy squid reporting!

In January I attended the annual Australian Linux and Open Source conference (LCA). But since I was sick all of January and had a lot to catch up on, I never got around to sharing all the talks that I gave during that time.

Drupal Down Under

It started with a talk at Drupal Down Under, which happened the weekend before LCA. I gave a talk titled “HTML5 video specifications” (video, slides).

I spoke about the video and audio element in HTML5, how to provide fallback content, how to encode content, how to control them from JavaScript, and briefly about Drupal video modules, though the next presentation provided much more insight into those. I explained how to make the HTML5 media elements accessible, including accessible controls, captions, audio descriptions, and the new WebVTT file format. I ran out of time to introduce the last section of my slides which are on WebRTC.

Linux.conf.au

On the first day of LCA I gave a talk both in the Multimedia Miniconf and the Browser Miniconf.

Browser Miniconf

In the Browser Miniconf I talked about “Web Standardisation – how browser vendors collaborate, or not” (slides). Maybe the most interesting part about this was that I tried out a new slide “deck” tool called impress.js. I’m not yet sure if I like it but it worked well for this talk, in which I explained how the HTML5 spec is authored and who has input.

I also sat on a panel of browser developers in the Browser Miniconf (more as a standards than as a browser developer, but that’s close enough). We were asked about all kinds of latest developments in HTML5, CSS3, and media standards in the browser.

Multimedia Miniconf

In the Multimedia Miniconf I gave a “HTML5 media accessibility update” (slides). I talked about the accessibility problems of Flash, how native HTML5 video players will be better, about accessible video controls, captions, navigation chapters, audio descriptions, and WebVTT. I also provided a demo of how to synchronize multiple video elements using a polyfill for the multitrack API.

I also provided an update on HTTP adaptive streaming APIs as a lightning talk in the Multimedia Miniconf. I used an extract of the Drupal conference slides for it.

Main conference

Finally, and most importantly, Alice Boxhall and myself gave a talk in the main linux.conf.au titled “Developing Accessible Web Apps – how hard can it be?” (video, slides). I spoke about a process that you can follow to make your Web applications accessible. I’m writing a separate blog post to explain this in more detail. In her part, Alice dug below the surface of browsers to explain how the accessibility markup that Web developers provide is transformed into data structures that are handed to accessibility technologies.

Paul Wayper writes about the merits of using toll transponders to pay for parking.

I can report that I'm able to use my FasTrak tag to pay for parking at San Francisco International Airport, and it does indeed rock.

I'm unaware of anywhere else accepting it as a form of payment though.

Arrays

An array is a list surrounded by square brackets and separated by commas.

[1, 2, 3]  is an array of numbers.

['coat', 'mittens', 'snowboard']  is an array of strings.

Think of it as a caterpillar which has been stapled into your code. The two square brackets are staples which keep the caterpillar from moving, so you can keep track of which end is the head and which is the tail. The commas are the caterpillar’s legs, wiggling between each section of its body.

Once there was a caterpillar who had commas for legs. Which meant he had to allow a literary pause after each step. The other caterpillars really respected him for it and he came to have quite a commanding presence. Oh, and talk about a philanthropist! He was notorious for giving fresh leaves to those less-fortunate.

Yes, an array is a collection of things, but it also keeps those things in a specific order.

This is really a hilarious book!

I felt compelled to blog that quote   and now, I will continue reading…

http://mislav.uniqpath.com/poignant-guide/

“Hopefully, you’re seeing some patterns in Ruby. If not, just shake your head vigorously while you’ve got these examples in your mind. The code should break apart into manageable pieces.”

I am also reading “Learn You a Haskell for Great Good!”, in hardcopy from the Coburg library. It’s not quite so humorous as the Poignant guide, but has its moments of hilarity… and Haskell is an awesome language.

http://learnyouahaskell.com/

Something interesting I discovered about Python and --prefix that I can't see a lot of documentation on...

When you build Python you can use the standard --prefix flag to configure to home the installation as you require. You might expect that this would hard-code the location to look for the support libraries to the value you gave; however in reality it doesn't quite work like that.

Python will only look in the directory specified by prefix after it first searches relative to the path of the executing binary. Specifically, it looks at argv[0] and works through a few steps — is argv[0] a symlink? then dereference it. Does argv[0] have any slashes in it? if not, then search the $PATH for the binary. After this, it starts searching for dirname(argv[0])/lib/pythonX.Y/os.py, then dirname(argv[0])/../lib and so on, until it reaches the root. Only after these searches fail does the interpreter then fall back to the hard-coded path specified in the --prefix when configured.

What is the practical implications? It means you can move around a python installation tree and have it all "just work", which is nice. In my situation, I noticed this because we have a completely self-encapsulated build toolchain, but we wish to ship the same interpreter on the thing that we're building (during the build, we run the interpreter to create .pyc files for distribution, and we need to be sure that when we did this we didn't accidentally pick up any of the build hosts python; only the toolchain python).

The PYTHONHOME environment variable does override this behaviour; if it is set then the search stops there. Another interesting thing is that sys.prefix is therefore not the value passed in by --prefix during configure, but the value of the current dynamically determined prefix value.

If you run an strace, you can see this in operation.

readlink("/usr/bin/python", "python2.7", 4096) = 9 readlink("/usr/bin/python2.7", 0xbf8b014c, 4096) = -1 EINVAL (Invalid argument) stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory) stat64("/usr/bin/lib/python2.7/os.py", 0xbf8af090) = -1 ENOENT (No such file or directory) stat64("/usr/bin/lib/python2.7/os.pyc", 0xbf8af090) = -1 ENOENT (No such file or directory) stat64("/usr/lib/python2.7/os.py", {st_mode=S_IFREG|0644, st_size=26300, ...}) = 0 stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory) stat64("/usr/bin/lib/python2.7/lib-dynload", 0xbf8af0a0) = -1 ENOENT (No such file or directory) stat64("/usr/lib/python2.7/lib-dynload", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

Firstly it dereferences symlinks. Then it looks for Modules/Setup to see if it is running out of the build tree. Then it starts looking for os.py, walking its way upwards. One interesting thing that may either be a bug or a feature, I haven't decided, is that if you set the prefix to / then the interpreter will not go back to the root and then look in /lib. This is probably pretty obscure usage though!

All this is implemented in Modules/getpath.c which has a nice big comment at the top explaining the rules in detail.

This post is password protected. To view it please enter your password below:

Password:

• @kattekrab @KathyReid @johnnymestizo take a look at http://t.co/evlWMf7z — I think this might solve our problem in reply to kattekrab #
• If we all end up going to jail for downloading music, I at least hope they separate us by music genre. #
• Apple: Protect Workers Making iPhones in Chinese Factories http://t.co/9I6vgeuY via @change #
• An article on innovation, comparing Google and Apple http://t.co/2ahgwTX9 #

• Straight couples opting for civil partnerships http://t.co/Fifq2xU6 #
• @KathyReid sweet, thanks! in reply to KathyReid #
• @KathyReid that’s a good idea! There’s the LA jobs mailing list, but I think we can do better. in reply to KathyReid #
• Some insight into how we sleep, by @arjenlentz http://t.co/ppKiVxiO #
• @KathyReid Yup! Lots of projects have their own ‘help wanted’ lists, but a unified one for the whole FLOSS community would rock. in reply to KathyReid #
• @johnnymestizo @kathyreid I would post on it and I already do heaps of pro bono work (in addition to paid work) for #olpcau in reply to johnnymestizo #
• @johnnymestizo @kathyreid I think it would also be good for people to advertise their skills and availability to do pro bono work in reply to johnnymestizo #
• @KathyReid @johnnymestizo yep aware of that My point is we all do good work. Would be good to make it easier for others to get involved in reply to KathyReid #
• @KathyReid @johnnymestizo yes more tech specific. Should it be FOSS specific? in reply to KathyReid #
• An article about my #lca2012 talk and the #olpcau education programme http://t.co/M3BRaTYo #olpc #

I’ve previously written Some Notes on DRBD [1] and a post about DRBD Benchmarking [2].

Previously I had determined that replication protocol C gives the best performance for DRBD, that the batch-time parameters for Ext4 aren’t worth touching for a single IDE disk, that barrier=0 gives a massive performance boost, and that DRBD gives a significant performance hit even when the secondary is not connected. Below are the results of some more tests of delivering mail from my Postal benchmark to my LMTP server which uses the Dovecot delivery agent to write it to disk, the rates are in messages per minute where each message is an average of 70K in size. The ext4 filesystem is used for all tests and the filesystem features list is “has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize“.

p4-2.8 Default Ext4 1663 barrier=0 2875 DRBD no secondary al-extents=7 645 DRBD no secondary default 2409 DRBD no secondary al-extents=1024 2513 DRBD no secondary al-extents=3389 2650 DRBD connected 1575 DRBD connected al-extents=1024 1560 DRBD connected al-extents=1024 Gig-E 1544

The al-extents option determines the size of the dirty areas that need to be resynced when a failed node rejoins the cluster. The default is 127 extents of 4M each for a block size of 508MB to be synchronised. The maximum is 3389 for a synchronisation block size of just over 13G. Even with fast disks and gigabit Ethernet it’s going to take a while to synchronise things if dirty zones are 13GB in size. In my tests using the maximum size of al-extents gives a 10% performance benefit in disconnected mode while a size of 1024 gives a 4% performance boost. Changing the al-extents size seems to make no significant difference for a connected DRBD device.

All the tests on connected DRBD devices were done with 100baseT apart from the last one which was a separate Gigabit Ethernet cable connecting the two systems.

Conclusions

For the level of traffic that I’m using it seems that Gigabit Ethernet provides no performance benefit, the fact that it gave a slightly lower result is not relevant as the difference is within the margin of error.

Increasing the al-extents value helps with disconnected performance, a value of 1024 gives a 4% performance boost. I’m not sure that a value of 3389 is a good idea though.

The ext4 barriers are disabled by DRBD so a disconnected DRBD device gives performance that is closer to a barrier=0 mount than a regular ext4 mount. With the significant performance difference between connected and disconnected modes it seems possible that for some usage scenarios it could be useful to disable the DRBD secondary at times of peak load – it depends on whether DRBD is used as a really current backup or a strict mirror.

Future Tests

I plan to do some tests of DRBD over Linux software RAID-1 and tests to compare RAID-1 with and without bitmap support. I also plan to do some tests with the BTRFS filesystem, I know it’s not ready for production but it would still be nice to know what the performance is like.

But I won’t use the same systems, they don’t have enough CPU power. In my previous tests I established that a 1.5GHz P4 isn’t capable of driving the 20G IDE disk to it’s maximum capacity and I’m not sure that the 2.8GHz P4 is capable of running a RAID to it’s capacity. So I will use a dual-core 64bit system with a pair of SATA disks for future tests. The difference in performance between 20G IDE disks and 160G SATA disks should be a lot less than the performance difference between a 2.8GHz P4 and a dual-core 64bit CPU.

• [1] http://etbe.coker.com.au/2011/12/17/drbd-notes/
• [2] http://etbe.coker.com.au/2012/01/05/drbd-benchmarking/

Related posts:

1. DRBD Benchmarking I’ve got some performance problems with a mail server that’s...
2. Some Notes on DRBD DRBD is a system for replicating a block device across...
3. Ethernet bonding Bonding is one of the terms used to describe multiple...

What on earth was Ubuntu thinking when it introduced Unity. I dislike it a lot. Infact I heard today they dropped the development team working on Kubuntu too. What on earth is going on.

http://blogs.discovery.com/dfh-sara-novak/2011/12/second-guesses-question-their-way-to-depression.html

You surely know someone like this or maybe that someone is you. Second guessers research and research decisions before hesitantly stepping forward only to wonder whether they made the right decision in the in.

If you run Simpana 9.0 and have some issues backing up Windows System State via CentOS/RHEL 6.2 MediaAgents, don’t fear SP5a will fix some quirks.

I had an issue matching the above and seems it will be resolved in SP5a (when released). If you have any issues, I recommend opening a support ticket with CommVault to have it checked out.

To: Roam Tollways

To: Wilson Parking

Re: paying for parking.

Dear people,

Many people have those little e-tags in their cars these days. They allow us to drive along tollways without having to stop and throw money into a machine. Another area where people have to stop their cars and throw money into a machine is in parking stations. We also have to grab the card that it spits out, carry it around and remember to pay for it before we leave, and if it doesn't validate or we lose it we have trouble getting out. However, you people have the solution for that.

Instead, we could drive up to the entrance gate of a parking station, the toll sensor would go 'beep', the boom gate would open and we'd drive in. Then, when we wanted to leave, we'd drive up to the exit gate, the toll sensor would go 'beep', the boom gate would open, we'd drive out again and the parking cost would automatically be debited to our account.

This would save us lots of time - time otherwise spent getting a ticket, paying for it, and feeding it into the exit gate. It would save you a lot of cost maintaining and repairing those machines. I'm sure you're already doing data mining on the journeys people take - this gives you a lot more interesting data. And you get a lot more people wanting to use your e-tags - people who like the convenience of being able to drive right into a shopping centre but aren't already toll users.

Go ahead and use this idea, I don't need any credit - just improve the planet.

Have fun,

Paul