Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 54 min 8 sec ago

Mark Terle: Oh Canada!

1 hour 58 min ago

Dear readers,

In my last instalment I left you having left Brisbane and arrived into Sydney ready to depart Australia.  Apologies for the lack of updates for the past few weeks, I’ve been enjoying the trip, devoting my all my brain cycles to my adventures, and spare moments to catching sleep and relaxing.

After a very long Thursday (courtesy Air Canada), was met at the other end by long time friends, A and E.   Had to remember, or be reminded, to get in the correct side of the car for Canada.  Canada not being of the 76 countries that drive on the left.   Was entertaining for the A and E to watch my reactions to being taken in traffic in the front passenger seat for the first time.

Headed off to A and E’s apartment to freshen up after the long flight.  As A famously boasts, “Closer to the airport than the nearest hotel!”.   Their apartment has a birds eye view looking out over the Vancouver Airport.  The area of Richmond that they live in is being slowly developed and turned into apartment towers.   I’m slightly jealous of the view and that it would be a nice place to spend a lazy winters afternoon looking out of the window.

Of my Vancouver experiences, two stick in my mind.

The first was ringing at my bell tower in a foreign country [1] at the Holy Rosary Cathedral.  I managed to ring there twice, once for a wedding and once for service.   Unsurprisingly, ringers are not a diverse bunch, easily spotted lurking outside towers, and fit into some stereotypes.    That said I had a great time and would love to visit them again.

The second was a trip out to the Richmond Night Markets with A, E and Z.   We worshipped at the stall supplying Rotatos (potatoes cut in a swirl on a stick and covered in cinnamon).  Yum!   Wandered past lots of other delicious food smells coming from the other vendors.   Looked at the dozens of stalls selling iPhone covers and shaking my head.   It was a great market experience.

Sadly, I had to leave Canada for the rest of my trip into the the US of A and crossed the border at of all places, Vancouver Airport.   I think I get bonus points for avoid LAX

[1] No, Queensland doesn’t count.

Mark Terle: South of the border one day, Queensland the next

1 hour 58 min ago

Arrived into Brisbane late morning on Saturday 4th August, it was great to finally get out of the chilly parts of Australia back into a nice, warm sunny part of the country.

First memorable experience of my time in Queensland was getting dragged along to a publicity sing for QUMS at the University of Queensland (UQ) Open Day.   Having participated in UWA Open Days I had some idea of what to expect.   Given it was a publicity sing and they needed all the voices they mustered, joined in and sung a lot of the usual songbook repertoire.   Several people were impressed at the sound tech controlling his rig with an iPad, and he was impressed with the volume that we made when we sang Bogoridtyse Deyvo.

I managed to also attend a couple of QUMS Rehearsals and Coffees during my extended stay in  Queensland.   The coffee at S place was enjoyable with lots of spirited post rehearsal conversation, scheming about BIV and working out QUMS fundraising ideas.  Hot dog, anyone?

Catching up with friends in Brisbane was also important.   Had lunch with R at the local golf club that is near his place, unfortunately in the electorate of that awful Campbell Newman person, but a good afternoon with chats about trains, choristers and beer.

The second lunch that was organised was going out to Brisbane’s Worst Vegetarian restaurant, the Norman Hotel, with I and colleague.  A nice T bone steak was consumed and a lot of very geeky conversation ensued and some discussion on careers.

I also got the chance to cook during one of my evenings in Brisbane.  I’ve grown quite fond of cooking bangers and mash.

Celebrated K’s birthday by heading out to dinner at Southbank.   The food was excellent and was the company.

Lots of ringing, rehearsals, service and even a quarter peal behind.  I’m looking forward to visiting more towers later in London.  I, K and M were excellent company during my ringing in Brisbane.

The second weekend in Brisbane was spent with a trip up to Mount Tambourine with S and K to eat lunch at the Fox and Hounds.   A very good faux English Pub   The trip also involved a wander around the markets, buying avocados, eating cheese, drinking beer and purchasing some very sinful fudge.

Packing and preparing for the overseas trip became the focus of the last few days in Brisbane.  I’m growing more accustomed to travelling with less, but there were decisions to be made about what ended up getting packed in the back pack and what got left behind.     This also meant the end of the driving trip across Australia, I’d covered somewhere around 7000 km (will need to check fuel records) over the length of the whole journey.    I don’t think I’ll undertake another long drive for a while again, although I’ll need to head down to Busselton for my school reunion when I get back to Australia.

Driving around Brisbane was much nicer than Sydney or Melbourne, just as long as you get the right exit and don’t disappear off onto the Inner City Bypass…. 25 minutes later ….. anyhow, hills are still exciting for a flat town person like me.

Woke up on the morning of the 16th August and took the taxi out to Brisbane Airport.   A mixture of sadness at leaving Brisbane with the excitement of heading onwards to Vancouver for the first leg of my international trip.  The trip down to Sydney with Qantas was uneventful but otherwise enjoyable.   I’ll leave off here and will continue to write about the next part of my trip in another post.

 

Mark Terle: The United States of New South Wales

1 hour 58 min ago

This blog entry has taken some time to put together and write up as other things have been keeping me occupied and enough time for reflective thoughts and writing has been proving elusive.

Drove out of Sydney heading out into regional New South Wales with a mission to get to Parkes in one day.  This is only 400 or so kilometres.  On WA roads, it would be easily achievable and quickly.   I managed it however, but NSW does have a different idea of what a highway is.

Passed through Bathurst, on the way to Parkes, indulged my inner motor head and drove around the Mt Panorama circuit.   Head onwards to Orange and then continued to Parkes.

My GPS decided to take me the dirt track way to get to the telescope and managed to arrive about fifteen minutes before closing at the gift shop.  Took some photos, bought some merchandise and indulged the geek in me.  Collected some geeky telecommunications photos too.   I’m a bit sad that I arrived there several days early, I think it would have been a much more exciting time as the Curiousity Rover was landing.

Returned to Orange for the evening and caught up with J and her band of ringers at Holy Trinity.   A very fun ring with some practice of call changes.   J and her husband put me up overnight which I was thankful for.

It would have been remiss of me to visit Orange without going and seeing B M OAM.   Had a lovely cuppa with him and chatted about all things choral and IV, and then let him return to his home renovations.

Drove onwards and arrived just after dusk in Tamworth.   Found a motel to stay in, after some confusion with the GPS, and settled for the night.   The meal at the restaurant attached was notable for its blandness.

The next day started with a visit to the Big Guitar and was then onwards through Armidale, down the side of the mountain at Dorrigo to Bellinen, a cut across inland to Varley and then onwards to Coffs Harbour.

The motel at Coffs Harbour felt like something out of survivor, but it had a stove and I took the opportunity to cook for myself for a change whilst travelling.   In the morning, the Big Banana was visited and then onwards to Bryon Bay.

Byron Bay reminded me very much of Margaret River, however with something mildly wrong and much more exploitative of the tourist.  In quaint country town style I had arrived on the day of the Coffs Cup and some stores that I wished to use were closed.   Egads!

At K’s suggestion, went and wandered up to the lighthouse and looked out into the Pacific Ocean.  Wandered down the hill to the most easterly point of the Australian continent.   I then head back into town and found a nice BnB with the most blue room ever to stay for the night.

The next day was the final major driving leg of this trip to Brisbane, and crossing the border into Queensland.

Returning back to the topic of this post, one of my favourite bits of trivia about the east cost of Australia (as opposed to Western Australia) is that Tasmania, South Australia, Victoria, Queensland and the ACT were all part of New South Wales at some point in their history.

Driving around the east coast of Australia you can see this influence, both current and historic, of Sydney and New South Wales pervade the built environment and culture.

As I crossed the mountains outside of Sydney and got back onto the plane, the areas there felt like a completely different state.   Victoria, now in comparison, seems lost elsewhere in time.   Canberra and ACT seem schizophrenic, not knowing if they want to be New South Wales or Victoria.

I’ve enjoyed seeing the bits of Australia in between and may go and visit there again.

 

 

 

Mark Terle: Crossing the coathanger

1 hour 58 min ago

Arrived Sydney late Wednesday afternoon into Paramatta.  Visited ringing practice at All Saints.  Lovely bells and a nice band, then headed onwards onto D’s place in Petersham.

Understandably snoozed a lot on the Thursday.    Friday, I first visited a contact who works for AARNet and touched based on a couple of projects that are under way and caught up with D and Q after they got home.

Saturday was catching up with geek friend, M.

A very good evening was had with L and A and their friend M.  M and her friend M also turned up.  Met A?.  A also dragged M along.  Lots of very yummy home made pizzas were made

Sunday was catching up with S and G for lunch, briefly met their son D.   Lots of change in their life with a new house and a change in employment situation.   The Sunday evening was spent with the cousins, who are always great company.

Monday night was ringing practice at St Andrews, and then Tuesday was getting back on the road again.

Crossed the harbour via the bridge and tunnel multiple times and got used to driving in Sydney… not that scary!

Mark Terle: The woman from snowy river

1 hour 58 min ago

This slightly delayed post covers Canberra.   Delayed mostly from not having very much computer time over the past few days to reconnect back into the geeksphere.

Have now done a lot of the tourist things in Canberra – National Carillion, National Portrait Gallery, National Library, Old and New Parliament Houses, Royal Australian Mint and the very solemn Australian War Memorial.   There are still a few left like the place with the miniature things.

Enjoyed wandering around the Parliament Houses and looking at the architecture.   Shall have to visit again when the House of Representatives is sitting.

Rung for service at St Pauls in Manuka.  Nice bells and a sociable bunch of ringers, including a Canberra chorister that I already knew.

Went with S and L to a bonfire over the border in New South Wales for Sunday night.  Was a fun experience on a cold winters night and had lots of yummy Vegan food.

Organised a dinner gathering on the Monday night at a cafe called Cream.  N, K, A, C, G, M, R, P, S, D, L, S all turned up throughout the course of events.  Introduced M to P and I’m sure they’ll enjoy going for motorbike rides and honing their ninja skills.  S came with myself, L and S to get soy fried ice cream elsewhere in Canberra.  It was a great experience to introduce completely (for values of Canberra) separate friends to each other and have them get along.

Departed Canberra and headed off to Jindabyne.  Saw snow again in the distance (I’m progressively getting closer…) and caught up with D.  Had some excellent Indian, chatted about life, IV and floristry     Continued heading onwards to Sydney but that is for another post….

Mark Terle: Inland cities….

1 hour 58 min ago

Continued travel up from Melbourne through Albury, Gundagai, Yass and onwards to Canberra which is where I am currently writing to you from.

Having lived all my life effectively on the coast, inland population centres both fascinate and confuse me.   Why would anyone life so far from the ocean?   What do you do with the extra land that is in one of your four compass directions?

From this visit to Canberra, in comparison to previous ones, the place seems to be going through a bit of a growth spurt.   Talking with the lovely P last night, I was amazed at the comparatively low housing costs compared to Perth.   This is probably making the place attractive to live for some.

Didn’t do as much tourist stuff as I had hoped today.  Was a bit sad when I visited the Telstra Tower to hear that the museum that was there is no more.

Notable achievement for the day was organising a catch up with all my favourite Canberra folk on Monday night.

Mark Terle: Walking ’round the rainy city

1 hour 58 min ago

This was about my eight visit to Melbourne (as far as I can remember/work out) and I’ve finally worked out the zen of the city.  This time was different, I had a car and was able to drive around the city.

Melbournians seem unware of the impact that geography has on their perceptions of their own city and desire to look outside it.   One thing I noticed was that the distances that I was travelling were tiny by Perth comparisons but similar travel times.  Everything seemed close and easily reachable.  Melbourne being a large city also has everything.

Being able to see this is difficult if you are only travelling on foot or public transport.  It is reflected in the nature of the roads and traffic of the city.

I’ve got a lot more rants about Melbourne driving, but I’ll spare you ..

I left today and drove to Albury.  Saw the cultural change the moment I crossed the bridge over the Murray.

Really enjoyed my time at IV.  The Berlioz was special, the Brahams a wonderful challenge.   Looking forward to AIV and the Rachmaninoff Vespers.

I’m onto the next major phase of my trip – the meandering via Canberra and Sydney up to Brisbane.  Looking forward to that too. Need to catch up on some of the administrivia of life over the next couple of days post IV.

Mark Terle: Climbed one mountain.

1 hour 58 min ago

Had the concert for the 63rd Intervarsity Choral Festival peformance of the Berlioz Requiem on Saturday night.   It was a great sing and it did literally feel like I sung my lungs out.  It will go down in my memories as one of my favourite concerts ever.

Dragged C along to the PCP where he provided medicinal cider for my vocal chords and was told to join MonUCS by one and all.

Attended the AIVCC meeting on Sunday.  Why do I keep doing this to myself?   Apart from that, the meeting went well.

Monday, I, like the rest of the choir, have hit the wall of exhaustion.  Rehearsals for the Brahams is going to take a lot of willpower to climb that next mountain.   I’m knackered so I’m going to try and get what little sleep I can.

Mark Terle: The Big O

1 hour 58 min ago

Had the first orchestral rehearsal with the orchestra for the Berlioz last night.

At one point, the hairs on my neck stood up and my knees began to melt. A truly special musical moment.

Any folk in Melbourne who miss the performance tonight in the Town Hall are going you feel sorry fit themselves. (More details about tickets at http://www.miv.aicsa.org.au/

Anyhow, off to the dress rehearsal.

Mark Terle: In the most isolated capital in the world ….

1 hour 58 min ago

I’ve been back in Perth for about a month now and am only starting to re-adjust to it.   After having travelled around the world, I still feel very much like a visitor in town (admittedly, one who knows how to get around)

I’ve been extremely lax about writing up my international adventures. It was hard during the trip due to constantly being on the move with sucky internet access and a case of writers block.    I’ve decided to write up some of the international sections as vignettes so based on requests I’ll write something about those cities first.   In order, Vancouver, San Francisco, Chicago, Grand Rapids, Boston, New York, London, Amsterdam (Haarlem), Oslo and Hong Kong (with a bonus Brisbane and Sydney in there too)

I still have to complete my other journey that requires leaving a very familiar place after 19 years, but I suspect that will happen in the new year.  (ask me in person and I’ll be less cryptic)

Anyhow, comments and requests welcome.

 

 

Sridhar Dhanapalan: Twitter posts: 2014-09-15 to 2014-09-21

2 hours 56 min ago

Andrew Pollock: [life] Day 233: An abortive attempt at bowling and a vet visit

Sat, 2014-09-20 20:25

Taking Zoe bowling has been on my to do list for a while, and yesterday the schedule was pretty open, so I thought I'd try doing it first thing in the morning.

Who would have thought that a bowling alley would be booked solid with senior citizens from 9am onwards? I certainly didn't. So I was a bit disappointed when we rocked up at 9:30am and were told that there weren't any lanes available until 11am.

We stuck around and watched for a little bit anyway, so I could explain stuff to Zoe. There were three lanes that weren't in use the entire time we were there, so I don't know what was up with that.

We had to be home at 11am for a vet visit for Smudge, and so after a brief stop off at the Garage to grab some tomatoes for lunch, we got home just as the vet arrived.

Dr Anthony is the most lovely and charismatic vet you could ever hope to have in your home, and Zoe took to him straight away. She wanted to show him all sorts of random stuff, and had to show him her bedroom before he left. It was very cute, and he was very nice and played along.

I'd asked him to come out to take a look at Smudge because I'd noticed excessive fur missing on the insides of her back legs. Turns out she's just over-grooming, which is probably a stress reaction. Not sure if it's related to reducing her food portions or maybe some random construction in the neighbourhood. The prescription is some Feliway, which like everything else in Australia is exorbitantly expensive. $11.99 on Amazon, $56.70 at Petbarn. Absolutely outrageous.

After lunch, we returned to the bowling alley for our bowling game. Zoe found a 6 pound ball a bit heavy to swing, and didn't really have enough oompf to get it down the lane. We ended up with quite a few balls stuck in the gutter. She did much better with the ball ramp.

I was disappointed that the technician couldn't take us out the back to see the behind the scenes workings, as I thought Zoe would have liked seeing that. I also forgot to get a printout of our score.

She had a great time "playing" on all the car racing arcade games, and we had a game of air hockey, which she was a good height for playing before returning home.

I had a few things that I had to get done, so I let Zoe watch some TV while I did them, and then Sarah arrived to pick her up. I then spent the rest of the night furiously trying to finish off all the written exams I needed to do for my weekend rock climbing course.

linux.conf.au News: Linux.conf.au 2015 welcomes the Deploying OpenStack Miniconf

Sat, 2014-09-20 15:27

OpenStack is Open Source software for building public and private clouds through a series of Infrastructure as a Service building blocks. OpenStack offers virtualized infrastructure -- compute, networking and storage -- as well as orchestration and management tools. Built with the support of a large number of organizations, many of whom sponsor linux.conf.au, there are now thousands of installations around the world. You quite possibly know someone running OpenStack right now.

This year's linux.conf.au miniconf is going to try taking a new approach -- we're going to cover the issues that are important to a System Admin in deploying OpenStack into their environment, while attempting to tell the story of a hypothetical OpenStack deployment from end to end. How does OpenStack integrate with your existing LDAP or Active Directory? What choices need to be made about how to configure storage on your compute nodes? How does scaling your object storage work? What are the networking options you might like to consider? What hypervisor is the right choice for your needs?

We'll also cover the existing configuration management options, including puppet, chef and HP's deployment system tripleo.

The focus of this year's miniconf is explicitly on the deployers of OpenStack, rather than the developers of it. We won't cover developer-centric issues like the latest tweaks to our CI system, or what the state of development is with the Kilo release. We pinky swear there will be no talks on the governance of the OpenStack Foundation.

So, are you interested in deploying cloud infrastructure at your organization? If so, the OpenStack miniconf is the event for you. Also, we'll have stickers. Just sayin'.



Linux.conf.au

linux.conf.au is one of the foremost open source conferences in the world, and is considered the most prestigious in the Southern Hemisphere. The next conference, LCA2015, will be held on 12 - 16 January 2015 at the University of Auckland in Auckland, New Zealand.

Linux.conf.au Miniconfs

Miniconfs are one day miniature conferences, within the main LCA2015 conference, which are targeted towards specific communities of interest and offer delegates an opportunity to network with other enthusiasts while immersing themselves in a specific topic or project.

LCA2015 miniconfs will be run on Monday 12th and Tuesday 13th of January 2015. Each miniconf's schedule is organised by the miniconf's organiser and will be published ahead of time, listing speakers and sessions for the day.

Linux.conf.au 2015 Date: 12 - 16 January 2015

Venue: Owen G Glenn Building, University of Auckland, Grafton Road, Auckland.

City: Auckland, New Zealand

Web: http://linux.conf.au/

Facebook: https://www.facebook.com/LCA2015

Twitter: @linuxconfau

Hashtag: #lca2015

linux.conf.au News: Writing Openly; Open Source Documentation Miniconf at Linux.conf.au 2015

Sat, 2014-09-20 11:27

People often complain about the quality of open source project documentation. At the same time, documentation is a great place to get started contributing to an open source community.

This miniconf will explore practical aspects of Open Source documentation, with an eye to applying them right away.

We will look at:

  • popular markup languages (Docbook, DITA, markdown, etc)
  • version control systems for writers (SVN, git, etc)
  • getting started as a contributor (how to pick a project, getting an account, meeting the community, your first commit, etc)
  • documentation skills and methodologies (topic-based authoring, single sourcing, minimalism, etc)

We'll then be able to start contributing documentation. The pacing of this session will be largely driven by participant interest. It might be that we fly through the concepts straight to a frenetic docs hack fest. It might be that we get a lively argument about the best markup language, or whether minimalism is all hype.

Tim Hildred was a technical writer at Red Hat. Before that a barista at the Linuxcaffe in Toronto



Linux.conf.au

linux.conf.au is one of the foremost open source conferences in the world, and is considered the most prestigious in the Southern Hemisphere. The next conference, LCA2015, will be held on 12 - 16 January 2015 at the University of Auckland in Auckland, New Zealand.

Linux.conf.au Miniconfs

Miniconfs are one day miniature conferences, within the main LCA2015 conference, which are targeted towards specific communities of interest and offer delegates an opportunity to network with other enthusiasts while immersing themselves in a specific topic or project.

LCA2015 miniconfs will be run on Monday 12th and Tuesday 13th of January 2015. Each miniconf's schedule is organised by the miniconf's organiser and will be published ahead of time, listing speakers and sessions for the day.

Linux.conf.au 2015 Date: 12 - 16 January 2015

Venue: Owen G Glenn Building, University of Auckland, Grafton Road, Auckland.

City: Auckland, New Zealand

Web: http://linux.conf.au/

Facebook: https://www.facebook.com/LCA2015

Twitter: @linuxconfau

Hashtag: #lca2015

Andrew McDonnell: Evaluating the security of OpenWRT (part 2)

Sat, 2014-09-20 01:26

In my last post I covered how I setup an OpenWRT build, to examine  a small subset of indicators of security of the firmware.

In this follow-up post we will examine in detail the analysis results of one of the indicators: specifically, the RELRO flag.

A first look – what are the defaults?

The analysis here is specific to the Barrier Breaker release of OpenWRT, but it should be noted that during experiments with the OpenWRT development trunk the results are much the same.

Before diving into RELRO, lets take a look at the overall default situation.

Here is the checksec report for the Carambola2 device (MIPS processor) build.

It is a sea of red…

The ‘run as root’ errors can be ignored: those programs are actually absolute symbolic links which do not resolve in the host system. Relative symbolic links resolve correctly but are filtered out of the analysis.

The x86 build paints a similar picture:

(Notably for x86, the NX flag is correctly set, but that is a topic for another time.)

Note, the rest of this post describes how to modify OpenWRT to enable RELRO.  There may be perfectly valid reasons to not enable the flag (for example, using RELRO may have a performance impact, and for a given system the adverse security risk may be judged low), so I have ensured that the suggested mitigation if applied remains a choice in the configuration menu of the system.  For the moment my patch also retains backward compatibility by defaulting to off.

Inside the OpenWRT build system

After a brief look at the build logs, the reason is obvious: the typical gcc linker command is missing the flags needed to enable RELRO:  -Wl,-z,relro -Wl,-z,now (or the direct linker equivalents, -z relro -z now)

What could be done to address this?

OpenWRT provides a hook for appending to the global compiler CFLAGS but there is no similar hook for the linker stage. We could add those flags to the global CFLAGS and they can in fact flow through to the linker for many programs, but that would also be redundant as the flags are irrelevant to the compiler.  In the end I decided I would modify the OpenWRT source to add a new global CONFIG option, which adds -Wl,-z,relro -Wl,-z,now to the global LDFLAGS instead.

The following patch achieves that (note, I have left out some of the help for brevity):

diff --git a/rules.mk b/rules.mk index c9efb9e..e9c58d8 100644 --- a/rules.mk +++ b/rules.mk @@ -177,6 +177,10 @@ else endif endif +ifeq ($(CONFIG_SECURITY_USE_RELRO_EVERYWHERE),y) + TARGET_LDFLAGS+= -Wl,-z,relro -Wl,-z,now +endif + export PATH:=$(TARGET_PATH) export STAGING_DIR export SH_FUNC:=. $(INCLUDE_DIR)/shell.sh; diff --git a/toolchain/Config.in b/toolchain/Config.in index 7257f1d..964200d 100644 --- a/toolchain/Config.in +++ b/toolchain/Config.in @@ -38,6 +38,19 @@ menuconfig TARGET_OPTIONS Most people will answer N. + config SECURITY_USE_RELRO_EVERYWHERE + bool "Enable RELRO and NOW for binaries and libraries" if TARGET_OPTIONS + default n + help + Apply -z relro -z now flag to the linker stage for all ELF binaries and libraries. menuconfig EXTERNAL_TOOLCHAIN bool

Having attched OpenWRT, and enabled the new flag, lets rebuild everything again and run another checksec scan.

What a difference!

The results shown above are for x86, the picture is similar for the Carambola2 MIPS image.

The new results indicate that the RELRO flag is present on some binaries but not all of them. From this we can predict that some packages do not fully honour the global OpenWRT build system linker flags. I soon confirmed this**; the implication is that the new flag CONFIG_SECURITY_FORCE_RELRO  is useful, however, a caveat in the Kconfig help is required. In particular, a statement to the effect that the efficacy depends on proper coding of OpenWRT packages (with ideally all packages maintained by the project being fixed to honour the flag.)

** For example: the package that builds libnl-tiny.so does not pass LDFLAGS through to the linker; this and some other base system packages needed patching to get complete coverage.  it is likely that there are other packages that I did not have selected that may also need tweaking.

Another notable package is busybox.  Busybox it turns out uses ld directly for linking, instead of indirectly via gcc, and thus requires the flags in the pure form -z relro -z now. (The busybox OpenWRT package Makefile also happens to treat the global TARGET_LDFLAGS differently from the TARGET_CFLAGS although I am unsure if this is a bug; but that turned out to be a red-herring.) Oddly, this solution worked for MIPS when I tried it previously, but is presently not successful for the x86 build, so further investigation is needed here; possibly I incorrectly noted the fix in previous experiments.

Fun and Games with uClibc and busybox

The other recalcitrant is the uClibc library. I spent quite a bit time trying to work out why this was not working, especially having  confirmed with verbose logging that the flags are being applied as expected. Along the way I learned that uClibc already has its own apply RELRO config item, which was already enabled. Even more oddly, RELRO is present on some uClibc libraries and not others, that as far as I could tell were being linked with identical linker flag sets.

After some digging I discovered hints of bugs related to RELRO in various versions of binutils, so I further patched OpenWRT to use the very latest binutils release.   However that made no difference.  At this point I took a big diversion and spent some time building the latest uClibc externally, where I discovered that it built fine using the native toolchain of Debian Wheezy (including a much older binutils!)  After some discussion on the uClibc mailing list I have come to the conclusion that there may be a combination of problems, including the fact that uClibc in OpenWRT is a couple of years old (and additionally has a set of OpenWRT specific patches.)   I could go further and patch OpenWRT to use the trunk uClibc but then I would have to work through refreshing the set of patches which I really don’t have time or inclination to do, so for the moment I have deferred working on resolving this conundrum.  Eventually someone at OpenWRT may realise that uClibc has undergone a flurry of development in recent times and may bump to the more recent version.

Comments

Along the way, I discovered that Debian actually runs security scans across all packages in the distribution – take a look at https://lintian.debian.org/tags/hardening-no-relro.html.

It is worth noting that whenever changing any build-related flag it is worth cleaning and rebuilding the toolchain as well as the target packages and kernel; I found without doing this, flag changes such as the RELRO flag don’t fully take effect as expected.

For maximum verboseness, run with make V=csw although I had to dig through the code to find this out.

I was going to repeat all the testing against a third target, another  MIPS-based SOC the  RALINK 3530 but at this point I don’t really have the time or inclination, I am sure the results will be quite similar.  It would probably be useful to try with an ARM-based target as well.

I should also try repeating this experiment with MUSL, which is an alternative C library that OpenWRT can be built with.

Conclusion

Out of the box, OpenWRT has very limited coverage of the RELRO security mitigation in a standard firmware build.  By applying the suggested patches it is possible to bring OpenWRT up to a level of coverage, for RELRO, to that approaching a hardened Gentoo or Ubuntu distribution, with only a small subset of binaries missing the flag.

References

My Github account includes the repository openwrt-barrier-breaker-hardening. The following branch include the completed series of patches  mentioned above:  owrt_analysis_relro_everywhere  I hope it will remain possible to apply these changes against the official release for a while yet.

The patch that enables the latest binutils is not in that branch, but in this commit.

Paul Wayper: Learning as we go

Fri, 2014-09-19 23:25
Today I broke my record for distance on the electric motorbike: 70km on a charge.

The battery at the end of that was at 121.8V, which is about 3.2V per cell, and the lowest cell was 3.16V. Watching the voltage as I was riding (taking quick glances) showed that even under load it wasn't dipping below 110V, so there's a good chance that most cells were still running just fine. It's possible that there's a cell with lower capacity, but I think as I ride it and the battery gets more chance to level out I'm actually improving its range.

Unfortunately the battery meter on the bike thinks that power is leaking out when it isn't, so that doesn't tell me much. The meter on the BMS read "0%" at 38Km (the last time I read it) and "0%" before I started charging it, so I have no idea whether that's being caused by a low cell or some other random error. Either way, it's still trial and error to see how much distance I can actually get out of the battery.

According to the meter on the wall socket, that was half a kilowatt-hour. According to my calculations, that was about 11 amps at 120 volts over four hours, so about five kilowatt-hours. Either the decimal point is wrong on the meter and it's an order of magnitude too low, I'm reading it wrong, or it's just plain incorrect. Five kilowatt-hours is in the right ball park. At $0.17 per kilowatt-hour I just paid 89 cents to fill up the bike. So it's about 1.2 cents per kilometre, at this rough guess.

Interesting!

The other amusing thing is that I'm having to get used to coming back to the bike to find people peering intently at it. Fluoro-clad workers, motorbike enthusiasts, general passers by - I get all types. There are still lots of people who walk on by, so I don't think I've really changed the planet. But it's still fun to explain it and to see people's different reactions, all without exception positive. That's pretty cool.

Steven Hanley: [mtb/events] The Surf Coast Century 2014 - Anglesea, Victoria - Surf Coast track and Otway ranges

Fri, 2014-09-19 17:25


Near Bells beach around 18km into the 100km run fullsize)

Earlier this year I competed in The North Face 100 km run in the Blue Mountains, it has a fairly large amount of climbing (over 4000 metres) and though it is remarkably pretty it is a tough day out. When a friend suggested doing the Surf Coast Centruy I thought why the heck not, less than half the climbing and I have never spent much time in this part of Victoria.

Report and photos are online. I was happy with my time, had a good day out and definately have to hand it to the locals, they live in a beautiful place. I also got a PB time for the distance by a fairly large amount (almost 3 hours faster than my TNF100 time)

Steven Hanley: [mtb/events] The Coastal Classic 2014 - Otford to Bundeena in the Royal National Park

Fri, 2014-09-19 17:25


Running along a beach 10km in to the event fullsize)

I had the opportunity to run in the Coastal Classic two weeks ago, Max Adventure do a great event here by getting access to the coastal walkiing track for this run and I recommend a visit to this track by anyone looking for a great coastal wlak or run.

Report and photos are online. It was a really pretty run and I had never been along the track so it was a great day out.

Stewart Smith: Some current MySQL Architecture writings

Fri, 2014-09-19 16:26

So, I’ve been looking around for a while (and a few times now) for any good resources that cover a bunch of MySQL architecture and technical details aimed towards the technically proficient but not MySQL literate audience. I haven’t really found anything. I mean, there’s the (huge and very detailed) MySQL manual, there’s the MySQL Internals manual (which is sometimes only 10 years out of date) and there’s various blog entries around the place. So I thought I’d write something explaining roughly how it all fits together and what it does to your system (processes, threads, IO etc).(Basically, I’ve found myself explaining this enough times in the past few years that I should really write it down and just point people to my blog). I’ve linked to things for more reading. You should probably read them at some point.

Years ago, there were many presentations on MySQL Architecture. I went to try and find some on YouTube and couldn’t. We were probably not cool enough for YouTube and the conferences mostly weren’t recorded. So, instead, I’ll just link to Brian on NoSQL – because it’s important to cover NoSQL as well.

So, here is a quick overview of executing a query inside a MySQL Server and all the things that can affect it. This isn’t meant to be complete… just a “brief” overview (of a few thousand words).

MySQL is an open source relational database server, the origins of which date back to 1979 with MySQL 1.0 coming into existence in 1995. It’s code that has some history and sometimes this really does show. For a more complete history, see my talk from linux.conf.au 2014: Past, Present and Future of MySQL (YouTube, Download).

At least of writing, everything here applies to MariaDB and Percona Server too.

The MySQL Server runs as a daemon (mysqld). Users typically interact with it over a TCP or UNIX domain socket through the MySQL network protocol (of which multiple implementations exist under various licenses). Each connection causes the MySQL Server (mysqld) to spawn a thread to handle the client connection.

There are now several different thread-pool plugins that instead of using one thread per connection, multiplex connections over a set of threads. However, these plugins are not commonly deployed and we can largely ignore them. For all intents and purposes, the MySQL Server spawns one thread per connection and that thread alone performs everything needed to service that connection. Thus, parallelism in the MySQL Server is gotten from executing many concurrent queries rather than one query concurrently.

The MySQL Server will cache threads (the amount is configurable) so that it doesn’t have to have the overhead of pthread_create() for each new connection. This is controlled by the thread_cache_size configuration option. It turns out that although creating threads may be a relatively cheap operation, it’s actually quite time consuming in the scope of many typical MySQL Server connections.

Because the MySQL Server is a collection of threads, there’s going to be thread local data (e.g. connection specific) and shared data (e.g. cache of on disk data). This means mutexes and atomic variables. Most of the more advanced ways of doing concurrency haven’t yet made it into MySQL (e.g. RCU hasn’t yet and is pretty much needed to get 1 million TPS), so you’re most likely going to see mutex contention and contention on cache lines for atomic operations and shared data structures.

There are also various worker threads inside the MySQL Server that perform various functions (e.g. replication).

Until sometime in the 2000s, more than one CPU core was really uncommon, so the fact that there were many global mutexes in MySQL wasn’t really an issue. These days, now that we have more reliable async networking and disk IO system calls but MySQL has a long history, there’s global mutexes still and there’s no hard and fast rule about how it does IO.

Over the past 10 years of MySQL development, it’s been a fight to remove the reliance on global mutexes and data structures controlled by them to attempt to increase the number of CPU cores a single mysqld could realistically use. The good news is that it’s no longer going to max out on the number of CPU cores you have in your phone.

So, you have a MySQL Client (e.g. the mysql client or something) connecting to the MySQL Server. Now, you want to enter a query. So you do that, say “SELECT 1;”. The query is sent to the server where it is parsed, optimized, executed and the result returns to the client.

Now, you’d expect this whole process to be incredibly clean and modular, like you were taught things happened back in university with several black boxes that take clean input and produce clean output that’s all independent data structures. At least in the case of MySQL, this isn’t really the case. For over a decade there’s been lovely architecture diagrams with clean boxes – the code is not like this at all. But this probably only worries you once you’re delving into the source.

The parser is a standard yacc one – there’s been attempts to replace it over the years, none of which have stuck – so we have the butchered yacc one still. With MySQL 5.0, it exploded in size due to the addition of things like SQL2003 stored procedures and it is of common opinion that it’s rather bloated and was better in 4.1 and before for the majority of queries that large scale web peeps execute.

There is also this thing called the Query Cache – protected by a single global mutex. It made sense in 2001 for a single benchmark. It is a simple hash of the SQL statement coming over the wire to the exact result to send(2) over the socket back to a client. On a single CPU system where you ask the exact same query again and again without modifying the data it’s the best thing ever. If this is your production setup, you probably want to think about where you went wrong in your life. On modern systems, enabling the query cache can drop server performance by an order of magnitude. A single global lock is a really bad idea. The query cache should be killed with fire – but at least in the mean time, it can be disabled.

Normally, you just have the SQL progress through the whole process of parse, optimize, execute, results but the MySQL Server also supports prepared statements. A prepared statement is simply this: “Hi server, please prepare this statement for execution leaving the following values blank: X, Y and Z” followed by “now execute that query with X=foo, Y=bar and Z=42″. You can call execute many times with different values. Due to the previously mentioned not-quite-well-separated parse, optimize, execute steps, prepared statements in MySQL aren’t as awesome as in other relational databases. You basically end up saving parsing the query again when you execute it with new parameters. More on prepared statements (from 2006) here. Unless you’re executing the same query many times in a single connection, server side prepared statements aren’t worth the network round trips.

The absolute worst thing in the entire world is MySQL server side prepared statements. It moves server memory allocation to be the responsibility of the clients. This is just brain dead stupid and a reason enough to disable prepared statements. In fact, just about every MySQL client library for every programming language ever actually fakes prepared statements in the client rather than trust every $language programmer to remember to close their prepared statements. Open many client connections to a MySQL Server and prepare a lot of statements and watch the OOM killer help you with your DoS attack.

So now that we’ve connected to the server, parsed the query (or done a prepared statement), we’re into the optimizer. The optimizer looks at a data structure describing the query and works out how to execute it. Remember: SQL is declarative, not procedural. The optimizer will access various table and index statistics in order to work out an execution plan. It may not be the best execution plan, but it’s one that can be found within reasonable time. You can find out the query plan for a SELECT statement by prepending it with EXPLAIN.

The MySQL optimizer is not the be all and end all of SQL optimizers  (far from it). A lot of MySQL performance problems are due to complex SQL queries that don’t play well with the optimizer, and there’s been various tricks over the years to work around deficiencies in it. If there’s one thing the MySQL optimizer does well it’s making quick, pretty good decisions about simple queries. This is why MySQL is so popular – fast execution of simple queries.

To get table and index statistics, the optimizer has to ask the Storage Engine(s). In MySQL, the actual storage of tables (and thus the rows in tables) is (mostly) abstracted away from the upper layers. Much like a VFS layer in an OS kernel, there is (for some definition) an API abstracting away the specifics of storage from the rest of the server. The API is not clean and there are a million and one layering violations and exceptions to every rule. Sorry, not my fault.

Table definitions are in FRM files on disk, entirely managed by MySQL (not the storage engines) and for your own sanity you should not ever look into the actual file format. Table definitions are also cached by MySQL to save having to open and parse a file.

Originally, there was MyISAM (well, and ISAM before it, but that’s irrelevant now). MyISAM was non-transactional but relatively fast, especially for read heavy workloads. It only allowed one writer although there could be many concurrent readers. MyISAM is still there and used for system tables. The current default storage engine is called InnoDB. It’s all the buzzwords like ACID and MVCC. Just about every production environment is going to be using InnoDB. MyISAM is effectively deprecated.

InnoDB originally was its own independent thing and has (to some degree) been maintained as if it kind of was. It is, however, not buildable outside a MySQL Server anymore. It also has its own scalability issues. A recent victory was splitting the kernel_mutex, which was a mutex that protected far too much internal InnoDB state and could be a real bottleneck where NRCPUs > 4.

So, back to query execution. Once the optimizer has worked out how to execute the query, MySQL will start executing it. This probably involves accessing some database tables. These are probably going to be InnoDB tables. So, MySQL (server side) will open the tables, looking up the MySQL Server table definition cache and creating a MySQL Server side table share object which is shared amongst the open table instances for that table. See here for scalability hints on these (from 2009). The opened table objects are also cached – table_open_cache. In MySQL 5.6, there is table_open_cache_instances, which splits the table_open_cache mutex into table_open_cache_instances mutexes to help reduce lock contention on machines with many CPU cores (> 8 or >16 cores, depending on workload).

Once tables are opened, there are various access methods that can be employed. Table scans are the worst (start at the start and examine every row). There’s also index scans (often seeking to part of the index first) and key lookups. If your query involves multiple tables, the server (not the storage engine) will have to do a join. Typically, in MySQL, this is a nested loop join. In an ideal world, this would all be really easy to spot when profiling the MySQL server, but in reality, everything has funny names like rnd_next.

As an aside, any memory allocated during query execution is likely done as part of a MEM_ROOT – essentially a pool allocator, likely optimized for some ancient libc on some ancient linux/Solaris and it just so happens to still kinda work okay. There’s some odd server configuration options for (some of the) MEM_ROOTs that get exactly no airtime on what they mean or what changing them will do.

InnoDB has its own data dictionary (separate to FRM files) which can also be limited in current MySQL (important when you have tens of thousands of tables) – which is separate to the MySQL Server table definitions and table definition cache.

But anyway, you have a number of shared data structures about tables and then a data structure for each open table. To actually read/write things to/from tables, you’re going to have to get some data to/from disk.

InnoDB tables can be stored either in one giant table space or file-per-table. (Even though it’s now configurable), InnoDB database pages are 16kb. Database pages are cached in the InnoDB Buffer Pool, and the buffer-pool-size should typically be about 80% of system memory. InnoDB will use a (configurable) method to flush. Typically, it will all be O_DIRECT (it’s configurable) – which is why “just use XFS” is step 1 in IO optimization – the per inode mutex in ext3/ext4 just doesn’t make IO scale.

InnoDB will do some of its IO in the thread that is performing the query and some of it in helper threads using native linux async IO (again, that’s configurable). With luck, all of the data you need to access is in the InnoDB buffer pool – where database pages are cached. There exists innodb_buffer_pool_instances configuration option which will split the buffer pool into several instances to help reduce lock contention on the InnoDB buffer pool mutex.

All InnoDB tables have a clustered index. This is the index by which the rows are physically sorted by. If you have an INT PRIMARY KEY on your  InnoDB table, then a row with that primary key value of 1 will be physically close to the row with primary key value 2 (and so on). Due to the intricacies of InnoDB page allocation, there may still be disk seeks involved in scanning a table in primary key order.

Every page in InnoDB has a checksum. There was an original algorithm, then there was a “fast” algorithm in some forks and now we’re converging on crc32, mainly because Intel implemented CPU instructions to make that fast. In write heavy workloads, this used to show up pretty heavily in profiles.

InnoDB has both REDO and UNDO logging to keep both crash consistency and provide consistent read views to transactions. These are also stored on disk, the redo logs being in their own files (size and number are configurable). The larger the redo logs, the longer it may take to run recovery after a crash. The smaller the redo logs, the more trouble you’re likely to run into with large or many concurrent transactions.

If your query performs writes to database tables, those changes are written to the REDO log and then, in the background, written back into the table space files. There exists configuration parameters for how much of the InnoDB buffer pool can be filled with dirty pages before they have to be flushed out to the table space files.

In order to maintain Isolation (I in ACID), InnoDB needs to assign a consistent read view to a new transaction. Transactions are either started explicitly (e.g. with BEGIN) or implicitly (e.g. just running a SELECT statement). There has been a lot of work recently in improving the scalability of creating read views inside InnoDB. A bit further in the past there was a lot of work in scaling InnoDB for greater than 1024 concurrent transactions (limitations in UNDO logging).

Fancy things that make InnoDB generally faster than you’d expect are the Adaptive Hash Index and change buffering. There are, of course, scalability challenges with these too. It’s good to understand the basics of them however and (of course), they are configurable.

If you end up reading or writing rows (quite likely) there will also be a translation between the InnoDB row format(s) and the MySQL Server row format(s). The details of which are not particularly interesting unless you’re delving deep into code or wish to buy me beer to hear about them.

Query execution may need to get many rows from many tables, join them together, sum things together or even sort things. If there’s an index with the sort order, it’s better to use that. MySQL may also need to do a filesort (sort rows, possibly using files on disk) or construct a temporary table in order to execute the query. Temporary tables are either using the MEMORY (formerly HEAP) storage engine or the MyISAM storage engine. Generally, you want to avoid having to use temporary tables – doing IO is never good.

Once you have the results of a query coming through, you may think that’s it. However, you may also be part of a replication hierarchy. If so, any changes made as part of that transaction will be written to the binary log. This is a log file maintained by the MySQL Server (not the storage engines) of all the changes to tables that have occured. This log can then be pulled by other MySQL servers and applied, making them replication slaves of the master MySQL Server.

We’ll ignore the differences between statement based replication and row based replication as they’re better discussed elsewhere. Being part of replication means you get a few extra locks and an additional file or two being written. The binary log (binlog for short) is a file on disk that is appended to until it reaches a certain size and is then rotated. Writes to this file vary in size (along with the size of transactions being executed). The writes to the binlog occur as part of committing the transaction (and the crash safety between writing to the binlog and writing to the storage engine are covered elsewhere – basically: you don’t want to know).

If your MySQL Server is a replication slave, then you have a thread reading the binary log files from another MySQL Server and then another thread (or, in newer versions, threads) applying the changes.

If the slow query log or general query log is enabled, they’ll also be written to at various points – and the current code for this is not optimal, there be (yes, you guess it) global mutexes.

Once the results of a query have been sent back to the client, the MySQL Server cleans things up (frees some memory) and prepares itself for the next query. You probably have many queries being executed simultaneously, and this is (naturally) a good thing.

There… I think that’s a mostly complete overview of all the things that can go on during query execution inside MySQL.

linux.conf.au News: Open Hardware Miniconf to be at Linux.conf.au 2015

Fri, 2014-09-19 14:27

Linux.conf.au is pleased to announce that an Open Hardware Miniconf will be run the Linux.conf.au 2015 conference in Auckland, New Zealander during January 2015 .

The concept of Free / Open Source Software, already well understood by LCA attendees, is complemented by a rapidly growing community focused around Open Hardware and "maker culture". One of the drivers of the popularity of the Open Hardware community is easy access to cheap devices such as Arduino, which is a microcontroller development board originally intended for classroom use but now a popular building block in all sorts of weird and wonderful hobbyist and professional projects.

Interest in Open Hardware is high among FOSS enthusiasts but there is also a barrier to entry with the perceived difficulty and dangers of dealing with hot soldering irons, unknown components and unfamiliar naming schemes. The Miniconf will use the Arduino microcontroller board as a stepping stone to help ease software developers into dealing with Open Hardware. Topics will cover both software and hardware issues, starting with simpler sessions suitable for Open Hardware beginners and progressing through to more advanced topics.

The day will run in two distinct halves. The first part of the day will be a hands-on assembly session where participants will have the chance to solder together a special hardware project developed for the miniconf. Instructors will be on hand to assist with soldering and the other mysteries of hardware assembly. The second part of the day will be presentations about Open Hardware topics, including information on software to run on the hardware project built earlier in the day.

Please see www.openhardwareconf.org for more info.

Miniconf organiser Jon Oxer Jon Oxer has been hacking on both hardware and software since he was a little tacker. Most recently he's been focusing more on the Open Hardware side, co-founding Freetronics as a result of organising the first Arduino Miniconf at LCA2010 and designing the Arduino-based payloads that were sent into orbit in 2013 on board satellites ArduSat-X and ArduSat-1. His books include "Ubuntu Hacks" and "Practical Arduino". Linux.conf.au

linux.conf.au is one of the foremost open source conferences in the world, and is considered the most prestigious in the Southern Hemisphere. The next conference, LCA2015, will be held on 12 - 16 January 2015 at the University of Auckland in Auckland, New Zealand.

Linux.conf.au Miniconfs

Miniconfs are one day miniature conferences, within the main LCA2015 conference, which are targeted towards specific communities of interest and offer delegates an opportunity to network with other enthusiasts while immersing themselves in a specific topic or project.

LCA2015 miniconfs will be run on Monday 12th and Tuesday 13th of January 2015. Each miniconf's schedule is organised by the miniconf's organiser and will be published ahead of time, listing speakers and sessions for the day.

Linux.conf.au 2015 Date: 12 - 16 January 2015

Venue: Owen G Glenn Building, University of Auckland, Grafton Road, Auckland.

City: Auckland, New Zealand

Web: http://linux.conf.au/

Facebook: https://www.facebook.com/LCA2015

Twitter: @linuxconfau

Hashtag: #lca2015