Planet Linux Australia

Syndicate content
Planet Linux Australia -
Updated: 1 hour 7 min ago

Jonathan Adamczewski: A little bit of floating point in a memory allocator — Part 2: The floating point

Sat, 2018-01-06 17:04


This post contains the same material as this thread of tweets, with a few minor edits.

In IEEE754, floating point numbers are represented like this:


nnn is the exponent, which is floor(log2(size)) — which happens to be the fl value computed by TLSF.

sss… is the significand fraction: the part that follows the decimal point, which happens to be sl.

And so to calculate fl and sl, all we need to do is convert size to a floating point value (on recent x86 hardware, that’s a single instruction). Then we can extract the exponent, and the upper bits of the fractional part, and we’re all done :D

That can be implemented like this:

double sf = (int64_t)size; uint64_t sfi; memcpy(&sfi, &sf, 8); fl = (sfi >> 52) - (1023 + 7); sl = (sfi >> 47) & 31;

There’s some subtleties (there always is). I’ll break it down…

double sf = (int64_t)size;

Convert size to a double, with an explicit cast. size has type size_t, but using TLSF from, the largest supported allocation on 64bit architecture is 2^32 bytes – comfortably less than the precision provided by the double type. If you need your TLSF allocator to allocate chunks bigger than 2^53, this isn’t the technique for you :)

I first tried using float (not double), which can provide correct results — but only if the rounding mode happens to be set correctly. double is easier.

The cast to (int64_t) results in better codegen on x86: without it, the compiler will generate a full 64bit unsigned conversion, and there is no single instruction for that.

The cast tells the compiler to (in effect) consider the bits of size as if they were a two’s complement signed value — and there is an SSE instruction to handle that case (cvtsi2sdq or similar). Again, with the implementation we’re using size can’t be that big, so this will do the Right Thing.

uint64_t sfi; memcpy(&sfi, &sf, 8);

Copy the 8 bytes of the double into an unsigned integer variable. There are a lot of ways that C/C++ programmers copy bits from floating point to integer – some of them are well defined :) memcpy() does what we want, and any moderately respectable compiler knows how to select decent instructions to implement it.

Now we have floating point bits in an integer register, consisting of one sign bit (always zero for this, because size is always positive), eleven exponent bits (offset by 1023), and 52 bits of significant fraction. All we need to do is extract those, and we’re done :)

fl = (sfi >> 52) - (1023 + 7);

Extract the exponent: shift it down (ignoring the always-zero sign bit), subtract the offset (1023), and that 7 we saw earlier, at the same time.

sl = (sfi >> 47) & 31;

Extract the five most significant bits of the fraction – we do need to mask out the exponent.

And, just like that*, we have mapping_insert(), implemented in terms of integer -> floating point conversion.

* Actual code (rather than fragments) may be included in a later post…

Ben Martin: That gantry just pops right off

Sat, 2018-01-06 00:17
Hobby CNC machines sold as "3040" may have a gantry clearance of about 80mm and a z axis travel of around 55mm. A detached gantry is shown below. Notice that there are 3 bolts on the bottom side mounting the z-axis to the gantry. The stepper motor attaches on the side shown so there are 4 NEMA holes to hold the stepper. Note that the normal 3040 doesn't have the mounting plate shown on the z-axis, that crossover plate allows a different spindle to be mounted to this machine.

The plan is to create replacement sides with some 0.5inch offcut 6061 alloy. This will add 100mm to the gantry so it can more easily clear clamps and a 4th axis. Because that would move the cutter mount upward as well, replacing the z-axis with something that has more range, say 160mm becomes an interesting plan.

One advantage to upgrading a machine like this is that you can reassemble the machine after measuring and designing the upgrade and then cut replacement parts for the machine using the machine.

The 3040 can look a bit spartan with the gantry removed.

The preliminary research is done. Designs created. CAM done. I just have to cut 4 plates and then the real fun begins.

Pia Waugh: Pivoting ‘the book’ from individuals to systems

Thu, 2018-01-04 11:01

In 2016 I started writing a book, “Choose Your Own Adventure“, which I wanted to be a call to action for individuals to consider their role in the broader system and how they individually can make choices to make things better. As I progressed the writing of that book I realised the futility of changing individual behaviours and perspectives without an eye to the systems and structures within which we live. It is relatively easy to focus on oneself, but “no man is an island” and quite simply, I don’t want to facilitate people turning themselves into more beautiful cogs in a dysfunctional machine so I’m pivoting the focus of the book (and reusing the relevant material) and am now planning to finish the book by mid 2018.

I have recently realised four paradoxes which have instilled in me a sense of urgency to reimagine the world as we know it. I believe we are at a fork in the road where we will either reinforce legacy systems based on outdated paradigms with shiny new things, or choose to forge a new path using the new tools and opportunities at our disposal, hopefully one that is genuinely better for everyone. To do the latter, we need to critically assess the systems and structures we built and actively choose what we want to keep, what we should discard, what sort of society we want in the future and what we need to get there.

I think it is too easily forgotten that we invented all this and can therefore reinvent it if we so choose. But to not make a choice is to choose the status quo.

This is not to say I think everything needs to change. Nothing is so simplistic or misleading as a zero sum argument. Rather, the intent of this book is to challenge you to think critically about the systems you work within, whether they enable or disable the things you think are important, and most importantly, to challenge you to imagine what sort of world you want to see. Not just for you, but for your family, community and the broader society. I challenge you all to make 2018 a year of formative creativity in reimagining the world we live in and how we get there.

The paradoxes in brief, are as follows:

  • That though power is more distributed than ever, most people are still struggling to survive.
    It has been apparent to me for some time that there is a growing substantial shift in power from traditional gatekeepers to ordinary people through the proliferation of rights based philosophies and widespread access to technology and information. But the systemic (and artificial) limitations on most people’s time and resources means most people simply cannot participate fully in improving their own lives let alone in contributing substantially to the community and world in which they live. If we consider the impact of business and organisational models built on scarcity, centricity and secrecy, we quickly see that normal people are locked out of a variety of resources, tools and knowledge with which they could better their lives. Why do we take publicly funded education, research and journalism and lock them behind paywalls and then blame people for not having the skills, knowledge or facts at their disposal? Why do we teach children to be compliant consumers rather than empowered makers? Why do we put the greatest cognitive load on our most vulnerable through social welfare systems that then beget reliance? Why do we not put value on personal confidence in the same way we value business confidence, when personal confidence indicates the capacity for individuals to contribute to their community? Why do we still assume value to equate quantity rather than quality, like the number of hours worked rather than what was done in those hours? If a substantial challenge of the 21st century is having enough time and cognitive load to spare, why don’t we have strategies to free up more time for more people, perhaps by working less hours for more return? Finally, what do we need to do systemically to empower more people to move beyond survival and into being able to thrive.
  • Substantial paradigm shifts have happened but are not being integrated into people’s thinking and processes.
    The realisation here is that even if people are motivated to understand something fundamentally new to their worldview, it doesn’t necessarily translate into how they behave. It is easier to improve something than change it. Easier to provide symptomatic relief than to cure the disease. Interestingly I often see people confuse iteration for transformation, or symptomatic relief with addressing causal factors, so perhaps there is also a need for critical and systems thinking as part of the general curriculum. This is important because symptomatic relief, whilst sometimes necessary to alleviate suffering, is an effort in chasing one’s tail and can often perpetrate the problem. For instance, where providing foreign aid without mitigating displacement of local farmer’s efforts can create national dependence on further aid. Efforts to address causal factors is necessary to truly address a problem. Even if addressing the causal problem is outside your influence, then you should at least ensure your symptomatic relief efforts are not built to propagate the problem. One of the other problems we face, particularly in government, is that the systems involved are largely products of centuries old thinking. If we consider some of the paradigm shifts of our times, we have moved from scarcity to surplus, centralised to distributed, from closed to openness, analog to digital and normative to formative. And yet, people still assume old paradigms in creating new policies, programs and business models. For example how many times have you heard someone talk about innovative public engagement (tapping into a distributed network of expertise) by consulting through a website (maintaining central decision making control using a centrally controlled tool)? Or “innovation” being measured (and rewarded) through patents or copyright, both scarcity based constructs developed centuries ago? “Open government” is often developed by small insular teams through habitually closed processes without any self awareness of the irony of the approach. And new policy and legislation is developed in analog formats without any substantial input from those tasked with implementation or consideration with how best to consume the operating rules of government in the systems of society. Consider also the number of times we see existing systems assumed to be correct by merit of existing, without any critical analysis. For instance, a compliance model that has no measurable impact. At what point and by what mechanisms can we weigh up the merits of the old and the new when we are continually building upon a precedent based system of decision making? If 3D printing helped provide a surplus economy by which we could help solve hunger and poverty, why wouldn’t that be weighed up against the benefits of traditional scarcity based business models?
  • That we are surrounded by new things every day and yet there is a serious lack of vision for the future
    One of the first things I try to do in any organisation is understand the vision, the strategy and what success should look like. In this way I can either figure out how to best contribute meaningfully to the overarching goal, and in some cases help grow or develop the vision and strategy to be a little more ambitious. I like to measure progress and understand the baseline from which I’m trying to improve but I also like to know what I’m aiming for. So, what could an optimistic future look like for society? For us? For you? How do you want to use the new means at our disposal to make life better for your community? Do we dare imagine a future where everyone has what they need to thrive, where we could unlock the creative and intellectual potential of our entire society, a 21st century Renaissance, rather than the vast proportion of our collective cognitive capacity going into just getting food on the table and the kids to school. Only once you can imagine where you want to be can we have a constructive discussion where we want to be collectively, and only then can we talk constructively the systems and structures we need to support such futures. Until then, we are all just tweaking the settings of a machine built by our ancestors. I have been surprised to find in government a lot of strategies without vision, a lot of KPIs without measures of success, and in many cases a disconnect between what a person is doing and the vision or goals of the organisation or program they are in. We talk “innovation” a lot, but often in the back of people’s minds they are often imagining a better website or app, which isn’t much of a transformation. We are surrounded by dystopic visions of the distant future, and yet most government vision statements only go so far as articulating something “better” that what we have now, with “strategies” often focused on shopping lists of disconnected tactics 3-5 years into the future. The New Zealand Department of Conservation provides an inspiring contrast with a 50 year vision they work towards, from which they develop their shorter term stretch goals and strategies on a rolling basis and have an ongoing measurable approach.
  • That government is an important part of a stable society and yet is being increasingly undermined, both intentionally and unintentionally.
    The realisation here has been in first realising how important government (and democracy) is in providing a safe, stable, accountable, predictable and prosperous society whilst simultaneously observing first hand the undermining and degradation of the role of government both intentionally and unintentionally, from the outside and inside. I have chosen to work in the private sector, non-profit community sector, political sector and now public sector, specifically because I wanted to understand the “system” in which I live and how it all fits together. I believe that “government” – both the political and public sectors – has a critical part to play in designing, leading and implementing a better future. The reason I believe this, is because government is one of the few mechanisms that is accountable to the people, in democratic countries at any rate. Perhaps not as much as we like and it has been slow to adapt to modern practices, tools and expectations, but governments are one of the most powerful and influential tools at our disposal and we can better use them as such. However, I posit that an internal, largely unintentional and ongoing degradation of the public sectors is underway in Australia, New Zealand, the United Kingdom and other “western democracies”, spurred initially by an ideological shift from ‘serving the public good’ to acting more like a business in the “New Public Management” policy shift of the 1980s. This was useful double speak for replacing public service values with business values and practices which ignores the fact that governments often do what is not naturally delivered by the marketplace and should not be only doing what is profitable. The political appointment of heads of departments has also resulted over time in replacing frank, fearless and evidence based leadership with politically palatable compromises throughout the senior executive layer of the public sector, which also drives necessarily secretive behaviour, else the contradictions be apparent to the ordinary person. I see the results of these internal forms of degradations almost every day. From workshops where people under budget constraints seriously consider outsourcing all government services to the private sector, to long suffering experts in the public sector unable to sway leadership with facts until expensive consultants are brought in to ask their opinion and sell the insights back to the department where it is finally taken seriously (because “industry” said it), through to serious issues where significant failures happen with blame outsourced along with the risk, design and implementation, with the details hidden behind “commercial in confidence” arrangements. The impact on the effectiveness of the public sector is obvious, but the human cost is also substantial, with public servants directly undermined, intimidated, ignored and a growing sense of hopelessness and disillusionment. There is also an intentional degradation of democracy by external (but occasionally internal) agents who benefit from the weakening and limiting of government. This is more overt in some countries than others. A tension between the regulator and those regulated is a perfectly natural thing however, as the public sector grows weaker the corporate interests gain the upper hand. I have seen many people in government take a vendor or lobbyist word as gold without critical analysis of the motivations or implications, largely again due to the word of a public servant being inherently assumed to be less important than that of anyone in the private sector (or indeed anyone in the Minister’s office). This imbalance needs to be addressed if the public sector is to play an effective role. Greater accountability and transparency can help but currently there is a lack of common agreement on the broader role of government in society, both the political and public sectors. So the entire institution and the stability it can provide is under threat of death by a billion papercuts. Efforts to evolve government and democracy have largely been limited to iterations on the status quo: better consultation, better voting, better access to information, better services. But a rethink is required and the internal systemic degradations need to be addressed.

If you think the world is perfectly fine as is, then you are probably quite lucky or privileged. Congratulations. It is easy to not see the cracks in the system when your life is going smoothly, but I invite you to consider the cracks that I have found herein, to test your assumptions daily and to leave your counter examples in the comments below.

For my part, I am optimistic about the future. I believe the proliferation of a human rights based ideology, participatory democracy and access to modern technologies all act to distribute power to the people, so we have the capacity more so than ever to collectively design and create a better future for us all.

Let’s build the machine we need to thrive both individually and collectively, and not just be beautiful cogs in a broken machine.

Further reading:

Pia Waugh: Chapter 1.2: Many hands make light work, for a while

Thu, 2018-01-04 11:01

This is part of a book I am working on, hopefully due for completion by mid 2018. The original purpose of the book is to explore where we are at, where we are going, and how we can get there, in the broadest possible sense. Your comments, feedback and constructive criticism are welcome! The final text of the book will be freely available under a Creative Commons By Attribution license. A book version will be sent to nominated world leaders, to hopefully encourage the necessary questioning of the status quo and smarter decisions into the future. Additional elements like references, graphs, images and other materials will be available in the final digital and book versions and draft content will be published weekly. Please subscribe to the blog posts by the RSS category and/or join the mailing list for updates.

Back to the book overview or table of contents for the full picture. Please note the pivot from focusing just on individuals to focusing on the systems we live in and the paradoxes therein.

“Differentiation of labour and interdependence of society is reliant on consistent and predictable authorities to thrive” — Durkheim

Many hands makes light work is an old adage both familiar and comforting. One feels that if things get our of hand we can just throw more resources at the problem and it will suffice. However we have made it harder on ourselves in three distinct ways:

  • by not always recognising the importance of interdependence and the need to ensure the stability and prosperity of our community as a necessary precondition to the success of the individuals therein;
  • by increasingly making it harder for people to gain knowledge, skills and adaptability to ensure those “many hands” are able to respond to the work required and not trapped into social servitude; and
  • by often failing to recognise whether we need a linear or exponential response in whatever we are doing, feeling secure in the busy-ness of many hands.

Specialisation is when a person delves deep on a particular topic or skill. Over many millennia we have got to the point where we have developed extreme specialisation, supported through interdependence and stability, which gave us the ability to rapidly and increasingly evolve what we do and how we live. This resulted in increasingly complex social systems and structures bringing us to a point today where the pace of change has arguably outpaced our imagination. We see many people around the world clinging to traditions and romantic notions of the past whilst we hurtle at an accelerating pace into the future. Many hands have certainly made light work, but new challenges have emerged as a result and it is more critical than ever that we reimagine our world and develop our resilience and adaptability to change, because change is the only constant moving forward.

One human can survive on their own for a while. A tribe can divide up the labour quite effectively and survive over generations, creating time for culture and play. But when we established cities and states around 6000 years ago, we started a level of unprecedented division of labour and specialisation beyond mere survival. When the majority of your time, energy and resources go into simply surviving, you are largely subject to forces outside your control and unable to justify spending time on other things. But when survival is taken care of (broadly speaking) it creates time for specialisation and perfecting your craft, as well as for leisure, sport, art, philosophy and other key areas of development in society.

The era of cities itself was born on the back of an agricultural technology revolution that made food production far more efficient, creating surplus (which drove a need for record keeping and greater proliferation of written language) and prosperity, with a dramatic growth in specialisation of jobs. With greater specialisation came greater interdependence as it becomes in everyone’s best interests to play their part predictably. A simple example is a farmer needing her farming equipment to be reliable to make food, and the mechanic needs food production to be reliable for sustenance. Both rely on each other not just as customers, but to be successful and sustainable over time. Greater specialisation led to greater surplus as specialists continued to fine tune their crafts for ever greater outcomes. Over time, an increasing number of people were not simply living day to day, but were able to plan ahead and learn how to deal with major disruptions to their existence. Hunters and gatherers are completely subject to the conditions they live in, with an impact on mortality, leisure activities largely fashioned around survival, small community size and the need to move around. With surplus came spare time and the ability to take greater control over one’s existence and build a type of systemic resilience to change.

So interdependence gave us greater stability, as a natural result of enlightened self interest writ large where ones own success is clearly aligned with the success of the community where one lives. However, where interdependence in smaller communities breeds a kind of mutual understanding and appreciation, we have arguably lost this reciprocity and connectedness in larger cities today, ironically where interdependence is strongest. When you can’t understand intuitively the role that others play in your wellbeing, then you don’t naturally appreciate them, and disconnected self interest creates a cost to the community. When community cohesion starts to decline, eventually individuals also decline, except the small percentage who can either move communities or who benefit, intentionally or not, on the back of others misfortune.

When you have no visibility of food production beyond the supermarket then it becomes easier to just buy the cheapest milk, eggs or bread, even if the cheapest product is unsustainable or undermining more sustainably produced goods. When you have such a specialised job that you can’t connect what you do to any greater meaning, purpose or value, then it also becomes hard to feel valuable to society, or valued by others. We see this increasingly in highly specialised organisations like large companies, public sector agencies and cities, where the individual feels the dual pressure of being anything and nothing all at once.

Modern society has made it somewhat less intuitive to value others who contribute to your survival because survival is taken for granted for many, and competing in ones own specialisation has been extended to competing in everything without appreciation of the interdependence required for one to prosper. Competition is seen to be the opposite of cooperation, whereas a healthy sustainable society is both cooperative and competitive. One can cooperate on common goals and compete on divergent goals, thus making best use of time and resources where interests align. Cooperative models seem to continually emerge in spite of economic models that assume simplistic punishment and incentive based behaviours. We see various forms of “commons” where people pool their resources in anything from community gardens and ’share economies’ to software development and science, because cooperation is part of who we are and what makes us such a successful species.

Increasing specialisation also created greater surplus and wealth, generating increasingly divergent and insular social classes with different levels of power and people becoming less connected to each other and with wealth overwhelmingly going to the few. This pressure between the benefits and issues of highly structured societies and which groups benefit has ebbed and flowed throughout our history but, generally speaking, when the benefits to the majority outweigh the issues for that majority, then you have stability. With stability a lot can be overlooked, including at times gross abuses for a minority or the disempowered. However, if the balances tips too far the other way, then you get revolutions, secessions, political movements and myriad counter movements. Unfortunately many counter movements limit themselves to replacing people rather than the structures that created the issues however, several of these counter movements established some critical ideas that underpin modern society.

Before we explore the rise of individualism through independence and suffrage movements (chapter 1.3), it is worth briefly touching upon the fact that specialisation and interdependence, which are critical for modern societies, both rely upon the ability for people to share, to learn, and to ensure that the increasingly diverse skills are able to evolve as the society evolves. Many hands only make light work when they know what they are doing. Historically the leaps in technology, techniques and specialisation have been shared for others to build upon and continue to improve as we see in writings, trade, oral traditions and rituals throughout history. Gatekeepers naturally emerged to control access to or interpretations of knowledge through priests, academics, the ruling class or business class. Where gatekeepers grew too oppressive, communities would subdivide to rebalance the power differential, such a various Protestant groups, union movements and the more recent Open Source movements. In any case, access wasn’t just about power of gatekeepers. The costs of publishing and distribution grew as societies grew, creating a call from the business class for “intellectual property” controls as financial mechanisms to offset these costs. The argument ran that because of the huge costs of production, business people needed to be incentivised to publish and distribute knowledge, though arguably we have always done so as a matter of survival and growth.

With the Internet suddenly came the possibility for massively distributed and free access to knowledge, where the cost of publishing, distribution and even the capability development required to understand and apply such knowledge was suddenly negligible. We created a universal, free and instant way to share knowledge, creating the opportunity for a compounding effect on our historic capacity for cumulative learning. This is worth taking a moment to consider. The technology simultaneously created an opportunity for compounding our cumulative learning whilst rendered the reasons for IP protections negligible (lowered costs of production and distribution) and yet we have seen a dramatic increase in knowledge protectionism. Isn’t it to our collective benefit to have a well educated community that can continue our trajectory of diversification and specialisation for the benefit of everyone? Anyone can get access to myriad forms of consumer entertainment but our most valuable knowledge assets are fiercely protected against general and free access, dampening our ability to learn and evolve. The increasing gap between the haves and have nots is surely symptomatic of the broader increasing gap between the empowered and disempowered, the makers and the consumers, those with knowledge and those without. Consumers are shaped by the tools and goods they have access to, and limited by their wealth and status. But makers can create the tools and goods they need, and can redefine wealth and status with a more active and able hand in shaping their own lives.

As a result of our specialisation, our interdependence and our cooperative/competitive systems, we have created greater complexity in society over time, usually accompanied with the ability to respond to greater complexity. The problem is that a lot of our solutions to change have been linear responses to an exponential problem space. the assumption that more hands will continue to make light work often ignores the need for sharing skills and knowledge, and certainly ignores where a genuinely transformative response is required. A small fire might be managed with buckets, but at some point of growth, adding more buckets becomes insufficient and new methods are required. Necessity breeds innovation and yet when did you last see real innovation that didn’t boil down to simply more or larger buckets? Iteration is rarely a form of transformation, so it is important to always clearly understand the type of problem you are dealing with and whether the planned response needs to be linear or exponential. If the former, more buckets is probably fine. If the latter, every bucket is just a distraction from developing the necessary response.

Next chapter I’ll examine how the independence movements created the philosophical pre-condition for democracy, the Internet and the dramatic paradigm shifts to follow.

Colin Charles: Premier Open Source Database Conference Call for Papers closing January 12 2018

Tue, 2018-01-02 23:01

The call for papers for Percona Live Santa Clara 2018 was extended till January 12 2018. This means you still have time to get a submission in.

Topics of interest: MySQL, MongoDB, PostgreSQL & other open source databases. Don’t forget all the upcoming databases too (there’s a long list at db-engines).

I think to be fair, in the catch all “other”, we should also be thinking a lot about things like containerisation (Docker), Kubernetes, Mesosphere, the cloud (Amazon AWS RDS, Microsoft Azure, Google Cloud SQL, etc.), analytics (ClickHouse, MariaDB ColumnStore), and a lot more. Basically anything that would benefit an audience of database geeks whom are looking at it from all aspects.

That’s not to say case studies shouldn’t be considered. People always love to hear about stories from the trenches. This is your chance to talk about just that.

Craige McWhirter: Resolving a Partitioned RabbitMQ Cluster with JuJu

Tue, 2018-01-02 17:44

On occasion, a RabbitMQ cluster may partition itself. In a OpenStack environment this can often first present itself as nova-compute services stopping with errors such as these:

ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager._sync_power_states: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c ... TRACE nova.openstack.common.periodic_task self._raise_timeout_exception(msg_id) TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/", line 218, in _raise_timeout_exception TRACE nova.openstack.common.periodic_task 'Timed out waiting for a reply to message ID %s' % msg_id) TRACE nova.openstack.common.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c

Merely restarting the stopped nova-compute services will not resolve this issue.

You may also find that querying the rabbitmq service may either not return or take an awful long time to return:

$ sudo rabbitmqctl -p openstack list_queues name messages consumers status

...and in an environment managed by juju, you could also see JuJu trying to correct the RabbitMQ but failing:

$ juju stat --format tabular | grep rabbit rabbitmq-server false local:trusty/rabbitmq-server-128 rabbitmq-server/0 idle 0/lxc/12 5672/tcp rabbitmq-server/1 error idle 1/lxc/8 5672/tcp hook failed: "config-changed" rabbitmq-server/2 error idle 2/lxc/10 5672/tcp hook failed: "config-changed"

You should now run rabbitmqctl cluster_status on each of your rabbit instances and review the output. If the cluster is partitioned, you will see something like the below:

ubuntu@my_juju_lxc:~$ sudo rabbitmqctl cluster_status Cluster status of node 'rabbit@192-168-7-148' ... [{nodes,[{disc,['rabbit@192-168-7-148','rabbit@192-168-7-163', 'rabbit@192-168-7-174']}]}, {running_nodes,['rabbit@192-168-7-174','rabbit@192-168-7-148']}, {partitions,[{'rabbit@192-168-7-174',['rabbit@192-168-7-163']}, {'rabbit@192-168-7-148',['rabbit@192-168-7-163']}]}] ...done.

You can clearly see from the above that there are two partitions for RabbitMQ. We need to now identify which of these is considered the leader:

maas-my_cloud:~$ juju run --service rabbitmq-server "is-leader" - MachineId: 0/lxc/12 Stderr: | Stdout: | True UnitId: rabbitmq-server/0 - MachineId: 1/lxc/8 Stderr: | Stdout: | False UnitId: rabbitmq-server/1 - MachineId: 2/lxc/10 Stderr: | Stdout: | False UnitId: rabbitmq-server/2

As you see above, in this example machine 0/lxc/12 is the leader, via it's status of "True". Now we need to hit the other two servers and shut down RabbitMQ:

# service rabbitmq-server stop

Once both services have completed shutting down, we can resolve the partitioning by running:

$ juju resolved -r rabbitmq-server/<whichever is leader>

Substituting <whichever is leader> for the machine ID identified earlier.

Once that has completed, you can start the previously stopped services with the below on each host:

# service rabbitmq-server start

and verify the result with:

$ sudo rabbitmqctl cluster_status Cluster status of node 'rabbit@192-168-7-148' ... [{nodes,[{disc,['rabbit@192-168-7-148','rabbit@192-168-7-163', 'rabbit@192-168-7-174']}]}, {running_nodes,['rabbit@192-168-7-163','rabbit@192-168-7-174', 'rabbit@192-168-7-148']}, {partitions,[]}] ...done.

No partitions \o/

The JuJu errors for RabbitMQ should clear within a few minutes:

$ juju stat --format tabular | grep rabbit rabbitmq-server false local:trusty/rabbitmq-server-128 rabbitmq-server/0 idle 0/lxc/12 5672/tcp 19 rabbitmq-server/1 unknown idle 1/lxc/8 5672/tcp 19 rabbitmq-server/2 unknown idle 2/lxc/10 5672/tcp

You should also find the nova-compute instances starting up fine.

Simon Lyall: Donations 2017

Mon, 2018-01-01 11:03

Like in 2016 and 2015 I am blogging about my charity donations.

The majority of donations were done during December (I start around my birthday) although after my credit card got suspended last year I spread them across several days.

The inspiring others bit seems to have worked a little. Ed Costello has blogged his donations for 2017.

I’ll note that throughout the year I’ve also been giving money via Patreon to several people whose online content I like. I suspended these payments in early-December but they have backed down on the change so I’ll probably restart them in early 2018.

As usual my main donation was to Givewell. This year I gave to them directly and allowed them to allocate to projects as they wish.

  • $US 600 to Givewell (directly for their allocation)

In march I gave to two organization I follow online. Transport Blog re-branded themselves as “Greater Auckland” and is positioning themselves as a lobbying organization as well as news site.

Signum University produce various education material around science-fiction, fantasy and medieval literature. In my case I’m following their lectures on Youtube about the Lord of the Rings.

I gave some money to the Software Conservancy to allocate across their projects and again to the Electronic Frontier Foundation for their online advocacy.

and lastly I gave to various Open Source Projects that I regularly use.

Craige McWhirter: First Look at Snaps

Wed, 2017-12-27 22:37

I've belatedly come to have a close up look at both Ubuntu Core (Snappy), Snaps and the Snappy package manager.

The first pass was to rebuild my rack of Raspberry Pi's from Debian armhf to Ubuntu Core for the Raspberry Pi.

This proved to be the most graceful install I've ever had on any hardware, ever. No hyperbole: boot, authenticate, done. I repeated this for all six Pi's in such a short time frame that I was concerned I'd done something wrong. Your SSH keys are already installed, you can log in immediately and just get on with it.

Which is where snaps come into play.

Back on my laptop, I followed the tutorial Create Your First Snap which uses GNU Hello as an example snap build and finishes with a push to the snap store at

I then created a Launchpad Repo, related a snap package, told it to build for armhf and amd64 and before long, I could install this snap on both my laptop and the Pi's.

Overall this was a pretty impressive and graceful process.

Russell Coker: Designing Shared Cars

Thu, 2017-12-21 01:03

Almost 10 years ago I blogged about car sharing companies in Melbourne [1]. Since that time the use of such services appears to have slowly grown (judging by the slow growth in the reserved parking spots for such cars). This isn’t the sudden growth that public transport advocates and the operators of those companies hoped for, but it is still positive. I have just watched the documentary The Human Scale [2] (which I highly recommend) about the way that cities are designed for cars rather than for people.

I think that it is necessary to make cities more suited to the needs of people and that car share and car hire companies are an important part of converting from a car based city to a human based city. As this sort of change happens the share cars will be an increasing portion of the new car sales and car companies will have to design cars to better suit shared use.

Personalising Cars

Luxury car brands like Mercedes support storing the preferred seat position for each driver, once the basic step of maintaining separate driver profiles is done it’s an easy second step to have them accessed over the Internet and also store settings like preferred radio stations, Bluetooth connection profiles, etc. For a car share company it wouldn’t be particularly difficult to extrapolate settings based on previous use, EG knowing that I’m tall and using the default settings for a tall person every time I get in a shared car that I haven’t driven before. Having Bluetooth connections follow the user would mean having one slave address per customer instead of the current practice of one per car, the addressing is 48bit so this shouldn’t be a problem.

Most people accumulate many items in their car, some they don’t need, but many are needed. Some of the things in my car are change for parking meters, sunscreen, tools, and tissues. Car share companies have deals with councils for reserved parking spaces so it wouldn’t be difficult for them to have a deal for paying for parking and billing the driver thus removing the need for change (and the risk of a car window being smashed by some desperate person who wants to steal a few dollars). Sunscreen is a common enough item in Australia that a car share company might just provide it as a perk of using a shared car.

Most people have items like tools, a water bottle, and spare clothes that can’t be shared which tend to end up distributed in various storage locations. The solution to this might be to have a fixed size storage area, maybe based on some common storage item like a milk crate. Then everyone who is a frequent user of shared cars could buy a container designed to fit that space which is divided in a similar manner to a Bento box to contain whatever they need to carry.

There is a lot of research into having computers observing the operation of a car and warning the driver or even automatically applying the brakes to avoid a crash. For shared cars this is more important as drivers won’t necessarily have a feel for the car and can’t be expected to drive as well.

Car Sizes

Generally cars are designed to have 2 people (sports car, Smart car, van/ute/light-truck), 4/5 people (most cars), or 6-8 people (people movers). These configurations are based on what most people are able to use all the time. Most car travel involves only one adult. Most journeys appear to have no passengers or only children being driven around by a single adult.

Cars are designed for what people can drive all the time rather than what would best suit their needs most of the time. Almost no-one is going to buy a personal car that can only take one person even though most people who drive will be on their own for most journeys. Most people will occasionally need to take passengers and that occasional need will outweigh the additional costs in buying and fueling a car with the extra passenger space.

I expect that when car share companies get a larger market they will have several vehicles in the same location to allow users to choose which to drive. If such a choice is available then I think that many people would sometimes choose a vehicle with no space for passengers but extra space for cargo and/or being smaller and easier to park.

For the common case of one adult driving small children the front passenger seat can’t be used due to the risk of airbags killing small kids. A car with storage space instead of a front passenger seat would be more useful in that situation.

Some of these possible design choices can also be after-market modifications. I know someone who removed the rear row of seats from a people-mover to store the equipment for his work. That gave a vehicle with plenty of space for his equipment while also having a row of seats for his kids. If he was using shared vehicles he might have chosen to use either a vehicle well suited to cargo (a small van or ute) or a regular car for transporting his kids. It could be that there’s an untapped demand for ~4 people in a car along with cargo so a car share company could remove the back row of seats from people movers to cater to that.

Related posts:

  1. Designing Unsafe Cars The LA Times has an interesting article about problems with...
  2. A Better Design for Child Seats The current method of carrying young children (less than 4-6...
  3. Hyperthermia and Children in Cars Bruce Schneier writes about the risks involving children abandoned in...

Colin Charles: Percona Live Santa Clara 2018 CFP

Mon, 2017-12-18 21:01

Percona Live Santa Clara 2018 call for papers ends fairly soon — December 22 2017. It may be extended, but I suggest getting a topic in ASAP so the conference committee can view everything fairly and quickly. Remember this conference is bigger than just MySQL, so please submit topics on MongoDB, other databases like PostgreSQL, time series, etc., and of course MySQL.

What are you waiting for? Submit TODAY!
(It goes without saying that speakers get a free pass to attend the event.)

OpenSTEM: Celebration Time!

Fri, 2017-12-15 11:05
Here at OpenSTEM we have a saying “we have a resource on that” and we have yet to be caught out on that one! It is a festive time of year and if you’re looking for resources reflecting that theme, then here are some suggestions: Celebrations in Australia – a resource covering the occasions we […]

Russell Coker: Huawei Mate9

Thu, 2017-12-14 23:02
Warranty Etc

I recently got a Huawei Mate 9 phone. My previous phone was a Nexus 6P that died shortly before it’s one year warranty ran out. As there have apparently been many Nexus 6P phones dying there are no stocks of replacements so Kogan (the company I bought the phone from) offered me a choice of 4 phones in the same price range as a replacement.

Previously I had chosen to avoid the extended warranty offerings based on the idea that after more than a year the phone won’t be worth much and therefore getting it replaced under warranty isn’t as much of a benefit. But now that it seems that getting a phone replaced with a newer and more powerful model is a likely outcome it seems that there are benefits in a longer warranty. I chose not to pay for an “extended warranty” on my Nexus 6P because getting a new Nexus 6P now isn’t such a desirable outcome, but when getting a new Mate 9 is a possibility it seems more of a benefit to get the “extended warranty”. OTOH Kogan wasn’t offering more than 2 years of “warranty” recently when buying a phone for a relative, so maybe they lost a lot of money on replacements for the Nexus 6P.


I chose the Mate 9 primarily because it has a large screen. It’s 5.9″ display is only slightly larger than the 5.7″ displays in the Nexus 6P and the Samsung Galaxy Note 3 (my previous phone). But it is large enough to force me to change my phone use habits.

I previously wrote about matching phone size to the user’s hand size [1]. When writing that I had the theory that a Note 2 might be too large for me to use one-handed. But when I owned those phones I found that the Note 2 and Note 3 were both quite usable in one-handed mode. But the Mate 9 is just too big for that. To deal with this I now use the top corners of my phone screen for icons that I don’t tend to use one-handed, such as Facebook. I chose this phone knowing that this would be an issue because I’ve been spending more time reading web pages on my phone and I need to see more text on screen.

Adjusting my phone usage to the unusually large screen hasn’t been a problem for me. But I expect that many people will find this phone too large. I don’t think there are many people who buy jeans to fit a large phone in the pocket [2].

A widely touted feature of the Mate 9 is the Leica lens which apparently gives it really good quality photos. I haven’t noticed problems with my photos on my previous two phones and it seems likely that phone cameras have in most situations exceeded my requirements for photos (I’m not a very demanding user). One thing that I miss is the slow-motion video that the Nexus 6P supports. I guess I’ll have to make sure my wife is around when I need to make slow motion video.

My wife’s Nexus 6P is well out of warranty. Her phone was the original Nexus 6P I had. When her previous phone died I had a problem with my phone that needed a factory reset. It’s easier to duplicate the configuration to a new phone than restore it after a factory reset (as an aside I believe Apple does this better) I copied my configuration to the new phone and then wiped it for my wife to use.

One noteworthy but mostly insignificant feature of the Mate 9 is that it comes with a phone case. The case is hard plastic and cracked when I unsuccessfully tried to remove it, so it seems to effectively be a single-use item. But it is good to have that in the box so that you don’t have to use the phone without a case on the first day, this is something almost every other phone manufacturer misses. But there is the option of ordering a case at the same time as a phone and the case isn’t very good.

I regard my Mate 9 as fairly unattractive. Maybe if I had a choice of color I would have been happier, but it still wouldn’t have looked like EVE from Wall-E (unlike the Nexus 6P).

The Mate 9 has a resolution of 1920*1080, while the Nexus 6P (and many other modern phones) has a resolution of 2560*1440 I don’t think that’s a big deal, the pixels are small enough that I can’t see them. I don’t really need my phone to have the same resolution as the 27″ monitor on my desktop.

The Mate 9 has 4G of RAM and apps seem significantly less likely to be killed than on the Nexus 6P with 3G. I can now switch between memory hungry apps like Pokemon Go and Facebook without having one of them killed by the OS.


The OS support from Huawei isn’t nearly as good as a Nexus device. Mine is running Android 7.0 and has a security patch level of the 5th of June 2017. My wife’s Nexus 6P today got an update from Android 8.0 to 8.1 which I believe has the fixes for KRACK and Blueborne among others.

Kogan is currently selling the Pixel XL with 128G of storage for $829, if I was buying a phone now that’s probably what I would buy. It’s a pity that none of the companies that have manufactured Nexus devices seem to have learned how to support devices sold under their own name as well.


Generally this is a decent phone. As a replacement for a failed Nexus 6P it’s pretty good. But at this time I tend to recommend not buying it as the first generation of Pixel phones are now cheap enough to compete. If the Pixel XL is out of your price range then instead of saving $130 for a less secure phone it would be better to save $400 and choose one of the many cheaper phones on offer.

Remember when Linux users used to mock Windows for poor security? Now it seems that most Android devices are facing the security problems that Windows used to face and the iPhone and Pixel are going to take the role of the secure phone.

Related posts:

  1. Another Broken Nexus 5 In late 2013 I bought a Nexus 5 for my...
  2. Samsung Galaxy Note 3 In June last year I bought a Samsung Galaxy Note...
  3. The Nexus 5 The Nexus 5 is the latest Android phone to be...

Russell Coker: Thinkpad X301

Wed, 2017-12-13 19:02
Another Broken Thinkpad

A few months ago I wrote a post about “Observing Reliability” [1] regarding my Thinkpad T420. I noted that the T420 had been running for almost 4 years which was a good run, and therefore the failed DVD drive didn’t convince me that Thinkpads have quality problems.

Since that time the plastic on the lid by the left hinge broke, every time I open or close the lid it breaks a bit more. That prevents use of that Thinkpad by anyone who wants to use it as a serious laptop as it can’t be expected to last long if opened and closed several times a day. It probably wouldn’t be difficult to fix the lid but for an old laptop it doesn’t seem worth the effort and/or money. So my plan now is to give the Thinkpad to someone who wants a compact desktop system with a built-in UPS, a friend in Vietnam can probably find a worthy recipient.

My Thinkpad History

I bought the Thinkpad T420 in October 2013 [2], it lasted about 4 years and 2 months. It cost $306.

I bought my Thinkpad T61 in February 2010 [3], it lasted about 3 years and 8 months. It cost $796 [4].

Prior to the T61 I had a T41p that I received well before 2006 (maybe 2003) [5]. So the T41p lasted close to 7 years, as it was originally bought for me by a multinational corporation I’m sure it cost a lot of money. By the time I bought the T61 it had display problems, cooling problems, and compatibility issues with recent Linux distributions.

Before the T41p I had 3 Thinkpads in 5 years, all of which had the type of price that only made sense in the dot-com boom.

In terms of absolute lifetime the Thinkpad T420 did ok. In terms of cost per year it did very well, only $6 per month. The T61 was $18 per month, and while the T41p lasted a long time it probably cost over $2000 giving it a cost of over $20 per month. $20 per month is still good value, I definitely get a lot more than $20 per month benefit from having a laptop. While it’s nice that my most recent laptop could be said to have saved me $12 per month over the previous one, it doesn’t make much difference to my financial situation.

Thinkpad X301

My latest Thinkpad is an X301 that I found on an e-waste pile, it had a broken DVD drive which is presumably the reason why someone decided to throw it out. It has the same power connector as my previous 2 Thinkpads which was convenient as I didn’t find a PSU with it. I saw a review of the T301 dated 2008 which probably means it was new in 2009, but it has no obvious signs of wear so probably hasn’t been used much.

My X301 has a 1440*900 screen which isn’t as good as the T420 resolution of 1600*900. But a lower resolution is an expected trade-off for a smaller laptop. The T310 comes with a 64G SSD which is a significant limitation.

I previously wrote about a “cloud lifestyle” [6]. I hadn’t implemented all the ideas from that post due to distractions and a lack of time. But now that I’ll have a primary PC with only 64G of storage I have more incentive to do that. The 100G disk in the T61 was a minor limitation at the time I got it but since then everything got bigger and 64G is going to be a big problem and the fact that it’s an unusual 1.8″ form factor means that I can’t cheaply upgrade it or use the SSD that I’ve used in the Thinkpad T420.

My current Desktop PC is an i7-2600 system which builds the SE Linux policy packages for Debian (the thing I compile most frequently) in about 2 minutes with about 5 minutes of CPU time used. the same compilation on the X301 takes just over 6.5 minutes with almost 9 minutes of CPU time used. The i5 CPU in the Thinkpad T420 was somewhere between those times. While I can wait 6.5 minutes for a compile to test something it is an annoyance. So I’ll probably use one of the i7 or i5 class servers I run to do builds.

On the T420 I had chroot environments running with systemd-nspawn for the last few releases of Debian in both AMD64 and i386 variants. Now I have to use a server somewhere for that.

I stored many TV shows, TED talks, and movies on the T420. Probably part of the problem with the hinge was due to adjusting the screen while watching TV in bed. Now I have a phone with 64G of storage and a tablet with 32G so I will use those for playing videos.

I’ve started to increase my use of Git recently. There’s many programs I maintain that I really should have had version control for years ago. Now the desire to develop them on multiple systems gives me an incentive to do this.

Comparing to a Phone

My latest phone is a Huawei Mate 9 (I’ll blog about that shortly) which has a 1920*1080 screen and 64G of storage. So it has a higher resolution screen than my latest Thinkpad as well as equal storage. My phone has 4G of RAM while the Thinkpad only has 2G (I plan to add RAM soon).

I don’t know of a good way of comparing CPU power of phones and laptops (please comment if you have suggestions about this). The issues of GPU integration etc will make this complex. But I’m sure that the octa-core CPU in my phone doesn’t look too bad when compared to the dual-core CPU in my Thinkpad.


The X301 isn’t a laptop I would choose to buy today. Since using it I’ve appreciated how small and light it is, so I would definitely consider a recent X series. But being free the value for money is NaN which makes it more attractive. Maybe I won’t try to get 4+ years of use out of it, in 2 years time I might buy something newer and better in a similar form factor.

I can just occasionally poll an auction site and bid if there’s anything particularly tempting. If I was going to buy a new laptop now before the old one becomes totally unusable I would be rushed and wouldn’t get the best deal (particularly given that it’s almost Christmas).

Who knows, I might even find something newer and better on an e-waste pile. It’s amazing the type of stuff that gets thrown out nowadays.

Related posts:

  1. Observing Reliability Last year I wrote about how great my latest Thinkpad...
  2. I Just Bought a new Thinkpad and the Lenovo Web Site Sucks I’ve just bought a Thinkpad T61 at auction for $AU796....
  3. Thinkpad T420 I’ve owned a Thinkpad T61 since February 2010 [1]. In...

Francois Marier: Using all of the 5 GHz WiFi frequencies in a Gargoyle Router

Mon, 2017-12-11 13:03

WiFi in the 2.4 GHz range is usually fairly congested in urban environments. The 5 GHz band used to be better, but an increasing number of routers now support it and so it has become fairly busy as well. It turns out that there are a number of channels on that band that nobody appears to be using despite being legal in my region.

Why are the middle channels unused?

I'm not entirely sure why these channels are completely empty in my area, but I would speculate that access point manufacturers don't want to deal with the extra complexity of the middle channels. Indeed these channels are not entirely unlicensed. They are also used by weather radars, for example. If you look at the regulatory rules that ship with your OS:

$ iw reg get global country CA: DFS-FCC (2402 - 2472 @ 40), (N/A, 30), (N/A) (5170 - 5250 @ 80), (N/A, 17), (N/A), AUTO-BW (5250 - 5330 @ 80), (N/A, 24), (0 ms), DFS, AUTO-BW (5490 - 5600 @ 80), (N/A, 24), (0 ms), DFS (5650 - 5730 @ 80), (N/A, 24), (0 ms), DFS (5735 - 5835 @ 80), (N/A, 30), (N/A)

you will see that these channels are flagged with "DFS". That stands for Dynamic Frequency Selection and it means that WiFi equipment needs to be able to detect when the frequency is used by radars (by detecting their pulses) and automaticaly switch to a different channel for a few minutes.

So an access point needs extra hardware and extra code to avoid interfering with priority users. Additionally, different channels have different bandwidth limits so that's something else to consider if you want to use 40/80 MHz at once.

Using all legal channels in Gargoyle

The first time I tried setting my access point channel to one of the middle 5 GHz channels, the SSID wouldn't show up in scans and the channel was still empty in WiFi Analyzer.

I tried changing the channel again, but this time, I ssh'd into my router and looked at the errors messages using this command:

logread -f

I found a number of errors claiming that these channels were not authorized for the "world" regulatory authority.

Because Gargoyle is based on OpenWRT, there are a lot more nnwireless configuration options available than what's exposed in the Web UI.

In this case, the solution was to explicitly set my country in the wireless options by putting:

country 'CA'

(where CA is the country code where the router is physically located) in the 5 GHz radio section of /etc/config/wireless on the router.

Then I rebooted and I was able to set the channel successfully via the Web UI.

If you are interested, there is a lot more information about how all of this works in the kernel documentation for the wireless stack.

OpenSTEM: Happy Holidays, Queensland!

Fri, 2017-12-08 15:04
It’s finally holidays in Queensland! Yay! Congratulations to everyone for a wonderful year and lots of hard work! Hope you all enjoy a well-earned rest! Most other states and territories have only a week to go, but the holiday spirit is in the air.- Should you be looking for help with resources, rest assured that […]

Lev Lafayette: A Tale of Two Conferences: ISC and TERATEC 2017

Tue, 2017-12-05 17:04

This year the International Supercomputing Conference and TERATEC were held in close proximity, the former in Frankfurt from June 17-21 and the latter in Paris from June 27-28. Whilst the two conferences differ greatly in scope (one international, one national) and language (one Anglophone, the other Francophone), the dominance of Linux as the operating system of
choice at both was overwhelming.

read more

David Rowe: How Inlets Generate Thrust on Supersonic aircraft

Sun, 2017-12-03 13:04

Some time ago I read Skunk Works, a very good “engineering” read.

In the section on the SR-71, the author Ben Rich made a statement that has puzzled me ever since, something like: “Most of the engines thrust is developed by the intake”. I didn’t get it – surely an intake is a source of drag rather than thrust? I have since read the same statement about the Concorde and it’s inlets.

Lately I’ve been watching a lot of AgentJayZ Gas Turbine videos. This guy services gas turbines for a living and is kind enough to present a lot of intricate detail and answer questions from people. I find his presentation style and personality really engaging, and get a buzz out of his enthusiasm, love for his work, and willingness to share all sorts of geeky, intricate details.

So inspired by AgentJayZ I did some furious Googling and finally worked out why supersonic planes develop thrust from their inlets. I don’t feel it’s well explained elsewhere so here is my attempt:

  1. Gas turbine jet engines only work if the air is moving into the compressor at subsonic speeds. So the job of the inlet is to slow the air down from say Mach 2 to Mach 0.5.
  2. When you slow down a stream of air, the pressure increases. Like when you feel the wind pushing on your face on a bike. Imagine (don’t try) the pressure on your arm hanging out of a car window at 100 km/hr. Now imagine the pressure at 3000 km/hr. Lots. Around a 40 times increase for the inlets used in supersonic aircraft.
  3. So now we have this big box (the inlet chamber) full of high pressure air. Like a balloon this pressure is pushing equally on all sides of the box. Net thrust is zero.
  4. If we untie the balloon neck, the air can escape, and the balloon shoots off in the opposite direction.
  5. Back to the inlet on the supersonic aircraft. It has a big vacuum cleaner at the back – the compressor inlet of the gas turbine. It is sucking air out of the inlet as fast as it can. So – the air can get out, just like the balloon, and the inlet and the aircraft attached to it is thrust in the opposite direction. That’s how an inlet generates thrust.
  6. While there is also thrust from the gas turbine and it’s afterburner, turns out that pressure release in the inlet contributes the majority of the thrust. I don’t know why it’s the majority. Guess I need to do some more reading and get my gas equations on.

Another important point – the aircraft really does experience that extra thrust from the inlet – e.g. it’s transmitted to the aircraft by the engine mounts on the inlet, and the mounts must be designed with those loads in mind. This helps me understand the definition of “thrust from the inlet”.

Pia Waugh: My Canadian adventure exploring FWD50

Fri, 2017-12-01 17:01

I recently went to Ottawa for the FWD50 conference run by Rebecca and Alistair Croll. It was my first time in Canada, and it combined a number of my favourite things. I was at an incredible conference with a visionary and enthusiastic crowd, made up of government (international, Federal, Provincial and Municipal), technologists, civil society, industry, academia, and the calibre of discussions and planning for greatness was inspiring.

There was a number of people I have known for years but never met in meatspace, and equally there were a lot of new faces doing amazing things. I got to spend time with the excellent people at the Treasury Board of Canadian Secretariat, including the Canadian Digital Service and the Office of the CIO, and by wonderful coincidence I got to see (briefly) the folk from the Open Government Partnership who happened to be in town. Finally I got to visit the gorgeous Canadian Parliament, see their extraordinary library, and wander past some Parliamentary activity which always helps me feel more connected to (and therefore empowered to contribute to) democracy in action.

Thank you to Alistair Croll who invited me to keynote this excellent event and who, with Rebecca Croll, managed to create a truly excellent event with a diverse range of ideas and voices exploring where we could or should go as a society in future. I hope it is a catalyst for great things to come in Canada and beyond.

For those in Canada who are interested in the work in New Zealand, I strongly encourage you to tune into the D5 event in February which will have some of our best initiatives on display, and to tune in to our new Minister for Broadband, Digital and Open Government (such an incredible combination in a single portfolio), Minister Clare Curran and you can tune in to our “Service Innovation” work at our blog or by subscribing to our mailing list. I also encourage you to read this inspiring “People’s Agenda” by a civil society organisation in NZ which codesigned a vision for the future type of society desired in New Zealand.


  • One of the great delights of this trip was seeing a number of people in person for the first time who I know from the early “Gov 2.0″ days (10 years ago!). It was particularly great to see Thom Kearney from Canada’s TBS and his team, Alex Howard (@digiphile) who is now a thought leader at the Sunlight Foundation, and Olivia Neal (@livneal) from the UK CTO office/GDS, Joe Powell from OGP, as well as a few friends from Linux and Open Source (Matt and Danielle amongst others).
  • The speech by Canadian Minister of the Treasury Board Secretariat (which is responsible for digital government) the Hon Scott Brison, was quite interesting and I had the chance to briefly chat to him and his advisor at the speakers drinks afterwards about the challenges of changing government.
  • Meeting with Canadian public servants from a variety of departments including the transport department, innovation and science, as well as the Treasury Board Secretariat and of course the newly formed Canadian Digital Service.
  • Meeting people from a range of sub-national governments including the excellent folk from Peel, Hillary Hartley from Ontario, and hearing about the quite inspiring work to transform organisational structures, digital and other services, adoption of micro service based infrastructure, the use of “labs” for experimentation.
  • It was fun meeting some CIO/CTOs from Canada, Estonia, UK and other jurisdictions, and sharing ideas about where to from here. I was particularly impressed with Alex Benay (Canadian CIO) who is doing great things, and with Siim Sikkut (Estonian CIO) who was taking the digitisation of Estonia into a new stage of being a broader enabler for Estonians and for the world. I shared with them some of my personal lessons learned around digital iteration vs transformation, including from the DTO in Australia (which has changed substantially, including a name change since I was there). Some notes of my lessons learned are at
  • My final highlight was how well my keynote and other talks were taken. People were really inspired to think big picture and I hope it was useful in driving some of those conversations about where we want to collectively go and how we can better collaborate across geopolitical lines.

Below are some photos from the trip, and some observations from specific events/meetings.

My FWD50 Keynote – the Tipping Point

I was invited to give a keynote at FWD50 about the tipping point we have gone through and how we, as a species, need to embrace the major paradigm shifts that have already happened, and decide what sort of future we want and work towards that. I also suggested some predictions about the future and examined the potential roles of governments (and public sectors specifically) in the 21st century. The slides are at and the full speech is on my personal blog at

I also gave a similar keynote speech at the NerHui conference in New Zealand the week after which was recorded for those who want to see or hear the content at

The Canadian Digital Service

Was only set up about a year ago and has a focus on building great services for users, with service design and user needs at the heart of their work. They have some excellent people with diverse skills and we spoke about what is needed to do “digital government” and what that even means, and the parallels and interdependencies between open government and digital government. They spoke about an early piece of work they did before getting set up to do a national consultation about the needs of Canadians ( which had some interesting insights. They were very focused on open source, standards, building better ways to collaborate across government(s), and building useful things. They also spoke about their initial work around capability assessment and development across the public sector. I spoke about my experience in Australia and New Zealand, but also in working and talking to teams around the world. I gave an informal outline about the work of our Service Innovation and Service Integration team in DIA, which was helpful to get some feedback and peer review, and they were very supportive and positive. It was an excellent discussion, thank you all!

CivicTech meetup

I was invited to talk to the CivicTech group meetup in Ottawa ( about the roles of government and citizens into the future. I gave a quick version of the keynote I gave at 2017 (, which explores paradigm shifts and the roles of civic hackers and activists in helping forge the future whilst also considering what we should (and shouldn’t) take into the future with us. It included my amusing change.log of the history of humans and threw down the gauntlet for civic hackers to lead the way, be the light

CDS Halloween Mixer

The Canadian Digital Service does a “mixer” social event every 6 weeks, and this one landed on Halloween, which was also my first ever Halloween celebration  I had a traditional “beavertail” which was a flat cinnamon doughnut with lemon, amazing! Was fun to hang out but of course I had to retire early from jet lag.

Workshop with Alistair

The first day of FWD50 I helped Alistair Croll with a day long workshop exploring the future. We thought we’d have a small interactive group and ended up getting 300, so it was a great mind meld across different ideas, sectors, technologies, challenges and opportunities. I gave a talk on culture change in government, largely influenced by a talk a few years ago called “Collaborative innovation in the public service: Game of Thrones style” ( People responded well and it created a lot of discussions about the cultural challenges and barriers in government.


Finally, just a quick shout out and thanks to Alistair for inviting me to such an amazing conference, to Rebecca for getting me organised, to Danielle and Matthew for your companionship and support, to everyone for making me feel so welcome, and to the following folk who inspired, amazed and colluded with me  In chronological order of meeting: Sean Boots, Stéphane Tourangeau, Ryan Androsoff, Mike Williamson, Lena Trudeau, Alex Benay (Canadian Gov CIO), Thom Kearney and all the TBS folk, Siim Sikkut from Estonia, James Steward from UK, and all the other folk I met at FWD50, in between feeling so extremely unwell!

Thank you Canada, I had a magnificent time and am feeling inspired!

OpenSTEM: This Week in HASS – term 4, week 9

Fri, 2017-12-01 13:05
Well, we’re almost at the end of the year!! It’s a time when students and teachers alike start to look forward to the long, summer break. Generally a time for celebrations and looking back over the highlights of the year – which is reflected in the activities for the final lessons of the Understanding Our […]

Francois Marier: Proxy ACME challenges to a single machine

Wed, 2017-11-29 17:10

The Libravatar mirrors are setup using DNS round-robin which makes it a little challenging to automatically provision Let's Encrypt certificates.

In order to be able to use Certbot's webroot plugin, I need to be able to simultaneously host a randomly-named file into the webroot of each mirror. The reason is that the verifier will connect to, but there's no way to know which of the DNS entries it will hit. I could copy the file over to all of the mirrors, but that would be annoying since some of the mirrors are run by volunteers and I don't have direct access to them.

Thankfully, Scott Helme has shared his elegant solution: proxy the .well-known/acme-challenge/ directory from all of the mirrors to a single validation host. Here's the exact configuration I ended up with.

DNS Configuration

In order to serve the certbot validation files separately from the main service, I created a new hostname,, pointing to the main Libravatar server:

CNAME acme Mirror Configuration

On each mirror, I created a new Apache vhost on port 80 to proxy the acme challenge files by putting the following in the existing port 443 vhost config (/etc/apache2/sites-available/libravatar-seccdn.conf):

<VirtualHost *:80> ServerName __SECCDNSERVERNAME__ ServerAdmin __WEBMASTEREMAIL__ ProxyPass /.well-known/acme-challenge/ ProxyPassReverse /.well-known/acme-challenge/ </VirtualHost>

Then I enabled the right modules and restarted Apache:

a2enmod proxy a2enmod proxy_http systemctl restart apache2.service

Finally, I added a cronjob in /etc/cron.daily/commit-new-seccdn-cert to commit the new cert to etckeeper automatically:

#!/bin/sh cd /etc/libravatar /usr/bin/git commit --quiet -m "New seccdn cert" seccdn.crt seccdn.pem seccdn-chain.pem > /dev/null || true Main Configuration

On the main server, I created a new webroot:

mkdir -p /var/www/acme/.well-known

and a new vhost in /etc/apache2/sites-available/acme.conf:

<VirtualHost *:80> ServerName ServerAdmin DocumentRoot /var/www/acme <Directory /var/www/acme> Options -Indexes </Directory> </VirtualHost>

before enabling it and restarting Apache:

a2ensite acme systemctl restart apache2.service Registering a new TLS certificate

With all of this in place, I was able to register the cert easily using the webroot plugin on the main server:

certbot certonly --webroot -w /var/www/acme -d

The resulting certificate will then be automatically renewed before it expires.