Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 7 min 11 sec ago

Michael Still: Rejected talk proposal: Design at scale: OpenStack versus Kubernetes

Sun, 2018-06-17 19:00

This proposal was submitted for pyconau 2018. It wasn’t accepted, but given I’d put the effort into writing up the proposal I’ll post it here in case its useful some other time. The oblique references to OpensStack are because pycon had an “anonymous” review system in 2018, and I was avoiding saying things which directly identified me as the author.

OpenStack and Kubernetes solve very similar problems. Yet they approach those problems in very different ways. What can we learn from the different approaches taken? The differences aren’t just technical though, there are some interesting social differences too.

OpenStack and Kubernetes solve very similar problems – at their most basic level they both want to place workloads on large clusters of machines, and ensure that those placement decisions are as close to optimal as possible. The two projects even have similar approaches to the fundamentals – they are both orchestration systems at their core, seeking to help existing technologies run at scale instead of inventing their own hypervisors or container run times.

Yet they have very different approaches to how to perform these tasks. OpenStack takes a heavily centralised and monolithic approach to orchestration, whilst Kubernetes has a less stateful and more laissez faire approach. Some of that is about early technical choices and the heritage of the projects, but some of it is also about hubris and a desire to tightly control. To be honest I lived the OpenStack experience so I feel I should be solidly in that camp, but the Kubernetes approach is clever and elegant. There’s a lot to like on the Kubernetes side of the fence.

Its increasingly common that at some point you’ll encounter one of these systems, as neither seems likely to go away in the next few years. Understanding some of the basics of their operation is therefore useful, as well as being interesting at a purely hypothetical level.

The post Rejected talk proposal: Design at scale: OpenStack versus Kubernetes appeared first on Made by Mikal.

Michael Still: Rejected talk proposal: Learning from the mistakes that even big projects make

Sun, 2018-06-17 19:00

This proposal was submitted for pyconau 2018. It wasn’t accepted, but given I’d put the effort into writing up the proposal I’ll post it here in case its useful some other time. The oblique references to OpensStack are because pycon had an “anonymous” review system in 2018, and I was avoiding saying things which directly identified me as the author.

Since 2011, I’ve worked on a large Open Source project in python. It kind of got out of hand – 1000s of developers and millions of lines of code. Yet despite being well resourced, we made the same mistakes that those tiny scripts you whip up to solve a small problem make. Come learn from our fail.

This talk will use the privilege separation daemon that the project wrote to tell the story of decisions that were expedient at the time, and how we regretted them later. In a universe in which you can only run commands as root via sudo, dd’ing from one file on the filesystem to another seems almost reasonable. Especially if you ignore that the filenames are defined by the user. Heck, we shell out to “mv” to move files around, even when we don’t need escalated permissions to move the file in question.

While we’ll focus mainly on the security apparatus because it is the gift that keeps on giving, we’ll bump into other examples along the way as well. For example how we had pluggable drivers, but you have to turn them on by passing in python module paths. So what happens when we change the interface the driver is required to implement and you have a third party driver? The answer isn’t good. Or how we refused to use existing Open Source code from other projects through a mixture of hubris and licensing religion.

On a strictly technical front, this is a talk about how to do user space privilege separation sensibly. Although we should probably discuss why we also chose in the last six months to not do it as safely as we could.

For a softer technical take, the talk will cover how doing things right was less well documented than doing things the wrong way. Code reviewers didn’t know the anti-patterns, which were common in the code base, so made weird assumptions about what was ok or not.

On a human front, this is about herding cats. Developers with external pressures from their various employers, skipping steps because it was expedient, and how throwing automation in front of developers because having a conversation as adults is hard. Ultimately we ended up being close to stalled before we were “saved” from an unexpected direction.

In the end I think we’re in a reasonable place now, so I certainly don’t intend to give a lecture about doom and gloom. Think of us more as a light hearted object lesson.

The post Rejected talk proposal: Learning from the mistakes that even big projects make appeared first on Made by Mikal.

Donna Benjamin: The Five Whys

Sat, 2018-06-16 11:02
Saturday, June 16, 2018 - 09:16

Imagine you work in a hardware store. You notice a customer puzzling over the vast array of electric drills.

She turns to you and says, I need a drill, but I don’t know which one to pick.

You ask “So, why do you want a drill?

“To make a hole.” she replies, somewhat exasperated. “Isn’t that obvious?”

“Sure,” you might say, “But why do you want to drill a hole? It might help us decide which drill you need!” “

Oh, okay," and she goes on to describe the need to thread cable from one room, to another.

From there, we might want to know more about the walls, about the type and thickness of the cable, and perhaps about what the cable is for. But what if we keep asking why? What if the next question was something like this?

“Why do you want to pull the cable from one room to the other?”

Our customer then explains she wants to connect directly to the internet router in the other room. "Our wifi reception is terrible! This seemed the fastest, easiest way to fix that."

At this point, there may be other solutions to the bad wifi problem that don’t require a hole at all, let alone a drill.

Someone who needs a drill, rarely wants a drill, nor do they really want a hole.

It’s the utility of that hole that we’re trying to uncover with the 5 Whys.

Acknowledgement

I can't remember who first told me about this technique. I wish I could, it's been profoundly useful, and I evangelise it's simple power at every opportunity. Thank you who ever you are, I honour your generous wisdom by paying it forward today.

More about the Five whysImage credits

Creative Commons Icons all from the Noun Project

  • Drill by Andrejs Kirma
  • Mouse Hole by Sergey Demushkin
  • Cable by Amy Schwartz
  • Internet by Vectors Market
  • Wifi by Baboon designs
  • Not allowed by Adnen Kadri

Lev Lafayette: Being An Acrobat: Linux and PDFs

Sat, 2018-06-16 09:04

The PDF file format can be efficiently manipulated in Linux and other free software that may not be easy in proprietary operating systems or applications. This includes a review of various PDF readers for Linux, creation of PDFs from office documents using LibreOffice, editing PDF documents, converting PDF documents to images, extracting text from non-OCR PDF documents, converting to PostScript, converting restructuredText, Markdown, and other formats, searching PDFs according to regular expressions, converting to text, extracting images, separating and combining PDF documents, creating PDF presentations from text, creating fillable PDF forms, encrypting and decrypting PDF documents, and parsing PDF documents.

A presentation to Linux Users of Victoria, Saturday June 16, 2018

OpenSTEM: Assessment Time

Fri, 2018-06-15 15:05
For many of us, the colder weather has started to arrive and mid-year assessment is in full swing. Teachers are under the pump to produce mid-year reports and grades. The OpenSTEM® Understanding Our World® program aims to take the pressure off teachers by providing for continuous assessment throughout the term. Not only are teachers continually […]

Donna Benjamin: DrupalCon Nashville

Fri, 2018-06-15 11:02
Saturday, March 17, 2018 - 22:01

I'm going to Nashville!!

That is all. Carry on. Or... better yet - you should come too!

https://events.drupal.org/nashville2018

Donna Benjamin: Leadership, and teamwork.

Fri, 2018-06-15 11:02
Friday, April 13, 2018 - 04:09

I'm angry and defensive. I don't know why. So I'm trying hard to figure that out right now.

Here's some words.

I'm writing these words for myself to try and figure this out.
I'm hoping these words might help make it clear.
I'm fearful these words will make it worse.

But I don't want to be silent about this.

Content Warning: This post refers to genocide.

This is about a discussion at the teamwork and leadership workshop at DrupalCon. For perhaps 5 mins within a 90 minute session we talked about Hitler. It was an intensely thought provoking, and uncomfortable 5 minute conversation. It was nuanced. It wasn't really tweetable.

On Holocaust memorial day, it seems timely to explore whether or not we should talk about Hitler when exploring the nature of leadership. Not all leaders are good. Call them dictators, call them tyrants, call them fascists, call them evil. Leadership is defined differently by different cultures, at different times, and in different contexts.

Some people in the room were upset and disgusted that we had that conversation. I'm really very deeply sorry about that.

Some of them then talked about it with others afterwards, which is great. It was a confronting conversation, and one, frankly, we should all be having as genocide and fascism exist in very real ways in the very real world.

But some of those they spoke with, who weren't there, seem to have extrapolated from that conversation that it was something different to what I experienced in the room. I feel they formed opinions that I can only call, well, what words can I call those opinions? Uninformed? Misinformed? Out of context? Wrong? That's probably unfair, it's just my perspective. But from those opinions, they also made assumptions, and turned those assumptions into accusations.

One person said they were glad they weren't there, but clearly happy to criticise us from afar on twitter. I responded that I thought it was a shame they didn't come to the workshop, but did choose to publicly criticise our work. Others responded to that saying this was disgusting, offensive, unacceptable and inappropriate that we would even consider having this conversation. One accused me of trying to shut down the conversation.

So, I think perhaps the reason I'm feeling angry and defensive, is I'm being accused of something I don't think I did.

And I want to defend myself.

I've studied World War Two and the Genocide that took place under Hitler's direction.

My grandmother was arrested in the early 1930's and held in a concentration camp. She was, thankfully, released and fled Germany to Australia as a refugee before the war was declared. Her mother was murdered by Hitler. My grandfather's parents and sister were also murdered by Hitler.

So, I guess I feel like I've got a pretty strong understanding of who Hitler was, and what he did.

So when I have people telling me, that it's completely disgusting to even consider discussing Hitler in the context of examining what leadership is, and what it means? Fuck that. I will not desist. Hitler was a monster, and we must never forget what he was, or what he did.

During silent reflection on a number of images, I wrote this note.

"Hitler was a powerful leader. No question. So powerful, he destroyed the world."

When asked if they thought Hitler was a leader or not, most people in the room, including me, put up their hand. We were wrong.

The four people who put their hand up to say he was NOT a leader were right.

We had not collectively defined leadership at that point. We were in the middle of a process doing exactly that.

The definition we were eventually offered is that leaders must care for their followers, and must care for people generally.

At no point, did anyone in that room, consider the possibility that Hitler was a "Good Leader" which is the misinformed accusation I most categorically reject.

Our facilitator, Adam Goodman, told us we were all wrong, except the four who rejected Hitler as an example of a Leader, by saying, that no, he was not a leader, but yes, he was a dictator, yes he was a tyrant. But he was not a leader.

Whilst I agree, and was relieved by that reframing, I would also counter argue that it is English semantics.

Someone else also reminded us, that Hitler was elected. I too, was elected to the board of the Drupal Association, I was then appointed to one of the class Director seats. My final term ends later this year, and frankly, right now, I'm kind of wondering if I should leave right now.

Other people shown in the slide deck were Oprah Winfrey, Angela Merkel, Rosa Parks, Serena Williams, Marin Alsop, Sonia Sotomayor, a woman in military uniform, and a large group of women protesting in Tahrir Square in Egypt.

It also included Gandhi, and Mandela.

I observed that I felt sad I could think of no woman that I would list in the same breath as those two men.

So... for those of you who judged us, and this workshop, from what you saw on twitter, before having all the facts?
Let me tell you what I think this was about.

This wasn't about Hitler.

This was about leadership, and learning how we can be better leaders. I felt we were also exploring how we might better support the leaders we have, and nurture the ones to come. And I now also wonder how we might respectfully acknowledge the work and effort of those who've come and gone, and learn to better pass on what's important to those doing the work now.

We need teamwork. We need leadership. It takes collective effort, and most of all, it takes collective empathy and compassion.

Dries Buytaert was the final image in the deck.

Dries shared these 5 values and their underlying principles with us to further explore, discuss and develop together.

Prioritize impact
Impact gives us purpose. We build software that is easy, accessible and safe for everyone to use.

Better together
We foster a learning environment, prefer collaborative decision-making, encourage others to get involved and to help lead our community.

Strive for excellence
We constantly re-evaluate and assume that change is constant.

Treat each other with dignity and respect
We do not tolerate intolerance toward others. We seek first to understand, then to be understood. We give each other constructive criticism, and are relentlessly optimistic.

Enjoy what you do
Be sure to have fun.

I'm sorry to say this, but I'm really not having fun right now. But I am much clearer about why I'm feeling angry.

Photo Credit "Protesters against Egyptian President Mohamed Morsi celebrate in Tahrir Square in Cairo on July 3, 2013. Egypt's armed forces overthrew elected Islamist President Morsi on Wednesday and announced a political transition with the support of a wide range of political and religious leaders." Mohamed Abd El Ghany Reuters.

Donna Benjamin: Makarrata

Fri, 2018-06-15 11:02
Thursday, June 14, 2018 - 20:19

The time has come
To say fairs fair...

Dear members of the committee,

Please listen to the Uluru statement from the heart. Please hear those words. Please accept them, please act to adopt them.

Enshrine a voice for Australia’s first nation peoples in the Australian constitution.

Create a commission for Makarrata.

Invest in uncovering and telling the truth of our history.

We will be a stronger, wiser nation when we truly acknowledge the frontier wars and not only a stolen generation but stolen land, and stolen hope.

We have nothing to lose, and everything to gain through real heartfelt recognition and reconciliation.

Makarrata. Treaty. Sovereignty.

Please. I am Australian. I want this.

I felt sick shame when the prime minister rejected the Uluru statement. He did not, does not, speak for me.

Donna Benjamin
Melbourne, VIC.

Donna Benjamin: Makarrata

Thu, 2018-06-14 21:02
Thursday, June 14, 2018 - 20:19

The time has come
To say fairs fair...

Dear members of the committee,

Please listen to the Uluru statement from the heart. Please hear those words. Please accept them, please act to adopt them.

Enshrine a voice for Australia’s first nation peoples in the Australian constitution.

Create a commission for Makarrata.

Invest in uncovering and telling the truth of our history.

We will be a stronger, wiser nation when we truly acknowledge the frontier wars and not only a stolen generation but stolen land, and stolen hope.

We have nothing to lose, and everything to gain through real heartfelt recognition and reconciliation.

Makarrata. Treaty. Sovereignty.

Please. I am Australian. I want this.

I felt sick shame when the prime minister rejected the Uluru statement. He did not, does not, speak for me.

Donna Benjamin
Melbourne, VIC.

Julien Goodwin: Custom uBlox GPSDO board

Tue, 2018-06-12 23:03
For the next part of my ongoing project I needed to test the GPS reciever I'm using, a uBlox LEA-M8F (M8 series chip, LEA form factor, and with frequency outputs). Since the native 30.72MHz oscillator is useless for me I'm using an external TCVCXO (temperature compensated, voltage controlled oscillator) for now, with the DAC & reference needed to discipline the oscillator based on GPS. If uBlox would sell me the frequency version of the chip on its own that would be ideal, but they don't sell to small customers.

Here's a (rather modified) board sitting on top of an Efratom FRK rubidium standard that I'm going to mount to make a (temporary) home standard (that deserves a post of its own). To give a sense of scale the silver connector at the top of the board is a micro-USB socket.



Although a very simple board I had a mess of problems once again, both in construction and in component selection.

Unlike the PoE board from the previous post I didn't have this board manufactured. This was for two main reasons, first, the uBlox module isn't available from Digikey, so I'd still need to mount it by hand. The second, to fit all the components this board has a much greater area, and since the assembly house I use charges by board area (regardless of the number or density of components) this would have cost several hundred dollars. In the end, this might actually have been the sensible way to go.

By chance I'd picked up a new soldering iron at the same time these boards arrived, a Hakko FX-951 knock-off and gave it a try. Whilst probably an improvement over my old Hakko FX-888 it's not a great iron, especially with the knife tip it came with, and certainly nowhere near as nice to use as the JBC CD-B (I think that's the model) we have in the office lab. It is good enough that I'm probably going to buy a genuine Hakko FM-203 with an FM-2032 precision tool for the second port.

The big problem I had hand-soldering the boards was bridges on several of the components. Not just the tiny (0.65mm pitch, actually the *second largest* of eight packages for that chip) SC70 footprint of the PPS buffer, but also the much more generous 1.1mm pitch of the uBlox module. Luckily solder wick fixed most cases, plus one where I pulled the buffer and soldered a new one more carefully.

With components, once again I made several errors:
  • I ended up buying the wrong USB connectors for the footprint I chose (the same thing happened with the first run of USB-C modules I did in 2016), and while I could bodge them into use easily enough there wasn't enough mechanical retention so I ended up ripping one connector off the board. I ordered some correct ones, but because I wasn't able to wick all solder off the pads they don't attach as strongly as they should, and whilst less fragile, are hardly what I'd call solid.
  • The surface mount GPS antenna (Taoglas AP.10H.01 visible in this tweet) I used was 11dB higher gain than the antenna I'd tested with the devkit, I never managed to get it to lock while connected to the board, although once on a cable it did work ok. To allow easier testing, in the end I removed the antenna and bodged on an SMA connector for easy testing.
  • When selecting the buffer I accidentally chose one with an open-drain output, I'd meant to use one with a push-pull output. This took quite a silly long time for me to realise what mistake I'd made. Compounding this, the buffer is on the 1PPS line, which only strobes while locked to GPS, however my apartment is a concrete box, with what GPS signal I can get inside only available in my bedroom, and my oscilloscope is in my lab, so I couldn't demonstrate the issue live, and had to inject test signals. Luckily a push-pull is available in the same footprint, and a quick hot-air aided swap later (once parts arrived from Digikey) it was fixed.

Lessons learnt:
  • Yes I can solder down to ~0.5mm pitch, but not reliably.
  • More test points on dev boards, particularly all voltage rails, and notable signals not otherwise exposed.
  • Flux is magic, you probably aren't using enough.

Although I've confirmed all basic functions of the board work, including GPS locking, PPS (quick video of the PPS signal LED), and frequency output, I've still not yet tested the native serial ports and frequency stability from the oscillator. Living in an urban canyon makes such testing a pain.

Eventually I might also test moving the oscillator, DAC & reference into a mini oven to see if a custom OCXO would be any better, if small & well insulated enough the power cost of an oven shouldn't be a problem.

Also as you'll see if you look at the tweets, I really should have posted this almost a month ago, however I finished fixing the board just before heading off to California for a work trip, and whilst I meant to write this post during the trip, it's not until I've been back for more than a week that I've gotten to it. I find it extremely easy to let myself be distracted from side projects, particularly since I'm in a busy period at $ORK at the moment.

Francois Marier: Mysterious 'everybody is busy/congested at this time' error in Asterisk

Mon, 2018-06-11 12:01

I was trying to figure out why I was getting a BUSY signal from Asterisk while trying to ring a SIP phone even though that phone was not in use.

My asterisk setup looks like this:

phone 1 <--SIP--> asterisk 1 <==IAX2==> asterisk 2 <--SIP--> phone 2

While I couldn't call SIP phone #2 from SIP phone #1, the reverse was working fine (ringing #1 from #2). So it's not a network/firewall problem. The two SIP phones can talk to one another through their respective Asterisk servers.

This is the error message I could see on the second asterisk server:

$ asterisk -r ... == Using SIP RTP TOS bits 184 == Using SIP RTP CoS mark 5 -- Called SIP/12345 -- SIP/12345-00000002 redirecting info has changed, passing it to IAX2/iaxuser-6347 -- SIP/12345-00000002 is busy == Everyone is busy/congested at this time (1:1/0/0) -- Executing [12345@local:2] Goto("IAX2/iaxuser-6347", "in12345-BUSY,1") in new stack -- Goto (local,in12345-BUSY,1) -- Executing [in12345-BUSY@local:1] Hangup("IAX2/iaxuser-6347", "17") in new stack == Spawn extension (local, in12345-BUSY, 1) exited non-zero on 'IAX2/iaxuser-6347' -- Hungup 'IAX2/iaxuser-6347'

where:

  • 12345 is the extension of SIP phone #2 on Asterisk server #2
  • iaxuser is the user account on server #2 that server #1 uses
  • local is the context that for incoming IAX calls on server #1

This Everyone is busy/congested at this time (1:1/0/0) was surprising since looking at each SIP channel on that server showed nobody as busy:

asterisk2*CLI> sip show inuse * Peer name In use Limit 12345 0/0/0 2

So I enabled the raw SIP debug output and got the following (edited for clarity):

asterisk2*CLI> sip set debug on SIP Debugging enabled == Using SIP RTP TOS bits 184 == Using SIP RTP CoS mark 5 INVITE sip:12345@192.168.0.4:2048;line=m2vlbuoc SIP/2.0 Via: SIP/2.0/UDP 192.168.0.2:5060 From: "Francois Marier" <sip:67890@192.168.0.2> To: <sip:12345@192.168.0.4:2048;line=m2vlbuoc> CSeq: 102 INVITE User-Agent: Asterisk PBX Contact: <sip:67890@192.168.0.2:5060> Content-Length: 274 -- Called SIP/12345 <--- SIP read from UDP:192.168.0.4:2048 ---> SIP/2.0 100 Trying Via: SIP/2.0/UDP 192.168.0.2:5060 From: "Francois Marier" <sip:67890@192.168.0.2> To: <sip:12345@192.168.0.4:2048;line=m2vlbuoc> CSeq: 102 INVITE User-Agent: snom300 Contact: <sip:12345@192.168.0.4:2048;line=m2vlbuoc> Content-Length: 0 <-------------> --- (9 headers 0 lines) --- <--- SIP read from UDP:192.168.0.4:2048 ---> SIP/2.0 480 Do Not Disturb Via: SIP/2.0/UDP 192.168.0.2:5060 From: "Francois Marier" <sip:67890@192.168.0.2> To: <sip:12345@192.168.0.4:2048;line=m2vlbuoc> CSeq: 102 INVITE User-Agent: snom300 Contact: <sip:12345@192.168.0.4:2048;line=m2vlbuoc> Content-Length: 0

where:

  • 12345 is the extension of SIP phone #2 on Asterisk server #2
  • 67890 is the extension of SIP phone #1 on Asterisk server #2
  • 192.168.0.4 is the IP address of SIP phone #2
  • 192.168.0.1 is the IP address of Asterisk server #2

From there, I can see that SIP phone #2 is returning a status of 408 Do Not Disturb. That's what the problem was: the phone itself was in DnD mode and set to reject all incoming calls.

Chris Samuel: Submission to Joint Select Committee on Constitutional Recognition Relating to Aboriginal and Torres Strait Islander Peoples

Sun, 2018-06-10 23:01

Tonight I took some time to send a submission in to the Joint Select Committee on Constitutional Recognition Relating to Aboriginal and Torres Strait Islander Peoples in support of the Uluru Statement from the Heart from the 2017 First Nations National Constitutional Convention held at Uluru. Submissions close June 11th so I wanted to get this in as I feel very strongly about this issue.

Here’s what I wrote:

To the Joint Select Committee on Constitutional Recognition Relating to Aboriginal and Torres Strait Islander Peoples,

The first peoples of Australia have lived as part of this continent for many times longer than the ancestors of James Cook lived in the UK(*), let alone this brief period of European colonisation called Australia.

They have farmed, shaped and cared for this land over the millennia, they have seen the climate change, the shorelines move and species evolve.

Yet after all this deep time as custodians of this land they were dispossessed via the convenient lie of Terra Nullius and through killing, forced relocation and introduced sickness had their links to this land severely beaten, though not fatally broken.

Yet we still have the chance to try and make a bridge and a new relationship with these first peoples; they have offered us the opportunity for a Makarrata and I ask you to grasp this opportunity with both hands, for the sake of all Australians.

Several of the component states and territories of this recent nation of Australia are starting to investigate treaties with their first peoples, but this must also happen at the federal level as well.

Please take the Uluru Statement from the Heart to your own hearts, accept the offering of Makarrata & a commission and let us all move forward together.

Thank you for your attention.

Your sincerely,
Christopher Samuel

(*) Australia has been continuously occupied for at least 50,000 years, almost certainly for at least 60,000 years and likely longer. The UK has only been continuously occupied for around the last 10,000 years after the last Ice Age drove its previous population out into warmer parts of what is now Europe.

Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

This item originally posted here:

Submission to Joint Select Committee on Constitutional Recognition Relating to Aboriginal and Torres Strait Islander Peoples

Ben Martin: A new libferris is coming! 2.0.x

Sun, 2018-06-10 17:11
A while back I ported most of the libferris suite over to using boost for smart pointers and for signals. The later was not such a problem but there were always some fringe cases to the former and this lead to a delay in releasing it because there were some known issues.

I have moved that code into a branch locally and reverted back to using the Modern C++ Loki library for intrusive reference counting and sigc++. I imported my old test suite into the main libferris repo and will flesh that out over time.

I might do a 2.0.0 or 1.9.9 release soonish so that the entire stack is out there. As this has the main memory management stuff that has been working fine for the last 10 years this shouldn't be more unstable than it was before.

I was tempted to use travis ci for testing but will likely move to using a local vm. Virtualization has gotten much more convenient and I'm happy to setup a local test VM for this task which also breaks a dependency on companies which really doesn't need to be there. Yes, I will host releases and a copy of git in some place like github or gitlab or whatever to make that distribution more convenient. On the other hand, anyone could run the test suite which will be in the main libferris distro if they feel the desire.

So after this next release I will slowly at leisure work to flesh out the testsuite and fix issues that I find by running it over time. This gives a much more incremental development which will hopefully be more friendly to the limited time patches that I throw at the project.

One upside of being fully at the mercy of my time is that the project is less likely to die or be taken over by a company and lead in an unnatural direction. The downside is that it relies on my free time which is split over robotics, cnc, and other things as well as libferris.

As some have mentioned, a flatpak or docker image for libferris would be nice. Ironically this makes the whole thing a bit more like plan9 with a filesystem microkernel like subsystem (container) than just running it as a native though rpm or deb, but whatever makes it easier.

OpenSTEM: This Week in Australian History

Fri, 2018-06-08 17:05
Today we introduce a new category for OpenSTEM® Blog articles: “This Week in Australian History”. So many important dates in Australian history seem to become forgotten over time that there seems to be a need to highlight some of these from time to time. For teachers of students from Foundation/Prep/Kindy to Year 6 looking for […]

Matthew Oliver: Keystone Federated Swift – Separate Clusters + Container Sync

Fri, 2018-06-08 15:04

This is the third post in the series of Keystone Federated Swift. To bounce back to the start you can visit the first post.

Separate Clusters + Container Sync

The idea with this topology is to deploy each of your OpenStack federated clusters each with their own unique swift cluster and then use another swift feature, container sync, to push objects you create on one federated environment to another.

In this case the keystone servers are federated. A very similar topology could be a global Swift cluster, but each proxy only talks to single region’s keystone. Which would mean a user visiting a different region would authenticate via federation and be able to use the swift cluster, however would use a different account name. In both cases container sync could be used to synchronise the objects, say from the federated account to that of the original account. This is because container sync can synchronise both between containers in separate clusters or in the same.

 

Setting up container sync

Setting up container sync is pretty straight forward. And is also well documented. At a high level to goes like this. Firstly you need to setup a trust between the different clusters. This is achieved by creating a container-sync-realms.conf file, the online example is:

[realm1]
key = realm1key
key2 = realm1key2
cluster_clustername1 = https://host1/v1/
cluster_clustername2 = https://host2/v1/

[realm2]
key = realm2key
key2 = realm2key2
cluster_clustername3 = https://host3/v1/
cluster_clustername4 = https://host4/v1/

 

Each realm is a set of different trusts. And you can have as many clusters in a realm as you want, so as youcan see you can build up different realms. In our example we’d only need 1 realm, and lets use some better names.

[MyRealm]
key = someawesomekey
key2 = anotherkey
cluster_blue = https://blueproxyvip/v1
cluster_green = https://greenproxyvip/v1

NOTE: there is nothing stopping you from only having 1 cluster defined as you can use container sync within a cluster, or adding more clusters to a single realm.

 

Now in our example both the green and blue clusters need to have the MyRealm realm defined in their /etc/swift/container-sync-realms.conf file. The 2 keys are there so you can do key rotation. These keys should be kept secret as these keys will be used to define trust between the clusters.

 

The next step is to make sure you have the container_sync middleware in your proxy pipeline. There are 2 parts to container sync, the backend daemon that periodically checks containers for new objects and sends changes to the other cluster, and the middleware that is used to authenticate requests sent by container sync daemons from other clusters. We tend to place the container_sync middleware before (to the left of) any authentication middleware.

 

The last step is to tell container sync what containers to keep in sync. This is all done via container meta-data which is controlled by the user. Let’s assume we have 2 accounts, AUTH_matt on the blue and AUTH_federatedmatt on the green. And we wanted to sync a container called mycontainer. Note, the containers don’t have to be called the same. Then we’d start by making sure the 2 containers have the same container sync key, which is defined by the owner of the container, this isn’t the realm keys but work in a similar way. And then telling 1 container to sync with the other.
NOTE: you can make the relationship go both ways.

 

Let’s use curl first:

$ curl -i -X POST -H 'X-Auth-Token: <token>' \
-H 'X-Container-Sync-Key: secret' \
'http://blueproxyvip/v1/AUTH_matt/mycontainer'

$ curl -i -X POST -H 'X-Auth-Token: <token>' \
-H 'X-Container-Sync-Key: secret' \
-H 'X-Container-Sync-To: //MyRealm/blue/AUTH_matt/mycontainer' \
'http://greenproxyvip/v1/AUTH_federatedmatt/mycontainer'

Or via the swift client, noting that you need to change identities to set each account.

# To the blue cluster for AUTH_matt
$ swift  post -k 'secret' mycontainer

 

# To the green cluster for AUTH_federatedmatt
$ swift  post \
-t '//MyRealm/blue/AUTH_matt/mycontainer' \
-k 'secret' mycontainer

In a federated environment, you’d just need to set some key for each of your containers you want to work on while your away (or all of them I guess). Then when you visit you can just add the sync-to metadata when you create containers on the other side. Likewise, if you knew the name of your account on the other side you could make a sync-to if you needed to work on something over there.

 

To authenticate containersync generates and compares a hmac on both sides where the hmac consists of both the realm and container keys, the verb, object name etc.

 

The obvious next question is great, but then do I need to know the name of each cluster, well yes, but you can simply find them by asking swift via the info call. This is done by hitting the /info swift endpoint with whatever tool you want. If your using the swift client, then it’s:

$ swift info

Pros and cons Pros

The biggest pro for this approach is you don’t have to do anything special, if you have 1 swift cluster or a bunch throughout your federated environments the all you need to do it setup a container sync trust between them and the users can sync between themselves.

 

Cons

There are a few I can think off the top of my head:

  1. You need to manually set the metadata on each container. Which might be fine if it’s just you, but if you have an app or something it’s something else you need to think about.
  2. Container sync will move the data periodically, so you may not see it in the other container straight away.
  3. More storage is used. If it’s 1 cluster or many, the objects will exist in both accounts.
Conclusion

This is an interesting approach, but I think it would be much better to have access to the same set of objects everywhere I go and it just worked. I’ll talk about how to go about that in the next post as well as talk about 1 specific way I got working as a POC.

 

Container sync is pretty cool, Swiftstack have recently open sourced a another tool 1space, that can do something similar. 1space looks awesome but I haven’t have a chance to play with it yet. And so will add it to the list of Swift things I want to play with whenever I get a chance.

Gary Pendergast: Podcasting: Tavern Style

Thu, 2018-06-07 15:03

Earlier today, I joined JJJ and Jeff on episode 319 of the WP Tavern’s WordPress Weekly podcast!

We chatted about GitHub being acquired by Microsoft (and what that might mean for the future of WordPress using Trac), the state of Gutenberg, WordCamp Europe, as well as getting into a bit of the philosophy that drives WordPress’ auto-update system.

Finally, Jeff was kind enough to name me a Friend of the Show, despite my previous appearance technically not being a WordPress Weekly episode.

Russell Coker: BTRFS and SE Linux

Wed, 2018-06-06 23:02

I’ve had problems with systems running SE Linux on BTRFS losing the XATTRs used for storing the SE Linux file labels after a power outage.

Here is the link to the patch that fixes this [1]. Thanks to Hans van Kranenburg and Holger Hoffstätte for the information about this patch which was already included in kernel 4.16.11. That was uploaded to Debian on the 27th of May and got into testing about the time that my message about this issue got to the SE Linux list (which was a couple of days before I sent it to the BTRFS developers).

The kernel from Debian/Stable still has the issue. So using a testing kernel might be a good option to deal with this problem at the moment.

Below is the information on reproducing this problem. It may be useful for people who want to reproduce similar problems. Also all sysadmins should know about “reboot -nffd”, if something really goes wrong with your kernel you may need to do that immediately to prevent corrupted data being written to your disks.

The command “reboot -nffd” (kernel reboot without flushing kernel buffers or writing status) when run on a BTRFS system with SE Linux will often result in /var/log/audit/audit.log being unlabeled. It also results in some systemd-journald files like /var/log/journal/c195779d29154ed8bcb4e8444c4a1728/system.journal being unlabeled but that is rarer. I think that the same
problem afflicts both systemd-journald and auditd but it’s a race condition that on my systems (both production and test) is more likely to affect auditd.

root@stretch:/# xattr -l /var/log/audit/audit.log security.selinux: 0000 73 79 73 74 65 6D 5F 75 3A 6F 62 6A 65 63 74 5F system_u:object_ 0010 72 3A 61 75 64 69 74 64 5F 6C 6F 67 5F 74 3A 73 r:auditd_log_t:s 0020 30 00 0.

SE Linux uses the xattr “security.selinux”, you can see what it’s doing with xattr(1) but generally using “ls -Z” is easiest.

If this issue just affected “reboot -nffd” then a solution might be to just not run that command. However this affects systems after a power outage.

I have reproduced this bug with kernel 4.9.0-6-amd64 (the latest security update for Debian/Stretch which is the latest supported release of Debian). I have also reproduced it in an identical manner with kernel 4.16.0-1-amd64 (the latest from Debian/Unstable). For testing I reproduced this with a 4G filesystem in a VM, but in production it has happened on BTRFS RAID-1 arrays, both SSD and HDD.

#!/bin/bash set -e COUNT=$(ps aux|grep [s]bin/auditd|wc -l) date if [ "$COUNT" = "1" ]; then echo "all good" else echo "failed" exit 1 fi

Firstly the above is the script /usr/local/sbin/testit, I test for auditd running because it aborts if the context on it’s log file is wrong. When SE Linux is in enforcing mode an incorrect/missing label on the audit.log file causes auditd to abort.

root@stretch:~# ls -liZ /var/log/audit/audit.log 37952 -rw-------. 1 root root system_u:object_r:auditd_log_t:s0 4385230 Jun 1 12:23 /var/log/audit/audit.log

Above is before I do the tests.

while ssh stretch /usr/local/sbin/testit ; do ssh stretch "reboot -nffd" > /dev/null 2>&1 & sleep 20 done

Above is the shell code I run to do the tests. Note that the VM in question runs on SSD storage which is why it can consistently boot in less than 20 seconds.

Fri 1 Jun 12:26:13 UTC 2018 all good Fri 1 Jun 12:26:33 UTC 2018 failed

Above is the output from the shell code in question. After the first reboot it fails. The probability of failure on my test system is greater than 50%.

root@stretch:~# ls -liZ /var/log/audit/audit.log 37952 -rw-------. 1 root root system_u:object_r:unlabeled_t:s0 4396803 Jun 1 12:26 /var/log/audit/audit.log

Now the result. Note that the Inode has not changed. I could understand a newly created file missing an xattr, but this is an existing file which shouldn’t have had it’s xattr changed. But somehow it gets corrupted.

The first possibility I considered was that SE Linux code might be at fault. I asked on the SE Linux mailing list (I haven’t been involved in SE Linux kernel code for about 15 years) and was informed that this isn’t likely at
all. There have been no problems like this reported with other filesystems.

Related posts:

  1. SE Linux in Debian I have now got a Debian Xen domU running the...
  2. SE Linux Status in Debian 2012-01 Since my last SE Linux in Debian status report [1]...
  3. More BTRFS Fun I wrote a BTRFS status report yesterday commenting on the...

Michael Still: Mirroring all your repos from github

Wed, 2018-06-06 09:00

So let me be clear here, I don’t think its a bad thing that Microsoft bought github. No one is forcing you to use their services, in fact they make it trivial to stop using them. So what’s the big deal.

I’ve posted about a few git mirror scripts I run at home recently: one to mirror gerrit repos; and one to mirror arbitrary github users.

It was therefore trivial to whip up a slightly nicer script intended to help you forklift your repos out of github if you’re truly concerned. Its posted on github now (irony intended).

Now you can just do something like:

$ pip install -U -r requirements.txt $ python download.py --github_token=foo --username=mikalstill

I intend to add support for auto-creating and importing gitlab repos into the script, but haven’t gotten around to that yet. Pull requests welcome.

The post Mirroring all your repos from github appeared first on Made by Mikal.

Simon Lyall: Audiobooks – May 2018

Tue, 2018-06-05 11:03

Ramble On by Sinclair McKay

The history of walking in Britain and some of the author’s experiences. A pleasant listen. 7/10

Inherit the Stars by James P. Hogan

Very hard-core Sci Fi (all tech, no character) about a 50,000 year old astronaut’s body being found on the moon. Dated in places (everybody smokes) but I liked it. 7/10

Sapiens: A Brief History of Humankind by Yuval Noah Harari

A good overview of pre-history of human species plus an overview of central features of cultures (government, religion, money, etc). Interesting throughout. 9/10

The Adventures of Sherlock Holmes II by Sir Arthur Conan Doyle, read by David Timson

Another four Holmes stories. I’m pretty happy with Timson’s version. Each is only about an hour long. 7/10

The Happy Traveler: Unpacking the Secrets of Better Vacations by Jaime Kurtz

Written by a “happiness researcher” rather than a travel expert. A bit different from what I expected. Lots about structuring your trips to maximize your memories. 7/10

Mrs. Kennedy and Me: An Intimate Memoir by Clint Hill with Lisa McCubbin

I’ve read several of Hill’s books of his time in the US Secret Service, this overlaps a lot of these but with some extra Jackie-orientated material. I’d recommend reading the others first. 7/10

The Lost Continent: Travels in Small Town America by Bill Bryson

The author drives through small-town American making funny observations. Just 3 hours long so good bang for buck. Almost 30 years old so feels a little dated. 7/10

A Splendid Exchange: How Trade Shaped the World by William J. Bernstein

A pretty good overview of the growth of trade. Concentrates on the evolution of  routes between Asia and Europe. Only brief coverage post-1945. 7/10

The Adventures of Sherlock Holmes III by Sir Arthur Conan Doyle

The Adventure of the Cardboard Box; The Musgrave Ritual; The Man with the Twisted Lip; The Adventure of the Blue Carbuncle. All well done. 7/10

The Gentle Giants of Ganymede (Giants Series, Book 2) by James P. Hogan

Almost as hard-core as the previous book but with less of a central mystery. Worth reading if you like the 1st in the series. 7/10

An Army at Dawn: The War in North Africa, 1942-1943 – The Liberation Trilogy, Book 1 by Rick Atkinson

I didn’t like this as much as I expected or as much as similar books. Can’t quite place the problem though. Perhaps works better when written. 7/10

The Adventures of Sherlock Holmes IV by Sir Arthur Conan Doyle

A Case of Identity; The Crooked Man; The Naval Treaty; The Greek Interpreter. I’m happy with Timson’s version . 7/10

Michael Still: Quick note: pre-pulling docker images for ONAP OOM installs

Tue, 2018-06-05 11:00

Writing this down here because it took me a while to figure out for myself…

ONAP OOM deploys ONAP using Kubernetes, which effectively means Docker images at the moment. It needs to fetch a lot of Docker images, so there is a convenient script provided to pre-pull those images to make install faster and more reliable.

The script in the OOM codebase isn’t very flexible, so Jira issue OOM-655 was filed for a better script. The script was covered in code review 30169. Disappointingly, the code reviewer there doesn’t seem to have actually read the jira issue or the code before abandoning the patch — which isn’t very impressive.

So how do you get the nicer pre-pull script?

Its actually not too hard once you know the review ID. Just do this inside your OOM git clone:

$ git review -d 30169

You might be prompted for your gerrit details because the ONAP gerrit requires login. Once git review has run, you’ll be left sitting in a branch from when the review was uploaded that includes the script:

$ git branch master * review/james_forsyth/30169

Now just rebase that to bring it in mine with master and get on with your life:

$ git rebase -i origin Successfully rebased and updated refs/heads/review/james_forsyth/30169.

You’re welcome. I’d like to see the ONAP community take code reviews a bit more seriously, but ONAP seems super corporate (even compared to OpenStack), so I’m not surprised that they haven’t done a very good job here.

The post Quick note: pre-pulling docker images for ONAP OOM installs appeared first on Made by Mikal.