Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 1 hour 52 min ago

Dave Hall: Wix aren't the Only Ones Violating the GPL

Wed, 2018-07-25 03:04

Recently Matt Mullenweg called out Wix for violating the GPL. Wix's CEO, Avishai Abrahami's post on the company blog missed some of the key issues when it comes to using code licensed under the terms of the GNU General Public License. The GPL doesn't require you to only release the changes made to a dependency. If you use a GPL library in your project, then the entire project is deemed to be a derivate work. This is why the GPL is called a viral license, because if one part of your project is covered by the GPL, then it applies to the entirety of your app. Other people more qualified than me have explained this in more detail, more elegantly than I ever could.

This reminded me of a popular platform that is built on a GPL violation. FlightRadar24 allows users to track flights in real time. FR24's data is collected from multiple sources, including the US FAA and a network of volunteers that run tracking equipment. One of the most common tracking setups uses a Raspberry Pi hooked up to a cheap DVB-T USB tuner. librtlsdr is used to tune the USB stick to the right frequency so it can listen to the tracking signals.

The FlightRadar24 software, FR24Feed, is a single binary that includes a statically linked copy of librtlsdr. On multiple occasions I have asked for a full copy of the source code for the FR24Feed binaries that FlightRadar24 are distributing. They have refused.

FlightRadar are under the mistaken belief that they only have to distribute the source for their modifications to librtlsdr. They have made their patched version of the code available on the FlightRadar24 downloads page. FlightRadar24 say they won't release the source to ensure the quality of their data. I'm sorry, but the GPL doesn't include a clause that excuses you from distributing the source code if you have a bad architecture.

It is disappointing to see FlightRadar24 refusing to comply with the terms of the GPL. I still have everything I need to run a tracking station sitting in a drawer. I'll set up my tracking station once FlightRadar24 release all the full source for FR24Feed.

Dave Hall: Many People Want To Talk

Wed, 2018-07-25 03:04

WOW! The response to my blog post on the future of Drupal earlier this week has been phenomenal. My blog saw more traffic in 24 hours than it normally sees in a 2 to 3 week period. Around 30 comments have been left by readers. My tweet announcing the post was the top Drupal tweet for a day. Some 50 hours later it is still number 4.

It seems to really connected with many people in the community. I am still reflecting on everyone's contributions. There is a lot to take in. Rather than rush a follow up that responds to the issues raised, I will take some time to gather my thoughts.

One thing that is clear is that many people want to use DrupalCon Baltimore next week to discuss this issue. I encourage people to turn up with an open mind and engage in the conversation there.

A few people have suggested a BoF. Unfortunately all of the official BoF slots are full. Rather than that be a blocker, I've decided to run an unofficial BoF on the first day. I hope this helps facilitate the conversation.

Unofficial BoF: The Future of Drupal

When: Tuesday 25 April 2017 @ 12:30-1:30pm
Where: Exhibit Hall - meet at the Digital Echidna booth (#402) to be directed to the group
What: High level discussion about the direction people think Drupal should take.
UPDATE: An earlier version of this post had this scheduled for Monday. It is definitely happening on Tuesday.

I hope to see you in Baltimore.

Dave Hall: Continuing the Conversation at DrupalCon and Into the Future

Wed, 2018-07-25 03:04

My blog post from last week was very well received and sparked a conversation in the Drupal community about the future of Drupal. That conversation has continued this week at DrupalCon Baltimore.

Yesterday during the opening keynote, Dries touched on some of the issues raised in my blog post. Later in the day we held an unofficial BoF. The turn out was smaller than I expected, but we had a great discussion.

Drupal moving from a hobbyist and business tool to being an enterprise CMS for creating "ambitious digital experiences" was raised in the Driesnote and in other conversations including the BoF. We need to acknowledge that this has happened and consider it an achievement. Some people have been left behind as Drupal has grown up. There is probably more we can do to help these people. Do we need more resources to help them skill up? Should we direct them towards WordPress, backdrop, squarespace, wix etc? Is it is possible to build smaller sites that eventually grow into larger sites?

In my original blog post I talked about "peak Drupal" and used metrics that supported this assertion. One metric missing from that post is dollars spent on Drupal. It is clear that the picture is very different when measuring success using budgets. There is a general sense that a lot of money is being spent on high end Drupal sites. This has resulted in less sites doing more with Drupal 8.

As often happens when trying to solve problems with Drupal during the BoF descended into talking technical solutions. Technical solutions and implementation detail have a place. I think it is important for the community to move beyond this and start talking about Drupal as a product.

In my mind Drupal core should be a content management framework and content hub service for building compelling digital experiences. For the record, I am not arguing Drupal should become API only. Larger users will take this and build their digital stack on top of this platform. This same platform should support an ecosystem of Drupal "distros". These product focused projects target specific use cases. Great examples of such distros include Lightning, Thunder, Open Social, aGov and Drupal Commerce. For smaller agencies and sites a distro can provide a great starting point for building new Drupal 8 sites.

The biggest challenge I see is continuing this conversation as a community. The majority of the community toolkit is focused on facilitating technical discussions and implementations. These tools will be valuable as we move from talking to doing, but right now we need tools and processes for engaging in silver discussions so we can build platinum level products.

Dave Hall: Migrating AWS System Manager Parameter Store Secrets to a new Namespace

Wed, 2018-07-25 03:04

When starting with a new tool it is common to jump in start doing things. Over time you learn how to do things better. Amazon's AWS System Manager (SSM) Parameter Store was like that for me. I started off polluting the global namespace with all my secrets. Over time I learned to use paths to create namespaces. This helps a lot when it comes to managing access.

Recently I've been using Parameter Store a lot. During this time I have been reminded that naming things is hard. This lead to me needing to change some paths in SSM Parameter Store. Unfortunately AWS doesn't allow you to rename param store keys, you have to create new ones.

There was no way I was going to manually copy and paste all those secrets. Python (3.6) to the rescue! I wrote a script to copy the values to the new namespace. While I was at it I migrated them to use a new KMS key for encryption.

Grab the code from my gist, make it executable, pip install boto3 if you need to, then run it like so:

copy-ssm-ps-path.py source-tree-name target-tree-name new-kms-uuid

The script assumes all parameters are encrypted. The same key is used for all parameters. boto3 expects AWS credentials need to be in ~/.aws or environment variables.

Once everything is verified, you can use a modified version of the script that calls ssm.delete_parameter() or do it via the console.

I hope this saves someone some time.

Dave Hall: Remote Presentations

Wed, 2018-07-25 03:04

Living in the middle of nowhere and working most of my hours in the evenings I have few opportunities to attend events in person, let alone deliver presentations. As someone who likes to share knowledge and present at events this is a problem. My work around has been presenting remotely. Many of my talks are available on playlist on my youtube channel.

I've been doing remote presentations for many years. During this time I have learned a lot about what it takes to make a remote presentation sucessful.

Preparation

When scheduling a remote session you should make sure there is enough time for a test before your scheduled slot. Personally I prefer presenting after lunch as it allows an hour or so for dealing with any gremlins. The test presentation should use the same machines and connections you'll be using for your presentation.

I prefer using Hangouts On Air for my presentations. This allows me to stream my session to the world and have it recorded for future reference. I review every one of my recorded talks to see what I can do better next time.

Both sides of the connection should use wired connections. WiFi, especially at conferences can be flakely. Organisers should ensure that all presentation machines are using Ethernet, and if possible it should be on a separate VLAN.

Tips for Presenters

Presenting to a remote audience is very different to presenting in front of a live audience. When presenting in person you're able to focus on people in the audience who seem to be really engaged with your presentation or scan the crowd to see if you're putting people to sleep. Even if there is a webcam on the audience it is likely to be grainy and in a fixed position. It is also difficult to pace when presenting remotely.

When presenting in person your slides will be diplayed in full screen mode, often with a presenter view in your application of choice. Most tools don't allow you to run your slides in full screen mode. This makes it more difficult as a presenter. Transitions won't work, videos won't autoplay and any links Keynote (and PowerPoint) open will open in a new window that isn't being shared which makes demos trickier. If you don't hide the slide thumbnails to remind you of what is coming next, the audience will see them too. Recently I worked out printing thumbnails avoids revealing the punchlines prematurely.

Find out as much information as possible about the room your presentation will be held in. How big is it? What is the seating configuration? Where is the screen relative to where the podium is?

Tips for Organisers

Event organisers are usually flat out on the day of the event. Having to deal with a remote presenter adds to the workload. Some preparation can make life easier for the organisers. Well before the event day make sure someone is nominated to be the point of contact for the presenter. If possible share the details (name, email and mobile number) for the primary contact and a fallback. This avoids the presenter chasing random people from the organising team.

On the day of the event communicate delays/schedule changes to the presenter. This allows them to be ready to go at the right time.

It is always nice for the speaker to receive a swag bag and name tag in the mail. If you can afford to send this, your speaker will always appreciate it.

Need a Speaker?

Are you looking for a speaker to talk about Drupal, automation, devops, workflows or open source? I'd be happy to consider speaking at your event. If your event doesn't have a travel budget to fly me in, then I can present remotely. To discuss this futher please get in touch using my contact form.

Dave Hall: Drupal, We Need To Talk

Wed, 2018-07-25 03:04

Update 21 April: I've published a followup post with details of the BoF to be held at DrupalCon Baltimore on Tuesday 25 April. I hope to see you there so we can continue the conversation.

Drupal has a problem. No, not that problem.

We live in a post peak Drupal world. Drupal peaked some time during the Drupal 8 development cycle. I’ve had conversations with quite a few people who feel that we’ve lost momentum. DrupalCon attendances peaked in 2014, Google search impressions haven’t returned to their 2009 level, core downloads have trended down since 2015. We need to accept this and talk about what it means for the future of Drupal.

Technically Drupal 8 is impressive. Unfortunately the uptake has been very slow. A factor in this slow uptake is that from a developer's perspective, Drupal 8 is a new application. The upgrade path from Drupal 7 to 8 is another factor.

In the five years Drupal 8 was being developed there was a fundamental shift in software architecture. During this time we witnessed the rise of microservices. Drupal is a monolithic application that tries to do everything. Don't worry this isn't trying to rekindle the smallcore debate from last decade.

Today it is more common to see an application that is built using a handful of Laravel micro services, a couple of golang services and one built with nodejs. These applications often have multiple frontends; web (react, vuejs etc), mobile apps and an API. This is more effort to build out, but it likely to be less effort maintaining it long term.

I have heard so many excuses for why Drupal 8 adoption is so slow. After a year I think it is safe to say the community is in denial. Drupal 8 won't be as popular as D7.

Why isn't this being talked about publicly? Is it because there is a commercial interest in perpetuating the myth? Are the businesses built on offering Drupal services worried about scaring away customers? Adobe, Sitecore and others would point to such blog posts to attack Drupal. Sure, admitting we have a problem could cause some short term pain. But if we don't have the conversation we will go the way of Joomla; an irrelevant product that continues its slow decline.

Drupal needs to decide what is its future. The community is full of smart people, we should be talking about the future. This needs to be a public conversation, not something that is discussed in small groups in dark corners.

I don't think we will ever see Drupal become a collection of microservices, but I do think we need to become more modular. It is time for Drupal to pivot. I think we need to cut features and decouple the components. I think it is time for us to get back to our roots, but modernise at the same time.

Drupal has always been a content management system. It does not need to be a content delivery system. This goes beyond "Decoupled (Headless) Drupal". Drupal should become a "content hub" with pluggable workflows for creating and managing that content.

We should adopt the unix approach, do one thing and do it well. This approach would allow Drupal to be "just another service" that compliments the application.

What do you think is needed to arrest the decline of Drupal? What should Drupal 9 look like? Let's have the conversation.

Dave Hall: Per Environment Config in Drupal 8

Wed, 2018-07-25 03:04

One of the biggest improvements in Drupal 8 is the new configuration management system. Config is now decoupled from code and the database. Unlike Drupal 6 and 7, developers no longer have to rely on the features module for moving configuration around.

Most large Drupal sites, and some smaller ones, require per environment configuration. Prior to Drupal 8 this was usually achieved using a combination of hard coding config variables and features. Drupal 8 still allows users to put config variables in the settings.php file, but putting config in code feels like a backward step given D8 emphasis on separating concerns.

For example we may have a custom module which calls a RESTful API of a backend service. There are dev, stage and production endpoints that we need to configure. We also keep our config out of docroot and use drush to import the config at deployment time. We have the following structure in our git repo:

/ +- .git/ | +- .gitignore | +- README.md | +- config/ | | | +- README.md | | | +- base/ | | | +- dev/ | | | +- prod/ | | | +- stage/ | +- docroot/ | +- scripts/ | +- and-so-on/

When a developer needs to export the config for the site they run drush config-export --destination=/path/to/project/config/base. This exports all of the configuration to the specified path. To override the API endpoint for the dev environment, the developer would make the config change and then export just that piece of configuration. That can be done by runing drush config-get mymodule.endpoint > /path/to/project/config/dev/mymodule.endpoint.yml.

Drupal 8 and drush don't allow you to import the 2 config sets at the same time, so we need to run 2 drush commands to import our config. drush config-import --source=/path/to/project/config/base && drush config-import --partial --source=/path/to/project/config/dev. The first command imports the base config and the second applies any per environment overrides. The --partial flag prevents drush deleting any missing config. In most cases this is ok, but watch out if you delete a view or block placement.

Best practices are still emerging for managing configuration in Drupal 8. While I have this method working, I'm sure others have different approaches. Please leave a comment if you have an alternative method.

Update 11-Mar-2016:Removed --partial from base import. This prevented old configuration being removed during updates.

Russell Coker: Passwords Used by Daemons

Mon, 2018-07-23 19:03

There’s a lot of advice about how to create and manage user passwords, and some of it is even good. But there doesn’t seem to be much advice about passwords for daemons, scripts, and other system processes.

I’m writing this post with some rough ideas about the topic, please let me know if you have any better ideas. Also I’m considering passwords and keys in a fairly broad sense, a private key for a HTTPS certificate has more in common with a password to access another server than most other data that a server might use. This also applies to SSH host secret keys, keys that are in ssh authorized_keys files, and other services too.

Passwords in Memory

When SSL support for Apache was first released the standard practice was to have the SSL private key encrypted and require the sysadmin enter a password to start the daemon. This practice has mostly gone away, I would hope that would be due to people realising that it offers little value but it’s more likely that it’s just because it’s really annoying and doesn’t scale for cloud deployments.

If there was a benefit to having the password only in RAM (IE no readable file on disk) then there are options such as granting read access to the private key file only during startup. I have seen a web page recommending running “chmod 0” on the private key file after the daemon starts up.

I don’t believe that there is a real benefit to having a password only existing in RAM. Many exploits target the address space of the server process, Heartbleed is one well known bug that is still shipping in new products today which reads server memory for encryption keys. If you run a program that is vulnerable to Heartbleed then it’s SSL private key (and probably a lot of other application data) are vulnerable to attackers regardless of whether you needed to enter a password at daemon startup.

If you have an application or daemon that might need a password at any time then there’s usually no way of securely storing that password such that a compromise of that application or daemon can’t get the password. In theory you could have a proxy for the service in question which runs as a different user and manages the passwords.

Password Lifecycle

Ideally you would be able to replace passwords at any time. Any time a password is suspected to have been leaked then it should be replaced. That requires that you know where the password is used (both which applications and which configuration files used by those applications) and that you are able to change all programs that use it in a reasonable amount of time.

The first thing to do to achieve this is to have one password per application not one per use. For example if you have a database storing accounts used for a mail server then you would be tempted to have an outbound mail server such as Postfix and an IMAP server such as Dovecot both use the same password to access the database. The correct thing to do is to have one database account for the Dovecot and another for Postfix so if you need to change the password for one of them you don’t need to change passwords in two locations and restart two daemons at the same time. Another good option is to have Postfix talk to Dovecot for authenticating outbound mail, that means you only have a single configuration location for storing the password and also means that a security flaw in Postfix (or more likely a misconfiguration) couldn’t give access to the database server.

Passwords Used By Web Services

It’s very common to run web sites on Apache backed by database servers, so common that the acronym LAMP is widely used for Linux, Apache, Mysql, and PHP. In a typical LAMP installation you have multiple web sites running as the same user which by default can read each other’s configuration files. There are some solutions to this.

There is an Apache module mod_apparmor to use the Apparmor security system [1]. This allows changing to a specified Apparmor “hat” based on the URI or a specified hat for the virtual server. Each Apparmor hat is granted access to different files and therefore files that contain passwords for MySQL (or any other service) can be restricted on a per vhost basis. This only works with the prefork MPM.

There is also an Apache module mpm-itk which runs each vhost under a specified UID and GID [2]. This also allows protecting sites on the same server from each other. The ITK MPM is also based on the prefork MPM.

I’ve been thinking of writing a SE Linux MPM for Apache to do similar things. It would have to be based on prefork too. Maybe a change to mpm-itk to support SE Linux context as well as UID and GID.

Managing It All

Once the passwords are separated such that each service runs with minimum privileges you need to track and manage it all. At the simplest that needs a document listing where all of the passwords are used and how to change them. If you use a configuration management tool then that could manage the passwords. Here’s a list of tools to manage service passwords in tools like Ansible [3].

Related posts:

  1. Email Passwords I was doing some routine sysadmin work for a client...
  2. SE Linux Play Machine and Passwords My SE Linux Play Machine has been online again since...
  3. Case Sensitivity and Published Passwords When I first started running a SE Linux Play Machine...

Michael Still: The last week for linux.conf.au 2019 proposals!

Mon, 2018-07-23 09:00

Dear humans of the Internet — there is ONE WEEK LEFT to propose talks for linux.conf.au 2019. LCA is one of the world’s best open source conferences, and we’d love to hear you speak!   Unsure what to propose? Not sure if your talk is what the conference would normally take? Just want a chat? You’re welcome to reach out to papers-chair@linux.org.au to talk things through.   https://linux.conf.au/call-for-papers/

David Rowe: A Visit to the Home of FreeDV and Codec 2

Fri, 2018-07-20 13:04

Peter, VK2TPM, visited my home this week and has written nice blog post showing FreeDV development in action. Really enjoyed your visit Peter!

Matthew Oliver: Monasca + Swift: Sending all your Swift metrics Monasca’s way

Mon, 2018-07-16 19:04

Last week was SUSE Hackweek. A week every employee is given to go have fun hacking something or learning something they find interesting. It’s an awesome annual event that SUSE runs. It’s my second and I love it.

While being snowed in in Dublin at the Dublin PTG a while ago I chatted with Johannes, a monasca dev and very intelligent team mate at SUSE. And I heard that Monasca has a statsd endpoint as a part of the monasca agent you can fire stats at. As a Swift developer this interests me greatly. Every Swift daemon dumps a plethora of statsd metrics. So can I put the 2 together? Can I simply install monasca-agent to each storage and proxy node and then point the statsd endpoints for all swift services locally?

 

I started the week attempting to do just that. Because I’m new to monasca, and didn’t want to go attempt to set it up, I just run a devsack + SAIO environment.

The devstack was just a simple monasa + keystone + horizon configuration and the SAIO was a standard Swift All In One.

 

Next I installed the monasca-agent to the SAIO and then updated Swift to point at it. In Swift each config supports a statsd server endpoint configuration options:

 

# You can enable StatsD logging here:
# log_statsd_host =
# log_statsd_port = 8125
# log_statsd_default_sample_rate = 1.0
# log_statsd_sample_rate_factor = 1.0
# log_statsd_metric_prefix =

 

So pointing swift is easy. I then uploaded as few objects to swift and bingo, inside Monasca’s influxdb instance I can see the Swift measurements.

 

account-auditor.passes
account-auditor.timing
account-replicator.attempts
account-replicator.no_changes
account-replicator.successes
account-replicator.timing
account-server.GET.timing
account-server.HEAD.timing
account-server.PUT.timing
account-server.REPLICATE.timing
container-auditor.passes
container-auditor.timing
container-replicator.attempts
container-replicator.no_changes [41/49393]
container-replicator.successes
container-replicator.timing
container-server.GET.timing
container-server.PUT.timing
container-server.REPLICATE.timing
container-updater.no_changes
container-updater.successes
container-updater.timing
monasca.collection_time_sec
monasca.thread_count
object-auditor.timing
object-replicator.partition.update.count.sdb1
object-replicator.partition.update.count.sdb2
object-replicator.partition.update.count.sdb3
object-replicator.partition.update.count.sdb4
object-replicator.partition.update.timing
object-replicator.suffix.hashes
object-server.HEAD.timing
object-server.PUT.sdb1.timing
object-server.PUT.sdb2.timing
object-server.PUT.sdb3.timing
object-server.PUT.sdb4.timing
object-server.PUT.timing
object-server.REPLICATE.timing
object-updater.timing
proxy-server.account.GET.200.first-byte.timing
proxy-server.account.GET.200.timing
proxy-server.account.GET.200.xfer
proxy-server.object.HEAD.404.timing
proxy-server.object.HEAD.404.xfer
proxy-server.object.PUT.201.timing
proxy-server.object.PUT.201.xfer
proxy-server.object.policy.1.HEAD.404.timing
proxy-server.object.policy.1.HEAD.404.xfer
proxy-server.object.policy.1.PUT.201.timing
proxy-server.object.policy.1.PUT.201.xfer

 

NOTE: This isn’t the complete list, as the measures are added when new metrics are fired, and the SAIO is a small healthy swift cluster, so there isn’t many 500 series errors etc. But it works!

 

And better yet I have access to them in grafana via the monasca datasource!

 

swift_recon check plugin

I thought that was easy, but Swift actually provides more metrics then just that. Swift has a reconnaissance API (recon) on all the wsgi servers (account, container and object servers). That you can hit either via REST or the swift-recon tool. So next I thought I wonder how hard it would be to write a swift_recon check plugin for Monasca.

Some of the recon metrics you can get aren’t really grafana friendly. But some would be awesome to have in the same place and closer to horizon where ops are looking.

 

So I went and wrote one. Like I said I couldn’t get all the metrics, but I got most:

 

swift_recon.account.account_auditor_pass_completed [2/49393]
swift_recon.account.account_audits_failed
swift_recon.account.account_audits_passed
swift_recon.account.account_audits_since
swift_recon.account.attempted
swift_recon.account.failure
swift_recon.account.replication_last
swift_recon.account.replication_time
swift_recon.account.success
swift_recon.container.attempted
swift_recon.container.container_auditor_pass_completed
swift_recon.container.container_audits_failed
swift_recon.container.container_audits_passed
swift_recon.container.container_audits_since
swift_recon.container.container_updater_sweep
swift_recon.container.failure
swift_recon.container.replication_last
swift_recon.container.replication_time
swift_recon.container.success
swift_recon.disk_usage.mounted
swift_recon.object.async_pending
swift_recon.object.attempted
swift_recon.object.auditor.object_auditor_stats_ALL.audit_time
swift_recon.object.auditor.object_auditor_stats_ALL.bytes_processed
swift_recon.object.auditor.object_auditor_stats_ALL.errors
swift_recon.object.auditor.object_auditor_stats_ALL.passes
swift_recon.object.auditor.object_auditor_stats_ALL.quarantined
swift_recon.object.auditor.object_auditor_stats_ALL.start_time
swift_recon.object.auditor.object_auditor_stats_ZBF.audit_time
swift_recon.object.auditor.object_auditor_stats_ZBF.bytes_processed
swift_recon.object.auditor.object_auditor_stats_ZBF.errors
swift_recon.object.auditor.object_auditor_stats_ZBF.passes
swift_recon.object.auditor.object_auditor_stats_ZBF.quarantined
swift_recon.object.auditor.object_auditor_stats_ZBF.start_time
swift_recon.object.expirer.expired_last_pass
swift_recon.object.expirer.object_expiration_pass
swift_recon.object.failure
swift_recon.object.object_updater_sweep
swift_recon.object.replication_last
swift_recon.object.replication_time
swift_recon.object.success
swift_recon.quarantined
swift_recon.unmounted

 

Some of the metric names might need to tidy up, but so far, so good. Some of the really interesting metrics Swift Ops usually want to keep an eye on is when have all the replicators completed a cycle. Why? Well one example is while ring rebalancing on a large and busy cluster you want to avoid too much data movement, so when adding new drives you will rise their weights slowly. But you also want to make sure a complete replication cycle is complete before you rebalance again. So knowing when you pushed a new ring out and the timestamps of the last run replication tells you when it’s safe. These are coming through nicely:

 

 

Unfortunately there are some metrics I can’t quite get though. You can use recon to get md5s of the rings and configs on each node. But I found md5s can’t get pushed through. You can also ask recon what version of swift is installed on each node (nice is a large deployment and when upgrading). But the version number also had issues. Both of these are probably not insurmountable, but I still need to figure out how.

 

swift_handoffs check plugin

I’ve been involved in the Swift community for quite a while now, and I’d had heard of another awesome metric one of the Swiftstack cores came out with to give an awesome visualisation of the Swift cluster. He even provided a gist to the community others would use and adapt. I thought, why not make sure everyone could use it, lets add it as another check plugin to the monasca agent.

 

Everything in Swift is treated as an object, and an object has a number of devices in the cluster who are considered primary (who store that object). When a drive gets full or there is too much load on say an object PUT, if a primary is unavailable to meet the durability contract another node will store the object (this node would be called a handoff for that object), the handoff node will push the handoff object to the primary as soon as it can (drive is replaced, or comes back online, etc).

Further, a ring in Swift is divided into logical segments called partitions. And it’s these partitions that devices are responsible for storing (or think of it as, it has to store all objects that belong to a partition). When we rebalance the ring, either by adding or removing drives or changing weights, these partitions shift around the cluster. Either to say drain a drive or to move to where where is more space. Swift is really good as minimising this movement to the minimum. So after a rebalance, nodes that used to be primaries for some partitions wont be anymore. They’ll suddenly be handoffs, and the back-end consistency engine will move them to their new home.

So what’s interesting to note there is, it all involves handoff partitions.

 

Turns out, by just watching the number of partitions vs the number of handoffs on each storage node gives you a great health indicator. When should I do a rebalance? when the handoffs numbers are down. There seem to be a build up of handoffs in a region, maybe write affinity and WAN links are saturated or there is some network/disk/server issue on one of the nodes around there etc.

Here are the metrics:

 

swift_handoffs.handoffs
swift_handoffs.primary

 

And here is a simplified snapshot. This is my SAIO with 4 simulated nodes. This is watching the storage nodes as a whole but you can break down to the drive. There is a graph for each node and each Swift ring. This rise in handoffs (Object – Policy 0 SAIO[1-3]) is due to me turning of the consistency engine and then changing the weight back to a nicely weighted cluster:

See Object - Policy 0. SAIO0’s weight has increased, so the other nodes now have handoff partitions to give him. If I now went and turned the consistency engine back on, you’d see more primary nodes on SAIO0.

 

Wheres the code

UPDATE: I’ve now pushed up the checks to monasca. They can be found here:

  • https://review.openstack.org/#/c/583876/
  • https://review.openstack.org/#/c/585067/

Lev Lafayette: Transparency and Immersion in HPC

Thu, 2018-07-12 19:05

The development of the graphic user-interface is widely considered a major phenomenological contribution to the Human-Computer Interaction (HCI) by providing an intuitive framework for data storage and processing, encapsulated in the term "user friendly". Whilst for a very large number of everyday computational tasks this Windows-Icons-Menu-Pointer (WIMP) interface has been highly successful, the field of high performance computing (HPC) continues to use the command line interface.

The reasons for this illustrate an epistemological divide in the type of computation problems with an ontological foundation. In high performance computing, a deeper level of understanding of the complex processes is unavoidably requisite, where simplified and automated ignorance is inadequate for performance benefits that such architectures provide. Further, high performance computing is increasingly important in research as datasets for processing are growing faster than improvements in hardware or efficiency in software, and as a result, there is a correlation between research output with access and training to HPC systems.

Despite the necessity of command-line knowledge for HPC systems that does not mean that more transparent and immersive user experiences are not possible in this environment as continuing improvements to the shell environment illustrate and notably by free software and peer adoption. Shell environments such as fish (Friendly Interactive SHell) provide syntax highlighting, autosuggestions, and tab completions, with reduced configuration, all with the user experience in mind. Finally, further developments are illustrated for shell environments that have a stronger correlation to human languages with multilingual libraries.

Transparency and Immersion in HPC. Presentation to Human-Technology Relations Conference, University of Twente, The Netherlands, July 11-13, 2018

David Rowe: Horus Binary Telemetry and FreeDV

Tue, 2018-07-10 09:04

Since our last post on Testing HAB Telemetry Protocols, Mark and I have been working steadily on making the Horus Binary protocol easier to use for High Altitude Balloon (HAB) telemetry.

The objectives for this work are:

  1. Test our 4FSK Binary open source balloon telemetry over the air on a real HAB mission.
  2. Develop a GUI version of the decoder to make it usable by a wider audience. Now anyone with a SSB radio (or SDR) and Windows machine can crowd source telemetry using the new protocol.
  3. Move the art of open source, high performance telemetry forward. We now have a C API suitable for integrating telemetry protocols into other projects, and a cross platform GUI for end users. In our use case, we are outperforming Lora by a factor of 10.
  4. It’s a cool AREG club project. Now we’ve moved it off the Linux command line, club members who prefer Windows and SSB radios can get involved, along with people who like Linux and SDRs.
  5. Show how we develop open source protocols, by blogging on our work, sharing our design and test techniques and most importantly the source code.

RS41 Transmitter

Mark has been collecting RS41 Radiosondes. Conveniently, our local Bureau of Meterology launches two every day. They can be tracked, and many of them eventually fall from the sky into the hands of eager South Australian Radio Amateurs. Batteries are even included in the deal.

Mark has built a nice system to help track and recover them. Turns out they are rather hackable so he reprogrammed one to be a Horus Binary transmitter using this source code:

FreeDV GUI changes

FreeDV GUI is a cross platform (Linux/OSX/Windows) GUI program for digital voice. It’s designed to demodulate modem signals from sound cards, and I’m its maintainer so we decided to press it into service as a HAB telemetry decoder.

I spent a few hours each day for a week refactoring the code to handle 48 kHz sample rate we use for the Horus API. This code needed some maintenance so it was a useful step for my digital voice work as well. I post experimental Windows versions here. Or you can just build it from the source code.

Here is a weak (-7db SNR) Horus Binary mode test signal. You may be able to hear some tinkling sounds buried in the noise – that’s the 4FSK:

You can download this weak test signal to test FreeDV GUI using the Tool-Start/Stop Play File from Radio menu option.

Now FreeDV GUI is not for everyone. The good news is you can use the Horus API to roll your own GUI front end if that’s the itch you’d like to scratch. I have limited time, and my main focus is Digital Voice over HF radio. So I would really appreciate other people from the HAB community stepping up to help with further HAB features. If you can’t code, please consider supporting this work via Pateron or PayPal.

Bench Testing MDS

On Sat 23 June Mark and I performed some Minimum Detectable signal (MDS) tests on the telemetry stack. We used a RS41 as the Tx (13 dBm output power), many dBs of attenuators, and careful experimental design to attenuate the signal down to very low levels.

This post also describes how we do MDS tests, and includes some video.

For these tests we used my FT817 SSB radio and a RTL-SDR, both with a LNA. There was up to 160dB of fixed and switched attenuators. The Tx was placed in a metal box, and we added 100dB of attenuation directly at the output. We then routed the signal to the Rx in another room using good quality coax (so it wouldn’t leak RF). A switched attenuator was placed in front of the Rx so we could adjust the attenuation in 1dB steps.

It’s important to make sure the Rx signal is coming from the desired path. You can test this using a spectrum analyser or in our case the calibrated SNR meter in FreeDV GUI. Insert more attenuation and note how far the signal level drops. If you insert 10dB more attenuation and it drops just 1dB, well you have some RF feeding through another path!

Our calculated MDS (50% packet loss) was:

MDS = EbNo + 10log10(Rb) + NF - 174 = 4.5 + 10log10(200) + 1 - 174 = -145.45 dBm

Note the baud (symbol) rate is 100 symbols/s, but 4FSK sends 2 bits/symbol, so our bit rate is 200 bits/s. Our experiments on HAB protocols gave us the Eb/No figure. The Noise Figure (NF) of our preamp is an estimate from previous work and the pre-amp data sheet.

We obtained consistent results down to about -143dBm, but from there on it was difficult to get repeatable results. The signal is so weak, and the RF finds other paths. But hey – that lets us know we are close and we can conclude there are no obvious problems in the integrated system.

The comparable Lora mode 10 with a similar (1 second) packet duration has a MDS of -132dBm. Our mode requires at least 10dB less power, which gives it four times the range using the same transmit power (every 6dB doubles the line of sight range). Or we can transmit the same distance on 10% of the power of Lora. Our solution is based on good engineering, open source software, and commodity hardware rather than a chipset. The chipset form-factor has some advantages, and we hope to develop a similar form factor solution in the future. You can help if you like.

Test Flight

On Sunday 8 July the 4FSK Binary Horus system had a work out on a real Horus HAB flight. A lot of things can go wrong on a real flight but it worked well. Many people set up receive stations for the binary protocol, and crowd sourced the uploading of packets.

Callsign Packets Rx-ed % of Flight VK5AKH 1511 82.4% VK5APR 1541 84.0% VK5FJGM 1234 67.3% VK5FLJG 1577 86.0% VK5FTAZ 832 45.4% VK5IX 1491 81.3% VK5KJP 1471 80.2% VK5RR 288 15.7% VK5ST 1663 90.7% VK5TRM 1719 93.7% VK5QI-9 (Mobile Station) 1560 85.1% VK5WTF (Mobile Station) 1039 56.7%

Here is a plot of the telemetry collated by HabHub. Note on the far right hand side that the telemetry is still being received on the ground, as the recovery team tracked through the bush to collect the various payloads – fortunately Mark’s car was nearby so the signal was strong enough to be picked up even after landing.

Next Steps

The current system requires both FreeDV GUI and a Python script to to upload packets to HabHub. An obvious next step is to put some C++ code into FreeDV GUI to create a single application. Would anyone like to help us with this task?

I’d like to see more Open source telemetry, and encourage people to build their own protocols using open source components. We have demonstrated that we can outperform closed source systems like Lora. A great next step would be our own radio hardware, such as a simple FSK transmitter that can be easily integrated with a microcontroller.

Reading Further

FreeDV GUI for Windows with Horus Binary support.
Horus binary set up instructions with FreeDV.
AREG post on the flight.
RS41 source code to make it a Horus Binary telemetry transmitter.
Wenet – 115 kbit/s system for HD images that uses the same FSK modem.
All Your Modem are Belong To Us – the start of our open source modem development for HAB.
Binary Telemetry Protocol – first flight of a similar binary protocol in 2016, using 2FSK.
Testing HAB Telemetry Protocols – bench testing the 4FSK binary protocol against various RTTY demodulators.
High Speed Balloon Data Link – start of the Wenet work, and a nice example of MDS testing @ home.

Ian Wienand: uwsgi; oh my!

Mon, 2018-07-09 15:01

The world of Python based web applications, WSGI, its interaction with uwsgi and various deployment methods can quickly turn into a incredible array of confusingly named acronym soup. If you jump straight into the uwsgi documentation it is almost certain you will get lost before you start!

Below tries to lay out a primer for the foundations of application deployment within devstack; a tool for creating a self-contained OpenStack environment for testing and interactive development. However, it is hopefully of more general interest for those new to some of these concepts too.

WSGI

Let's start with WSGI. Fully described in PEP 333 -- Python Web Server Gateway Interface the core concept a standardised way for a Python program to be called in response to a web request. In essence, it bundles the parameters from the incoming request into known objects, and gives you can object to put data into that will get back to the requesting client. The "simplest application", taken from the PEP directly below, highlights this perfectly:

def simple_app(environ, start_response): """Simplest possible application object""" status = '200 OK' response_headers = [('Content-type', 'text/plain')] start_response(status, response_headers) return ['Hello world!\n']

You can start building frameworks on top of this, but yet maintain broad interoperability as you build your application. There is plenty more to it, but that's all you need to follow for now.

Using WSGI

Your WSGI based application needs to get a request from somewhere. We'll refer to the diagram below for discussions of how WSGI based applications can be deployed.

In general, this is illustrating how an API end-point http://service.com/api/ might be connected together to an underlying WSGI implementation written in Python (web_app.py). Of course, there are going to be layers and frameworks and libraries and heavens knows what else in any real deployment. We're just concentrating on Apache integration -- the client request hits Apache first and then gets handled as described below.

CGI

Starting with 1 in the diagram above, we see CGI or "Common Gateway Interface". This is the oldest and most generic method of a web server calling an external application in response to an incoming request. The details of the request are put into environment variables and whatever process is configured to respond to that URL is fork() -ed. In essence, whatever comes back from stdout is sent back to the client and then the process is killed. The next request comes in and it starts all over again.

This can certainly be done with WSGI; above we illustrate that you'd have a framework layer that would translate the environment variables into the python environ object and connect up the processes output to gather the response.

The advantage of CGI is that it is the lowest common denominator of "call this when a request comes in". It works with anything you can exec, from shell scripts to compiled binaries. However, forking processes is expensive, and parsing the environment variables involves a lot of fiddly string processing. These become issues as you scale.

Modules

Illustrated by 2 above, it is possible to embed a Python interpreter directly into the web server and call the application from there. This is broadly how mod_python, mod_wsgi and mod_uwsgi all work.

The overheads of marshaling arguments into strings via environment variables, then unmarshaling them back to Python objects can be removed in this model. The web server handles the tricky parts of communicating with the remote client, and the module "just" needs to translate the internal structures of the request and response into the Python WSGI representation. The web server can manage the response handlers directly leading to further opportunities for performance optimisations (more persistent state, etc.).

The problem with this model is that your web server becomes part of your application. This may sound a bit silly -- of course if the web server doesn't take client requests nothing works. However, there are several situations where (as usual in computer science) a layer of abstraction can be of benefit. Being part of the web server means you have to write to its APIs and, in general, its view of the world. For example, mod_uwsgi documentation says

"This is the original module. It is solid, but incredibly ugly and does not follow a lot of apache coding convention style".

uwsgi

mod_python is deprecated with mod_wsgi as the replacement. These are obviously tied very closely to internal Apache concepts.

In production environments, you need things like load-balancing, high-availability and caching that all need to integrate into this model. Thus you will have to additionally ensure these various layers all integrate directly with your web server.

Since your application is the web server, any time you make small changes you essentially need to manage the whole web server; often with a complete restart. Devstack is a great example of this; where you have 5-6 different WSGI-based services running to simulate your OpenStack environment (compute service, network service, image service, block storage, etc) but you are only working on one component which you wish to iterate quickly on. Stopping everything to update one component can be tricky in both production and development.

uwsgi

Which brings us to uwsgi (I call this "micro-wsgi" but I don't know if it actually intended to be a μ). uwsgi is a real Swiss Army knife, and can be used in contexts that don't have to do with Python or WSGI -- which I believe is why you can get quite confused if you just start looking at it in isolation.

uwsgi lets us combine some of the advantages of being part of the web server with the advantages of abstraction. uwsgi is a complete pluggable network daemon framework, but we'll just discuss it in one context illustrated by 3.

In this model, the WSGI application runs separately to the webserver within the embedded python interpreter provided by the uwsgi daemon. uwsgi is, in parts, a web-server -- as illustrated it can talk HTTP directly if you want it to, which can be exposed directly or via a traditional proxy.

By using the proxy extension mod_proxy_uwsgi we can have the advantage of being "inside" Apache and forwarding the requests via a lightweight binary channel to the application back end. In this model, uwsgi provides a uwsgi:// service using its internal protcol on a private port. The proxy module marshals the request into small packets and forwards it to the given port. uswgi takes the incoming request, quickly unmarshals it and feeds it into the WSGI application running inside. Data is sent back via similarly fast channels as the response (note you can equally use file based Unix sockets for local only communication).

Now your application has a level of abstraction to your front end. At one extreme, you could swap out Apache for some other web server completely and feed in requests just the same. Or you can have Apache start to load-balance out requests to different backend handlers transparently.

The model works very well for multiple applications living in the same name-space. For example, in the Devstack context, it's easy with mod_proxy to have Apache doing URL matching and separate out each incoming request to its appropriate back end service; e.g.

  • http://service/identity gets routed to Keystone running at localhost:40000
  • http://service/compute gets sent to Nova at localhost:40001
  • http://service/image gets sent to glance at localhost:40002

and so on (you can see how this is exactly configured in lib/apache:write_uwsgi_config).

When a developer makes a change they simply need to restart one particular uwsgi instance with their change and the unified front-end remains untouched. In Devstack (as illustrated) the uwsgi processes are further wrapped into systemd services which facilitates easy life-cycle and log management. Of course you can imagine you start getting containers involved, then container orchestrators, then clouds-on-clouds ...

Conclusion

There's no right or wrong way to deploy complex web applications. But using an Apache front end, proxying requests via fast channels to isolated uwsgi processes running individual WSGI-based applications can provide both good performance and implementation flexibility.

OpenSTEM: Rising sea levels and our children’s future

Fri, 2018-07-06 17:05
Global warming and sea level rise sometimes seems like a lot of hype bandied about by the media and politicians. However, it has a different level of importance for scientists in a range of disciplines around the world. The Australian Curriculum emphasises that students should study sustainability and environment issues in HASS and Science subjects, […]

David Rowe: Codec 2 at 450 bit/s

Fri, 2018-07-06 07:04

Thomas Kurin is a Masters student at the Institute of Electronics Engineering, part of the Friedrich-Alexander University Erlangen-Nuremberg. Thomas and his supervisor Stefan Erhardt recently contacted me with some exciting news – a version of Codec 2 running at 450 bit/s, including a 16 kHz mode!

I’m very happy that Codec 2 can be used as a starting point for academic research, and thrilled about the progress Thomas has made. It’s also great for me to have a contribution to Codec 2 on such a deep technical level. Thomas has done a lot of work in vector quantisation, an area that I struggle in, and has developed an innovative 16 kHz mode.

Any speech codec running as low at 450 bit/s is pretty special, and this work is also a great starting point for further innovation and quality improvements.

Thomas has sent me some samples, and is working with me to merge his new mode into codec2-dev, and has kindly offered to tell us his story below. Well done Thomas

Here are Thomas (left) and Stefan (right) working together on their innovative 450 bit/s Codec 2 mode at the University of Erlangen:

Samples

Now this is bleeding edge, very low bit rate speech coding. Don’t expect Hi-Fi. The use case is channels where no other speech signal can get through. Ham radio operators will know what I mean.

If integrated with FreeDV, a 450 bit/s codec translates to a 2dB improvement in SNR over FreeDV 700D. This would support digital speech over HF radio at lower than -4dB SNR. Weak signal or FreeDV EME anyone?

Listen to the samples, and please add your comments. How does it compare to Codec 2 at 1300 and 700 bit/s? Could you communicate using this mode?

Sample 1300 700C 450 8kHz 450 16kHz hts1a Listen Listen Listen Listen hts2a Listen Listen Listen Listen forig Listen Listen Listen Listen ve9qrp_10s Listen Listen Listen Listen mmt1 Listen Listen Listen Listen vk5qi Listen Listen Listen Listen cq_ref Listen Listen Listen Listen

The samples I use are deliberately chosen to give codecs a hard time. mmt1 is the worst, it has high level truck background noise added to it.

Thomas’ Story

As a masters student at FAU Erlangen-Nürnberg I have been working on and with Codec 2 and especially with 700C for the last few months. I experimented with the source code and the Vector Quantisation. This lead to the 450 bit/s Codec which shall be shown in the following post.

The 450 Codec builds on the 700C Codec. It consists of 3 major changes:

  1. The training data used for Vector Quantisation (VQ) was changed to include multiple languages, as for other languages the VQ sometimes performed poorly because certain vectors were missing. The used dataset consisted of approximately 30 minutes English, 30 minutes German, 15 minutes Russian, 15 minutes Chinese and 15 minutes Spanish. This Codebook allowed a switch from the two stage VQ with 9 * 2 = 18 bit per frame to a one stage VQ with just 9 bit per frame for VQ. The audio quality is of course somewhat changed, but still understandable.
  2. The energy quantisation was changed from 4 bit to 3 bit as no change in quality for the one stage VQ was detectable. This leads to a reduction of 9 + 1 = 10 bit, from a 28 bit frame to a 18 bit frame. This means a 450 bit/s Codec.
  3. The biggest change was the inclusion of a 16k mode. This means the 8kHz sampled and encoded signal can be converted to a 16kHz sampled signal at the decoder. To achieve this, the codebook was trained with data sampled at 16kHz. Then only the vectors for the 0-4 kHz frequencies (= 8kHz sample rate) are used for encoding. The decoder uses the indices to look up the 16kHz sampled and trained vectors. This works because a vector that is similar between 0-4 kHz mostly also looks similar in 4-8 kHz. This results in a pseudo-wideband format without any additional bits. For some speakers this improves quality, but for some speakers it just creates noise. But that’s the beauty of the system, as the same data can both be decoded to a 8kHz sampled signal, and to a 16kHz sampled signal. Therefore the transmitted signal stays the same and the receiver can choose dependent on quality which mode he wants to listen on.The 16k mode still needs some refinement but its fascinating that it works at all.

Over the next week or so I will patch the changes with David against codec2-dev. As it is a new codec, it shouldn’t change any of the other codecs when patching, as most of the patching will be to include new files.

Reading Further

Codec 2
Codec 2 700C

Ben Martin: LinuxCNC and latency

Tue, 2018-07-03 12:34
When you enter the 4th and 5th axis world for cnc things start getting interesting. I have started looking at how LinuxCNC works with a plan to use it as the primary gcode runner on a higher axis machine.

Of the many options that LinuxCNC offers, I might start out using a parallel port and small breakout board to software signal some stepper controllers. These controllers have an enable, step, and direction pin and can be wired up in a few ways. This involves if the signal goes to the A+ pin or if you run high voltage to those A+ pins and run the signal to the A- pin. See for example about half way down this smoothieboard page.

Anyway, back to the LinuxCNC topic. The main thing that controls how well you can control stepper motors over the parallel port is how accurately the LinuxCNC process can schedule itself. if there is a big delay or jitter between calls that are supposed to be at a fixed distance in time apart then the machine control will not be as you might like. The latency-test program can be used to see the jitter.

The LinuxCNC images are based on Debian wheezy. This distribution includes a Linux kernel designed for supporting real time operation. There are a few real time Linux options floating around and the kernel on stretch (rt preempt) that is installed by installing linuxcnc-uspace is different to the one used by the wheezy image (RTAI). IIRC.

For controlling the machine I put together a combination for around $200 including an Intel Pentium G4560 on a Asus H110M-C2 with 4gb of 2400Mhz RAM. This relies on having cases, psu, drives etc already. This has an Intel NIC and onboard prt header (no pci bracket). The CPU has 2 cores and can do two HT.

Booting the current LinuxCNC iso image and running the latency-test gave about 3200 idle and 5100 under load. The two downsides are that the kernel was old enough that M.2 didn't work and the NIC was not detected.

Installing Debian Stretch on the machine the M2 and nic were both detected and all was well. I then installed the LinuxCNC packages which brought in the preempt rt kernel:

Linux bitbreaker 4.9.0-6-rt-amd64 #1 SMP PREEMPT RT Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux

The latency-test came out at about 40,000 idle and up to 70,000 under load. Not quite what I had hoped for. Adding "isolcpus=2,3 idle=poll intel_idle.max_cstate=0 processor.max_cstate=0" to the kernel cmdline and using taskset -cp 2 $pid moved the numbers to 22,000 and 40,000 respectively. Better, but not what I had seen on the official recommended ISO bootup.

To cut a long story short(er) doing a telinit 3 and going non graphical on the control machine and using ssh -Y to login and run the latency-test I was able to get down to 4000 under load. This is again by forcing the rtapi_app to use a reserved CPU core.

Maybe using a different graphics card or some other graphics options would allow the latency to be lower on the display of the machine. But I'm fairly happy to run the GUI to display on the laptop to harvest the better latency numbers.

It is handy to be able to get closer to the jitter numbers that are given when using the official ISO for LinuxCNC as it gives the feeling that I'm not loosing out on good machine control because I'm on stretch and a customized setup. Ironically, using the nice new hardware for the build actually made initial setup harder. Though I imagine that is a temporary situation.

Hopefully those better jitter numbers can be had for real machining over the network too.

Michael Davies: LCA2019 Call for Proposals (CFP) is now open!

Mon, 2018-07-02 16:42
On behalf of the LCA2019 team we are pleased to announce that the Call
for Proposals for linux.conf.au 2019 is now open! This Call for
Proposals will close on July 30.

linux.conf.au is one of the best-known community driven Free and Open
Source Software conferences in the world. In 2019 we welcome you to
join us in Christchurch, New Zealand on Monday 21 January through to
Friday 25 January.

For full details including those not covered by this announcement
visit https://linux.conf.au/call-for-papers/

== IMPORTANT DATES ==

* Call for Proposals Opens: 2 July 2018
* Call for Proposals Closes: 30 July 2018 (no extensions)
* Notifications from the programme committee: early-September 2018
* Conference Opens: 21 January 2019

== HOW TO SUBMIT ==

Create an account or login at https://linux.conf.au/dashboard/ and
click the link to submit your proposal.

== ABOUT LINUX.CONF.AU ==

linux.conf.au is a conference where people gather to learn about the
entire world of Free and Open Source Software, directly from the
people who contribute.  Many of these contributors give scheduled
presentations, but much interaction occurs in-between and after formal
sessions between all attendees. Our aim is to create a deeply
technical conference where we bring together industry leaders and
experts on a wide range of subjects.

linux.conf.au welcomes submissions from first-time and seasoned
speakers, from all free and open technology communities, and all walks
of life. We respect and encourage diversity at our conference.

== CONFERENCE THEME ==

Our theme for linux.conf.au 2019 is "The Linux of Things". Building on
the role that Linux plays in our everyday lives, we will address
IoT-related opportunities and concerns from the purely technical
through environmental, health, privacy, security and more. Please let
this inspire you, but not restrict you - we will still have many talks
about other interesting things in our community.

For some suggestions to get you started with your proposal ideas
please visit the CFP page on the linux.conf.au website.

== PROPOSAL TYPES ==

We're accepting submissions for three different types of proposal:

* Presentation (45 minutes): These are generally presented in lecture
  format and form the bulk of the available conference slots.
* Tutorial (100 minutes): These are generally presented in a classroom
  format. They should be interactive or hands-on in nature. Tutorials
  are expected to have a specific learning outcome for attendees.
* Miniconf (full-day): Single-track mini-conferences that run for the
  duration of a day on either Monday or Tuesday. We provide the room,
  and you provide the speakers. Together, you can explore a field in
  Free and Open Source software in depth.

== PROPOSER RECOGNITION ==

In recognition of the value that presenters and miniconf organisers
bring to our conference, once a proposal is accepted, one presenter or
organiser per proposal is entitled to:
* Free registration, which holds all of the benefits of a Professional
  Delegate Ticket
* A complimentary ticket to the Speakers' Dinner, with additional
  tickets for significant others and children available for purchase.
* 50% off the advertised price for sponsorship at the White-Flippered
  Blue Penguin tier.

If your proposal includes more than one presenter or organiser, these
additional people will be entitled to:
* Professional or hobbyist registration at the Early Bird rate,
  regardless of whether the Early Bird rate is generally available
* Speakers' dinner tickets available for purchase at cost

Important Note for miniconf organisers: These discounts apply to the
organisers only. All participants in your miniconf must arrange or
purchase tickets for themselves via the regular ticket sales process
or they may not be able to attend!

As a volunteer-run non-profit conference, linux.conf.au does not pay
speakers to present at the conference; but you may be eligible for
financial assistance.

== FINANCIAL ASSISTANCE ==

linux.conf.au is able to provide limited financial assistance for some speakers.

Financial assistance may be provided to cover expenses that might
otherwise prohibit a speaker from attending such as:
* Cost of flight
* Accommodation
* Other accessibility related costs

To be considered for assistance you can indicate this when making your
proposal. We will try to accommodate as many requests for assistance
as possible within our limited budget.

== ACCESSIBILITY ==

linux.conf.au aims to be accommodating to everyone who wants to attend
or present at the conference. We recognise that some people face
accessibility challenges. If you have special accessibility
requirements, you can provide that information when submitting your
proposal so that we can plan to properly accommodate you.

We recognise that childcare and meeting dietary requirements also fall
under the general principle of making it possible for everyone to
participate, and will be announcing our offering for these in the near
future; if you have concerns or needs in these areas, or in any area
that would impact your ability to participate, please let us when
submitting your proposal.

== CODE OF CONDUCT ==

By agreeing to present at or attend the conference you are agreeing to
abide by the terms and conditions
(https://linux.conf.au/attend/terms-and-conditions/). We require all
speakers and delegates to have read, understood, and act according to
the standards set forth in our Code of Conduct
(https://linux.conf.au/attend/code-of-conduct/).

== RECORDING ==

To increase the number of people that can view your presentation,
linux.conf.au will record your talk and make it publicly available
after the event. We plan to release recordings of every talk at the
conference under a Creative Commons Share-Alike Licence. When
submitting your proposal you may note that you do not wish to have
your talk released, although we prefer and encourage all presentations
to be recorded.

== LICENSING ==

If the subject of your presentation is software, you must ensure the
software has an Open Source Initiative-approved licence at the time of
the close of our CFP.

Gary Pendergast: General Ann Dunwoody is Joining Automattic!

Sun, 2018-07-01 19:03

I’m excited that General Dunwoody is joining the Automattic board: not just for the new perspective to help bring different ideas, but also for how important diversity is to her.

Personally, I’ve found that my work is only improved when I’m working with a diverse group of people, who don’t think, or act, or look like… well, me! The tech industry as a whole suffers from a myopic lack of diversity: if Automattic can help improve that, then I’m proud to play my part.

Automattic’s First New Board Member: General Ann Dunwoody

As Automattic keeps growing we’ve been bringing in a lot of talented people behind the scenes to help expand on our vision as we go from hundreds to thousands of colleagues, and hundreds of million…

Matt Mullenweg

Simon Lyall: Audiobooks – June 2018

Sun, 2018-07-01 09:03

Flash Boys by Michael Lewis

Well written story of how Wall Street firms didn’t even realize high-speed trading existed while they were losing hundreds of millions to traders practicing it, until a couple of guys told them. 8/10

Woodsman: Living in a Wood in the 21st Century by Ben Law

Write-ups about building his house and the area around it along with various descriptions of traditional crafts, his businesses, appearance on TV etc. Seems to balance well. 7/10

Mars Rover Curiosity: An Inside Account from Curiosity’s Chief Engineer by Rob Manning

Interesting stories about the project with the evolution on the landing method, experiments and project problems. A bit on the short side & published soon after landing. 6/10

The Adventures of Sherlock Holmes V by Sir Arthur Conan Doyle. Read by David Timson

The Adventure of the Reigate Squire, The Adventure of Beryl Coronet, The Boscombe Valley Mystery and The Yellow Face. All good stories, although ‘The Yellow Face’ has not aged well. 7/10

The Dark Forest by Cixin Liu

The 2nd book in the Three-Body Trilogy. A bit closer to the hard-core Sci Fi that I like although the author isn’t going for high accuracy. Feels real and kept my interest. 8/10

A History of Britain, Volume 1: At the Edge of the World? 3000 B. C. – A. D. 1603 by Simon Schama

Fairly straight history of Britain. An overview of everything so covers the main points rather than going into details. 7/10

Audacity: How Barack Obama Defied His Critics and Created a Legacy That Will Prevail by Jonathan Chait

A little overtaken by events (written before Trump elected). Highlights Obama’s wins that the author feels was overlooked. Okay but should have waited 10 years. 6/10

The Adventures of Sherlock Holmes VI by Sir Arthur Conan Doyle

Stories: The Adventure of the Final Problem, The Adventure of “Gloria Scott”, The Adventure of the Noble Bachelor, The Adventure of the Resident Patient. More read by Timson. Good as usual. 7/10

Amazing Stories of the Space Age: True Tales of Nazis in Orbit, Soldiers on the Moon, Orphaned Martian Robots, and Other Fascinating Accounts from the Annals of Spaceflight by Rod Pyle

A mix of the missions that never were & dramatic/odd missions 7/10