Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 1 hour 10 min ago

Linux Users of Victoria (LUV) Announce: LUV November 2018 Workshop: Introduction to shell programming

Sun, 2018-11-11 19:04
Start: Nov 17 2018 12:30 End: Nov 17 2018 16:30 Start: Nov 17 2018 12:30 End: Nov 17 2018 16:30 Location:  Infoxchange, 33 Elizabeth St. Richmond Link:  http://luv.asn.au/meetings/map

Introduction to shell programming

A lot of things in Linux are done on the command line. This talk starts with a very basic knowledge of the command line and describes how to leverage this into a powerful programming system. Transform those snips of commands into powerful self documenting scripts to super charge your productivity.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121.  Late arrivals please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

November 17, 2018 - 12:30

read more

Simon Lyall: DevOpsDaysNZ 2018 – Day 2 – Session 4

Tue, 2018-11-06 15:04

Allen Geer, Amanda Baker – Continuously Testing govt.nz

  • Various .govt.nz sites
  • All Silverstripe and Common Web Platform
  • Many sites out of date, no automated testing, no test metrics, manual testing
  • Micro-waterfall agile
  • Specification by example (prod owner, Devops, QA)  created Gherkin tests
  • Standardised on CircleCI
  • Visualised – Spec by example
  • Prioritised feature tests
  • Ghirkinse
  • Test at start of dev process. Bake Quality in at the start
  • Visualise and display metrics, people could then improve.
  • Path to automation isn’t binary
  • Involve everyone in the team
  • Automation only works if humanised

Jules Clements – Configuration Pipeline : Ruling the One Ring

  • Desired state
  • I didn’t quite understand what he was saying

Nigel Charman – Keep Calm and Carry On Organising

  • 71 Conferences worldwide this year
  • NZ following the rules
  • Lots of help from people
  • Stuff stuff stuff

Jessica DeVita – Retrospecting our Retrospectives

  • Works on Azure DevOps
  • Post-mortems
  • What does it mean to have robust systems and resilience? Is resilience even a property? It just Is. When we fly on planes, we’re trusting machines and automation. Even planes require regular reboots to avoid catastrophic failures, and we just trust that it happen
  • CEO after a million dollar outage said “Can you get me a million dollars of learning out of this?”
  • After US Navy had accidents caused by slept deprivation switched to new watch structure
  • Postmortems are not magic, they don’t automatically make things change
  • http://stella.report
  • We dedicate a lot of time to to below the line, looking at the technology. Not a lot of conversation about above-the-line things like mental models.
  • Resilience is above the line
  • Catching the Apache SNAFU
  • The Ironies of Automation – Lisanne Bainbridge
  • Well facilitated debriefings support recalibration of mental models
  • US Forest Service – Learning Review – Blame discourages people speaking up about problems
  • We never know where the accident boundary is, only when we have crossed it.
    • SRE, Chaos Engineer and Human Factors help hadle
  • In postmortems please be mindful of judging timelines without context. Saying something happened in a short or long period of time is damanging
  • Ask “what made it hard to get that team on the phone?” , “What were you trying to achieve”
  • Etsy Debriefing Guide – lots of important questions.
  • “Moving post shallow incident data” – Adaptive Capacity Labs
  • Safety is a characteristics of Systems and not of their components
  • Ask people about their history, ask every person about what they do and how they got there because that is what shapes your culture as an organisation

Simon Lyall: DevOpsDaysNZ 2018 – Day 2 – Session 3

Tue, 2018-11-06 15:04

Kubernetes

I’ll fill this in later.

Observability

  • Honeycomb, Sumologic. Use AI to look at what happened at same time and magically correlate
  • Expensive or hard to send all logs as volumes go up
  • What is the logging is wrong or missing?
  • Metrics
    • Export in prometheus format
    • Read RED and USE paper
    • Create a company schema with half a dozen metrics that all services expose
  • Had and event or transaction ID that flows across all the microservices sorry logs can be correlated
  • Non technical solutions
    • Refer to previous incident logs
    • Part of deliverables for product is SLA stats which require logs etc
  • Testing logs
    • Make sure certain events produce a log
  • Chaos Monkey

ANZ Drivetrain

  • Change control cares about
    • Avaiability
    • Risk
    • Dependencies
    • Rollback
  • But the team doing the change knows about these all
  • Saw tools out there that seem very opinated
  • Drivetrain
    • Automated Checklist
    • Work with Change people to create checklist
    • Pipeline talks to drivetrain and tells it what has been down
    • Slack messages sent for manual changes (they login to app to approve)
  • Looked at some other tools (eg chef automate, udeploy )
    • Forced team to work in a certain pattern
  • But use ServiceNow tool as official corporate standard
    • Looking at making DriveTrail fill in ServiceNow forms
  • People worried about stages in tool often didn’t realise the existing process had same limitations
  • Risk assessed at the Story and Feature level. Not release level
  • Not suitable for products that due huge released every few months with a massive number of changes.

 

 

Simon Lyall: DevOpsDaysNZ 2018 – Day 2 – Session 2

Tue, 2018-11-06 11:04

Interesting article I read today

Why Doctors Hate their Computers by Atul Gawande

Mrinal Mukherjee – A DevOps Confessional

  • Not about accidents, it is about Planned Blunders that people are doing in DevOps
  • One Track DevOps
    • From Infrastructure background
    • Job going into places, automated the low hanging fruit, easy wins
    • Column of tools on resume
    • Started becoming the bottleneck, his team was the only one who knew how the infrastructure worked.
    • Not able to “DevOps” a company since only able to fix the infrastructure, not able to fix testing etc so not dilvering everything that company expected
    • If you are the only person who understands the infrastructure you are the only one blamed when it goes wrong
    • Fixes
      • Need to take all team on a journey
      • But need to have right expectations set
      • Need to do learning in areas where you have gaps
      • DevOps is not about individual glory, Devops is about delivering value
      • HR needs to make sure they don’t reward the wrong thing
  • MVP-Driven Devops
    • Mostly working on Greenfields products that need to be delivered quickly
    • MVP = Maximum Technical Debt
    • MVP = Delays later and Security audits = Your name attached to the problem
    • Minimum Standard of Engineering
      • Test cases, Documentation, Modular
      • Peer review
    • Evolve architecture, not re-architect
  • Judgemental Devops
    • That team sucks, they are holding things up, playing a different game from us
    • Laughing at other teams
    • Consequence – Stubbornness from the other team
    • Empathy
      • Find out why things are they way they are
    • Collaborate to find common ground and improve
    • Design my system to I plan to work within constraints of the other team

Simon Lyall: DevOpsDaysNZ 2018 – Day 2 – Session 1

Tue, 2018-11-06 09:04

Alison Polton-Simon – The DevOps Experiments: Reflections From a Scaling Startup

  • Software engineer at Angaza, previously Thoughtworks, “Organizational Anthropologist”
  • Angaza
    • Enable sales of life-changing products (eg solar chargers, water pumps, cook stoves in 3rd world countries)
    • Originally did hardware, changed to software company doing pay-as-you-go of existing devices
    • ~50 people. Kenya and SF, 50% engineering
    • No dedicated Ops
    • Innovate with empathy, Maximise impact
    • Model is to provide software tools to activate/deactivate products sold to peopel with low credit-scores. Plus out software around the activity like reports for distributors.
  • Reliability
    • Platform is business critical
    • Outages disrupt real people (households without light, farmers without irrigation)
    • Buildkite, Grafana, Zendesk
  • Constraints
    • Operate in 30+ countries
    • Low connectivity, 2G networks best case
    • Rural and peri-urban areas
    • Team growing by 50% in 2018 (2 eng teams in Kenya + 1 QA)
    • Most customers in timezone where day = SF night
  • Eras of experimentation
    • Ad Hoc
    • Tributes (sacrifice yourself for the stake of the team)
    • Collectives (multiple teams)
    • Product teams
  • ad Hoc – 5 eng
    • 1 eng team
    • Ops by day – you broke, you fix
    • Ops by night – Pagerduty rotation
    • Paged on all backend exception, 3 pages = amnesty
    • Good
      • Small but senior
      • JIT maturity
      • Everyone sitting next to each other
    • Bad
      • Each incident higherly disruptive
      • prioritized necessity over sustainability
  • Tribute – 5-12 eng
    • One person protecting team from interuptions
    • Introduced support person and triage
    • Expanded PD rotation
    • Good
      • More sustainable
      • Blue-Green deploys
      • Clustered workloads
    • Not
      • Headcount != horizontal scaling
      • Hard to hire
      • Customer service declined
  • Collectives 13-20 engs
    • Support and Ops teams – Ops staffed with devs
    • Other teams build roadmaps and requests
    • Teams rotate quarterly – helps onboarding
    • Good
      • Cross train ppl
      • Allow for focus
      • allowed ppl to get depth
    • Bad
      • Teams don’t op what they built
      • Quarter flies by quickly
      • Context switch is costly
      • Still a juggling act
      • 1m ramping up, 6w going okay, 2w winging down
  • Product teams  21 -? eng
    • 5 teams, 2 in Nairobi
    • Teams allighned with business virticals, KPIs
    • Dev, own and maintain services
    • Per-team tributes
    • No [Dev]Ops team
    • Intended goals
      • Independent teams
      • own what build
      • Support biz KPIs
      • cross team coordination
    • Expected Chellenges
      • Ownership without responsbility
      • Global knowledge sharing
      • Return to tribute system (2w out of the workflow)
  • Next
    • Keep growing team
    • Working groups
    • Eventual SRE
    • 24h global coverage
  • Case a “Constitution” of values that everybody who is hired signs
  • Takeaways
    • Maximise impact
      • Dependable tools over fashionable ones
      • Prefer industry-std tech
      • But get creative when necessary
    • Define what reliability means for your system
    • Evolve with Empathy
      • Don’t be dogmatic without structure
      • Serve your customers and your team
      • Adapt when necessary
      • Talk to people
      • Be explicit as to the tradeoffs you are making

Anthony Borton – Four lessons learnt from Microsoft’s DevOps Transformation

  • Microsoft starting in 1975
  • 93k odd engineers at Microsoft
    • 78k deployments per day
    • 2m commits per month
    • 4.4 builds/month
    • 500 million tests/day
  • 2018 State of Devops reports looks at Elite performers in the space
  • TFS – Team Foundation Server
    • Move product to the cloud
    • Moved on-prem to one instance
    • Each account had it’s own DB (broke stuff at 11k DBs)
  • 4 lessons
    • Customer focussed
      • Listen to customers, uservoice.com
      • Lots of team members keep eye on it
      • Stackoverflow
      • Embed with customers
      • Feedback inside product
      • Have to listen in all the channels
    • Failure is an opportunity to learn
    • Definition of done
      • Live in prod, collecting telemetary that examines hypotheses that it was created to prove
    • “For those of you who don’t know who Encarta is, look it up on Wikipedia”
  • Team Structure
    • Combined engineering = devs + testing
      • Some testers left team or organisation
    • Feature team
      • Physical team rooms
      • Cross discipline
      • 10-12 people
      • self managing
      • Clear charter and goals
      • Intact for 12-18 months
    • Sticky note exercise, people pick which teams they would like to join (first 3 choices)
      • 20% choose to change
      • 100% get the choice
  • New constants and requirements
    • Problems
      • Tests took too long – 22h to 2days
      • Tests failed frequently – On 60% passed 100%
      • Quality signal unreliable in master
    • Publish VSTS Quality vision
      • Sorted by exteranl dependancies
      • Unit tests
        • L0 – in-memory unit tests
        • L1 – More with SQL
      • Functional Tests
        • L2 – Functional tests against testable service deployment
        • L3 – Restricted class integration tests that run against production
      • 83k L0 tests run agains all pulls very fast
  • Deploy to rings of users
    • Ring 0 – Internal Only
    • Ring 1 – Small Datacentre 1-1.5m accounts in Brazil (same TZ as US)
    • Ring 2 – Public accounts, medium-large data centre
    • Ring 3 – Large internal accounts
    • Ring 4-5 – everyone else
    • Takes about a week for normal releases.
    • Binaries go out and then the database changes
    • Delays of minutes (up to 75) during the deploys to allow bugs to manafest
    • Some customers have access to feature flags
    • Customers who are risk tolerant can opt in to early deploys. Allows them to get faster feedback from people who are able to provide it
  • More features delivered in 2016 than previous 4 years. 50% more in 2017

 

Simon Lyall: DevOpsDaysNZ 2018 – Day 1 – Session 4

Mon, 2018-11-05 15:04

Everett Toews – A Trip Down CI/CD Lane

I missed most of this talk. Sounded Good.

Jeff Smith – Creating Shared Contexts

  • Ideas and viewpoints are different from diff people
  • Happens in organisation, need to make sure everybody is on the same page
  • Build a shared context via conversations
  • Info exchange
  • Communications tools
  • Context Tools
  • X/Y Problem
  • Data can bridge conversations. Shared reality.
  • Use the same tools as other teams so you know what they are doing
  • Give the context to your requests, ask for them and it will automatically happen eventually.

Peter Sellars – 2018: A Build Engineers Odyssey

  • Hungry, Humble and Smart

Katrina Clokie – Testing in DevOps for Engineers

  • We can already write, so how hard can it be to write a novel?
  • Hopefully some of you are doing testing already
  • Problem is that people overestimate their testing skills, not interested in finding out anything else.
  • The testing you are doing now might be with one tool, in one spot. You are probably finding stuff but missing other things
  • Why important
    • Testing is part of you role
    • In Devops testing goes though Operations as well
    • Testing is DevOps is like air, it is all around you in every role
    • Roles of testers is to tech people to breath continuously and naturally.
  • Change the questions that you ask
    • How do you know that something is okay? What questions are you asking of your product?
    • Oracles are the ways that we recognise a problem
    • Devs ask: “Does it work how we think it should?”
    • Ops ask: “Does it work how it usually works?”
    • Devs on claims, Ops on history
    • Does it work like our competors, does it meet it’s purpose without harmful side effects, doesn’t it meet legal requirements, Does it work like similar services.
    • HICCUPPS – Testing without a Map – Michael Bolton, 2005
    • How do we compare to what other people are doing?  ( Not just a BA’s job , cause the customer will be asking a question and so should you)
    • Flip the Oracle, compare them against other things not just the usual.
    • Audit – Continuous compliance, Always think about if it works like the standards say it should.
    • These are things that the business is asking. If you ask then gain confidence of business
  • Look for Answers in Other Places
    • Number of tests: UI <  Service < Unit
    • The Test Pyramid as a bug catcher. Catch the Simple bugs first and then the subtle ones
  • Testing mesh
    • Unit tests – fine mesh
    • Intergration – Bigger/Fewer tests but cover more
    • Next few layers: End to End, Alerting, Monitoring, Logging. Each stops different types of bugs
    • Conversation should be “Where do we put our mesh?”, “How far can this bug fall?”.
    • If another layer will pick the bug up do we need a test.
  • Use Monitoring as testing
    • Push risk really late, no in all cases but can often work
  • A/B testing
    • Ops needs to monitor
    • Dev needs framework to role out and put in different options
  • Chaos Engineering
    • Start with something small, communicate well and do during daylight hours.
    • Yours customers are testing in production all the time, so why arn’t you too?
  • https://leanpub.com/testingindevops

 

Simon Lyall: DevOpsDaysNZ 2018 – Day 1 – Session 3

Mon, 2018-11-05 15:04

Open Space 1 – Prod Support, who’s responsible

  • Problem that Ops doesn’t know products, devs can’t fix, product support owners not technical enough
  • Xero have embedded Ops and dev in teams. Each person oncall maybe 2 weeks in 20
  • Customer support team does everything?
  • “Ops have big graphs on screens, BI have a couple of BI stats on screens, Devs have …youtube videos”
  • Tiers support vs Product team vs Product support team
  • Tiered support
    • Single point of entry
    • lower paid person can handle easy stuff
    • Context across multiple apps
  • Product Team
    • Buck stops with someone
    • More likely to be able to action
    • Ownership of issues
    • Everyone must be enabled to do stuff
    • Everyone needs to be upskilled
  • Prod Support
    • Big skilled can fix anything team
    • Devs not keen
    • Even the best teams don’t know everything

Open Space 2 – DevOps at NZ Scale

  • Devops team, 3rd silo
    • Sometimes they are the new team doing cool stuff
    • One model is evangelism team
  • Do you want devops culture or do you just want somebody to look after your pipeline?
  • Companies often don’t know what they want to hire
  • Companies get some benefit with the tools (pipelines, agile)  but not the culture. But to get the whole benefit they need to adopt everything.
  • The Way of Ways article by John Cutler

Open Space 3 – Responding Quickly

I was taking notes on the board.

Simon Lyall: DevOpsDaysNZ 2018 – Day 1 – Session 2

Mon, 2018-11-05 11:04

Mark Simpson, Carlie Osborne – Transforming the Bank: pipelines not paperwork

  • Change really is possible even in the least likely places
  • Big and risk adverse
    • Lots of paperwork and progress, very slow
  • Needed to change – In the beginning – 18 months ago
    • 6 months talking about how we could change things
  • Looked for a pilot project – Internet Banking team – ~80 people – Go-money platform
    • Big monolith, 1m lines of code
    • New release every 6 weeks
    • 10 weeks for feature from start to production
    • Release on midnight on a Friday night, 4-5 hours outage, 20-25 people.
    • Customer complaints about outage at midnight, move to 2am Sunday morning
  • Change to release every single week
    • Has to be middle of the day, no outage
    • How do we do this?
  • Took whole Internet banking team for 12 weeks to create process, did nothing else.
  • What we didn’t do
    • Didn’t replatform, no time
  • What we did
    • Jenkins – Created a single Pipeline, from commit to master all the way to projection
    • Got rid of selenium tests
    • Switched to cypress.io
      • Just tested 5 key customer journeys
    • Drivetrain – Internal App
      • Wanted to empower the teams, but lots of limits within industry/regulations
      • Centralise decision making
      • Lightweight Rules engine, checks that all the requirements have been done by the team before going to the next stage.
    • Cannery Deployments
      • Two versions running, ability to switch users to one or other
  • Learning to Break things down into small chunks
  • Change Process
    • Lots of random rules, eg mandatory standdown times
    • New change process for teams using Drivetrain, certified process no each release
  • Lots of times spent talking to people
    • Had to get lots of signoffs
  • Result
    • Successful
    • 16 weeks rather than 12
    • 28 releases in less than 6 months (vs approx 4 previously)
    • 95% less toil for each release
  • Small not Big changes
    • Now takes just 4-5 weeks to cycle though a feature
    • Don’t like saying MVP. Pitch is as quickly delivering a bit of value
    • and iterating
    • 2 week pilot, not iterations -> 8 week pilot, 4 iterations
    • Solution at start -> Solution derived over time
  • Sooner, not later
    • Previously
      • Risk, operations people not engaged until too late
      • Dev team disengaged from getting things into production
    • Now
      • Everybody engaged earlier
  • Other teams adopting similar approach

Ryan McCarvill – Fighting fires with DevOps

  • Lots of information coming into a firetruck, displayed on dashboard
  • Old System was 8-degit codes
  • Rugged server in each each truck
    • UPS
    • Raspberry Pi
    • Storage
    • Lots of different networking
  • Requirements
    • Redundant Comms
    • Realtime
    • Offline Mpas
    • Offline documentation, site reports, photos, building info
    • Offline Hazzards
    • Allow firefighters to update
    • Track appliance and firefighter status
    • Be a hub for an incident
    • Needs to be very secure
  • Stack on the Truck
    • Ansible, git, docker, .netcode, redis, 20 micoservices
  • What happens if update fails?
  • More than 1000 trucks, might be offline for months at a time
  • How to keep secure
  • AND iterate quickly
  • Pipeline
    • Online update when truck is at home
    • Don’t update if moving
    • Blue/Green updates
    • Health probes
  • Visual Studio Team Services -> Azure cont registry
  • Playbooks on git , ansible pull,
  • Nginx in front of blue/green
  • Built – there were problems
    • Some overheating
    • Server in truck taken out of scope, lost offline strategy
    • No money or options to buy new solution
  • MVP requirements
    • Lots of gigs of data, made some so only online
    • But many gigs still needed online
    • Create virtual firetruck in the sky, worked for online
    • Still had communication device – 1 core, minimum storage, locked down Linux
  • Put a USB stick in the back device and updated it
    • Can’t use a lot of resources or will inpact comms
    • Hazard search
      • Java/python app, no much impact on system
      • Re-wrote in rust, low impact and worked
      • Changed push to rsync and bash
  • Lessons
    • Automation gots us flexability to change
    • Automation gave us flexability to grow
    • Creativity can solve any problem
    • You can solve new problems with old technology
    • Sometimes the only way to get buy in is to just do it.

Simon Lyall: DevOpsDaysNZ 2018 – Day 1 – Session 1

Mon, 2018-11-05 09:04

Jeff Smith – Moving from Ops to DevOps: Centro’s Journey to the Promiseland

  • Everyone’s transformation will look a little different
  • Tools are important but not the main problem (dev vs Ops)
  • Hiring a DevOps Manager
    • Just sounds like a normal manager
    • Change it to “Director of Production Operations”
  • A “DevOps Manager” is the 3rd silo on top of Dev and Ops
  • What did peopel say was wrong when he got there?
    • Paternalistic Ops view
      • Devs had no rights on instances
      • Devs no prod access
      • Devs could not create alerts
  • Fix to reduce Ops load
    • Devs get root to instances, but access to easily destroy and recreate if they broke it
    • Devs get access to common safe tasks, required automation and tools (which Ops also adopted)
    • Migrated to datadog – Single tool for all monitoring that anyone could access/update.
    • Shared info about infrastructure. Docs, lunch and learns. Pairing.
  • Expanding the scope of Ops
    • Included in the training and dev environment, CICD. Customers are internal and external
    • Used same code to build every environment
    • Offering Operation expertise
      • Don’t assume the people who write the software know the best way to run it
  • Behaviour can impact performance
    • See book “Turn the Ship around”
    • Participate in Developer rituals – Standups, Retros
    • Start with “Yes.. But” instead of “No” for requests. Assume you can but make it safe
    • Can you give me some context. Do just do the request, get the full picture.
  • Metrics to Track
    • Planned vs unplanned work
    • What are you doing lots of times.
  • What we talk about?
    • Don’t allow your ops dept to be a nanny
    • Remove nanny state but maintain operation safety
    • Monitor how your language impacts behavour
    • Monitor and track the type of work you are doing

François Conil – Monitoring that cares (the end of user based monitoring)

  • User Based monitoring (when people who are affected let you know it is down)
  • Why are we not getting alerts?
    • We are are not measuring the right thing
    • Just ignore the dashboard (always orange or red)
    • Just don’t understand the system
  • First Steps
    • Accept that things are not fine
    • Decide what you need to be measuring, who needs to know, etc. First principals
    • A little help goes a long way ( need a team with complementary strengths)
  • Actionable Alerts
    • Something Broken, User affected, I am the best person to fix, I need to fix immediately
    • Unless all 4 apply then nobody should be woken up.
      • Measured: Take to QA or performance engineers to find out the baseline
      • User affected: If nobody is affected do we care? Do people even work nights? How do you gather feedback?
      • Best person to fix: Should ops guys who doesn’t understand it be the first person to page?
      • Do it need to be fixed? – Backup environment, Too much detail in the alerts, Don’t alert on everything that is broken, just the one causing the problem
  • Fix the cause of the alerts that are happening the most often
  • You need time to get things done
    • Talk to people
    • Find time for fixes
  • You need money to get things done
    • How much is the current situation costing the company?
    • Tech-Debt Friday

Linux Users of Victoria (LUV) Announce: LUV November 2018 Main Meeting: Computerbank / Designing Open Democracy

Fri, 2018-11-02 15:04
Start: Nov 7 2018 18:30 End: Nov 7 2018 20:30 Start: Nov 7 2018 18:30 End: Nov 7 2018 20:30 Location:  Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053 Link:  http://www.melbourne.vic.gov.au/community/hubs-bookable-spaces/kathleen-syme-lib...

PLEASE NOTE CHANGE OF DAY DUE TO MELBOURNE CUP HOLIDAY

6:30 PM to 8:30 PM Wednesday, November 7, 2018
Training Room, Kathleen Syme Library, 251 Faraday Street Carlton VIC 3053

Speakers:

Many of us like to go for dinner nearby after the meeting, typically at Brunetti's or Trotters Bistro in Lygon St.  Please let us know if you'd like to join us!

Linux Users of Victoria is a subcommittee of Linux Australia.

November 7, 2018 - 18:30

read more

Francois Marier: Lean data in practice

Fri, 2018-11-02 02:05

Mozilla has been promoting the idea of lean data for a while. It's about recognizing both that data is valuable and that it is a dangerous thing to hold on to. Following these lean data principles forces you to clarify the questions you want to answer and think hard about the minimal set of information you need to answer these questions.

Out of these general principles came the Firefox data collection guidelines. These are the guidelines that every team must follow when they want to collect data about our users and that are enforced through the data stewardship program.

As one of the data steward for Firefox, I have reviewed hundreds of data collection requests and can attest to the fact that Mozilla does follow the lean data principles it promotes. Mozillians are already aware of the problems with collecting large amounts of data, but the Firefox data review process provides an additional opportunity for an outsider to question the necessity of each piece of data. In my experience, this system is quite effective at reducing the data footprint of Firefox.

What does lean data look like in practice? Here are a few examples of changes that were made to restrict the data collected by Firefox to what is truly needed:

  • Collecting a user's country is not particularly identifying in the case of large countries likes the USA, but it can be when it comes to very small island nations. How many Firefox users are there in Niue? Hard to know, but it's definitely less than the number of Firefox users in Germany. After I raised that issue, the team decided to put all of the small countries into a single "other" bucket.

  • Similarly, cities generally have enough users to be non-identifying. However, some municipalities are quite small and can lead to the same problems. There are lots of Firefox users in Portland, Oregon for example, but probably not that many in Portland, Arkansas or Portland, Pennsylvania. If you want to tell the Oregonian Portlanders apart, it might be sufficient to bucket Portland users into "Oregon" and "not Oregon", instead of recording both the city and the state.

  • When collecting window sizes and other pixel-based measurements, it's easier to collect the exact value. However, that exact value could be stable for a while and create a temporary fingerprint for a user. In most cases, teams wanting to collect this kind of data have agreed to round the value in order to increase the number of users in each "bucket" without affecting their ability to answer their underlying questions.

  • Firefox occasionally runs studies which involve collecting specific URLs that users have consented to share with us (e.g. "this site crashes my Firefox"). In most cases though, the full URL is not needed and so I have often been able to get teams to restrict the collection to the hostname, or to at least remove the query string, which could include username and passwords on badly-designed websites.

  • When making use of Google Analytics, it may not be necessary to collect everything it supports by default. For example, my suggestion to trim the referrers was implemented by one of the teams using Google Analytics since while it would have been an interesting data point, it wasn't necessary to answer the questions they had in mind.

Some of these might sound like small wins, but to me they are a sign that the process is working. In most cases, requests are very easy to approve because developers have already done the hard work of data minimization. In a few cases, by asking questions and getting familiar with the problem, the data steward can point out opportunities for further reductions in data collection that the team may have missed.

Francois Marier: Installing Vidyo on Ubuntu 18.04

Tue, 2018-10-30 09:46

Following these instructions as well as the comments in there, I was able to get Vidyo, the proprietary videoconferencing system that Mozilla uses internally, to work on Ubuntu 18.04 (Bionic Beaver). The same instructions should work on recent versions of Debian too.

Installing dependencies

First of all, install all of the package dependencies:

sudo apt install libqt4-designer libqt4-opengl libqt4-svg libqtgui4 libqtwebkit4 sni-qt overlay-scrollbar-gtk2 libcanberra-gtk-module

Then, ensure you have a system tray application running. This should be the case for most desktop environments.

Building a custom Vidyo package

Download version 3.6.3 from the CERN Vidyo Portal but don't expect to be able to install it right away.

You need to first hack the package in order to remove obsolete dependencies.

Once that's done, install the resulting package:

sudo dpkg -i vidyodesktop-custom.deb Packaging fixes and configuration

There are a few more things to fix before it's ready to be used.

First, fix the ownership on the main executable:

sudo chown root:root /usr/bin/VidyoDesktop

Then disable autostart since you don't probably don't want to keep the client running all of the time (and listening on the network) given it hasn't received any updates in a long time and has apparently been abandoned by Vidyo:

sudo rm /etc/xdg/autostart/VidyoDesktop.desktop

Remove any old configs in your home directory that could interfere with this version:

rm -rf ~/.vidyo ~/.config/Vidyo

Finally, launch VidyoDesktop and go into the settings to check "Always use VidyoProxy".

Simon Lyall: Audiobooks – October 2018

Mon, 2018-10-29 09:04

How to Rig an Election by Nic Cheeseman & Brian Klaas

The authors take experiences in various countries (mostly recent 3rd-world examples) as to how elections are rigged. Some advice for reducing it. 8/10

The Hound of the Baskervilles by Sir Arthur Conan Doyle. Read by Stephen Fry

Once again great reading by Fry and a great story. Works very well with all Holmes and Watson action and no giant backstory. 8/10

Our Oriental Heritage: Story of Civilization Series, Book 1 by Will Durant

Covers the early history of Egypt, the Middle East, India, China and Japan. In some cases up to the 20th Century. The book cover arts, religion and philosophy as well Kings and dates. This was written in the 1930s so has some stuff that has been superseded and out of date attitudes to race and religion in places. It long (50 hours) with another 11 volumes still to go but it is pretty good if you don’t mind these problems. 7/10

Chasing New Horizons: Inside the Epic First Mission to Pluto by Alan Stern & David Grinspoon

Stern was one of the originators and principal investigator of the mission so lots of firsthand details about all stages of the project from first conception though various versions.

Donna Benjamin: Six years and 9 months...

Sat, 2018-10-27 13:03
Saturday, October 27, 2018 - 13:05

Six years and 9 months... is a relatively long time. Not as long as some things, longer than others. Relative. As is everything.

But Six years and 9 months is the length of time I've been on the board of the Drupal Association.

I was elected to serve on the board by the community in February 2012, and then nominated to serve for another two terms. That second term expires on 31 October. My original candidate statement makes somewhat nostalgic reading now... and it's now that I wonder, what I achieved. If anything?

But that's the wrong question. There's nothing useful to be gained in trying to answer it.

Instead - I want to reflect on what I learned.

I learned something from everyone at that table. Honestly, I never really lost my sense of imposter syndrome, and I'm freely and gleefully willing to admit that.

Cary Gordon - we shared a passion for DrupalCon. That show grew into the incredible event it is because of seeds you sewed. And your experience running big shows, and supporting small community libraries seemed to be the perfect mix for fueling what Drupal needed.

Steve Purkiss - we were elected together! Your passion for cooperatives, for Drupal, and for getting on with it, and making things happen was infectious! Thank you for standing with me in those weird first few months of being in this weird new place, called the board of the Drupal Association!

Pedro Cambra - I wish I'd heed the lesson you taught me more often. Listen carefully. Speak only when there's something important to say, or to make the case for a perspective that's being missed. But also good humour. And Thank you for helping make the election process better, and helping the DA "own" the mechanics.

Morten - brother. I can't even find the words to say. Your passion for Drupal, for theming, and for our community always inspired me. I miss your energy.

Angie "webchick" Byron - mate! I still can't fathom how you did what you do so effortlessly! Well, I know it's not effortless, but you make it look that way. Your ability to cut through noise, sort things out, get things done, and inspire the Drupal masses to greatness is breathtaking.

Matthew Saunders - you made me appreciate the importance of governance from a different perspective. Thank you for the work you did to strengthen our board processes.

Addison Berry - Sorry Addi - this is a bit shameful, but it was the mezcal, tequila and bourbon lessons that really stuck.

Danese Cooper - I was so grateful for your deep wisdom of Open Source, and the twists and turns of the path it's followed over such a long time. Your eye to pragmatism over zealotry, but steadfast in the important principles.

Shyamala Rajaram - Oh Shyamala! I can't believe we only first met at DrupalCon Mumbai, or perhaps it was only the first time, this time! Thank you for teaching us all how important it is for us to be in India, and embrace our global community.

Ryan Szrama - you stepped onto the board at such a tough moment, but you stepped up into the role of community elected Director, and helped make sense out of what was happening. Sorry not to see you in Drupal Europe.

Rob Gill - Running. I didn't learn this. Sorry.

Tiffany Farriss - You're formidable! You taught me the importance of having principles, and sticking to them. And then using them to build a foundation in the bedrock. You do this with such style, and grace, and good humour. I'm so thankful I've had this time with you.

Jeff Walpole - You made me question my assumptions all the time! You made me laugh, and you gave me excellent bourbon. You always had a way of bringing us back to the real world when we waded too deep into the weeds.

Vesa Palmu - So many things - but the one that still resonates, is we should all celebrate failure. We should create ritual around it, and formalise the lessons failure teaches. We all learn so much more from mistakes, than from successes.

Sameer Verna - For a time, we were the only linux users at the table, and then I defected back to MacOS - I still feel a bit guilty about this, I admit. You championed Free Software at every step - but also, so often, guided us through the strategic mumbo jumbo, to get to the point we needed to.

Steve Francia - "It's not as bad as you all seem to think it is" I don't know why, but I hear this mantra, spoken with your voice, whenever I think of you. Thank you for your Keynote in Nashville, and for everything.

Mike Lamb - I've not yet put into practice the lesson I need to learn from you. To switch off. To really go home, and be home, and switch off the world. I need me some of that, after all of this. Thank you so much for all you've done, but more for your positive, real world perspective. Ta!

Annie - I missed your presence in Germany so much - I feel like I've still got so much to learn from you. You bridged the worlds of digital and marketing, and brought much needed perspective to our thinking. Twas an honour to serve with you.

Audra - With you too, I feel like I was only beginning to get into the groove of the wisdom you're bringing to the table. I hope our paths continue to cross, so I can keep learning!

Baddy Sonja Breidert - A powerful lesson - as volunteers, we have to account for the time, passion and energy we borrow from the rest of our lives, when we give it to Drupal. And Drupal needs to properly recognise it too.

Ingo Rübe - You taught me how to have courage to bring big ideas to the table, and show grace in letting them go.

Michel van Velde - You taught me to interrogate my assumptions, with fun, with good humour, and honest intention of doing good.

George Matthes - You taught me the power of questioning the received wisdom from history. You reminded me of the importance of bringing fresh eyes to every challenge.

Adam Goodman - a simple, but important lesson. That leadership is about caring for people.

Suzanne Dergacheva - newly elected, and about to start your term - I had too little chance to learn from you at the board table, but I already learned that you can teach the whole community kindness by giving them carnations! #DrupalThanks to you too. And power to your arms as you take the oars as a community elected director, and help row us forward!

And to all the staff who've served over the years, your dedication to this organisation and community it serves is incredible. You've all made a difference, together, to all of us. Special mentions for four of you...

Kris - from Munich to Vienna - my constant companion, and my dive bar adventure buddy. Til next time there is cheese...

Holly - Inspiring me to knit! Or, more accurately, to wish I could knit better than I can. To knit with conviction! It's a metaphor for so much, but also very very literally. Also I miss you.

Steph - Your vibrant enthusiasm, and commitment to DrupalCon always inspired me. Your advice on food trucks in Portland nourished me.

Megan - where to start? I'd never finish. Kindness, compassion, steely focus, commercial reality, "operational excellence", and cactus margaritas.

I save my penultimate words for Dries... Thank you for having faith in me. Thank you for creating Drupal, and for sharing it with all of us. Also, thank you sharing many interesting kinds of Gin!

These final words are for Tim - as you take the reins of this crazy sleigh ride into the future - I feel like I'm leaving just before the party is really about to kick off.

Go you good thing.

Good bye, so long, and thanks for all the fish.

The DA does amazing work.
If you rely on Drupal, you rely on them.

Please consider becoming a member, or a supporting partner.

Gary Pendergast: Iterating on Merge Proposals

Fri, 2018-10-26 15:05

Developing new WordPress features as plugins has been a wonderfully valuable process for all sorts of features to come into being, from the MP6 Dashboard Redesign, to oEmbed endpoints, and including multiple Customiser enhancements over the years. Thanks to the flexibility that this model offers, folks have been able to iterate rapidly on a wide range of features, touching just about every part of WordPress.

The “Features as Plugins” idea was first introduced during the WordPress 3.7 development cycle, during which the features were merged after a short discussion during a core chat: it was only in the WordPress 3.8 cycle that the idea of a merge proposal post (called “Present Your Feature” back then) came into being. It was envisioned as a way to consult with WordPress leaders, key contributors, and the wider WordPress community on the readiness of this feature to be released. Ultimately, WordPress leaders would make a decision on whether the feature was right for WordPress, and the release lead would decide if it was ready for that release.

Since then, most feature plugins have published some form of merge proposal post before they were ultimately merged into WordPress, and they’ve nearly all benefited to some degree from this process.

The merge proposal process has worked well for smaller features, but it struggled with larger changes.

The REST API is a great example of where the merge proposal process didn’t work. The REST API was a significant change, and trying to communicate the scope of that change within the bounds of a single merge proposal post didn’t really do it justice. It was impossible to convey everything that was changing, how it all worked together, and what it meant for WordPress.

I’d go so far as to say that the shortcomings of the merge proposal process are at least partially responsible for why the REST API hasn’t seen the level of adoption we’d hoped for. It’s managed to gain a moderate amount of popularity with WordPress development agencies, and a handful of plugins use it in some ways, but it never really entered into mainstream usage in the ways it could’ve.

In a project that prides itself on being willing to try new ideas, the merge proposal process has remained largely static for many years.

Gutenberg is the first opportunity since the REST API was merged where we can examine the shortcomings of the merge proposal process, and see how we can apply the original intent of it to the Gutenberg project’s scope and long term vision.

Merge Consultation

Over the last six months, Gutenberg project leads have been consulting with teams across the WordPress project. Helping them get involved when they didn’t have any Gutenberg experience, explaining how their focus fit into the vision for Gutenberg, and listening to feedback on where things needed to be improved. In many circumstances, this consultation process has been quite successful: the WordPress Media and REST API teams are great examples of that. Both teams have got up to speed on the Gutenberg project, and have provided their valuable experience to make it even better.

That’s not to say it’s been entirely successful. There’s been a lot of discussion about Gutenberg and Accessibility recently, much of it boils down to what Joe Dolson summarised as being “too little, too late”. He’s correct, the Accessibility team should’ve been consulted more closely, much earlier in the process, and that’s a mistake I expect to see rectified as the Gutenberg project moves into its next phase after WordPress 5.0. While Gutenberg has always aimed to prioritise accessibility, both providing tools to make the block editor more accessible, as well as encouraging authors to publish accessible content, there are still areas where we can improve.

While there’s much to be discussed following WordPress 5.0, we can already see now that different teams needed to be consulted at different points during the project. Where Gutenberg has aimed to consult with teams earlier than a previous feature plugin would’ve, we need to push that further, ensuring that teams are empowered to get involved earlier still in the process.

All feature plugins in the future, great and small, will benefit from this iteration.

Creating a framework for more fluid feedback over the entire lifecycle of a feature project is beneficial for everyone. WordPress teams can ensure that their feedback is taken on board at the right time, project leads gain experience across the broad range of teams that work on WordPress, and projects themselves are able to produce a better resulting feature.

They important thing to remember throughout all of this is that everything is an experiment. We can try an approach, discover the weaknesses, and iterate. We’re all only human, we all make mistakes, but every mistake is an opportunity to ensure the same mistake can’t happen again. Sometimes that means changing the software, and sometimes that means changing the processes that help build the software. Either way, we’re always able to iterate further, and make WordPress fun for everyone.

Lev Lafayette: Performance Improvements with GPUs for Marine Biodiversity: A Cross-Tasman Collaboration

Thu, 2018-10-18 11:05

Identifying probable dispersal routes and for marine populations is a data and processing intensive task of which traditional high performance computing systems are suitable, even for single-threaded applications. Whilst processing dependencies between the datasets exist, a large level of independence between sets allows for use of job arrays to significantly improve processing time. Identification of bottle-necks within the code base suitable for GPU optimisation however had led to additional performance improvements which can be coupled with the existing benefits from job arrays. This small example offers an example of how to optimise single-threaded applications suitable for GPU architectures for significant performance improvements. Further development is suggested with the expansion of the GPU capability of the University of Melbourne’s “Spartan” HPC system.

A presentation to EResearchAustralasia 2018.

OpenSTEM: Children in Singapore will no longer be ranked by exam results. Here’s why | World Economic Forum

Tue, 2018-10-16 11:06
https://www.weforum.org/agenda/2018/10/singapore-has-abolished-school-exam-rankings-here-s-why The island nation is changing its educational focus to encourage school children to develop the life skills they will need when they enter the world of work.

Lev Lafayette: International HPC Certification Program

Fri, 2018-10-12 11:05

The HPC community has always considered the training of new and existing HPC practitioners to be of high importance to its growth. The significance of training will increase even further in the era of Exascale when HPC encompasses even more scientic disciplines. This diversification of HPC practitioners challenges the traditional training approaches, which are not able to satisfy the specific needs of users, often coming from non-traditionally HPC disciplines and only interested in learning a particular set of skills. HPC centres are struggling to identify and overcome the gaps in users' knowledge. How should we support prospective and existing users who are not aware of their own knowledge gaps? We are working towards the establishment of an International HPC Certification program that would clearly categorize, define and examine them similarly to a school curriculum. Ultimately, we aim for the certificates to be recognized and respected by the HPC community and industry.

International HPC Certification Program, International Supercomputing Conference, Frankfurt, June, 2018

Julian Kunkel (University of Reading), Kai Himstedt (Universität Hamburg), Weronika Filinger (University of Edinburgh), Jean-Thomas Acquaviva (DDN), William Jalby (Université de Versailles Saint-Quentin), Lev Lafayette (University of Melbourne)

Simon Lyall: Audiobooks – September 2018

Wed, 2018-10-10 21:04

Lone Star: A History of Texas and the Texans by T. R. Fehrenbach

About 80% of the 40 hour book covers the period 1820-1880. Huge amounts of detail during then but skips over the rest quickly. Great stories though. 8/10

That’s Not English – Britishisms, Americanisms, and What Our English Says About Us by Erin Moore

A series of short chapters (usually one per word) about how the English language is used differently in England from the US. Fun light read. 7/10

The Complacent Class: The Self-Defeating Quest for the American Dream by Tyler Cowen

How American culture (and I’d extend that to countries like NZ) has stopped innovating and gone the safe route in most areas. Main thesis is that pressure is building up and things may break hard. Interesting 8/10

A History of Britain, Volume 2 : The British Wars 1603 – 1776 by Simon Schama

Covering the Civil War, Glorious Revolution and bits of the early empire and American revolution. A nice overview. 7/10

I Find Your Lack of Faith Disturbing: Star Wars and the Triumph of Geek Culture by A. D. Jameson

A personal account of the author’s journey though Geekdom (mainly of the Sci-Fi sort) mixed in with a bit of analysis of how the material is deeper than critics usually credit. 7/10

sthbrx - a POWER technical blog: Open Source Firmware Conference 2018

Tue, 2018-10-09 10:08

I recently had the pleasure of attending the 2018 Open Source Firmware Conference in Erlangen, Germany. Compared to other more general conferences I've attended in the past, the laser focus of OSFC on firmware and especially firmware security was fascinating. Seeing developers from across the world coming together to discuss how they are improving their corner of the stack was great, and I've walked away with plenty of new knowledge and ideas (and several kilos of German food and drink..).

What was especially exciting though is that I had the chance to talk about my work on Petitboot and what's happened from the POWER8 launch until now. If you're interested in that, or seeing how I talk after 36 hours of travel, check it out here:

OSFC have made all the talks from the first two days available in a playlist on Youtube
If you're after a few suggestions there was, in no particular order:

Ryan O'Leary giving an update on Linuxboot - also known as NERF, Google's approach to a Linux bootloader all written in Go.

Subrate Banik talking about porting Coreboot on top of Intel's FSP

Ron Minnich describing his work with "rompayloads" on Coreboot

Vadmin Bendebury describing Google's "Secure Microcontroller" Chip

Facebook presenting their use of Linuxboot and "systemboot"

And heaps more, check out the full playlist!