Planet Linux Australia

Syndicate content
Planet Linux Australia - http://planet.linux.org.au
Updated: 34 min 42 sec ago

Michael Still: Quartz trig

Mon, 2015-07-13 17:29
A morning of vacation geocaching, wandering, and walking to quartz trig. Quartz was a disappointment as its just a bolt in the ground, but this was a really nice area I am glad I wandered around in. This terrain would be very good for cubs and inexperienced scouts.



                                       



See more thumbnails



Interactive map for this route.



Tags for this post: blog pictures 20150713-quartz photo canberra bushwalk trig_point

Related posts: Goodwin trig; Big Monks; Narrabundah trig and 16 geocaches; Cooleman and Arawang Trigs; One Tree and Painter; A walk around Mount Stranger



Comment

Sridhar Dhanapalan: Twitter posts: 2015-07-06 to 2015-07-12

Mon, 2015-07-13 00:27

Chris Samuel: Git: Renaming/swapping “master” with a branch on Github

Sun, 2015-07-12 19:26

I was playing around with some code and after having got it working I thought I’d make just one more little quick easy change to finish it off and found that I was descending a spiral of additional complexity due to the environment in which it had to work. As this was going to be “easy” I’d been pushing the commits to master on Github (I’m the only one using this code) and of course a few reworks in I’d realised that this was never going to work out well and needed to be abandoned.

So, how to fix this? The ideal situation would be to just disappear all the commits after the last good one, but that’s not really an option, so what I wanted was to create a branch from the last good point and then swap master and that branch over. Googling pointed me to some possibilities, including this “deprecated feedback” item from “githubtraining” which was a useful guide so I thought I should blog what worked for me in case it helps others.

  1. git checkout -b good $LAST_GOOD_COMMIT # This creates a new branch from the last good commit
  2. git branch -m master development # This renames the "master" branch to "development"
  3. git branch -m good master # This renames the "good" branch to "master".
  4. git push origin development # This pushes the "development" branch to Github
  5. In the Github web interface I went to my repos “Settings” on the right hand side (just above the “clone URL” part) and changed the default branch to “development“.
  6. git push origin :master # This deletes the "master" branch on Github
  7. git push --all # This pushes our new master branch (and everything else) to Github
  8. In the Github web interface I went and changed my default branch back to “master“.

…and that was it, not too bad!

You probably don’t want to do this if anyone else is using this repo though.

David Rowe: Trellis Decoding for Codec 2

Sun, 2015-07-12 11:30

OK, so FreeDV 700 was released a few weeks ago and I’m working on some ideas to improve it. Especially those annoying R2D2 noises due to bit errors at low SNRs.

I’m trying some ideas to improve the speech quality without the use of Forward Error Correction (FEC).

Speech coding is the art of “what can I throw away”. Speech codecs remove a bunch of redundant information. As much as they can. Hopefully with whats left you can still understand the reconstructed speech.

However there is still a bit of left over redundancy. One sample of a model parameter can look a lot like the previous and next sample. If our codec quantisation was really clever, adjacent samples would look like noise. The previous and next samples would look nothing like the current one. They would be totally uncorrelated, and our codec bit rate would be minimised.

This leads to a couple of different approaches to the problem of sending coded speech over channel with bit errors:

The first, conventional approach is to compress the speech as much as we can. This lowers the bit rate but makes the coded speech very susceptible to bit errors. One bit error might make a lot of speech sound bad. So we insert Forward Error correction (FEC) bits, raising the overall bit rate (not so great), but protecting the delicate coded speech bits.

This is also a common approach for sending data over dodgy channels. For data, we cannot tolerate any bit errors, so we use FEC, which can correct every error (or die trying).

However speech is not like data. If we get a click or a pop in the decoded speech we don’t care much. As long as we can sorta make out what was said. Our “Brain FEC” will then work out what the message was.

Which leads us to another approach. If we leave a little redundancy in the coded speech, we can use that to help correct or at least smooth out the received speech. Remember that for speech, it doesn’t have to be perfect. Near enough is good enough. That can be exploited to get us gain over a system that uses FEC.

Turns out that in the Bit Error Rate (BER) ranges we are playing with (5-10%) it’s hard to get a good FEC code. Many of the short ones break – they introduce more errors than they correct. The really good ones are complex with large block sizes (1000s of bits) that introduce unacceptable delay. For example at 700 bit/s, a 7000 bit FEC codeword is 10 seconds of coded speech. Ooops. Not exactly push to talk. And don’t get me started on the memory, MIPs, implementation complexity, and modem synchronisation issues.

These ideas are not new, and I have been influenced by some guys I know who have worked in this area (Philip and Wade if you’re out there). But not influenced enough to actually look up and read their work yet, lol.

The Model

So the idea is to exploit the fact that each codec model parameter changes fairly slowly. Another way of looking at this is the probability of a big change is low. Take a look at the “trellis” diagram below, drawn for a parameter that is represented by a 2 bit “codeword”:

Lets say we know our current received codeword at time n is 00. We happen to know it’s fairly likely (50%) that the next received bits at time n+1 will be 00. A 11, however, is very unlikely (0%), so if we receive a 11 after a 00 there is very probably an error, which we can correct.

The model I am using works like this:

  1. We examine three received codewords: the previous, current, and next.
  2. Given a received codeword we can work out the probability of each possible transmitted codeword. For example we might BPSK modulate the two bit codeword 00 as -1 -1. However when we add noise the receiver will see -1.5 -0.25. So the receiver can then say, well … it’s most likely -1 -1 was sent, but it also could have been a -1 1, and maybe the noise messed up the last bit.
  3. So we work out the probability of each sequence of three codewords, given the probability of jumping from one codeword to the next. For example here is one possible “path”, 00-11-00:

    total prob =

    (prob a 00 was sent at time n-1) AND

    (prob of a jump from 00 at time n-1 to 11 at time n) AND

    (prob a 11 was sent at time n) AND

    (prob of a jump from 11 at time n to 00 at time n+1) AND

    (prob a 00 was sent at time n+1)

  4. All possible paths of the three received values are examined, and the most likely one chosen.

The transition probabilities are pre-computed using a training database of coded speech. Although it is possible to measure these on the fly, training up to each speaker.

I think this technique is called maximum likelihood decoding.

Demo and Walk through

To test this idea I wrote a GNU Octave simulation called trellis.m

Here is a test run for a single trellis decode. The internal states are dumped for your viewing pleasure. You can see the probability calculations for each received codeword, the transition probabilities for each state, and the exhaustive search of all possible paths through the 3 received codewords. At the end, it get’s the right answer, the middle codeword is decoded as a 00.

For convenience the probability calculations are done in the log domain, so rather than multiplies we can use adds. So a large negative “probability” means really unlikely, a positive one likely.

Here is a plot of 10 seconds of a 4 bit LSP parameter:

You can see a segments where it is relatively stable, and some others where it’s bouncing around. This is a mesh plot of the transition probabilities, generated from a small training database:

It’s pretty close to a “eye” matrix. For example, if you are in state 10, it’s fairly likely the next state will be close by, and less likely you will jump to a remote state like 0 or 15.

Here is test run using data from several seconds of coded speech:



octave:130> trellis

loading training database and generating tp .... done

loading test database .... done

Eb/No: 3.01 dB nerrors 28 29 BER: 0.03 0.03 std dev: 0.69 1.76

We are decoding using trellis based decoding, and simple hard decision decoding. Note how the number of errors and BER is the same? However the std dev (distance) between the transmitted and decoded codewords is much better for trellis based decoding. This plot shows the decoder errors over 10 seconds of a 4 bit parameter:

See how the trellis decoding produces smaller errors?

Not all bit errors are created equal. The trellis based decoding favours small errors that have a smaller perceptual effect (we can’t hear them). Simple hard decision decoding has a random distribution of errors. Sometimes you get the Most Significant Bit (MSB) of the binary codeword flipped which is bad news. You can see this effect above, with a 4 bit codeword, a MSB error means a jump of +/- 8. These large errors are far less likely with trellis decoding.

Listen

Hear are some samples that compare trellis based decoding to simple hard decision decoding, when applied to Codec2 at 700 bit/s on a AWGN channel using PSK. Only the 6 LSP parameters are tested (short term spectrum), no errors or correction are applied to the excitation parameters (voicing, pitch, energy).

Eb/No (dB) BER Trellis Simple (hard dec) big 0.00 Listen Listen 3.0 0.02 Listen Listen 0.0 0.08 Listen Listen

At 3dB, the trellis based decoding removes most of the effects of bit errors, and it sounds similar to the no error reference. Compared to simple decoding, the bloops and washing machine noises have gone away. At 0dB Eb/No, the speech quality is improved, with some exceptions. Fast changes, like the “W” in double-you, and the “B” in Bruce become indistinct. This is because when the channel noise is high, the probability model favours slow changes in the parameters.

Still – getting any sort of speech at 8% bit error rates with no FEC is pretty cool.

Further Work

These techniques could also be applied to FreeDV 1600, improving the speech quality with no additional overhead. Further work is required to extend these ideas to all the codec parameters, such as pitch, energy, and voicing.

I need to train the transition probabilities with a larger database, or make it train in real time using off air data.

We could include other information in the model, like the relationship of adjacent LSPs, or how energy and pitch change slowly in strongly voiced speech.

Now 10% BER is an interesting, rarely explored area. The data guys start to sweat above 1E-6, and assume everyone else does. At 10% BER FEC codes don’t work well, you need a really long block size or a low FEC rate. Modems struggle due to syncronisation issues. However at 10% the Eb/No versus BER curves start to get flat, so a few dB either way doesn’t change the BER much. This suggests small changes in intelligibility (not much of a threshold effect). Like analog.

Summary

For speech, we don’t need to correct all errors; we just need to make it sound like they are corrected. By leaving some residual redundancy in the coded speech parameters we can use probability models to correct errors in the decoded speech with no FEC overhead.

This work is another example of experimental work we can do with an open source codec. It combines knowledge of the channel, the demodulator and the codec parameters to produce a remarkable result – improved performance with no FEC.

This work is in it’s early stages. But the gains all add up. A few more dB here and there.

References

  1. I found this description of Soft Decision Viterbi decoding useful.
  2. Last year I did some related work on natural versus gray coding.

Binh Nguyen: Electronics (TV) Repair, Working at Amazon, and Dealing With a Malfunctioning Apple iDevice

Sat, 2015-07-11 16:38
I obviously do component level electronics repair from time to time (I've been doing electronics repair/modification since I was fairly young on devices ranging from food processers all the way up to advanced electronic component level repair such as laptops). One of recent experiments was with large screen flat panel (Plasma, LCD, LED, etc...) television sets. Some general notes:



- take precautions. If you've ever watched some of those guys on YouTube, you'll realise that they are probably amateur electrcians and have probably never been shocked/electrocuted before. It's one thing to work with small electronic devices. It's an entirely different matter to be working with mains voltage. Be careful...

- a lot of the time electronic failure will take occur gradually over time (although the amount of time can vary drastically obviously)

- don't just focus on repairing it so that power can flow through the circuit once more. It's possible that it will just fail once more. Home in on the problem area, and make sure everything's working. That way you don't have to keep on dealing with other difficulties down the track

http://electronics.stackexchange.com/questions/25055/slow-blow-vs-fast-acting-fuse

http://en.wikipedia.org/wiki/Fuse_%28electrical%29

- it may only be possible to test components outside of circuit. While testing components with a multimeter will help you may need to purchase more advanced and expensive diagnostic equipment to really figure out what the true cause of the problem is

http://www.eevblog.com/forum/beginners/is-there-any-way-to-test-capacitors-while-on-the-circuit-board/

http://electronics.stackexchange.com/questions/95663/how-to-test-capacitors-of-non-working-circuit-board-using-capacitor-meter

http://en-us.fluke.com/training/training-library/test-tools/digital-multimeters/how-to-measure-capacitance-with-a-digital-multimeter.html

http://www.learningaboutelectronics.com/Articles/How-to-test-a-capacitor

http://www.tpub.com/neets/book7/24l.htm

http://www.ladyada.net/library/metertut/resistance.html?PageSpeed=noscript

https://en.wikipedia.org/wiki/Oscilloscope

- setup a proper test environment. Ideally, one where you have a seperate circuit and where there are safety mechanisms in place to reduce the chances of a total blackout in your house and to increase your personal safety

- any information that you take from this is at your own risk. Please don't think that any of the information here will turn you into a qualified electronics technician or will allow you to solve most problems that you will face

- a lot of the time information on the Internet can be helpful but only applies to particular conditions. Try to understand and work the problem rather than just blindly following what other people do. It may save you a bit of money over the long term

http://en.wikipedia.org/wiki/Failure_modes_of_electronics

http://en.wikipedia.org/wiki/Cascading_failure

https://ca.answers.yahoo.com/question/index?qid=20100213153443AAc1xPb

http://electronics.stackexchange.com/questions/8129/how-do-components-fail

http://www.angelfire.com/planet/funwithtransistors/Book_TS_CHAP-3.html

http://www.angelfire.com/electronic/funwithtubes/Restore_cap.html

http://www.avsforum.com/forum/167-plasma-flat-panel-displays/1490867-samsung-pn60e550d1f-no-picture-has-sound-capacitors-look-ok.html

http://www.avsforum.com/forum/167-plasma-flat-panel-displays/1490976-samsung-51d490-sound-but-no-picture-issue.html

http://forum.allaboutcircuits.com/threads/lg-42-lcd-tv-42lg60fr-picture-goes-black-sound-still-on.86475/

http://www.gamersyde.com/forum_lg_lcd_tv_47lg50_blacked_out_-33_34874_1_en.html

http://www.avsforum.com/forum/166-lcd-flat-panel-displays/1337855-tv-turns-there-sound-but-no-picture.html

https://www.youtube.com/channel/UCUfgq9Gn8S041qQFl0C-CEQ

https://www.youtube.com/user/GrantsPassTVRepairs?feature=hovercard

http://www.explainthatstuff.com/plasmatv.html

http://www.badcaps.net/forum/showthread.php?t=18542

http://www.badcaps.net/forum/showthread.php?t=33333

https://www.ifixit.com/Answers/View/136156/color+on+LCD+fades+to+white+after+10+seconds

http://support-us.samsung.com/cyber/popup/iframe/pop_troubleshooting_fr.jsp?idx=28929&modelname=LT-P227W

https://us.en.kb.sony.com/app/answers/detail/a_id/42696/~/the-tv-turns-off-and-on-by-itself,-reboots,-or-the-standby-light-is-blinking

https://answers.yahoo.com/question/index?qid=20080129052848AAdAtpt

http://forums.cnet.com/7723-13973_102-530929/horizontal-black-lines-on-samsung-50-plasma-tv/

http://forums.cnet.com/7723-13973_102-511585/plasma-tv-has-3-thin-black-lines/

http://everydaylife.globalpost.com/causes-horizontal-lines-lcd-panel-37988.html

Philips 32PFL5522D/05 - Completely dead (no power LED or signs of life) - Diagnosis and repair

https://www.youtube.com/watch?v=TrphsEw8slw how fix tv site:blogspot.com

- electronics repair is becoming increasingly un-economical. Parts may be impossible to find and replacing the TV rather than fixing it may actually be cheaper (especially when the screen is cracked. It's almost certain that a new replacement is going to cost more than the set itself). The only circumstances where it's likely to be worth it is if you have cheap spare parts on hand or the type of failure involves a relatively small, minor, component. The other thing you should know is that while the device may be physically structured in such a way to appear modularised it may not fail in such a fashion. I've been reading about boards which fail but actually have no mechanism to stop it from bleeding into other modules which means you end up in an infinite, failure loop. Replace one bad component with a good one and the leftover apparently good component fails and takes out the new, good board eventually. The cycle then continues on forever before the technician realises this or news of such design spreads. You may have to replace both boards at the same time which then makes the repair un-economical

http://resistor.cherryjourney.pt/

http://www.eeweb.com/toolbox/4-band-resistor-calculator

http://www.1728.org/resisclr.htm

http://www.camradio.net/resistors.html

- spare parts can be extremely difficult to source or are incredibly expensive. Moreover, the quality of the replacement parts can vary drastically in quality. If at all possible work with a source of known quality. Else, ask for demo parts particularly with Asian suppliers who may provide them for free and as a means of establishing a longer term business relationship

http://www.aliexpress.com/store/product/37PFL7422-93-37TA2800-Power-Supply-715T2484-5-Original-parts/715046_32223080629.html

http://www.aliexpress.com/store/product/Non-Replacement-37PFL7422-Power-Board-715T2484-5/1395137_32228251009.html

http://en.wikipedia.org/wiki/Mains_electricity_by_country

http://www.worldstandards.eu/electricity/plug-voltage-by-country/

https://answers.yahoo.com/question/index?qid=20060619172834AADo9qt

http://electronics.stackexchange.com/questions/120983/what-happens-if-a-240v-appliance-is-connected-in-a-120v-ac-power-supply

http://electronics.stackexchange.com/questions/29259/charging-devices-voltage-and-amperage

https://answers.yahoo.com/question/index?qid=20060619172834AADo9qt

- be careful when replacing parts. Try to do your bet to replace like for like. Certain systems will operate in a degraded state if/when using sub-par replacements but will ultimately fail down the line

http://electronics.stackexchange.com/questions/86530/substituting-capacitors

http://www.antiqueradio.org/recap.htm

- use all your senses (and head) to track down a failure more quickly (sight and smell in particular for burnt out components). Sometimes, it may not be obvious where the actual failure is as opposed to where it may appear to be coming from. For instance, one set I looked at had a chirping power supply. It had actually suffered from failures of multiple components which made it appear/sound as though the transformer had failed. Replacement of all relevant components (not the transformer) resulted in a functional power supply unit and stopping of the chirping sound

http://www.cnet.com/au/news/samsung-power-defect-causes-some-tvs-to-fail-and-a-class-action-suit-follows/

http://techreport.com/forums/viewtopic.php?f=37&t=62360&start=960

http://www.tomshardware.com/forum/363126-28-making-clicking-chirping-noises

https://answers.yahoo.com/question/index?qid=20111006212112AAbUauu

http://www.badcaps.net/forum/showthread.php?t=24657

http://www.badcaps.net/forum/showthread.php?t=18603

http://www.badcaps.net/forum/showthread.php?t=17379

http://superuser.com/questions/487729/motherboard-makes-high-pitched-chirping-noise-will-it-stop-by-itself-or-what-d

http://sound.westhost.com/troubleshooting.htm

http://www.instructables.com/id/Repair-your-electronics-by-replacing-blown-capacit/

http://www.justanswer.com/tv-repair/62e22-philips-42-inch-lcd-no-picture-just-chirping-sound-when-turned.html

http://www.fixya.com/tags/chirping/flat_panel_televisions/philips

http://www.ehow.com/how_7387574_philips-turn-making-chirping-noise.html

http://www.capacitorlab.com/visible-failures/

http://www.badcaps.net/pages.php?vid=5

http://www.angelfire.com/electronic/funwithtubes/Testing_caps.html

http://conradhoffman.com/capchecktut.htm

http://www.justanswer.com/tv-repair/71602-fix-vertical-lines-samsung-plasma-tv.html

http://www.digikey.com.au/product-detail/en/ECE-A1AKA470/P806-ND/6913

- as with musical instruments, teardowns may be the best that you can get with regards to details of how a device should work. This is nothing like school/University where you are given a rough idea of how it should work. You may be completely blind here...

http://dtbnguyen.blogspot.com.au/2015/06/repairing-musical-instrumentselectrical.html

- components may be shared across different manufacturers. It doesn't mean that they will work if swapped though. They could be using different version of the same base reference board (similar to the way in which graphics, sound, telecommunications, and network cards rely on reference designs in the ICT sector)

https://www.ifixit.com/

715t2484-3-schematic.pdf

Magnavox has a very similar layout to a similar size Phillips LCD TV

https://www.youtube.com/watch?v=qSS6ycUxS98



Apparently, Amazon are interested in some local talent.

http://aws.amazon.com/careers/aus-nz-event/

There are some bemusing tales of recruitment and the experience of working there though.

http://gawker.com/working-at-amazon-is-a-soul-crushing-experience-1573522379

http://www.theguardian.com/money/2014/nov/28/being-homeless-is-better-than-working-for-amazon

http://www.glassdoor.com.au/Overview/Working-at-Amazon-com-EI_IE6036.11,21.htm

http://www.salon.com/2014/02/23/worse_than_wal_mart_amazons_sick_brutality_and_secret_history_of_ruthlessly_intimidating_workers/



If your iPhone, iPad, or iPod touch doesn't respond or doesn't turn on. If your device is in a lot of trouble I often just run the following command on the storage, 'dd if=/dev/zero of=/dev/[iPod storage node]'. This will create a corrupted filesystem and force restoration of the iOS to factory settings/setup.

https://support.apple.com/en-au/HT201412

https://www.ifixit.com/Wiki/iPod_Classic_Troubleshooting

https://www.ifixit.com/Answers/View/15461/Won%27t+turn+on+or+charge

https://discussions.apple.com/thread/2468023?tstart=0

https://support.apple.com/en-au/HT201263

http://www.howtogeek.com/216839/what-to-do-when-your-iphone-or-ipad-won%E2%80%99t-turn-on/?PageSpeed=noscript



Sometimes digitizers play up. Apparently, a lot of strange behaviour can occur if certain cables are bent improperly or if there isn't enough space/insulation between certain components.

https://www.ifixit.com/Answers/View/137418/Digitizer+freaks+out+when+laid+flat+on+the+frame



Identify your iPad model.

https://support.apple.com/en-au/HT201471



If your device is suffering from device corruption issues you may need to backup your music first...

http://www.wikihow.com/Copy-Music-from-Your-iPod-to-Your-Computer 

http://www.syncios.com/how-to-backup-ipod-music-to-computer.html

http://lifehacker.com/5869827/how-to-copy-music-from-your-iphone-ipad-or-ipod-touch-to-your-computer-for-free

https://discussions.apple.com/thread/5025610

http://geeknizer.com/how-to-fix-corrupted-ipod/



A lot of substances can be used to remove scratches from your electronic device. Some of them not so obvious in the way that they actually work (solvents and abrasives are the most common techniques that are used).

http://www.macworld.com/article/1046291/scratchremove.html

http://www.instructables.com/id/Buffing-Your-Ipod/

Paul Wayper: Labor on refugees

Sat, 2015-07-11 15:26
Sorry, technical folk, this is going to be a political blog post.

I recently got an email from my local member, Andrew Leigh, that raised an issue I feel passionately about; here is my response.

On 09/07/15 14:55, Andrew Leigh wrote:[snip] > > ▪ Some people have asked me *why Labor supported the government’s bill to > continue regional processing*. This is a tough question, on which reasonable > people can disagree, but the best answer to this is to read Bill Shorten’s > speech to the House of Representatives > > on the day the legislation was introduced. Hi Andrew,

I'm sorry, but I cannot agree with the logic Bill Shorten and the Labor party has expressed in that speech.

Firstly, anyone watching the international problems with refugees will realise that Australia's intake is pitiful and stingy compared to some of its key allies and comparable nations and especially when compared to its population size and lifestyle. It is hypocritical to say "we don't want people to risk journeying across the sea from Indonesia, but we're happy for them to remain illegal immigrants there", especially when you look at the life that those people face as refugees there.

As an aside, though, I would say that it is still partly correct - it is more humane for them to remain in Indonesia than to be detained indefinitely in the inhuman, underresourced and tortuous conditions on Manus Island and Nauru. It is shameful to me that the Labor party can ignore this obvious contradiction.

But more importantly, the logic that we're somehow denying "people smugglers a product to sell" by pushing boats back into international waters shows no understanding of people smuggling as a business. Australia is still very much a destination, it's just that people now come with visas on planes and they pay even more for this than they used to. There is still a thriving trade in getting people into Australia, it's just been made more expensive - in the same way that making heroin illegal has not caused it to suddenly vanish from the face of the earth.

All we're doing by punishing people who come by boat to seek refuge in Australia is punishing the very desperate, the worst off, the people who have literally fled with their clothes and nothing else.

Other people with money still arrive, overstay their visas, get jobs as illegal immigrants or on tourism visas. The ABC has exposed some of these ridiculous, unethical companies trading on foreign tourists and grey market labourers. The Labor party, of all parties, should be standing up for these people's rights yet it seems remarkably silent on this issue.

The point that I think Labor needs to learn and the point I ask you to express to your colleagues there is that we don't want Labor to return to its policies in 2010. We thought those were inhuman and unjust then, and we still do now. Invoking them as a justification for supporting the Government now is bad.

Personally, I want Labor to do three things with regard to refugees:

  1. Move back to on-shore detention and processing. The current system is vastly more expensive than it needs to be, and makes it more difficult for UN officials and our own members of parliament and judiciary to be able to examine the conditions of detention. The Coalition keeps telling everyone about how expensive their budget is but seems remarkably silent on why we're paying so much to keep refugees offshore.
  2. Provide better ways of settling refugees, such that one can cut the "people smuggler" middle men out of the deal.

    For example, set up refugee processing in places such as Sri Lanka and Afghanistan where many refugees come from. Set a fixed price per person for transportation and processing in Australia, such that it undercuts the people smugglers - according to figures I read in 2010 this could be $10,000 and still be 50% less than black market figures.

  3. Ensure accountability and transparency of the companies such as Serco that are running these centres. If the government was running them and people were being abused, the government would be held accountable; when private companies do this the government wipes its hands and doesn't do a thing.
And on a more conversational note, I'd be interested in your views on this as an economist. There is obviously an economy of people smuggling - do we understand it? Is there any economic justification for offshore detention? All markets must work with a certain amount of illegal activity - can we work _with_ the black market rather than trying to work against it?

I do appreciate your updates and information and I look forward to more of your podcasts.

All the best,

Paul

Simon Lyall: Gather 2015 – Afternoon Sessions

Sat, 2015-07-11 13:28

Panel: “How we work” featuring Lance Wiggs, Dale Clareburt, Robyn Kamira, Amie Holman – Moderated by Nat Torkington

  • Flipside of Startups given by Nat
  • Amie – UX and Services Designer for the Govt, thinks her job is pretty cool. Puts services online.
  • Lance – Works for NZTE better by capital programme. Also runs an early stage fund. Multiple starts and fails
  • Dale – Founded of Weirdly. Worked her way up to top of recruitment company (small to big). Decided to found something for herself.
  • Robyn – Started business 25 years ago. IT consultant, musician, writer.
  • Nat – Look at what you are getting from the new job. Transition to new phase in life. Want ot be positive.
  • Types of jobs: Working for someone else, work for yourself, hire other people, investor. Each has own perks, rewards and downsides.
  • Self employed
    • Big risk around income, peaks and troughs. Robyn always lived at the bottom of the trough level of income. Some people have big fear where next job is coming from.
    • Robyn – Charged Govt as much as possible. Later on charged just below what the really big boys charged. Also has lower rates for community orgs. Sniffed around to find out the rates. Sometimes asked the client. Often RFPs don’t explicityly say so you have to ask.
    • Pricing – You should be embarrassed about how much you charge for services.
    • Robyn – Self promotion is really hard. Found that contracts came out of Wellington. Book meetings in cafes back to back. Chat to people, don’t sell directly.
  • Working for others
    • Amie – Working in a new area of government. But it an area that is growing. Fairly permissive area, lots of gaps that they can fill.
    • Dale – Great experience as an employee. In environment with lot of autonomy in a fast growing company.
    • Lance – Worked from Mobile – Lots of training courses, overseas 6 months after hired. 4 years 4 different cities, steep learning curve, subsidized housing etc. “Learning curve stopped after 4 years and then I left”.
    • Big companies downside: Multiple stakeholders, Lots of rules
    • Big company upside: Can do startup on the side, eg a Family . Secure income. Get to play with big money and big toys.
  • Startup
    • Everything on steroids
    • Really exciting
    • Starting all parts of a company at once
    • Responsibility for business and people in it
    • Crazy ups and downs. Brutal emotional roller-coaster
    • Lance lists 5 businesses off the top of his head that failed that he was at. 3 of which he was the founder
    • Worst that can happen is that you can lose your house
    • Is this life for everyone? – Dale “yes it can be, need to go in with your eyes open”.  “Starting a business can be for everyone. I’m the poorest I’ve ever been now but I’m the happiest I’ve ever been”
    • At a startup you are not working for yourself, you are working for everybody else. Dale says she trys to avoid that.
    • Robyn – “If you life is gone when you are in a business then you are doing it wrong.”
    • If you are working from home you can get isolated, get some peer support and have a drink, coffee with some others.
  • Robyn – Recomends “How to make friends and influence People”
  • Dale
    • Jobhunters – Look for companies 1st and specific job 2nd
    • Startup – Meet everyone that you know and ask their opinion on your pitch
    • Young People going to Uni – You have to get work experience, as a recruiter she looks at experience 1st and pure academic history second.
  • Lance
    • Balance between creating income, creating wealth, learning
    • Know what you are passionate about and good at
    • It is part of our jobs to support everyone around us. Promote other people
  • Amie
    • Find the thing that is your passion
    • When you are deliverying your passion then you are delivering sometime relevant

 Pick and Mix

  • Random Flag generator – @polemic
    • See Wikipedia page for parts of a flag
    • 3 hex numbers are palet
    • 4 numbers represent the pattern
    • Next number will be the location
    • next number which color will be assigned
    • Last number will be a tweak number
    • Up to 8 or 9 of the above
    • Took python pyevolve and run evolution on them.
  • Alex @4lexNZ , @overtime
    • E-sports corporate gaming league
    • untested in NZ
    • Someone suggested cold calling CEOs or writing them letter
  • Simon @slyall (yes me)
    • Low volume site for announcements
  •  Mutate testing
    • Tweak test values of code, to reverse fuzzing
  • Landway learning  – @kiwimrdee
    • Looking for computers to borrow for class
    • They teach lots of stuff
  • Poetry for computers – @kiwimrdee
    • Hire somebody english/arts background who understand language rather than somebody from a CS background who understand machines
  • the.dosprompt.com
    • Lossless image compression for the web
    • Tools vary across the platform
  • Glen – Make computers learn to play Starcraft 1
    • Takes replays of humans playing starcraft
    • Getting computer to learn to play from that DB
    • It is struggling
  • Emergent political structures in tabletops games

Never check in a bag – How to pack

  • 48 hour bag
    • Laptop and power
    • Always – Zip up pouch, tissues , hand sanitizer, universal phone charger, breath mints, the littlest power plug (check will work in multiple voltages), Food bar, chocolate.
    • If more than 48 hours – notebook, miso soup, headphones, pen, laptop charger, apple plugs ( See “world travel kit” on apple site)
    • Get smallest power plug that will charge your laptop
    • Bag 3 – Every video adapter in the world, universal power adapter, airport express.
    • TP-link battery powered wifi adapters
    • If going away just moves laptop etc to this bag
    • Packing Cell
      • Enough clothes to get me through 48 hours
      • 2 * rolled tshirts (ranger rolling)
      • 2 pairs of underwear
      • 2 pairs of socks
      • Toileties. Ziplock back that complies with TSA rules for gels etc.
      • Other toiletries in different bag
      • Rip off stuff from hotels, also Kmart and local stores.
      • Put toiletries ziplock near door to other bag so easy to get out for security.
      • Leave packing cell in Hotel when you go out
    • Learn to Ranger roll socks and shirts etc.
  • 6 weeks worth of stuff
    • In the US you can have huge carry-on
    • Packs 2 weeks worth of clothes
    • Minaal Bag (expensive but cool).
    • Schnozzel bag – Vacuum pack clothing bag
  • Airlines allow 1 carryon bag up to 7 kgs + 1 bag for other items (heavy stuff can go into that)
  • Pick multi-color packing sell so you can color-code them.
  • Elizabeth Holmes and Matilda Kahl and Steve Jobs all wear same stuff every day.
  • Wear Ballet Heals on the plane
  • Woman no more than 2 pairs of shoes every, One of which must be good for walking long distances
  • Always be charging

 Show us your stack

  • I was running this session so didn’t take any notes.
  • We had people from about 5 compnies give a quick overview of some stuff they are running.
  • A bit of chat beforehand
  • Next year if I do this I probably need to do 5 minutes time limits for everyone

Close from Rochelle

  • Thanks to Sponsors
  • Thanks to Panellists
  • Thanks to catering and volunteer teams
  • Will be back in 2016

 

Share

Simon Lyall: Gather 2015 – Morning Sessions

Sat, 2015-07-11 08:28

Today I’m at the Gather 2015 conference. This was originally “Barcamp Auckland” before they got their own brand and went off to do random stuff. This is about my 5th year or so here (I missed one or two).

Website is gathergather.co.nz . They do random stuff as well as the conference.

Welcome

  • Welcome and intro to conference history from Ludwig
  • Rochelle thanks the sponsors
  • Where to go for dinner, no smoking, watch out for non-lanyard people., fire alarms, etc
  • Quiet room etc

Lessions learnt from growing a small team from 5-15

  • Around 30 people. Run by Ben, works at sitehost, previously worked at Pitch
  • Really hard work. Takes  a lot of time and real effort to build a great team
  • Need dedicate time and resources to growing team, Need someone who is focussed on growing the team and keeping the current team working
  • Cringe when people say “HR” but you need some in the sort of role and early on.
    • At around 16 people and doesn’t have full HR person yet. Before FT have someone with scheduled time to focus on team or company culture. In ideal world that person might not be in a manager role but be a bit senior (so they hear what the lower level employees say.
  • Variety and inclusion are keep to happy team
    • Once you are at 10+ members team will be diverse so “one size fits all” won’t work anymore. Need to vary team activities, need to vary rewards. Even have team lunches at different places.
  • Hire for culture and fit
    • From the first person
    • Easier to teach someone skills than to be a good team member
    • Anecdote: Hired somebody who didn’t fit culture, was abrasive, good worker but lost productivity from others.
    • Give people a short term trial to see if they fit in.
  • You will need to change the way communicate as a team as it grows
    • A passing comment is not enough to keep everybody in the loop
    • Nobody wants to feel alienated
    • Maybe chat software, noticeboard, shared calendar.
  • Balance the team work the members do
    • Everybody needs to enjoy the work.
    • Give people interesting rewarding work, new tech, customer interaction
    • Share the no-fun stuff too. Even roster if you have to. Even if somebody volunteers don’t make them always do it.
  • Appreciate you team members
    • Praise them if they have put a lot of work into something
    • Praise them before you point out the problems
    • Listen to ideas no matter who they come from.
    • 5 Questions/Minutes rule
  • If someone is working not well, wonder if problem is elsewhere in their life. Maybe talk to them. Job of everyone in the team
  • Appreciate your teams work, reward them for it
  • Do what feels right for your team. What works for some teams might not work for all. No “one size fits all”
  • Building great teams isn’t science it is an art. Experiment a bit.
  • Taking the time to listen to 10 people instead of just 5 takes longer. Maybe this can be naturally taken on by others in the team, no just the “boss”.
  • Have a buddy for each new hire. But make sure the buddys don’t get overloaded my constantly doing this with each new hire.
  • Going from 10 to 100 ppl. They same thing doesn’t work at each company size.
  • The point where you can get everybody in a room till when you can’t. At that point you have multiple teams and tribalism.
  • If you have a project across multiple teams then try and put everybody in that project together in a room.
  • Have people go to each others standups
  • Hire people who can handle change
  • Problem if you you buy a small company, they small company may want to keep their culture.
  • Company that does welcome dinners not farewell dinners
  • Make sure people can get working when they arrive, have an email address etc, find out if they have preferences like nice keyboard.
  • Don’t hire when you are extremely busy that you can’t properly get them onboard (or you may pick the wrong person). Never hire impulsively. Hire ahead of time.
  • Don’t expect them to be fully productive straight away. Give them something small to start on, no too complicated, no to crazy dependant on your internal crazy systems. But make sure it is within their skill level in case they struggle.
  • Maybe summer student projects. Find good people without being stuck with someone. Give them a project that isn’t high enough priority for the FT people.
  • Create training material

 Writing for fun and profit

  • Run by Peter Ravlich
  • Scrivener – IDE for writing
  • Writing full time (with support from partner), currently doing 4 projects simitaniously
  • Less community for Fantasy Writers than for literary writers. Bias in NZ against Genre fiction
  • Community – SpecficNZ – For speculative fiction. SciFi con each year and have a stand at Armageddon each year. $30 per year
  • If you write roleplaying games look at selling via rpgnow.com
  • Remember if publishing with Amazon then remember to be non-exclusive
  • For feature writing you need to know editors who like you and like your work.
  • “Just keep writing” , only way you’ll ever get better
  • Writing a weekly column:
    • The Best way: Write articles week ahead of time, edited by his wife, sent to the editor well in advance.
    • Leaving to last minute, not pre-editing quality varies, speakers column got dropped
  • Find the type of writing that you like and are good at.
  • Run everything past a reading group. “Am I on the right track?”
  • Treated writing as a jobs. Scheduled “Write for an hour, edit for 30 minutes, lunch, then repeat”. Make yourself.
  • Lots of sites that that push you to write a set number of words. Give you badges, pictures of kittens or punishment to keep you to a wordcount
  • Join a online writing group and post regular updates and get a bit of feedback
  • Daily Routines Blog or spinoff book for some ideas
  • Developmental editor or Structural editor
    • Developmental editor – Go to early, guidelines of what you should be doing, what direction you should be going. What is missing. Focused at plot level.
    • Structural Editor – Goes though line-by-line
  • Need to find editor who suits your style of writing, knows genre is important. Looks for those who have edited books/authors in your area.
  • Self editing – set aside novel, change font, new device, read though again. Change context so looking at it with new eyes.
  • Get contract with editor reviewed by Lawyer with experience in the industry (and on your side)
  • Most traditional publishers expect to see an edited novel
  • Talk to agents, query those who work with authors in similar areas to you.
  • Society of Authors
    • Have some legal experts, give you a reference
  • Kindle K-boards, a bit romance orientated but very good for technical stuff.
  • Go to poetry or reading/writing group. Get stuff out to other people. Once you have got it out to some, even just a small group then small jump to send it out to billions.
  • Have a stretegy on how to handle reviews, probably don’t engage with them.
  • Anne Friedman – Disapproval Matrix
  • You are your own toughest reviewer
  • Often people who went to journalism school, although not many actual journalists
  • Starling Literary Journal
  • Lists of Competitions and festivals in various places
  • Hackathon ( Step it up 2015 ) coming up, one group they want is for journalists who want to get more money into the job

The World of Vexillology – Flag Design

  • Dan Newman
  • flagdesign.nz + flagtest.nz
  • NZ Flag design cutoff this coming Thursday (the 16th of July)
  • People interesting in how the flag design originates, eg how Navel custom influences designs
  • 6000 odd submissions -> 60 shortlist -> 4 voted in referendum -> 1 vs current
  • 60 people at meeting in Wellington, less in other places.
  • Government Website
  • first time a country changed a flag by referendium not at the time of signifcant event (eg independence)
  • A lot of politicians are openly republican, but less push and thought in rest of population
  • Concern that silver flag looks like corporate logo
  • Easier to pretend you are an Australian and ask them “What would the NZ flag look like?” . Eg “Green Kangaroo on yellow” , “While silver fern or Kiwi on Black background”
  • Also lots of other countries use the Southern Cross
  • most countries the National team colors are close to that of the flag
  • Feeling if even flag changes now, then after “full independence” will change again
  • What will happen if Celebs come out if favour of a specific design
  • Different colours have different associations ( in different places )
  • All sorts of reasons why different colours are on a flag
  • The Silver fan looks like a fish to some
  • Needs to look good scaled down to emoji size

Bootstrapping your way to freedom

  • From Mark Zeman – Speedcurve
  • Previous gather sessions have been orientated toward VC and similar funding
  • There is an alternative where you self-fund
  • Design teacher – all students wanted to work on LOTR cause it was where all the publicity was.
  • Boostrapping – Doing it your way, self funded, self sustaining, usually smaller
  • Might take Capital later down the track
  • 3Bs seen as derogatory
  • Lots of podcasts, conferences and books etc
  • See Jason Cohen, many bits in present taken from him
  • The “ideal” bootstrapped business. Look at it from your own constraints
  • Low money, low time, self funded, try to create a cash machine
  • SAAS business lower end is very low. Very small amount per year
  • Low time if working on the side
  • Trying to get to maybe $10k/month to go fulltime
  • Reoccurring revenue. 150 customers at $66/month. Not many customers, not huge value product but has to be a reasonable amount.
  • Maybe not one-off product
  • Enterprise vs consumer space
  • Hard to get there with $0.99 one-offs in App store
  • Annual plans create cashflow
  • Option Boutique product. Be honest about who you are, how big you really are, don’t pretend to be a big company
  • B2B is a good space to be in. You can call 150 business and engage with them.
  • Not critical, Not real time (unless you want to be up at 3am)
  • Pick something that has “naturally reoccurring pain”. eg “Not a wedding planner” , probably multiple times per month
  • Aftermarkets. eg “Plugins for wordpress. Something small, put 20 hours into it, put it up for $5″. See also Xero, Salesforce.
  • Pick Big Markets, lots of potential customers
  • “Few NZ clients great for GST since I just get refunds”
  • Better By design. Existing apps mean there is already a market. Took an existing Open source product (webpagetest.org) and put a nice wrapper on it
  • Number of companies have published their numbers. Look at the early days of them while it took them to get to $10k/month (eg many took a year or two to get there).
  • Option to do consultancy on the side if you go “full time”. Cover the gap between you new business and your old wage. Had a 1 year contract that let him go half time on new biz but cover old expenses.
  • Don’t have false expectations on how quickly it will happen
  • Hard when it was a second job. Good because it was different from the day-job, but a lot of work.
  • Prototype and then validate. In most cases you should go the other way around.
  • If you want to talk to someone have something to offer. Have a pay it forward.
  • Big enterprises have people too. Connect to one guy inside and they can buy your product out of his monthly credit card bill.
  • Not everybody is doing all the cool techniques. Even if you are a “B” then you are ahead of a lot of the “C”s . eg creating sites with responsive design.
  • 1/3 each – Building Business, Building Audience, Building Product
  • Loves doing his GST etc
  • In his case he did did each in turn. Product , then Audience then Business
  • Have a goal. Do you want to be a CEO? Or just a little company?
  • His Success measures – Fun, time with kids, travel, money, flexability, learning, holidays, adventures, ideas, sharing
  • Resources: Startups for the Rest of us. A Smart Bear Blog, Amy Hoy – Unicorn Free, GrowthHacker TV, Microconf Videos

Share

Donna Benjamin: Landscape design: a great analogy for the web

Fri, 2015-07-10 17:27
Friday, July 10, 2015 - 17:16

I often find myself describing the digital domain to people who don't live and breathe it like I do. It's an intangible thing, and many of the concepts are coded in jargon. It doesn't help that every technology tool set uses it's own specific language, sometimes using the same words for very different things, or different words for the same things. What's a page? A widget? A layout? A template? A module, plugin or extension? It varies. The answer "depends".

Analogies can be a helpful communication tool to get the message across, and get everyone thinking in parallel.

One of my favourites, is to compare a web development project, to a landscape design project.

One of the first things you need to know, is who is this landscape for and what sort of landscape is it? The design required for a public park is very different to one suitable for the back courtyard of an inner city terrace house.

You also need to know what the maintenance resources will be. Will this be watered and tended daily? What about budget? Can we afford established plants, or should we plan to watch the garden grow from seeds or seedlings?

The key point of comparison, is that a garden, whether big or small, is a living thing. It will change, it will grow. It may die from neglect. It may become an un-manageable jungle without regular pruning and maintenance.

What analogies do you use to talk about digital design and development?

Image: XIIIfromTOKYO - Plan of the gardens of Versailles - Wikipedia - CC-BY-SA 3.0

Simon Lyall: NetHui 2015 – Friday afternoon

Fri, 2015-07-10 14:28

Safety and security in SMEs

  • Biggest challenge for one SME IT person very bad password practises
  • PABX issues, default passwords on voicemail resulting in calls getting forwarded overseas, racking up a big bill
    • Disable countries you don’t need
    • Credit Limits on your account
    • Good firewall practice
    • Good pin/password practice
  • SMEs wanted problem to go away since they had a business to run.
  • No standards for IT in small business, everywhere is setup different
  • 9 times out of 10 IT stifles business and makes things worse.
  • Small businesses recognise value, don’t want to spend on stuff that doesn’t return value
  • So many attack directions very hard to secure.
  • If you let other people using your business devices its a huge risk. Do you let your kids play with your work phone/laptop?
  • Biometrics don’t seem to be there yet
  • Maybe cloud-based software is a solution.

Disaster recovery

  • Pictures of before/after of satellite downlink and comms centre in Vanuatu after Cyclone
  • Cellular network survived, Datacentre survived, Fibre network survived
  • One month after disaster 80% of comms were restored
  • NZ team just sent over material via Govt CIO
  • Various other groups on the ground
  • Lots of other people doing stuff. Some were uncoordinated with main efforts
  • NZ people (Dean, Andy) Spent 90% of time on logistics and 10% of time on IT stuff
  • Vanuatu people very busy. eg offshore people had own mailing list to discuss things and then filter them through to people on the ground
  • Lots of offers from people.
  • Plan not in place in Vanuata, they now have one though
  • What people wanted was Generators and Satellite phones. Both of them are hard to ship via air due to Petrol/Lithium.
  • Very hard for non-regular (not the top 5 NGOs) to get access to shipping in military planes etc
  • Echo from people who had similar problems in Christchurch working with the regular agencies
  • Guy from vodafone said their company (globally) has a cellphone site that can be split between normal plane luggage
  • Twitter accounts for Wellington suburbs had a meeting with council
  • Some community outreach from the councils to coordinate with others. community resilience. Paying for street BBQs etc. “Neighbours day”
  • Vital infrastructure needs to have capacity in disaster.
  • Orgs need to have plans in place beforehand
  • Good co-operation between telcos in Christchurch Earthquake
  • Mobile app for 111 currently being looked at
  • Some parts of the privacy act can be loosened when disasters are declared to enable information sharing with agencies
  • Options for UPS on UFB “modems”

Panel: Digital inclusion – Internet for everybody

  • Panelists: Vanisa Dhiru (2020 Communications Trust), Bob Hinden (Internet Society), Professor Charles Crother (Auckland University of Technology), Robyn Kamira (Mitimiti on the Grid Project).
  • Charles
    • Cure-all quick technical fix
    • attitude to non-users
    • Recognise the dark-side of the Internet
    • What sorts of uses do we want to see?
    • Facilitating active vs passive users
    • Various stats on users. At around 80%. Elderly catching up with other groups
  • Vanisa
    • Digital inclusion projects, best know is “computers in homes”, In 1500 homes per year
    • Does digital disadvantaged just mean poor or other groups?
  • Tim
    • Network for Learning ( N4L)
    • Connecting up schools to managed network, many over RBI
    • School gets: router with firewall, some services on top of that
    • Means teachers don’t have to worry about the technical issues
    • map.n4l.co.nz is website with map of connected schools
    • Target 90% by end of the year. Getting down to smaller and more remote schools
    • Not just about having fibre connections and handing out tablets to every student
    • Raspberry Pi at each site they can to remote in to test network
  • Robyn
    • Issues 10 years ago about theft of data and concepts
    • Today we still see instances where models will have [Maori Chin Tattoo] and similar
    • Wellbeeding – health, education
    • Cultural preservation: creation too, not a museum piece
    • Economic development: how to we participate in dev of NZ
    • Mitimiti on the Grid. Very small school in Hokianga Harbour
  • I was tweeting a bit too much rather than typing here.
  • What is “inclusion”
  • Where the leadership be coming from
    • “everyone” . We live in a country small enough for everyone to do that.

 

General NetHui Feedback, some minor nagatives..

  • Need filtering of questions. Too many in all sessions turned into long statements. Val Aurora outlines a good method to prevent this.
  • I went to the “Quiet Room” once and there were people holding a noisy conversation
  • Heard there was a bit of an agressive questionare during the e-Mental health session

Share

Simon Lyall: NetHui 2015 – Friday Morning

Fri, 2015-07-10 08:28

Panel: Adapt or die? News media, new media, transmedia

  • Panelists: Megan Whelan (Radio New Zealand), Alex Lee (Documentary Edge Festival), Walid Al-Saqaf (Internet Society), Tim Watkin (Executive Producer of The Nation and blogger), Carrie Stoddart-Smith (blogger).
  • Panel moderator: Paul Brislen.
  • Intro Megan
    • Been at Radio NZ for 10 years. Website back then just frequencies and fax number
    • Good at Radio, not doing Internet very well
    • New Job as community engagement editor
    • Internet completely changed how the job is done.
    • Sacrifice accuracy and context sometimes to get the story out fast
    • Because people can now get their first and publish. They are no longer the gatekeepers of information. Getting used to others knowing more thna we do
  • Alex
    • Sees himself as creative entrepreneur
    • Content a few years ago seeing documentaries play in the cinema
    • Storytelling being distributed. Communities already telling their own stories.
    • 2 types of people in Audience. Skimmers and people wanting to do a deep dive
    • Story tellers only know who to tell the story, sometimes not so much on the technology
    • Developing collaboration between technologists and creatives together
  • Carrie
    • Blogging and social media provided new spaces for stories
    • Maori TV. Maori people in the “Ngati blogosphere”
    • Telling our own stories not just having others telling them
    • Media still highlights negative rather than positive stories about Maori
    • “Social media & blogging and facilitate stories and getting to know each other online”
    • But Internet allows Maori to bypass media to get positive stories out to to National/International audience
  • Walid
    • Internet should be empowering tool
    • Problem with Internet are people on it not the Internet themselves
    • Characteristics of traditionalist media is that there is a gatekeeper
    • New media is that everybody is responsible for their own actions
    • 60% of what is on social media is fake.
  • Tim
    • Every newsroom in NZ is running digital-first
    • No sustainable profit model for media orgs online
    • Digital tools give media a lot more tools and ability to create to stories
    • Speed comes a loss of quality, loss of subeditors
    • Internet has sucks a lot of money out of journalism (especially loss of classifieds)
    • Nostalgia has forgotten how bad journalism has used to be.
    • So much pressure on resources but less money
    • Example of real-time fact checking during interviews
  • Question for Alex.
    • You want people to Interaction with docus, but past has shown people don’t really?
    • Alex says that people have in the past
    • Refers to national Film board of Canada websites and interaction with their documentaries
    • These days all need docs are required by broadcasters and funders to have interaction and social media strategy
  • Mixing of Advertising and journalism undermine content?
    • A bit but it is a source of money that helps keep the rest afloat.
  • Is mainstream media actually verified compared to Social media
    • Yes it is
    • Use varified accounts on twitter to at least ensure the person is real
  • Opinion on tools such as “data miner” which takes news across internet and aggregates it?
    • Newsrooms have a lot of expertise
    • But less now as newsrooms get hallowed out
    • 8 feature editors at NZ Herald 10 years ago. Just 1.5 now
  • People can some fact-check journalism instantly
    • Good in one way
    • But diversity of knowledge means fact checking harder
  • What the Economic side of this? Where do you see economic support for high-quality contact coming from?
    • Sugar Daddy. Eg Washington Post supported by Jeff Bezos
    • Some kind of paywall seems be an main option
  • Responsibility to highlight stories and come back to old/ongoing stories
    • Yes they are revisited by media
  • How far though a digital day could somebody go and only experience Maori?
    • Some people only tweet in Maori
    • Work at places where people primary work in Maori
  • If money is tight and media companies consolidate does media have the room to push against the “powers that be”
    • Pretty much has always been the case
    • Getting harder but not astronomically harder than it used to be.

NZ culture online

  • Facilitators: Amber Craig, Bronwyn Holloway-Smith, Dan Shannan
  • Amber
    • How does NZ tell our story online with youtube etc
    • How to compete with other countries
  • Dan
    • Documentary NZ Trust
    • Looking into content and presentation of content
    • Lots of new platforms. Hard to negotiate with each of them
    • Have to reach people outside the main centres and people not able to attend festival events
    • Funding bodies want to be about NZ only. Won’t fund NZers telling stories about other places on non-NZ stuff. Told for Nz point of view
  • Brownwyn
    • Artist
    • People using various media

I switched to a different session after 10 minutes

Slowing the fast lane – Net Neutrality

  • Theresa legacy pervades in NZ Internet “Marketing by Confusion”
  • Incumbents offer inferior products to smaller ISP and exploit consumer ignorance
  • Almost all NZ ISPs do “not net neutral” stuff to save costs and improve user experience. eg they cache Google.
  • Netflix effect driving up demand across network ( 40% growth in 3 months) . Need to find a way of pricing that. Why shouldn’t we look at options to manage the network and push pricing signals back up the line
  • Why does Spark not have Netflix caches? Why does Spark not peer? Spark guy refuses to answer.
  • Spark gets away since it is the legacy incumbent Telco, the default ISP to get away with a lot of stuff.
  • Spark expect at manipulating the outcome
  • UFB levels the playing field. Market Failure will come from small ISPs not getting the scale to compete.
  • Datacaps now high enough that Zero-rating are now longer a thing. However packet prioritisation is still a thing that ISPs can hang over providers.
  • Alternative IXs being created to dis-satisfaction regarding port costs at the current ISPs
  • Prediction is that if NZ goes down net neutrality path it will fail, cause it won’t moderate unfair use of market power. Legislation will be narrow and based on the technology of the day, will be left behind too quickly.
  • Vote with feet away from Spark and Vodafone. Why they have the share they will keep abusing it.
  • Spark customer: Spark works okay, don’t care about the random polities, they work okay.
  • Peering can go via the Telecom Commissioner, doesn’t need politician
  • The Peering policy is damaging the NZ Internet.
  • ISPs that do not peer are less robust than those who do (especially in emergencies)

Copyright on the Internet

  • Facilitators: Hadyn Green, Matthew Jackson, Trish Hepworth
  • Trish
    • The Internet makes it easy to copy things and violate copyright
    • From the Internet point of view copying is the core function, resptricting that directly impacts the internet
    • Website blocking – Providers get sites blocked by copyright holders. Collatoral damage problem, other sites and other services
    • Track people and sending warning notices. To what extent should monitoring be allowed “just” to prevent copyright infringment
    • Should technologies like VPNs be allowed if they are copyright-circumvention applications
    • Should TPM/DRM be put on everything?
    • Can you copyright an API? Can you restrict people for using it everywhere. How important is interoperability without explicit permission
    • Copyright/Patents over software
    • How do you regulate a digital single market across multiple countries and multiple jurisdictions and cultures?
    • Should data be predicted by copyright? text mining allowed?
  • Paula Browning
    • From the “We create” , creative sector
    • Instead of thinking copyright is broekn cause you cannot want GoT at 1pm. The Internet is a massive opportunity for NZ. Copyright is needed for the industry to make money.
    • Games industry will generate $500 million this year
  • Paddy Buckley – Quickflicks
    • Problem is the current licensing model of content. Licensed by territory
    • Challenge people to name 5 TV series you cannot watch via NZ services
    • Need to keep territory-specific licensing. Cause else services are not going to have any local focus
    • Also content creators make the rules and they want territory licensing
  • You gotta respect their rules cause it is their content.. Maybe not cause random rules sometimes don’t make sense anymore
  • 28% of people use VPNs or some other place changing technology
  • Should we take things to count to get judicial clarity
  • Copyright had originally to do with regulation of printing press (eg “technology” ) not the regulation of content.
  • Copyright originally and up to now always orientation towards large-many distribution. New systems are many-many that involves “copying” for all interactions.
  • Why are VPNs a problem since money is still going to creators? – That is not how the model works, you have to pay for something first before creating it. The selling to different territories is how stuff is funded. Future business models are still not developed enough for everyone.
  • Worry about the amount of effort that goes into enforcing the current business models rather than looking at new ones. Especially what is this happening in NZ which is not solved by the current model.
  • TPP criminalise breaking DRM even when you legally have access to the resulting content
  • Vertical integration like with Disney allows resell of content across songs, TV, parks, re-releases, e over a period of 50+ years
  • Regional Licensing allows a single provider to pay for something and allow the content creator to get a lump-sum of money. They local provider can also prote the show locally.
  • Content providers already do the sums of global vs regional deals
  • Physical good already restricted to different places via exclusive import agreements.
  • Same with software, Example of NZ version of software was 10x more expensive and 2 versions behind.
  • “It seems clumsy attempts to secure copyright are still driving users to piracy. Cost and complexity are not being addressed”
  • Copyright laws should not be written solely by the content industry since they will solely reflect the interest of that industry.
  • Is there are shortage of Music, Movies, TV shows right now? Is copyright rally killing the industry.
  • Bollywood movie industry often legitimately released on Youtube after a few years of standard release.
  • Only reason that publishers have control of copyright (of books) is market failure. You have to go via intermediaries because you can’t do it yourself.
  • VPNs have lots of other uses beyond copyright evasion. Shouldn’t be banned just to prevent that.
  • Getting rid of “Work for hire” would seem to be a problem when something like a film has 5,000 people worked on a Movie.
  • Suggestion that there are should be a license like APRA to allow people to download what they want for a fixed license each year.
  • Need to sort out the problem with lengthening copyright period and orphan works.

 

Share

Maxim Zakharov: dpsearch-4.54-2015-07-06

Fri, 2015-07-10 08:25

A new snapshot version of DataparkSearch Engine has been released. You can get it on Google Drive.

Here is the list of changes since previous snapshot:

  • Crossword section is now includes value of TITLE attribute of IMG tag and values of ALT and TITLE attributes of A and LINK tags found on documents pointing to the indexing document
  • Meta PROPERTY is now indexing
  • URL info data is now stored for all documents with HTTP status code < 400
  • configure is now understands --without-libextractor switch to build dpsearch without libextractor support even it has been installed
  • robots.txt support is enabled for sites crawling using HTTPS scheme
  • AuthPing command has been added to send authorisation request before getting documents from a web-site. See details below.
  • Cookie command has been added.
  • Add support for SOCKS5 proxy without authorisation and with username authorisation. See details below.
  • A number of minor fixes

AuthPing command

Some web-sites may serve different content to a logged in user. In most cases logging in process consists of sending a POST or GET HTTP request to a specific URL before you start to receive targeted content. You may use AuthPing command to send such authentication request before requesting any document from the web-site.

E.g.:

AuthPing "POST https://commercial-site.ext.au/user/login.php u=bot%40user.ext.au&p=super%40pass"

This command specify a POST request to be send to the URL address https://commercial-site.ext.au/user/login.php with the following CGI loading: u=bot%40user.ext.au&p=super%40pass

AuthPing command should be specified before each Server/Realm/Subnet command it affects. And specified request is sent each time an indexing thread access a web-server for the first time in a run session.

Using SOCKS5 proxy

Proxy command is now accepting proxy type option with value either http either socks5. If you need to use username authentication with SOCKS5 proxy please use ProxyAuthBasic command to specify username and password.

E.g.:

Proxy socks5 localhost:9050

In this example a SOCKS5 proxy connection to local Tor system is specified which uses no authentication method for connection.

Binh Nguyen: Ableton and Ableton Push Hacking

Thu, 2015-07-09 23:47
For those who have been tracking this blog, it's been obvious that I've recently been spending more and more time with the Ableton Push.



If you don't know what this is please see the following...

https://www.ableton.com/en/help/learn-push/

http://blog.dubspot.com/learned-30-days-ableton-push/

http://sonicbloom.net/en/two-weekends-with-ableton-push-part-1-review/

http://emilehoogenhout.com/reviews/ableton-push-an-exploration-emile-hoogenhout/

https://www.youtube.com/results?search_query=Ableton+Live+9+%26+Push+Masterclass+Webinar  

Easy to miss Push features

https://forum.ableton.com/viewtopic.php?f=55&t=196560

Helpful Push information.

https://forum.ableton.com/viewtopic.php?f=55&t=215744



Jeremy Ellis meets Ableton Push

https://www.youtube.com/watch?v=EdCgA54g-aM

Mad Zach Push Performance Walkthrough

https://www.youtube.com/watch?v=iSZm4ECbE5k

Decap Push Performance

https://www.youtube.com/watch?v=ZOtj5WtkR1Q



It's basically an advanced, modern musical instrument/MIDI controller.



There have been others who have attempted to de-compile and extend/modify the behaviour of the device but while information and the extensions that have been provided have been interesting and useful they have been somewhat limited.

https://github.com/gluon/AbletonLive9_RemoteScripts

http://julienbayle.net/ableton-live-9-midi-remote-scripts/

http://julienbayle.net/ableton-push/

ableton: just release the py midi remote scripts

https://forum.ableton.com/viewtopic.php?f=1&t=134338

Live 9 MIDI Remote Scripts revealed...

https://forum.ableton.com/viewtopic.php?f=1&t=190069

http://livecontrol.q3f.org/ableton-liveapi/articles/introduction-to-the-framework-classes/



I'm beginning to understand why. The following link provides an update to automatically generated documentation (via epydoc) of decompiled Ableton Remote Script code (my scripts for decompilation and automated documentation have been included in the package). https://sites.google.com/site/dtbnguyen/Decompiled-Documentation-Ableton-MIDI-Remote-Scripts.zip

https://wiki.python.org/moin/DocumentationTools

http://stackoverflow.com/questions/1125970/python-documentation-generator

http://sphinx-doc.org/

https://packages.debian.org/search?keywords=python-sphinx

https://en.wikipedia.org/wiki/Comparison_of_documentation_generators

https://packages.debian.org/stable/python/python-epydoc

https://packages.debian.org/wheezy/python-epydoc



If you want to make any additional modifications of behaviour you'll need to be aware of the following:

- you'll need to catch up on your Python coding

http://stackoverflow.com/questions/36901/what-does-double-star-and-star-do-for-python-parameters

http://stackoverflow.com/questions/400739/what-does-asterisk-mean-in-python

https://en.wikipedia.org/wiki/Pickle_%28Python%29

http://stackoverflow.com/questions/514371/whats-the-bad-magic-number-error

http://stackoverflow.com/questions/12233837/bad-magic-number-while-trying-to-import-pyc-module

http://stackoverflow.com/questions/6034621/bad-magic-number-error-persists-even-after-rebuilding-the-pyc-file

https://shankaraman.wordpress.com/tag/how-to-fix-runtimeerror-bad-magic-number-in-pyc-file/

http://www.tutorialspoint.com/python/python_exceptions.htm

http://www.linuxtopia.org/online_books/programming_books/python_programming/python_ch17s03.html

- you'll need knowledge of how the device works, music and mathematical theory, Ableton, and core computing knowledge. It is not sufficient to know how they work seperately. You need to know how everything fits together.

- sounds obvious but start small and move up. This is critical particularly with reference to the awkward style of programming that they can sometimes resort to. More on this to below

- the code can vary in quality and style quite significantly at times. At times it seems incredibly clean, elegant, and well documented. At other times, it there is no documentation at all and doesn't seem to be well designed or engineered or have keep maintenance in mind. For instance, a commonly used design pattern is MVC. This doesn't seem to follow that. They use a heap of sentinels throughout there code. Moreover, the characters that are used can be a bit confusing. They don't use preprocesser directives/constants where they may be better suited. If you break certain aspects of the code you can end up breaking a whole lot of other parts. This may be deliberate (to reduce the chances of third party modification which is likely particularly as there seems to be some authentication/handshake mechasnisms in the code to stop it from working with 'uncertified devices') or not (they lack resources or just have difficult timelines to deal with)

https://mail.python.org/pipermail/tutor/2003-July/024206.html

http://stackoverflow.com/questions/2682745/creating-constant-in-python

- be prepared to read through a lot of code just to understand/make a change to something very small. As stated previously strictly speaking at times they don't adhere to good practice. That said, other aspects can be changed extremely easily without breaking other components

- due to the previous two poits it should seem obvious that it can be very difficult to debug things sometimes. Here's the other thing you should note,

- the reason why Ableton suffers from strange crashes and hangs from time to time becomes much more obvious when you look at the way they code. In the past, I've built programs (ones which rely on automated code generation in particular) that relied on a lot of consecutive steps that required proper completion/sequencing for things to work properly. When things work well, things are great. When things break, you feel and look incredibly silly

- you may need to figure out a structure for ensuring and maintaining a clean coding environment. I try to have two screens with one for clean code and another for modified code. Be prepared to restart from scratch by reverting to a clean pyc code and only one or a small number of modified py files.

- caching occurs in situations where you may not entirely expect. If you can not explain what is happening and suspect caching just restart the system. Better yet, maintain your development environment in a virtual machine to reduce hardware stress caused by continual restarts.

- you will need patience. As stated previously, due to the way code has been structured (sometimes) you'll need to understand it properly to allow you to make changes without breaking other parts. Be prepared to modify, delete, or add code just to help you understand it

- if you've ever dealt with firmware or embedded devices on a regular basis you would be entirely familiar with some of what I'm talking about. Like a lot of embedded devices you'll have limited feedback if something goes wrong and you'll be scratching your head with regards to how to work the problem

http://dtbnguyen.blogspot.com.au/2012/07/if-only-reading-were-easier.html

http://dtbnguyen.blogspot.com.au/2012/08/funky-firmware.html

You may require a lot of Linux/UNIX based tools and other debugging utilities such as IDA Pro, Process Explorer and Process Monitor from the Sysinternals Suite. Once you examine Ableton using such utilities, it becomes much clearer how the program has been structured, engineered, and designed. One thing that can cause mayhem in particular is the Ableton Indexer which when it kicks in at the wrong time can make it feel as though the entire system has frozen.

https://technet.microsoft.com/en-us/sysinternals/bb545021.aspx

https://en.wikipedia.org/wiki/Sysinternals

Ablton indexing Crashes

https://forum.ableton.com/viewtopic.php?f=1&t=206205

(42474187) Disable "Ableton Index" possible?

https://forum.ableton.com/viewtopic.php?f=52&t=191573

http://audiosex.pro/index.php?/topic/11798-ableton-9-killing-index-process-to-speed-up-workflow/

http://audiosex.pro/index.php?/topic/11815-a-little-app-to-speed-up-live-9-mac-only/

https://www.ableton.com/en/help/article/indexer-crash/

The actual index file/s are located at

C:\Users\[username]\AppData\Roaming\Ableton\Live 9.1.7\Database

- the most relevant log file is located at,

C:\Users\[username]\AppData\Roaming\Ableton\Live 9.1.7\Preferences\Log.txt

The timestamps works on the basis on amount of time since program startup. Time of startup is clearly outlined.

Ableton takes 130 - 2 mins to start up ?

https://forum.ableton.com/viewtopic.php?f=1&t=6308

Delete it if you need to if you get confused about how it works.

https://www.ableton.com/en/help/article/live-crash-packs/

https://www.ableton.com/en/help/article/live-crashing-what-should-i-do-now/

https://www.ableton.com/en/help/article/how-to-out-of-memory/

- be aware that there are some things that you can't do anything about. The original Novation Launchpad was considered somewhat sluggish in terms of refresh rate and latency. The electronics were subsequently updated in the Novation Launchpad S to deal with it. You may encounter similar circumstances here.

Push browser - slow, freezing, sluggish :(

https://forum.ableton.com/viewtopic.php?f=55&t=215539

- they have a strong utility/systems engineering mentality. A lot of files are archives which include relatively unobfuscated content. For instance, look in a

C:\Users\[username]\AppData\Roaming\Ableton\Live 9.1.7\Live Reports

and you'll find a lot of 'alp' files. These are 'Crash Reports' which are sent to Ableton to help debug problems. Rename them to a gz file extension and run it through hexedit. Same with 'adg' audio device group files. Rename to gz and gunzip to see a flat XML file containing some encoded information but mostly free/human readable content. It will be interesting to see how much of this can be manually altered achieving flexibility in programming without having to understand the underlying file format.

https://www.ableton.com/en/help/article/filetypes-used-by-ableton/

- each version of Ableton seems to have a small version of Python included. To make certain, advanced extensions others have suggested installing a libraries seperately or a different version of Python...

- be prepared to learn multiple protocols and languages in order to make the changes that you want

How to control the Push LCD text with sysex messages

https://forum.ableton.com/viewtopic.php?f=55&t=193744http://www.snoize.com/MIDIMonitor/http://archive.monome.org/community/discussion/1648/share-your-arduinomonome-code/p1

http://hackaday.com/2008/08/13/rgb-monome-clone/

https://en.wikipedia.org/wiki/Protofuse

http://julienbayle.net/works/creation/protodeck-midi-controller-for-ableton-live/



For a lot of people, the device seems incredibly expensive for what amounts to a MIDI controller. It was much the same with me. The difference is that it's becoming increasingly clear how flexible the device can be with adequate knowledge of the platform.

http://www.nativekontrol.com/

http://blog.dubspot.com/5-ableton-push-software-devices/Push feature requests

https://forum.ableton.com/viewtopic.php?f=55&t=192110

The Ableton Push is a good platform but it will never realise it's full potential if the software isn't upgraded.



If you are interested in signing up to test the latest Beta version of Ableton please see the following...

https://ableton.centercode.com/signup/

Simon Lyall: NetHui 2015 – Thursday Afternoon

Thu, 2015-07-09 14:28

Domains: growth, change, transition

  • Transition of .nz to second level domains
  • Some stuff re moving root zone control away from the US
  • Problem with non-ascii domains (IDNs). They work okay, but not 3rd party apps or apps in Organisations. Eg can’t register on Facebook or other websites.
  • 60% of Government Depts don’t accept IDNs as email addresses, lots of other orgs
  • 1/3 of all new .nz domains created at second level
  • Around 95k or 600k .nz domains now at second level (about 2/3s of these from rights are 3LD holder)
  • Some people when you give them your address.nz change it into address.co.nz
  • 1st principles of .nz whois public policy.
  • People are in danger if they address is published
  • But what the ability to contact the real owner of a domain
  • 4 people in room with signed domains
  • 300 signed .nz domains. 150 with DS record
  • Around 3 people in room with new TLDs. See ntldstats.com for current stats

Internet of Things

  • Where does the data from your house appliances go?
  • Forwarded to other companies
  • Issues need to be understandable by ordinary citizens especially terms and conditions
  • Choose the data that you choose to share with the company rather than company choosing what it shares with you (and others)
  • In health care area people worried about sharing data if it will affect their insurance premiums or coverage
  • Many people don’t understand what their data is, they don’t understand that if every time they do something (on a device) it is stored and can be used later. How to educate people without sounding paranoid?
  • “IoT is connecting things whose primary purpose is not connecting to the Internet”
  • “The cost of sharing is bearable, because the sharing is valuable.”
  • More granularities of trust. No current standards or experience or feeling for this since such a new area and rapidly evolving
  • NZ law should override overly aggressive agreements (by overseas companies)
  • Some discussion about standards, lots of them, full stack, piecemeal, rapidly changing
  • Will the IoT make everything useless after the zombie apocalypse?
  • “Denial of Service attack on your IoT pill bottle would be bad!”
  • Concern that something like a pill bottle failing can put life in danger. Very high level of reliability needed which is rare and hard in software

Panel: Parliamentary Internet Forum

  •  With Gareth Hughes (Green Party), Clare Curran (Labour Party), Brett Hudson (National Party), Ria Bond (NZ First), Karen Melhuish Spencer (Core Education), Nigel Robertson (University of Waikato)
  • What roles does the Education system play in the Internet
    • National guy mostly talked about UFB and RBI programmes, computers in homes
    • Gareth Hughes adopts the “I went out to XYZ School” story. Pushes Teachers not trained and 1 in 4 homes don’t have Internet access.
    • Claire – Got distracted about discussion re her pants. But she said 40% of jobs at risk over next 10-15 years due to impact of technology
    • Karen – I got distracted about another clothing related discussion on twitter
    • Nigel – 1. Use the Internet to do what we already do better. Help people to use the Internet better (digital literacy)
  • Lots of discussion about retraining older people to handle jobs in the future as their present jobs go away
  • How much should government be leading vs getting out of the way and just funding it?
    • Nigel – Government should provide direction. Different in tertiary and other sectors
    • Karen – Collaborative and connected but not mandating
  • “We need to prepare people not just for the jobs of the future, but also to create the companies of the future” – Martin Danner
  • Lots of other stuff but I got distracted.

Share

Colin Charles: #PerconaLive Amsterdam – schedule now out

Thu, 2015-07-09 13:25

The schedule is out for Percona Live Europe: Amsterdam (September 21-23 2015), and you can see it at: https://www.percona.com/live/europe-amsterdam-2015/program.

From MariaDB Corporation/Foundation, we have 1 tutorial: Best Practices for MySQL High Availability – Colin Charles (MariaDB)

And 5 talks:

  1. Using Docker for Fast and Easy Testing of MariaDB and MaxScale – Andrea Tosatto (Colt Engine s.r.l.) (I expect Maria Luisa is giving this talk together – she’s a wonderful colleague from Italy)
  2. Databases in the Hosted Cloud Colin Charles (MariaDB)
  3. Database Encryption on MariaDB 10.1 Jan Lindström (MariaDB Corporation), Sergei Golubchik (Monty Program Ab)
  4. Meet MariaDB 10.1 Colin Charles (MariaDB), Monty Widenius (MariaDB Foundation)
  5. Anatomy of a Proxy Server: MaxScale Internals Ivan Zoratti (ScaleDB Inc.)

OK, Ivan is from ScaleDB now, but he was the SkySQL Ab ex-CTO, and one of the primary architects behind MaxScale! We may have more talks as there are some TBD holes to be filled up, but the current schedule looks pretty amazing already.

What are you waiting for, register now!

Simon Lyall: NetHui 2015 – Thursday Morning

Thu, 2015-07-09 08:28

Ministerial address: Hon. Amy Adams, Minister for Communications

  • Mentions she was at community group meeting where people were “shocked” when it was suggested that minutes be sent via email
  • Talk up of the UFB rollout. Various stats about how it is going
  • Also mentioned that Mobile build is part of UFB, better cellular connectivity in rural regions
  • Notes that this will never be 100% complete. The bar keeps moving
  • Very different takeup in different regions. 2% in some 19% in others. Local organisations pushing
  • Good Internet is especially important for remote countries like New Zealand
  • Talk about getting better access in common areas (eg shared driveways) for network builds
  • Notes how Broadcasting and Communications as well as other areas are converging. Previously they were separate silos. Similar for other areas.
  • Harmful Digital Communications Act.
    • Says new framework, adjustment may be needed and bedding down the courts.
    • Says that majority of cases will go to mediation
    • Similar Act in Australia very few things going to courts
    • Gave similar silly literal readings of others acts ( RMA requires a permit to sneeze )
  • 5 “Questions” to minister. 2 on TPP, 1 on Captions, 1 pushing some project and one actual question that she got to answer.
  • Maybe they should look at this idea for the Questions

Keynote: Kathy Brown, ISOC CEO

  • GDP of a National is highly correlated with the growth of the Internet
  • 75% of the benefit of the Internet goes to existing businesses
  • ISOC Global Internet Report 2015
  • Huge growth in Mobile Internet
  • “94% of the global population is covered by mobile networks. Mobile broadband covers 48% of global population”
  • Huge gap between developed and developing counties
  • Report is Online and “Interactional”
  • Challenges
    • Openness of the Internet means information is out there, exposed and gettable by the wrong people sometimes
    • Generational divide in attitude to privacy
  • Privacy is a matter of personal choice. The tools should be available should you wish to use them

Govt 2.0: Digital by default

  • Rachel Prosser and David Farrar facilitating.
  • Room full
  • Result 10 programme background
  • NZ Government Web toolkit
  • 50,000 registered with NZ Realme site
  • Shared rules between local governments, problems with same rules everywhere. Some limitations,. Perhaps at least similar technical standards
  • People don’t care about governments structure, they just want a service, don’t care how depts are arranged.

Share

Simon Lyall: NetHui 2015 – InTac afternoon

Wed, 2015-07-08 13:28

Building an access network for demand and scale – new challengesKurt Rogers, Chorus

  • Over 1 million broadband connections on access network
  • 70-80% of BB connections
  • Average connection sped now near 20Mb/s due to VDSL and Fibre
  • Busiest 15 minute period (around 9pm Thursday) of week averaging 0.5Mb/s per user ( up from 100kb/s just 3 years ago )
  • Jump in mid-2013 when Netflix and Lightbox launched
  • Average bandwidth per user growing 50%/year. Grown that much in 1st half of 2015
  • Quite a few people still on ADSL1 modems when ADSL2 would work
  • Same a lot of people can get VDSL that don’t realize
  • Lots of people on 30Meg fibre plan at the start, now most going for 100Mb/s
  • Rural broadband (RBI)
    • 85k lines upgraded to FTTN
    • Average speed jumped 5.6Mb/s to 15Mb/s after a single rural cabinet upgraded cause everybody could now use ADSL2 and faster uplink. One fibre guy got 48Mb/s on VDSL, other 37Mb/s
    • More speed out there than some people realize
  • VDSL bandplan moving from 997 to 998. Trail average speed increases were from 32 to 46Mb/s for downstream. Minimal change on upstream speed.
  • Capacity
    • Aggregation link bandwidth. Alert threshold at 70%, Max threshold at 90%
  • Technology down the road to speed up aggregation links with Next Generation PON technology

The new smart ISPColin Brown, GM of Networks at Spark

  • Working on caching infrastructure, bigger and closer to their edge
  • Big traffic growth this year
  • Big growth in mobile traffic especially upload
  • 60% of phones in stores are 4G capable
  • Providers investing a lot of money , profits lower. Less like banks, more like airlines
  • Technology refresh every 5 years rather than every 10

Share

Rusty Russell: The Megatransaction: Why Does It Take 25 Seconds?

Wed, 2015-07-08 13:28

Last night f2pool mined a 1MB block containing a single 1MB transaction.  This scooped up some of the spam which has been going to various weakly-passworded “brainwallets”, gaining them 0.5569 bitcoins (on top of the normal 25 BTC subsidy).  You can see the megatransaction on blockchain.info.

It was widely reported to take about 25 seconds for bitcoin core to process this block: this is far worse than my “2 seconds per MB” result in my last post, which was considered a pretty bad case.  Let’s look at why.

How Signatures Are Verified

The algorithm to check a transaction input (of this form) looks like this:

  1. Strip the other inputs from the transaction.
  2. Replace the input script we’re checking with the script of the output it’s trying to spend.
  3. Hash the resulting transaction with SHA256, then hash the result with SHA256 again.
  4. Check the signature correctly signed that hash result.

Now, for a transaction with 5570 inputs, we have to do this 5570 times.  And the bitcoin core code does this by making a copy of the transaction each time, and using the marshalling code to hash that; it’s not a huge surprise that we end up spending 20 seconds on it.

How Fast Could Bitcoin Core Be If Optimized?

Once we strip the inputs, the result is only about 6k long; hashing 6k 5570 times takes about 265 milliseconds (on my modern i3 laptop).  We have to do some work to change the transaction each time, but we should end up under half a second without any major backflips.

Problem solved?  Not quite….

This Block Isn’t The Worst Case (For An Optimized Implementation)

As I said above, the amount we have to hash is about 6k; if a transaction has larger outputs, that number changes.  We can fit in fewer inputs though.  A simple simulation shows the worst case for 1MB transaction has 3300 inputs, and 406000 byte output(s): simply doing the hashing for input signatures takes about 10.9 seconds.  That’s only about two or three times faster than the bitcoind naive implementation.

This problem is far worse if blocks were 8MB: an 8MB transaction with 22,500 inputs and 3.95MB of outputs takes over 11 minutes to hash.  If you can mine one of those, you can keep competitors off your heels forever, and own the bitcoin network… Well, probably not.  But there’d be a lot of emergency patching, forking and screaming…

Short Term Steps

An optimized implementation in bitcoind is a good idea anyway, and there are three obvious paths:

  1. Optimize the signature hash path to avoid the copy, and hash in place as much as possible.
  2. Use the Intel and ARM optimized SHA256 routines, which increase SHA256 speed by about 80%.
  3. Parallelize the input checking for large numbers of inputs.
Longer Term Steps

A soft fork could introduce an OP_CHECKSIG2, which hashes the transaction in a different order.  In particular, it should hash the input script replacement at the end, so the “midstate” of the hash can be trivially reused.  This doesn’t entirely eliminate the problem, since the sighash flags can require other permutations of the transaction; these would have to be carefully explored (or only allowed with OP_CHECKSIG).

This soft fork could also place limits on how big an OP_CHECKSIG-using transaction could be.

Such a change will take a while: there are other things which would be nice to change for OP_CHECKSIG2, such as new sighash flags for the Lightning Network, and removing the silly DER encoding of signatures.

Stewart Smith: The sad state of MySQL and NUMA

Wed, 2015-07-08 12:27

Way back in 2010, MySQL Bug 57241 was filed, pointing out that the “swap insanity” problem was getting serious on x86 systems – with NUMA being more and more common back then.

The swapping problem is due to running out of memory on a NUMA node and having to swap things to other nodes (see Jeremy Cole‘s blog entry also from 2010 on the topic of swap insanity). This was back when 64GB and dual quad core CPUs was big – in the past five years big systems have gotten bigger.

Back then there were two things you could do to have your system be usable: 1) numa=off as kernel boot parameter (this likely has other implications though) and 2) “numactl –interleave all” in mysqld_safe script (I think MariaDB currently has this built in if you set an option but I don’t think MySQL does, otherwise perhaps the bug would have been closed).

Anyway, it’s now about 5 years since this bug was opened and even when there’s been a patch in the Twitter MySQL branch for a while (years?) and my Oracle Contributor Agreement signed patch attached to bug 72811 since May 2014 (over a year) we still haven’t seen any action.

My patch takes the approach of you want things allocated at server startup to be interleaved across nodes (e.g. buffer pool) while runtime allocations are probably per connection and are thus fine (in fact, better) to do node local allocations.

Without a patch like this, or without running mysqld with the right numactl incantation, you end up either having all your memory on one NUMA node (potentially not utilising full memory bandwidth of the hardware), or you end up with swap insanity, or you end up with some other not exactly what you’d expect situation.

While we could have MySQL be more NUMA aware and perhaps do a buffer pool instance per NUMA node or some such thing, it’s kind of disappointing that for dedicated database servers bought in the past 7+ years (according to one comment on one of the bugs) this crippling issue hasn’t been addressed upstream.

Just to make it even more annoying, on certain workloads you end up with a lot of mutex contention, which can end up meaning that binding MySQL to fewer NUMA nodes (memory and CPU) ends up increasing performance (cachelines don’t have as far to travel) – this is a different problem than swap insanity though, and one that is being addressed.

Update: My patch as part of https://bugs.mysql.com/bug.php?id=72811 has been merged! MySQL on NUMA machines just got a whole lot better. I just hope it’s enabled by default…

Simon Lyall: NetHui 2015 – InTac morning

Wed, 2015-07-08 09:28

IntroductionDean Pemberton, InternetNZ

Dean was going to do an intro but got cock-blocked by some guy in a High-Vis vest.

The People Factor: what users wantPaul Brislen, ex-CEO of TUANZ

  • Working from home since 1999, 30kb/s at first. Made it work
  • Currently has 10Mb/s shared with busy family, often congested, not using much TV yet
  • Television driving demand.
  • Some infrastructure showing the strain
  • Southern cross replacement will be via Sydney. A couple of thousand km in the wrong direction when going to the US
  • Rural broadband still to deliver on the promise, no uptake stats, not great service level
  • Internet access critical path for economic development. lack of political will
  • Dean got to do his intro talk now.
  • Will Internet be priced on peak usage? A: Already offpeak discounts, some ISPs manage home/biz customer ratio to keep traffic balanced
  • Average usage per customer is 5Mb/s for ISP with streaming orientated ISP (acct sold with device).
  • 60% of International traffic going to Aus (to CDNS)
  • Consumers don’t accept buffering, high quality video (bitrate and production quality). Want TV to just-work.
  • NZ doesn’t want to be a “rural” level of internet access, equiv to a farm in more connected countries
  • Could multicast work for live events like sport?
  • Hard to get overage to work to work when people leave TV on all day
  • Plenty of people in Auckland not getting UFB till 2017 (or later)

The connected home and the Internet of ThingsAmber Craig, ANZ

  • At top of Hype cycle
  • Has home Switches on Wemo (have to get upgraded)
  • Lots of devices generating a lot of data
  • Video Blogging – 10GB of raw data, 1GB of finished for just 5 minutes. Uploading to shared drives, sending back and forth through multiple edits
  • Network capacity if probably not much for IoT compared to video, but home will be a source of a lot more uploads
  • With IPv6 maybe less NAT, harder to manage (since people are not used to it).
  • Whose responsibility is it to ensure that Internet works in every room
  • Building standards, what are customers, government, ISP each prepared to pay for?
  • What about medical dependency people who need Internet. A lot of this goes over GSM since that is more “reliable”

Lightbox – content delivery in New ZealandKym Nyblock, Chief Executive of Lightbox

  • Lightbox is part of Spark ventures, morepork, skinny, bigpipe
  • Lighbox – On line TV service, $12.99/month thousands of hours of online content
  • 40% of US household have SVOD, but pay-TV only down 25%
  • Many providers around the world, multiple providers in many countries. Youtube also bit player in the corner
  • SVOD have some impact on piracy, especially those who only pirate cause they want content same day as programme airs in the US
  • Lots of screens now in the house, TV not only viewed on TVs
  • Lightbox challenges
    • Rights issues, lots of competition with other providers, some with fuzzy launch dates
    • NZ Internet not too bad
    • Had to work within an existing company
  • Existing providers
    • Sky – 850k homes, announced own product, has most sports
    • Netflix – approx 30k homes, coming to NZ soon
  • From Biz plan to launch in 12 months
  • Marketing job to be very simple – “Grandma Rule” ( can be explained to Grandma, used by her)
  • Express service delivers content right after views in the US. Lots of views for the episodes that are brand new. One new episode can be 10% of days total views
  • Very agile company, plans changed a lot.
  • Future
    • Customers will have several providers and change often
    • Multiple providers in the market, more to come
    • Premium and exclusive content will drive, simple interface will keep it
    • Rights issues are a problem but locked into the studio system
    • Try to “grow the category”, majority on consumers still using linear, scheduled TV
    • Try to address local rights ownership. This is the bit where they dug at US based providers and people using them.
    • Working on a Sports offering
    • and then she showed a Lightbox ad
    • Question costs of other ISPs of getting good lightbox due to charges from Spark-Wholesale for bandwidth exchanged. Not really answered

Quickflix – another view of content delivery in New ZealandPaddy Buckley, MD of Quickflix NZ

  • 1st service to launch in March 2012
  • Subscription service for movies and TV shows and Standalone pay-per-view service for new-release movies and some TV shows
  • Across lots of devices, Smart TVs, phones, computers, games consoles, tablets, tivo, chromecast. No Linux Client
  • Just 15% of views via the website now
  • Content: New release movies, subscriptions content movies, TV shows
  • Uses Akamai for delivery. Hosting Centers in Sydney and Perth. AWS/Azure
  • Unwritten 5 second rule. Content should play within 5 seconds of pressing play
  • The future
    • Multiple Models, Not just SVOD, eg TVOD, AVOD, EVOD, EST
    • More fibre, fast home wifi and better hardware
    • VOD content getting nearer to the viewer. HbbTV combines broadcast and on-demand being done by freeview
    • Android TV
    • Viewing levels to increase (volume and frequency), people will pick and mix between providers
    • Aiming at 50% of households, 1 million is quite a lots for any scale.
  • Coming soon
    • 1080p/4K , 5.1 surround sound
    • Fewer device limits. All services and all devices
    • More streams
    • Changing release windows
    • Live streaming
    • PPV options to compliment
    • Download now, view later
  • What we need from ISPs
    • Significant bandwidth
    • Mooorrreee bandwidth
    • People will change ISPs if the ISP can’t provide the level of service
    • Netflix is naming and shaming. Netflix best/worst list
  • Prediction that NZ could hit 50% SVOD within a couple of years
  • Asked if they will be going broke in next few months. Says he’s done deal with Presto in Aus and will ease funding problems but business as normal in the NZ
  • SVOD has evolved from back-catalog TV shows a few years ago to first-run now. Will probably keep going forward with individual shows being provider-exclusive for now, especially since services are fairly low cost per month
  • A few questions about subtitles. Usually available (although can cost extra) but not good support with end devices to turn on/off .

Share