Planet Linux Australia
Written by a Victoria Cross recipient, this is the true story of a messed up kid who made something of himself. Mark's dad died of cancer when he was young, and his mum was murdered. Mark then went through a period of being a burden on society, breaking windows for fun and generally being a pain in the butt. But then one day he decided to join the army...
This book is very well written, and super readable. I enjoyed it a lot, and I think its an important lesson about how troubled teenagers are sometimes that way because of pain in their past, and can often still end up being a valued contributor to society. I have been recommending this book to pretty much everyone I meet since I started reading it.
Tags for this post: book mark_donaldson combat sas army afghanistan
Related posts: Goat demoted Comment Recommend a book
Tags for this post: coding_club python game tkinter
Related posts: More coding club; Implementing SCP with paramiko; Coding club day one: a simple number guessing game in python; Packet capture in python; mbot: new hotness in Google Talk bots; Calculating a SSH host key with paramiko
You also might be interested to know that Yandex, a Russian google, has their Yandex Translation Service running, which in many cases gives better translation for Russian - English pair of languages.
“How’d you solve the icing problem?”
“Might want to look into it.”
— Iron Man (2008)
On a particularly cold Saturday morning a couple of years ago, my mobile phone couldn’t get any signal for a few hours. But I didn’t really care, because I had breakfast to eat, animals to feed, and nobody I urgently needed to talk to at the time. Also it came good again shortly after midday.
The following week the same thing happened, but for rather longer, i.e. well into the evening. This was enough to prompt me to use my landline to call Optus (our mobile phone provider) and report a fault. The usual dance ensued:
“Have you tried turning it off and on again?”
“Have you tried a different handset?”
“A different SIM?”
“Holding the phone in your other hand?”
“Sacrificing a lamb to the god Mercury?”
I might be misremembering the details of the conversation, but you get the idea. Long story short, I got a fault lodged.
Later I received a call – on my mobile – asking if my mobile was working again. “Indeed it is, and you wouldn’t have been able to make this call if it wasn’t”, I replied. Then I asked what the problem had been. “Let me check”, said the support person. “Uhm… It says here there was… 100mm of ice on the local tower.”
Flash forwards to a couple of days ago, when snow fell down to sea level for the first time since 2005, and my mobile phone was dead again. I can only assume they haven’t solved the icing problem, and that maybe the local NBN fixed wireless tower suffers from the same affliction, as that was dead too for something like 24 hours.
It was very pretty though.
from launchpadlib import launchpad
LP_INSTANCE = 'production'
CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache/')
def main(username, project):
lp = launchpad.Launchpad.login_with(username, LP_INSTANCE, CACHE_DIR)
for bug in lp.projects[project].searchTasks(status=["New",
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Fetch bugs from launchpad')
args = parser.parse_args()
Tags for this post: launchpad api
Related posts: Taking over a launch pad project; Juno nova mid-cycle meetup summary: the next generation Nova API
There’s a significant debate going on at the moment in the Bitcoin world; there’s a great deal of information and misinformation, and it’s hard to find a cogent summary in one place. This post is my attempt, though I already know that it will cause me even more trouble than that time I foolishly entitled a post “If you didn’t run code written by assholes, your machine wouldn’t boot”.The Technical Background: 1MB Block Limit
The bitcoin protocol is powered by miners, who gather transactions into blocks, producing a block every 10 minutes (but it varies a lot). They get a 25 bitcoin subsidy for this, plus whatever fees are paid by those transactions. This subsidy halves every 4 years: in about 12 months it will drop to 12.5.
Full nodes on the network check transactions and blocks, and relay them to others. There are also lightweight nodes which simply listen for transactions which affect them, and trust that blocks from miners are generally OK.
A normal transaction is 250 bytes, and there’s a hard-coded 1 megabyte limit on the block size. This limit was introduced years ago as a quick way of avoiding a miner flooding the young network, though the original code could only produce 200kb blocks, and the default reference code still defaults to a 750kb limit.
In the last few months there have been increasing runs of full blocks, causing backlogs for a few hours. More recently, someone deliberately flooded the network with normal-fee transactions for several days; any transactions paying less fees than those had to wait for hours to be processed.
There are 5 people who have commit access to the bitcoin reference implementation (aka. “bitcoin-core”), and they vary significantly in their concerns on the issue.The Bitcoin Users’ Perspective
From the bitcoin users perspective, blocks should be infinite, and fees zero or minimal. This is the basic position of respected (but non-bitcoin-core) developer Mike Hearn, and has support from bitcoin-core ex-lead Gavin Andresen. They work on the wallet and end-user side of bitcoin, and they see the issue as the most urgent. In an excellent post arguing why growth is so important, Mike raises the following points, which I’ve paraphrased:
- Currencies have network effects. A currency that has few users is simply not competitive with currencies that have many.
- A decentralised currency that the vast majority can’t use doesn’t change the amount of centralisation in the world. Most people will still end up using banks, with all the normal problems.
- Growth is a part of the social contract. It always has been.
- Businesses will only continue to invest in bitcoin and build infrastructure if they are assured that the market will grow significantly.
- Bitcoin needs users, lots of them, for its political survival. There are many people out there who would like to see digital cash disappear, or be regulated out of existence.
At this point, it’s worth mentioning another bitcoin-core developer: Jeff Garzik. He believes that the bitcoin userbase has been promised that transactions will continue to be almost free. When a request to change the default mining limit from 750kb to 1M was closed by the bitcoin lead developer Wladimir van der Laan as unimportant, Jeff saw this as a symbolic moment:
— Jeff Garzik (@jgarzik) July 21, 2015What Happens If We Don’t Increase Soon?
Mike Hearn has a fairly apocalyptic view of what would happen if blocks fill. That was certainly looking likely when the post was written, but due to episodes where the blocks were full for days, wallet designers are (finally) starting to estimate fees for timely processing (miners process larger fee transactions first). Some wallets and services didn’t even have a way to change the setting, leaving users stranded during high-volume events.
It now seems that the bursts of full blocks will arrive with increasing frequency; proposals are fairly mature now to allow users to post-increase fees if required, which (if all goes well) could make for a fairly smooth transition from the current “fees are tiny and optional” mode of operation to a “there will be a small fee”.
But even if this rosy scenario is true, this begsavoids the bigger question of how high fees can become before bitcoin becomes useless. 1c? 5c? 20c? $1?So What Are The Problems With Increasing The Blocksize?
In a word, the problem is miners. As mining has transitioned from a geek pastime, semi-hobbyist, then to large operations with cheap access to power, it has become more concentrated.
The only difference between bitcoin and previous cryptocurrencies is that instead of a centralized “broker” to ensure honesty, bitcoin uses an open competition of miners. Given bitcoin’s endurance, it’s fair to count this a vital property of bitcoin. Mining centralization is the long-term concern of another bitcoin-core developer (and my coworker at Blockstream), Gregory Maxwell.
Control over half the block-producing power and you control who can use bitcoin and cheat anyone not using a full node themselves. Control over 2/3, and you can force a rule change on the rest of the network by stalling it until enough people give in. Central control is also a single point to shut the network down; that lets others apply legal or extra-legal pressure to restrict the network.What Drives Centralization?
Bitcoin mining is more efficient at scale. That was to be expected. However, the concentration has come much faster than expected because of the invention of mining pools. These pools tell miners what to mine, in return for a small (or in some cases, zero) share of profits. It saves setup costs, they’re easy to use, and miners get more regular payouts. This has caused bitcoin to reel from one centralization crisis to another over the last few years; the decline in full nodes has been precipitous by some measures and continues to decline.
Consider the plight of a miner whose network is further away from most other miners. They find out about new blocks later, and their blocks get built on later. Both these effects cause them to create blocks which the network ignores, called orphans. Some orphans are the inevitable consequence of miners racing for the same prize, but the orphan problem is not symmetrical. Being well connected to the other miners helps, but there’s a second effect: if you discover the previous block, you’ve a head-start on the next one. This means a pool which has 20% of the hashing power doesn’t have to worry about delays at all 20% of the time.
If the orphan rate is very low (say, 0.1%), the effect can be ignored. But as it climbs, the pressure to join a pool (the largest pool) becomes economically irresistible, until only one pool remains.Larger Blocks Are Driving Up Orphan Rates
Large blocks take longer to propagate, increasing the rate of orphans. This has been happening as blocks increase. Blocks with no transactions at all are smallest, and so propagate fastest: they still get a 25 bitcoin subsidy, though they don’t help bitcoin users much.
Many people assumed that miners wouldn’t overly centralize, lest they cause a clear decentralization failure and drive the bitcoin price into the ground. That assumption has proven weak in the face of climbing orphan rates.
And miners have been behaving very badly. Mining pools orchestrate attacks on each other with surprising regularity; DDOS and block withholding attacks are both well documented. A large mining pool used their power to double spend and steal thousands of bitcoin from a gambling service. When it was noticed, they blamed a rogue employee. No money was returned, nor any legal action taken. It was hoped that miners would leave for another pool as they approached majority share, but that didn’t happen.
If large blocks can be used as a weapon by larger miners against small ones, it’s expected that they will be.
More recently (and quite by accident) it was discovered that over half the mining power aren’t verifying transactions in blocks they build upon. They did this in order to reduce orphans, and one large pool is still doing so. This is a problem because lightweight bitcoin clients work by assuming anything in the longest chain of blocks is good; this was how the original bitcoin paper anticipated that most users would interact with the system.The Third Side Of The Debate: Long Term Network Funding
Before I summarize, it’s worth mentioning the debate beyond the current debate: long term network support. The minting of new coins decreases with time; the plan of record (as suggested in the original paper) is that total transaction fees will rise to replace the current mining subsidy. The schedule of this is unknown and generally this transition has not happened: free transactions still work.
The block subsidy as I write this is about $7000. If nothing else changes, miners would want $3500 in fees in 12 months when the block subsidy halves, or about $2 per transaction. That won’t happen; miners will simply lose half their income. (Perhaps eventually they form a cartel to enforce a minimum fee, causing another centralization crisis? I don’t know.)
It’s natural for users to try to defer the transition as long as possible, and the practice in bitcoin-core has been to aggressively reduce the default fees as the bitcoin price rises. Core developers Gregory Maxwell and Pieter Wuille feel that signal was a mistake; that fees will have to rise eventually and users should not be lulled into thinking otherwise.
Mike Hearn in particular has been holding out the promise that it may not be necessary. On this he is not widely supported: that some users would offer to pay more so other users can continue to pay less.
It’s worth noting that some bitcoin businesses rely on the current very low fees and don’t want to change; I suspect this adds bitterness and vitriol to many online debates.Summary
The bitcoin-core developers who deal with users most feel that bitcoin needs to expand quickly or die, that letting fees emerge now will kill expansion, and that the infrastructure will improve over time if it has to.
Other bitcoin-core developers feel that bitcoin’s infrastructure is dangerously creaking, that fees need to emerge anyway, and that if there is a real emergency a blocksize change could be rolled out within a few weeks.
At least until this is resolved, don’t count on future bitcoin fees being insignificant, nor promise others that bitcoin has “free transactions”.
 http://www.coindesk.com/bitcoin-mining-pools-ddos-attacks/ “Bitcoin Mining Pools Targeted in Wave of DDOS Attacks” Coinbase 2015
 http://blog.bettercrypto.com/?p=1131 “Block Withholding Attacks – Recent Research” N T Courtois 2014
 https://bitcointalk.org/index.php?topic=327767.0 “GHash.IO and double-spending against BetCoin Dice” mmtech et. al 2013
 https://www.reddit.com/r/Bitcoin/comments/3c8tq2/questions_about_the_july_4th_bip66_fork/cstgajp “Questions about the July 4th BIP66 fork”
 https://twitter.com/petertoddbtc/status/608475414449778688 “350,000 full nodes to 6,000 in two years…” P Todd 2015
 https://getaddr.bitnodes.io/dashboard/?days=365 “Reachable nodes during the last 365 days.” Bitnodes.io
 https://bitcointalk.org/index.php?topic=532.msg6306#msg6306 “Re: Scalability and transaction rate” Satoshi 2010
 http://firstname.lastname@example.org/msg08161.html “[Bitcoin-development] Mining centralization pressure from non-uniform propagation speed” Pieter Wuille 2015
My good friends at the Amateur Radio Experimenters Group (AREG), an Adelaide based Ham radio club, have organised a special FreeDV QSO Party Weekend on Sat/Sun September 12/13th. This is a great chance to try out FreeDV, work VK5 using open source HF digital voice, and even talk to me!
All the details including paths, frequencies, and times over on the AREG site.
This is the entire complaint that was received:
We are an IT security company from Spain.
We have detected sensitive information belonging to Banesco Banco Universal, C.A. clientes.
As authorized representative in the resolution of IT security incidents affecting Banesco Banco Universal, C.A., we demand the deletion of the content related to Banesco Banco Universal, C.A, clients. This content violates the law about electronic crime in Venezuela (see “Ley Especial sobre Delitos Informáticos de Venezuela”, Chapter III, Articles 20 y 23).
Note the complete lack of any information regarding what URLs, or even which site(s), they consider to be problematic. Nope, just “delete all our stuff!” and a wave of the “against the law somewhere!” stick.
Carina gave a fantastic talk of the very real consequences of our interactions with data mining and the algorithms used to target us. A must watch when the video comes out.
A humorous and contructive talk on the pitfalls of pip. Strongly recommended that people no longer use it.Arrested Development - surviving the awkward adolescence of a microservices-based application
- Logging is a super power.
- Be explicit
- Measure everything
- Scale via automation
by Katie Bell
- Run coding camps and competitions for kids with no prior experience required.
- Courses and competitions designed to bootstrap kids into programming basics.
- Bad tests are better than no tests.
- Write tests early.
- Results in better interfaces.
- Clearer separation.
- Write tests first.
- An interface is anything that gives OK isolation to the code you want to test.
- Good interfaces tests will test very branch.
- Measure all the things.
- Measure setup time
- Measure execution time
- Run tests thoroughly nightly.
- Make it easy to translate a bug report into a test case.
- Retain your test cases and run them regularly.
by Tom Eastman
- Minimise the attack surface.
- Reduce the privileges of the account used by the app.
- Eliminate SQL injections.
- White list input validation.
- "Escape" all data appropriately.
- Use Content Security Policy to protect against cross site scripting.
- Don't have all your eggs in one basket.
- You can always have better logging. Off site logging is useful.
- Take a look at the ELK stack.
- 10 Awful (But Funny) Résumés That People Actually Submitted http://t.co/KZqroMVRWp 16:33:02, 2015-08-01
- Assessment: Identify Your Entrepreneurial Personality Type (HBR) http://t.co/pVLTqfdQyp 14:19:00, 2015-08-01
- Why Gen-Y men aren’t the Dads they thought they’d be http://t.co/5qd3oHO7iW 14:19:02, 2015-07-31
- Coal not good for reducing poverty, Oxfam report says http://t.co/zKH5xmSTU7 #auspol 10:42:12, 2015-07-31
- Lack of transparency in political donations is a recipe for corruption http://t.co/NkBbtFycyh #auspol 10:42:01, 2015-07-28
- Virtual Reality for the masses could account for a third of all digital video consumption within two years http://t.co/ACt7K1ryeq 16:33:08, 2015-07-27
I don't read as much as I should these days, but one author I always make time for is John Scalzi. This is the next book in the Old Man's War universe, and it continues from where The Human Division ended on a cliff hanger. So, let's get that out of the way -- ending a book on a cliff hanger is a dick move and John is a bad bad man. Then again I really enjoyed The Human Division, so I will probably forgive him.
I don't think this book is as good as The Human Division, but its a solid book. I enjoyed reading it and it wasn't a chore like some books this far into a universe can be (I'm looking at you, Asimov share cropped books). The conclusion to the story arc is sensible, and not something I would have predicted, so overall I'm going to put this book on my mental list of the very many non-terrible Scalzi books.
Tags for this post: book john_scalzi combat aliens engineered_human old_mans_war age colonization human_backup cranial_computer personal_ai
Related posts: The Last Colony ; The Human Division; Old Man's War ; The Ghost Brigades ; Old Man's War (2); The Ghost Brigades (2) Comment Recommend a book
We’ve finally had time to finalise Korora 22 and images are now available. I strongly recommend downloading with BitTorrent if you can.
We are not shipping Adobe Flash by default from 22 onwards, due to consistent security flaws. We still include the repository however, so users can install via the package manager or command line if they really want it:
sudo dnf install flash-plugin
Alternatively, install Google Chrome which includes the latest version of Flash.
Also, KDE 4 is not available for this release, so if you are not ready to move to KDE 5, then please stick to Korora 21.
The list of available wifi channels is slightly different from country to country. To ensure access to the right channels and transmit power settings, one needs to set the right regulatory domain in the wifi stack.Linux
For most Linux-based computers, you can look and change the current regulatory domain using these commands:iw reg get iw reg set CA
where CA is the two-letter country code when the device is located.
On Debian and Ubuntu, you can make this setting permanent by putting the country code in /etc/default/crda.
Finally, to see the list of channels that are available in the current config, use:iwlist wlan0 frequency OpenWRT
In order to persist your changes though, you need to use the uci command:uci set wireless.radio0.country=CA uci set wireless.radio1.country=CA uci commit wireless
where wireless.radio0 and wireless.radio1 are the wireless devices specific to your router. You can look them up using:uci show wireless
To test that it worked, simply reboot the router and then look at the selected regulatory domain:iw reg get Scanning the local wifi environment
Once your devices are set to the right country, you should scan the local environment to pick the least congested wifi channel. You can use the Kismet spectools (free software) if you have the hardware, otherwise WifiAnalyzer (proprietary) is a good choice on Android (remember to manually set the available channels in the settings).
Excellent keynote on using Python in education. Many interesting insights into what's being done well, what needs improving and how to contribute.Slow Down, Compose Yourself - How Composition Can Help You Write Modular, Testable Code
- Modularity and testability traits are desirable.
- Tests are the only way ti ensure your code works
- When designing a system, plan for these traits
- Fake components can be used to return hard to replicate conditions ie: a database returning "no more space"
- Ensure you have correctness at every level.
- Suggests using practices from better (functional programming) languages like Haskell with Python.
- Trosnoth is a 2D side scrolling game.
- Team strategy game
- Involve people at every step of the process.
- Have a core group of interested people and allow them to contribute and become involved to increase commitment.
- Build community.
- Get people excited
- Be prepared to be wrong.
- Encourage people to be involved.
- Start with "good enough".
- Re-writing everything blocked contributions for the period of the re-write.
- Look for ideas anywhere.
- Mistakes are a valuable learning opportunity.
- Your people and your community are what makes the project work.
by Todd Owen (http://2015.pycon-au.org/schedule/30057/view_talk?day=saturday)
- Ansible is a push model of configuration management and automation.
- Pull model of Chef and Puppet is unnecessarily complex.
- Ansible's simplicity is a great asset
- Graceful and clear playbooks.
- Simple is better than complex.
- Complex is better than complicated.
- Flat is better than nested.
- Ansible modules are organised without hierarchy.
- Ansible only uses name spaces for roles.
- Explicit is better than implicit. Implicit can become invisible to users.
- Ansible id very consistent.
- Simple by design.
- Easy to learn.
by Tim Butler
- Prefers to think of containers as application containers.
- Docker is an abstraction layer focussed on the application it delivers.
- Easy, repeatable deployments.
- Not a fad, introduced 15 years again.
- Google launch over 2 billion containers per week.
- Docker is fast, lightweight, isolated and easy.
- Rapidly changing and improving.
- Quickly changing best practices.
- Containers are immutable.
- No mature multi-host deployment.
- Security issues addressed via stopping containers and starting a patched one (milliseconds)
- Simple and clear subcommands.
- Docker compose for orchestration.
- Docker machine creates to the Docker host for you.
- Volume plugin
- New networking system so it actually works at scale.
- Salt stack provides:
- configuration management
- loosely coupled infrastructure coordination.
- remote execution
- unittest 101
- create a test directory
- write a test
- gave an example of a "hello world" of tests.
- uses unittest.main.
- Nose extends unittest to better facilitate multiple unit tests.
- Mock allows the replacement of modules for testing.
- Keep tests small.
- Claims Python on mobile is very viable.
- Most of the failing bugs are not services you want on mobile anyway.
- libffi problems needs to be resolved.
- Compiler issues not quite resolved.
- libffi problems needs to be resolved.
- What not Jython? Does not compile on Android. Nor will many of it's dependencies.
- Jython is probably not the right solution anyway.
- Thinks you may be able to compile Python directly to Java...uses Byterun as an example.
- Kivy works now and worth using.
- His project Toga is coming along nicely.
- Admits he may be on a fools errand but thinks this is achievable.
by Victor Palma
- The sysadmin mindset of get things fixed as fast as possible needs to be shed.
- Take the time to step back and consider problems.
- Keep things simple, explicit and consistent.
- Do comments in reStructured Text.
- Unless you're testing, you're not really coding.
- Tools include tox and nose.
- Don't be afraid to experiment.
- Find something you are passionate about and manipulate that data.
- Gerrit does code review.
- Zuul tests things right before they merge and will only merge them if they pass.
- Only Zuul can commit to master.
- Zuul uses gearman to manage Jenkins jobs.
- Uses NNFI - nearest non-failing item
- Use jenkins-gearman instead of jenkins-gerrit to reproduce the work flow.
by Monty Taylor
- Create truth in realistic acting
- Know what problem you're trying to solve.
- Develop techniques to solve the problem.
- Don't confuse the techniques with the result.
- Willingness to change with new information.
What Monty Wants
- Provide computers and networks that work.
- Should not chase 12-factor apps.
- Kubernetes / CoreOS are already providing these frameworks
- OpenStack should provide a place for these frameworks to work.
- By default give a directly routable IP.
inaugust.com/talks/a-vision-for-the-future.htmlThe Future of Identity (Keystone) in OpenStack
- Moving to Fernet Tokens as the default, everywhere.
- No database requirement
- Limited token size
- Will support all the features of existing token types.
- Problems with UUID or PKI tokens:
- SQL back end
- PKI tokens are too large.
- Moving from bespoke WSGI to Flask
- Moving to a KeystoneAuth Library to remove the need for the client to be everywhere.
- Keystone V3 API...everywhere. Focus on removing technical debt.
- V2 API should die.
- Deprecating the Keystone client in favour of the openstack client.
- Paste.ini functionality being moved to core and controlled via policy.json
- Gave a great overview of OpenStack / CoreOS / Containers
- All configuration management sucks. Ansible sucks less.
- CI/CD pipelines are repeatable.
by Jamie Lennox
- SAML is the initially supported WebSSO.
- Ipsilon has SAML frontend, supports SSSD / PAM on the backend.
- Requires Keystone V3 API everywhere.
- Jamie successfully did live demo that demonstrated the work flow.
by Angus Lees
- Uses Linux kernel separation to restrict available privileges.
- Gave a brief history of rootwrap`.
- Fast and safe.
- Still in beta
by Monty Taylor
- Shade's existence is a bug.
- Take OpenStack back to basics
- Keeps things simple.
Once in a while you want to start a daemon with differing parameters from the norm.
For example, the default parameters to Fedora's packaging of ladvd give too much access to unauthenticated remote network units when it allows those units to set the port description on your interfaces. So let's use that as our example.
With systemd unit files in /etc/systemd/system/ shadow those in /usr/lib/systemd/system/. So we could copy the ladvd.service unit file from /usr/lib/... to /etc/..., but we're old, experienced sysadmins and we know that this will lead to long run trouble. /usr/lib/systemd/system/ladvd.service will be updated to support some new systemd feature and we'll miss that update in the copy of the file.
What we want is an "include" command which will pull in the text of the distributor's configuration file. Then we can set about changing it. Systemd has a ".include" command. Unfortunately its parser also checks that some commands occur exactly once, so we can't modify those commands as including the file consumes that one definition.
In response, systemd allows a variable to be cleared; when the variable is set again it is counted as being set once.
Thus our modification of ladvd.service occurs by creating a new file /etc/systemd/system/ladvd.service containing:.include /usr/lib/systemd/system/ladvd.service [Service] # was ExecStart=/usr/sbin/ladvd -f -a -z # but -z allows string to be passed to kernel by unauthed external user ExecStart= ExecStart=/usr/sbin/ladvd -f -a
 At the very least, a security issue equal to the "rude words in SSID lists" problem. At it's worst, an overflow attack vector.
I’ve been working on the Linear Predictive Coding (LPC) modeling used in the Codec 2 700 bit/s mode to see if I can improve the speech quality. Given this mode was developed in just a few days I felt it was time to revisit it for some tuning.
LPC fits a filter to the speech spectrum. We update the LPC model every 40ms for Codec 2 at 700 bit/s (10 or 20ms for the higher rate modes).
Speech Codecs typically use a 10th order LPC model. This means the filter has 10 coefficients, and every 40ms we have to send them to the decoder over the channel. For the higher bit rate modes I use about 37 bits/frame for this information, which is the majority of the bit rate.
However I discovered I can get away with a 6th order model, if the input speech is filtered the right way. This has the potential to significantly reduce the bit rate.
Our ear perceives speech based on the frequency of peaks in the speech spectrum. When the peaks in the speech spectrum are indistinct, we have trouble understanding what is being said. The speech starts to sound muddy. With analog radio like SSB (or in a crowded room), the troughs between the peaks fill with noise as the SNR degrades, and eventually we can’t understand what’s being said.
The LPC model is pretty good at representing peaks in the speech spectrum. With a 10th order LPC model (p=10) you get 10 poles. Each pair of poles can represent one peak, so with p=10 you get up to 5 independent peaks, with p=6, just 3.
I discovered that LPC has some problems if the speech spectrum has big differences between the low and high frequency energy. To find the LPC coefficients, we use an algorithm that minimises the mean square error. It tends to “throw poles” at the highest energy part of signal (frequently near DC), while ignoring the still important, lower energy peaks at higher frequencies above 1000Hz. So there is a mismatch in the way LPC analysis works and how our ears perceive speech.
For example I found that samples like hts1a and ve9qrp code quite well, but cq_ref and kristoff struggle. The former have just 12dB between the LF and HF parts of the speech spectrum, the latter 40dB. This may be due to microphones, input filtering, or analog shaping.
Another problem with using an unconventionally low LPC order like p=6 is that the model “runs out of poles”. Some speech signals may have 4 or 5 peaks, so the poor LPC model gets all confused and tries to reach a compromise that just sounds bad.
I messed around with a bunch of band pass filters that I applied to the speech samples before LPC modeling. These filters whip the speech signal into a shape that the LPC model can work with. I ran various samples (hts1a, hts2a, cq_ref, ve9qrp_10s, kristoff, mmt1, morig, forig, x200_ext, vk5qi) through them to come up with the best compromise for the 700 bits/mode.
Even though the latter sample is band limited, it is easier to understand as the LPC model is doing a better job of clearly representing those peaks.
After some experimentation with sox I settled on two different filter types: a sox “bandpass 1000 2000″ worked on some, whereas on others with more low frequency content “bandpass 1500 2000″ sounded better. Some helpful discussions with Glen VK1XX had suggested that a two band AGC was common in broadcast audio pre-processing, and might be useful here.
However through a process of frustrated experimentation (I was stuck on cq_ref for a day) I found that a very sharp skirted filter between 300 and 2600Hz did a pretty good job. Like p=6 LPC, a 2600Hz cut off is quite uncommon for speech coding, but SSB users will find it strangely familiar…….
Note that for the initial version of the 700 bit/s mode (currently in use in FreeDV 700) I have a different band pass filter design I chose more or less at random on the day that sounds like this with p=6 LPC. This filter now appears to be a bit too severe.
Here is a little chunk of speech from hts1a:
Below are the original (red) and p=6 LPC models (green line) without and with a sox “bandpass 1000 2000″ filter applied. If the LPC model was perfect green and red would be superimposed. Open each image in a new browser tab then jump back and forth. See how the two peaks around 550 and 1100Hz are better defined with the bandpass filter? The error (purple) in the 500 – 1000 Hz region is much reduced, better defining the “twin peaks” for our long suffering ears.
Here are three spectrograms of me saying “D G R”. The dark lines represent the spectral peaks we use to perceive the speech. In the “no BPF” case you can see the spectral peaks between 2.2 and 2.3 seconds are all blurred together. That’s pretty much what it sounds like too – muddy and indistinct.
Note that compared to the original, the p=6 BPF spectrogram is missing the pitch fundamental (dark line near 0 Hz), and a high frequency peak at around 2.5kHz is indistinct. Turns out neither of these matter much for intelligibility – they just make the speech sound band limited.
OK, so over the last few weeks I’ve spent some time looking at the effects of microphone placement, and input filtering on p=6 LPC models. Now time to look at quantisation of the 700 mode parameters then try it again over the air and see if the speech quality is improved. To improve performance in the presence of bit errors I’d also like to get the trellis based decoding into a real world usable form. When the entire FreeDV 700 mode (codec, modem, error handling) is working OK compared to SSB, time to look at porting to the SM1000.
Command Line Magic
I’m working with the c2sim program, which lets me explore Codec 2 in a partially quantised or incomplete state. I pipe audio in and out between various sox stages.
Note these simulations sound a lot better than the final Codec 2 at 700 bit/s as nothing else is quantised/decimated, e.g. it’s all at a 10ms frame rate with original phases. It’s a convenient way to isolate the LPC modeling step with as much fidelity as we can.
If you want to sing along here are a couple of sample command lines. Feel free to ask me any questions:
sox -r 8000 -s -2 ../../raw/hts1a.raw -r 8000 -s -2 -t raw - bandpass 1000 2000 | ./c2sim - --lpc 6 --lpcpf -o - | play -t raw -r 8000 -s -2 -
sox -r 8000 -s -2 ../../raw/cq_ref.raw -r 8000 -s -2 -t raw - sinc 300 sinc -2600 | ./c2sim - --lpc 6 --lpcpf -o - | play -t raw -r 8000 -s -2 -
Interactive map for this route.
Tags for this post: blog pictures 20150729 photo sydney
Related posts: In Sydney!; In Sydney for the day; A further update on Robyn's health; RIP Robyn Boland; Weekend update; Bigger improvements
See more thumbnails
I also made a quick and dirty 360 degree video of the view of LA from the top of the nike control radar tower:
Interactive map for this route.
Tags for this post: blog pictures 20150727-nike_missile photo california
Related posts: First jog, and a walk to Los Altos; Did I mention it's hot here?; Summing up Santa Monica; Noisy neighbours at Central Park in Mountain View; So, how am I getting to the US?; Views from a lookout on Mulholland Drive, Bel Air