New Job! DevOps Engineer at University of Melbourne

I’ve started a new job! Some of you guys might know that I’ve left NUS about 1 month ago, and that I was moving to Melbourne. I left kind of abruptly and did not gave much details, as everything was pretty much unconfirmed even before I flew off. Now that the dust has finally settled, I felt that it is time to write about what’s been happening for the last one month.

I’ve started work at University of Melbourne, as a DevOps Engineer working on NeCTAR Research Cloud. NeCTAR RC is a big federated OpenStack cloud made up of nodes all around Australia. I joined the Melbourne Node, working with three other colleagues. I’ve always been interested in OpenStack so I guess it’s a good change for me!

I’m still catching up on everything, so I’m just trying to respond to support tickets whenever I can, and poking around the cloud when I have time. We just upgrade one of our sites from Precise to Trusty, and I’m currently doing some maintenance work and writing tools.

The research cloud is quite big. There are pretty graphs at the status page, and Melbourne node is one of the biggest at about 1500 instances in 3500 cores! (P.S: I hope on of the tools I’m working on can push that up ~5%!)

Most of the ‘development’ seems to come from the lead node guys (sitting a few steps away). They are really quite good at cells and messaging and all that. To my knowledge, pretty much only NeCTAR RC, RackSpace and CERN runs nova-cells now. So, it’s really cool to be able to see the workings of a big federated cloud from the inside!


Disabling 802.11b

We’ve recently tried to disable the old legacy 802.11b on 1 of our wireless controllers to free up more airtime for the newer protocol. What a difference it made in Channel Utilization!wlc3-public

We will be rolling this out for the other controllers soon. Bye bye 802.11b!


We are now running a ceph cluster, which I find is awesome. Who doesn’t like distributed, easily scalable storage pools?

However, the ceph storage is pretty useless if the clients can’t mount it. Given that most clients talk NFS, SMB, iSCSI and not ceph, an intermediate node needs to be created for exporting ceph to the clients of the world. Enters nfsceph.

nfsceph is something I’ve written off and on over the past few weeks. It is a set of scripts that allows you to create rbds (rados block device) on ceph, maps them, formats them and exports them to the world. More concisely terms, rbd create, rbd map, mkfs.ext3, exportfs.

Let’s see how it makes our (my) life easier!


‘nfsceph create’ creates a filesystem on ceph

[root@nfs1 ~]# nfsceph create backup 10000
Creating rbd... Success.
Mapping rbd...Success.
Making filesystem...Success.
Mounting filesystem...Success.


‘nfsceph list’ lists our filesystems

[root@nfs1 ~]# nfsceph list
backup 10.48576 GB


‘nfsceph export <filesystem> <ip>’ nfs exports a filesystem to the ip specified
‘nfsceph export’ shows the exports you have

[root@nfs1 ~]# nfsceph export backup
[root@nfs1 ~]# nfsceph export

At this point, the filesystem is ready to be mounted on the client. You can specify multiple clients, and also netblock (

More Information

The ceph rbd is mounted on /dev/rbd<x>

[root@nfs1 ~]# mount | grep backup
/dev/rbd6 on /export/backup type ext3 (rw)

The filesystem is exported with the following options for best performance and compatibility.

[root@nfs1 ~]# exportfs -v | grep backup

There’s also a set of initscripts that saves the current state to a file, and makes the exports persistent across reboot. If you’d like to play with it, the source can be found on github.

With this architecture, we can scale out quite easily by just adding more intermediate nodes to ease the load. Cheap, (practically) unlimited NFS storage. Awesome. 🙂

OpenStack Active Directory / LDAP authentication

OpenStack (Grizzly) allows keystone to authenticate to different backends. The default backend is an SQL database, storing both user information (username/password) and also tenant information (which user belongs to which group). Although you can update this to a LDAP based backed, it would mean having to take care of tenant information in LDAP too (which means tedious things like creating new LDAP DC, which no self-respecting LDAP admin will let you do arbitrarily). But what if you just wants OpenStack to authenticate to a LDAP server, like Active Directory in an enterprise setting?

Luckily, keystone allows you to extend authentication easily. What the following patch does is to allow you to set up to 3 LDAP servers, which keystone will attempt to bind to using provided username / password when a user logs in. It can also fall back to use the user information in SQL if it fails to bind to LDAP servers by setting FALLBACK = True.

First of all, you need to create your own Identity backend with _check_password() function. Please check out on my github. Put this file into keystone/identity/backends.

Next, you will need to update to read some new configurations in your keystone.conf.

The full patch is in my github.

After this, you can update keystone.conf to specify the LDAP servers that you want to authenticate with. Example:

server1_host = ldap://
server2_host = ldap://
server3_host = ldap://
server1_domain = DOMAIN1
server2_domain = DOMAIN2
server3_domain = DOMAIN3
fallback = True

Watching Netflix in Singapore

roku netflix

Recipe for Netflix in Singapore



  • Sign up for MyRepublic Fibre Broadband Service
  • Sign up for Netflix free trial through their website [3]
  • Purchase Roku 3 through Amazon (free shipping to Singapore)
  • Twiddle thumbs till Roku 3 arrives
  • IMPORTANT: Set up a Roku account with a US Country and Zip Code. Use credit card.
  • Plug in Roku 3 (might need a 220v to 110v step down transformer, but users have reported success without)
  • Run through setup.
  • Start Netflix
  • Watch Netflix


  • Cancel Starhub 🙂

[1] My Republic Teleport is free till 31 Dec, $5 a month afterwards (I really hope they don’t charge!)
[2] I’ve heard Apple TV works, and WD TV Live too. Let me know if your device works for you
[3] Free trial for 1 month, so that you don’t lose anything if it doesn’t work. You need to pay for it after free trial.

[edits:] added information that you need to create a Roku US account BEFORE activating Roku

Bad experience with IPMIRROR robot review

A good friend recently wanted to register a .SG domain, and he asked me about SG registrars. This reminded me of a IPMIRROR fiasco a few years ago, and I just wanted to let people know about it. In case someone asks me again, then I can point them to this webpage and I don’t have to repeat myself.

Every year during National Day, SGNIC will run a promotion offering .sg domains for a steal (like $8, instead of $40+ normally). Looking through, I decided to go with ipmirror as the registrar. However, as ipmirror was still charging the full price in their system. I emailed them about this and they replied that their system is not updated, but this is not a problem as they will take note and rebate me the difference after I registered. So I went ahead and register and emailed them.

Posted On: 23 Aug 2009 11:05 AM
Dear Wai Peng YIP,
First Name: Wai Peng
Last Name: YIP
Company Name:
Contact Number:
Email Address:
Email to:
Subject: I Love sg 2009 promotion
Message: I registered Please help process the rebate

They replied promptly, which was cool

From: Jenelle Bi <>
Sent: Monday, 24 August 2009 11:41 AM
Subject: [#SWK-630635]: I Love sg 2009 promotion

Dear Wai Peng,

Thank you for choosing IP Mirror as your preferred registrar.

We have received your domain name registration under the I 2009 promotion. As you have paid the full price at the point of submission, we have calculated and refunded the excess amount into your credit balance account. This amount can be used for your future domain name transactions and if you have further inquires pertaining to this refund, we will be most pleased to assist.
Best regards,

Jenelle Bi
IP Mirror Pte Ltd
47 Duxton Road, IP Mirror TechHaus, Singapore 089511
Tel: +65 6222-0105 Fax: +65 6222-0210

Fast forward a few years, I didn’t want the domain anymore and wanted to close the account, getting my refund back. I emailed them.

From: “Yip Wai Peng” < >
Sent: Monday, July 4, 2011 1:32:17 PM
Subject: withdraw credit in account

Dear ipmirror,

I would like to withdraw my credits in my account. My username is <likeyouwanttoknow>.

If you can refund via a check, please send a crossed cheque to

Name: YIP Wai Peng

Address: <redacted>

I got this back in reply

From: ” ” < >
To: Yip Wai Peng < >
Cc: IP Mirror Billing < >
Sent: Tuesday, 5 July 2011 11:35 AM
Subject: Re: withdraw credit in account

Dear Wai Peng,

For refund, there is a 10% admin charge, therefore we will only issue a cheque of $43.38.
Attached refund policy for your reference

Best regards,

IP Mirror Pte Ltd
47 Duxton Road, IP Mirror TechHaus, Singapore 089511
Tel: +65 6222-0105 Fax: +65 6222-0210
Reg No: 200003703C

What? You took my money for 2 years and now you’re charging me an admin charge?!

Thinking that this was a automated reply, I told them of the circumstances why I had a credit, hoping a human can step in and set things right.

From: “Yip Wai Peng” < >
Sent: Tuesday, July 5, 2011 1:36:23 PM
Subject: Re: withdraw credit in account

Dear ipmirror,

This refund was because of the purchase of .sg domain during the SGNIC promotion previously. Your system was unable to charge the promotion pricing, thus I was asked to pay the full price and get the refund. I do not think it is fair that I am charged a penalty for this.

– WP

The same robotic reply came back

Sent: Thursday, 7 July 2011 4:01 PM

Dear Wai Peng,

We regret to inform you that we will only refund $43.38 cause there is a 10% admin charge. Please refer to our web for refund policy. 

Best regards,

Undeterred, I tried again.

Sent: Thursday, July 7, 2011 6:20:58 PM

Dear IP Mirror,
Please try to understand the situation.

1. .SG domains were supposed to go for $5 during the I Love SG promotion. This is a nation wide promotion by SGNIC. All the registrars are supposed to charge $5.
2. When I choose you guys as my registrar, your systems were still charging the non-sale price.
3. I was told that your systems have not been updated, I should pay the full price and the remainder will be refunded.
4. There was no notification of any admin fee or any other fees at that time.

I didn’t mind paying more and getting the full refund as your systems were not updated. However, it is unfair to the customer to be charged an admin fee just because your systems were not updated. If you had that constraint, you should make it clear up front.

Your policies are set up by humans, Please try to understand the situation and help.

– WP

Same robotic reply

Sent: Monday, 11 July 2011 4:49 PM

We are so sorry to inform you that we will only refund $43.38 cause it is in our refund policy. Please refer to our web for refund policy.

Annoyed, I replied

Sent: Monday, 11 July 2011 10:38 PM

I really understand you have a policy. Do you understand what I have been trying to explain to you so far? By the way, may I know who I am talking to? If you are unable to help me from your position, I would like to escalate this please.

The billing robot didn’t reply after this. A few days later, I sent

Sent: Saturday, July 16, 2011 8:55:16 AM
Subject: Re: withdraw credit in account

Dear IP Mirror,
Any updates o this please?

Robot again

Sent: Monday, 18 July 2011 12:40 PM
Subject: Re: withdraw credit in account

Dear Wai Peng,

We understand what you are trying to explain and you are talking to the refund department. Please refer to our web site for refund policy and we will only refund $43.38. 

I gave up. Never argue with an idiot adage.

Sent: Friday, July 29, 2011 6:23:32 PM
Subject: Re: withdraw credit in account

Ok please refund me $43.38. Thanks.

And the reply

Date: Mon, 01 Aug 2011 11:29:43 +0800 (SGT)

We have issue a cheque UOB 730487 amount $43.38 and has mail out on 28 July 2011.

Best regards,
Jasmine Loh

Finally, a human!

Shame on you, ipmirror. Shame on you, Jasmine and the helpdesk / billing robots. You people deserve better then to work for such a douchebag company.

How MyRepublic Teleport works

I’ve just signed up with MyRepublic on their Pure HD service, mostly due to their Teleport service. Briefly, Teleport allows you to watch US only service like Netflix and Hulu+ from Singapore.

In addition, I also purchased a WD TV Live to watch Netflix on my big screen TV. However when I set it up, I realized that the WD TV Live does not work with Netflix! 😦

After feeling sorry for myself, I decided to figure out how Teleport works, and maybe try to fix the issue with Netflix and WD TV Live.

First of all, I heard that many WD TV Live users have managed to let Netflix work using Unblock-Us. I went ahead and tried configuring Unblock-Us, and sure enough, it works! This made me further believe that the issue is not with WD TV Live nor Netflix, and surely is with Teleport.

I set up my laptop to NAT all traffic in and out of the WD TV, so that I could listen to all the traffic.

In short, MyRepublic Teleports uses their DNS to redirect you to an Amazon instance in the US for specific domains – mostly the authentication / setup part of streaming services like Netflix. The main bulk of the streaming content afterwards comes from CDNs, which I believe does not need to go through the US link. Let’s take a look.

The WD TV Live starts off by connecting to If you look it up using MyRepublic DNS servers, you can see that it resolves to an Amazon EC2 instance in the US WEST.

$ dig @

$ dig -x
<snip> 300  IN      PTR

After that, it connects to 2 other domains, and This is where the problem lies – MyRepublic still resolves these two to the same EC2 instance. 0 IN A 0 IN A

As far as I can tell, both and connections are HTTPS, which means they can’t share the same IP. To test my theory, I set up a DNS server that responds with the Unblock-Us DNS servers for and It works!

I guess the fix for MyRepublic is simple – they just have to create another 2 instances to take care of the traffic going to the 2 affected domains, and everything should work!

I’ve forwarded them the information, hopefully it’ll help them.