APNIC 32 - Destination::IPv6

Transcript - Panel discussion


Due to the difficulties capturing a live speaker's words, it is possible this transcript may contain errors and mistranslations. APNIC accepts no liability for any event or action resulting from the transcripts.

APNIC 32 Conference
Thursday, 1 September 2011
09:00 (UTC +9)
Panel discussion

Philip Smith: I think we should make a start, if everybody can take their seats.

Good morning, everybody.  I think we have had probably a pretty good social evening.  There seems to be a significant absence of women in the room, so I think the Women in ICT Event must have been very good last night.  I don't think I've ever seen such a ratio in an APNIC Meeting before.  I imagine the other participants will be drifting in as we proceed in the morning.

This is a bit of, I suppose, a nice technical 90 minutes before we head off into the duties of the member meeting.

We are going to hold a panel discussion.  I suppose the motivation for that is that we have had two fairly significant natural disasters within the Asia Pacific Region over the last year, and both of them in some sense are still ongoing in the impacts they are having on the community.  We thought we would invite four folks who have been, I suppose, deeply involved in a lot of activities dealing with those disasters, at least from the Internet perspective.

We have invited them on to the panel here to give their impressions and some feedback, some advice, from their local industry about how their industry dealt with keeping the Internet running during those natural disasters.

We have seen in the press and we have seen in a lot of technical journals about how much in both communities the Internet still kept working, in spite of what mother nature was doing to the environment around us.

The panelists we have, from New Zealand we have Dean Pemberton and Andy Linton, and from Japan we have Katsuyasu Toyama and Tomoya Yoshida.

I am going to invite each one to come up to the podium to give a presentation on their feedback, their impressions.  We will do these four presentations in sequence, and then at the end we will have time for questions and answers and general discussion from the audience and from the panelists.

Andy will be first up.  I would like to invite Andy Linton to start us off here.

Andy Linton: Dean and I are going to do this as a tag team.  Halfway through this, he will take over the slides, because we have worked on it together.

Some things we have put together here, based on experience in New Zealand.  Dean and I were both in Hong Kong last year when the earthquake happened.  We were at the APRICOT-APNIC Meeting in Hong Kong and people were nudging each other in the session, saying, "Oh, shit.

Do you see what's happened?"  It was kind of scary.

While neither of us live in Christchurch, we were concerned that friends of ours and family might be involved.

New Zealand being a very small country, it's almost impossible not to have some connection with somebody, either a friend of a friend or that sort of thing.

We left that meeting and went back to Wellington.

Certainly for me, this isn't about a personal experience of what I did, this is a report about what other people did.  But let's have a look.

Just to set the scene, this is a picture from Port-au-Prince in Haiti.  I got this picture from a news website, I think it was The Guardian, and the next slide talks about it.

Christchurch suffered damage like this, and our colleagues in Japan and people in Japan suffered similar devastation.  I think we were lucky that we were doing a little bit better than someone drawing a cart by hand.

We had trucks and so on.  But you can see the sort of devastation that goes on.

Here is the quote.  The picture and the quote came from an article in The Guardian.  This woman, Kate Buckley, said: "Rescue missions after the Sri Lankan tsunami and Haitian earthquake were boosted by using mobile phone networks, satellite and other computer software."

So we weren't the first to think of this, this sort of approach has been going on for a number of years.

I am going to talk about three things.  Dean will talk more about keeping the network operational and some of the war stories from that.

I want to talk about three things that were done, which seemed to help.  The three things are the Christchurch Earthquake Map, the Student Volunteer Army and Trade Me.  Trade Me is an auction site in New Zealand, if you think of the New Zealand eBay.  EBay never got a foothold in New Zealand because these guys have done such a good job of stitching up the local market.

The Christchurch Recovery Map was built on a piece of software called Ushahidi.  This software was developed in Kenya in the post-election violence thing and it was a way for people in the local community to report problems -- don't go into this part of town because this is happening, sort of thing.  That software was used in a number of elections in Africa, it was used in India to monitor elections and was used in the Sri Lankan tsunami and was used extensively in Haiti.  When the earthquake happened in Haiti, some people had cell phones working so they were able to text their friend in the States and say, listen, such and such a building, I can hear noises of people, and whatever, and volunteers in the States actually put together a map which was used to coordinate.  We will see some examples of that.

The idea is that this is a system that can take text message, SMS and pictures, MMS, and also take email and start to coordinate them and put the results up on a map.

The next thing about this information was that it was able to be used in New Zealand by multiple websites.

The maps from this website were embedded in the Herald and Stuff, the New Zealand Herald is one of the major newspapers and Stuff is an online news website which is linked to the other paper, The Dominion Post.

It was also fed into the Google Crisis Response Page, and Eqviewer, so a lot of cooperation went on here.

I have talked about Ushahidi.  You can look at this.

One of the nice things you get with Ushahidi is, as well as being able to look at the stuff on a web page, you can get an application for smartphones.  On my Android phone, I have a little Ushahidi thing.  Not everybody has these, but an awful lot of people do have these, so you can look at the map and see what's going on in my area.

Remember, Christchurch is a city of a couple of hundred thousand people, so it's not a big city by world standards.  It's our second largest city.

Remember, we have 4.5 million people in New Zealand, about the same population as Busan, spread over a much wider geographic area and it's long and skinny, those of you who remember the map.  Communication from one end of the country by physical means takes two to three hours to get up and down the country.

One of the real benefits of this was that people were able to see what was happening on the street.

I will talk a little bit about the short code thing in a second.

The nice thing about this was we had our standard responses from the civil defence, which said we are posting information saying -- water was disrupted badly in Christchurch.  Christchurch is built basically on a river delta, an alluvial plain, and the ground turned to jelly, it just liquefied, and people were sitting with a foot or 30cm of silt in their living rooms.  Their houses had slipped off the foundations and cracks would open in the concrete slabs of people's houses and silt came up.  You would be sitting there and all of a sudden it would come up into your living room.

There was no water, no sewage, people were relying on chemical toilets, yet they still had the Internet working.  It is a strange paradox that all those basic things of civilized society were in trouble, and the luxury things to some degree were working.  Mobile phones were working pretty much.  They got heavily loaded.

One of the nice things was that civil defence would put up something, "There will be a water tanker at the junction of Smith Street and Blogg Avenue," and people were going up with water bottles and collecting water.

You sit at home and get that off the radio or whatever, and say, OK, I'll nip down and get some water.

But if you have somebody on the street with a cell phone who can patch into a message system which says, "That's great, but that tanker has run out of water and has gone back to get more, but we went down a street three blocks further down and this tanker still has water," it saves a whole lot of confusion and so on.

Other things people needed to do, people have medical conditions like diabetes or so on, they need prescriptions.  Which chemist do I go to, which pharmacy do I go to, to get my prescription filled?  If somebody finds an open pharmacy, they report in that there is an open pharmacy.

This is not just in the real disaster -- the first phase of the disaster, where you are trying desperately to dig bodies out of rubble and so on, this is in the aftermath when people are trying to piece their lives back together and get their systems up and moving again.

Christchurch has had nearly a year now of this stuff, with constant earthquakes, and I know now, I have talked to people, they are really tired, they are woken up several times a week, several times a night even with small quakes, and think, is this another one.

Anything that is done to help people get their lives back on track is fantastic stuff.

One of the things that was really interesting was we got a short code, an SMS short code, four digits, livened up really, really quickly.  A company in Wellington called Catalyst, who deal in open source software, had this SMS code for testing purpose.  They talked to the three major telcos in New Zealand and said, we want you to make this real and live, and not charge us for this and we will put this to good use.

And they did, it was great.  The telcos came to the party and did the job here.

That short code was up within hours and people were able to say, use this short code to send your messages.

There are a couple of example websites, which you can look at.  I think everybody in the room probably has a laptop so you can have a look at these.  These ones are live.  The Christchurch site has been taken down, the original earthquake map, because its job is basically done and the normal systems have started to kick back in.

The Syrian one is dealing with the problems going on in Syria, with the political unrest and so on, and you may or may not agree with what's going on, but that's not the point, go and look at the software and see what it's able to do.

A much smaller scale one is one called alerts.crowdmap.com.  Samoa, which is a set of basically two islands in the Pacific, had some natural disaster stuff with a tsunami and have had some heavy rains and flooding.

The Samoa site has maybe only 60 or 70 things on it, but you have on there a school with no water, there was a report about a kid being swept down a river into the sea.  That probably didn't turn out very well, but you can use this for relatively fast turnaround stuff.

Those are interesting sites to go to.

The other piece that went on with this software is because this had been worked on in New Zealand with a bunch of dedicated volunteers, they found some things that needed fixing and they patched it.  Then a few weeks later this software was taken to help deal with the tsunami in Japan.  So there is a site in Japan, I think it is sensai.com, which was based on all the pieces of work done on Ushahidi, from Kenya to Sri Lanka and whatever, then worked on in New Zealand and basically moved up to Japan to do the same sort of effort, to help people find things and so on.

One of the worst problems for people in this, I got an email at one stage from Randy Bush, who you all know, and Randy said, "There's a friend of mine in Christchurch and a friend of mine is trying to find him.

Can you help?"

The chances were this guy was OK -- but.  I was able to go to this map and find out some stuff about it.  So it was really good at a personal level.

Just some summary on it again -- I am talking faster than the slides are coming up -- open source.

If this had been a commercial system and you were trying to hack on it, you would have been in trouble.

The real strength in this situation for that.

Those improvements were taken up globally.

The company Catalyst were fantastic.  I remember being in Hong Kong at this time six months ago and the guy from Catalyst, Don Christie, a friend of mine, was saying, "Our people will work on this."  They just did a great job.  This thing picked up and dealt with 300 messages in the first week.

This is also part of a thing called "Crisis Commons".  People are aware of Creative Commons, talking about licensing for data, but this group has actually taken the "commons" word up and gone ahead with it.

We in New Zealand had help from people from all over the world.  It was fantastic.  People volunteered, people who didn't have any connection whatsoever, volunteered.  Crisis Commons is certainly something you should look at and have a think about as part of your disaster recovery stuff.

Locally, we trained over 300 people, to a greater or lesser extent.  140 people worked on this and provided edits to the code and so on.  That's a real success story.

LINZ is Land Information New Zealand, so they are the people who hold all the map data for New Zealand.

The Government of New Zealand has taken a fairly progressive stance on information and all our information is released under the Creative Commons licence.  All the maps and so on, unless they are a sublicence from somebody else, are freely available, along with a bunch of other data.

DIA is our Department of Internal Affairs, so they also have data on where people are and what's doing, and they helped.

The websites had no logos.  There was no glory here, no branding, this was just a plain website.  It also makes life simpler for getting it up quickly.

It didn't look like Civil Defence, it wasn't a government department or an agency of any sort, trying to say, "This is the message."

There were a lot of people who tried to help, wanted to help, but they didn't know about open source so they couldn't.  I don't want to sound like an open source evangelist here, but there is a message here.

There is a high barrier to entry.  People need to be self-driven and motivated to help.  Lots of people wanted to help.  Of course you do.  You look at a natural disaster, and think, "How can I help?"  For some of us, it's a time for putting our hand in our pocket and saying, "Here is some money to help", but if there's tangible stuff like this we can do, it's great.

We did see some issues.  All the work done here was put under Creative Commons, which really means there are some conditions about how you use it, you have to attribute it and so on.  One of our issues was that a number of the official agencies saw what we were doing as some sort of distraction or even some attempt to usurp their authority or whatever.  Christchurch Council took some of the material and claimed it as their own, which was a bit naughty of them.

There were delays in getting the thing launched, because relationships weren't in place.  You are the Civil Defence Authority and a bunch of smelly geeks come up to you and say, "We can help", and they go, "Not so sure.  Actually, we've got this under control, don't worry."  So that's hard.

If you are an official government site, you don't really want people putting up stuff saying, "My cafe is open", because that sounds like advertising.  But in this situation, a cafe being open is exactly the right thing to happen, you can go and get a cup of coffee and something to eat if your kitchen is a pile of rubble.

Some people didn't know they were contributing to the site.  People on Twitter were putting stuff into Twitter and all of a sudden found themselves on the site because people were retweeting it and passing it on.

There are some interesting thoughts in there.

Let's talk about Trade Me.  The auction site is a phenomenal success story commercially, 4.5 million people, 2.8 million people in the population are members of the site.  That's market share to die for.


It meant that eBay really never got a foothold in New Zealand and these guys sell everything from -- they have dating, they have cars, they have jobs, rental apartments -- they cover everything.

These guys have spent a lot of time working on making their systems go like sting, work really fast.

I worked for a company about four years ago, up to three years ago, one of the jobs we did with them in the early days was worked with these guys to work on load balancing and how to make it go fast.

Trade Me saw their message boards jump triple in volume after the quake struck.  They really came to the party.  They built a subsite overnight, they spoke to the government and said, "We will do this."  They made the whole thing totally free, didn't charge anybody for it, and that's 2.8 million accounts, so that's covering almost every household in the country that is connected to the net.

They had little things like, can I volunteer to provide some labour?  Can I provide accommodation?  We had people all over the country saying, "I've got a spare bed at my house in Nelson or Dunedin and if somebody is really stuck and wants to get away for two weeks until this calms down, I've got a free bed."

That's great.

People lost their pets.  If you don't have a dog or a cat, you go, oh, well, get tough.  People were devastated that their cat or dog had gone missing and were trying to find them.  It also talked about transport and so on.

That site is still there.  You can go and look at it.  There was a reference earlier.

They had a lot of traffic.  They did 110,000 listings, 5 million page impressions and 267,000 unique browsers in March.  For the size of the population, those are large numbers.  They found a really tangible way to help.

We have one other aspect, then I will hand over to Dean.  We had a Student Volunteer Army.  One area that was hit in Christchurch was the university.  So classes were off.  We had 10,000 students sitting around going, "What should I do?  How can I help?"

The students were mobilized and were busily chatting on Facebook, "We could help.  We could do something."

There were a couple of figures who came to the fore, took a leadership role and said, "Let's go and help."

One of the interesting things was that people found Facebook more interactive than Twitter and there was no limit to the number of characters.  If you are trying to organize something, 160 characters is difficult, whereas on Facebook you can say a lot more.

GeoOps got modified to deal with thousands of requests coming in.

The students were organized into -- I talked about all this sand coming up through the floor.  Organizing the students to get into working parties, to turn up with shovels and say to a little old lady in her house, the house is devastated and full of sand, "We will dig it out for you.  We might look rough, long-haired lads with piercings and so on, but we will help sort it out."

But the students didn't like being taken advantage of, so think about that when you mobilize people.  What tended to happen was they got all the crap jobs to deal with and in some cases they literally were those jobs.

The council didn't really trust them.  They were saying, "They are students."  That sort of attitude didn't really help.

The message I get from the people involved here was that this was the biggest single thing to do well here.

The building of prior relationships.  Strong relationships that you have with people before this happens are the ones that will stand up during the thing.  People will get thrown together in the disaster and have to work with each other.  But where you have somebody you can trust and somebody you know, if they say they will do something for you, those are the people who worked well.

We have a thing in New Zealand called Foo Camp, which is sort of a weekend camp where people go and have a -- it's like this but much less formal and there is no particular agenda.  People turn up and say, "I have these things I would like to talk about."  It's like a brainstorming session, whatever you want to call it.  A lot of people there were people who were technical people -- we got our Deputy Prime Minister to come to one of them.  So we had met people up and down the chain and those relationships really stood up during this disaster.

I would encourage people in the technical community to engage with people in Civil Defence or whatever, at some point, and try to build a relationship which says: we can help you in a natural disaster.

Civil Defence didn't have that relationship.  Fulton Hogan are a very large construction company based in New Zealand and they worked well with that relationship because there were prior linkages.

A whole bunch of work has been done on a template for this, so the other universities in New Zealand can have a structure in place, should we get another one.

I live in Wellington and I anticipate it.  It will happen.  We have had them there before, right on the fault line, so the university where I work has picked this up and started doing it.

I am happy to answer questions later on this, and Dean will talk some more.

Dean Pemberton: Thank you, Andy.

One of the most important points to get across -- Andy made it earlier -- is that not only did the Christchurch earthquake, the large one, happen almost exactly six months ago while we were sitting in Hong Kong, but the effects of it are still really being felt today.  People still have chemical toilets in the street that they share with their entire neighbourhood, people still have sand all through their houses.  While we have talked about the networking aspects, it's really the whole of people's lives aspects that are still being impacted today.

I want to tell you that I went around and interviewed a lot of network engineers who were both on the ground in Christchurch but also weren't in Christchurch but had to deal with the situation in Christchurch, and that was just as interesting.

When all of your normal systems that you use to do remote management fail, and it's still your job to get everything up and going, how do you build new remote management stuff almost on the fly?

Without exception, everyone I spoke to, when I said, "What was the largest problem?  Did BGP flap?  Were there all these technical issues?"  They said, "No, power."  Power was by far the largest thing they had to work with.  They had about two hours of power after the earthquake, then some of them had nothing for three weeks.

It wasn't just that there were power fluctuations or power went down and then they could sort it out.

Sometimes, three weeks later certain sites still had nothing.

There are backups and diesel generators, but as we will see later, we were getting to the sort of disaster where your backups and backups and backups are failing as well.

One of the problems associated with power is, if you are looking at on-grid power, the grid has to be able to deliver the power to you.

In Christchurch, the main 66kV feed into the city was underground, and the earthquake fault ran across it and snapped the cable, boom.  A huge black-out in the city.

When they finally got around to finding the fault and digging this cable up -- and that's not something they got to do in the first hour, they were worried about getting people out of buildings.  This was sometimes days later they found the fault, and it eventually gets down their priority list.  They dug up the cable, a big break, there you go, fix the cable.

Why isn't it working?  Because 10m further down there's another break.

So they fixed that, then, no, 100m further along, another break.

The earthquake had shattered the cable into so many different bits that it wasn't worth fixing, so they had to go back to Parliament and get an emergency resource consent to build a new overhead cable to restore mains power into Christchurch.

You can see that if you are thinking about your network infrastructure and about how you are going to maintain it, you don't just need to think about how many hours worth of battery back-up you have got, you have to think, what if this entire city is going to be without power for weeks?

That's OK, I'll go out and get a generator.  Cool.

The stories of people stealing generators aside -- and they were very real stories, someone got arrested for stealing upwards of three generators.  If you have got your generator, then you have another problem -- fuel.

The first problem was where is your generator?

Because if you put your generator in the basement of the building where your POP is, that's cool, except (a) you might not have a building; if you have a building, you might not have a basement; if you have a basement, you might not have access to it.

The entire middle CBD of Christchurch is still to this day out of bounds of people.  It's cordoned off.

You can't get to it.

Imagine the CBD in the place where you guys live, cafes, parks, imagine not being able to go there for the last six months.  It's not just the honour system, it's police and army controlled.  So they arrest people that break curfew and go in there.

People's node rooms are in there.  You may not be able to get in to refuel your generator.

Assuming that's all fine and your node room was outside there, all good, but still has no power.  How do you actually keep on getting fuel?

Well, the service stations ran out of fuel pretty quickly.  With all the liquefaction, which is what the silt and sand is called, all over the roads, two feet deep.  We got snow recently which was one foot deep and that caused havoc.  Imagine silt and sand this deep.

There weren't petrol tankers coming in to refuel service stations, so fuel was hard to get.

There was one story where the entirety of the Christchurch area was getting so low on fuel, they had probably about a day of fuel left, so they needed to get more diesel and petrol into Christchurch.

Luckily, there is a port just over the hills from Christchurch, and there was a tanker which had just arrived full of petrol.  Awesome.  Our prayers had been answered.

Unfortunately, they couldn't offload the fuel from the ship.  Now, it wasn't anything to do with the fuel pipeline or not having power, get the Student Army out there to do it with jugs, if you have to.  The problem was that the water main was fractured and there was a policy decision that if you do not have firefighting ability, you cannot offload fuel from the ship.

The fuel was sitting there in the ship, the diesel generators are running low, "Danger, Will Robinson!", and you can't get it.

In the end, they built another pipeline to a river and had a pump set up as back-up.  If there was a fire when they were offloading the ship, they were going to empty the river on to the ship.

In speaking to people, it really became apparent that it was not just your first back-up which has failed, it's not your second back-up which has failed, you are down to plan J, plan Z in some cases.  It really was, what on earth can go wrong now?

In terms of buildings, there really were real cases of sometimes your building no longer exists.  Sorry, but your node in the CTV Building is no longer there because neither is the CTV Building.

In some case, buildings just aren't structurally sound.  I don't know whether people saw a lot of the images of the Hotel Grand Chancellor in Christchurch, but the top of it was leaning upwards of 2 or 3m, it looked like it was meant to be in Pisa.  It was crazy.

The hotel we were all in, two or three years ago in Christchurch, will be demolished.  That hotel will not exist any more.

You can't go into a building to effect any change, to get your router, to repatch, to fill the jinny, if it's not structurally sound; they won't let you.  And if it was in the middle blocks of the CBD there's no way you can get in there anyway.  Sometimes the buildings are sound but you can't get to them.  Sometimes they are structurally sound and have been cleared for access, but the fibre pit you need to get to in the front is covered in squashed cars or full of pieces of the building or full of liquefaction.

In order for anything to work at all, everything had to line up, and pretty much nothing was lining up.

There were massive breakdowns in communications over which of the stuff was working.  How did you find out if your building was OK or not?

When things started coming back on, were you important enough to be in the first wave of getting things fixed?  Were you a hospital?  Were you a disaster centre?  No, you were just for people to read their email.  Well, the Civil Defence didn't put that as high as they should have.  From Andy's presentation, people were getting a lot more benefit from Twitter and their email than from a lot of other things.

A question about the networking, a lot of the telecom suburban exchanges, the old ones, were built like bunkers, no windows, concrete and generators and back-up, and they are out in the suburbs, and those things survived.  Great.

A lot of the streetside cabinets, not so much.  A very small amount of batteries, built on a shaky foundation.  So when the power went out, those streetside cabinets had a little bit of life in them, then nothing.  No one will go around and connect up generators to hundreds and hundreds and hundreds of streetside cabinets.  If that is where you had moved all your active network equipment to, that's dead.

There was also a lot of focus from people about people.  People will step up.  People will take on massive amounts of responsibility and massive amounts of workload.

If you don't have people who are willing to do this, then you are pretty much stuffed.  But make sure that you rotate people out when you get a chance.  People had a lot to think about, some people didn't have homes any more, some people have had to wait until this week to find out whether they are going to be able to rebuild their homes.  People have a lot on their mind and not all of it is in terms of the job you are paying them to do.

Stories abound about people who went above and beyond the call of duty, but if you are managing them, bear in mind that they have homes to go to and you need to swap them out.

Time and again, people relayed that it was the fact that they had to just get up and go.  I want you all to imagine now, if I say, right, get up out of your seat, leave your computer where it is, take nothing, leave this room, that's the last time you will ever see it.

Martin Levy (Hurricane Electric): No!

Dean Pemberton: Will that cause a problem?  There are a whole lot of people here saying, "No, all my stuff is in the cloud, that's fine, I'll go and buy a new laptop, that's fine."  That's good.  But for a lot of small businesses, florists, law firms and all that who were in the CBD, still have not got their computers, so all their customer data, financial records and tax stuff -- the Inland Revenue Department in New Zealand has had to be very flexible on tax returns because some people do not have the data.  If you had to get up and leave your computer and your premises or your building fell down and you never got to see the data again, would it be a problem?  It has been a problem for a lot of people in Christchurch.

In terms of network design, if we build networks to only work on the best of days, then it's almost certain they will never work on the worst of days.  There has been a lot of feedback about how we seem to be building networks closer and closer to the wind and they work but we are over-subscribing them with more and more customers, and it works fine downhill with a tail wind, but the minute it's not the best of days any more, then those networks that sail too close to the wind are the ones that fail first.  The ones where you have only just provisioned enough bandwidth for your peak times during a normal week -- yeah, that's not going to cope at all when you are taking 3, 5 or 10 times the amount of bandwidth in terms of hits.

One of the mobile carriers in New Zealand found that people didn't like staying in Christchurch when there were magnitude 5 aftershocks most nights.  So they moved to the surrounding smaller cities and just overloaded the mobile network.

Those small second tier cities did not expect to have 10 or 20 times increase in population, and it just stressed those networks amazingly.

Keep the networking simple and it's more likely to stay around.  The network operators that had very complicated, very high touch networks, found out that it didn't take very many of those failures to get down the track far enough so that they couldn't manage their networks any more, let alone architect around them.  The people who had very simple networks were able to bring up very simple workarounds and get them going.

It didn't help that some carriers had full change freezes on anything touching Christchurch, which meant making those workarounds was a lot harder.  You call up and say, "I just need to get a temporary link between here and here."  "Sorry, we have got a change freeze in Christchurch.  There was an earthquake, didn't you hear?"

Contractors and on-site access, time restrictions on when you can access buildings.  You can't do it at night, there is no power.  All service impacting changes had to happen during the day.

The contractors who you thought would be able to do this have just found out they have personal issues that take far more priority.

Some things that did work.  Individuals and other organizations helped to re-establish connectivity.

There was a really good feeling within the New Zealand community about: if my company can help and your company needs help, let's just do it, don't worry about anything stupid like commercials, let's just sort it out.

Know your neighbours in the same building.  They can tell you things, especially if you're remote.  So calling up to find out when the diesel is going to run out, maybe no one is answering your phone, but the accounting firm above you will have the same sort of problem, so know them as well.

As I was saying before, guys on the ground went to extraordinary lengths, working stupid hours, despite not having anywhere to go home to -- or maybe that's why they were doing stupid hours.  They had nowhere to go and you would think that was their focus, but they put in some really long hours of work.

The take-away from this is to develop some resistance to disasters.  Disasters are very much like diseases.  If a new one comes along and you have never seen anything like it before and have no resistance to it, you are pretty much at its mercy.  If it's bad, it's going to be bad for you.  If it's mild, you are lucky, but you didn't really have any say in that.

The more resistance you can get to a natural disaster, the better off you will be when one comes along.  Just like a disease, the more your immune system has seen something similar, the better off you will be.

You can generate some disasters to bootstrap this process, whether they are just a planning thought process or a manufactured disaster, or like Andy was saying, sign up to Crisis Commons and deal with some real disasters.  They don't have to be in your backyard for you to be getting some experience here.

We have all seen stories of the survivalist people who live in the wilderness and have their bunker with 100 years worth of canned beans.  You don't need to go to that extent in order to be able to survive a natural disaster but you should have at least been camping once.

If you have never done anything then you are at the mercy of the disaster.  If you have done something then it's better.

I have some examples from Foreign Affairs, but I might leave them until the panel discussion.

Thank you.


Philip Smith: Thank you very much, Andy and Dean.  Now we have Katsuyasu Toyama, who will carry on the presentation.

Katsuyasu Toyama: Good morning.  My name is Katsuyasu Toyama of JPNIC and Tomoya and I will give a talk about the Japanese disaster.

My slide here is based on the presentation which was made by Dr Ishida of JPIX and I.  We made a presentation at the JPIX meeting the day before yesterday.

First of all, I would like to thank you very much for every heart warming message and support from all over the world, especially immediately after the earthquake occurred.  We had a lot of messages from you, asking about ourselves or whether our families are safe or not.  Those kinds of messages are very helpful to us, because we feel encouraged, so thank you very much to all of you.

The outline: I would like to talk about the overview of the Japan earthquake and the importance of the Internet working, our Internet exchange and data centres, and domestic backbones and submarine cable lines and access lines, which will be the presentation by Tomoya.

The overview of the disaster.  The earthquake happened at 14:46 on 11 March and that was a magnitude 9.0 earthquake.  The earthquake occurred at that point in the red area, which was severely damaged and shaken, and more damage was caused by the tsunami, as you know, which came to the coast, about 20 minutes or later after the earthquake.

The tsunami was 15 metres or higher in some places.

Maybe you have seen some pictures on the website or video and YouTube, of the hospital which has four storeys, was almost covered by water.

It was a very huge tsunami which attacked our land.

This slide shows the damage situation from the weeks before, which were released by the National Policy Agency.  The amount of damage to people, 15,000 or more people were killed and we still have more than 4,000 missing people.  A lot of people are refugees and they do not have their own house and are still now living, staying in some evacuation place, still 100,000 people are in such kinds of shelters.

Amazingly, 92.5 per cent of the people killed were drowned.  This data is slightly old, but almost 90 per cent of those killed were drowned, and this shows that the tsunami hit and caused very hard damage.

On the other hand, I would like to talk about the telecommunications in Tokyo on the day of the earthquake.  We could not use mobile phones for voice, that was useless.  The outgoing voice calls were restricted by the carrier, so we could not call our families or our companies.  Also the short message SMS did not work properly for an hour or two.

Later, we got a lot of messages on the phone, but it was useless.

On the other hand, mobile data communication worked well.  Also, the PCs in offices which were connected to the Internet worked really well.  So people relied heavily on the Internet, just like SNS type of things or Twitter, Facebook and the Japanese SNS site, mixi.

Also, people were watching the streams, on Ustream or Nicovideo or YouTube.  So people thought that the Internet was the only medium that survived.

The other biggest problem was in the nuclear power plants.  The Tokyo Electric Power Company, which we call TEPCO, had the Fukushima Power Plant Number 1 which was washed by the tsunami.

This map shows on the left-hand side the map of Japan.  I am sorry, but it is a very small one.  The red cross shows the Fukushima Power Plant, which is located 200km or more from Tokyo.

On the same side of the map, Asia, maybe you can imagine in your country how far from Tokyo the power plant is.

At first, the earthquake occurred, the power plant automatically suspended the plant and immediately started the cooling system with emergency power generators.

But then approximately 40 minutes later the tsunami reached, approximately 10m or more, and that washed out the emergency generators and the facilities.  That caused a big problem.

The impact of the nuclear power failure was very huge, because the TEPCO service area is not only in Tokyo but also the surrounding prefectures.  In that area there are a lot of residences and a lot of factories.

TEPCO generated power before the earthquake of approximately 60,000MW per hour, but immediately after the quake the number reduced to almost half.  Currently we have about 50,000, just recovered, but still we have a shortage of power.

In March, we had a planned outage, a blackout, of the power.  The blackout area was rotated one by one.

TEPCO segmented some areas, three hours here today and after that three hours in this area and that area, so it was a rotated outage which occurred.

This had a very huge impact on the economies, because in the factories and hospitals they cannot cope with a planned outage.  So we are very heavily relying on our electric powers, so the few hours of outage had a very big impact on our economy.

Gradually, our lives returned to normal, but still the visitors from abroad are not so many.  Also, our huge effort to save energy, especially because Japan is very hot and humid in the summer-time and these days, we have a lot of air-conditioners in buildings and houses, so we have to save energy.  We have tried very hard to do it.

Air-conditioners, setting the temperature at 28 Celsius, so very hot and humid in the office, so I cannot work well inside the office.

Some companies are forced to shift their holidays into weekdays, so they work Saturdays and Sundays, but instead, for example, Wednesday and Thursday they have off.

Moving on to the Internet working part, a summary of the impact of the earthquake is described here.  Almost all services and servers survived and also the Internet IX/IDC survived.  Because the main services and servers are in the Tokyo area, and in the Tokyo area the datacentre was very good, so the services survived.

But the domestic backbone, especially in the north-east area, and the submarine cables, and of course the access lines in that area were very few damaged.

Traffic wise, Renesas reported these kinds of figures.  This is the traffic status at JPIX.  So after the earthquake, on the left-hand side, all the traffic was immediately reduced or decreased.

This graph is JPNAP, almost the same.  The black and purple lines shows a week or two weeks ago the traffic, and the green one shows the traffic on the day of the earthquake.

The same, it is steeply down after the earthquake.

Dr Ishida said they recovered almost one week or two to normal, but in JPNAP's case, it took almost one month to recover to normal.

So earthquake damage on the IX datacentres was comparatively negligible.  Our switches worked well, but the customer traffic vanished.

Also, we had some minor issues, for example the patch cables were loosened, and the optical cables were degraded because of some kind of damage to the optical fibres.  But they were minor issues, there was almost no impact on the traffic itself.

Why no big damages?  This is because seismically isolated buildings and also we carefully mounted the equipment on the racks, and we prepared the UPS or generators with securing fuel, and of course we have circuit diversity, and the most important part is securing human resources.

This is a typical Japanese earthquake resistance structure.  On the left-hand side is the earthquake-proof structure, which is the weakest among them, and the damping structure.  These days, the new datacentre which was built 10 years ago is mainly a seismically isolated structure.  This is very strong for earthquakes.  The building can shake for a very long time, very slowly, but the equipment in the racks are firmly bolted, so there was no damage to the servers.

The biggest problem for us after the earthquake is the outage.  Especially in summer, we have to save energy.

In the planned outage, when we had the planned outage, we had to check the UPS and generators and also we had to secure the fuel.  We have a prioritized contract with the fuel companies, but maybe the traffic is congested or such kind of things, so securing the fuel is very important.

Also, in the messy situation, we have to secure or gather the human resources, people and operators and engineers had to go to the places.  So these kinds of things are very important.  We had to prepare for it.

What we should do for the future.  Of course, we are doing that.  When we got to the area exchange, avoiding a single point of failure, of course, and also we need some geographically dispersing area exchange.  We also have that point, so that ISPs should connect to the network in Nagoya and Osaka and Tokyo, which will help the Internet traffic supply.

Also, the datacentre should be strengthened against disasters and diversifying into other regions, those kinds of things are very important.  OK.  Next is Tomoya.


Tomoya Yoshida: Good morning.  My name is Tomoya Yoshida of Internet Multifeed.  I am sorry this is a sad picture, but this is real in Japan.

At first I would like to say thank you very much for your great support from all over the world.  It is very helpful for us.

We should remember this earthquake, so we remember this disaster and move forward.

At the time I was in my office on the 17th floor, in my last office, and I was thinking about I should share and notify our situation to all over the world.  So 33 minutes later I sent an email to this NOG list.  Why I chose this NOG list is because at the time I did not have much time to decide which mailing list is better, so I just used the NOG list report destination address.

So I picked up the mailing list and I posted, just using one sentence, "Japan had so big terrible earthquake."

Afterwards, so many people sent email, usually on the mailing list, and after update the situation, so that was very nice.

Nine minutes later, some JPRS guy, Morishita-san, sent to me, "JP DNS is working."  So that was very nice.

Afterwards, he sent it to the APOPS mailing list, that all the JPRS servers were still fine, so that was very nice.

In the 40 minutes after the quake, we had the great east Japan earthquake, and 30 minutes later we had a big aftershock.

I think in the middle of the aftershock I sent to the NOG mailing list, and Morishita-san said JP DNS is working.  But between this event, I did many things.

I called my home but not available, sent email to my wife on the mobile phone.  But Skype was available, so I could each my home using the Skype-out, which was very nice.

After that, I sent an empty email from nttv6 domain to ntt.com domain, just checking whether mail of ntt.com could be received on the Internet.  That's why I sent the empty email from one domain to the other domain.  We had a big aftershock, and after that I sent a message to my parents, "I am OK", sent to the NOG mailing list, and received "We are OK" from my parents, and afterwards the message from Morishita.  So in the 40 minutes I did many things, this my log at the time.

The traffic, at the time, I will show you about the percentage of traffic.  Toyoma mentioned the JPNAP traffic and JPIX traffic went down after the earthquakes and one or two weeks later it slightly recovered to normal.

These are the BPG routes statistics.  We have withdrawn inside the northeast area and also the BPG routes are recovering one week or two weeks later, but we had many withdrawn BPG routes.

This is the traffic of other disasters.  On the east side we had about 30 per cent jump of the traffic, and the bottom is the IX traffic.  At this time, this shows the JPNAP traffic and the other ISPs traffic, and it is the same decreased ratio.

We can see on the west side, some lines is also the same decrease, because they adopt the closest existing routing, so just go through the packet in the west area, so that packet is going to the Tokyo east north area, so this packet doesn't slow the traffic down as well as the east side.  Also, some other graphs show the jump up.

So this could be the back-up traffic from east side to west side.  It was very nice.

The traffic decreased a little bit in the west area.

So there is almost 30 per cent decrease after the quake.

Why this happened, the big impact is the influence of the blackout, so the most area that influenced the blackout, I think one day or 20 hours there was many big impact of the blackout. But now it's almost recovered.

This is the cut-off of a domestic relay cable of the NTT Com.  We have three routes from the Tokyo area to the Sendai area; one is the Japan Sea route, the middle route, and the Pacific Ocean route.  This time we lost two of the three cables.  I didn't believe at the same time the cable was destroyed, but this was real.

After a while we recovered bypass fibre across the telephone pole, so now we already have three routes.

But in the Pacific Ocean area, this has caused nuclear power plant issues.  This is not normal, but already we think about fibre.

This is the detail, and after the quake we moved to the Miyagi Prefecture by helicopter the next day and we specified the broken spot and arranged local workers and articles.

On the 13th, two days after, NTT Com specified the specific cut-off spot.  They set up the bypass fibre between the telephone poles and also set up a pipe next to the manhole and the cable was constructed to a telephone pole.  So this was NTT Com's restoration of the cable.

This is the submarine cable outage.  Japan-US, APCN2, China-US and PC-1 are injured, but they were restored in sequence.  Red one is already almost recovered.  Blue one is not influenced at that time.

But many cables were cut.

Also, in the case of KDDI it is almost the same, one operational area and one damaged area, and that is similar to NTT Com's case, and I believe the cable is already fixed.

This is the damage to the access line.  This is a summary of the numbers, compared with other areas.  The red one is on the earthquake and the second one was the big earthquake in Niigata Prefecture and the other one was the earthquake in the Hanshin west area.  You can see the numbers are totally different.  Especially you can see the telco buildings, and this time many telco buildings were destroyed.  Also, the cable length is 8,000km, which is a very huge distance.

This is the NTT Telco Building destroyed by the tsunami.

This picture is from Akiyama-san of NTT East, and this is inside the building, which is destroyed very much.

The mobile lab or equipment cannot be used.  Also, we are repairing the building and installing the electric power equipment, and also the reinforcement of the outer wall.

This was at night, so they used headlights or a lamp.  The power is very limited, so the power should be used for the equipment, so they used just a light or a lamp.

TEPCO is now recovering very successfully and also the access line is being repaired.  But the opposite side is not yet.

We are concentrating on the fibre or power for the NTT building, but this side is not good.

This is the last slide.  I think we have many issues, and the next step.  One is more layer-1 level redundant design.  Of course, we already assigned the layer-1.  As I mentioned, the three parts we have, it depends on the service or the service level.

I discussed about KDDI guys or the software guys and other guys, so they can do more and more design, so we need more reliable design.

Desperation to Osaka: the backbone level is almost desperation, it was OK, but the content level, so that most content is still in Tokyo only.  Of course, we have much content all over the country, but much of the content was in Tokyo, so we should design a model for redundancy.  Also, the application desperation should be needed.

And the traffic control at the disaster, so that Twitter or Facebook or social networking was working very successfully, but some people still see video or YouTube, of course they want to see.  But we should, for important packets we should have first to move forward.

So the traffic control is one issue for us for the disaster.

The desperation of operation, mainly we operate in Tokyo but if we have a disaster in the Tokyo area, how can we operate the Internet in Japan?  So this is one issue I think.

Also the Internet Disaster Simulation Day, this is something like an idea that we should simulate for the future, so this is one of the issues and the next step.

Thank you very much for all of your support.

Thank you.


Philip Smith: Thank you very much, Tomoya.

We have only five minutes left for questions from the audience.  Do we have any questions for any of the panelists or is it still too early in the morning?

No questions.  OK.

Andy Linton: One observation that came out of the New Zealand one, and I was catching it listening to the other speakers talk, is that our Civil Defence people and Canterbury Council and so on have understood the value of some of this stuff, so they have encouraged their Civil Defence people to sign up for things like Crisis Commons and do this as part of their job description.  That might be a message that you can take away, that this would be something that could be really valuable.

Masato Yamanishi: I am maintaining our international backbone, so let me add additional information about the submarine cable.

Some rumours said the reason why the cable was cut was the cable station was sunk, but it's not correct.

It happened by cable cut undersea and that cable cut was caused by sand slide under the sea.  After the big earthquake, a sand slide happened and it flowed, normally around the ocean trench, and it is very slow but it is very powerful.  Once it leads to the cable point, it pushed the cable and cut the cable.  That is the reason why the cable cut happens after a big earthquake, and this earthquake was very big, so the sand slide was also very powerful.

As a result, the last cable cut happened 11 hours after the initial earthquake.  It means the earthquake was very large and it was very powerful.

Philip Smith: Any comments from the panelists?

Dean Pemberton: New Zealand didn't see a lot of undersea cable cuts, mainly because Christchurch is nowhere near where any of the cables go.  But Andy and I were saying before that New Zealand is very at risk from this.  If the natural disaster had happened anywhere around Auckland, most of the cables in and out of New Zealand come through Auckland.  As a country, we should start thinking what would happen, and maybe saying that we should bring cables in somewhere else, even if it doesn't make amazing financial sense.

Andy Linton: We also observed, looking at the map for Japan, nearly all the cables come in on the east coast, but apparently very little comes in on the west coast.

It's a similar question.  If all your eggs are in one basket, and it doesn't matter whether it's cables, datacentres or any of these things, redundancy is the key to this stuff.

Dean and I were talking about having stuff that isn't overly complex, the simplicity allows you to patch around things.  We were watching when you said that you patched around some problems and put up some stuff up a telephone pole, we had those same issues and those are the things that helped.

Xing Li (CERNET): Thanks for the presentations.  Back to 2008, there was a big earthquake in China and we learned two lessons.  The first is air-conditioners.  Sometimes you have the power for routers, but not air-conditioners, and that was shut down.

The second and extreme case, only the satellite can help.  Others won't help, so solar powered satellite is helpful in an extreme case.

Gaurab Raj Upadhaya (Limelight Networks): We know about the really massive earthquake in 2007, when lots of fibre near Hong Kong and Taiwan was taken off line, then we saw a raft of new investments in alternative cable paths going through the Philippines out east and west.

At that point I looked at the new fibres and realized that, holy cow, they all go to the same old cable landing stations in every country.  Literally, for those of us who work in this industry, you can go and point on a map to the four cable stations most critical in the entire region and if they are gone, it would be a far worst disaster than sending ships out to repair the cables at sea.

That is probably something we should take out of these events.  Probably when we are designing new submarine systems, if they are not already on the ground, we probably want a bit more diversity in cable landing stations.

Having said that, a question for the Kiwis.  The new Pacific fibre talked about at the last NZNOG, which gets a lot of news, are they changing the landing station to somewhere other than Auckland yet?

Dean Pemberton: They are in the process now of finalizing all that.  I believe at the moment it is still Auckland, but I will have a chat with them and see.  I think it really is a good point.

On an Asia scale, building into those four places might make perfect commercial sense, those are the easiest places, those are the places where all the ops work, but again it comes back to building the networks for the best day you are having, rather than thinking about it for the worst day you are having.

Just like insurance and backups and all the stuff you have to pay for that you don't get immediate benefit that day, what's the trade-off between paying a little bit more at the start, but when you need it, it's there.

Andy Linton: If you move up the stack in New Zealand, one of the sites that failed really badly was the Red Cross.

When you are saying to people, let's send donations to help with this, and they couldn't take the donations because they had built their site for the best day, not the worst day.

Miwa Fujii (APNIC): Thank you very much for an interesting presentation.  I would like to make a personal comment to Tomoya-san's presentation.  I was in Brisbane, but I am originally from Japan and I had friends and family members and was really worried about what was going on.

As both of you presented, the mobile phone was the first network to go down and I could not communicate with anyone and the Internet stayed to the end and I think it remained quite robust.  Most of my information about this devastating earthquake and nuclear power plant disaster came from YouTube, Skype, Twitter and Facebook.  I knew, from working behind the scenes, I could visualize every one of your faces, including Yamanishi-san, I was communicating with him too and he was working overnight to fix the submarine cable.

I want to say thank you very much for your great help.  The world was watching you and we appreciated your network engineers' effort, who worked really, really, really hard during the most difficult time.  So thank you so much.

Philip Smith: Thank you.

I want to wrap this up.  I would like the panelists to give a 15 second summary, if possible, so we can go off and have coffee and the Secretariat can get ready for the member meeting.

Katsuyasu Toyama: This earthquake reminded me that our communications system relies heavily on electric power, so we have to think about how secure the electric power is or other power.  Thank you.

Tomoya Yoshida: Thank you very much for having this opportunity to share our disaster.  I think just we move forward, but in Japan we had many disasters past or future, so we need to have reliable Internet.  But at the time the Internet was working, so that's very nice.

Thank you very much.

Dean Pemberton: I think this process has really taught me that when I'm designing networks, normally I would go, if that fails we will go to this, if that fails, we will go to this, and any more failures are unlikely.  I have now learnt to expect the unexpected and there are a whole lot of disasters which have the capability of meaning you need a lot more backups or a lot more flexibility, so you can make the backups on the fly, and allowing yourself that flexibility.  Thank you.

Andy Linton: I think the thing I got out of this was that this idea of having community, an event like this will pull your community together, it will be a massive disruption, but having the predefined links or prebuilt relationships beforehand will allow things to happen much more quickly.

My favourite quote from all this was somebody talked about "random hacks of kindness" -- people doing some hacks which made a difference for people.  I suppose that was the big one, my favourite quote out of the whole thing.

Philip Smith: Thank you very much.  Thank you to the panelists, Katsuyasu Toyama, Tomoya Yoshida, Dean Pemberton and Andy Linton, I appreciate your contributions.

Sunny Chendi: This was the last session in the agenda.  We tried to figure out what to do, and we came up with this topic and thought it was very topical to Busan.

The APNIC Secretariat appreciates the speakers for agreeing to participate in the panel.  I would like to request you all to give them a round of applause, and to Philip Smith for chairing the session.


Sunny Chendi: Now I would like to request all the participants in this room to take their belongings and move out for morning tea, because we are going to change the room layout for the APNIC Members Meeting.

When you come back at 11 o'clock, the seating arrangement will be a little bit different for the members meeting.  Thank you for your support, and we will see you at 11 o'clock.

Key Info


Paradise Hotel,
Busan, South Korea


28 August -
1 September 2011

Program included:

AMM, Policy SIG, IPv6 plenary, APOPS

Please wait...