Nir Simionovich on EC2

Event:

There is a full text transcript of this hour below.

Part 1: (Talkshoe) [audio:http://media.blubrry.com/winelover/recordings.talkshoe.com/TC-22622/TS-224823.mp3|titles=Nir Simionovich Asterisk on EC2 Instance]

Part 2 (ZipDX recording served from CloudFront): [audio:http://cloud.vuc.meZipDX20090619.mp3|titles=VUC Part 2  Fridays at 12 Noon Eastern Time]

We’ve tried to get people to talk about this before, but we didn’t get a deep explanation. Here’s a chance to ask Nir, who did a compelling presentation at AMOOCON on Asterisk on EC2, to clear up any of those nagging questions you might have.

Nir’s AMOOCON presentation, Dynamic Asterisk Scalability with Amazon EC2 and videos are available on the AMOOCON site. Nir’s company is Greenfield Tech.

Nir is also the author of Asterisk Gateway Interface 1.4 and 1.6 Programming

Jerry Shuman of Perssonas (@theagent) joined the call with some great input, too.

IRC Transcript 2009-06-19

Follow on Twitter

@voipusers @e4voip @mjgraves @viperdudeuk @steely_glint @teamforrest @fredposner @asteriskbot

Transcription

Nir Simionovich: I have to admit that the first person that got me heavily into EC2 is actually here and it’s John Todd. I’ve been dealing over the past two years a lot with Asterisk and Virtualization mostly VMware, and one day on the list there was this question about Asterisk and VMware if it’s doable, what is the performance, and I actually replied to it, and John came up with the idea, “Well, there’s been some talk about EC2 and Cloud computing,” and I said, “Well, why not? Let’s give it a try and see what we can do.” As time progress and what’s really progressively fast00 was the time of the Israeli elections and I was working with a company in Israel that was doing outbound dialing to get people to come into to vote. So, we ended up building the entire infrastructure of EC2 because in order to generate as many calls as we needed to, there was simply no infrastructure big enough here in Israel to hold like 32 servers. So, we ended up opening multiple EC2 accounts and doing everything from EC2 and it worked really well and that actually sprung everything into like real rapid motion and it got me heavier and heavier into EC2.

Zeeek: I think the first question in most people’s mind before they know anything about EC2 and I don’t know how many people know much, I don’t know a lot. I know about S3. I’ve used it. We use S3 and CloudFront of services extensively as somebody just posted the Amoocon talks, that nearest talk is number 27 and if you’re looking in IRC, you can see that, the link, otherwise, just find the talk number 27 on Amoocon.de.

Anyway, Nir, the first perception of EC2, you have to wonder because for years, Asterisk has been this full thing. You’ve got a Linux box and you download it and you install it. It’s fairly easy to install if you have any experience in Linux at all, and then the thing is working and you got it on your machine and you play with it, and you know, it’s free and blah blah blah, but when you’re doing an EC2 Instance of it, I guess one of the main reasons to do this is because it’s inexpensive but it’s extensible, right? During the elections if there were thousands of thousands of calls being made, you’re not going to be able to do that on your old 386 that you threw together, right?

Nir Simionovich: No. That would have been impossible. The thing was that we actually needed a way to expand really fast. Now, we have an infrastructure here in Israel of servers. We have like six different tools beyond servers and because we were doing also media and outbound dialing and we’re handling a lot of capacity here in Israel, we got limited by the number of concurrent calls we’re able to per machine. So, sure we’re able to push it up to about 240 concurrent calls, 260 per server. So that wasn’t big deal but we needed more and the idea of using the EC2 just sprung up saying, “OK, let’s see what we can get out of that specific system.” So, we ended up going over there initiating what’s called the medium AMI. That’s AMI based on, I believe it’s a dual core and it has 8 gigs of RAM and 160 gigabyte of disk, and we created our own AMI image, sprung those up and each one was able to get about 120 concurrent channels on it with media and we’re really happy because we could now expand as much as we want.

As you said, it’s pretty easy to get Asterisk up and running on your own machine. It’s pretty easy also to get Asterisk up and running on Amazon EC2. The main problem after you get it up and running is to make sure that the work that you’ve done actually stays there because the one thing that, for me at least, is pretty annoying about Amazon EC2 is that once you bring up an instance inside EC2, if you shut it down, it will simply go away. It will get deleted. So you have to maintain your work somewhere else and going about on expanding your system with EC2 requires a fairly new skill set that most Asterisk developers currently don’t yet have, but it’s like completely mashing up the entire voice world that Asterisk is pretty much dominant in the web world where most, let’s say, IVR developers are not that accustomed to working with.

Fred: Hi, this is Fred. I was wondering on 260 things, you said that you’re limited to about 260 calls per server and that seems a little low for a dedicated hardware back and I was wondering what kind of setup you were using or what the limiting factor was.

Nir Simionovich: Well, actually the limiting factor was not Asterisk itself. It was actually the number of file descriptors we had opened on the box and the network that we had, and the boxes themselves were just dual cores. These were old IBM x335 servers and we were getting about 240, 260 concurrent calls, concurrent channels with media which were interacting with the database and external sources and the application itself was pretty heavy. So, we ended up running into a load average on the box itself around four, 4 ½ and we didn’t want to breach that one. We could have gone higher. That is for sure. That shouldn’t be a problem. Yeah, we could have gone higher to about 360, 400 but we didn’t want to. We were afraid of actually harming the quality of the call.

Man 3: Yeah. I’m curious if you’re using Amazon Elastic Block Store for persistent drive space for any of this architecture and if you’re noticing any sort of timing issues with it?

Nir Simionovich: Well, I have to admit that we hadn’t used it. We didn’t see any reason to do so but the minute I’ll do a test of that, I’ll be sure to publish those.

Man 3: Thanks.

Zeeek:  I can kind of see what EC2 is about but as far as Asterisk goes, first of all, I assume that someone has—you have the possibility now just finding instances that are already built and kind of throwing them up there? In other words, rather than starting from the beginning, someone’s already done this work, correct?

Nir Simionovich: That is correct. There was a lot of work done by Eric Chamberlain from Voxilla. The main issue that EC2 suffered from, the main kernel issue that each of the ES, let’s say, the standard AMI images that EC2 used was a different clock being compiled into the kernel and that caused issues with things about, let’s say, conference calls. It will just come up wrong. What Eric did is create a new image that is based off a 1,000 hertz clock which is pre-set. I think its 1,000 kilohertz, sorry. The thing was that—what Eric created was something really optimized for Asterisk to be working off Amazon EC2 and that made a lot of things in a lot of the work. It took a lot of the edge of working with Asterisk and the EC2 because it came in a pre-packaged AMI. You can just take it, you can modify it, whatever you want, and you can get your systems up and running really, really fast.

I know of a few companies that actually managed, like these virtual call centers. What they do is actually, they initiate small instances of EC2 according to the customers they have and what they do, they just initiate the instances as they go along. That means that if somebody right now is working and they need like four hours to do an outbound campaign or 10 hours to do an inbound campaign, then they’ll just bring out a proper EC2 image saying, “OK, work off of that,” and they will just rented you a call center per hour and it’s a completely new way of thinking about how to do call centers and how to visualize PBX systems and host PBX systems. So it brings a lot of possibilities back into the mix.

Zeeek: Jim is asking you, Nir, in IRC about how you distribute the load among multiple EC2 instances. It’s a good question. I’m trying to picture this stuff and it’s pretty foreign to me.

Nir Simionovich: Well, basically what we did, we used carriers that we’re able to distribute the load according to what we needed. We just gave them all the IP numbers, all the public IP numbers of our EC2 Instances and they actually distributed the loads for us, but we could have also easily taken, let’s say, an open sear, put it in front of the system and use that to distribute the load to each of the instances.

Zeeek: Other question I had was with regard to configuring. So, I don’t even understand how you access this. This looks to you like you’re going to login to it and it just looks like any box, is that it? You go with SSH into it?”

Nir Simionovich:  …basically, Amazon EC2 is based on Xen.  So, essentially what you’re getting is, you’re getting your own Xen domain.  Basically, imagine that, if you’re familiar with Xen, then imagine that EC2 is like this enormous Xen dom zero that operates, and whatever host, whatever AMI you issue, or you initiate, you go about and you actually create your own dom inside that, let’s say that big cloud.  Now what you get is your own virtual machine.  So, you get a machine that you can SSH into it, and work with it, and do whatever you want.  It behaves exactly like a normal machine.  Sure, you have no control, well, you have some level of control about where it is located in the world, in terms of geographically saying, “Okay, I want to be in this specific data center or in that specific data center”.  But, again, that level of control doesn’t give you a lot of ability to say, “Okay, if I initiate two, or four, or five different AMI instances”, that they will be located in the same area.  And that is the main problem with using EC2, because the normal, let’s say, coupling of servers that we are very much accustomed to when building Asterisk environments doesn’t exist there.  There is no ability to say, “Okay, if I initiate database, then the database is sitting right next to the application, and there’s no latency between those two.  So, if you do that, that’s kind of a hassle.  So, you need to go about, and say, “Okay, we need to decouple our application.  We need to decouple our information storage”, and say, “Okay, we’re going to move everything outward, into a completely new location, and have Asterisk work off of that”.  And that requires new skills.

Zeeek:  No question about that.

Nir Simionovich:  By the way, somebody on the IRC just said that Open Series is now called Open SIPS, that’s not entirely correct.  Open Series is now referred to as Camie-Camiellio[sp?], and Open SIPS is a fork of Open [unintelligible], so these are two different things.

Dean Collins:  Randy, it’s Dean Collins.  I just wanted to jump in.  So, are you going to set up a consultancy practice offering this type of technology, and if so, what’s the URL for your company, so people know how to get in touch with you?

Nir Simionovich:  Well, okay.  The easiest way to get to my website is [omitted, since it was an incorrect address].

Zeeek:  We’ll have to remember to post that in the…

Nir Simionovich:  Yeah, well actually, it’s…

Zeeek:  …BBC site.  Oops.  Ooh, bad URL.  Bad.

Nir Simionovich:  [laughs]  Yeah.  Or, if you’re able, well, because most of the audience here are English speakers, it’ll be a lot easier if you go into www.greenfieldtech.net, then you’ll get to the same place, it seems.

Zeeek:  How did John Todd get involved in EC2, by the way?

John Todd:  Because it seemed to be fairly obvious, that in order to do something really big that scaled, that EC2 is a good platform for it.  There have been, previously there have been some people doing work with EC2 doing experiments, really.  But kind of a formalized test with actual results that’s been kind of lacking.  We still don’t have really good, quantifiable tests with EC2, saying we can handle this many media channels on a particular instance, before we start to see pattern problems.  And, well, really, the whole point of my talking with Neer[sp?], actually, and a couple of other people, was to see if we could get some quantifiable results on Amazon’s network.  Because, of course, Voice Over IP is very subject to things like packet loss and jitter.  I wanted to see if Amazon’s network was going to hold up to Asterisk, or RTP, in general, operating over their fairly large infrastructure, and all, I guess, non quantifiable reports seemed to indicate that it does work just fine.  But I still am having…

John Todd:  I don’t have any actual numbers saying that, you know, for these three days, we tested RTP streams, and we saw this packet loss to this provider.  My goal is to get Amazon to offer, and this still may happen, we had some brief conversations with the folks at Amazon, but I’d love to see them offer a service where service providers, meaning PSTN termination providers, could interconnect with Amazon’s EC2 infrastructure at no cost, and those voice packets would be transmitted at no cost to the instance users.  In other words, don’t pay for the bits in and out of Amazon’s network if you’re using this particular service provider, who is bearing the burden of network interconnect.  That would make this much more appealing for VoIP services, and application providers to outsource their activity to Amazon, and I think that that’s really just waiting for kind of a critical mass of EC2 infrastructure stuff to happen…

Zeeek:  And of course, you were channeling Don Quixote, as always, John.

John Todd:  Yeah, to some degree.  But actually, Amazon is pretty clueful, if they see this as a way to increase the number of hours people use the machines, then I don’t think that, you know, their network costs, I think, are pretty close to break even, unless you’re doing something like video.  But I think anything you’ve got to do to encourage people to get onto their system, they would be interested in.  One of the big services that is based on EC2 and Asterisk is Toolio[sp?].  All of their infrastructure right now, from what I’ve understood them to say, is that it is running on EC2, scalably, with Asterisk as the back end.  And, as a matter of fact, they’re going to be giving a talk about that at AstriCon this fall, in October.

Zeeek:  There was a question about, and a very good question from Jim, about bandwidth cost calculations.  Now, you did give us some idea, Neer[sp?], during your presentation about the costs.  Why don’t you run over that a little?  Whatever you can, or want to reveal about the cost of the campaign, some of the numbers.

Nir Simionovich:  Well, the calculation is based on…well, okay.  Let’s start from square one.  Amazon EC2 economics, the way they calculate your usage is a little different.  That means that they calculate your usage according to your instances.  For example John, I’ll just punch it up for a second.  For example, they would go about and say, “Okay, for every gigabyte that you consume, up to a certain level, you will pay something like, let’s say a cent”.  Now, if you know that specific calculation, and you’re able able to calculate what is your projected traffic, and you can calculate you’re projected traffic according to the codecs you use, and how many concurring calls you have, and how many sessions you…and how much time you will be operating the system, you’re pretty much easy to say, “Okay, I can go about and say, okay, I’ll be utilizing, let’s say 800 gigabytes over the next two weeks of traffic”.  So, you’re able to say, “Okay, 800 gigabytes of traffic, multiplied by one cent, that’s eight dollars worth of traffic”.  So, that’s how the math actually works.  Now, to say that there is a calculator that enables you to calculate that exactly, it’s a little hard to say.  I don’t think that one exists, but the math, in itself, isn’t that complex to do.  In general, just use any type of bandwidth calculator that you’ll be using for any VoIP application, and just calculate your, let’s say, megabit per second ratio, and once you have that, just multiply that by the number of seconds you have over a course of your operation, and you’ll have your number.

Zeeek:  It’s interesting, Dave EG[sp?], from EGW, we have a customer who uses EC2 for their telephony apps, multiple Asterisk instances, front ended by Open SIPS, you have some logic which brings up and down additional Asterisk instances for peak load times, which is basically, Neer[sp?], what you did during that election campaign.  Interesting.

Nir Simionovich:  That’s exactly the same.  Same idea.

Zeeek:  Yeah.  No question, this is great for things that you pointed out, and one of those was, hey, you’ve got this project that lasts, what was that, three days, is that right?  Something like that?

Nir Simionovich:  It was three days worth of dial out, and it was operating about ten to twelve hours every day.  So, we got out about 36 hours, 38 hours worth of dial out.

Zeeek:  It would be incredibly hard to put together a hardware platform, and stupid [laughs], to put together a hardware platform for three days, unless, you know, you worked in a hardware store, or something, to do this project of over a couple of days.  Even if it was a few weeks, whereas this is one of the things where…

Nir Simionovich:  That’s exactly the case.

Zeeek:  The whole EC2 thing shines.

Nir: This is exactly what it pays. Awhile back, John and I did some mass going about and saying, trying to understand if there is a validity to actually going about and running a host PBX on top of Amazon EC2, and we realized that if you operate for 24 hours, a full system is operating 24 hours, then there is no real usage for Amazon EC2 as is. If you’re a business which is doing a hosted environment, then Amazon EC2 is the best you’ll find, really. It gives you that immediate quantifiable capacity that is really, really fast,but if you’re on business, let’s say, if you’re in an office and needed a PBX system, then Amazon EC2 isn’t really the thing for you because you can get an equivalent server with inside dedicated hosting facility today for like $59 a month and that one already has 16 gigabytes of traffic with it, maintenance, managed hosting and so on and so on. So, some services make sense working off of EC2, some don’t. If you’re going to say, “OK, we want to move into the Cloud,” that specific move has to be calculated and has to be tested, let’s say, economic wise.

Zeeek: The agent has posted—now, I’ve seen calculators for S3 before and this is it, right? Well, I’m going to read this because I don’t have time to make a short URL, Calculator.S3.AmazonEWS.com/calc5.html.

Nir: That will give some insights as to how much money you’ll need to pay.

Zeeek: I think it’s worth trying a project, maybe messing with it to see. So once you’ve got your instance ready to go, I mean, if the elections are held next weekend, are you ready to bring that same instance up? What’s involved in bringing that back to life? Can it be done?

Nir: Yeah, it’ll take about five minutes.

Zeeek: So where is this thing stored? I mean, it’s in your account and you just click, you just check a box and click submit and you can bring it back up or is it something locally stored somewhere with you or what?

Nir: It’s exactly how you described. What you do is, once you build your own box, you can actually create your own AMI image that contains all your software. That’s exactly what we did. Once we had our own AMI image, the private image, we can then store inside S3 and what will happen is, the minute we may want it, we’ll just go into our AWS account and we’ll just say, “OK, we need now five instances of this specific AMI running,” and we’ll just click that and punch it up immediately. No big deal.

Zeeek: Interesting. The agent is talking about the ElasticFox extension and start playing with EC2. I have absolutely no notion of what that is but hopefully, either you do or he’ll call in and tell us about it.

Nir: Yeah. ElasticFox is a Firefox extension that enables you to play with Amazon EC2 instead of working with the web interface. It’s only for Firefox, that’s one. I think a combination of both ElasticFox and the web interface as is because the new web interfaces they’ve created which is currently and still in beta is, in my view, just a bit easier to use than the ElasticFox, just a bit.

Zeeek: OK, and as I mentioned, IRC, I use a Firefox plug-in sometimes for S3 or for CloudFront. So, I assume it’s a similar concept. You just kind of dumps it down.

Nir: Yeah, similar concept. Yeah.

Zeeek: It’s kind of amazing to me that—and this goes back to what we were saying with John Todd that there’s no good reason to think that Amazon’s network would allow for real time. The CloudFront thing, the idea there is that you can be serving video and stuff like that and that it will push it out to the edge servers that will be the most efficient. On the other hand, I’m not sure how that translates to VoIP and I’m surprised to learn that they’re able to do the real time. Who knew, you know. That’s the way I feel about it. So it’s kind of surprising that they have the capacity—you don’t have any problem with call quality apparently, right?

Nir: No. Well, let’s say this, as long as you’re working with carriers which were interconnected with, let’s say…

[Music playing]

Zeeek: Yeah. Let me try to find the source of that.

Nir: OK.

Zeeek: That may have been Digium. Hello, Digium. Thank you. They may not. Sorry.

Nir: That was a good one.

Zeeek: Yeah. Well, that’s the Asterisk built-in…

Nir: We can talk over the music.

Zeeek: Frankly, I think that was somebody in Huntsville or else I just happened to click it by accident. Sorry. That happens from time to time.

Nir: Yeah.

Zeeek: Where were we?

Nir: In any case—where were we? Yeah, in any case, we were working with three or four different providers in the U.S. which are considered somewhere around, let’s say the tier 2. Then, usually, these carriers are interconnected on the backbone with links that are very, very efficient and reliable over to Amazon EC2 as long as you’re hosting your instances in the U.S. So, the end result that were, let’s say one of the carriers that we were using was called TSG Global and we were getting about 24 millisecond roundtrip from Amazon EC2 to TSG Global. Now, you’ll run voice over IP on that and will have perfect quality always.

Zeeek: I’m trying of course the agent to call in, trying to figure out because it would be more interesting to have a discussion here than reading the stuff in IRC.

Nir: Yeah, absolutely.

Zeeek: OK. My arguments have won over. He’s going to call. He or she, you never know, right?

Nir: Well, he brought up a very interesting point of how to bundle your own AMI and this is actually what I was talking about, creating your own AMI image. It’s not that simple. I have to be honest. It took me about four or five hours to actually understand how to do it right and it never works from the first try. It’s really annoying. What happens is that you need to take your running machine, your running AMI and you’re creating a raw06:25 image of the hard drive, of the virtual hard drive and then you upload that into S3. That’s complicated to do but if done right it will work on the second or third try. It never works on the first try, never.

Zeeek: OK and we have a bleep from Southwest British Columbia. Can you hear us? Could be the agent, you never know.

Nir: Actually, I’d love to hear about what he just wrote on Openfire XMPP and Red5 Flash Server for video conferencing. That is really interesting because I’ve been trying to play around with Red5 but I have to admit that Flash isn’t really my thing. So, I’d love to hear about that.

Zeeek: OK, the agent is coming out of the wild of Southern California. Who is in the Southwest British Columbia? Say hello. Come on, don’t be shy. We’re waiting for the agent to call in. We have time.

Man 2: What do you do for your logs then here, just logs if you’re just bringing up these instances and using for so many hours and shutting them down? Are you uploading those at some point, back up to S3 or do you just not care about them?

Nir: Well, the application as is doesn’t really carry that much logging into the actual server itself but I have to admit that I never really needed to go about and offload the logs anywhere because we don’t really are about those. CDR’s are kept completely separated from the system but you could offload those into a different machine or you can, before shutting down the system, go back and offload those out of EC2. It shouldn’t be that much of a problem. Or if you really want to, you can use the—it’s not S3—what was that again—it’s called…

Zeeek: CloudFront?

Nir: No, not CloudFront. They’ve got servers where you can actually map a storage device to your AMI and then you can share…”

Nir Simionovich: This is exactly what it pays. Awhile back, John and I did some mass going about and saying, trying to understand if there is a validity to actually going about and running a host PBX on top of Amazon EC2, and we realized that if you operate for 24 hours, a full system is operating 24 hours, then there is no real usage for Amazon EC2 as is. If you’re a business which is doing a hosted environment, then Amazon EC2 is the best you’ll find, really. It gives you that immediate quantifiable capacity that is really, really fast,but if you’re on business, let’s say, if you’re in an office and needed a PBX system, then Amazon EC2 isn’t really the thing for you because you can get an equivalent server with inside dedicated hosting facility today for like $59 a month and that one already has 16 gigabytes of traffic with it, maintenance, managed hosting and so on and so on. So, some services make sense working off of EC2, some don’t. If you’re going to say, “OK, we want to move into the Cloud,” that specific move has to be calculated and has to be tested, let’s say, economic wise. Moderator: The agent has posted—now, I’ve seen calculators for S3 before and this is it, right? Well, I’m going to read this because I don’t have time to make a short URL, Calculator.S3.AmazonEWS.com/calc5.html. Nir Simionovich: That will give some insights as to how much money you’ll need to pay. Moderator: I think it’s worth trying a project, maybe messing with it to see. So once you’ve got your instance ready to go, I mean, if the elections are held next weekend, are you ready to bring that same instance up? What’s involved in bringing that back to life? Can it be done? Nir Simionovich: Yeah, it’ll take about five minutes. Moderator: So where is this thing stored? I mean, it’s in your account and you just click, you just check a box and click submit and you can bring it back up or is it something locally stored somewhere with you or what? Nir Simionovich: It’s exactly how you described. What you do is, once you build your own box, you can actually create your own AMI image that contains all your software. That’s exactly what we did. Once we had our own AMI image, the private image, we can then store inside S3 and what will happen is, the minute we may want it, we’ll just go into our AWS account and we’ll just say, “OK, we need now five instances of this specific AMI running,” and we’ll just click that and punch it up immediately. No big deal. Moderator: Interesting. The agent is talking about the ElasticFox extension and start playing with EC2. I have absolutely no notion of what that is but hopefully, either you do or he’ll call in and tell us about it. Nir Simionovich: Yeah. ElasticFox is a Firefox extension that enables you to play with Amazon EC2 instead of working with the web interface. It’s only for Firefox, that’s one. I think a combination of both ElasticFox and the web interface as is because the new web interfaces they’ve created which is currently and still in beta is, in my view, just a bit easier to use than the ElasticFox, just a bit. Moderator: OK, and as I mentioned, IRC, I use a Firefox plug-in sometimes for S3 or for CloudFront. So, I assume it’s a similar concept. You just kind of dumps it down. Nir Simionovich: Yeah, similar concept. Yeah. Moderator: It’s kind of amazing to me that—and this goes back to what we were saying with John Todd that there’s no good reason to think that Amazon’s network would allow for real time. The CloudFront thing, the idea there is that you can be serving video and stuff like that and that it will push it out to the edge servers that will be the most efficient. On the other hand, I’m not sure how that translates to VoIP and I’m surprised to learn that they’re able to do the real time. Who knew, you know. That’s the way I feel about it. So it’s kind of surprising that they have the capacity—you don’t have any problem with call quality apparently, right? Nir Simionovich: No. Well, let’s say this, as long as you’re working with carriers which were interconnected with, let’s say… [Music playing] Moderator: Yeah. Let me try to find the source of that. Nir Simionovich: OK. Moderator: That may have been DGM. Hello, DGM. Thank you. They may not. Sorry. Nir Simionovich: That was a good one. Moderator: Yeah. Well, that’s the Asterisk built-in… Nir Simionovich: We can talk over the music. Moderator: Frankly, I think that was somebody in Huntsville or else I just happened to click it by accident. Sorry. That happens from time to time. Nir Simionovich: Yeah. Moderator: Where were we? Nir Simionovich: In any case—where were we? Yeah, in any case, we were working with three or four different providers in the U.S. which are considered somewhere around, let’s say the tier 2. Then, usually, these carriers are interconnected on the backbone with links that are very, very efficient and reliable over to Amazon EC2 as long as you’re hosting your instances in the U.S. So, the end result that were, let’s say one of the carriers that we were using was called TSG Global and we were getting about 24 millisecond roundtrip from Amazon EC2 to TSG Global. Now, you’ll run voice over IP on that and will have perfect quality always. Moderator: I’m trying of course the agent to call in, trying to figure out because it would be more interesting to have a discussion here than reading the stuff in IRC. Nir Simionovich: Yeah, absolutely. Moderator: OK. My arguments have won over. He’s going to call. He or she, you never know, right? Nir Simionovich: Well, he brought up a very interesting point of how to bundle your own AMI and this is actually what I was talking about, creating your own AMI image. It’s not that simple. I have to be honest. It took me about four or five hours to actually understand how to do it right and it never works from the first try. It’s really annoying. What happens is that you need to take your running machine, your running AMI and you’re creating a raw06:25 image of the hard drive, of the virtual hard drive and then you upload that into S3. That’s complicated to do but if done right it will work on the second or third try. It never works on the first try, never. Moderator: OK and we have a bleep from Southwest British Columbia. Can you hear us? Could be the agent, you never know. Nir Simionovich: Actually, I’d love to hear about what he just wrote on Openfire XMPP and Red5 Flash Server for video conferencing. That is really interesting because I’ve been trying to play around with Red5 but I have to admit that Flash isn’t really my thing. So, I’d love to hear about that. Moderator: OK, the agent is coming out of the wild of Southern California. Who is in the Southwest British Columbia? Say hello. Come on, don’t be shy. We’re waiting for the agent to call in. We have time. Man 2: I’m here. Moderator: Yeah. Man 2: What do you do for your logs then here, just logs if you’re just bringing up these instances and using for so many hours and shutting them down? Are you uploading those at some point, back up to S3 or do you just not care about them? Nir Simionovich: Well, the application as is doesn’t really carry that much logging into the actual server itself but I have to admit that I never really needed to go about and offload the logs anywhere because we don’t really are about those. CDR’s are kept completely separated from the system but you could offload those into a different machine or you can, before shutting down the system, go back and offload those out of EC2. It shouldn’t be that much of a problem. Or if you really want to, you can use the—it’s not S3—what was that again—it’s called… Moderator: CloudFront? Nir Simionovich: No, not CloudFront. They’ve got servers where you can actually map a storage device to your AMI and then you can share…

Open Fire XMPP and the Red Five. Let’s get into that. You guys… Sure. …go ahead. I mean, I’d love to hear about that one. Yeah, so. Well, let me, let me, kind of, lay out the landscape a little bit for why I would even go down this path. I don’t know if you’ve heard of a specific podcast or video podcast that’s done and has been done for a couple of years, called Gilmor Gang It’s run by a guy named Steve Gilmore who’s a well know technology columnist here in, or up in Silicon Valley. Anyways, I’m basically an executive producer of that show and for the longest period of time we’ve been trying to put together, you know, a remote TV studio on the fly. And the biggest thing with this scenario without doing any hardware was really the concept of having, having remote individuals around the world that could dial in, in whatever manner they had available to them. If they had the ability to use a video teleconference capability either through a flash or through a Skype connection or if they just had a basic call-in functionality we needed to incorporate all those things because we wanted to see, we want to use a remote video teleconference capability as an actual remote camera so that we can bring them into the show. We can switch in and out of the different guests because generally Gilmor Gang handles 4-6 different guests on a given show. There’s a lot of banter. So, really, this has been an ongoing investigation of mine for probably over a year. And I’ve been using EC2’s primarily for, at this point, for Personis itself. That is completely hosted on EC2 all the way, all the way through. S3, EC2 SQS. I mean, I probably use almost every service floating around inside of Amazon at this point. And, I also had played a number of months ago, using, putting Asterisk up and running it there, playing around with it to see how well that would work. Which, it works extremely well. I mean, you’re able to take advantage of Amazon’s significant infrastructure, their tremendous pipe, that they have going throughout their data centers and their multiple peering relationships that they have for Internet access. So, I was very pleased with that. So we’re moving on to the next go-round. Okay. Hold on. Sorry about that. It’s okay. It’s hard to predict. But once I see the wave front, it’s over. Go ahead. Anyways. I thought, I thought the angels were coming for me or something. So. The a, so anyway, we’ve gone to this capability in an attempt to basically create this real-time audio-video capability that can be brought up or down on the fly. And that’s, kind of, the real story. Because, I mean, as he was mentioning regarding the phone, the service that they were running where it’s, they only need it for a short period of time. That’s the extreme value of Amazon. I mean, that’s where you really, really, really can take advantage of the system. Because you only need to bring up an instance for as long as you need it and then you can bring the instance down. You know, you can use it in such a, in that manner for a [R?] escape function, so that if you have, you know you’re going to have a burst of traffic coming in. You know, depending on how you set it up, you can pre-stage servers sitting there waiting for the burst or you can even do, you can get a little more sophisticated and create an auto-scale mechanism that, as traffic builds, it automatically instantiates new instances to handle the traffic burst and shuts them down accordingly as the traffic falls off. So, I mean, if you looking at, just to give you a price understanding, from an Amazon perspective. Just on a small instance. If you’re running an instance. The smallest instance that Amazon has. It costs you approximately, and this is without, you know, significant bandwidth going in or out. It’s only $70, it’s about 70 bucks a month. Now, the real significance, though, is that if you don’t need it up all the time, take it down. And that’s, kind of, what we’re doing with this whole remote video communication capability, in that, we can pull-up or pull-down anybody actually. We’re setting it up so that we can have remote. If somebody else wants to do a show that we want to add to our network, we’re going to have this all pre-configured so that they’ll have, they’ll have video-conferencing, they’ll have, and that’s the whole Red Five to XMPP integration. So we have our own chat functionality through XMPP. Don’t need IRC. We also have all the video capability that is, and SIP capabilities that are inherent inside of XMPP itself with Open Fire. And then we have the complete telecommunications capability that we’re embedding with Asterisk. And then, of course, the Red Five stuff, which is really handling the, you know, the multiple videos simultaneously function. That’s why the Red Five exists. So that I can have a room, literally, of ten, you know, video, teleconference, or video pieces coming in from all the participants. So those are the types of things that we’re currently working, or I’m currently on, that hopefully, we’ll see as, basically, a package that we’re going to load out into the world, that people can run. And that’s the other part of it. You can take advantage of the really big thing about the AMI’s, the other big part that I take a lot of, that I take advantage of all the time, is just that there’s a lot of people out there packaging these AMI’s. They, if they put them out in the public space, you can just go and click on one of them and say, “hey, what is this one, oh, look, there’s a Red Five server already configured by somebody. I’m going to go ahead and launch that AMI.” You can customize that AMI to your heart’s content and then ultimately save that to your, save that yourself as a private version of that instance. There’s a vast amount of power sitting within the Amazon infrastructure to take advantage of.

Nir: Jerry, one small question in regards to Red5, and this is something I have been toying around with, and maybe you’ll have an answer, because I didn’t get much work into it. Are you working with the, by any chance, with the Fontventa extensions for 3G, 3.5G cellular video transmission? Jerry: No I’m not. It’s an interesting… I’ve seen the extensions, I haven’t played with them at all. That would be kind of a next evolution for us. The other thing is that right now I’m kind of using, it’s kind of a limited version of Red5, in the sense that I’m actually running, and I didn’t even know this could exist until I found the ability through some board. There is actually, you can take Openfire and you can actually run Red5. Somebody’s written an extension that actually runs Red5 inside of Openfire. So, you don’t need to set up multiple servers. It’s basically an instance that’s prepackaged and you can basically bring it up and you’ve got full blown Jabber and you’ve got full blown Open5 sitting there, I mean Red5 sitting there, ready for you to use. The extensions that could take advantage of 3G, 3.5G, and 4, I mean those are all things that would be very, very cool to be able to put in. Because as you can see with things KiteTV and QIK, and a bunch of the other ones, you know there is no doubt that kind of capability from a video teleconference capability is going to be big, and big in the very near term.
Nir: I have to admit that I didn’t have much time to play around with… Well l did play around with the Fontventa extensions, and I’ve done some work with that, in trying to integrate video, IP Video cams and Asterisk, and have, I’d say, some fair success with that. The thing is a lot of people now are talking about it. They are actually integrating a flash environment back into your cellular phone over 3G instead of writing your complete IBPR and doing that through Red5 seems like a good idea. But I will have to admit, that I hadn’t been playing with that lately. Again it’s very reassuring for me saying – OK there’s actually somebody out there really doing it. I mean, really progressing in that direction and we’re doing that on top of EZ2 is really exciting. Jerry: Yeah, and we think it’s really powerful, too. Again, the concept is that once this is a kind of an operational tapestry that we’ve kind of, or are orchestrating, that we really plan on bundling it as a kind of a preconfigured service that people in our… You know the concept, as we can all tell, media is changing for the worst, for the better, however you want to look at it doesn’t really matter to me, but bottom line is this entire users interface content and community that’s floating around out there and taking over, and watching mainstream media die. I mean, the idea that we can have this full blown, on the fly network capability brought up at will any where in the world and have multiple correspondence with basically multiple instances of both video and voice, there is an exorbitant amount of power there, especially when you start looking at what’s happening with the real time, in the real-time space, with things like Twitter and others. So, I think this whole new real-time space is actually going to usher in with Twitter and others, the ability to usher in this whole new paradigm for real-time, instantaneous, dynamic virtual, in essence radio stations, television stations, that can pop up on the fly and shut down on the fly at will. And it’s all going to be run through, basically, a cloud. So, that’s kind of what I’m focusing in on.
Nir: Well I totally agree that there is a shift. Where we can all… well you know people who are already in it and are dealing with the shift, they are all feeling it pretty strongly. I remember when they started talking about cloud computing here in Israel that actually, the thing is that I was in this, there’s the Grid Community here in Israel, they were doing this event and I was looking at it saying – Well it’s pretty interesting but it doesn’t do much for real-time. Then I got back home and started to talk again with John Todd and saying – Well why not do it? It’s pretty interesting that there are real usage of real-time applications that can be done on the cloud and that’s really interesting to me, and really exciting. Not only from a commercial point of view or the economical point of view, but also from a technical point of view, because we have to rethink every thing. Where we’re really used to using like this enormous servers and saying – OK I’ve got 8 cores, I’ve got 16 cores, I’ve got 32 gigabytes of ram, and suddenly we don’t have that and it brings up a lot of the old paradigms that I used to work on back in the days of lets say Commodore 64 and saying – OK we need to be something really, really optimal, and the optimization is really interesting. It’s bringing back really old paradigms in terms of optimizations and really new paradigms in terms of economics.

Jerry: Yeah, yeah. I agree with you. I think what’s been happening, I mean it’s kind of how software leads hardware in some ways. You know, over the course of the past decade or more, the extreme focus on, or at least in the places I’ve played, a very laser guided focus on abstraction and decoupling. I mean that’s been a big part of a lot of systems that I’ve been a part of in the past 10 years. I think now it’s interesting to see cloud computing and the way it’s basically being pushed out from a hardware standpoint and how all the aspects of abstraction and decoupling are instrumental in being able to do these systems. Because you really do need a fairly significant abstractional error because you don’t know where the hardware exists. You don’t know how much memory you’re necessarily playing with. So, this whole decoupling is pretty paramount but it’s also been a trend that’s been going on for at least a decade or more.

Nir: Yeah, it’s been around here but not many people were using it, you know. I started my, let’s say, I started using the main frame computer somewhere around the age 18 or 19, when I was in the Army, and decoupling applications and the world of the main frame just sounds logical. It’s there, there’s nothing else. But when you are used to working on PC’s and servers, decoupling applications looked really weird. When you come to the company saying – OK you need to decouple this and they go – What? What are you talking about? What I don’t have access to the data base? That doesn’t sound right. The paradigm shift is going in a few directions and I’ve seen some companies that they, company like Giga Spaces, it’s an Israeli company, and they build like this enormous middle layer between your application and the Amazon EC2 cloud, which gives you the, let’s say, the imaginary effect of working inside a coupled system while you are completely decoupled. Which is really interesting. Jerry: Yeah, yeah.
Nir: There is a lot of work being done right now. I think we are heading, in terms of technology, we’re heading for the next, I believe, is like the next three years is going to be really interesting in terms of cloud computing. Jerry: Yeah. I think the biggest thing for me, I’m somewhat of serial entrepreneur, and have been that way for a couple of decades now, but I think the thing that really excited me about the advent of things like EZ2 or Rackspace’s mossomo, or any of the other ones that are floating around there. There’s Slicehost which actually is now owned by Rackspace. It’s just the cost at which somebody could bootstrap up something, an idea, a concept. To be able to sit there and create a new business in this new paradigm and do it at a cost that is unheard of. Having done a few startups in my lifetime, you ultimately throw out an inurnment amount of money upfront just to handle, from a capital standpoint, the hardware that you are going to run on, especially if you’re a service offering. So I mean, that’s one of the things here, where you can get away with some, a couple of weeks worth of coding, you know, a $70 fee and you’ve got a service up and running in an hour or more, or a little bit more. But it’s nothing. It’s nothing to do and it’s cost effective, and if it doesn’t pan out, you basically shut down the AMI and you walk away from it.

Comments on this entry are closed.

Previous post:

Next post: