OnLive: how it works

After launching in the UK a few months ago, cloud gaming service OnLive is now bringing PC and console games to mobile devices, including the iPad and iPhone. A lot of people are still unclear on exactly how it all works, though, so we've spoken to OnLive's CEO, Steve Perlman, and OnLive UK's general manager, Bruce Grove, to put together this technical guide, which will hopefully shed some light on what makes the service tick. But be warned – it's pretty thorough…

For those unfamiliar with the service, OnLive allows you to play high-end games on any device, because the all the hard work is being done by servers in the cloud, with the video output being sent to you over the internet. We've got a full introduction in our video walkthrough below.

Why don't we let Steve Perlman, OnLive's CEO, explain: "So you hit the controller button, keyboard, mouse or touchscreen, and that controller action gets sent through the internet to the data centre in the cloud, and we actually have the games running. The game responds to the input, calculates the next frame, and then using a proprietary video compression technology, we stream it very rapidly down to appear on your screen, but perceptually it seems like the game is running locally."

If you've already been using the service on your computer, or maybe with OnLive's microconsole, you'll be used to using a controller or keyboard and mouse to control the games, but obviously things have to be a little different on the iPad.

For a start, OnLive has always recommended to date that you connect to your router directly with an ethernet cable for playing OnLive, but this clearly isn't possible with tablets or mobile phones. Normally, OnLive recommends a 5Mbps broadband connection to play, but that's lowered for the iPad.

"We recommend around a 2Mbps connection – that makes it very coffee-shop friendly or hotel-lobby friendly, or wherever, but also means you can just play at home over your Wi-Fi if you're just sitting next to the TV playing the games," said Bruce Grove, OnLive UK's general manager.

This lower speed recommendation is partly due to tweaks to ensure it works better of the limitations of Wi-Fi compared to ethernet, but fact that the iPad has a lower resolution than OnLive's default 720p also helps.

"It's as much about the resolution as anything else. If you wanted to pump it out over HDMI and get HD on the TV, then you'd be back to needing to bump the bandwidth up to get the right quality. But the play on a device like this, it's fine for 2 – 2.5Mbps," said Grove, raising an interesting point.

Using Apple's Digital AV Adapter, you can output over HDMI from an iPad 2 or iPhone 4S to an HDTV, and the video will go back up to OnLive's native 720p. As Grove says above, though, you'll want a faster internet connection if that's your intention.

Taking control

The other big change from computers to tablets is how you control them. OnLive's games are designed for keyboards and controllers, not touchscreens.

"We've been working with the publishers on working some form of native touch into the games. So we demoed From Dust doing this, but here's a brand new build of Defense Grid. What you'll notice is that this is the same Defense Grid that you get elsewhere, but when you bring it up on a mobile device, we now have some additional features in there that are specific for playing on this type of device," Grove explained.

In the case this game, there are new buttons around the edge of the screen that aren't present when you play the game on non-touchscreen devices. These new controls have been built in by the developer, to make sure they fit in with the game overall. Rockstar Games has done the same with the game LA Noire, released on consoles earlier this year.

However, not all developers have tweaked their games for touch, so OnLive has developed a system of touch overlays for the other games.

"We have on-screen controls that allow you to play the game, and each of these controls are tuned for the game, so on a game-by-game basis you'll see these buttons move around. We put them in places that we think have a good look and feel for each of these devices. What you'll notice on the Arena is that, because this is just a button overlay for you in the app, you don't get them on the screen when spectating."

Of course, it's fair to say that while a touch overlay might suffice for many games, it isn't the ideal way to play something that was designed for a controller. As a result, OnLive has launched the Universal Controller, which can connect to any Bluetooth device, and comes with a USB dongle for anything that doesn't have Bluetooth built it.

"The Universal Controller is Bluetooth 4.0, so it's very low latency," Grove tells us. "If it sees the dongle, it gives you the lowest possible latency. But it can also work with other Bluetooth versions across multiple devices, so here it just pairs to the iPad."

We tried the controller and the touch controls when we spoke to Grove, and it seemed to us at the time that the controller felt slightly more responsive.

"That's quite possible. Touch controls have their own latency that's inherent in the screen response, but the controller is going straight through. So it's feasible that it's more responsive. On a game-by-game basis, one of the things we're doing is seeing what games it makes sense to add overlays to, but the controller just works with it all."

In fact, the iPhone 4S has Bluetooth 4.0 built in already, so should work perfectly with the controller, and with its ability to output OnLive's full 720p resolution to an HDTV over HDMI, it's actually one of the devices best suited to OnLive – a high-end games console you carry everywhere.

Going up against the App Store

One of the main stumbling blocks for OnLive is likely to be pricing. With so many games in the App Store available for 69p, or not much more, how will the console prices that OnLive charges stand up, especially when there are versions of some games available on both services?

"There's going to be overlap," admits Perlman. "You've got LEGO Harry Potter on the App Store, we have LEGO Harry Potter on OnLive, and so on the iPad you have two LEGO Harry Potters to choose from. The one that's in the App Store you can play on an aeroplane or away from any connectivity, but if you're playing LEGO Harry with OnLive it's a much richer experience. They don't have to simplify the game, but you've got to be near an access point somewhere. I think the two versions can coexist fine."

But what about pricing?

"It's something we've talked to the developers and the publishers about," says Grove. "And right now, the obvious pitch is: 'This is a console that plays across all of these devices', so what you're paying for is making games playable across all these devices.

"The other part of it is, when you look at the games in the App Store, and you look at the Dead Spaces or Infinity Blades that could cost up to £10 a game or £5 depending on which ones, they tend to be shorter. They tend to have four, maybe eight hours of gameplay in, whereas this might have 40 hours of gameplay in. So we've already started talking to some of the developers and publishers about exactly this, which is: 'What if we break games down into episodes, and we give out or unlock the episodes for a price?' Because that way you can get your 10 hours of gameplay for £10 or £5, and then you go onto the next chapter for £10, or maybe you go and buy some side quests for a few pounds.

"Between us and the publishers, what we're doing is thinking about what models make it more mobile friendly. We know one of the things that's going to happen is that it will open up an audience that wasn't there through the microconsole, or wasn't there through PC or Mac, because there's so many iPads and iPhones in people's hands, and so we have to go up against the App Store games, just because we're both on the same device, and so visible to the user.

"One of the things we're thinking about is, with a good network connection, we could have multiple-screen playing. I could could have one screen on my microconsole, and each tablet could have a different view of the world, interacting in their own way. Maybe as multiplayer, or interacting with your own part of the campaign. At that point, from a dev's standpoint, they're not restricted by the front-end hardware, and we'll build the hardware to support them on the back end – we'll support them to let them figure out what to do next, and create content to take advantage of this multi-screen world."

Multiplayer

One of the potential stumbling blocks for OnLive is that it's multiplayer components are mostly restricted to play against others within the OnLive service. We asked Perlman if that would always be the case, and if it was a cause for concern when multiplayer is such a big draw for games these days.

"We don't have an agenda for that, but we do have challenges for bringing it out, because the versions of the games that we have are slightly different to the PC games," he explained. "Sometimes they're subtle differences, and some are not-so-subtle differences. If you've got a laptop or tablet, for example, you might go out of Wi-Fi range. We lose contact with your device, so what we'll do is pause the game, and we'll give you five minutes to reconnect and pick up where you were. Even that little thing changes the PC game. The multiplayer systems for PCs don't expect that – they don't expect one of the games to be paused. But that said, we are working with different PC multiplayer services to see or not whether there are at least some games we can play together.

"What I'm trying to say is that we're not trying to go and cordon off people from playing multiplayer outside, it's just not as simple as it sounds. For multiplayer to work, you've got to reconcile things perfectly. You can't have a situation where one person has one slight different in the way they see the frame to another person, otherwise the games will get out of sync.

"The other thing is, when you're playing multiplayer on OnLive, it's actually like a LAN party, because the servers – particularly the servers in Europe – essentially one cluster, so everyone you're playing has less than 1ms ping between the different machines. Now, there's this latency between you and the game that you're playing, but particularly as more people are playing on fibre, you'll find that it's quite low, and with the new games that are coming out, it's imperceptible. The delay that you have through the network is actually less the delay you have from the prerendered view you'd have in a PC game today."

Streaming the video

When OnLive was first announced, it was met with no small amount of skepticism. It was mostly thought that it just wasn't possible to play a game over the internet like that and have it feel remotely responsive. In fact, Perlman says that wasn't really the problem that needed focussing on.

"A lot of people don't realise that we solved the problem of low-latency video compression years ago many years ago," he told us. "The thing we've been working on over the last few years has been largely the problem of delivering video with very, very high reliability over the internet."

Obviously, you don't want there to ever be any pausing while you're playing a game, and there can't be any buffer for when your connection falters, as there is when streaming linear video over the internet, because there's nothing to buffer – you haven't played that part of the game yet. Essentially, every frame has to reach you perfectly as soon as it's created. What sort of technology can do that?

"So we're hoping – this has been a decade of work, right – we were hoping that we would find the holy grail. One great algorithm that can handle any type of device, any type of internet situation, but we just couldn't find it," Perlman explained.

"So people very often ask what video technology, singular, we're using. It's actually many different ones, and it depends on many different things. For example, to run on the iPad, or to run on an Android tablet, we're using two different algorithms, just because the processors are different, which are different again to what's being used on, say, a Macintosh, which is an X86 processor. Which is yet again different from the algorithm used for the microconsole. So just on the device alone side, we use different algorithms to accommodates the limitations of the device.

"Then you get into characterising the connection. No matter what you do, the internet is going to be a variable. Demonstrating in your office today, you guys publish gaming stuff and people are doing big downloads, and we've seen a couple of bumps in the road, but we've watched the algorithm adapt. There's been a couple of point where you actually see there's a problem for a second, and then it kicks in. What's typically happening there is it's saying, 'OK, the situation has changed', and it'll actually switch to a different algorithm, or change the parameters of a certain algorithm.

"You could actually be partially through a frame when we tweak what you're receiving. We typically adjust the algorithms about every 4ms, and a frame time is about 17ms when you're running 60fps.

"The other side of it is that the games themselves are being optimised to work in a cloud gaming context. Previously, we've largely started with PC versions of games, and they've made some changes to them, but not changes that would specifically, for example, tune up the game to take advantage of the way that we compress the video to get the best quality image, and so on. But we're now including games that have some optimisations for our systems."

This sounds impressive, and comprehensive, from OnLive's end, but this is obviously best-case scenario stuff – the optimum tweaks made by OnLive's servers. The real question then becomes the quality of internet service between the servers and where you're playing, and even anything on your network that could interfere.

"People have asked how the internet compares in the UK with the States," Perlman says. "They're saying the internet can't be as good here as the States, and it's quite the contrary, actually. The States has all these different companies, and each has their own networks that are peered – they're connected together – and we have to unravel all that to make OnLive work with low latency in the states. But in the UK, you have a single backbone that ties everything together, and BT are the experts. On a wholesale level, they're very knowledgeable, and we worked with their experts to design the way that we provide the connectivity into the backbone of the UK network.

"It's still new, – there's gonna be a couple of rough edges in the early days, but for all those reasons you'll be seeing exceptionally good performance. I actually think you'll have a little better performance here than in the States in most cases."

"The pickup here's been huge," adds Grove. "So big that we had to airlift a bunch of servers from the US. We planned a four-month capacity increase, and we ended up pulling that into four weeks. That was a lot of work. We put a lot of servers onto planes to get them over here and beef up capacity. The pickup continues to grow here, and the response we're seeing has been hugely positive.

"We're always going to see the people who say 'it doesn't work for me', but one of the things we've seen here is people are reacting more to 'my network isn't good enough for this', rather than 'the service doesn't work'. In the US, we had to fight for a long time against 'the service doesn't work', and overcome the initial bias that it couldn't possibly work. In the UK, having come from over a year of launch in the US, we had enough credibility, I think, and people recognised the fact that it does work, and works well enough that a lot of people are very happy with it. "

"The ISPs in the States are getting to know us as the canaries in the coal mine for them," laughs Perlman. "And they love it. I mean, they want to provide good service, and most of the time one user calls up with a problem, and they can't necessarily know if it's that person's problem or whether it's a general problem.

Even the very best Internet connection you'll find, because of the way the internet was designed, will have packet loss. Even if you have fibre, there is packet loss. It just has to happen, otherwise it's impossible to build a system. The thing is, if I had a 100Mb connection, and all I needed was 3 or 4 or 5Mb out of it, and I could choose which 3 or 4 or 5Mb that was, I'd be fine, because I could always avoid the packets that are lost. But you can't. There's a certain statistical probability that it's going to be the packet you need, and conventional compression algorithms that exist today, if you have one bad packet, it can cause a whole series of frames to be destroyed. So what we've designed is a set of technologies that are highly, highly tolerant, up to a certain level.

I was dealing with some guys last night who were having problems with OnLive. It turned out to be their router that was the problem, but after one day of the service struggling, it adapted, and they were playing fine the next day. Now, they were seeing 6% packet loss, and it was still running. Think about it: if your ISP was seeing a 6% packet loss in their systems, that'd be a service call. There's something wrong. A typical ISP would like like to see less than half a percent packet loss. So that fact that we can tolerate 6% packet loss has been years and years of evolution of the technology. "

Adapting for the iPad

OnLive doesn't just use multiple algorithms to deliver its video, it also uses a proprietary video codec. The iPad has a very limited range of internet video options that it supports, and OnLive's isn't one of them. Decompressing video is needs a lot of power, so there's usually dedicated hardware to do it. The iPad features hardware decompression for the H.264 codec, but little else, leaving software decompression as the only other choice. But that has some serious problems:

"I remember when we first were talking about doing OnLive on the iPad, we were working with another partner of ours who was involved with Apple on the iPad, and they had worked with other companies who had tried to do software decompression on the iPad, and they were saying that it runs at about 5 frames per second on the ARM processor that's in the iPad," Perlman recalls.

"That's why everybody uses QuickTime and the hardware decompressor in it, but Apple will only allow QuickTime to reach that chip, so even if we wanted to use some parts of that chip for OnLive, we can't. To make this work, we had to go and optimise the codec to work specifically with the ARM chip in there. A lot of it is machine-coded instructions that are using some special features of the ARM chip, and we also take advantage of the graphics processor in here.

"GPUs are used for 3D animation and so forth, but they're not normally used for video decompression. But they're fast. We couldn't get the ARM to do it fast enough, so we created an algorithm just for iPad that was able to go and formulate some decompression functions into what looks like a shader – a 3D shader. And so the decompression comes from a combination of tightly coded ARM code and custom-coded GPU code, and it gives you the instantaneous response you need. So we're not using the built-in video silicon at all in the iPad to make this work."

The use of the graphics processor is intriguing. Using a GPU for computing tasks is not a new concept – it's usually called GPGPU (General Purpose computing on Graphics Processing Units), but it's not something that's hit mobile devices in a big way yet. What Perlman describes above certainly isn't a standard GPGPU implementation – we asked if OnLive is effectively tricking the GPU into doing the work the CPU can't.

"It thinks it's shading a big old polygon here, or a bunch of polygons that make up the image. I wouldn't say tricking it, but we have certain mathematical things we're doing, and we reformulated them so that they match the mathematical capabilities of the GPU. GPGPU things would be more algorithmic, and there's not enough time to do that – we've gotta throw these frames up too quickly – so we're using the GPU in the way it's primarily designed, which is a straight pipeline through. And it works quite well!

"The good thing about the way we're doing it is that if you leave this iPad running, and you leave another iPad running that's decompressing QuickTime, playing LOVEFiLM or iTunes or something like that, our battery life's about the same. That's another problem we had: how can we design an algorithm that's heavily using the CPU, and at the same time isn't burning thorough the battery? We had to gear back and find out, for example, if we were constantly hitting the RAM, or certain things like that. We figured out what things would burn through the battery more.

"So getting this to work overall on the iPad was very, very difficult, and yet the Android stuff – another challenge. A whole new set of problems!

"We do have access to some of the silicon that we don't in the iPad, but they're using different silicon, and different Android devices are using different chips, and…" Perlman sighs. "The silicon in the microconsole that we're using is our silicon, though, and we are using hardware for decompression for the most part, although we're using software for handling all the analysis of the channel – checking if there's congestion, if there's packet loss – to figure out what to do and to provide the feedback.

"Oh, and everything's encrypted as well, so the decryption has to happen in all these devices, which again uses CPU cycles! So we have to see if there's a way we can use some hardware capability that's in there for that, too.

"Now, the next generation of tablets that are coming, and the next generation of TVs, and because the manufacturers all want to run OnLive very efficiently, we've been able to work with the silicon vendors. In fact, we have a number of them that we've announced with – Nvidia, AMD, Marvell and Intel – and all of these guys are adding stuff into their silicon now that they know what our requirements are. When you have a zero million install base, it's hard to persuade chip vendors to redesign their chips, but once you get into larger numbers, then they suddenly get interested.

"Now we've launched, and we've got 150 games, we're in a position now where we can say, yes, we're working with all these guys. Why wouldn't they want to support OnLive? It doesn't cost anything – we don't anything to the chip, it's usually a rearrangement. Frequently it's removing a buffer that introduces delay that's needed for normal video decoding to deal with packet loss, which is not needed for what we're doing. In a lot of them, the buffer length is programmable, and we make it zero. No hardware changes needed!"