The Book of Xen Part 1

The Book of Xen.

by Chris Takemura; Luke S. Crawford.

FOREWORD

Virtualization is cool. I've always had a soft spot for virtualization, since as a lifelong sysadmin I get pretty tired of the endless fine-tuning that goes into building a successful network "host." Especially when that fine-tuning evolves into upgrades involving screwdrivers, recabling, and dust.

While Xen wasn't the first serious virtualization platform, it was the first serious open source open source virtualization platform, so it was the first that I was willing to invest my time in learning about, and the first I'd consider basing any production-level systems on. Open source isn't just a preference for me-I dislike lock-in, so I hardly ever buy or deploy or depend on something that I couldn't replace with a different product offered by a competing vendor sometime in the future. virtualization platform, so it was the first that I was willing to invest my time in learning about, and the first I'd consider basing any production-level systems on. Open source isn't just a preference for me-I dislike lock-in, so I hardly ever buy or deploy or depend on something that I couldn't replace with a different product offered by a competing vendor sometime in the future.

Like any serious open source system, Xen has the power of an ecosystem in which anybody who wants to vend can pick a spot and start hacking, but Xen also has the backing of a strong company whose employees contribute to the open source version of their product. This kind of vertical openness makes it possible for anyone (a hobbyist or a Fortune 500 company) to jump into Xen, buy only what they want or need (or just get it all for free), and have it remain compatible with the rest of the ecosystem. Thank you, XenSource and Citrix, for all this.

Confession time: I don't use Xen for any of my personal projects. I just don't have enough systems in any one location, nor can I plan far enough in advance-I'm too small to be able to afford virtualization's efficiencies. Whenever I do need separation of privilege on the same physical servers, I've been able to get away with FreeBSD jails or User Mode Linux. I also do a fair amount of real-time work, in which I need my code to be close to the hardware for design-and sometimes performance-reasons.

For professional work, my company uses a mixture of proprietary (VMware) and open source (Xen) virtualization and the results are outstanding. Whether it's to save money on hardware, save money on sysadmin time, or enable new kinds of computing, virtualization is a winner and it's here to stay. I've seen Amazon and Google build gigantic clouds of virtualized servers for their own use and for rental to customers, and this method has driven down IT costs for both new and established companies of all sizes. It probably saves power and lowers the industry's carbon footprint as well.

I'm struggling to find a way to communicate how amazingly cool cool this is. We try to write programs that fit into a single process, but they end up taking a whole Unix system because of all the processes and databases and sh.e.l.l scripts and file systems and UIDs they slop over. So we end up dedicating physical servers to applications that have no performance- or security-related reason to be on dedicated servers; but each one takes up some rack s.p.a.ce and some sysadmin time, and each one generates some minimum amount of heat, and so on. Then along comes virtualization, and we're back to adding physical servers only when we've got a good reason to do so, which usually means for capacity reasons. this is. We try to write programs that fit into a single process, but they end up taking a whole Unix system because of all the processes and databases and sh.e.l.l scripts and file systems and UIDs they slop over. So we end up dedicating physical servers to applications that have no performance- or security-related reason to be on dedicated servers; but each one takes up some rack s.p.a.ce and some sysadmin time, and each one generates some minimum amount of heat, and so on. Then along comes virtualization, and we're back to adding physical servers only when we've got a good reason to do so, which usually means for capacity reasons.

Note that while I admire cloud computing, I also fear it. Amazon and Google have their own virtualization APIs, and anyone who builds "version 1" of a system to live inside one of these commercial clouds is probably signing up to put "version 2" into the same cloud. Compet.i.tion requires differentiation and most vendors want to be different in capability, not just in cost efficiency. In other words, lock-in is great for sellers but not so great for buyers. Thus my attraction to enterprise virtualization-and specifically to open source enterprise virtualization, with the resulting vertically open ecosystem. I'll build my own clouds whenever I need them-and with Xen, so can you.

A word about Luke. He was a kid who lived down the street from my sister, and she asked me to give him a chance. So I hired him at an anti-spam company called MAPS (yes, that's spam spelled backwards, pretty neat, huh?), and he turned out to be a dumba.s.s kid, like we all were at that age. In the time since then, he has distinguished himself as a virtualizer and now, with this book, as a writer. Xen is cool stuff, but it's also deep and wide and dense-that is to say, it's a hard topic. Luke and Chris have unscrambled Xen into the linear form of a printed book in about the best way I can imagine anybody doing it, and I learned quite a bit about Xen from reading my advance copy. The book is also fun to read without the fun being distracting or dilutive.

Go forth and virtualize!

Paul Vixie La Honda, California September 2009

ACKNOWLEDGMENTS

First, we would like to thank No Starch Press. Without them, this book would never have been imagined, much less published. In particular, we'd like to thank our editor, Tyler Ortman, who had the thankless tasks of making us write and cutting our dumb jokes. We'd also like to especially thank Rami Rosen, who provided us with an excellent technical review; Jeanne Hansen, our long-suffering copyeditor; and Bill Pollock, who paid for it all (and who made sure we actually finished it). And to everyone else on No Starch's team: we couldn't have done it without you. It was a humbling experience to see so many people scrutinizing our work, and the book is much, much better for it.

We also want to thank all the people who worked on prgmr.com during its checkered history. Without help from many skilled people willing to work at below market rates, the company would have folded long ago. So, heartfelt thanks go to Thuy Vu, Neal Krummell, Will Crawford, and Nick Schmalenberger, and to everyone else who has worked here for shorter periods of time. Neal deserves a special mention. Aside from introducing Chris and Luke, Neal provided encouragement and help during the critical early phases of the project.

Maybe most of all, we want to thank the customers of prgmr.com for giving us a lab with real users to test all this stuff.

Chris would like to add: And to Alan, Chris, Ian, and Ken: The book's done now, so stop teasing me about it. Thanks for the encouragement, everyone.

Luke's personal acknowledgments: I want to thank my dad. (Sorry you got beat out for the dedication. I'm sure you understand.) Without his encouragement, my natural entrepreneurial spark would never have developed into the flaming inferno it is. And I want to thank my other dad, too. When I make fun of enterprise software, I compare it to stuff I wrote with my stepfather's copy of FoxPro when I was 14.

And extra thanks to Paul Vixie, who both gave me my first real job and agreed to write the foreword for this book: If I'm a good sysadmin today, my time at MAPS has quite a lot to do with that.

INTRODUCTION

Being an account of the struggles and travails encountered by Our Hero in his zealous quest for performance: In which there is brief confusion and a beginning.

Once upon a time, in the land of Armonk-where-the-shadows-lie, a band of fiendish programmers were weaving their evil schemes. And it seemed that dark days were upon the earth at last, and for all time, for the programmers seemed so very clever that no one would ever be able to stand against them. And even if some hero were, through great fortune or unimaginable heroism, to bring one low, then there would still be an innumerable quant.i.ty remaining, each more fiendish and subtle than the last.

Wait. That's not right at all. In fact, that's the beginning of an entirely different book. Let's try that again.

This book is about Xen. It's not about Zen. It will not show you a path to enlightenment, expressed as a release from the floating world. We will not give you advice on the Eightfold Path, or enumerate the Four n.o.ble Truths. Those are beyond our purview. But if all goes well, this book will make you happy.

Virtualization: A Brief History In this case, the vehicle for happiness will be virtualization. It sounds bizarre, but people have attempted to become happy through virtualization since the Dawn Of Time. (In computing terms, that's the 1970s.) IBM's team of programmers in Armonk produced the first VM (virtual machine) solution that we know of, VM/370, to ensure that their new machine would be able to run programs developed for an older model. Customers loved it back in 1979, and the Xen developers credit it as a major inspiration. A similar, more modern example might be the Xbox 360's software emulation of the original Xbox.

For a while, not much came of it. Virtualization continued to play a part in computing, mostly in the very top end of the market, but people continued to obstinately require a complete machine for most tasks until about 2001.

2001, everyone had to admit, looked very different from 1979.[1] Computers had become small and ubiquitous. The great time-sharing machines had given way to PCs. Batch processing was a rarity, and fully interactive desktop applications had become the Computers had become small and ubiquitous. The great time-sharing machines had given way to PCs. Batch processing was a rarity, and fully interactive desktop applications had become the raison d'etre raison d'etre of computing. Most important, from our perspective, the single computer had been eclipsed by the network: Most computers worth having were connected to the Internet, and each of them required various services. of computing. Most important, from our perspective, the single computer had been eclipsed by the network: Most computers worth having were connected to the Internet, and each of them required various services.

These services, in turn, were designed in such a way that they could be readily provided by even the cheapest and smallest server, often many times over.[2] Suddenly the people operating these services had a terrible surplus of computing power, devouring electricity all out of proportion to the actual services they provided. Something had to be done. The stage was set for virtualization to re-emerge, this time as a means of server Suddenly the people operating these services had a terrible surplus of computing power, devouring electricity all out of proportion to the actual services they provided. Something had to be done. The stage was set for virtualization to re-emerge, this time as a means of server consolidation consolidation.

Some clever gentlemen at Cambridge decided that this idea could be extended even further-if virtualization allows an individual or company to consolidate their machines, they reasoned, shouldn't it also enable multiple multiple organizations to consolidate their machines and reap even bigger benefits? That's the goal of Xen. It treats virtualization as a technology that allows people to ignore the hardware entirely. Computing, in this model, becomes a service or a commodity, "creating a world in which XenoServer execution platforms are scattered across the globe and available for any member of the public." organizations to consolidate their machines and reap even bigger benefits? That's the goal of Xen. It treats virtualization as a technology that allows people to ignore the hardware entirely. Computing, in this model, becomes a service or a commodity, "creating a world in which XenoServer execution platforms are scattered across the globe and available for any member of the public."[3]

That's where we are today. Although the XenoServer platform was never released, its vision survives today as "cloud computing," made possible by Xen (and, admittedly, other virtualization systems). Xen fits into this grand cloud computing scheme by enabling sites to create "nodes" that can be managed, transferred, and billed in ways that aren't feasible with other computing-as-service mechanisms.

[1] And, to our great dismay, also very different from the movie. And, to our great dismay, also very different from the movie.

[2] We know, there are many applications where this is not the case-but there are still a lot of small web servers (for example) out there. We know, there are many applications where this is not the case-but there are still a lot of small web servers (for example) out there.

[3] Hand et al., "Controlling the XenoServer Open Platform," (University of Cambridge, England, 2003). Abstract. Hand et al., "Controlling the XenoServer Open Platform," (University of Cambridge, England, 2003). Abstract.

So What's Xen Again? (And Why Should I Use It?) Even if you're not interested in this sort of grid computing thing, Xen offers some advantages to both the system administrator and the home user.

Xen is a piece of software that enables one machine to behave as if it were many virtual virtual machines. Each of these machines can run its own operating system and exist almost independently of the other virtual machines running on the same hardware. Each virtual machine (an machines. Each of these machines can run its own operating system and exist almost independently of the other virtual machines running on the same hardware. Each virtual machine (an instance instance, or domain domain in Xen parlance) has its own apparent network interfaces, disks, and memory. in Xen parlance) has its own apparent network interfaces, disks, and memory.

At first, this makes Xen seem no different from an emulator emulator, such as VMware, Microsoft's Virtual PC, or the open source QEMU.[4] However, these traditional emulators work by running software on a simulated processor that is, itself, also software-a rather slow proposition. Xen actually runs all software directly on the processor at full speed, with only a very small overhead for some resource management tasks. However, these traditional emulators work by running software on a simulated processor that is, itself, also software-a rather slow proposition. Xen actually runs all software directly on the processor at full speed, with only a very small overhead for some resource management tasks.

This leads to the first, and probably the most important, advantage of Xen: Xen runs fast fast in comparison with traditional emulators. Preliminary results in "Xen and the Art of Virtualization"-one of the seminal Xen papers-indicated performance degradation of less than 2 percent for a standard workload and between 10 and 20 percent for a worst-case scenario. Since then, Xen has improved. We usually just consider Xen's performance to be "sufficient" and leave it at that. (Readers desiring a more precise answer might want to read in comparison with traditional emulators. Preliminary results in "Xen and the Art of Virtualization"-one of the seminal Xen papers-indicated performance degradation of less than 2 percent for a standard workload and between 10 and 20 percent for a worst-case scenario. Since then, Xen has improved. We usually just consider Xen's performance to be "sufficient" and leave it at that. (Readers desiring a more precise answer might want to read Chapter10 Chapter10, which discusses benchmarking Xen's performance with your particular application.) Xen's advantages also show up in contrast to a standalone machine, even beyond the consolidation argument mentioned earlier. Like a traditional emulator, Xen provides robust fault isolation-that is, any software problem that affects one virtual machine is unlikely to affect the real machine or other virtual machines running on the same hardware. This makes it especially useful in environments where you can't be certain of the intentions or skill level of the users.

Also like traditional emulators, Xen provides an additional layer of abstraction between the machine and the user, allowing the administrator increased flexibility-suddenly the application can be decoupled from the hardware almost completely; stopped, started, moved around; made into a genuine service.

But Xen's main advantage is, in a sense, psychological: It makes it possible to think of computer time as even more of a commodity than it already is.[5] With Xen, you can run your own virtual computer for as much or as little time as you need, with resources tailored to the desired application. With Xen, you can run your own virtual computer for as much or as little time as you need, with resources tailored to the desired application.

Further, Xen gives you the ability to run whatever configuration you happen to need at a given time. For example, the web developer who wants to test a new page against different versions of Microsoft's Internet Explorer doesn't have to maintain a farm of Windows boxes, each with different Windows versions, different patch levels, and different versions of Internet Explorer. Instead, it's possible to just keep different OS images on the hard drive and start them as needed.

Xen's Limitations All right, we're getting carried away. Xen's not perfect, nor is it any sort of computing panacea. It has both disadvantages and limitations.

Xen's main disadvantage is that it only works with operating systems that have been specifically modified to support it. (But note that unmodified guest OSs are possible with sufficiently advanced hardware. We'll talk about that later, in Chapter12 Chapter12.) Xen's also more work to set up than a pure software emulator, requiring the user to work entirely in a guest domain (albeit a special, privileged guest domain) rather than simply starting an external emulation program as desired.

Additionally, the state of the Xen doc.u.mentation is pretty dreadful. (That's what we're here for, you might say.) People are, of course, working on it, but everyone knows it's more fun to write code than to doc.u.ment it. Also, Xen's under such active development that much of the doc.u.mentation that exists is out of date.

These are significant disadvantages, but they aren't so bad that you should be discouraged from running Xen.

Finally, though there are also some situations in which Xen-and virtualization itself-simply isn't useful. Xen isn't especially useful to people with a constant, CPU-limited workload, for example. It's not great in large server farms, where individual nodes are already scaled to their jobs. In these situations, Xen is probably not what you want, although the developers (and the open source community) are working on compelling features even for environments like these.

But, in the end, it's not Xen itself that's interesting-it's what you can use it for.

So, Why Should I Use Xen?

The short answer is, because it will make your life easier because it will make your life easier. Don't trust a piece of software? Spin off a virtual machine and see how you like it. Need to test a network app? Start up a few machines and see how well they talk to each other. Have a cl.u.s.ter that you want to test some new software on but can't afford a second "test" cl.u.s.ter? Xen offers a solution. Want decent snapshot backups? Xen could be your answer, with its ability to pause and back up a running machine within, literally, seconds. Need to provide hosting for dozens of users, each of whom wants complete authority to mess with their configuration? Well, that's what we do, and Xen's the way we do it. (The astute reader might notice in our writing a certain bias toward that last application. That's why.) On a more fundamental level, Xen lets you take a machine, stop it, send it somewhere else, and resume it at will. It's one less thing to think about-suddenly the hardware is no longer important. A good thing for both users and administrators!

Finally, there's one last good reason to run Xen, one that's so big and mundane it often gets ignored: Xen is simply cheaper than running multiple boxes. CPU usage in data centers ranges between 5 percent and 40 percent-a fairly unimpressive figure.[6] Xen lets you put some of those unused cycles to use, without sacrificing reliability, performance, or scalability. Xen lets you put some of those unused cycles to use, without sacrificing reliability, performance, or scalability.

Unlike the virtualization technologies of a few decades ago, Xen virtualizes cheap commodity hardware; this might not make sense at first, until you realize that much of the market is very very price sensitive, and power is becoming quite expensive. It's much cheaper to run one big dual quad-core rig than it is to run eight single-core boxes, and with Xen, you can easily split that quad-core system into individual systems. price sensitive, and power is becoming quite expensive. It's much cheaper to run one big dual quad-core rig than it is to run eight single-core boxes, and with Xen, you can easily split that quad-core system into individual systems.

[4] In fact, Xen uses QEMU extensively, as we'll see. In fact, Xen uses QEMU extensively, as we'll see.

[5] This is sort of like cell phones. People use them, not as a subst.i.tute for landlines, but as a subst.i.tute for traditional planning. This is sort of like cell phones. People use them, not as a subst.i.tute for landlines, but as a subst.i.tute for traditional planning.

[6] This is a generally held belief, but one oft-cited source is the presentation "Virtualization: Taking Charge of Your Servers" by Thomas Bittman. This is a generally held belief, but one oft-cited source is the presentation "Virtualization: Taking Charge of Your Servers" by Thomas Bittman.

Overview of the Book All right, enough hype. Now for nuts and bolts.

We've organized this book (mostly) alternating between theoretical and practical discussion. In our experience, an admin needs both practical experience and a firm theoretical grounding to effectively solve problems, and that's what we aim to provide.

Chapter1 is an overview of Xen and virtualization technologies in general. We try to outline how Xen works, what distinguishes it from other virtualization packages, and why you might (or might not) want to use it. This one is theory-intensive. is an overview of Xen and virtualization technologies in general. We try to outline how Xen works, what distinguishes it from other virtualization packages, and why you might (or might not) want to use it. This one is theory-intensive.

Chapter2 is a step-by-step quick start based on the rationale that there's no subst.i.tute for experience. We install Xen from base principles on a CentOS system. is a step-by-step quick start based on the rationale that there's no subst.i.tute for experience. We install Xen from base principles on a CentOS system.

Chapter3 describes manually creating virtual machine images to use with Xen. describes manually creating virtual machine images to use with Xen.

Chapter4 covers storage. It sounds kind of mundane, but storage is actually a vital part of virtualization-if storage is tied to a particular machine or hardware configuration, then many of Xen's coolest features won't work. We cover various storage options, laying the groundwork for subsequent discussion of migration and snapshots. covers storage. It sounds kind of mundane, but storage is actually a vital part of virtualization-if storage is tied to a particular machine or hardware configuration, then many of Xen's coolest features won't work. We cover various storage options, laying the groundwork for subsequent discussion of migration and snapshots.

We talk about networking in Chapter5 Chapter5-how to set it up and what options you have when doing so. Both this chapter and the previous focus a bit more on theory.

Chapter6 is about a couple of popular packaged frontends that can be used with the open source Xen hypervisor to automate the routine drudgery of VM administration. We also talk about scripting Xen, if you'd rather build your own frontend. is about a couple of popular packaged frontends that can be used with the open source Xen hypervisor to automate the routine drudgery of VM administration. We also talk about scripting Xen, if you'd rather build your own frontend.

Chapter7 goes back to the practical case studies to talk about Xen for shared hosting. It's one of the big applications that's driving early adoption of Xen, and we've got a lot of experience doing it. goes back to the practical case studies to talk about Xen for shared hosting. It's one of the big applications that's driving early adoption of Xen, and we've got a lot of experience doing it.

Moving on from shared hosting, in Chapter8 Chapter8 we discuss possible alternatives to Linux, both as a "host" and "guest" OS. we discuss possible alternatives to Linux, both as a "host" and "guest" OS.

In Chapter9 Chapter9 we describe migration, both in theory and practice. we describe migration, both in theory and practice.

Chapter10 is about performance a.n.a.lysis with Xen. We discuss Xen's robust support in this area, which doesn't seem to get mentioned nearly as often as it deserves. is about performance a.n.a.lysis with Xen. We discuss Xen's robust support in this area, which doesn't seem to get mentioned nearly as often as it deserves.

With Chapter11 Chapter11 we diverge a bit to cover the commercial product that XenSource (now a division of Citrix) has built around Xen. we diverge a bit to cover the commercial product that XenSource (now a division of Citrix) has built around Xen.

Chapter12 is about Xen's HVM support-that is to say, the hardware virtualization supported by Intel's and AMD's newest processors. is about Xen's HVM support-that is to say, the hardware virtualization supported by Intel's and AMD's newest processors.

Chapter13 covers Windows. We talk about using it with Xen, making it play nicely, how you can expect to access it, and what you might expect to do with it once you've got it working. covers Windows. We talk about using it with Xen, making it play nicely, how you can expect to access it, and what you might expect to do with it once you've got it working.

Chapter14 is a collection of extremely practical tips for Xen admins. is a collection of extremely practical tips for Xen admins.

Chapter15 is a troubleshooting chapter-a collection of problems that we've run into and how we've solved them. is a troubleshooting chapter-a collection of problems that we've run into and how we've solved them.

We've also included appendixes on Xen's domain configuration files and xm xm's syntax.

But I Am Impatient!

If you're really impatient to get started with Xen, skip to Chapter2 Chapter2 and follow our step-by-step instructions. Then skim the rest of the book as the fancy strikes you. and follow our step-by-step instructions. Then skim the rest of the book as the fancy strikes you.

If you're planning to deploy Xen as a service provider, we suggest following the steps in Chapter2 Chapter2, then reading Chapters Chapter3 Chapter3, Chapter4 Chapter4, Chapter5 Chapter5, Chapter6 Chapter6, Chapter7 Chapter7, Chapter8 Chapter8, and probably 11, and finally just reading the whole book.

For those of you who are contemplating a large deployment of Xen, you'll probably be most interested in Chapters Chapter3 Chapter3, Chapter4 Chapter4, Chapter5 Chapter5, Chapter6 Chapter6, and Chapter7 Chapter7, with an excursion to 13 to consider the commercial XenSource product. But again, we think we've put useful information throughout the book.

NoteWe've tried to keep this book as distribution- and version-independent as possible, except in the tutorial sections, where we try to be extremely specific and detailed, and in the distro-specific notes, which are necessarily, er, distro-specific.

Often we will get carried away and make some ridiculous broad generalization, like "only an idiot would use Linux as an NFS server."[7] Where reasonable, we've tried to add footnotes that qualify and temper the occasionally strident claims we make. Where reasonable, we've tried to add footnotes that qualify and temper the occasionally strident claims we make.

[7] Actually, we've seen morons and imbeciles do this too. Actually, we've seen morons and imbeciles do this too.